Gray was a graduate of Boston Latin School. From there, he went on to Harvard University, where he earned his bachelor of arts degree in 1859, and Harvard Law School, where he earned his law degree in 1861. He was admitted to the bar in 1862, and thereafter served a tour in the Union Army in the American Civil War.

In 1865, after the end of the Civil War, Gray established his law practice in Boston, Massachusetts, which would eventually evolve into the modern firm of Ropes and Gray. In 1869, he began teaching at Harvard Law School, first as a lecturer, and became a full professor in 1875. In 1883, he was named Royall Professor of Law (a chair named for Isaac Royall, Jr.), a position he would hold for 20 years. He received honorary Doctor of Laws degrees from Yale University in 1894, and from Harvard in 1895.

Two years after retiring from teaching, he died at Boston, Massachusetts on February 25, 1915.

Gray wrote two books on future interests, Restraints on the Alienation of Property (1883), and The Rule against Perpetuities (1886). His best known work is his survey of the common law, The Nature and Sources of the Law (1909). Gray's writings were so influential that they are still used in American law schools and cited in law journals to this day.