Born in New York City, John Hart Ely graduated from Princeton University in 1960 and Yale Law School in 1963. As a summer clerk at Arnold, Fortas, & Porter, a Washington, D.C. law firm, he assisted Abe Fortas in the landmark case of Gideon v. Wainwright, writing a first draft of a brief on behalf of Clarence Earl Gideon, a Florida drifter who had been tried and convicted without a lawyer.

In the fall of 1963, Ely trained with Company A of the U.S. Army's Military Police School at Fort Gordon, Georgia. Ely served as the youngest staff member of the Warren Commission, which investigated the assassination of President John F. Kennedy. He went on to clerk for Chief Justice Earl Warren, whom he considered a hero, and to whom he dedicated his landmark book, Democracy and Distrust (1980). As a clerk for Warren, Ely authored the landmark decision Hanna v. Plumer.

Joining the faculty of Yale Law School in 1968, and moving to Harvard Law School in 1973, Ely wrote several influential law review articles, including a highly critical analysis of the Supreme Court's decision in Roe v. Wade in an article entitled "The Wages of Crying Wolf," published in the Yale Law Journal, wherein he argued that the Court's decision protecting abortion rights was wrong "because it is not constitutional law and gives almost no sense of an obligation to try to be."

Ely's most notable work was his 1980 book Democracy and Distrust, which ranks as one of the most influential works about constitutional law ever written. In it, he argues against "interpretivism" of which Hugo Black was an exponent, "originalism" advanced by Robert Bork, and "textualism" advanced by Antonin Scalia, by contending that "strict construction" fails to do justice to the open texture of many of the Constitution's provisions; at the same time, though, he maintains that the notion that judges may infer broad moral rights and values from the Constitution is radically undemocratic whether the "moralism" of Ronald Dworkin or the libertarian Richard Epstein. Instead, Ely argued that the Supreme Court should interpret the Constitution so as to reinforce democratic processes and popular self-government, by ensuring equal representation in the political process (as in the Court's decision in Baker v. Carr [1961]). He argues ejusdem generis that the Constitution's unenumerated rights (such as the 9th Amendment or the Privileges or Immunities clause of the 14th Amendment) are procedural in nature rather than substantive, thus protecting rights to democratic processes but not rights of a substantive nature. Justice Stone's Footnote Four from United States v. Carolene Products Co. (1938) is a chief inspiration for Ely's theory of judicial review.

He went on to serve as dean of Stanford Law School from 1982 to 1987, and remained on the faculty until 1996. Prompted by his love of scuba diving, he visited the University of Miami School of Law in 1996. Discovering he liked the city and the faculty, he chose to stay and became the first holder of the law school's first endowed chair, named after Richard A. Hausler. Ely was there when he died of cancer, aged 64.

He was married to Gisela Cardonne Ely, who is a state judge in Miami, Florida.