He was born on 13 November 1906 at New Oxley, Bushbury, near Wolverhampton, and died on 24 January 1992 at Iffley, near Oxford. His father was Isaac Saredon Sparrow, a barrister who had inherited wealth through the family business as prominent Midland ironmasters. John Sparrow was the eldest of five children, born an illegitimate child of his mother, Margaret Sparrow (née Macgregor).

John Sparrow briefly attended the junior house of Wolverhampton Grammar School, but was soon moved to Brockhurst at Church Stretton in Shropshire as a boarder. Not long after, in September 1916, when he was nearly ten, he was sent to a preparatory school called The Old Hall at Wellington in Shropshire. His formal education followed at Winchester College and New College, Oxford.

He was elected Fellow of All Souls College, Oxford (1929), winning a prize fellowship the same year HLA Hart sat (unsuccessfully) for the first time.He became Warden of All Souls (1952-77) in an election in which he famously defeated A. L. Rowse. He was also Fellow of Winchester (1951-81) and Honorary Fellow of New College (1956-1992). In Oxford he was well-known as a book-collector and bibliographer, and became President of the Oxford University Society of Bibliophiles, in which role he influenced a generation of Oxford bookmen and women.

He famously wrote an article for Encounter on Lady Chatterley's Lover (after the obscenity trial) arguing that the acquittal was wrong, as the novel promoted the illegal practice of sodomy.

Sparrow was called to the Bar at the Middle Temple (1931, Honorary Bencher 1952), practising in the Chancery Division (1931-39, 1946-51).

He published more than fifty books and essays on topics including epigraphy, painting, and Latin and English poetry.

One of his best known remarks was describing the dog as "That indefatigable and unsavoury engine of pollution".