Stephen was born at Kensington Gore in London, the brother of James Fitzjames Stephen and son of Sir James Stephen. His family had belonged to the Clapham Sect, the early 19th century group of mainly evangelical Christian social reformers. At his father's house he saw a good deal of the Macaulays, James Spedding, Sir Henry Taylor and Nassau Senior. After studying at Eton College, King's College London and Trinity Hall, Cambridge, where he graduated B.A. (20th wrangler) in 1854 and M.A. in 1857, Stephen remained for several years a fellow and tutor of his college. He recounted some of his experiences in a chapter in his
Life of Fawcett as well as in some less formal Sketches from
Cambridge: By a Don (1865). These sketches were reprinted from the
Pall Mall Gazette, to the proprietor of which, George Smith, he had been introduced by his brother. It was at Smith's house at Hampstead that Stephen met his first wife, Harriet Marian (1840—1875), daughter of William Makepeace Thackeray, with whom he had a daughter, Laura Makepeace Stephen (1870—1945); after her death he married Julia Prinsep Jackson (1846—1895), widow of Herbert Duckworth. With her he had four children:
- Vanessa (1879—1961) married Clive Bell
- Thoby (1880—1906)
- Virginia (1882—1941) married Leonard Woolf
- Adrian (1883—1948)
In the 1850s, Stephen and his brother James Fitzjames Stephen were invited by Frederick Denison Maurice to lecture at The Working Men's College. Leslie Stephen became a member of the College's governing College Corporation.