Spence was born on 28 April 1699, at Kingsclere, Hampshire, the son of Joseph (Rector of Winnal in Winchester and Precentor of Winchester Cathedral) and Mirabella (née Collier, granddaughter of Sir Thomas Lunsford).

In 1709, Spence attended school in Mortimer, near his birthplace, and later attended Eton College and then Winchester College. Spence matriculated to Magdalen Hall, Oxford on 11 April 1717, but did not go up until admitted as scholar or probationary fellow at New College on 22 April 1720. On 30 April 1722, he received a full fellowship, taking his Bachelor of Arts degree on 9 March 1724 and Master of Arts on 2 November 1727. Spence was ordained in the Oxford diocese on 5 June 1726. Early literary friends of Spence included fellow Wykehamists Robert Lowth, Christopher Pitt, and Edward Young.

Spence wrote an essay on Alexander Pope's translation of Homer's Odyssey in 1726 which gave rise to a friendship between Spence and Pope. In 1727 Pope published his comments on the second part of Spence's essay.

Spence was elected as the Oxford Professor of Poetry on 11 July 1728, holding the post for maximum term allowed (ten years). In the same month he was given the New College living of Birchanger, Essex, but continued to live in Oxford when not travelling abroad.

Recommended by Alexander Pope, Spence became a travelling companion of Charles Sackville, 6th Earl of Dorset and 1st Earl of Middlesex, on a Grand Tour lasting from December 1730 to July 1733. In 1731, while Spence was abroad, his Account of Stephen Duck was published. On his return to Oxford he lectured and made mundane contributions to poetry collections marking royal weddings, births, and deaths. In 1736 he published An Account of Lord Buckhurst and an edition of Gorboduc.

Spence was a companion to John Morley Trevor in his tour of the Netherlands, Flanders, and France between May 1737 and February 1738, and between September 1739 and November 1741 travelled in Italy with Henry Pelham-Clinton. While touring, Spence frequently wrote letters to his mother, which he later edited for publication but never published.

On 4 June 1742 Spence became Regius Professor of Modern History and exchanged Birchanger for Great Horwood, Buckinghamshire, which he visited each year and distributed charity. Around 1742 he gave up his fellowship at New College and settled with his mother in London and often visited Alexander Pope.

After much work, Spence published Polymetis as an illustrated folio in February 1747. He had first thought of such a work during his first visit to Italy and used much of the material he had collected there, intending to demonstrate the relationship between the works of ancient artists and of Roman poets. The work was criticsed by Gotthold Ephraim Lessing in his 1766 work, Laokoon, which subsequently adversely affected its standing.

In 1748 Henry Pelham-Clinton gave Spence the use of a house he owned at Byfleet, Surrey, which Spence and his mother moved into. Having designed Birchanger's garden along the lines of Pope's garden in Twickenham, and planting extensively at Great Horwood, as well as designing gardens and making notes for friends London, Spence further explored his interest in landscape gardening at Byfleet - developing the 30 acre estate as a ferme ornée. Though making extensive notes and translating Jean Denis Attiret's account of the Emperor of China's gardens, Spence left his gardening treatise, Tempe, unfinished. His translation of Attiret was published in 1752 under the pen name of "Sir Harry Beaumont", which he also used for Crito, published in 1752, and Moralities, published in 1753.

In 1752-3 Spence published accounts of Robert Hill (tailor) and Thomas Blacklock, and promoted a subscription edition of Blacklock's poems. In 1758, Spence travelled with James Dodsley to visit Blacklock in Scotland, and en-route, visited William Shenstone at The Leasowes. Spence had his work Parallel in the Manner of Plutarch printed at Horace Walpole's Strawberry Hill Press in 1758, in order to raise money for Hill. In 1753 Spence provided notes for an edition of Virgil written by Joseph Warton, and in 1768 edited the Remarks on Virgil by Edward Holdsworth, the authors being friends and Old Wykehamists.

Spence was installed as a prebendary of Durham Cathedral on 24 May 1754, a posting on the generosity of the Bishop of Durham, Richard Trevor. Spence continued to live at Byfleet and spent more than the minimum time required for his prebendal duties of three weeks' residence at Durham. In Durham, he not only began to improve the garden of his prebendal estate, but also those of his neighbours, the bishop and the Earl of Darlington, Henry Vane, sometimes spending weeks at a time travelling to landscaped gardens where his advice had been sought.

Spence suffered a mild stroke during his annual journey north in June 1766, and made his will at Sedgefield, Durham, on 4 August 1766. On 24 March 1767, Spence sold the copyrights, including those on his unpublished works, to James Dodsley for £100. On 20 August 1768 Spence was found lying face down in the shallow ornamental waters of his garden at Byfleet and was buried four days later at St Mary's, Byfleet.

Spence's published works works include An Essay on Pope's Odyssey (1726) and Polymetis (1747). Spence's unpublished works include his edition of travelling letters, notes for a gardening treatise, notes for a biographical history of English poetry, and his anecdotes, which include tales about Alexander Pope and other literary figures such as John Arbuthnot, Isaac Newton, and Stephen Duck.