Garnett was born in Brighton as the only child of Edward Garnett and Russian translator Constance Garnett. As a conscientious objector in the First World War, he worked on fruit farms in Suffolk and Sussex with his lover, Duncan Grant.

A prominent member of the Bloomsbury Group, Garnett received literary recognition when his novel Lady into Fox was awarded the 1922 James Tait Black Memorial Prize for fiction. He ran a bookshop near the British Museum with Francis Birrell during the 1920s. He also founded (with Francis Meynell) the Nonesuch Press. He wrote the novel Aspects of Love (1955), on which the later Andrew Lloyd-Webber musical was based.

His first wife was illustrator Rachel "Ray" Marshall (1891—1940), sister of Frances Partridge whose woodcuts appear in some of his books. He and Ray had two sons, but she died relatively young of breast cancer.

Although Garnett was primarily heterosexual, he had affairs with Francis Birrell and Duncan Grant. He was present at the birth of Grant's daughter, Angelica Garnett (née Bell), on 25 December 1918, and wrote to a friend shortly afterwards, "I think of marrying it. When she is 20, I shall be 46 — will it be scandalous?". When Angelica was in her early twenties, they did marry (on 8 May 1942), to the horror of her parents.

They had four daughters (Amaryllis, Henrietta, and twins Nerissa and Frances), but later separated. Their eldest daughter Amaryllis Garnett (1943—1973) was an actress. Henrietta Garnett, their second daughter, eventually married Burgo Partridge, her father's nephew by his first wife Ray; she oversees the legacies of both David Garnett and Duncan Grant.

After his separation from Angelica, Garnett moved to France and lived at the Chateau de Charry, Montcuq (near Cahors), where he died in 1981.