She was the illegitimate daughter of the painters Duncan Grant and Vanessa Bell, niece of Virginia Woolf, and was a member of the Bloomsbury Group. She had two half-brothers, the poet Julian Bell, who was killed during the Spanish Civil War in 1937; and art historian Quentin Bell.
Her mother's husband, Clive Bell, was not her biological father, but was fully supportive of her mother's love affair with Grant, and willingly allowed Angelica to bear his name and to regard him as her father in order that his conservative family not disinherit her. She was not told of her true parentage until she was 17, although she had grown up living with Grant and her mother at Charleston Farmhouse in Sussex/England, which her mother had rented and shared with other members of the Bloomsbury Group. The farmhouse is now a museum.
Angelica Garnett is the author of a memoir, Deceived with Kindness, which focuses on her relationship with both of her biological parents. Its somewhat bitter view of both Bell and Grant has proven controversial. The memoir was awarded the J. R. Ackerley Prize for Autobiography in 1985.
She married David Garnett, the former lover of her biological father, Duncan Grant, in 1942, but they later separated. They had four daughters: Amaryllis Virginia (1943-1973), an actress; Henrietta Garnett, a writer and custodian of the family legacy (b. 1945); and twins Nerissa Stephen (1946-2004), called Nel, a painter, photographer and ceramics artist; and Frances, called Fanny (also b. 1946).