Levey was born in Wimbledon, London, and grew up in Leigh-on-Sea, Essex. He attended The Oratory School, a Catholic boarding school near Reading. He was called up for National Service in 1945 and served it largely in Egypt. After demobilisation in 1948 Levey went to Exeter College, Oxford to read English; he graduated with first class honours after only two years' study.

In 1951 Levey joined the National Gallery as assistant to the Keeper, Sir Martin Davies. He combined administrative duties with scholarly work, producing his first catalogue, on the Gallery's 18th-century Italian paintings, in 1956. In the 1960s, affordable art books with colour reproductions for the general reader began to appear, and Levey was commissioned to write an overview of Western painting for Thames & Hudson's World of Art series. The resulting book, A Concise History of Painting: From Giotto to Cézanne (1962), remains a classic overview of European art history from the introduction of perspective in Italy to the beginnings of modern art at the start of the 20th century.

From 1963 to 1964 Levey was Slade Professor of Fine Art at Cambridge University; his lectures were published as From Rococo to Revolution in 1966. The Early Renaissance, written a year later, is considered another milestone in popular art publishing, and was the first non-fiction work to win the Hawthornden Prize for Literature. Levey became deputy Keeper of the National Gallery in 1966, Keeper in 1968, and Director in 1973. He was knighted in 1981.

He relinquished his directorship to care for his wife, the novelist and critic Brigid Brophy, after she was diagnosed with multiple sclerosis in 1985; the disease ultimately claimed her life. Brophy and Levey were married in 1954.

Levey was a Distinguished Supporter of the British Humanist Association.