Mahon was born into a wealthy Anglo-Irish family and was heir to the Guinness Mahon banking fortune. After attending Eton, he enrolled at Christ Church, Oxford University, where he received an M.A. He spent a year working at the Ashmolean Museum under the supervision of Kenneth Clark, then in 1933 he enrolled at the Courtauld Institute of Art in London. It was here that he was introduced to Italian Baroque painting in a series of lectures by Nikolaus Pevsner, who also gave him private tuition. He bought his first artwork, Guercino's Jacob Blessing the Sons of Joseph in 1934 in Paris for £120. He subsequently met art historian Otto Kurz, whom he frequently used as an Italian translator, in the late 1930s, and together they travelled to Russia to study Italian masters.

Mahon's Studies in Seicento Art and Theory, a series of essays promoting Italian art of the 17th century, was published in 1947. In the 1950s, he became a trustee of the National Gallery. In the 1960s, Mahon and Sir Anthony Blunt became embroiled in a nasty, public feud over the iconography of paintings of Nicolas Poussin, a subject where both were recognized experts who had published extensively on Poussin.

In the 1990s, Mahon donated his entire art collection to various museums in the United Kingdom, the Pinacoteca Nazionale in Bologna and the National Gallery of Ireland in Dublin. In a complex legal arrangement, a number of these paintings are on permanent loan from The Art Fund, which owns them. There are two conditions to the loans: that the museums in question never deaccession any of their works, or charge for admission; if either condition is broken the paintings are withdrawn from the museums. Mahon was knighted in 1986, was made a Companion of Honour in 2002 for his services to art, and has received honorary doctorates from the universities of Newcastle, Oxford, Rome, and Bologna.

In December 2007, a painting Mahon bought for £50,400 the previous year (and which was considered to be the work of an anonymous follower of Caravaggio) was authenticated by him as a true Caravaggio. It is an early version of the painting The Cardsharps. The painting is thought to be worth up to £50 million.