Ashbee was born in Southwark, London. He was by occupation a textile trader, the senior partner in the London branch of the firm of Charles Lavy & Co. He travelled extensively during his life, including Europe, Japan, and San Francisco, collaborating with Alexander Graham on Travels in Tunisia, published in 1887. He was an avid book collector, with perhaps the world's most extensive collections of Cervantes and erotica.

Ashbee was a part of a loose intellectual fraternity of English gentlemen who discussed sexual matters with a freedom that was at odds with Victorian mores; this fraternity included Richard Francis Burton, Richard Monckton Milnes, Algernon Charles Swinburne, and others. He also amassed thousands of volumes of pornography in several languages. He wrote on sex under the pseudonyms "Fraxinus" (Ash) and "Apis" (Bee), and sometimes combined them as "Pisanus Fraxi".

Ashbee's will left his entire collection to the British Museum, with the condition that the erotic works had to be accepted along with the conventional items. Because the trustees wanted the materials related to Cervantes, they decided to accept the bequest. The trustees exploited a loophole to destroy some of the erotica, although some of the works are in the British Library, including a work by William Simpson Potter, forming the core of the Private Case.

Ashbee married Elisabeth Lavy in Hamburg, Germany in 1862. They had one son, Charles (the designer Charles Robert Ashbee, born 1863), and three daughters. His family life grew unhappier as he aged. As he became more conservative, his family followed the progressive movement of the era. "The 'excessive education' of his daughters irritated him, his Jewish wife's pro-suffragism infuriated him, and he became tragically estranged from his socialist homosexual son, Charles". Henry and Elisabeth separated in 1893.

Ashbee's most famous works were his three bibliographies of erotic works:

• Index Librorum Prohibitorum: being Notes Bio- Biblio- Icono- graphical and Critical, on Curious and Uncommon Books. London, privately printed, 1877. (The name is a reference to the Catholic Church's list of banned books "Index Librorum Prohibitorum").
• Centuria Librorum Absconditorum: being Notes Bio- Icono- graphical and Critical, on Curious and Uncommon Books. London, privately printed, 1879.
• Catena Librorum Tacendorum: being Notes Bio- Icono- graphical and Critical, on Curious and Uncommon Books. London, privately printed, 1885

The Index was arranged by author, the Centuria and Catena by subject. Ashbee includes plot summaries of the works listed, with liberal quotations. Of particular note are the 300 pages of the "Centuria" devoted to anti-Catholic pornography. Initially only 250 copies of each volume were printed.

Ashbee is also suspected to be "Walter", the author of My Secret Life, a lengthy sexual memoir of a Victorian gentleman. Gershon Legman was the first to link "Walter" and Ashbee in his introduction to the 1962 reprints of Ashbee's bibliographies; the 1966 Grove Press edition of My Secret Life included an expanded version of that essay.

A character based on him is central to Sarah Waters's award-winning novel Fingersmith: a man obsessively collecting and indexing pornography and works about human sexuality, in an atmosphere of oppressive Victorian hypocrisy.