Hollander was born Xaviera de Vries in Soerabaja, Dutch East Indies (now Indonesia), to a Dutch Jewish father and a French-German mother. She spent the first two years of her life in a Japanese internment camp.

In her early 20s, she left Amsterdam for Johannesburg, South Africa, where her stepsister lived. There she met and became engaged to Carl Gordon, an American economist. When the engagement was broken off, she left South Africa for New York.

In 1968, she left her job as the secretary of the Dutch consulate in Manhattan to become a call girl, where she made $1,000 a night. A year later she opened her own brothel called the Vertical Whorehouse and soon became New York City's leading madam. In 1971, she was arrested for prostitution by New York police and was forced to leave the U.S.

In 1971 Hollander published a memoir, The Happy Hooker: My Own Story, that she co-authored with Robin Moore and Yvonne Dunleavy. The book was notable for its frankness by the standards of the time, and is considered a landmark of positive writing about sex. Hollander details in the book her life as a liberal and open-minded girl. She states that during the start of her career she did not ask for cash in exchange for sex, but her partners voluntarily gave her money and other presents.

Hollander has since written a number of other books and produced plays in Amsterdam. Her latest book, Child No More, is the heartfelt story of losing her mother. For 35 years she wrote an advice column for Penthouse magazine called Call Me Madam. For several years in the 1970s Hollander lived in Toronto, where she married a Canadian antique dealer and was a regular fixture in the downtown core.

In the early 1970s, she recorded a primarily spoken-word album titled Xaviera! for the Canadian GRT Records label (GRT 9230-1033), on which she discussed her philosophy regarding sex and prostitution, sang a cover version of The Beatles song, "Michelle", and recorded several simulated sexual encounters, including an example of phone sex, a threesome, and a celebrity encounter with guest "vocal" by Ronnie Hawkins. In 1975 she starred in the somewhat autobiographical My Pleasure is My Business. For the past seven years she has been operating Xaviera's Happy House, a bed and breakfast within her Amsterdam home.

Hollander says she "turned gay" around 1997, establishing a long-term relationship with a woman named Dia. In January 2007 she married a Dutch man 10 years her junior, Philip de Haan, in Amsterdam.

Hollander has been portrayed in film:

• The Happy Hooker (1975) by Lynn Redgrave
• The Happy Hooker Goes to Washington (1977) by Joey Heatherton
• "The Best Part of a Man" 1975 by Artistae Stiftung

And she appears in:

Non-fiction

*