Schiavelli was born in Brooklyn, New York, to a (Sicilian) Italian-American family. He attended Bishop Loughlin Memorial High School as a teen. He studied acting through the theatre programme at New York University. He began performing on stage in the 1960s.

Schiavelli's first movie role occurred in Milo? Forman's 1971 production Taking Off, in which he played a counselor who taught parents of runaway teens to smoke marijuana in order to better understand their children's experiences. Schiavelli's aptitude and distinctive angular appearance soon provided him with a steady stream of supporting roles, often in Milo? Forman films, namely One Flew Over the Cuckoo's Nest, Amadeus, The People vs. Larry Flynt, Valmont, and the 1999 biopic Man on the Moon. Schiavelli's role as a subway spirit in the 1990 romantic fantasy Ghost won him much critical acclaim.

Schiavelli also played Mr. Vargas the biology teacher in the 1982 hit comedy Fast Times at Ridgemont High, a role he reprised in the 1986 television spin-off Fast Times. He was cast in a similar role in the cult hit Better Off Dead in which he played Mr. Kerber, a strangely popular geometry teacher.

In 1987, he starred alongside Tim Conway in the short film comedy, Dorf on Golf, and Dorf and the First Games of Mount Olympus in 1988. In 1992, he played in Tim Burton's Batman Returns as the "Organ Grinder", one of the Penguin's henchmen. He appeared as another villain in the James Bond film Tomorrow Never Dies (1997), as a silent monk in The Frisco Kid (1979), and as John O'Connor, one of the evil Red Lectroids in the 1984 cult classic The Adventures of Buckaroo Banzai Across the 8th Dimension. In 1997, he was named one of America's best character actors by Vanity Fair magazine. In 2002, he played a children's television show host turned heroin addict named Buggy Ding Dong in Death To Smoochy.

His first television role came in 1972 as Peter Panama in The Corner Bar, the first sustained portrayal of a gay character on American television. His other television credits include Buffy the Vampire Slayer, and Taxi as the priest who marries Latka and Simka. He also appeared in the The Next Generation episode "The Arsenal of Freedom" as a holographic salesman and an uncredited role in an episode of Punky Brewster. Schiavelli also wrote a number of cookbooks and food articles for various magazines and newspapers. He received a James Beard Foundation Journalism Award in 2001 and was nominated on several other occasions.

Schiavelli served as honorary co-chair of the National Marfan Foundation, an organisation which serves those affected by Marfan syndrome, from which Schiavelli suffered.

He was married to actress Allyce Beasley from 1985 until their 1988 divorce and he once guest-starred as the love interest of Beasley's character on an episode of Moonlighting. Their son Andrea Schiavelli was born in 1987. In 1992 Schiavelli married American harpist Carol Mukhalian.

Schiavelli succumbed to lung cancer on December 26, 2005 at the age of 57.He died at his home in Polizzi Generosa, Italy, the Sicilian town from which his grandfather emigrated and about which he wrote in his 2002 book Many Beautiful Things: Stories and Recipes from Polizzi Generosa (ISBN 0-7432-1528-1).