Oppenheimer was born in Yonkers, New York, attended Cornell University for one year in 1948, spent less than one semester at the University of Chicago, and in 1950 enrolled at Black Mountain College in North Carolina. At Black Mountain, he studied with Paul Goodman and poet Charles Olson, became friends with Fielding Dawson and Ed Dorn, and worked in the school's print shop.

In his earliest poetry, Oppenheimer shows clearly the influence of William Carlos Williams, but he soon developed his own style. While at Black Mountain, Oppenheimer met and married his first wife, Rena Furlong. He left the school in January 1953 without taking a degree, eventually settling in New York and working in a print shop while continuing to write poetry.

His poetry has been collected in the following volumes: Robert J. Bertholf , ed. Collected Later Poems of Joel Oppenheimer ed. and intro. with eleven drawings by John Dobbs, The Poetry Collection, 1997; and Names & Local Habitations (Selected Earlier Poems 1951-1972) ed. Jonathan Williams, The Jargon Society, 1988.

He also published two nonfiction works, The Wrong Season, Bobbs-Merrill 1973, about the New York Mets, and Marilyn Lives, Delilah, 1984, on Marilyn Monroe. Drawing from Life, posthumously published in 1997, gathered 92 columns written for the Village Voice. Library Journal wrote that Drawing from Life "emphasizes several favorite themes: baseball, politics, and the role of the changing seasons in our lives".

A biography by Lyman Gilmore, Don’t Touch the Poet: The Life and Times of Joel Oppenheimer was published by Talisman Press in 1998.