Yerushalmi was born in the Bronx, New York City on May 20, 1932, to Yiddish-speaking Russian parents who had immigrated to the United States. His father was a Hebrew teacher.

In 1953, Yerushalmi received his bachelor’s degree from Yeshiva University. Later, in 1957 he was ordained as a Rabbi. He received a doctorate from Columbia University in 1966. Salo Baron was his dissertation director. From the time of receiving his doctorate until his appointment to the Columbia faculty, Yerushalmi taught at Harvard University, where he was Jacob E. Safra Professor of Jewish History and Sephardic Civilization and chairman of the Department of Near Easter Languages and Civilizations.

Professor Yerushalmi died of emphysema on December 8, 2009.