Abish was born in Vienna, Austria to Adolph and Frieda (Rubin). At a young age, his family fled from the Nazis, traveling first to Italy and Nice before settling in Shanghai from 1940 to 1949. In 1949, they moved to Israel, where Abish served in the army and developed an interest in writing. He moved to the United States in 1957 and became an American citizen in 1960. Since 1975, Abish has taught at several eastern universities and colleges. Abish received the PEN/Faulkner Award for Fiction in 1981 for his book How German Is It. He has also received a Guggenheim Fellowship and a MacArthur Fellowship, and sits on the contributing editorial board of the literary journal Conjunctions.

Abish is married to Cecile Gelb who is a photographer and sculptor.

He served on the board of International PEN from 1982-1988. He was on the board of governors for the New York Foundation for the Arts. He currently serves as a fellow for American Academy of Arts and Sciences.

He has worked/taught at Empire State College, Wheaton College, University at Buffalo, The State University of New York, Columbia University, Brown University, Yale University, and Cooper Union.

Abish's work shows both imaginative and experimental elements. In Alphabetical Africa, for instance, the first chapter consists entirely of words beginning with the letter "A." In the second chapter, words beginning with "A" and "B" appear, and so on through the alphabet. In the Future Perfect is a collection of short stories where words are juxtaposed in unusual patterns.