Gelb attended New Rochelle High School, in New Rochelle, New York, graduating in 1955. He received a B.A. from Tufts University in 1959, and an M.A. in 1961 and Ph.D. in 1964 from Harvard University. From 1964-1967 he was Assistant Professor of Government at Wesleyan University. He married his wife, Judith Cohen, on 2 August 1959 and lives in New York City. They have three children.

Gelb was director of Policy Planning and Arms Control for International Security Affairs at the Department of Defense from 1967 to 1969, winning the Pentagon's highest award, the Distinguished Service Award.Robert McNamara appointed Gelb as director of the project that produced the controversial Pentagon Papers on the Vietnam War.

He was diplomatic correspondent at The New York Times from 1973 to 1977.

He served as an Assistant Secretary of State in the Carter Administration from 1977 to 1979, serving as director of the Bureau of Politico-Military Affairs and winning the Distinguished Honor Award, the highest award of the US State Department.

He returned to the Times in 1981; from then until 1993, he was in turn its national security correspondent, deputy editorial page editor, editor of the Op-Ed Page, and columnist. This period included his leading role on the Times team that won a Pulitzer Prize for Explanatory Journalism in 1986 for a six-part comprehensive series on the "Star Wars" Strategic Defense Initiative).

Gelb became President of the Council on Foreign Relations in 1993 and is its President Emeritus.

He serves as the chairman of the advisory board for the progressive foreign policy think tank, National Security Network, on the board of directors of the Iraq and Afghanistan Veterans of America, member of the board of directors of the Truman scholars program, board of directors of the Nixon Center and the advisory board of United Against Nuclear Iran.

Gelb is a contributor to The Daily Beast, a news aggregation site.