Born on a farm in Holyoke, Massachusetts, Sheehan graduated from Mount Hermon School (later Northfield Mount Hermon) and Harvard University with a B.A. in 1958, served in the U.S. Army from 1959 to 1962. In 1962 he began working at the United Press International's Tokyo bureau, and spent the next two years covering the war in Vietnam as UPI's bureau chief. In 1963, during the Buddhist crisis, he and David Halberstam debunked the claim by the regime of Ngo Dinh Diem that the Army of the Republic of Vietnam regular forces had perpetrated the Xa Loi Pagoda raids, which the American authorities initially believed, and that instead, the Special Forces loyal to Diem's brother Ngo Dinh Nhu had done so to frame the army generals.

In 1964, he joined The New York Times. He worked the city desk before returning to the Far East to report from Indonesia and then to spend another year in Vietnam. In the fall of 1966 he became the newspaper's Pentagon correspondent and in 1968 began reporting on the White House. In 1971 he obtained the Pentagon Papers for the Times. He was a correspondent on political, diplomatic and military affairs.

In the New York Times Book Review, December 27, 1970 he claimed that Conversations With Americans by Mark Lane was a collection of Vietnam war crime stories with some obvious flaws which the author had not verified. Sheehan called for a more thorough and scholarly work to be done on the war crimes being committed in Vietnam.

He was awarded a non-fiction Pulitzer Prize in 1989 and a National Book Award for A Bright Shining Lie about the life of Lieutenant Colonel John Paul Vann and the United States involvement during the Vietnam War. (The book was published by Random House and edited by Robert Loomis.)

His wife, Susan Sheehan, also was awarded a Pulitzer Prize in 1983 for Is There No Place On Earth For Me?

As of 2009, Sheehan lived in Washington D.C.