Carroll was born in Chicago, the second of five sons of Joseph Carroll and his wife Mary. At the time, his father was a Special Agent of the FBI, which he remained until being seconded to, and later commissioned by, the US Air Force as an Intelligence Officer in 1948. After this, Carroll was raised in the Washington, D.C. area and in Germany. He was educated at Washington’s Priory School and at an American high school, the H. H. Arnold, in Wiesbaden, Germany He attended Georgetown University before entering St. Paul’s College, the Paulist Fathers’ seminary, where he received his B.A. and M.A. degrees.

He was ordained to the priesthood in 1969. Carroll served as Catholic chaplain at Boston University from 1969 to 1974. During that time, he studied poetry with George Starbuck and published books on religious subjects and a book of poems. He was also a columnist for the National Catholic Reporter (1972-1975) and was named Best Columnist by the Catholic Press Association. For his writing on religion and politics he received the first Thomas Merton Award from Pittsburgh’s Thomas Merton Center in 1972. Carroll left the priesthood to become a writer, and in 1974 was a playwright-in-residence at the Berkshire Theater Festival.

Carroll’s plays have been produced at the Berkshire Theater Festival and at Boston’s Next Move Theater. In 1976 he published his first novel, Madonna Red, which was followed by nine others. He has written for numerous publications, including The New Yorker, and his op-ed column appears weekly in the Boston Globe. He won the 1996 National Book Award for nonfiction for An American Requiem, a memoir of his relationships with his father, the American military, and the Catholic Church.

He is the author of other books on religion and politics, including House of War, which won the first PEN-Galbraith Award. Mr. Carroll's other works include the novels Secret Father, The City Below, Memorial Bridge, Prince of Peace, Mortal Friends, and Madonna Red, in addition to various plays and Forbidden Disappointments, a book of poetry published in 1974. Carroll's work has received the Melcher Book Award, the James Parks Morton Interfaith Award, and National Jewish Book Award in History, and has been frequently been named among the Notable Books of the Year by the New York Times.

Carroll has been a Shorenstein Fellow at the Kennedy School of Government at Harvard University and a Fellow at the Center for the Study of Values in Public Life at the Harvard Divinity School. He is a trustee of the Boston Public Library, a member of the Advisory Board of the International Center for Ethics, Justice, and Public Life at Brandeis University, and a member of the Dean’s Council at the Harvard Divinity School. Carroll is a Fellow of the American Academy of Arts and Sciences, and is a member of the Academy’s Committee on International Security Studies. He worked on his 2006 history of the Pentagon, House of War, as a Scholar-in-Residence at the Academy. Carroll is also a Distinguished Scholar-in-Residence at Suffolk University, where he wrote his latest book, Practicing Catholic, published in 2009.

Carroll wrote a long and detailed history of Christian attitudes and treatment of Jews, titled Constantine's Sword: The Church and the Jews (2001). In this work, he connects many personal experiences to the places and attitudes that he analyzes.

Carroll co-wrote the screenplay for the 2007 documentary Constantine's Sword with filmmaker Oren Jacoby. The book had been a New York Times Best-Seller.

In a 2009 book, he denounced Pope Benedict XVI as “the chief sponsor of the new Catholic fundamentalism, enforced with no regard for the real cost to human beings”.

In The Myth of Hitler's Pope, rabbi David G. Dalin, professor at the Roman Catholic Ave Maria University:

"anti-papal polemics of ex-seminarians like Garry Wills and John Cornwell (author of Hitler's Pope), of ex-priests like James Carroll, and or other lapsed or angry liberal Catholics exploit the tragedy of the Jewish people during the Holocaust to foster their own political agenda of forcing changes on the Catholic Church today."

Carroll is married to the novelist Alexandra Marshall. They have two grown children.