La Farge was born in New York City the son of Christopher Grant La Farge and Florence Bayard Lockwood, the niece of Senator Thomas F. Bayard. He was named for his ancestor Oliver Hazard Perry and his uncle, architect Oliver H.P. La Farge I. He received a Bachelor of Arts degree from Harvard University during 1924 and a Master's Degree from the same university in 1929.A member of the university rowing team, he rowed as captain of a men's eight-oared shell.

La Farge, a Newport, Rhode Island New Englander whose grandfather was of French descent and whose other ancestors included English colonists and (allegedly) Narragansett American Indians, was an anthropologist who discovered two previously unknown languages while on scientific expeditions to Central America and the American Southwest.He is well-known for his pioneering visit, with Frans Blom during 1925, to Mexico and what has since become known as the Olmec heartland, (re)discovering San Martin Pajapan Monument 1 and, most importantly, the ruins of La Venta, one of the major Olmec centers.

La Farge also spent much of his adult life championing American Indian rights and was president of the Association on American Indian Affairs for several years.

His published work included several scientific papers and non-fiction books, as well as several novels and a column for the Santa Fe newspaper The New Mexican. Some of his columns were released in book form under the title The Man With the Calabash Pipe.

La Farge had two children by his first wife, heiress Wanden Matthews: a son, Oliver Albee La Farge, and a daughter, Povy; and another son, John Pendaries "Pen" La Farge, by his second wife, Consuelo Baca de La Farge. His 1956 book Behind the Mountains was based on the reminiscences of Consuelo's family, who were ranchers in northern New Mexico. After La Farge and Matthews divorced in 1935, Oliver Albee changed his name to Peter La Farge and became a Greenwich Village folksinger with five Folkways Records albums.

La Farge died in Santa Fe, New Mexico, in 1963.