Born and raised in Todd County, Kentucky at her family's plantation home known as "Woodstock," Caroline Gordon received a high level of education at her father's Clarksville Classical School for Boys in neighboring Montgomery County, Tennessee. By 1916 she had graduated from West Virginia's Bethany College and obtained a job as a writer of society news for the Chattanooga Reporter newspaper. After eight years, she left Chattanooga and returned home, where, at the age of twenty-nine, she met Allen Tate, a free-spirited "bohemian" poet, commentator and essayist, four years her junior. They immediately embarked on a passionate love affair which culminated in a pregnancy and a May 15, 1925 wedding. Their daughter Nancy was born in September.

Over the next twenty years, Caroline Gordon (who retained her maiden name) and Allen Tate lived in Tate's house in Clarksville. Their guests included some of the best-known writers of their time, including F. Scott Fitzgerald, Ernest Hemingway, William Faulkner, Flannery O'Connor, T. S. Eliot, Robert Penn Warren, and Ford Madox Ford, the author whom Gordon considered her mentor. Ford counseled and prodded her into completing her first novel Penhally, published in 1931. Gordon received both of her awards, the Guggenheim and the O. Henry, during this early period. The O. Henry was a unique second-place prize awarded for her 1934 short story "Old Red", published in Scribner's Magazine. There were seventeen third-place recipients that year, including William Saroyan, Pearl Buck, Erskine Caldwell, William Faulkner, John Steinbeck, and Thomas Wolfe.

Between 1934 and 1972, Gordon published nine additional novels, five of which were written during the late 1930s and World War II.

Gordon's early fiction was influenced by her association with the Southern Agrarians. Paul V. Murphy writes that she "exhibited a southern nostalgia as strong as any member of the group, including Davidson, the most unreconstructed of the Agrarians".

Caroline Gordon's marriage to Allen Tate ended in divorce in 1945, followed by a 1946 remarriage and an ultimate divorce in 1959. They continued to correspond, however, and remained friends. On November 24, 1947, during another difficult period in her marriage, Gordon converted to Catholicism.

She continued to write until crippled by a March 1, 1981, stroke in San Cristóbal de las Casas, Chiapas, Mexico, where she lived in her later years. She died six weeks later, following surgery, at the age of 85.