Hemphill was born in 1936 in Birmingham, Alabama, where his father was a truck driver. He grew up Birmingham's Woodlawn neighborhood and attended Woodlawn High School there. He briefly played for the Class D minor league baseball Graceville Oilers of the Alabama-Florida League but was cut from the team at the start of spring training. Hemphill then played semi-pro baseball before switching to focus on college and writing. He graduated from Alabama Polytechnic Institute (later renamed Auburn University), working on the school newspaper, The Plainsman, earning a bachelor's degree in 1959. While in college, he worked as an intern at the Birmingham News, working his way up from covering little league to writing about high school sports.

He was a sports reporter for papers in Augusta, Georgia and Tampa, Florida before being hired in 1964 by the short-lived Atlanta Times. His writing led to a spot as a featured columnist in the Atlanta Journal shortly thereafter, where he became a reader favorite for his reporting on people and places from the South. He resigned despite all his experiences and opportunities with the paper, having felt that "with the next column due by dawn, I had run out of gas".

He started his first and most successful book, The Nashville Sound: Bright Lights and Country Music (1970), while at Harvard University on a Nieman Fellowship, a program designed to allow journalists the time to reflect on their careers and focus on honing their skills. The book was described by The New York Times as being "generally regarded as one of the best books on country music ever written". The book provided an eye on the scene around the Grand Ole Opry in Nashville, Tennessee at a time when country music was starting to achieve broader cultural recognition.

The Good Old Boys (1974) was the first collection of his newspaper pieces, featuring items about country singers, baseball players and other assorted characters. His 1973 novel about a minor league baseball team, Long Gone, was adapted as a 1987 move on HBO starring Virginia Madsen and William L. Petersen. Other novels included 1985's The Sixkiller Chronicles and his 1989 work King of the Road.

Though Hemphill had deeply respected his truck-driving father in his youth, he later began to despise him for his unchecked racism. His relationship with his father became the basis for his 1993 book Leaving Birmingham: Notes of a Native Son. Later books included the 1996 The Heart of the Game about a player for the Durham Bulls and Wheels: A Season on NASCAR's Winston Cup Circuit published in 1997. His 2005 Hank Williams biography Lovesick Blues marked a return to country music and his final book, the 2008 A Tiger Walk Through History was the story of Auburn Tigers football.

Hemphill served on the faculty at Emory University and the University of Georgia, where he taught writing.

Hemphill died at age 73 on July 11, 2009 from throat cancer that had metastasized to his lungs. He was survived by his third wife, Susan Percy, as well as three children from his first marriage, a daughter from his second marriage and six grandchildren.