John McElroy was born to Robert and Mary Henderson McElroy in Greenup County, Kentucky. When his father died, he traveled to St. Louis to become an apprentice in the printing business.

As a sixteen year old in 1863, McElroy enlisted in the Union Army as a private in Company L of the 16th Illinois Cavalry regiment, having earlier served with local Union troops in operations near St. Louis. In January 1864, he was among dozens of men captured in a skirmish near Jonesville, Virginia, by Confederate cavalrymen under William E. Jones. McElroy sent to a variety of camps before being assigned to Andersonville prison, where he remained for the rest of the war.

After the war ended, McElroy was released from captivity and transported back to the North. He settled in Chicago and resumed the printer's trade. He became a local reporter and newspaperman before moving to Toledo, Ohio, to become an editor of the Toledo Blade. He married Elsie Pomeroy of Ottawa, Ohio, and raised a family. In 1879, he wrote Andersonville: A Story of Rebel Military Prisons, a non-fiction work based on his experiences during his fifteen-month incarceration. It quickly became a bestseller and remained popular for the next twenty years.

In 1884, he moved from Toledo to Washington, D.C. to take over as editor and co-owner of the National Tribune. He was active in the local Grand Army of the Republic, serving as commander of the Department of the Potomac in 1896. In 1908, McElroy wrote The Economic Functions of Vice. The following year, he published Struggle for Missouri, a history of the bitter division over slavery that split the state's loyalties and led to armed conflict within its borders. In 1910, he wrote a Civil War novel entitled Si Klegg: His Transformation from a Raw Recruit to a Veteran.