Ballard was born in Salt Lake City, Utah, to Melvin Russell Ballard Sr. and his wife Geraldine Smith Ballard. As a young man he served as a missionary in England from 1948 to 1950. He met his wife while they were studying at the University of Utah. In 1974, Ballard was called to be the president of the Canada Toronto Mission of the church. He served as mission president until he was called to the First Quorum of the Seventy in 1976.

Following the death of apostle Bruce R. McConkie, Ballard was sustained to the Quorum of the Twelve Apostles on October 6, 1985, and ordained an apostle of the LDS Church on October 10, 1985.

Ballard is the grandson of apostles Melvin J. Ballard and Hyrum M. Smith and is a descendant of Hyrum Smith, brother of church founder Joseph Smith, Jr.

Professionally, Ballard became involved in several enterprises, including automotive, real estate, and investment businesses. He was the top-selling salesman for his father’s Nash car dealership when he left it in the early 1950s to pursue other business interests. In 1956, Ballard returned and took over the Ballard Motor Company from his father. During this period he also served in the United States Army Reserve, resigning his commission as a first lieutenant in 1957.

During the late 1950s, Ballard was recruited by the Ford Motor Company to become the first Edsel car dealer for Salt Lake City. After praying for guidance, he had the "clear impression" not to sign the franchise. He did anyway and incurred a huge loss, "without doubt the darkest period" of his business career.

One highlight of his business career was his presidency of the Valley Music Hall in Bountiful, Utah, which offered high-quality family entertainment. There Ballard worked with Art Linkletter, Danny Thomas, Bob Cummings, and other Hollywood celebrities who were advisers to the enterprise. Although the music hall failed financially, he ensured that investors recovered the money invested.

Ballard married Barbara Bowen in 1951. The Ballards are the parents of seven children. One of their daughters is married to Peter Huntsman, who is the brother of Jon Huntsman, Jr., governor of Utah, and the son of LDS Church area seventy and billionaire philanthropist Jon M. Huntsman.