Penick was born in Austin, Texas. He began his golf career as a caddy at the Austin Country Club in at age eight. He became the club's assistant pro five years later, and after his graduation from high school, was promoted to head professional in 1923, where he remained until 1973. After 1973, Penick continued teaching at the club.

Penick was the golf coach at the University of Texas from 1931 to 1963, coaching the Longhorns to 21 Southwest Conference championships in 33 years, including 20 out of 23 seasons from 1932 to 1954 (1932-38; 1940-47; 1949-52; 1954). He coached the following members of the World Golf Hall of Fame: Tom Kite, Ben Crenshaw, Mickey Wright, Betsy Rawls, and Kathy Whitworth. Other outstanding players coached by Penick include: Sandra Palmer, Judy Kimball, Terry Dill, Don Massengale, Rik Massengale, and Edward White. In 1989, Penick was honored by the PGA of America as Teacher of the Year.

In 1992, he co-authored (with Bud Shrake) Harvey Penick's Little Red Book; filled with insightful, easily understood anecdotes, it became the highest selling golf book ever published. While Penick was a strong all-around teacher of the game, he was perhaps the most gifted instructor of the mental game who ever lived. He advocated that "once you address the ball, hitting it to the desired target must be the only thing in your life. Allow no negative thoughts, and focus on your goal." Penick and Shrake collaborated on four more golf books on similar themes; all sold well. The final three Penick-Shrake books were published after Penick's 1995 death, based on Penick's extensive notes from his lifetime of golf instruction.

During his final illness, he gave lessons from his deathbed to longtime student Ben Crenshaw. The day after serving as a pallbearer at Penick's funeral, Crenshaw began play in the 1995 Masters Tournament. With the memory and spirit of his longtime friend and mentor to guide him, he became the second oldest Masters champion, winning his second Masters at the age of 43. In the post-tournament interview, Crenshaw said: "I had a 15th club in my bag," a reference to Penick. (The "15th club" reference is based on the golf rule that limits a player to carrying 14 clubs during a round.)

Penick died in Austin. In 2002, he was inducted into the World Golf Hall of Fame.