He earned a degree in journalism from New York University then served with the United States Army for four years. He joined the staff of the New York Times but remained only a short time before going to work at The Morning Telegraph, then the companion paper of the Daily Racing Form with which he became associated in 1954 and retired from as its executive columnist in 2003.

Often referred to as the "dean" of Thoroughbred racing writers, Hirsch American writers to win both the Eclipse Award for outstanding writing and the Lord Derby Award in London from the Horserace Writers and Reporters Association of Great Britain. Hirsch also received the Eclipse Award of Merit (1992), the Big Sport of Turfdom Award (1983), The Jockey Club Medal (1989), and was designated as the honored guest at the 1994 Thoroughbred Club of America's Testimonial Dinner. The annual Grade 1 Joe Hirsch Turf Classic Invitational at Belmont Park was named in his honor as are the press boxes at the Saratoga Race Course and Churchill Downs racecourses. The Breeders' Cup Ltd. presents the Joe Hirsch Award to a member of the media for their coverage of the Breeders' Cup.

In 2005, the University of Kentucky and the National Thoroughbred Racing Association (NTRA) announced the creation of the Joe Hirsch Scholarship to assist a worthy student interested in pursuing a career in Thoroughbred racing journalism. The first recipient of the scholarship was Ms Amanda Duckworth.

Hirsch is also known for being the roommate to Jets rookie Quarterback Joe Namath upon his arrival in New York City.

Until his death, Hirsch served as a member of the selection committee for the National Museum of Racing and Hall of Fame. He died early in the morning on January 9, 2009 at St. Luke's Hospital in New York.