Silverman was born on June 3, 1942, in Baltimore, Maryland, USA, the son of a grocer and a homemaker, and grew up on Queensberry Avenue in the city's Pimlico neighborhood. Silverman's exploits growing up in Forest Park during the 1950s and 1960s were later immortalized in the 1982 movie, Diner, which was directed by Barry Levinson. He attended the Forest Park High School where, among other things, he played lacrosse. In one of his books he describes himself as "a very uncoachable, selfish and lazy player." He and Levinson graduated in 1960, they would remain life-long friends and collaborate on several other projects. Silverman attended the University of Maryland, College Park and earned a bachelor's degree in history from the University of Baltimore in 1966. In 1975 he was awarded an M.S.from Morgan State University and later, earned another master's degree in public health from the Johns Hopkins School of Public Health and a doctorate in health care management from Century University in Albuquerque, N.M.

Silverman was an assistant dean and acting dean at Morgan State University during the 1970s and during his time at Morgan Silverman realized that many of the city's black high school lacrosse players had matriculated to Morgan but could not play lacrosse because Morgan had no team. In 1970 he founded and coached a lacrosse club at Morgan and in 1972 it was invited to join the United States Small College Lacrosse League. Silverman gradually augmented the team with black players from other hotbeds of lacrosse like Long Island, Philadelphia and upstate New York and his teams regularly beat the likes of Notre Dame. In 1975, he led the Morgan Bears lacrosse team to a stunning victory over top-ranked Washington and Lee. His fifth book, Ten Bears, co-written with Dr. Miles Harrison, a Baltimore surgeon, is about that team and was recently optioned to Warner Bros.. A movie based on the book is currently in production. In 1991, Silverman was inducted into the Morgan State University Hall of Fame.Silverman was also a Special Advisor to Maryland Governor William Donald Schaefer for Substance Abuse Policy and the Director of the Drug and Alcohol Administration; he worked for the state for 26 years. Silverman was also a part-time journalist, having written for the Baltimore News American, and produced and hosted segments of the Evening/PM Magazine television program on Baltimore's WJZ-TV.On March 6, 2008, Silverman lost his battle with malenoma and died in Baltimore.

In addition to Ten Bears, Silverman wrote or co-authored The Block, The Last Bookmaker with Bob Litwin, Aloha Magnum with Larry Manetti and Diner Guys.

Silverman appeared in the American films Diner and Tin Men as well as two episodes of the American television series Homicide: Life on the street.