In 1936, at the age of eleven Baker was put into an orphanage. He ran away frequently, and had to be retrieved by the staff from St. Louis, New York City, Chicago, and Pittsburgh. Eventually the orphanage quit looking for him, and at the age of 16 he stayed in New York City. He found work as a laborer and then a dishwasher. But, after hanging out in the pool halls of 26th Street, gave up work to become a full-time pool shark.

At nineteen, Baker decided to make a change in his life. He went back to dishwashing, and was determined to become a jazz musician. The trumpet was his first choice for an instrument, but with only fourteen dollars saved up, he could not find a pawnshop with anything but guitars for that price.

He enrolled at The New York School Of Music, but found the learning pace too slow. He dropped out and resolved to teach himself, but gave up shortly afterwards. Six months later he met a street guitarist who inspired him to start playing again. He continued taking private lessons from different teachers over the next few years, and, like many musicians of the day, tried to play his instrument like Charlie Parker played the saxophone.

By 1949, Baker had his own combo, and a few paying jobs. He decided to move west, but found that audiences there were not receptive to progressive jazz music. Baker was stranded without work in California when he saw a show by blues guitarist Pee Wee Crayton. Baker said of the encounter:

"I asked Pee Wee, 'You mean you can make money playing that stuff on guitar?' Here he was driving a big white Eldorado and had a huge bus for his band. So I started bending strings. I was starving to death, and the blues was just a financial thing for me then."

He found a few jobs in Richmond, California, and made enough money to return to New York.

After returning back east, Baker began recording for Savoy, King and Atlantic Records. He did sessions with The Drifters, Ray Charles, Ivory Joe Hunter, Ruth Brown, Big Joe Turner, Louis Jordan, Coleman Hawkins, and numerous other artists. During this time, Baker (along with either Paramour Crampton or Connie Kay on drums, Sam "The Man" Taylor on tenor, and Lloyd Crompton on bass) played on virtually every hit record by Atlantic, Savoy, and King.

Inspired by the success of Les Paul & Mary Ford, he formed the pop duo Mickey & Sylvia (with Sylvia Robinson, one of his guitar students) in the mid 1950s. Together, they had a hit single with "Love Is Strange" in 1957. The duo split up in the late 1950s, but continued to record off and on until the middle of the next decade. It was around this time that he moved to France, making a few solo records and working with some French pop and rock performers, including Ronnie Bird and Chantal Goya. Baker appeared at the 1975 version of the Roskilde Festival.

In 1999 Baker received a Pioneer Award from the Rhythm and Blues Foundation.

In 2003 he was listed at #53 on Rolling Stone's "100 Greatest Guitarists of All Time".

His self-tuition method book series, the Complete Course in Jazz Guitar is a mainstay for introducing students of guitar to the world of jazz. They have remained in print for over 50 years.

His guitar playing appeared on the following records:

• "Money Honey" - The Drifters
• "Shake, Rattle and Roll" - Big Joe Turner
• "It's Gonna Work Out Fine" - Ike & Tina Turner
• "Whole Lotta Shakin' Goin On" - Big Maybelle
• "Mama, He Treats Your Daughter Mean" - Ruth Brown