Johnson was born in Jacksonville, Florida, the son of Helen Louise Dillet and James Johnson. His brother was the composer J. Rosamond Johnson. Johnson was first educated by his mother (a musician and a public school teacher...the first female, black teacher in Florida at a grammar school) and then at Edwin M. Stanton School. His mother imparted to him her considerable love and knowledge of English literature and the European tradition in music. At the age of 16 he enrolled at Atlanta University, from which he graduated in 1894. In addition to his bachelor's degree, he also completed some graduate coursework there. The achievement of his father, headwaiter at the St. James Hotel, a luxury establishment built when Jacksonville was one of Florida's first winter havens, gave young Jimmie the wherewithal and the self-confidence to pursue a professional career. Molded by the classical education for which Atlanta University was best known, Johnson regarded his academic training as a trust given him in the expectation that he would dedicate his resources to black people. Johnson was also a prominent member of Phi Beta Sigma Fraternity, Inc.

He served in several public capacities over the next 35 years, working in education, the diplomatic corps, civil rights activism, literature, poetry, and music. In 1904 Johnson went on Theodore Roosevelt's presidential Campaign. Theodore Roosevelt appointed Johnson as U.S. consul at Puerto Cabello, Venezuela from 1906–1908 and then Nicaragua from 1909–1913.

In 1910, Johnson married Grace Nail while he was a United States Consul in Nicaragua. They had met several years earlier in New York when Johnson was working as a songwriter. A cultured and well-educated New Yorker, Grace Nail Johnson became an accomplished artist in pastels and collaborated with her husband on a screenwriting project.

In the summer of 1891 the Atlanta University freshman had gone to a rural district in Georgia to instruct the children of former slaves. "In all of my experience there has been no period so brief that has meant so much in my education for life as the three months I spent in the backwoods of Georgia," Johnson wrote. "I was thrown for the first time on my own resources and abilities." James Weldon Johnson graduated from Atlanta University in 1894. He would later receive an honorary Master's degree in 1904. After graduation he returned to Stanton, a school for African American students in Jacksonville, until 1906, where, at the young age of 23, he became principal. As principal Johnson found himself the head of the largest public school in Jacksonville regardless of race. For his work Johnson received a paycheck less than half of what was offered to a white colleague possessing a comparable position. Johnson improved education by adding the ninth and tenth grades. Algebra, English composition, physical geography and bookkeeping were a part of the added ninth grade course. The tenth grade course consisted of geometry, English literature, elementary physics, history and Spanish. Johnson later resigned from his position as principal.

In 1897, Johnson was the first African American admitted to the Florida Bar Exam since Reconstruction.He was also the first Negro in Duval County to seek admission to the state bar. In order to receive entry Johnson underwent a two-hour examination before three attorneys and a judge. He later recalled that one of the examiners, not wanting to see a black man admitted, left the room.

In December 1930, Johnson resigned from the leadership of the NAACP to accept the Spence Chair of Creative Literature at Fisk University in Nashville, where he lectured not only on literature but also on a wide range of issues to do with the life and civil rights of black Americans. The position had been especially created for him, largely out of recognition of his achievements as a poet, editor, and critic during the heyday of the Harlem Renaissance in the 1920s. He held this position until his death in an automobile accident in 1938.] He also wrote the melody for the song Dem Bones.

In 1922, he edited The Book of American Negro Poetry, which the Academy of American Poets calls "a major contribution to the history of African-American literature." One of the works for which he is best remembered today, Seven Negro Sermons in Verse, was published in 1927 and celebrates the tradition of the folk preacher. In 1917, Johnson published 50 Years and Other Poems.

While attending Atlanta University Johnson became known as an influential campus speaker. He won the Quiz Club Contest in English Composition and Oratory in 1892. The contest topic was "The Best Methods of Removing the Disabilities of Caste from the Negro". In addition, Johnson founded the newspaper the Daily American and in 1895 and became its editor. The newspaper concerned both political and racial topics. It was terminated a year later due to financial difficulty. These early endeavors were the start of what would prove to be a long period of activism.

Johnson became further involved with political activism during 1904 when he accepted a position as the treasurer of the Colored Republican Club started by Charles W. Anderson. A year later he became the president of the club. His duties as president included organizing political rallies. During 1914 Johnson became editor of the editorial page of the New York Age, an influential African American weekly newspaper that had supported Booker T. Washington in his propaganda struggle with fellow African American W. E. B. Du Bois during the early twentieth century. Johnson's writing for the Age displayed the political gift that soon made him famous.

In the fall of 1916, because Johnson excelled as a reconciler of differences among those whose ideological agendas seemed to preclude unified, cooperative action, he was asked to become the national organizer for the National Association for the Advancement of Colored People (NAACP). Opposing race riots in northern cities and the lynchings that pervaded the South during and immediately after the end of World War I, Johnson engaged the NAACP in mass tactics, such as a silent protest parade down New York's Fifth Avenue in which ten thousand African Americans took part on July 28, 1917. In 1920 Johnson was elected to manage the NAACP, the first African American to hold this position. While serving the NAACP from 1914 through 1930 Johnson started as an organizer and eventually became the first black male secretary in the organization's history. In 1920, he was sent by the NAACP to investigate conditions in Haiti, which had been occupied by U.S. Marines since 1915. Johnson published a series of articles in The Nation, in which he described the American occupation as being brutal and offered suggestions for the economic and social development of Haiti.These articles were reprinted under the title Self-Determining Haiti.