Baxter was born in Dunedin to Archibald Baxter and Millicent Brown and grew up near Brighton. He was named after James Keir Hardie, a founder of the British Labour Party. His father had been a conscientious objector during the First World War. His mother had studied at the Presbyterian Ladies' College, Sydney, the University of Sydney and Newnham College.
On his first day of school, Baxter burned his hand on a stove and later used this incident to represent the failure of institutional education. As a child he contrasted the social order represented by his maternal grandfather with the clan mentality of his Scottish father and frequently drew analogies between the Highland clans and the M?ori Tribes.
Baxter, like the comparable Francis Webb in Australia, claims to have begun writing poetry at the age of seven, and it is certain that he accumulated a large body of technically-accomplished work both before and during his teenage years. He continued to write throughout his lifetime, although his frequent shifts of religion and lifestyle were the center of much controversy and speculation.
Baxter typically wrote short lyrical poems or cycles of the same rather than longer poems.
In 1944, at age seventeen he joined the University of Otago and that year he published his first collection of poetry, Beyond the Palisades, to much critical acclaim. His work during this time was, as with his contemporary compatriots, most notably the experimental novelist Janet Frame, largely influenced by the modernist works of Dylan Thomas. He was a member of the so-called "Wellington Group" of writers that also included Louis Johnson, W.H. Oliver and Alistair Campbell.
Baxter failed to complete his course work at the University of Otago and was forced to take a range of odd jobs, most notably a cleaner at Chelsea Sugar Refinery, which inspired the poem "Ballad of the Stonegut Sugar Works". In 1948 he married Jacqueline Sturm, and at about the same time his interest in Christianity culminated in his joining the Catholic church.
In February 1951 Baxter enrolled at Wellington Teachers’ College. In 1952 his son, John, was born and a selection of poems in a collaborative volume, Poems unpleasant, was published. Having completed his course at teachers’ college in December, Baxter spent 1953 in full-time study at Victoria University College and published his third major collection, The fallen house. In 1954 he was appointed assistant master at Epuni School, Lower Hutt. He received a BA in 1956.
While at the University of Otago Baxter began drinking heavily. By 1954 he had joined Alcoholics Anonymous. By 1955 he had garnered a substantial legacy and could afford a comfortable house in Ngaio, Wellington. He left Epuni School early in 1956 to write and edit primary school bulletins for the Department of Education’s School Publications Branch. This period is likely to have influenced his writing providing material for numerous attacks on bureaucracy.
In 1957 Baxter took a course in Roman Catholicism, and his collection of poems In Fires of No Return, published in 1958, was influenced by his new faith. This was his first work to be published internationally, though English critics were largely nonplussed. His wife, a committed Anglican, was dismayed by his Catholicism, and they divorced in 1957. Through the late 50s and 60s Baxter visited the Southern Star Abbey a Cistercian monastery at Kapoa near Central Hawke's Bay.
The following year, 1958, Baxter received a UNESCO stipend and began an extended journey through Asia, and especially India, whose Rabindranath Tagore's university Shantiniketan was one of the inspirations for Baxter's later community at Jerusalem. Here he was reconciled with his wife and contracted dysentery. His writing after returning from India was more overtly critical of New Zealand society. In the 1960s he became a powerful and prolific writer of both poems and drama, and it was through his radio play Jack Winter's dream that he became internationally known.
The first half of the 1960s saw Baxter struggling to make ends meet on his postman's wage, having refused to take work as a schoolmaster. However, it was at this time that the collection of poems Pig Island Letters was published in which his writing found a new level of clarity. In 1966, Baxter took up the Robert Burns Fellowship at the University of Otago.
In 1968 Baxter claimed that he had been instructed in a dream to 'Go to Jerusalem'. Jerusalem was a small M?ori settlement (known by its M?ori transliteration, Hiruharama) on the Whanganui River. He left his University position and a job composing catechetical material for the Catholic Education Board, with nothing but a bible. This was the culmination of a short period in which he struggled with family life and his vocation as a poet.
Baxter spent some time in Grafton, Auckland where he set up a centre for drug addicts acting on the same principles as Alcoholics Anonymous. In 1969 he adopted the M?ori version of his name, Hemi, and moved to Jerusalem. He lived a sparse existence and made frequent trips to the nearby cities where he worked with the poor and spoke out against what he perceived as a social order that sanctions poverty. His poems of this time have a conversational style but speak strongly of his social and political convictions.
The harsh deprivations Baxter adopted at this time took their toll on his health. By 1972 he was too ill to continue living at Jerusalem and moved to a commune near Auckland. On October 22, 1972 he suffered a coronary thrombosis in the street and died in a nearby house, aged 46. He was buried at Jerusalem on M?ori land in front of "the Top House" where he had lived, in a ceremony combining M?ori and Catholic traditions.