James Hogg was born in a small farm near Ettrick, Scotland in 1770 and was baptized there on 9 December, his actual date of birth having never been recorded. His father, Robert Hogg (1729—1820), was a tenant farmer while his mother, Margaret Hogg (née Laidlaw) (1730—1813), was noted for collecting native Scottish ballads. James was the second eldest of four brothers, his siblings being William, David, and Robert (from eldest to youngest). Robert and David later emigrated to the United States, while James and William remained in Scotland for their entire lives.

James had little formal education, and became a shepherd, living in grinding poverty, hence his nickname, 'The Ettrick Shepherd'. His employer, James Laidlaw of Blackhouse in the Yarrow valley, seeing how hard he was working to improve himself, offered to help by making books available. Hogg used these to essentially teach himself to read and write (something he had achieved by the age of 14). In 1796 Robert Burns died, and Hogg, who had only just come to hear of him, was devastated by the loss. He struggled to produce poetry of his own, and Laidlaw introduced him to Sir Walter Scott, who asked him to help with a publication entitled The Minstrelsy of the Scottish Border.

In 1801, Hogg visited Edinburgh for the first time. His first collection, The Mountain Bard, was published in 1807 but he struggled to make an impact on the literary scene. Another venture, a magazine, The Spy failed after a year. But his epic story-poem, The Queen's Wake (the setting being the return to Scotland of Mary, Queen of Scots (1561) after her exile in France), was published in 1813 and was a success. Now a well-known literary figure (if often mocked for his rustic accent and appearance), he was recruited by William Blackwood for Blackwood's Magazine.

It was through Blackwood's that Hogg found fame, although it was not the sort that he wanted. Launched as a counter-blast to the Whig Edinburgh Review, Blackwood wanted punchy content in his new publication. He found his ideal contributors in John Wilson (who wrote as Christopher North) and John Gibson Lockhart (later Walter Scott's son-in-law and biographer). Their first published article, "The Chaldee Manuscript", a thinly disguised satire of Edinburgh society in biblical language which Hogg started and Wilson and Lockhart elaborated, was so controversial that Wilson fled and Blackwood was forced to apologise. Soon Blackwood's Tory views and reviews - often scurrilous attacks on other writers - were notorious, and the magazine, or "Maga" as it came to be known, had become one of the best-selling journals of its day. But Hogg quickly found himself forced out of the inner circle. As other writers such as Walter Maginn and Thomas de Quincey joined, he became not merely excluded from the lion's share of publication in Maga, but a figure of fun in its pages. Wilson and Lockhart were dangerous friends. Hogg's Memoirs of the Author's Life were savagely attacked by an anonymous reviewer, probably Wilson, and in 1822 the magazine launched the "Noctes Ambrosianae" or "Ambrosian Nights", imaginary conversations in a drinking-den between semi-fictional characters such as North, O'Doherty, The Opium Eater and the Ettrick Shepherd. The Shepherd was Hogg. The Noctes continued until 1834, the year of the real Hogg's death, and were written after 1825 mostly by Wilson, although other writers, including Hogg himself, had a hand in them.

The Shepherd of the Noctes is an extraordinary creation, part-animal, part-rural simpleton, part-savant, easily the most memorable character in the series, who speaks some of the richest and saltiest Scots ever written. He became one of the best-known figures in topical literary affairs, famous throughout Britain and its colonies. Quite what the real James Hogg made of this is mostly unknown, although some of his letters to Blackwood and others express outrage and anguish. What is known is that in 1824, no longer highly regarded in Edinburgh, largely excluded from Blackwood's, now in his fifties but with a young family, and writing desperately quickly for money to try to save his failing farm, Hogg wrote his famous tale of persecution, delusion, devilish mimickry and tortured consciousness: The Private Memoirs and Confessions of a Justified Sinner. It did not do well. Barely reviewed in Blackwood's, it became a largely forgotten book until over a century later.

Wordsworth's 1835 "Extempore Effusion upon the Death of James Hogg", written in the year of his death, includes the lines:

"The mighty Minstrel breathes no longer,

'Mid mouldering ruins low he lies;

And death upon the braes of Yarrow,

Has closed the Shepherd-poet's eyes."

This eulogy notwithstanding, Wordsworth's notes state "He was undoubtedly a man of original genius, but of coarse manners and low and offensive opinions."

In 1924 the French writer, André Gide, was loaned Justified Sinner by Raymond Mortimer. Gide was amazed, writing that "It is long since I can remember being so taken hold of, so voluptuously tormented by any book." Its republication in 1947, with an enthusiastic introduction by Gide, helped bring about the modern critical and academic appreciation of this novel.

The bulk of Hogg's writing was bowdlerised in the 19th century and neglected for most of the 20th. Apart from The Confessions, which even his detractors acknowledged as unusually powerful (and often attributed to someone else, usually Lockhart), his novels were regarded as turgid, his verse as light, his short tales and articles as ephemera. But growing interest in The Confessions led to the rediscovery and reconsideration of his other work in the late 20th and early 21st centuries. Now his novel The Three Perils of Woman is also considered a classic and all his work, including his letters, is undergoing major publication in the Stirling/Carolina editions. However, Justified Sinner remains his most important work and is now seen as one of the major Scottish novels of its time, and absolutely crucial in terms of exploring one of the key themes of Scottish culture and identity: Calvinism. In a 2006 interview with Melvyn Bragg for ITV1, Scottish novelist Irvine Welsh cited Hogg, especially The Confessions as a major influence on his writing. Hogg's story "The Brownie Of The Black Haggs" was dramatised for BBC radio 4 in 2003 by Scottish playwright Marty Ross as part of his "Darker Side Of The Border" series. More recently Ross returned to the villain of that story, Merodach, making him the villain of a Doctor Who audiobook, Night's Black Agents(Big Finish Productions 2010.), in which this demonic figure assumes the pose of a Minister of the Kirk!