Brown was born in Dubbo, New South Wales, and was the second son of a solicitor, Samuel Brown, and the grandson of a Methodist Minister, The Reverend E E Hynes. He was educated at state schools in Dubbo before attending Newington College, Sydney, as a boarder in 1963 and 1964. Brown's father had taught at Newington (1932-39) before becoming a legal practitioner and his grandfather was Chaplain at Newington whilst serving in the Stanmore parish. In 1965, Brown entered the University of Sydney to study law and was a resident of Wesley College until 1968 when he went down from the University and became a cadet journalist on the Dubbo newspaper, The Daily Liberal.
Brown was called up for National Service the following year and after Officer Training School graduated as a second lieutenant. He served as a platoon commander of the Third Training Battalion in Singleton, New South Wales.