He was born John Rees at Ponterwyd in Ceredigion, to a lead miner and farmer, Hugh Rees, and his wife. Rhys was educated at schools in Pantyffynnon and Ponterwyd before moving to the British School, a recently opened institution at Penllwyn, in 1855. Here Rhys was enrolled as a pupil and teacher, and after leaving studied at Bangor Normal College from 1860 to 1861. Upon leaving Bangor Normal College, Rhys gained employment as headmaster at Rhos-y-bol, Anglesey. It was here that Rhys was introduced to Dr Charles Williams, then the Principal of Jesus College, Oxford University, in 1865. This meeting eventually led to Rhys being accepted into the college, where he studied literae humaniores. In 1869, he was elected to a fellowship at Merton College.

Rhys also travelled and studied in Europe during this period, staying in Paris, Heidelberg, Leipzig, and Göttingen. He attended lectures by Georg Curtius and August Leskien whilst in Leipzig, and it was during this period that his interest in philology and linguistics developed. Rhys matriculated from Leipzig in 1871, and it was around this time that he adopted the Welsh spelling of his name. He returned to Wales as a government inspector of schools, covering Flint and Denbigh, and he settled in Rhyl. Rhys also began to write, with articles on the grammar of the Celtic language and articles on the glosses in the Luxembourg manuscript being printed, the latter in the Revue Celtique. In 1872 Rhys married Elspeth Hughes-Davies.

In 1874 Rhys delivered a series of lectures in Aberystwyth, later published as Lectures on Welsh Philology, which served to establish his reputation as a leading scholar of the Celtic language. This reputation saw him appointed as the first Professor of Celtic at Oxford University in 1877. He was also made a Fellow of Jesus College, Oxford. Rhys was elected bursar of the college in 1885, a position he held until 1895, when he succeeded Dr Hugo Harper as principal.

Rhys also maintained his interest in improving schools. In 1881 he was appointed to Lord Aberdare's departmental committee on Welsh education, whilst in 1889 he served as secretary to a commission established by Sir John Bridge. The commission was charged with examining tithe agitation in Wales. In 1889 Rhys was appointed secretary to the royal commission on Sunday closing in Wales. He also served on two other royal commissions, in 1893 and in 1901, the former concerning Welsh land tenure and the latter Irish university education.

Rhys gained his knighthood in 1907, and in 1911 was appointed to the Privy Council. Rhys was one of the founding Fellows of The British Academy when it was given its Royal Charter in 1902, and after his death the Academy established an annual lecture in his name, the Sir John Rhys Memorial Lecture. The Oxford Dictionary of National Biography declares him to be "foremost among the scholars of his time" in his published fields, noting that "his pioneering studies provided a firm foundation for future Celtic scholarship and research for many decades."