Irving Adler was born in Harlem, the third of five children. His parents emigrated to the United States from Poland, with his father coming in 1905 to seek work and his mother following five years later. His father, working first as a house-painter, earned enough money to start a small business selling ice, coal, wood, seltzer, and prohibition beer (less than 1/2 of 1% alcohol). Adler was given the Hebrew name Yitzchak, anglicized on his birth certificate as Isaac. His name was changed to Irving by a school clerk when he entered elementary school. Adler was accelerated in school five times, entering Townsend Harris High School at age eleven and beginning City College when he was fourteen. During his junior year he was awarded the Belden gold medal for excellence in mathematics silver medal for ranking second in the college. He graduated magna cum laude from college in 1931, when he was 18.

Adler began his teaching career with a one-year appointment as a teacher-in-training at Stuyvesant High School. After being licensed as a regular teacher, he taught for three years as a substitute teacher during a period when the Board of Education, in violation of state law, refused to fill vacancies with regular teachers entitled to full benefits. He joined the Unemployed Teachers Association, which filed a suit that resulted in 3,500 teachers, including Adler, being elevated from substitute to regular status in one day.

In the course of Adler's activities in the student peace movement of the 1930s, he met Ruth Relis, a Barnard College student whom he married when she graduated in 1935. Irving and Ruth Adler had two children, Stephen and Peggy.

Adler taught mathematics at various New York high schools during the 1930s and 1940s. He was chair of the math department at Textile High School from 1946 until 1952. He was also an active member of the New York Teachers' Union local of the American Federation of Teachers, and was drafted into a leadership role as a member of its executive board, chairman of the educational policy committee, and then as chairman of the salary and legislative committee.

After President Harry Truman issued an executive order in 1947 calling for loyalty investigations of federal employees, New York State adopted the "Feinberg Law" in 1949 providing for the dismissal of teachers who belonged to "subversive organizations." The New York Teachers' Union won a suit challenging the constitutionality of the Feinberg Law in the New York State Supreme Court, but the decision was reversed on appeal to the federal courts. The United States Supreme Court decided against the teachers in a 6-3 decision in 1952, in a case that became known as Adler vs. Board of Education because Adler was the plaintiff with the earliest name alphabetically.

Before the Feinberg Law was implemented, the New York Superintendent of Schools, William Jansen, began calling in teachers for questioning. Union leaders and active members were asked the same question being asked of those subpoenaed before the House Un-American Activities Committee, "Are you now or have you ever been a member of the Communist Party." On the advice of counsel, most refused to answer on the grounds that the question was a violation of section 26a of the New York Civil Service Law that prohibited questioning civil service employees about their political affiliation. Those who refused to answer the question, Adler among them, were dismissed for "insubordination and conduct unbecoming a teacher." Adler was suspended in 1952 and dismissed in 1954.

In 1967 the U.S. Supreme Court reversed itself in a subsequent case. The teachers who had been fired in the 1950s then sued for reinstatement. Adler was reinstated and retired from the city schools in 1977, with his pension rights restored.

Adler wrote his first book, a text for children called Secret of Light, in 1947 while still working as a teacher. In 1955 he wrote his first book for the John Day Company, Time in Your Life. He published the majority his books with John Day, but published seven books with Alfred A. Knopf under the pen name "Robert Irving," and later published with Golden Press and Doubleday under his own name. He wrote six books a year for many years, mostly books on scientific subjects for the junior-high and high-school levels.

A book Adler wrote for adults in 1958, The New Mathematics (ISBN 0-381-98002-2), was important in the "New Math" curriculum reform movement, and led to his frequent appearances at educational meetings throughout North America.

In 1959, Irving and Ruth Adler together began writing "The Reason Why" series of books about scientific concepts for elementary school children. Adler also wrote The Giant Golden Book of Mathematics, followed by a series of six arithmetic workbooks for grade-school children, aptly named Mathematics - Grade 1 through Mathematics - Grade 6. His workbooks eventually sold about 28 million copies worldwide.

The Adlers moved to Shaftsbury, Vermont in 1960. In 1961, Adler completed his doctorate in mathematics at Columbia University under supervision of Ellis Kolchin. He became the chairman of a committee of Vermont peace organizations that mobilized against atmospheric testing of atomic weapons, led a contingent from southern Vermont to the 1963 March on Washington, and was president of a group called the Vermont-in-Mississippi Corporation that supported civil rights activities in the southern U.S..

Ruth Adler died of cancer in early 1968. Later that year, Irving Adler married Joyce Sparer, a long-time family friend teaching in Guyana who Ruth had suggested he should marry after her death. Irving and Joyce Sparer Adler co-authored Language and Man (1970), after which she pursued her own writings. After the death of Joyce's daughter Ellen, Ellen's three children came to live with them in Shaftsbury in 1977, and Adler retired from writing full-time. Among many other travels, the Adlers gave an around-the-world lecture tour in 1984, speaking at universities in Australia, New Zealand, and several countries in Asia, and Europe.

In the late 1970s Adler turned his attention to the question of phyllotaxis, specifically to the arrangement of plant spirals according to the Fibonacci sequence. His papers in the Journal of Theoretical Biology, serves as the basis for a revival of interest in the subject. Adler has given lectures about phyllotaxis at many universities and conferences in the United States and internationally, including the University of Kansas, The Institute for Advanced Study and West Point.