Rumbaugh has a B.S. in physics from MIT, an M.S. in astronomy from Caltech, and a Ph.D. in computer science from MIT.

Rumbaugh worked at Digital Equipment Corporation (DEC) as a lead research scientist. He led the development of OMT while at General Electric Research and Development Center, where he worked for over 25 years. In 1994 he joined Rational Software, were he worked with Ivar Jacobson and Grady Booch ("the Three Amigos") to develop Unified Modeling Language (UML). Later they merged their software development methologies, OMT, OOSE and Booch into the Rational Unified Process (RUP). In 2003 he moved to IBM, after its acquisition of Rational Software. He retired in 2006.

Rumbaugh has written a number of books about UML and RUP together with Ivar Jacobson and Grady Booch. A selection:

• 1991. Object-Oriented Modeling and Design. With others. Prentice Hall, ISBN 0-13-629841-9
• 1996. OMT insights : perspectives on modeling from the Journal of Object-Oriented Programming. Foreword by James Coplien.
• 1999. Unified software development process
• 2005. Object-oriented modeling and design with UML