Rybakov was born in the city of Chernigov, Russian Empire (now Chernihiv, Ukraine) in a Jewish family. In 1934 he was arrested by the NKVD and exiled to Siberia. After the end of his exile, he worked as a transport worker. During World War II, he was a tank commander.

In 1948, he wrote the popular children's book Dirk (??????). In 1950, he published the novel Drivers (????????), then in 1979, the novel Heavy Sand (??????? ?????) about the fate of a Jewish family under Nazi occupation.

Heavy Sand is an epic story of four generations of a Jewish family living in Communist Russia and its life in a ghetto during the Nazi occupation, culminating in their participation in a ghetto uprising. Though the story of the ghetto uprising is fictional, some details of it seem to be based on the uprising in the Warsaw ghetto. It is believed that the novel is based on numerous stories collected by Rybakov from people who survived Nazi occupation of Ukraine. This story was dubbed the "first Russian Holocaust novel" by one of the Western newspapers of the time. The book became a television series in 2008.

His most popular novel Children of the Arbat was written and distributed via samizdat in the 1960's, but was not published until 1987 despite having been officially announced for publication in 1966 and 1978 (in both cases publication was canceled at the very last moment by the Soviet government). The eventual publication of the novel and its sequels - 1935 and Other Years («???????? ????? ? ?????? ????», 1989), Fear («?????», 1990) and Dust & Ashes («???? ? ?????», 1994) - were considered a landmark of the nascent glasnost, as the first in the trilogy was one of the earliest publications of previously forbidden anti-Stalin literature.

Rybakov was a laureate of the USSR and RSFSR state awards. He was awarded the Stalin Prize in 1948 and 1951. He received an honorary doctorate from Tel Aviv University.

Almost all his books have been made into movies. Rybakov’s books have been published in 52 countries, with overall distribution exceeding 20 million copies.

Marina Goldovskaya, a famed Russian-born documentary filmmaker, forged a deep friendship with Rybakov after meeting him at the French Consulate in Moscow. Goldovskaya filmed Rybakov for over a decade; In 2006, seven years after his death, she released her film, a documentary titled The Russian Story (???????? ???????: ???????????).