His father was Rabbi Dr. Joseph Carlebach, director of the Talmud Torah school at Hamburg (1925-1927), chief rabbi of Altona (1927-36) and the last chief rabbi of Hamburg (1936-1941). Joseph Carlebach is still held in great honour by the city of Hamburg and its remaining Jewish community. Part of the university campus in Hamburg was named as the Joseph-Carlebach-Platz since 1990. Upon his 120th birthday in 2003, a Joseph-Carlebach-Preis award was established for Jewish studies, given every two years, by the State University of Hamburg.

The Jewish community and its leading rabbi were deported 1941 from Hamburg to the Jungfernhof concentration camp near Riga in Latvia. Shlomo's father Rabbi Joseph, his mother Charlotte and his sisters Ruth, Noemi and Sara were murdered in a forest near Riga, Latvia in 1942. Shlomo, being the youngest son, survived the Holocaust suffering four years of internment in nine different concentration camps. His older four sisters and brothers were sent to England (his sister Miriam immigrated to Israel at the same time) by their parents and survived the Holocaust and the war.

Rabbi Carlebach talked with the authors of the book Die Carlebachs, eine Rabbinerfamilie aus Deutschland ("The Carlebachs, a Family of Rabbis from Germany", Ephraim-Carlebach-Stiftung[-Foundation] Edit: Dölling & Galitz. Hamburg 1995 ISBN 3-926174-99-4, in German) about his father and the time in the concentration camps. (Some details are in the German Wiki Konzentrationslager Jungfernhof.) Rabbi Carlebach has also written about this period in a biography of his father, Ish Yehudi — The Life and Legacy of a Torah Great. Joseph Tzvi Carlebach (Shearith Joseph Publications, New York 2008)

After the war Rabbi Carlebach was accepted as a student at the Yeshiva Rabbi Chaim Berlin where he became one of the closest disciples of its rosh yeshiva Rabbi Yitzchok Hutner. So much so, that Rabbi Carlebach was selected to write the brief welcoming introductions in Rabbi Hutner's works, the Pachad Yitzchok where he would sign himself as ???? ?? ??? ?' ???? ??? ??"? ?????? : "Shlomo the son of Rabbi Yosef Tzvi (may God avenge his blood) Carlebach"

Rabbi Carlebach was serving as a high school teacher of Torah studies and Talmud in the Yeshiva of Eastern Parkway in Brooklyn, when Rabbi Hutner called on him to become the new mashgiach ruchani of the yeshiva following the departure and long tenure of Rabbi Avigdor Miller in that position.

For most of his tenure Rabbi Carlebach was an extremely successful mashgiach ruchani and began to develop a series of lectures that he eventually published, first in pamphlet form and later in a full series that he would call Maskil Lishlomo.

Rabbi Hutner had always wanted to move to Israel to establish a new yeshiva and, because he had only one daughter, Rebbetzin Bruria David who had no children of her own who would be going with her father, Rabbi Hutner planned to leave the yeshiva he headed in Brooklyn (Yeshiva Rabbi Chaim Berlin and Kollel Gur Aryeh) to his disciples who were to remain behind. The designated new rosh yeshiva ("dean") was to be Rabbi Aaron Schechter and it was assumed that Rabbi Carlebach would continue as the official mashgiach ruchani. However, about three years prior to Rabbi Hutner's death in 1980 a serious dispute arose between Rabbi Carlebach and Rabbi Hutner and his disciples. The end of this internal struggle resulted in Rabbi Carlebach being denied access to the yeshiva. However he refused to relinquish the title of mashgiach ruchani. Rabbi Carlebach attempted to bring the termination of his employment and the manner in which it was done to adjudication with various batei din ("Jewish religious courts") but Rabbi Hutner maintained, after much persuasion by Rabbi Shechter, that he was not subject to such proceedings in this instance; that has been the position of his designated heirs who took complete control of Yeshiva Rabbi Chaim Berlin following Rabbi Hutner's death in 1980. However Rabbi Moshe Feinstein disagreed, and ruled that the Yeshiva must continue to pay Rabbi Carlebach a salary until they come to a Din Torah. Regardless, Yeshivas Rabeinu Chaim Berlin refused to comply, which caused Rabbi Moshe Feinstein to rule that Rabbi Shechter had the status of a lo tzayis dina (one who refuses to comply with a Din Torah), and therefore had lost the ability to make certain claims in Jewish courts. This cost the Yeshiva heavily when they wanted to buy a building on their block and would have definitely had the right to in Jewish courts being that they were the closest to the building, however since they lost their rights in Beis Din (Jewish courts) they lost the building to another Yeshiva.

Following his departure from his former yeshiva, Rabbi Carlebach set about lecturing at various Beis Yaakov schools and seminaries for young women. His eloquent speaking style won him a wide audience and he began to record and sell recorded tapes of his lectures. However, he began to pour the bulk of his energies into writing his life's work in Hebrew, which was to become the five volumed Maskil Lishlomo on the Humash (Pentateuch) that incorporated much of Rabbi Hutner's thought system in the ten volume Pachad Yitzchok. In recent years, Rabbi Carlebach has dedicated much of his time to writing a biography of his father, Ish Yehudi — The Life and Legacy of a Torah Great. Joseph Tzvi Carlebach (Shearith Joseph Publications, New York 2008). He is also working on translating his father's writings into English.

A number of Rabbi Carlebach's children are Orthodox rabbis. His eldest daughter is Professor Elisheva Carlebach who is the Salo Wittmayer Baron Professor of Jewish history, culture and society at Columbia University. His sister Professor Dr. Miriam Gillis-Carlebach heads the Joseph Carlebach Institute of the Bar-Ilan-University, Ramat Gan (Israel).