In the early 1980s, Roberts suffered from insomnia. Through self-experimentation, he set out to solve this problem by varying aspects of his lifestyle, like exercise and calcium intake. After many failures to see an improvement in his sleep, he eventually discovered that delaying breakfast, seeing faces in the morning light, and standing solved this problem. When Roberts discovers a trend or solution, he typically looks backwards evolutionarily for an explanation. Roberts would later apply this mentality to solving problems in health, sleep, and mood, among other things.

The accuracy of Robert's conclusions have been questioned by scientists who challenge that his experiments lack a control group and are subject to various biases.

Roberts' has been called the "champion of self-experimentation" for his contributions to the field. Professor Tyler Cowen called Roberts' theme of experimenting on the self the "highest stage of science."

As a graduate student, Roberts studied animal learning, specifically "rat psychology." After Roberts read a report by Israel Ramirez studied the effect of saccharin on weight gain in rats, he thought of his new theory "in seconds." Roberts tried about ten different variations, such as eating sushi, eating foods with low glycemic index, and drinking vinegar, before arriving at the Shangri La Diet.

Roberts argues that weight is controlled by maintaining body fat at a certain amount, referred to as a "set point." When weight is above the set point, appetite decreases, and it takes less food to feel full. When weight is below the set point, appetite increases, and it takes more food to feel full. He further states that eating strongly flavored, quickly digested foods (like soda or donuts) can raise the set point, whereas flavorless, slowly digested foods (like sugar water, canola oil, extra light olive oil) can lower the set point. These flavorless foods must be consumed in a "flavorless window," at least one hour after and one hour before consuming flavors. Consumption of 100-500 flavorless calories per day lowers the set point, and therefore, lowers weight.

The name is taken from the fictional Shangri-La, a reference Roberts explains by stating that Shangri-La is a "very peaceful place. My diet puts people at peace with food."

The diet has been endorsed or mentioned by Tyler Cowen, Stephen Dubner, Tim Ferriss, Tucker Max and Wired Magazine writer Gary Wolf. It was criticized by UCLA nutritionist Dr. John Ford.

Book

Through word of mouth, the book became a New York Times bestseller. It made it as high as #2 on Amazon.com's bestseller list. At one point in 2006, Roberts' book was ranked #3 on Amazon while on Freakonomics, a friend and early supporter of the book, it was ranked #4.

The Shangri-La Diet was also featured on Good Morning America where journalist Diane Sawyer tried a tablespoon of olive oil.

In September 2001, Dr. Ranjit Chandra, a prominent nutrition researcher, published a study about the effects of vitamin supplementation on the cognitive functions of the elderly. Roberts and Saul Sternberg, a psychology professor at the University of Pennsylvania, discovered inconsistencies in the data presented, specifically in the data distribution and standard deviation. Roberts is quoted as stating "the results were not just implausible, they were impossible."

The story received recognition in both the [[British Medical Journal]] and the [[New York Times]]. The CBC ran a three-part documentary about the controversy called The Secret Life of Dr. Chandra. In 2005, ''Nutrition'' issued a retraction of Chandra's original paper.