Phillips married his first wife actress Penelope Bartley on 30 May 1948. They had four children, Caroline, Claudia, Andrew and Roger, but by 1960 Phillips' work commitments began to take their toll on the marriage. As he put it in his autobiography: "Penny never objected to the money coming in, but she had begun to complain about my absences."
In 1962 Phillips starred in a play called
The Big Killing and was immediately attracted to the understudy Caroline Mortimer, daughter of writer Penelope Mortimer and stepdaughter of Sir John Mortimer. (Before marrying John Mortimer, Penelope had had four daughters by three men). At 19, Caroline was 18 years younger than Phillips however the attraction was mutual. In his autobiography he writes of his dilemma: despite his public image he had been a loyal and faithful husband up to this point and battled with his conscience. Though his wife Penelope had distanced herself from his career and there was a growing distance between them, still he felt torn. However, as the play's run continued he got more deeply involved with Mortimer, eventually meeting her mother.
"I was certain at first that they didn't consider me very suitable ... Penelope Mortimer took a long-nosed view of the kind of light comedy in which I'd had so much success. Perhaps, like many people, she didn't appreciate that from a technical point of view it was often a great deal harder to bring off than straight drama."
Ultimately Phillips was forced out of the family home against his wishes when a friend of his wife conveyed the information of his affair and Bartley decided she wanted a divorce. Phillips was spending more time with Mortimer's family, taking a foreign cruise with them and being a regular guest at the dinner table. As the divorce dragged on, life became difficult for Phillips who was attempting to maintain contact with his children, providing for them and his by now antagonistic wife, and continuing his relationship with Caroline Mortimer, in addition to dealing with the demands of his career. To compound the situation, the same friend who informed Bartley of his affair (and of whom Phillips has expressed his dislike) contrived to have his beloved dog Pippa put down. However, his contact with John Mortimer did at least result in him playing a judge in two of his television dramas. It was truly a turbulent time for the actor as he neared 40. There was respite in happiness at least, as he and Caroline found the then unspoiled island of Ibiza and bought a farmhouse there which played host over the years to Diana Rigg, Nigel Davenport, Laurence Olivier and Denholm Elliott amongst others. While there Mortimer embraced the hippy life and would drift around barefoot, wearing kaftans and bandanas. Although Phillips writes that he was deeply in love with her, by her mid-twenties she was expressing a desire to get married and have children. He cites his reluctance to do either as the cause of the demise of the relationship.
En route to his next job in Australia he was even more baleful when the inflight movie starred no other than Caroline Mortimer. However he soon embarked on a relationship with his co-star Vicki Luke, attracted to her by her sexual energy, attractiveness and unpredictability. After she turned up unannounced on his doorstep in Ibiza, the two embarked on a relationship characterised by risk-taking mainly on her part and which, despite the danger, Phillips found himself drawn to. (There was one instance when, returning from Bangkok, Luke revealed that she had smuggled drugs in Phillips' suitcase).
When he accepted a lead role in the play
Sextet in London, he was immediately drawn to his co-star (over whom he had casting approval) actress Angela Scoular, a former Bond girl. Scoular and Phillips began living together at his house in Maida Vale. Around the same time Penelope Bartley had a stroke which left her seriously debilitated. Phillips and Scoular helped her over the years as she struggled with her condition. Phillips has admitted that due to the time they'd spent together he still felt partly married to Bartley. In 1981 he and Scoular left on a tour of a Ray Cooney play in Australia. While there, his daughter Caroline phoned to say that Bartley had died in a fire. Phillips decided to stay in the production as his departure would have meant its closure, so that he didn't return to Britain until 1982. Although the first thing he did was to visit his ex-wife's grave in Horley, Surrey, he has acknowledged that his family has never forgiven him for not coming back to attend the funeral.
In 1982 Phillips and Scoular, 22 years his junior, were married, and remain together. He has admitted that he has never lost his appreciation for a well-made female figure
Phillips has been a lifelong fan of Tottenham Hotspur F.C. and even makes mention of the fact in his 2006 Autobiography
Hello:
"I was born in Tottenham just before the team scored a goal and they said my scream was louder than those in the ground. I supported Tottenham Hotspur and have done until this day".
