Charles Spencer is now happily married to his wife Karen, with whom he shares an 11-year-old daughter, Charlotte Diana – the youngest of his seven children.
However, in a recent conversation with one of his late sister's close friends, the Earl candid talked about how his love life had been "promiscuous" and "unhappy" when he was younger.
Speaking to Prince George's godmother Julia Samuel on an episode of her podcast Therapy Works, the Earl also said he'd previously had an "immature" attitude to marriage.
When Julia asked him: "How do you feel about love and trust, having had such a misshapen start?" The author and podcaster responded: "When I was definitely looking for the wrong things, I was very promiscuous, very unhappy in my love life… I didn't feel I deserved love."
Charles continued: "I had a great love in my early twenties, and I ran from her because it was just too threatening, so I destroyed it and that's the biggest regret of my life, and I've talked to her more recently about that and apologised, and she understands.
"I actually wrote to her mother at the time, when it ended, because she wrote me a very sweet letter, and I said 'I'm just not worthy of her,' I wrote that at the time." "Heartbreaking, isn't it, that young man," Julia commented.
Charles went on to marry his first wife, model Victoria Aitken, neé Lockwood, in 1989, at a ceremony that saw a young Prince Harry act as pageboy.
The former couple share daughters Lady Kitty Spencer, 33, twins Lady Eliza Spencer and Lady Amelia Spencer, 31 and son Louis Spencer, Viscount Althorp, who will inherit the Spencer estate.
After the former couple split in 1997, Charles was then married to Caroline Freud between 2001 and 2007, and they welcomed two children, the Honourable Edmund Spencer, 20, and Lady Lara Spencer, who just turned 18.
Charles was on the podcast to discuss his new book, a memoir called A Very Private School, in which he opens up publicly for the first time about his experience of physical and sexual abuse at boarding school between the ages of eight and 13.
He and Julia also discussed his family, including how angry he felt when his headmaster informed him that his father was getting re-married.
He said: "One of my major memories from the time at prep school was being summoned in the summer of 1976, I'd just turned 12, for the headmaster to tell me that my father was marrying my stepmother, and I was furious."
He went on: "I remember I said to the headmaster, who I was terrified of, 'I can't believe he hasn't got the courage to tell me himself,'" Charles confessed.
Sharing more insight into his dad's state of mind, the Earl continued: "I have no doubt my father loved us enormously, he was actually a very sweet man but...
"After my mother left, I think he limped along for the rest of his life as an undiagnosed depressive and it would have been against his culture, or his knowledge, actually, to have sought help."