The Portuguese boss has never stayed at a club longer than three years.
And his time at Spain ended on the back of falling out with key players such as Iker Casillas and Sergio Ramos.
Mourinho won one La Liga title, one Copa Del Rey and a Supercopa de Espana during his time at the Bernabeu.
But the Manchester United boss admits his departure came at the right time.
“It was my decision, my daughter wanted to stay at the University of London and I thought it was the right moment to leave,” Mourinho said on French TV station Telefoot.
“It was an important moment for my family.
“There are also professional reasons behind my decision. I reach the best results when I am happy and I was not happy in Madrid anymore.
"I wasn’t happy in my third year there.”
Mourinho has been linked with a move to Madrid's close rivals Barcelona in the past.
But the former Chelsea chief refused to reveal why he has never joined the club he used to work as a coach for.
“Maybe I will reveal the truth one day," he added.
“There is something about my career that I want to keep secret.
“I will tell the truth when I will be really, really, really old.”
Mourinho's United side are performing well this season, after a below-par league campaign last year.
The Red Devils are second, two points behind neighbours Manchester City.
