'Fiery Personalities' and Living Apart: Inside Halle Berry and Olivier Martinez's Split

From almost the start, it had been a rocky romantic road for Halle Berry and Olivier Martinez. Now, as the couple confirm they are splitting after two years of marriage, sources tell PEOPLE clashing priorities and headstrong temperaments ultimately doomed the pair.

“They have had many issues,” a source close to the couple tells PEOPLE. “They both have fiery personalities and don’t like to compromise. Halle has threatened to file for divorce several times after arguments.”

So what was the breaking point? According to the source, one major issue between the couple was their living situation.

When Martinez and a pregnant Berry tied the knot in 2013, their plan was to raise their family – including Berry’s daughter from a previous relationship, Nahla Ariela – in Europe. But when Berry got locked in a custody battle with Berry’s ex-boyfriend, model Gabriel Aubry, the plan changed.

Due to the custody agreement, Berry, 49, had to stay in Los Angeles in order to raise her daughter, forcing Martinez, 49, and their newborn son, Maceo-Robert, to relocate to L.A.

According to the source, Martinez had a hard time adjusting: “Olivier never loved living in L.A. full time. [He] just isn’t happy in L.A.”

Another issue, says another source who knows the couple, was Martinez’s temper.

“He has a violent temper,” the source told PEOPLE this summer. “[Friends say] he has an out of control hair-trigger temper.”

In 2012, Martinez got into a brawl with Aubrey at Berry’s house on Thanksgiving. This year, cameras caught Martinez shoving an empty car seat toward an LAX employee as he, Berry and the kids exited the airport. The employee is now suing the couple for $5 million.

The couple were also apparently at odds over how to balance family life with work, the source told PEOPLE just months before the split.

“Olivier is complaining that he and Halle are not spending much time together and that she is working too much,” said the initial source. “He never wanted her to sign on for [CBS’s Extant] and Olivier was hoping they would focus on family time together after Maceo was born.”

With Berry on the road to promote her acting career, Martinez found himself growing lonely.

“He complains he spends his days alone, or with friends,” said the first source. “He doesn’t understand why Halle wants to work so much and is still so focused on her career. Halle has instead been extremely busy and Olivier, who has not been working, is very bored in L.A.”

The second source confirmed to PEOPLE this summer that Martinez “goes to France a lot. He hasn’t worked in a long time.”

VIDEO: Halle’s Changing Looks!

After it became clear that they were both unhappy in their marriage, they tried to make their relationship work for the sake of their children. Their solution? Spend time away from one another, even living apart for extended periods of time.

“They are both aware that they have difficult personalities and have tried to stay in the marriage by not living together. Many times, Halle [had] been staying at the Malibu house and Olivier at the Chateau Marmont. They are much better when they don’t spent too much time together,” said the first source close to the couple, adding “They have gone weeks without seeing each other, but then has managed to patch things up again.

But with separate living arrangements and Berry’s hectic work schedule, the distance eventually drove them apart.

“Even though they are married, they always had very separate lives,” said the initial source. “Their marriage has never been what Olivier imagined it to be. It’s been a disappointment to him.”