Hugh Hefner's Longtime Love Barbi Benton on the Last Time She Saw Him: He Thought He Had 'Plenty of Time Left'

In his final months, Hugh Hefner surrounded himself with a small circle of friends and family — including longtime love Barbi Benton.

Speaking exclusively to PEOPLE in this week's issue, Benton, 67, opens up about her memories of the late publishing pioneer, who died Wednesday at 91.

"I cried all night [when I found out]," she says of his death. "I was a basket case."

Benton met Hefner in 1968 on the set of his TV show, Playboy After Dark. She was 18 and he was 42, and while she initially wasn't interested due to their age difference, the two ended up dating until 1976. She went on to appear on four Playboy covers as the pair traveled the globe in the 1970s before eventually moving into the famed Mansion in Los Angeles together.

Benton, who married L.A. entrepreneur George Gradow in 1979, says she saw Hefner about six months ago.

"As he was getting older, his hearing was diminishing, and he had a hard time talking on the phone," she says. "This started a couple of years ago, so my only contact was in person."

"I went up to the Mansion, and rather than sit where other people were, he wanted to sit with me privately," she recalls of her last visit. "When I was there for a party or a small get-together, I never got to talk to Hef. So we went to one of the rooms in the Mansion, and behind closed doors we had the most amazing conversation about old times."

"We just had a great talk," she adds. "I felt like he kind of knew that it was coming to an end. When I left, he said, 'I hope you come up and see me more often.'"

• For more on the late Hugh Hefner and his legacy, pick up this week's issue of PEOPLE, on newsstands Friday.

Hefner suffered from a long-term back condition that limited his mobility, and according to Benton — who noticed that his memory was "fading" — he admitted to being "in poor health," though he was certain that he would be around longer.

"He didn't think he was going to die anytime in the immediate future," she continues. "He was upset [with reports] that said he didn't have much time left. He said, 'I have plenty of time left! I'm not going anywhere just yet.'"

RELATED VIDEO: The Playboy Mansion — What Happens to the Famed Party Palace & Hugh Hefner's Beloved Home Now?

Benton says that Hefner wasn't afraid of death, but rather "was always humorous" about it.

"He wasn't looking forward to death, but knowing he was going to die — he has the best resting place," she says. "He has a plot next to Marilyn Monroe, so he was looking forward to lying next to the gal who kicked off Playboy. That's kind of the way he looked at death — nothing could be better than his life. And even though his health had deteriorated, he was still living a good life. He still had most of his faculties."

Recalling the first time they met nearly five decades ago, Benton, who was a UCLA student at the time, says Hefner came up to her while she was studying during a break on set.

"He came up to ask me what I was so diligently working on," she says. "I told him I was studying for school. He liked the fact that I was a co-ed and that I went to UCLA. We were talking about my schooling and what I was studying, and he asked me if I would go out with him."

"The first thing he asked me was if would mind if he held my hand," she continues. "I told him I had never been out with anybody over 24 and he said, 'Neither have I!' I looked at him like, is he joking, or is it true?"

Benton agreed to go out with him on a group date to the Candy Store, a disco in L.A.

"One of the main reasons I went is because I absolutely adored Shel Silverstein, and he came too," she admits. "Then Hef asked me out again and again, and before I knew it I was holding his hand all the time. I was also dating someone from UCLA and also I had another suitor, Jimmy Caan. So it was between my college boyfriend, Jimmy Caan and Hef. And Hef won. Within a few months, we were exclusive."

"He was very romantic," she gushes. "He was one of the most romantic men I've ever met. He was always rubbing my neck, holding my hand, squeezing me and making me feel loved. He was also affectionate with other girls … and he loved kissing. I didn't mind, because I was very secure in our relationship."

Benton says their whirlwind romance "was a totally exciting time" — and jam-packed with A-list celebrities.

"Hef loved movies. So every time that he met another movie star, he was tickled pink," she recalls. "He loved being around all these famous people, and of course they wanted to be around him. He was Hugh Hefner! So he had an in to meet anybody, and when he got the Mansion he just invited [everyone]. Everybody who was anybody was there. Warren Beatty was a regular, Sammy Davis, Jack Nicholson, Tony Bennett, Tony Curtis. The list is endless. Anybody that was famous found their way to the Playboy Mansion. It was exciting for me. But I didn't know half of them!"

But Benton, who was barely in her 20s at the time, got to know Hefner outside of the party palace at first.

"When we first dated, I would not go home with Hef, so the only way that he could see me was by taking me out," she says. "We went out quite a bit. There was the Daisy, the Candy Store, the Factory — and he liked to go out to dinner to Trader Vic's."

Over the course of their relationship, Hefner asked Benton to marry him four times.

"The first time he asked me was after I caught him in an affair," she reveals. "I confronted him. He didn't admit it at first, but it was kind of hard not to. I moved out every time!"

"The first three were early on, and the fourth was after I was already married," she adds. "We love each other. I will love him forever."

So why didn't it work out?

"One of the problems is I became a successful singer, and I was on tour much too much," she says. "He just couldn't be faithful that long. The more successful I became, the harder it was. It was hard on our relationship. And when you're hurt, it's not a good time to ask someone to marry them. It was his way of him getting me to come back. Because I moved out every time. By the third time, I moved out for good."

Nevertheless, Benton will always cherish her time with the late media icon.

"When he died, he took a big piece of me with him," she says. "I'm sure that every time I do something, I'll think about having done it with Hef. He had an enormous impact on culture and lives of people all over the world. I'll miss him immensely."

—WITH AURELIE CORINTHIOS