Robert Downey Jr. is taking a walk down memory lane.

The 51-year-old Captain America: Civil War actor was a guest on SiriusXM's The Howard Stern Show on Wednesday, where he reflected on his seven-year relationship with Sarah Jessica Parker.

WATCH: Robert Downey Jr. Reveals His Secret Dad Superpower

In the personal chat, Downey Jr., who dated the Sex and the City star from 1984 until 1991, shares with Stern why things probably didn't work out.

"We were at a very conservative relationship concerning the fact that she was normal and I was out of my mind," he explains. "I did the best I could."

While he was dating Parker, Downey Jr. starred in the war film 1969 with Kiefer Sutherland, which resulted in the trio "kind of" living together for three years.

"I would say, for Kiefer and I, [we] were almost always in a long-term, committed relationship and he is, and remains, a very disciplined guy despite whatever habits either of us had," he jokes. "When we did the movie, I remember he'd be like, 'Robert, it's 6:15, it's time to go to work.'"

NEWS: Robert Downey Jr.'s Daughter Reveals She's Team Iron Man

"He was probably staying with us for a while, and then I was having some trouble with Sarah Jessica, and he was looking after her," he continues. "It was all rather incestuous back then."

Stern started asking Downey Jr. if he was "pissed" about Sutherland trying to go after his girlfriend at the time, but the actor insisted he was just being a good pal.

"It's more that thing, like you're all close friends, and he was probably consoling her a bit," Downey Jr. says. "He didn't steal my girl, he remained in the relationship he was in. He was a shoulder [to cry on]. Better him than some stranger.'

Parker wasn't the only lady that rolled off his tongue during the candid interview, however. He also talked about getting steamy with his Iron Man co-star, Gwyneth Paltrow, something he claims is totally cool with his wife, Susan.

"I'm very happily married. I guess I could dream a little bit," he exclaims. "My free pass is, because her and Susan are such good friends, Paltrow. I gotta get her back in these movies so I can make out with her onscreen again."

When Stern asked if his wife ever gets jealous, he quickly responded, "She could care less."

WATCH: Robert Downey Jr. on Getting Closure With Ex Sarah Jessica Parker