Well, that was awkward.

Stephen Colbert is usually unshakable, but Monday’s interview with Casey Affleck on The Late Show went downhill fast.

WATCH: Jon Stewart Impersonates Donald Trump While Reuniting With Stephen Colbert on 'Late Show'

It all started after Colbert commented on Affleck casual attire, by sarcastically saying, “Thanks for dressing up.”

“I’m not saying you don’t look good,” he continued, causing the Triple 9 actor to call Colbert rude.

The two then started to bicker before Colbert joked that Affleck looked like “a street-corner Jesus.”

WATCH: Stephen Colbert Delivers Emotional Monologue Following Paris Attacks

Things appeared to be cooling off as Colbert started talking about the actor’s new film, Triple 9, until he joked the pair should just the fight physical because of how awkward everything was going.

“We should probably end up fighting,” Colbert joked. Not understanding the sarcasm Affleck replied, “I don’t want to do that.” Turns out neither did Colbert.

Watch the cringeworthy interview below.

Colbert is a seasoned pro and can usual deal with any situation. During one of his first tapings on The Late Show, a group of hecklers started shouting while he sat down with Uber CEO Travis Kalanick.

MORE: Stephen Colbert Dares Donald Trump to Appear as a Guest on 'The Late Show'

"When they initially started shouting, it was just loud and you couldn't really understand them," an eyewitness tells ET exclusively. "When Stephen looked up to listen to them, they calmed down a little bit and explained that they were angry that Uber is taking away jobs from cab drivers and is disrupting the New York City taxi system."

Watch the video below to see how the host took matters into his own hands during the interruption.