In a new interview with TIME, the 51-year-old comedian actually reveals the "very lovely evening" he had with David Letterman, before the iconic host went off the air in May.
"We had a couple bottles of water, and he answered questions," Colbert dishes. "It was two guys with similar jobs talking shop. And at the end of the night, he showed me how to run the freight elevator, which is how you get down to the theater."
"After that, I went across the street, got myself a cup of coffee and looked at the theater from the outside for about an hour, and I realized that nothing we do right now really matters," he recalls. "I mean, we'll do our best to have a good design and a good logo and a good marquee and hire all the right people and have the right sound and the right guests. But it doesn’t really matter until you go and do it."
Colbert is finally ready to let audiences see the real him, after playing his Colbert Report persona -- a fictional anchorman -- for almost a decade on Comedy Central.
"We've got a series of field pieces, packages that are ways for me to try to figure out who that is, as if I don't know who I am," he explains on how he plans to introduce himself. "The unexamined life can be extremely enjoyable, and who knows if I do know who I am. We're going to see whether I do. I'll have my own suppositions as to what these answers might be from people, and see if their memory of me is the same, or whether the police investigator we hired to investigate me to finds out. We're doing a series called 'Who Am Me?'"
Colbert is used to the naysayers, who predict he won't last actually being himself on television.
"They said, ‘You can’t do a nightly show in character -- it won't last until Christmas,'" he recalls about the early days of the Colbert Report. "And now there's a lot of 'You can't do the show not in character.' Evidently nobody has any belief that I can do anything."
But his longtime friend, former Daily Show host Jon Stewart, has absolutely no doubts about his ability.
"He's a better person than he is a performer, and he's the best performer I've ever worked with," Stewart gushes to the magazine. "I think he's a far more open performer than some of the greats of the past. The idea that we knew Johnny Carson as a person was ludicrous. But somehow it's a demand we have of Stephen, that we understand who he is underneath all of this. Fortunately for him, the foundation of it is wonderful."
Clearly, Colbert himself is entirely comfortable with who he is -- a "nerd" before it was cool.
"'Nerd' now is like someone discovering your favorite band. I was a nerd when nerd was nerd. OK? Alright?" he jokes. "There was no reward. No one catered to us. We weren't a demographic. We were a punching bag and a punch line. There was a movie called Revenge of the Nerds because the nerds needed revenge because of all of the things that were happening to them. That's a cultural artifact that people need that people need to understand. Revenge of the Nerds is proof that nerding has changed."
All jokes aside, Colbert is grateful that his huge new late-night gig has come at a mature point in his life.
"I feel very lucky that I got this kind of gig as old as I was," he surprisingly says. "I was 41 before anybody stopped me on the street, so I hope I had a sense of who I was."
And Colbert is certainly giving fans a star-studded first week on The Late Show. On Monday, it was revealed that George Clooney and Republican presidential hopeful Jeb Bush will be his first two guests, as well as Scarlett Johansson, Amy Schumer, and author Stephen King later on in the week.