Jennifer Aniston, Demi Moore, Charlize Theron, and Drew Barrymore had fun rapport with Letterman, but perhaps none more than Julia Roberts, who was known to plant a kiss on the funnyman's lips every now and again.
"We can all remember of course his relationship with Julia Roberts, you know, and she flirted right back," Shaffer said.
Letterman's not the only one who saw some action on the show. Shaffer remembered Barbara Walters being very friendly to him during one of her earlier appearances.
"The first time Barbara Walters came up she gave me a big kiss," Shaffer said.
Some of Rivers' favorite Late Show moments involved her late mother Joan, who was a favorite of Letterman's.
"She was money in the bank when she would come on this show," Shaffer raved. "What Dave loved in a guest was somebody with whom you could just sit back and let them work, and that was the case with her."
ET first visited Letterman and Shaffer 32 years ago when the show was still on NBC. Even then, Shaffer was shaping his legacy.