Joe Giudice says life is a bit of a struggle while his wife, Teresa Giudice, is behind bars.

Teresa has been serving time for fraud in a Connecticut prison since January and is set to be released in December. When the Real Housewives of New Jersey star gets out, Joe is scheduled to begin his own 41-month prison sentence. The plea deal made guaranteed that one member of the couple would be out and free to care for their children.

"Nothing gets easier, you know? It's always a pain in the ass without her here," Joe tells ET exclusively. "You know, when she was here it was a lot easier."

WATCH: EXCLUSIVE: Joe Giudice Opens Up About Teresa's Life Behind Bars: She’s Really Not OK

To get through the days, Teresa's husband has been taking it day by day.

"Few glasses of wine at night. Go to bed. Wake up the next day," he explains.

Despite living with a spouse in prison, Joe confesses that he's no more prepared than he was to face his own sentence, saying, "I mean, nothing's going to prepare you to go to jail. You know what I mean? You just got to go there and do your time."

WATCH: Teresa Giudice Poses With Her Family in Prison Jumpsuit

And while it has been hard for Joe raising the couple's four young daughters -- Gia, Gabriella, Milania, and Audriana -- he knows that Teresa has had a much rougher go of it in her absence.

"I mean, listen. It was hard for me but at least I'm here," he says. "It was much harder for her being not here, you know what I mean?"

Even so, the 43-year-old has been dogged by rumors that he has been seeing other people while his wife is serving her sentence.

PHOTOS: Busted! Hollywood's Most Memorable Mugshots

"No," Joe bluntly refutes the rumors. "I never go on dates. I would never go on a date with anybody. That would be stupid of me, number one, and I am not going to go on a date with anybody. That is just stupid."

Watch the video below, and tune into ET tonight for more of ET's exclusive sit-down with Joe Giudice.

MORE: First Look at Teresa Giudice Since Starting Prison Sentence

The three-part special following Joe and his family on The Real Housewives of New Jersey: Teresa Checks In premieres Sunday, Oct. 11 at 8 p.m. on Bravo.