The 40-year-old actor is the latest star from the series who could be making an exit.
The Affair may be facing more cast shakeups after Ruth Wilson's exit from the show during season four.
According to a source, Joshua Jackson, who plays Cole Lockhart, will also likely not be coming back for the fifth and final season of the series. If the 40-year-old actor does come back, the source heard, "it would only be for one episode."
The source also heard that Jackson was originally not supposed to return to The Affair beyond season three, but eventually agreed to sign back on after being pitched season four. Additionally, the source heard that series creator Sarah Treem has a plan for a potential one episode return of Jackson's character and is still trying to get him back for the final season, but "he left on very distinct terms" and does not want to return.
A separate source tells ET nothing is official and this may not be the last viewers see of Jackson's character.
Jackson's potential exit comes shortly after Wilson left the show earlier this season. Her character, Alison Lockhart, was recently killed off on the show under mysterious circumstances. In an interview with CBS This Morning in August, Wilson confirmed that she won't be back on the show.
"I'm not coming back, because she's dead," the 36-year-old actress said. She also told Gayle King during the interview, "I did want to leave, but I'm not allowed to talk about why."
Later that month, in a New York Times profile, Wilson also vaguely discussed her reasons for leaving, saying, "It isn't about pay parity, and it wasn't about other jobs. But I'm not really allowed to talk about it."
"There is a much bigger story," she added.
The Affair was renewed for its fifth and final season in July. ET has reached out to Showtime and Jackson's reps for comment.
For more on the show, watch the video below.
Reporting by Rande Iaboni.