The 'Queer Eye' star is stepping back from Twitter as the controversy continues to swirl.
"First, I have no say who is on the cast and didn't find out till this morning that he is on!" 38-year-old Brown replied. "But I'll tell you this... I'm excited to sit down w/ him and engage in a respectful conversations. Only way things get better is if we try to educate those who have different POV than us."
In another tweet, Brown also wrote that he "can't stand people" who post things "just to flame the fire and get a reaction. We fight harder than your ass ever has for people of color and members of the LGBTQIA community" after someone else called him out.
On Thursday, fellow Queer Eye star Bobby Berk defended Brown in a series of since-deleted tweets.
"As if being on @DancingABC was the stars wasn't already one of the most grueling things anyone can put themselves through, celebs now have to deal with the producers and casting planting political land mines to blow you up. I'm very saddened by what my cast mate @Karamo is having to deal with today."
"We've been trained over and over not to talk smack about anyone in an [interview] and to try and be as nice as possible," he continued. "When being backed in a corner about voicing his opinions about @seanspicer Karamo's media training kicked in as we are trained to do."
Berk also lambasted ABC for the position they put Brown and his DWTS competitors in by including Spicer.
"It's frankly a really [sh*tty] situation to be put in and @ABCNetwork should be ashamed of themselves for having to put the cast in the position of having to take a political side on a f**king dance competition."
Just hours before Brown responded to critics on Wednesday, DWTS host Tom Bergeron candidly shared in a tweet that he disagreed with the show's decision to cast Spicer.
"A few months ago, during a lunch with DWTS' new executive producer, I offered suggestions for season 28. Chief among them was my hope that DWTS, in its return from an unprecedented year-long hiatus, would be a joyful respite from our exhausting political climate and free of inevitably divisive bookings from any party affiliation," Bergeron wrote in a tweet. "I left that lunch convinced we were in agreement."
Referring to Spicer joining the cast, Bergeron added, "Subsequently (and rather obviously), a decision was made to, as we often say in Hollywood, 'go a different direction.'"
Brown and Bergeron aren't the only ones expressing their disapproval of having Spicer on the dancing competition show. The hashtag #BoycottDWTS began trending soon after the announcement on Wednesday. However, Andrew LIinares, the executive producer of DWTS, responded to Bergeron's comments by telling ET in a statement: "We've got a great and diverse cast. We are excited about the season."
Before the pushback over Spicer, ET spoke with Karamo about joining the show, where he shared his excitement about competing for the Mirrorball trophy in the upcoming season.
"This is a show that I've been a big fan of and I am ready to come out here and show that I've got rhythm! Baby, honey! He's ready to do it!" he said. "I'm not having no expectations. I just wanna enjoy myself and have fun and meet new friends."
ET also spoke with Spicer, who explained that he's ready to show a "different side" of himself.
"Most people probably saw one side, which is [me] just standing at a podium tussling with members of the media," the 47-year-old political aide stated. "Now it's an opportunity to see a different side that wasn't shown at that point."
Spicer added that he's in it to win it! "We're going full campaign -- email, social," he continued. "Then we're taking them out one at a time. You know, full TV."
Dancing With the Stars returns for its all-new season Monday, Sept. 16 at 8 p.m ET/PT on ABC.