"I will NOT continue to tolerate such lies about me regarding drug use with alleged meth, heroine, crack - it's not funny!" Carter wrote in a series of tweets.
The singer -- who has said in the past that his small frame is a result of a hiatal hernia
that he was diagnosed with at age 19, and not drug abuse or addiction -- said that the speculation about his weight was "male body shaming."
"This body shaming must end now! It's amazing how many of you who can hide behind your screens and type it, but wouldn't say it to my face," he wrote. "Ending this with the fact that there are so many bullies out there and this male body shaming must continue to be addressed."
Carter railed against his critics earlier in the day as well, sharing a shirtless snapshot of himself,
which he captioned, "Walk away from gossip and verbal defamation. Speak only the good you know of other people and encourage others to do the same."
He also wrote, "Never be bullied into silence. Never allow yourself to be made a victim. Accept no one's definition of your life; define yourself."
While Carter steadfastly denied using "heavy drugs" on Twitter, the singer's rep said on Sunday
that he "holds a medical license for medical marijuana for his long-standing anxiety."
The Backstreet Boys star wrote, "To my brother: I love u no matter what & if u feel the need to reach for help, I am here and willing to help you get better. Family isn't always easy, [but] we're all here for you."
In a statement released through his rep, Carter shot back at his brother's tweet, sharing, "If my own blood (Nick) truly cared about my well-being, why wouldn’t he call me directly and have a conversation instead of making this about him through a very public forum? That’s not cool at all to use me for his PR and kick me while I am down."