When asked if Simmons ever has friends over, Oliveira's message was clear: "Not at all."
"He has stopped talking to many loving, longtime friends, like Gerry Sinclair, the 95-year-old that is his second mother," Oliveira, who hasn't seen Simmons since April 2014, tells ET. "In fact, [Gerry] went to the house last year and the staff told her that Richard didn't want to see her."
Oliveira explains that this is a big change for Simmons, who was always "a people's person."
"He used to have friends over all the time, dinner parties all the time," he says. "That's why it bogs my mind that he decided to 'completely abandon the world.'"
"He always told me that his fans are the most important people in his life," he adds. "His fans are suffering. They wanted to Richard Simmons to have a formal goodbye."
Oliveira believes that Simmons has been "terribly advised" by the people he surrounds himself with, telling ET, "They are doing a terrible job to take care of his legacy of himself -- to take care of the amazing life that he had helping people."
So what does Oliveira think Simmons needs to do?
"Richard should at least do a little video on YouTube saying, 'I'm fine, thanks for the concern, but I wanted to be alone, please leave me alone,'" he suggests. "He needs to show his face and tell the fans he's OK. By having a formal goodbye, people will stop having speculations about his life and they will leave him alone and leave him in peace."
Oliveira also shares a story with ET about one of his last visits to Simmons' home in 2014.
"I found him in bed at 2 p.m.," he claims. "I went upstairs. He let me in. I said, 'Richard, you need to get up, you need to eat, you need to drink water.' He said, 'No, I want to sleep. Leave me alone.' I said, 'Richard, please, you need to do something. You need to be taken care of.'"
"I saw that he really was very sleepy. He wanted to be left alone, so I respected his privacy. I went downstairs and I knocked on [his housekeeper] Teresa [Reveles'] bedroom," he alleges. "I said, 'Teresa, Richard needs help!' She said simply, 'I don't care.' That's it."
When interest over the Sweatin' to the Oldies icon's disappearance first sparked last March, Simmons exclusively told ET he had decided he wanted to take time to be by himself. "No one should be worried about me. The people that surround me are wonderful people who take great care of me," he added.