The 44-year-old DWTS alum told ET on Monday night that, despite the mini stroke that prevented the reality star from making it back in time for competition, it wouldn't have been unfair to the other competitors to make an exception.
"I think the show handled it right," said Alfonso. "I think they handled it right because it could have been Kim going home tonight. And would it have been fair for her not to dance, and still come back next week, and have someone else go home? That wouldn't have been fair."
"So we don't know what the outcome was, but the rules say, you can't go the next week if you don't perform," he continued. "It truly is unfortunate that she wasn't able to make it here to perform."
Alfonso dealt with some trouble of his own that threatened his place on the show while competing in 2014. After suffering a groin injury earlier in the season, the actor injured his back during rehearsals with partner Witney Carson. However, despite the setback, he did ultimately dance through the pain during the show's semi-finals.
"I said, I don't care what happens," recalled Alfonso of dancing with his injury. "I'm gonna go on that dance floor, I'm gonna walk on that dance floor and whatever happens, happens."
Despite this, Tony hopes some kind of social outpouring from fans will get the two of them back on the show.
"I'm hoping the audience tweets or Instagrams and says, 'We want them back,' and maybe that will persuade them," he said, while still acknowledging Kim's health issues. "She did suffer a stroke. This is where you've got to put things in perspective."
"I want to thank everyone for their love and support over the past few days," Kim told ET through a statement from her rep on Sunday. "I am now home with my family, resting and taking care of my health. I can't express enough how much I love dancing and this whole experience."
Watch the video below to see our interview with Tony -- he wants to start a campaign to get her back on the show!