The 44-year-old DWTS alum told ET on Monday night that, despite the mini stroke that prevented the reality star from making it back in time for competition, it wouldn't have been unfair to the other competitors to make an exception.
"I think the show handled it right," said Alfonso. "I think they handled it right because it could have been Kim going home tonight. And would it have been fair for her not to dance, and still come back next week, and have someone else go home? That wouldn't have been fair."
"So we don't know what the outcome was, but the rules say, you can't go the next week if you don't perform," he continued. "It truly is unfortunate that she wasn't able to make it here to perform."
Alfonso dealt with some trouble of his own that threatened his place on the show while competing in 2014. After suffering a groin injury earlier in the season, the actor injured his back during rehearsals with partner Witney Carson. However, despite the setback, he did ultimately dance through the pain during the show's semi-finals.
"I said, I don't care what happens," recalled Alfonso of dancing with his injury. "I'm gonna go on that dance floor, I'm gonna walk on that dance floor and whatever happens, happens."
Despite agreeing with the show's decision, Alfonso is sad to see Kim go.
"I'm a fan of hers, I wanted to go get on a bus and bring her here," he said. "That bus takes too long!"
Kim and her partner Tony Dovolani, were forced to withdraw from competition after she was not able to fly back to Los Angeles from Atlanta on Monday due to her health scare.
"I could definitely be [doing] better," Tony admitted to ET. "The rules can't be broken, and obviously, because she's not physically here, if she's able or not, we get bowed out."
"I found out during the show, so that was a little heart-wrenching," he said. "The producers tried everything they could to try to keep us in the competition."
Despite this, Tony hopes some kind of social outpouring from fans will get the two of them back on the show.
"I'm hoping the audience tweets or Instagrams and says, 'We want them back,' and maybe that will persuade them," he said, while still acknowledging Kim's health issues. "She did suffer a stroke. This is where you've got to put things in perspective."
"My hat's off to her. She handled this entire situation with grace," said Tony."I'm very proud of her for that. All of us wish her a speedy recovery."
Even Kim's competitor, Carlos PenaVaga, said he wouldn't mind Kim getting a second chance.
"I really feel badly for her," he said. "She couldn't control what happened."
"Honestly, if they gave her a second shot, I'd be OK with it," he continued.
"I want to thank everyone for their love and support over the past few days," Kim told ET through a statement from her rep on Sunday. "I am now home with my family, resting and taking care of my health. I can't express enough how much I love dancing and this whole experience."
Watch the video below to see our interview with Tony -- he wants to start a campaign to get her back on the show!