"I was there for one week and it feels like I saw enough to be there for a month," said McConaughey, who spoke with ET's Deidre Behar at the premiere of his upcoming animated film, Kubo and the Two Strings, in Hollywood on Sunday.
"My wife and I jammed in as much as we could," McConaughey recounted. "From the beach, Copa Cabana, back to the Olympic Village and back again 10 times."
However, given his wife's Brazilian heritage, the cute couple did occasionally end up cheering on opposing teams, especially when the U.S. faced off against Brazil in beach volleyball -- although the actor was gracious in defeat.
"They kinda buzzsawed us, they won that one pretty cleanly," McConaughey said. "Then we had some swimming events where there was the Brazilians versus the Americans. I came out on top of those."
"I was in the American cheering section with all the other American swimmers, so when a Brazilian was in the race, she [Alves] made sure to get out of that section to go hang out with the Brazilians," he added.
Despite the moments when they were bitter enemies divided by national loyalty, McConaughey and Alves were one of the cutest couples in Rio, concentrating intensely from the stands as they watched the U.S. take on Australia in basketball, or took in the swimming semifinals at the Olympic Aquatics Stadium.
"I kept getting asked by friends, 'What films have you done that your kids love the most?' and I said, 'They haven't been able to watch any yet,'" McConaughey joked. "So that was part of the reason I was looking for something to do with an animated character."
Kubo and the Two Strings is the latest animated offering from stop-motion studio Laika, which created the acclaimed films Coraline, ParaNorman and The Boxtrolls.
The film, which also features the vocal talents of Charlize Theron, Ralph Fiennes, Rooney Mara and George Takei, hits theaters on Friday.