Why Stephen Baldwin Thinks One of His Brothers Would Fare Better on Special Forces: 'A Little Stud Muffin'

Warning: this story contains spoilers from the two-part season 3 premiere of Special Forces: World’s Toughest Test.

Stephen Baldwin has no doubts about which of his famous brothers would come out as top dog when it comes to competing on Special Forces: World’s Toughest Test.

The 58-year-old actor — who withdrew from the Fox reality show during the premiere of season 3 — detailed why his brother Daniel would fare better than Alec or Billy in the military-style challenges.

"My brother Daniel was the best athlete in the family," Stephen told Entertainment Weekly after his departure from the series. "Billy would do very well. He's like a little stud muffin, but I don't know. I think this show is just so physical. The only person that could ever of my family, that could do well, it would be Daniel."

Alec, 66, is the oldest Baldwin brother followed by Daniel, 64, and Billy, 61. At 58, Stephen is the youngest of the siblings. Daniel has appeared on Hawaii Five-O, The Sopranos, Grimm, Cold Case among other television shows. 

Stephen — who is dad to Hailey Baldwin Bieber and grandfather to Jack Blues —  withdrew from the competition after he requested a medical consultation due to his back and knee injuries after a  helicopter extraction challenge. The decision came after he explained to the show’s doctor that he refused to risk further injury that would put his other work in jeopardy, saying, “It's not worth it to have to go into a hospital for a couple weeks.”

The Usual Suspects star told the outlet that he had no regrets about leaving the competition, sharing, "I had to come back. I had more work. I had other stuff. I'm producing a movie in India later this year, and I'm also getting ready to produce two other projects, a game show and some other podcasts."

Stephen also detailed what he endured during his short time on the grueling reality show.

"The first day was beyond brutal," he recalled. "The 150 pushups we did in between those three events of the first day. It was just completely overwhelming for me physically. That plus my asthma situation just made it too much."

He added that he was "kind of glad I got out of there."

However, Stephen previously told PEOPLE there’s a chance he could appear on another reality show —  his brother Alec’s upcoming series. “Yeah, the new TLC show. We'll see,” he said, joking, “I don't know if they're ready for grandpa yet, but we'll see.”

Alec, 66, and his wife Hilaria Baldwin are gearing up for the launch of The Baldwins, their upcoming reality show documenting life with their seven children — Ilaria Catalina Irena, 2, María Lucía Victoria, 3, Eduardo "Edu" Pao Lucas, 4, Romeo Alejandro David, 6, Leonardo Ángel Charles, 8, Rafael Thomas, 9, and Carmen Gabriela, 11. 

The 30 Rock alum and podcaster, 40, first announced the show in a video posted to Instagram in June.

Never miss a story — sign up for PEOPLE's free daily newsletter to stay up-to-date on the best of what PEOPLE has to offer​​, from celebrity news to compelling human interest stories. 

Special Forces: World's Toughest Test airs Wednesdays at 8 p.m. ET on Fox.