Ousted Kermit the Frog Actor Steve Whitmire Fears Disney Could 'Destroy' Iconic Character

Steve Whitmire voiced Kermit the Frog for 27 years — and now, he’s concerned about the future of the beloved green Muppet.

Whitmire, 57, was let go by Disney in October due to what officials claimed was “unacceptable business conduct.” Family members of the late Muppets creator Jim Henson have also spoken out about the firing, with Jim’s son Brian telling The Hollywood Reporter that Whitmire used his status as the famous frog to make “outrageous demands” and often “played brinkmanship,” which he was told to stop as far back as the mid-1990s.

Whitmire has maintained he was fired over unwanted notes and a union agreement. Now the actor tells PEOPLE that while he was indeed “aggressively giving notes and trying to collaborate,” he insists he wasn’t doing so “in an unfriendly way” — and getting pushed out has been incredibly hard to accept.

“I’ve been with the Muppets since 1978, just short of 40 years, and 27 of those were as Kermit, along with many other characters too,” Whitmire tells PEOPLE. “It’s really been my life’s work. It’s the only thing I’ve done throughout my adult life.”

“I started out as a major Muppet fan and was able to work that into being contacted by Jim Henson,” he explains. “I met a man named Caroll Spinney, who does Big Bird, at a puppetry festival in Atlanta. I was this 18-year-old kid, and he recruited me onto the Muppets at a time when the Muppets were really one of the biggest things on the planet — I mean, they were huge during The Muppet Show, so it was an incredible time to join. It was still quite a small company, and I got to work with the originators of those characters and then had the opportunity, under Jim, to originate many of my own.”

Whitmire says he’s specifically “chosen to not try to refute” the Henson family’s allegations “out of respect for Jim.”

“I just don’t want to launch a counterattack because then it validates what I feel is trying be done here, which is to start a war of words between me and the Hensons,” he says. “I just don’t want to do that. I think people understand that they’ve been out of the picture for such a long time and that what they’re saying is not that imperative to the subject matter, so I’d rather for it to turn back to what happened with Disney.”

A spokesperson for The Muppets Studio issued the following statement to PEOPLE: “The role of Kermit the Frog is an iconic one that is beloved by fans and we take our responsibility to protect the integrity of that character very seriously. We raised concerns about Steve’s repeated unacceptable business conduct over a period of many years and he consistently failed to address the feedback. The decision to part ways was a difficult one which was made in consultation with the Henson family and has their full support.”

Nine months after being let go by the company, Whitmire says it still stings.

“It’s almost been more than a career — it’s a calling for me,” he says. “And to have that pulled away has been pretty tough to deal with. I’ve had nine months to begin to put it into perspective, but it still hurts.”

Matt Vogel, who voiced Kermit’s evil doppelgänger in the 2014 movie Muppets Most Wanted, has been announced as Whitmire’s successor.

“I love Matt — we’ve worked very closely together for a long time,” says Whitmire. “But I think it’s actually unfair of them to ask Matt to do this — and for him to take it. He’s a terrific performer, but the only issue is that he has no connection to Jim in terms of personal relationship. I feel that he’s not the right choice for Kermit.”

“It’s not something that can be taught, you know? Or at least not by anyone who hasn’t gone through it,” says Whitmire. “In an ideal world, what would happen in my case is what happened with Jim. I know the Hensons have said that I refused to have an understudy — and I don’t want to have an understudy, because Jim never had understudies. What an understudy does in the meantime is develop their own characters, and by the time I die or retire, they’re really too busy to do it anyway.”

FROM PEN: Patrick Warburton Opens Up About His Classic Seinfeld Role

Whitmire believes casting the roles of the Muppets shouldn’t be done via auditions, but rather carefully selected “based on a certain level of intuition.”

“When I hear that people are doing this via audition processes, where they march a bunch of people through, it tells me that unfortunately the folks making those choices don’t have any intuition as to who the character would be,” he says. “So they have to treat it like American Idol, like a reality competition show with Kermit as the prize, and that’s just going to destroy the character.”

“I think that there is a calculated hope on the part of the owners that the audience won’t notice [that Kermit has been replaced],” he adds. “And in some cases, that will be true. But the depth of that character is what makes him who he is, and it’s hard to impress upon people who only watch the Muppets from the outside, how important that is and what a hallmark of Jim Henson that was. Consistency is absolutely vital to these characters.”

Disney acquired the rights to the Muppets in 2004, officially acquiring the characters from the Jim Henson Company.

“What’s come down at Disney is that the Muppets have sort of been treated like a box of crayons,” says Whitmire. “They’ve handed that box of crayons off to a lot of really well-meaning people who just don’t understand the characters, and they let those people draw with those crayons.”

At the end of the day, Whitmore insists that his main focus is doing “what’s best for the Muppets.”

“That’s truly all I care about,” he says. “And I really believe that it’s better for them, at this stage, for me to stay involved. I’m willing to do that, because I think these executives at Disney who have taken me out are well-meaning people … and I think they deserve a chance to do what’s best for the characters.”

“I realized something throughout this nine-month process,” he adds. “I mean, I’ve felt every emotion a person could feel: anger, grief, sadness. But what I realized is that Jim worked with a certain group of people because he chose those people to work with him. His creative spirit, it’s not just in this box of puppets that were sold to Disney. It’s really inside of those of us who worked next to him, and I’m carrying that with me wherever I go.”