Peter Gallagher Almost Quit Acting Several Times: 'I Have Fought for Every Role I've Gotten'

He’s starred in an Oscar-winning movie and on pop culture-defining television, but for Peter Gallagher, the acting life hasn’t always come easy.

“Not getting the roles I’ve wanted has been true my whole career,” Gallagher, 64, tells PEOPLE. “I have fought for every role I’ve gotten.”

The New York native, who now stars on NBC’s Zoey’s Extraordinary Playlist as the father of a young woman who can perceive people’s thoughts through music, began acting in high school after auditioning for the school play on a whim.

• For more from Peter Gallagher, pick up this week’s issue of PEOPLE, on newsstands Friday

“I thought, if I can make a living in theater, I will die a happy man,” Gallagher recalls. “I never considered doing movies or TV. It was like, ‘Why don’t you become a bird? It’s not going to happen!’ ”

But Gallagher would go into film, with a breakout role in 1989’s Sex, Lies and Videotape followed by While You Were Sleeping in 1995 and American Beauty in 1999. (He says he based the character of real estate “king” Buddy Kane on Donald Trump!) In 2003, Gallagher landed the role of Sandy Cohen, the principled dad on the teen drama The O.C., which ran for four seasons.

“I’d rather play the ass—- or the bad guy in a movie I love and respect than play some hero in something I couldn’t quite get behind,” says Gallagher of his varied roles. “But there were so many roles I didn’t get. And there were many periods where I was ready to throw in the towel. I’ve lost all my confidence and needed to find it again.”

But now, with the critically- acclaimed Zoey’s (new episodes stream Feb. 17) and a comedic turn on Grace & Frankie, Gallagher, whose daughter Kathryn, 27, is also an actress, currently starring in Jagged Little Pill on Broadway, is feeling on top of the world.

“I was driving home [from work on Zoey’s] the other day and I was just suddenly overwhelmed with feelings,” Gallagher says. “I feel lucky that I get to work. And I feel like someone is on my side for the first time.”