Katie Maloney-Schwartz Says Lisa Vanderpump Asked Her 'Did You Even Try?' amid Fertility Struggle

Katie Maloney-Schwartz is explaining why a conversation with Lisa Vanderpump about her fertility struggle came off as "invalidating" to her.

On a new episode of her You're Gonna Love Me podcast, the Vanderpump Rules star, 34, talks trying for a baby with husband Tom Schwartz. She says that when several of her costars became pregnant around the same time as each other, she started "feeling like all eyes were on me" with pressure to be next.

"It was a lot. And I wasn't handling it very well," says Katie. "As much as we kept going and kept trying, at the end of last year I was like, 'This is not how I want this to go down. This is just so much pressure and I'm having anxiety even thinking about it.' I just said to Tom, 'Let's just take a break.' ... It was starting to feel forced."

She says she talked it out with Vanderpump, 61: "When I sit down and I'm talking to Lisa and she's like, 'Well, why are you guys going to see a fertility doctor? Did you even try?' That question, I get where it was coming from. It was coming from a good place. But it can feel very invalidating and invasive because it's like, yes I tried. Of course we tried."

"I think what people need to understand is ... I get it, people are curious and they wanna know, but it's rude. It just comes across as rude," she adds. "And I get tired of being asked that, and I get tired of answering that. I just wish Tom and I could just go about this at our own pace."

It feels like people are "breathing down our necks" waiting for a baby announcement, she says.

A rep for Vanderpump did not immediately respond to PEOPLE's request for comment.

Never miss a story — sign up for PEOPLE's free daily newsletter to stay up-to-date on the best of what PEOPLE has to offer, from juicy celebrity news to compelling human interest stories.

On a recent episode of Vanderpump Rules, Katie and Tom visited California Fertility Partners, where they were able to determine whether they could both have children. While Katie received "good news" about her ovaries' follicle count, Tom learned his sperm morphology was "a little bit on the lower side."

"Sperm look like tadpoles and the head of the sperm is where the DNA of the sperm is," the doctor explained. "We want to have normal-shaped sperm heads. Yours was right at 4 percent."

The doctor informed Katie and Tom they "can definitely work on" improving this, instructing Tom to cut back on the amount of alcohol he consumes per week. He was also told to abstain from using marijuana and taking hot baths.