President Donald Trump expressed frustration with national security adviser Michael Waltz Thursday in his conversations with people about his decision to withdraw Rep. Elise Stefanik’s nomination to be U.S. ambassador to the United Nations, according to two Republican sources with knowledge of the conversations.

Trump’s frustration is two-fold, the sources said. The president is angry that Waltz fumbled a Signal group chat created to coordinate among senior national security officials. According to screenshots published by The Atlantic, a Signal user with Waltz’s name added Jeffrey Goldberg, the magazine’s editor in chief, to the group chat. The episode has created a multi-day controversy for the White House.

But Trump is also annoyed that the race to replace Waltz in Congress seat is shaping up to be more competitive than it should be for Republicans — and next week’s special election in Florida only opened up because Trump selected Waltz for his post in the administration.

Even though GOP leaders are confident the Republican candidate in that special election, Randy Fine, will pull off a win, Trump is still worried that the optics are fueling a negative narrative and making the party look bad, the sources said. Trump is holding tele-town halls Thursday night for Fine and another Republican candidate in a second Florida special election. Had Stefanik been confirmed as U.N. ambassador, it would have created another special election and removed a Republican from the tightly divided House amid efforts to pass Trump’s legislative agenda.

The White House did not comment on questions about Trump’s conversations Thursday.

Trump’s frustration with Waltz comes as a chorus of allies have called for Trump to fire the national security adviser Waltz as a fall guy for the group chat fiasco.

But while Waltz comes in for criticism because Goldberg’s inclusion in the chat ultimately resulted in its publication, others have scrutinized Defense Secretary Pete Hegseth more closely, because he shared details of military plans over a commercial app instead of using traditional government channels for sensitive information.

Sen. Roger Wicker, R-Miss., the chair of the Senate Armed Services Committee, on Thursday asked the Defense Department’s inspector general to open an investigation into the Signal chat and, specifically, whether department policies on classification and sharing sensitive information were broken. Wicker the ranking member on the panel, Sen. Jack Reed, D-R.I., sent the letter to the inspector general together.

“The information as published recently appears to me to be of such a sensitive nature that, based on my knowledge, I would have wanted it classified,” Wicker told reporters on Wednesday.

Yet Hegseth has gotten full-throated backing from Trump and others.

“Hegseth is doing a great job. He had nothing to do with this. Hegseth? How do you bring Hegseth into it? He had nothing to do,” Trump said Wednesday in response to questions from reporters.

In the same media availability, Trump said of Waltz, “Mike Waltz, I guess he said he claimed responsibility,” before adding, “But again, the attacks were unbelievably successful, and that’s ultimately what you should be talking about.”

White House press secretary Karoline Leavitt said Wednesday that Trump “continues to have confidence in his national security team.”

Hegseth has defended himself by telling reporters there was “no classified information” in the chat, a refrain the administration and many other Republicans have adopted as well.

“There’s no units, no locations, no routes, no flight paths, no sources, no methods, no classified information,” Hegseth said Wednesday.

Other MAGA allies have come to Hegseth’s defense, too.

“We dare the media and these radical Democrats to come after Pete Hegseth,” said Steve Bannon, a longtime Trump ally and aide in his first administration, who said attacks on Hegseth We have his back.”

Still, one House Republican lawmaker with experience in national security matters told NBC News Thursday that Hegseth should resign, saying that if Biden-era Defense Secretary Lloyd Austin had done what Hegseth did, “we would have already drawn up, you know, impeachment and asked for his resignation.”