(CNN)A moderate House Republican is firing off a warning shot at House Minority Leader Kevin McCarthy as he caters to his right flank in a quest for the speaker's gavel.

       "He's taking the middle of the conference for granted," the GOP lawmaker told CNN, speaking on condition of anonymity to discuss internal conference dynamics. "McCarthy could have a bigger math problem with the moderates."

       Specifically, the lawmaker said a number of moderates are upset with how McCarthy, of California, has embraced some of the extremists in the GOP conference and warned it could hurt the party in swing districts and undermine their chances of winning back the majority.

       Marjorie Taylor Greene lays out demands for GOP House speaker vote if Republicans retake majority in 2022

       McCarthy has promised to reinstate Reps. Paul Gosar of Arizona and Marjorie Taylor Greene of Georgia to their committees if the GOP wins back the House. Greene -- who recently laid out a list of demands for her speaker's vote and said she has no respect for the current GOP leadership -- said Friday evening she had spoken to McCarthy by phone, describing it as a "good call." She also said she likes "what he has planned ahead."

       McCarthy said he spoke to Colorado Rep. Lauren Boebert Friday in the wake of her anti-Muslim remarks about Democratic Rep. Ilhan Omar, for which she has since apologized, and is encouraging she meet with the Minnesota lawmaker.

       Read More

       Boebert "has reached out to Congresswoman Omar to meet next week," McCarthy said in a statement. "I spoke with Leader Hoyer today to help facilitate that meeting so that Congress can get back to talking to each other and working on the challenges facing the American people."

       CNN reached out to McCarthy's office multiple times Friday for a comment about Boebert, but they did not provide a statement to CNN until Saturday.

       The moderate GOP lawmaker warned the two wings of the party were headed for a "collision course," and predicted other moderate lawmakers would start speaking out publicly if McCarthy doesn't do more to rein in the fringe members in the party.

       "Our side isn't going to take this much longer," the lawmaker said.

       This story has been updated with additional information Saturday.