The Prime Minister is said to have mouthed “Not for long” in response to Sir Lindsay telling him: “In this House, I am in charge.” Mr Johnson is believed to be smarting after being humiliated by the former Labour MP earlier this month.
One Tory minister told the Mail on Sunday Sir Lindsay had gone “way over the top”.
They added: “The PM's got a perfect right to respond to Starmer's jibes about Tory MPs' second jobs by asking for more details about the Opposition Leader's own outside work.”
Another, referring to predecessor John Bercow, suggested Sir Lindsay had developed “Bercow-esque tendencies”.
Nevertheless, others are understood to have been less supportive of the PM.
Sir Roger Gale, the Tory MP for North Thanet, is understood to have told Mr Johnson to “make his peace” with Sir Lindsay, adding: “You should not speak to the Speaker that way – it's not good.”
Mr Johnson’s bitter exchanges with Sir Lindsay came on Thursday after he clashed with Labour leader Sir Keir Starmer over allegations of Westminster sleaze.
JUST IN: Brexit LIVE - Final straw! Frost loses it as Queen dragged into EU row
The Prime Minister had repeatedly challenged his rival to reveal the source of fees paid for legal work before he became Labour leader.
Sir Lindsay told Mr Johnson: "I don't want to fall out about it, I've made it very clear.
“It is Prime Minister's Questions, it's not for the Opposition to answer your questions.
DON'T MISS
Brexit LIVE: Final straw! Frost loses it as Queen dragged into EU row [LIVE BLOG]
Sturgeon warned Scots face £250 hike in bills under SNP dream [INSIGHT]
Nexit fury erupts as Brussels pushes Dutch farmers out with hated rule [ANALYSIS]
"Whether we like it or not those are the rules of the game that we're all into and we play by the rules, don't we?
“And we respect this House, so let's respect the House."
After Mr Johnson attempted to ask again about the issue in a later exchange, the Speaker said: "Prime Minister, sit down.
"I’m not going to be challenged, you may be the Prime Minister of this country but in this House I'm in charge."
It was at this point that Mr Johnson has been accused of muttering his aside.
At the end of the tetchy exchanges between Mr Johnson and Sir Keir, the Speaker said: "I don't think this has done this House any good today.
“I’ll be quite honest, I think it's been ill-tempered, I think it shows the public that this House has not learnt from the other week, I need this House to gain respect but it starts by individuals showing respect for each other."
Sir Keir accused Mr Johnson of being "a coward, not a leader" during the session.
The Speaker later noted: “’Coward' is not what is used in this House."
Sir Keir replied: "I withdraw it, but he's no leader."
Express.co.uk has contacted Downing Street to ask whether Mr Johnson did use the words attributed to him.
Sir Lindsay Hoyle has served as Speaker since 2019.