You also have to see how Parliamentary Democracies work.
In this system, it's the "Party" that forms the Government, not the PM himself. And the Government can only be brought down by a "non-confidence vote" of the MPs in Parliament.
Labour got a huge majority (almost 2/3 of the seats) last election (July 2024), so the opposition parties don't have enough votes to bring them down.
And Labour isn't about to vote itself out of power, that just doesn't happen in politics.
If he gets to be too much of a liability, what they MIGHT do is get rid of just Starmer (by a simple internal majority vote of all their Labour MP's) and replace him with someone else ... but Labour, as a Party, will stay in power until the next election at least (2028/2029).