Labour MPs would be wrong to remove Keir Starmer, even though voters have given the party a “real kicking” at the ballot box and people feel “bitterly let down”, the education secretary, Bridget Phillipson, has said.

Phillipson said Labour had been “too gloomy” and had made a mistake in trying to withdraw the winter fuel allowance, after the party lost control of about 40 councils and 1,500 seats at the local elections on Thursday.

However, she said a leadership challenge from the Labour backbencher Catherine West would be “completely wrong”.

West, a former minister, said she would begin collecting the 80 necessary names for a challenge on Monday. She is not widely considered a viable candidate but if she triggers a contest then other contenders such as Wes Streeting, Angela Rayner or Ed Miliband could come forward.

A contest in the coming weeks would rule out a return for Andy Burnham, the mayor of Greater Manchester, who is not in parliament. The Guardian reported on Saturday that MPs from Labour’s left are expected to urge Miliband to consider a leadership bid in the coming days.

About 40 Labour MPs have said they think Starmer should quit or name a date to go. The latest to add his support to West was Josh Simons, the former Cabinet Office minister, who said Starmer must arrange an orderly transition to a new leader as he had “lost the country” and was incapable of “rising to this moment”. Writing in the Times, he said: “To avoid leadership chaos, senior figures across factions should come together to decide the best way forward.”

Facing a threat to his premiership, Starmer gave an interview to the Observer saying he wanted to serve for 10 years. He has also attempted to refresh his government by bringing back the former prime minister Gordon Brown as an adviser on finance, and Harriet Harman as an adviser on women and girls.

However, those in cabinet with leadership ambitions are weighing whether to attempt to oust him.

Phillipson joined other Starmer loyalists in saying it would be the wrong time to change leader.

Speaking on Sky News, she said: “Friday morning, I felt absolutely sick to the bottom of my stomach about the scale of the defeat that we’d suffered. We got a real kicking from the voters. There’s no escaping that, and we have to reflect seriously on that.”

But she added: “I’ve knocked on doors right across the country and in my own community, as colleagues will have done too and as party members will have done as well.

“And what I heard was not a desire for a leadership contest, for the Labour party to spend more time talking amongst ourselves.

“What I heard loud and clear from voters was their deep sense of frustration that they’d voted for change in 2024, they were hopeful that that change would be delivered and they don’t feel that we as a party or we as a Labour government have delivered what they wanted.”

She told BBC’s Sunday With Laura Kuenssberg: “I’m not going to come on here and say that there aren’t big and serious issues that we have to address.

“We can and we will. But I just do not believe that the message we should take from these elections is that we ought to spend time as a party amongst ourselves, arguing amongst ourselves, fighting amongst ourselves.

“We do need to tell a better story. We do need to deliver faster.”

However, Sharon Graham, the leader of the Unite trade union, a significant donor to Labour, said she wanted to see more concrete policies to help working-class voters who had deserted the party.

She said Labour had no right to exist and could become “extinct” unless it changed course.