Keir Starmer has told his cabinet he will not resign as prime minister, saying the threshold for a leadership challenge has not been met.

In comments that effectively dared the health secretary, Wes Streeting, to launch a challenge against him, Starmer said he intended to get on with governing.

“As I said yesterday, I take responsibility for these election results and I take responsibility for delivering the change we promised,” he told a cabinet meeting on Tuesday morning.

“The past 48 hours have been destabilising for government and that has a real economic cost for our country and for families. The Labour party has a process for challenging a leader and that has not been triggered.

“The country expects us to get on with governing. That is what I am doing and what we must do as a cabinet.”

Starmer convened the cabinet as at least 10 more MPs called on the prime minister to set a timetable to depart, taking the total to more than 80. The communities minister Miatta Fahnbulleh became the first minister to quit, on Tuesday morning, with more resignations expected.

Darren Jones, Starmer’s chief secretary, said the prime minister was “listening to colleagues” who were asking him to set out a timetable for departure but would make his own decisions about the way forward.

He warned the prime minister’s rivals that it was a “gruelling” job. “Anybody who thinks that they can just walk into the job of prime minister and, like the second coming of the messiah, fix all of our problems probably hasn’t really thought carefully enough about how difficult it is,” he said.

Fahnbulleh, who is close to the energy secretary, Ed Miliband, said she would “urge the prime minister to do the right thing for the country and the party and set a timetable for an orderly transition”. The MP for Peckham said the message on the doorsteps at local elections was that the prime minister had “lost the trust and the confidence of the public”.

The Guardian understands that four senior cabinet ministers – Shabana Mahmood, the home secretary, Yvette Cooper, the foreign secretary, John Healey, the defence secretary, and the deputy prime minister, David Lammy – were among those who spoke to Starmer on Monday.

Some told the prime minister he should oversee an orderly transition of power after crushing election defeats risked ringing the death knell on his premiership.

Others discussed with Starmer how they should take a “responsible, dignified, orderly” approach to what may follow. Several others, including Richard Hermer and Steve Reed, urged him to fight on.

Overnight, some Labour MPs began to voice public support for the prime minister. One, Neil Coyle, said he was “horrified at the elephant trap colleagues are falling into. Those who claimed council elections were about Keir had nothing to offer local communities.”

Another, Nick Smith, said. “A global security crisis and its economic impact on our country means we need political stability. Unity is strength.”