A cabinet minister has called for Peter Mandelson to hand back the payout he received after quitting as ambassador to the US last year, as pressure increased on the prime minister to quit for having appointed him in the first place.

Pat McFadden, the welfare secretary, said on Sunday he thought the Labour peer should give back his Foreign Office payout, which is reported to be as much as £55,000. The Foreign Office is understood to be reviewing the payment.

Mandelson quit last year as Washington ambassador after further details came to light about his relationship with the convicted child sex offender Jeffrey Epstein. This week, he said he would stand down from the House of Lords after yet more documents were published, showing the relationship between the two men to have been closer than thought.

Mandelson was reported on Sunday to have received a payment worth three months’ salary when he quit in September. “I think Peter should reflect on that and either return it or give it to an appropriate charity,” McFadden told the BBC’s Laura Kuenssberg.

The Foreign Office said: “Peter Mandelson’s civil service employment was terminated in accordance with legal advice and the terms and conditions of his employment.”

The department is understood to be reviewing the payment – though officials believed at the time they had no legal option but to pay it.

McFadden was a close ministerial ally of Mandelson in the last Labour government, working as his deputy in the business department. He said he had not known anything about his former boss’s relationship with Epstein, and said he had been dismayed by emails that appeared to show Mandelson passing sensitive government information to the late US financier.

“This is someone from whom I’ve sought political advice over the years, someone who I thought I knew well, but when I look at the emails that have been published in the last two weeks, there’s a whole side of that I knew nothing of,” he said.

McFadden insisted Keir Starmer should remain in his job despite rising pressure on the prime minister in the wake of the latest revelations.

“I don’t think it’s good for the country to be changing its prime minister every 18 months to two years,” he said. “It has an economic cost, it has a confidence cost, it has an international reputation cost.”

Steve Wright, the head of the Fire Brigades Union, on Sunday became the first boss of a Labour-affiliated union to call on the prime minister to stand down.

“I think we need to see change,” he told the BBC.

“I think 18 months ago, the general public wanted to see that change, and we’re not seeing it, we’re just seeing a continuation of what happened before. And I think that needs to be a leadership change, and I think MPs need to be calling for that.”

Many MPs, meanwhile, are agitating for Starmer to sack his chief of staff, Morgan McSweeney, who pushed personally for Mandelson to be given the job in Washington.

Rosena Allin-Khan, the Labour MP for Tooting, said on Sunday: “Someone needs to take responsibility, and a lot of the responsibility needs to lie with people who advise [Starmer]. It is well known that there is a boys’ club in No 10 of people who feel as though they can act with impunity, give advice. And it’s not worked out well.”

Another Labour MP added: “The idea of Morgan being allowed to resign makes the PM look even weaker. He should sack him or he will go down with Morgan. The longer he leaves it, the more likely it is the latter.”