The US commerce secretary, Howard Lutnick, had lunch with the convicted sex offender Jeffrey Epstein on the disgraced financier’s private island, he said on Tuesday, as he faces mounting calls to resign from lawmakers on both sides of the aisle.

“I did have lunch with him, as I was on a boat going across on a family vacation” in 2012, Lutnick said in testimony on Tuesday before the Senate appropriations committee.

Epstein pleaded guilty to state prostitution-related charges in Florida in 2008, including soliciting a minor, in a controversial deal that let him avoid federal charges.

“My wife was with me, as were my four children and nannies,” said Lutnick. “I had another couple with – they were there as well, with their children.”

“And we had lunch on the island, that is true, for an hour,” he added.

The US House member Thomas Massie, a Republican, called for the commerce secretary – a Wall Street billionaire, and longtime ally of Donald Trump – to resign over the weekend. During an appearance on CNN’s Inside Politics, Massie said: “Really, he should make life easier on the president, frankly, and just resign.”

Pointing to how the Epstein scandal has roiled UK politics, costing Andrew Mountbatten-Windsor his title as a prince and Peter Mandelson his role as UK ambassador to Washington, Massie added: “If this were Great Britain, he’d already be gone.”

“Lutnick must resign or be fired,” US House member Robert Garcia, a California Democrat, said in a Twitter/X post on Sunday. “And he must answer our questions.”

Other Democrats, including House members Ro Khanna, Ted Lieu, Melanie Stansbury, as well as Senator Adam Schiff, have also called for Lutnick to step down.

In his testimony on Tuesday morning, Lutnick insisted that he “barely had anything to do” with Epstein. “I’m glad to be here to make it clear that I met Jeffrey Epstein when he moved, when I moved to a house next door to him in New York,” he testified.

“Over the next 14 years, I met him two other times that I can recall, two times,” he said. “So six years later, I met him, and then a year and a half after that, I met him, and never again.”

Lutnick had previously claimed that he distanced himself from Epstein in 2005. But the latest tranche of files shows the commerce secretary discussing visiting the sex offender’s island in 2012 – four years after Epstein was sentenced to 13 months in jail for procuring a minor for prostitution.