The New South Wales premier, Chris Minns, says he will not apologise to the Muslim community after police disrupted a group of men praying during a Sydney protest against the visiting Israeli president, Isaac Herzog.

Muslim groups including the Lebanese Muslim Association (LMA) and the Australian Federal Imams Council (Afic) have called on the premier and the NSW police commissioner, Mal Lanyon, to issue a public apology.

Lanyon has apologised only “for any offence that may have been taken” after police dragged away people praying outside Sydney’s town hall on Monday evening.

The sheikh leading the prayer described the police behaviour as “unhinged and aggressive”.

The Australian National Imam’s Council (Anic) confirmed Lanyon had contacted it to apologise, but LMA and Afic said they had not been contacted. Asked on Thursday if he would apologise, Minns said: “Look, I won’t.”

“I don’t do that in an antagonistic way, but I think the circumstances are important, and I genuinely believe … that NSW police, their leadership, their regular officers, would never, ever have disrupted a prayer service, or individual Australians who were exercising their religion unless it was in the middle of a riot,” he told reporters.

_{Sign up: AU Breaking News email}

Bilal Rauf, a barrister and senior advisor to Anic, described Lanyon’s apology as “very qualified”, telling the ABC it was “not really directed at what occurred”.

Rauf said thousands of people from different backgrounds and faith communities had rallied on Monday “to express their concern about the visit of the president and, of course, the unfolding genocide in Gaza”.

“They are there to express that peacefully, and yet they are met with that level of seemingly disproportionate use of force to stifle, curb, and prevent the expression of that opposition,” he said. “It’s very troubling, and it needs to be looked at.”

Lanyon on Thursday told the ABC he had apologised, “but it needs to be put in context, what you can see is the police that had been confronted were moving forward to disperse everyone”.

“We [the police] have an excellent working relationship now with the Muslim community,” he said.

The commissioner has insisted that clashes began when officers started moving on protesters after they attempted to march to state parliament in contravention of anti-protest laws rushed through after the Bondi beach terror attack that killed 15 people at a Hanukah festival.

Lanyon denied on ABC radio any suggestion that police moved to break up the protest because an event attended by Herzog at the nearby International Convention Centre (ICC) was ending at that time.

Minns said he had had “lots of conversations in the last couple of days with members of the Islamic community”. The premier said the minister for social services, Jihad Dib, who is Muslim, had called “half of Sydney in the last 48 hours”.

In a post on social media on Tuesday, Dib said he was “deeply distressed” by footage showing police disrupting prayers. Dib said he’d had “direct conversations” with Minns and Lanyon about the incident.

“The scenes were confronting, and they never should have happened,” he wrote.

Minns said Dib had “said things I don’t agree with”, referring to the post.

“But he’s approaching the situation from the perspective of trying to make it better and repair the relationships,” the premier said. Dib was contacted for comment.

Despite videos showing police punching protesters, the premier has rejected calls for an independent inquiry – including from members of his government – into the force’s response on Monday night.

The upper house Greens member Sue Higginson referred the police’s conduct to the Law Enforcement Conduct Commission.

Human Rights Watch (HRW) said this week that videos it had verified showed police “violently dispersing people kneeling in prayer”.

It said the state government should “investigate the alleged use of excessive force by police and appropriately discipline or prosecute those responsible”.

“The NSW authorities’ adoption and use of unnecessary restrictions on legitimate protest doesn’t increase safety, but opens the door to abuse,” HRW researcher Annabel Hennessy said.

Sheikh Wesam Charkawi, who led the prayers on Monday evening, said in a statement on Thursday: “What does it say when peaceful worshippers needed civilians to shield them during prayer, not from others, but from the police themselves?”

Labor MLC Stephen Lawrence, who attended the rally, has said any investigation should also examine the political and legislative context.

Lawrence has said “major events” powers unsuccessfully challenged by Monday’s protest organisers, alongside the public assembly restriction declaration (Pard) made using laws rushed through after Bondi, had created a “pressure cooker” for police and protesters.

“This incident seems to me to have been a consequence of laws and then a declaration by the police commissioner that prevented those protest organisers going to the supreme court potentially and getting authorisation to march to parliament,” he told the ABC on Thursday.

“Rather, the laws that we passed in late December meant that that effectively was in the hands of the police commissioner. People tend to accept court decisions in the way that they wouldn’t necessarily accept police decisions.”

Asked on the ABC if all officers were recording footage using body-worn cameras, Lanyon said the “vast majority, I’m sure, did.” But the commissioner would not commit to making it public.

“The footage is definitely available; we will review it, it is not ordinarily made public,” Lanyon said.

LMA and Afic were part of a coalition of groups calling for Lanyon to resign over the prayer incident. Lanyon’s tenure, since his controversial appointment, has included the Bondi massacre, the worst terrorist attack on Australian soil, and multiple confrontations between police and pro-Palestine protesters.

Asked on Thursday if Lanyon should resign, Minns said: “No, I’m so grateful he’s in the job.”

“We’re all in an incredibly difficult situation … and I think the kind of bearing and leadership that he’s brought to the job as police commissioner is exactly what we need right now.”

Lanyon was contacted for comment.