Twelve people have been arrested, including three on suspicion of conspiracy to murder, over a suspected far-right threat against an Islamic event held over the weekend, police have said.

Counter-terrorism police are leading the investigation, which they said was related to “extreme rightwing terrorism” targeting an event held at Shrubland Hall in Suffolk.

Police said there was a “potential serious threat” to the event, which an estimated 15,000 people attended over four days.

Three of those arrested were detained on suspicion of conspiracy to murder, and eight others under terrorism laws. One man arrested is 82. A woman aged 48 was arrested in east London on suspicion of assisting an offender.

The event – UK Itjima, which means “gathering” – brought Muslims together for worship, study and reflection.

The Guardian understands the intelligence that led to the arrests emerged over the weekend. Arrests were made with the event already under way, which sources suggested indicated how late the intervention came. The event closed earlier than planned as a precaution and a major incident was declared.

Ruman Muhith, who helped organisers handle the media, said the gathering had been a “wonderful event” attended by people from all over the world with “a lot of discussions around integration”, and that on Saturday night attenders had “put their England tops on” and gone to the local pub to watch the match against Norway with soft drinks.

He said organisers and the venue owner told him on Sunday that “police had just asked us to finish the event early because there was a credible threat” .

Police said: “Due to the concerns over the possible threat, and as a precaution, advice was provided via Suffolk police to the organisers of the UK Ijtima event to close the event slightly earlier than planned on Sunday 12 July. There is not believed be any wider threat to the public connected to this matter.”

The gathering, organised by a Sunni Muslim missionary organisation, is believed to be one of the largest of its kind held on UK soil.

Counter-terrorism officers made arrests on Sunday and Monday in Greater Manchester, London, Essex, Ipswich and Surrey. They continue to search a series of addresses across Britain as the investigation continues.

Commander Helen Flanagan, the head of counter-terrorism policing London, said: “After becoming aware of a potential serious threat towards the Islamic event in Suffolk, we have moved extremely quickly to make a number of arrests in various locations across the country.

“I know this news may be concerning to the public and particularly those in the Muslim community, given that we believe the intended target was an Islamic event.”

A spokesperson for the Muslim Council of Britain said: “This cannot be treated as an isolated incident. Barely a month after racist pogroms in Belfast and a terror-related incident in Edinburgh that left five men injured, this once again highlights that the threat facing Muslim communities is real, not imagined.

“The MCB calls on ministers to match the language of condemnation with actual action and policy, not another round of statements that change nothing.”