UK ministers are to start removing post-Brexit residency rights from EU citizens who are no longer “continuously” living in the country.

The initiative is legal under the 2020 Brexit withdrawal agreement, but the decision to use travel data to partly determine absences has raised concerns after the HMRC fiasco that saw almost 20,000 parents stripped of child benefits because of inaccurate Home Office border data.

The Home Office said the crackdown was aimed at those who had received “pre-settled status” to remain in the UK before Brexit, a status that applied to anyone who had been in the UK for less than five years.

Officials will start with those believed to have left the country more than five years ago and there will be safeguards including consideration of reasons for prolonged absences.

The Home Office said the crackdown protected public services and was aimed at preventing unlawful immigration by abuse of the system. “In line with the withdrawal agreement, status will only be removed where it is proportionate to do so,” it said in its statement on a government website.

The latest Home Office data shows that of the 6.2 million who applied for UK immigration status after Brexit, 1.4 million are still on pre-settled status.

The Migration Observatory at Oxford University said it was difficult to say for certain how many of the 6.2 million remained in the country, but a combination of census and other data suggested it could be between 3 million and 4 million.

Under the rules those with settled status can be out of the country continuously for up to five years and still retain the right to live in the UK under the Brexit withdrawal agreement of 2020.

For those with pre-settled status, the Home Office said absences can be of any length, as long as they do not collectively amount to more than 30 months in the most recent 60-month period.

The Independent Monitoring Authority for the Citizens’ Rights Agreements (IMA), a statutory body, said it had “expressed concerns” to the Home Office about how the removals would be implemented. It said it was “difficult to know how caseworkers will make individual decisions in practice”.

The3million, which campaigns for the rights of EU citizens in the UK, expressed concern that “unsafe” decisions would be made on the basis of travel data. In a letter to the Home Office, it cited an individual who had applied to upgrade their pre-settled status to settled status who was then questioned by the Home Office on the basis of “obvious inaccuracies” in their travel data.

“Travel data contain journeys that were booked but not taken,” the3million said, adding that the Home Office’s letter to the individual did not recognise the inaccurate travel data even where the “inaccuracy is clearly and easily detectable”.

Their data included “two outbound journeys without any inbound journey between them” and journeys that were made on the same date but with different destinations.

Miranda Biddle, the chief executive of the IMA, said: “We recognise the concern, stress and uncertainty that this situation may cause for affected citizens. We have been engaging with the Home Office to secure assurances about the safeguards it is putting in place and the robustness of its decision-making. The IMA will continue to closely monitor how it implements the new guidance.”

The National Audit Office is investigating HMRC’s use of Home Office data despite clear flaws in their travel records.

An investigation by the Guardian and the Detail found that Home Office data did not always record return journeys by holidaymakers and business people. It also included airline manifests that did not take account of no-shows, a regular occurrence on low-cost airlines who make it difficult to cancel bookings.

The Home Office said it would start with those who had been out of the country the longest. A spokesperson said: “We have been clear that if an individual with pre-settled status has spent less than 30 months in the UK in the most recent 60-month period, they will have ceased to meet the eligibility requirements and may have their status removed.

“The vast majority of pre-settled status holders are completely unaffected by this change and only those with long absences from the UK will be asked to evidence their ongoing eligibility.”