Mark Hill, a conservative lawyer who served on the local school board, won a divisive mayoral runoff on Saturday in Frisco, Texas, that turned into a referendum on the city’s diversity and a testing ground for the power of anti-Muslim messaging.

Mr. Hill defeated Rod Vilhauer, a retired construction company owner who gained a following on the hard right with his promises to keep “terrorists” from gaining a foothold in Frisco and to prevent Shariah, or Islamic religious codes, from taking precedence over local and federal law.

The nonpartisan race was one of the earliest tests of whether attacks on Islam, increasingly prevalent among Republican politicians and primary candidates this year, would be effective with a broader set of voters.

Mr. Hill’s victory could dampen the enthusiasm among Republican candidates for such messages in general election contests, particularly in places like the suburbs of Dallas that are home to large numbers of Muslim and South Asian residents.

Mr. Vilhauer staked much of his campaign on pushing back against the rapid growth and changing demographics of Frisco, which is about 30 miles from Dallas. He also questioned Islam’s status as a religion.

“When they’re coming at us, you can’t tell me that Islam is a religion,” he said during a podcast interview in March. “It’s a terrorist group.”

Mr. Hill, who describes himself as a conservative Republican, united various opponents of Mr. Vilhauer’s with promises to turn down the political temperature and bring different groups of the city together. He warned that a victory by Mr. Vilhauer could drive people out of Frisco and discourage companies from relocating there in the future.

Once a small community, Frisco has ballooned into a dense and affluent suburb in recent decades. It has attracted corporate offices as well as transplants from other states and immigrants from abroad, particularly from South Asia. Its population of 245,000 is around a third Asian, which is about double what it was a decade ago, while its white population has declined to less than half, according to census figures.

In recent months, Frisco has become a flashpoint among many hard right influencers and activists. Some showed up at contentious meetings of the City Council to rail against local leaders.

Mr. Vilhauer emerged unexpectedly out of a crowded field of candidates in the first round of voting last month, offering a sharp break with the previous administration that had celebrated the city’s diversity.

But voters on Saturday rejected that kind of change, instead choosing Mr. Hill, who promised to repair Frisco’s image as a welcoming place for people from a range of backgrounds, and to focus on city affairs rather than culture war issues.