They think tensions would only worsen if the BJP wins a second term in the current general election, as exit polls released on Sunday indicate is likely.
And in a country where 14 per cent of the population are Muslim and 80 per cent Hindu, Nayabans reflects wider tensions in places where Muslim residents are heavily outnumbered by Hindu neighbours.
“Things were very good earlier. Muslims and Hindus were together in good and bad times, weddings to deaths. Now we live our separate ways despite living in the same village,” said Gulfam Ali, who runs a small shop selling bread and tobacco.
A report appearing in international media says Modi came to power in 2014 and the BJP took control of Uttar Pradesh state, which includes Nayabans, in 2017, partly on the back of a Hindu-first message. The state’s chief minister, Yogi Adityanath, is a hardline Hindu priest and senior BJP figure.
“Modi and Yogi have messed it up,” said Ali. “Dividing Hindus and Muslims is their main agenda, only agenda. It was never like this earlier. We want to leave this place but can’t really do that.”
He says about a dozen Muslim families have left in the past two years, including his uncle.
At the end of last year, Nayabans became a symbol of India’s deepening divide as some Hindu men from the area complained they had seen a group of Muslims slaughtering cows, which Hindus regard as sacred.
Angry Hindus accused police of failing to stop an illegal practice, and a Hindu mob blocked a highway, threw stones and burned vehicles. Two people were shot and killed - including a police officer.
Five months later many Muslims, who only number about 400 of the village’s population of more than 4,000, say the wounds haven’t healed.
The BJP denies it is seeking to make Muslims second-class citizens or is anti-Muslim.
“There have been no riots in the country under this government. It’s wrong to label criminal incidents, which we denounce, as Hindu-Muslim issues,” BJP spokesman Gopal Krishna Agarwal said.
“The opposition has been playing communal politics but we believe in neutrality of governance – neither appeasement of any, nor denouncement of any. Some people may be finding that they are not being appeased anymore.”
Some Muslim residents said Hindu hardliners started asserting themselves more in the village after Yogi took office in March 2017.
The atmosphere worsened around the Muslim holy month of Ramadan in 2017 - Hindu activists demanded Muslims stop using a microphone in their madrasa, which also acts as a mosque, to call people to prayer, arguing it disturbed the whole community.
The Muslims reluctantly agreed to stop using the mike and speaker – even though they say it had been operating for many years - to keep the peace, but the move created deep resentment.
“God knows what they are moaning about,” said Hindu elder Om Prakash, a 63-year-old tailor. “There’s peace here but we won’t tolerate any mike there. That’s a madrasa, not a mosque.”
“We can’t express our religion in any way here, but they are free to do whatever they want,” said Muslim law student Aisha, 21.
Aisha remembers when relations were better. “Earlier they would speak very nicely to us, but now they don’t,” said Aisha. “If there was any problem at all, or someone was sick in the family, all the neighbours would come over and help – whether Hindus or Muslims. Now that doesn’t happen.”
Sharfuddin Saifi, 38, who runs a cloth shop at a nearby market, was named in a complaint filed with the police by local Hindus over the cow incident last year.
After 16 days in jail, he was released as the police found he had nothing to do with the suspected slaughter, but said he found much had changed.
Hindus now shun his business. The money he spent on lawyers meant he had to stop going to Delhi to buy stock for the shop, which is largely empty. And he withdrew his 13-year-old son from a private school because he could no longer afford it.
He often thinks about leaving the village, he says, but tells himself: “I have not done anything wrong, why should I leave?”
Carpenter Jabbar Ali, 55, moved to a Muslim-dominated area in Masuri, closer to Delhi, buying a house with money he saved from working in Saudi Arabia.
He still keeps his house in Nayabans and visits occasionally but said he feels much safer in his new home, where all his immediate neighbours are Muslims.
“I’m fearful here,” he said. “Muslims may have to empty out this place if Modi gets another term, and Yogi continues here.”
Junaid, 22, comes from one of the most affluent Muslim families in the village. His father runs a gold shop in a town nearby. He said he wanted to move to New Delhi soon to study at a university there. “Things are not good here,” he said.
But Aas Mohammed, 42, the owner of a flourishing tiles and bathroom fixtures business in a nearby town, has decided to stay in the village, though he has a house on Delhi’s outskirts.
Mohammed helped arrange a lawyer for Saifi after his arrest over the cow incident. He is now lobbying to have the microphone brought back and fighting a legal battle to get a new mosque built.
“I will fight on,” he said. “I am not scared, but another term for Modi will make it very difficult for many other people to live here.”