Initial partial results allowed Sergei Sobyanin to say he was certain of victory though they showed him just topping the 50% barrier needed to win outright.
Vowing not to let a "single vote be stolen", Navalny said his campaign team's figures showed Sobyanin had fallen short of the mark and that he should face him in a second-round runoff.
The results put Navalny on just under 27% but he said his real support was at 35%. His remarks raise the prospect of a new electoral dispute in Russia after anti-Putin protests stalled last year when the former Soviet spy was re-elected and took a tough line on dissent.
"We do not accept the results that are being announced, and we will not give up a single vote that we received," said Navalny, a 37-year-old anti-corruption campaigner who emerged from anti-Putin protests last year as the opposition's leader.
"I call upon the Kremlin and the Moscow mayor's office to renounce falsifications and take the election into a second round," he said, urging volunteers at polling stations across the capital not to let "a single vote be stolen."
With more than 32% of the vote counted, the early results gave 55-year old Sobyanin 52.09% of votes and Navalny 26.65%. Two exit polls earlier put Navalny on about 30% of votes.
A low turnout of around 33% helped boost his numbers because the young people who form the bedrock of his support voted in droves and there was less mobilisation among elderly, more conservative voters.
Even if Navalny's challenge of the outcome does not succeed, such figures strike a blow for the opposition after a Western-style campaign that appeared to take the Kremlin and Navalny's rivals by surprise with its energy and professionalism.
Political observers say the Kremlin let Navalny run for mayor in the belief that he would suffer a humiliating defeat after being convicted of theft in July and sentenced to five years in prison, pending an appeal.
Allowing Navalny to run - and lose - would have increased Sobyanin's legitimacy, went the argument.
In a highly unusual ruling, a court in the city of Kirov released Navalny on bail the day after he was sentenced to allow him to run in the election in the capital, whose 12 million inhabitants account for more than a fifth of the Russian economy.
A pro-Navalny rally is planned Monday evening on the same Moscow square where he helped lead anti-Putin protests that erupted following allegations of widespread fraud in a December 2011 parliamentary election won by Putin's ruling party.
Navalny aides alleged that Moscow authorities increased the vote for Sobyanin by adding people to voter lists at the last minute and manipulating mobile polling stations designed to let people confined to their homes cast ballots.
Election officials said no major violations had been recorded.
If Sobyanin's first-round win is confirmed, Putin will have a close ally at the helm in Moscow until after the 2018 presidential election, in which the former Soviet KGB spy has not ruled out seeking a fourth term.
Sobyanin, a former head of the presidential administration and ex-governor of Siberia's oil-rich Tyumen region, was appointed mayor by the Kremlin in 2010 for a five-year term.
But he cut his term short and called an early election in a bid to boost his legitimacy and that of the Kremlin, which is trying to strengthen its hold on power after the protests.
Sobyanin is a senior member of the ruling United Russia party, which Navalny has denounced as a party of "swindlers and thieves" - a phrase that caught on over the Internet among opposition-minded Russians.
On 4 September, 52 members of the European Parliament across party lines sent a letter to the Council of Europe, the oledes European institution specialised in human rights, expressing their concern that human rights and the rule of law have increasingly came under attack in Russia.
The MEPs mention increased reports of violence by ultra-nationalistic persons beating up and torturing LGBT (Lesbian, gay, bisexual and transsexual) persons, migrants and ethnic minorities.
Last June Russia's lower house passed a law banning gay "propaganda", a measure that human rights groups say has already fuelled attacks on homosexuals as President Vladimir Putin pursues an increasingly conservative social agenda.
Critics of the anti-propaganda law have said it effectively disallows all gay rights rallies and could be used to prosecute anyone voicing support for homosexuals.
The law, as well as a ban on adoptions of children by same-sex couples, are part of a more conservative course taken by President Vladimir Putin on social issues since his return to the Kremlin in May 2012 (see background).
MEPs write that non-governmental organisations, civil society activists, migrants and LGTB people are increasingly intimidated and threatened both by vigilante groups and the police.
“These events bear witness to a sharp contraction of human rights and the rule of law in Russia, two pillars of European peace and stability. As a member of the Council of Europe, Russia committed itself to ‘accept the principles of the rule of law and of the enjoyment by all persons within its jurisdiction of human rights and fundamental freedoms’ (Article 3 of the Council of Europe Statute),” MEPs write.
The members of the European Parliament ask the high officials of the Council of Europe to raise these concerns “firmly” with their Russian counterparts, and “possibly consider that Russia has seriously violated Article 3 of the Statute”.