Britain's highest appellate court, the Supreme Court, said the government was in breach of an EU directive which put limits on nitrogen dioxide (NO2), a colourless, odourless gas produced by burning fuels which can damage people's breathing.
London has the highest levels of NO2 of any European capital. Around 29,000 early deaths a year in Britain are attributed to air pollution, according to a body which advises the British government.
Before deciding on further action, the Supreme Court referred a number of legal questions to the European Court of Justice in Luxembourg, which could take up to 18 months to answer.
The Supreme Court could eventually force the UK government to take certain steps to improve air quality but does not have the power to issue fines, said Alan Andrews, a lawyer at ClientEarth which brought the case against the government in 2011.
The environmental law firm wanted to force the government to come up with an air quality plan to comply with EU limits on N02 concentrations by 2015.
The High Court and the Court of Appeal refused to take action on the issue and the case went to the Supreme Court.
"This historic ruling marks a turning point in the fight for clean air and will pile the pressure on Owen Paterson," said James Thornton, chief executive of ClientEarth, referring to the UK Secretary of State for Environment, Food and Rural Affairs.
"He must now come up with an ambitious plan to protect people from carcinogenic diesel fumes," he added, which are largely blamed for creating NO2.