The election in North Rhine-Westphalia (NRW), a western German state with a bigger population than the Netherlands and an economy the size of Turkey, was held 18 months before a national vote in which Merkel will be fighting for a third term.
While Merkel remains popular because of the strength of the economy and her steady handling of the eurozone debt crisis, the sheer scale of the defeat in NRW leaves her vulnerable at a time when a backlash against her insistence on fiscal discipline is building across Europe.
According to projections, the centre-left Social Democrats (SPD) won 38.9% of the vote and will have enough to form a stable majority with the Greens.
Merkel's Christian Democrats (CDU) saw their support plunge to just 26.3%, down from nearly 35% in 2010, and the worst result in the state since World War Two.
"This is not a good evening for Merkel," said Gero Neugebauer, a political scientist at Berlin's Free University. "The SPD is strengthened by this election, which will stir things up in Berlin."
Elections in NRW have a history of influencing national politics. Seven years ago, a humiliating loss for then-Chancellor Gerhard Schröder's SPD in the state prompted him to call an early election, which he lost to Merkel.
For the past two years, the SPD and the Greens have run a fragile minority government under the leadership of the SPD's Hannelore Kraft, a tram-worker's daughter with a common touch whose victory on Sunday could propel her to national prominence.
Sigmar Gabriel, national leader of the SPD, said the convincing win could prompt speculation that Kraft would take on Merkel in the federal vote next year, even though she has vowed to stay in NRW. The SPD is due to pick a challenger to Merkel by the end of the year.
"This is a clear signal to Berlin," said Kraft, wiping tears from her eyes in a disco in the state capital Düsseldorf where jubilant SPD supporters held celebrations.
France's newly elected president, Socialist François Hollande, is due to visit Berlin shortly after he is sworn in on Tuesday (15 May) to press Merkel to shift away from austerity and place more emphasis on growth-oriented measures in Europe.
Other big countries like Italy also want Merkel to take a more balanced approach to the debt crisis and an election in Greece last week showed massive public resistance to tough austerity that has pushed unemployment close to 25%.
A Socialist victory in France, coupled with the NRW result, will give the SPD, which trails Merkel in national opinion polls, new momentum before the federal vote in September 2013.
The chancellor needs the support of her centre-left rivals to pass a new "fiscal compact" that is meant to anchor budget discipline across the EU.
But the SPD is pressing her to delay a parliamentary vote on the pact, keen for the government to commit to new growth measures beforehand.
North Rhine-Westphalia: A forerunner?
NRW, which shares a border with Belgium and the Netherlands, is one of Germany's most diverse states.
It is home to one-third of the country's blue-chip companies but also some of its poorest cities. Coal and steel firms in the Ruhr region where Kraft grew up once fuelled Germany's post-war economic miracle. Now many have been shuttered and unemployment in some areas is double the national average.
Many in Merkel's party will blame the result on regional leader Norbert Röttgen, Germany's environment minister, who bungled his campaign early on by refusing to commit to staying in the state in the event of a loss.
NRW is Germany's most indebted state and Röttgen ran on a platform of budget consolidation. Kraft advocated a go-slowly approach to debt reduction, emphasising the need to invest in cities, education and childcare.
The result will be seen by some as a double defeat for Merkel. Voters in NRW not only rejected her party but also the austerity measures that she has forced on struggling southern states like Greece, Spain and Portugal.
"The question arising from this election is whether people still follow Merkel's way of doing politics in Germany," said Erik Flögge, 26, a student and SPD supporter, who attended the party rally in Düsseldorf.
"People don't want us to make hard cuts in social funding, what we want is a 'New Deal' where both the social welfare state and fighting debt will work."
Opinion polls show, however, that a majority of Germans back Merkel's focus on debt reduction and that many don't want her to soften her stance towards struggling euro partners.
Liberals and Pirates
The Free Democrats (FDP), a pro-business party that rules in coalition with Merkel's conservatives at the federal level, scored 8.3% to make it back into the state assembly.
The party ended a string of humiliating regional performances in a state vote in Schleswig-Holstein last week and it hailed the NRW result as proof of a renaissance after a slide in national polls over the past three years.
The upstart Pirates, a party that campaigns for internet freedom and shot onto the national stage last year, continued a strong run at regional level, making it into the fourth straight state parliament with 7.8% of the vote.
The Greens scored 11.8%.