Some 80 members of Cameron's Conservative Party -- more than a quarter of the total -- defied their leader by voting for the motion, the biggest revolt against a Conservative prime minister on a European issue.
"We understand that many people who voted for it felt very strongly -- and we respect that. However, the government has to do what is in the national interest .... Britain's best interests are served by being in the EU," a statement from Cameron's office said after the vote.
The motion carried no legal weight, but the rebellion, the biggest of Cameron's 17-month-old government, raised the prospect of a return of Conservative divisions over Europe that dogged former Prime Minister John Major in the 1990s.
Cameron had ordered his predominantly eurosceptic Conservative Party to reject the motion, arguing that Britain needs a strong voice in Europe to safeguard its interests as the eurozone tackles a debt crisis.
The disunity embarrasses the prime minister and weakens his hand as he fights to keep Britain at the forefront of talks to reform the eurozone. Cameron wants Britain to remain in the EU while clawing back powers from Brussels.
He is due to attend an emergency European Council meeting on the eurozone crisis on Wednesday (26 October).
Tim Bale, politics professor at Sussex University, called the revolt by Conservatives "a major embarrassment."
"I would have thought there will be some partners in Europe who will be concerned about Britain's long-term commitment to the EU as a whole because of this," he said.
After more than five hours of fractious debate, the referendum call was easily defeated by 483-111 thanks to votes against from the Conservatives' pro-European Liberal Democrat coalition partners and opposition Labour politicians.
Conservatives sharply divided
Supporters of EU membership say Britain does 40 percent of its trade with the eurozone and that leaving the EU could damage the economy.
Eurosceptics dismiss such fears, and say Britain has handed too many powers to the European Union, which they see as undemocratic, over-regulated and a brake on economic growth.
Cameron thought he had done enough to pacify hardline eurosceptics by passing a law promising that no further powers would be transferred from London to Brussels without a referendum, but Monday's vote shows it does not go far enough.
The eurozone crisis has created a dilemma for Cameron's government, which does not want to stump up much money for a eurozone bailout. However, it fears the eurozone could exclude it from decisions on things that matter to Britain, particularly the financial services industry, where London dominates.
Opposition Labour Party leader Ed Miliband labelled the Conservative fissures as a "re-run of the old movie of the out-of-touch Tory (Conservative) Party tearing itself apart over Europe."
Conservative ministerial aide Adam Holloway indicated he was prepared to quit to support the motion, which was triggered by a public petition of more than 100,000 people. At least one other ministerial aide said they were also prepared to risk their job.
"This was such a critical thing really in terms of my reputation, and my relationship with my constituents that I didn't really have a choice," Holloway told the BBC.
COMMENTS
I followed the last three and a half hours of this five and a half hour debate, live, via the BBC. There were a lot of good arguments, easily beating the pro-government position. The measures taken to suppress the MP having a free vote t represent their constituents views will leave a bitter taste for time to come.
This will play into the hands of UKIP and I predict will split the right-wing vote in many constituencies up and down the country in the next General Election. The Conservatives will probably not get a majority again because of it.
The next election will be about the EU, it will be unavoidable and the actions by this government, lead by DC will leave no doubt in the minds of we, the voters, as to which party will definitely give us a referendum - UKIP. In fact the EU will be the main issue right up and until the next election.
Milliband will never be a British PM.
This is a victory for common sense over shameful, dishonest and sensationalist reporting by the British media where the bottom line is not quality editorial content but price for advertising space. Europe is not bad for the UK. It creates a strong market for British goods and provides strong representation in World Trade. All this concession of sovereignty is nonsense. As actors on the world stage all notions of absolute sovereignty are obsolete. Certain issues are purely the within competence of an individual state and others are the collective responsibilty of the Union. Positions that have been negotiated at Intergovermental conferences by the way.
EurActiv reserves its right to remove comments regarded as offensive, racist, or homophobic as well as hate-speech in general. Spamming or posts with an obvious commercial character will be removed as well. Thank you for your understanding.