A committee of the 170-member United Nations shipping body was unable to make "tangible progress" after a week of talks that ended late on Friday, delegates say.
An IMO spokeswoman said discussions on market-based measures, such as a levy on CO2 emissions and a cap-and-trade scheme, would resume in October when the Marine Environment Protection Committee meets again.
International shipping accounts for around 3% of global greenhouse gas emissions, but this could rise to 18% by 2050 if regulation is not in place, according to the IMO.
The EU executive has threatened to enforce its own shipping regulations if the IMO fails to act, as it has with aviation.
“While we have a clear preference for global action on measures to reduce emissions from shipping, we don't see the IMO on track to deliver reductions consistent with the globally accepted maximum two degrees Celsius objective,” a Commission spokesman said, referring to a threshold scientists say is needed to avert runaway climate change.
In April, an EU public consultation will close on four policy options, including a compensation fund, an emissions trading system, a fuel or carbon tax and a mandatory emission reduction per ship.
An impact assessment and Commission proposal will follow by June, with a final proposal to be presented in the fourth quarter.
IMO Secretary-General Koji Sekimizu has called for an impact assessment study on nine different market-based proposals to be finalised by 2013.
“Let us work together and set ourselves the challenge of completing all the work on the establishment of a market-based measure by a target year of 2015,” he said in a speech.
But the talks were bogged down by a debate about technical cooperation and technology transfer in relation to an agreement on an energy efficiency design index reached last year, delegates said.
In July, 48 countries voted in favour of adopting a mandatory energy efficiency design index (EEDI) for new ships and a voluntary energy efficiency management plant for all ships.
"There is bad feeling as a result of that," said John Maggs, policy advisor at Seas At Risk, a European association of non-governmental environmental organisations.
"Those who didn't support adoption of the EEDI last year are now causing difficulties for further progress on greenhouse gas measures," he said.