New rules coming into force in Europe this week to shine more light on the $700 trillion (€513 trillion) derivatives markets will take years to produce a clearer picture of these complex products which were at the heart of the financial crisis.
When Lehman Brothers collapsed in 2008 markets were in the dark over a tangle of derivatives on the US investment bank's books. Financial markets froze because of uncertainty about who was exposed to Lehman's derivatives, such as credit default swaps or interest rate swaps. US insurer AIG also ran up big losses linked to derivatives.
In response, politicians and regulators around the world called for action to make risks easier to spot in this opaque part of global financial markets.
The new EU rules, coming in on Wednesday, aim to increase transparency by requiring reporting of transactions.
But even if derivatives deals are captured in this process – a difficult enough task in itself – the data may be too fragmented to give a clear picture of the market and to show where risks might be building up.
In practice, a meaningful snapshot will not emerge for years, when all parties involved are reporting properly and all the data is linked. Until then risks in the markets may go unnoticed again until its too late.
"It is a challenge getting all this information into a format that is useful and usable," said Clive Ansell, head of infrastructure management at the International Swaps and Derivatives Association, the world's main derivatives industry body. "We suspect there will be some gaps and possible inconsistencies on day one."
From Wednesday, everyone who trades derivatives in the EU must report their transactions to one of six new bodies known as trade repositories.
Each trade report will have up to 80 pieces of data to provide regulators with detailed information to help to stop risky derivatives positions building up.
The rules cover private transactions – so-called over-the-counter deals – which make up the bulk of the derivatives business globally. But it also includes derivatives deals done on exchanges, which is where regulators want as many deals as possible done in future to improve transparency.
Analysts expect about 5 million trades from exchanges and about 20,000 contracts traded privately among banks or companies to be reported daily across the EU.
Chris Borg, a financial lawyer at Reed Smith, said regulators appreciated the magnitude of the task and everyone wanted to comply. "However, there is not sufficient clarity or indeed time, to enable them to do so," Borg said.
The United States introduced mandatory reporting in 2012. One US regulator from the Commodities Futures Trading Commission said late last year the data was still a mess.
The United States phased in reporting and requires it from only one party of an off-exchange trade. The EU regime is far broader and going for a "big bang" start with both sides of on- and off-exchange trades having to report from day one.
Not everybody ready
The 14 big banks, which trade about 65% of the world's derivatives, are largely ready in Europe.
But it is a different story for companies, which account for about 5-10% of the market, which use derivatives to insure against adverse price moves in raw materials, currencies or interest rates.
"It's not going to be a smooth start as I would be surprised if the majority of companies will be ready on day one," Michelle Price, a policy director at the Association of Corporate Treasurers, said.
Exchanges – such as Deutsche Boerse – wanted more time to prepare but were refused.
Everyone needs a so-called legal entity identifier or LEI, a unique code for regulators to identify who is trading but a great many participants will not have one by Wednesday.
"We expect that queue will still be stretching down the street as 12 February comes and goes," Reed Smith's Borg said.
The Financial Conduct Authority, which regulates the UK derivatives market, the EU's biggest, said reporting will not be perfect on Wednesday but people were busy preparing.
"We recognise these challenges and will be proportionate in our response to any instances of non-compliance. Firms should however recognise that non-compliance could lead to enforcement action," an FCA spokesman said.
Even when everyone can actually report, regulators still will not have that snapshot of risks they need to do their job.
"There are 22 trade repositories that don't talk to each other," said David Wright, secretary general of the International Organisation of Securities Commissions, a global body that brings together market regulators.
Five of the six trade repositories in the EU are backed by exchanges whose main reason for setting them up is to use them to woo customers to trade and clear derivatives.
The London Stock Exchange, CME, Warsaw Stock Exchange, Deutsche Boerse with the Madrid Exchange , and ICE see their repositories as simply a way of offering a one-stop service in derivatives.
This proliferation has forced the Financial Stability Board, a global regulatory body backed by the 20 leading economies (G20), to launch a public consultation last week on how to link and make sense of all the repository data globally.
"Even once reporting requirements are in place in all jurisdictions, no single authority or body will have a truly global view of the OTC derivatives market … unless a global aggregation mechanism is developed," the FSB report said.
The DTCC, a clearing house owned by its users such as banks, which operates a repository in the EU, said it will be ready to share information from Wednesday.
"As Europe accounts for the largest share of global derivatives trading, we anticipate seeing a significant increase in the volume of trades reporting once mandatory trade reporting commences," the DTCC said.
Fiona Syer of Deloitte's centre for regulatory strategy said a clear and accurate global snapshot for regulators is at least four years away – nearly a decade since Lehman crashed.
The Group of 20 nations agreed in 2009 that derivatives traded over-the-counter or privately among banks should be centrally cleared and reported to a repository by the end of 2012.
These commitments were taken in the wake of the 2008 financial crisis as a way to curb the kind of market volatility that threatened to send the world economy into a second Great Depression.
The bulk of the world's $700 trillion (€513 trillion) worth of derivatives are traded in London and New York, but while the EU and United States agree on the objectives, it has taken time to finalise the details.