EU looks to US model for measuring R&D impact


The EU may piggyback on US efforts to measure the impact of research spending, as Europe struggles to find more reliable ways to assess the return on its investment in science.

A major push is under way in the US to work out how to measure the long-term impact of public support for R&D. Policymakers on both sides of the Atlantic believe a common system would make sense given the global nature of science.

A large chunk of US President Barack Obama's stimulus package was devoted to research, but the administration's focus on transparency and accountability has heightened pressure to demonstrate a return on investment in science.

Dr Julia Lane, responsible for 'Science of Science & Innovation Policy Design' at the National Science Foundation (NSF) in the US, said there is widespread belief that investment in R&D breeds economic growth but this must be demonstrated using a robust performance measurement.

Speaking in the European Parliament on Wednesday (14 April), she said current ways of justifying research spending were too crude and that relying on anecdotal evidence of how science helps society was not good enough.

Many funding agencies simply describe the outcomes of their funding programmes, rather than scientifically measuring them and asking whether the money was invested efficiently.

Global approach needed to 'science of science policy'

However, Lane acknowledged that systems for addressing this problem are currently not well developed. Filling this gap is a problem common to all developed nations with an interest in innovation, she said, suggesting Europe, the US, Brazil, Japan and others should work together to agree a system for measuring return on investment.

The US is working on a new system known as Star Metrics – Science and Technology for America's Reinvestment: Measuring the Effect of Research on Innovation, Competitiveness and Science.

The initial motivation for this is to examine whether job creation has resulted from stimulus spending but Lane wants a more sophisticated system capable of accounting for the knock-on effects of science spending.

This could include direct job creation, helping to create a skilled workforce which benefits the private sector, generating patents, improving healthcare and the environment, she said.

"We talk about these kinds of benefits from science spending but we do so through anecdote rather than data," said Lane.

She said the focus should be on looking at what scientists at all levels do with public supports, adding that US federal support even played a small role in the genesis of companies like Google given that co-founder Sergey Brin was once a graduate student who benefited from taxpayers' money.

Scientists are often reticent when it comes to auditing systems. But the model under development in the US has sparked fears it could hurt basic research and foster a new layer of red tape.

This, says Lane, is not what will happen with Star Metrics. She believes there is enough data generated automatically to design an electronic system which "scrapes" the Internet for relevant information about publications, patents, company registrations and details of where science students work after graduation.

"Brazil has already done a great deal of work on this automated use of existing information, which minimises the burden on scientists but captures large amounts of data," Lane told an audience of MEPs and officials from the European Commission.

Developing this system remains a huge task, she said, but a global effort would help speed up the process.

The first step could be to develop a universal researcher identification system which would help "track and credit" researchers who move between continents.


Greek MEP Niki Tzavela (Freedom & Democracy Group) said that as Europe prepares to negotiate FP8 it has an opportunity to take stock of science policy.

She noted that the European Council has committed to 3% spending on R&D, "with the provision that there are strong monitoring mechanisms in place to ensure that funds are well spent and appropriated in a manner which increases the likelihood of the EU’s strategies producing results".

"Without a thorough understanding by policymakers of the issue surrounding science metrics, there is a danger that the policy we create will not be the right one," said Tzavela.  

Stefano Bertuzzi, Office of the National Institutes of Health (NIH) Director at the US Department of Health said the US has given a major funding boost to research funding agencies in response to the economic crisis.

"This is recognition that scientists work is directly connected to society. It has also sparked increased interest in accountability," he said.

Bertuzzi likened the US stimulus package to Franklin D. Roosevelt's response to the Great Depression in the 1930s. "FDR helped people go back to college – a policy that had knock-on effects for families for generations. I hope for similar stories as a result of our recent Recovery Act," he said.

Measuring the impact of health research is complex, said Bertuzzi, but there have been efforts by economists to look at the increase in life expectancy arising from better understanding of heart disease.

He said public spending on R&D had dramatically increased since the middle of the 20th century but there is no guarantee that it will continue indefinitely.

"We must go beyond anecdote to make our case for continued government support, especially given the competing priorities in areas like agriculture, roads, and education. You need data to buttress your case on what should and should not be funded," Bertuzzi said.

Richard Woodham, Director of R&D and Innovation at Intrasoft International, said there is a great deal of data on EU research spending, dating back to the 4th Framework Programme for Research (PF4).

He said he can see commitment from policymakers and researchers to creating metrics to measure the impact of R&D investment, but pointed to a number of obstacles that must be overcome.

"Data tends to be organised in silos and there is limited means for exchanging information. We need to work on consistency and usability of the existing information," he said.

Woodham believes adopting common standards on data management would help industry to support R&D and assist European companies seeking to expand globally.

Oswald Schröder, Principle Director at the European Patent Office (EPO), said funding fundamental science now is essential in order to have good applied science in the future.

He said vast quantifies of information on patents is available to the public but work is needed to make it more useful. Patent offices in Europe, the US, Japan, Korea and China are collaborating closely and should strive to improve data quality.

"If science has become global, intellectual property rights have to be considered in a similar way if we want solutions for big societal problems," saidSchröder.


Spending on research and development is seen as having a major impact on growth and competitiveness, but proving the precise relationship between investment and the economy is challenging.

The 'Europe 2020' strategy proposed by the European Commission reaffirms a target of spending 3% of GDP on research. The document also calls for a new indicator for tracking innovation.  

Finance ministers, keen to get public spending under control and plug fiscal deficits, resisted the inclusion of the 3% target in the 2020 strategy. They called for a new "outcome-oriented" measure for R&D and innovation (EURACTIV 17/03/10). 

EU leaders adopted the R&D spending goal at a summit in March, but the EU's innovation commissioner has committed to examining new indicators for research and innovation performance (EURACTIV 18/3/10EURACTIV 09/3/10EURACTIV 11/02/10).


  • September 2010: European Commission to publish Research & Innovation plan which will include EU position on measuring return on investment in science

Further Reading

Subscribe to our newsletters


Want to know what's going on in the EU Capitals daily? Subscribe now to our new 9am newsletter.