Research Strategies Australia


Menu Close

Month: August 2015

The ‘moral evaluation gap’

Here is a link to Sarah de Rijcke’s recent keynote at the European Sociological association conference on 

how indicators influence knowledge production in the life sciences and social sciences, and how in- and exclusion mechanisms get built into the scientific system through certain uses of evaluative metrics. 

De Rijcke argues that her findings demonstrate 

that we need an alternative moral discourse in research assessment, centered around the need to address growing inequalities in the science system. 

In de Rijcke’s words, there is an ‘evaluation gap’, a

Discrepancy between evaluation criteria and the social, cultural and economic functions of science

What she means in short is that our focus in research evaluation on measurements of quality have resulted in a system focused on meeting performance targets at the expense of creating a socially responsible research system.

While I agree with this, this assumes that we agree on what the social, cultural and economic functions of science and research are – it is good to talk about economic, social and cultural benefits, but we should first be able to answer the question, Benefits to whom? The social, cultural and economic functions of science are not given, universal nor eternal – one need only look to the case of Vannevar Bush and the Office of Scientific Research and Development, or the tragic case of Lysenkoism to understand this. And while these examples are hyperbole, they make the point – the social, economic and cultural are situated in the historical and political, and by extension so is science functioning in the service of the social, economic and cultural. 

This is not to say that there is a pure realm of research that exists beyond historical and political contingencies. On the contrary, it is to say that science and research are always the product of their time and place.

Those of us working in research evaluation, when we talk about measurement, have to depart from the understanding that what we are trying to measure, in the first instance, are political and historical interests, and that in simply measuring these, as de Rijcke demonstrates, we are going to push science and research in the direction of those interests. We spend a lot of time talking about the social and economic impacts of science, but far less time talking about how the social and economic impact on science.

Correlating SRE allocations with ARC grants

I saw this yesterday and couldn’t help but grab some Australian data to have a look. Below I have simply plotted the project dollars awarded to each Australian university by ARC for the years 2008-2010 against each university’s allocation under the Sustainable Research Excellence (SRE) Threshold 2 component of the Research Block Grants since 2012. The reasons for choosing these data are as follows:

I have added a linear trend line and some 95% prediction intervals – nothing fancy. But that is a good looking line!

SRE Threshold 2 vs ARC projects funding

Now in the UK case, the question has been posed

If the funding allocated to universities on the basis of the REF is correlated to the amount of grant income universities already receive, what is the point of the output assessment process?

But in Australia I tend to think about it differently – yes, those institutions that have received more ARC project funding are more likely to receive significant SRE shares on the back of their ERA outcomes, but isn’t that a good thing? Doesn’t that show that ARC project funding leads to higher quality research outcomes? And doesn’t that provide a decent mechanism for accounting around $900M of public money administered through the ARC last year (not to mention a large proportion of the nearly $1B research funding allocated by the NHMRC which also falls under the ERA/SRE mechnisms).

And this comes at a relatively modest cost to government as the ARC outlined in a 2014 submission to the Senate Economic References Committee Inquiry into Australia’s Innovation System

The financial cost of ERA to government, to January 2014, has been $48.1 million. This has funded the initial trial of ERA in 2009, the full rounds of ERA in 2010 and 2012, and the preparations so far for the upcoming ERA 2015 round.

While that doesn’t account for the compliance costs to the sector, it does seem a rather cost effective verification of the ARC’s allocations of public money.

SRE Threshold 2 vs ARC Project Grant (csv)


Two quick ideas to increase research impact

I was recently invited to participate in the Scholarly Communication Symposium at Griffith University  discussing ‘increasing research impact in sciences’. I couldn’t attend in person, but provided the following input to two questions posed below.

Q: What is one problem that you think needs to be addressed in order to maximise the impact of research in Australia?

A: Lack of support for long term funding of applied research in Australian universities

In Australia, increases in Higher Education Expenditure on R&D (HERD) have been accompanied by increases in applied research. In the period from 1992 to 2012, the shape of the Australian higher education research effort has significantly changed, from a sector characterised by basic research, to one characterised by applied research effort (Figure 1).

In the early 1990s, basic research accounted for 60 per cent (pure basic and strategic basic) of research activity, with applied research comprising only around 30 per cent. Until 2010 the focus of Australia’s universities remained basic research. However, in 2010, the balance shifted, with applied research reaching 47 per cent, overtaking basic research at 45 per cent for the first time.

Figure 1 Australian HERD expenditure by activity 1992-2010

increasing impact in science figure 1

Since the 1990s, project based funding has become a standard form of research activity in Australia. This has in part come at the expense of long term support for research activities. At present levels, 60 per cent of Australia Government support for HERD is delivered through project funding, with the remainder delivered through institution-based funding (Figure 2).

Figure 2 Government funded HERD, 2010

increasing impact in science figure 2

Institutional funding allows for long-term planning of research agendas. The major source of long term funding stability for university research is provided through the Research Block Grants (RBGs). At present, publications and Nationally Competitive Grants (HERDC Category 1) remain the focus of the RBGs, which only include a limited focus on research engagement through the Joint Research Engagement (JRE) funding pool.

The calculation of the JRE includes the following inputs:

  • Research income is weighted at 60 per cent and includes HERDC Category 2 (Other Public Sector Income), Category 3 (Industry and Other Income) and Category 4 (CRC Income) amounts;
  • Student load is weighted at 30 per cent ; and
  • Research publications are weighted at 10 per cent and include HERDC Category Books, Book Chapters, Journal Articles and Conference Papers.

In 2014, the JRE allocation was $342.6m, or 20 per cent of the total RBGs for 2014. In other words, Category 2-4 income accounted for 60 per cent of 20 per cent of the funding for university research.

The relative size of this reward is in stark contrast to the relative importance of Category 2-4 income to the sector. For the years 2008-2010, the proportion of the funding that these categories represented was close to 60 per cent of the HERDC income Categories. There is, in other words, a discrepancy between the focus of universities on engagement activities and the reward and RBG incentives that support long term strategic research planning, which are still reliant upon research publications and Category 1 funding outcomes. This will continue as universities focus more and more on applied research without a commensurate reward mechanism.


Q: What is one possible solution/opportunity to maximise research impact?

A: Make R&D tax incentives available for research in humanities and social sciences

Australia has amongst the lowest levels of direct Government funding for business R&D across comparator countries at 1.8 per cent (Figure 3).

Figure 3 Direct Government funding of business R&D, 2011

increasing impact in science figure 3

At the same time, Australia has some of very high proportions of support for business R&D provided through tax incentives (Figure 4). In fact, Australia has the second lowest level of direct funding to business R&D on the available data, second only to Mexico.

Figure 4 Direct Government funding of business R&D vs tax incentives, 2011

increasing impact in science figure 4

As it stands, tax incentives are not available to research conducted in the humanities and social sciences, which are excluded from the scheme. This disincentive to collaboration with the private sector is of particular note given the large proportion of Australia’s research effort that is conducted in these disciplines. Humanities and social sciences currently receive 16 per cent of Australia’s Category 3 (Industry and Other) income. This represents 16 per cent of what is already invested from the private sector into research engagement with universities that are not considered through the Government’s primary support mechanism.

In addition, excluding humanities and social science researchers from the R&D Tax incentive prevents 43 per cent of the Australian higher education research workforce – who produce around 30 per cent of university research outputs – from participating (Figure 5).

Figure 5 HASS vs non-HASS


Allowing humanities and social science research to be eligible for R&D tax incentives would likely increase the impact of Australia’s research significantly by unlocking new opportunities for collaboration between the private sector and universities.