Research Program

The Research Evaluation and Policy Project (REPP) is Australia’s leading centre for the systematic evaluation and mapping of research across all fields of scholarship. REPP attracts significant external funds through conducting bibliometric analyses to assess national and institutional strengths. Its current research projects focus on the need to develop novel qualitative and quantitative approaches to research assessment, and the need to generate ‘metrics’ or indicators sensitive to the research and dissemination practices of a variety of fields not well served by standard bibliometric approaches. With an increasing reliance on the use of quantitative measures in the assessment of research, members of the group play a central role in policy developments both within Australia, and internationally.

Projects

Project Retrosight: Returns on Investment in Cardiovascular Research

Project Retrosight is a three-year multinational study using a framework developed by the team, the Payback Framework, to investigate the impact of cardiovascular research, including how it is translated into clinical practice and ultimately how it effects health. The impacts of biomedical research occur over long time-scales and may take tortuous routes. The Payback Framework was originally designed by researchers at the Health Economics Research Group (HERG) at Brunel University to examine the benefits from health services research. Subsequently its scope was expanded to provide a way of handling the complexity of assessing the impact made by a wide range of biomedical research. The foundations of this work were a series of studies initially supported by the London NHS R&D Office. These were further developed through a number of studies led by RAND Europe, and conducted in collaboration with HERG, notably ‘The Returns from Arthritis Research’, a study for the UK’s Arthritis Research Campaign.

Researchers involved: L Butler and K Henadeera
Researcher collaborators: Martin Buxton and Steve Hanney (HERG), Jonathan Grant and Steve Wooding (RAND Europe)
Project partners: Heart and Stroke Foundation of Canada, National Heart Foundation of Australia

Development of measures of impact and quality for the RQF

REPP’s expertise on performance measures aimed at assessing the quality and impact of Australian university research across the full spectrum of disciplines was acknowledged in the appointment in 2006 of Ms Linda Butler (Metrics) and Dr Claire Donovan (Impact) to chair Department of Education Science and Training (DEST) Working Groups. These Groups were established to develop advice forthe Minister on the design of the Research Quality Framework (RQF). Ms Butler continued to play a central role in developments throughout 2007, undertaking further empirical research into issues that remain unresolved. The REPP team has been coordinating the construction of discipline-specific journal rankings, and has been conducting pilot tests to test the design of proposed citation metrics. More detail on RQF developments and relevant documentation can be found at: www.dest.gov.au/sectors/research_sector/policies_issues_reviews/key_issues/research_quality_framework/default.htm

Researchers involved: L Butler and C Donovan

Strategic Assessment of Research Performance Indicators
(ARC Linkage Project)

The use of quantitative measures of research performance in higher education sends powerful messages to those being evaluated, because the choice of measures signals what is considered most important. However, little critical assessment has been undertaken on the use of performance indicators or their impact on research practice. This project will create a knowledge base on performance measures - their validity, fairness, transparency and impact on research, and the cost of implementation. By assessing an extensive range of measures, it will provide Australian science management and science policy makers with rigorous information on which to base informed judgements on their utilisation.

Researchers involved: L Butler, G Laudel and C Donovan
Researcher collaborators: Frank Jackson (Philosophy, RSSS), David Siddle (University of Queensland) and Ian Lucas (Department of Education, Science and Training)

Impact of Evaluation-Based Funding on the Content of Scientific Research
(ARC Discovery Project, additional funding from BMBF, Germany)

Evidence suggests that evaluation-based funding mechanisms trigger adaptive behaviour in scientists, leading to unintended changes in knowledge production. However, there is no conclusive evidence about when (under what circumstances), how (by what processes) and with what effects researchers adapt to these mechanisms. This study aims to answer these crucial questions. To overcome the methodological weaknesses of earlier studies, a comparative investigation of several research fields and universities will be conducted, combining for the first time: qualitative interviews, ethnographic observations and bibliometric studies. It will enhance Australian research policy with a critical assessment of existing research funding models.

Researchers involved: J Gläser, G Laudel, L Butler
Collaborators: Uwe Schimank, Fern Universität in Hagen, Germany

read more