See below for list of indicators discussed at the workshop.
Background to the workshop
This workshop was run as part of the ARC Linkage project, “Strategic assessment of research performance indicators”. The participants in this project are the Research Evaluation and Policy Project (REPP) at the Australian National University, the University of Queensland, and the Department of Education Science and Training (DEST). The investigators on the project are:
- Linda Butler (REPP, Chief Investigator)
- Grit Laudel (REPP)
- Frank Jackson (ANU)
- David Siddle (UQld)
- Ian Lucas (DEST)
The project was funded for two years and is now drawing to an end. This workshop was set up to help us in one of the major goals of the project - the assessment of a range of quantitative performance measures.
More details on the project can be found at: repp.anu.edu.au/papers/2005_linkage_grant.html.
The aims of the Linkage project
The project was established to provide a knowledge base that supports informed decisions on the use of quantitative indicators of research performance. This knowledge base consists of:
- a comprehensive range of quantifiable performance indicators that could be used to assess research performance, including all those currently used in the Australian higher education sector, and those identified in an extensive literature review;
- an assessment of each indicator in terms of its validity, fairness, transparency, independence, cost effectiveness and behavioural impact;
- an evaluation of the likely effects on institutions, organisational units and individuals of using different performance measures; and
- an assessment of whether the application of field-specific weights to performance measures can provide a solution to inequalities that can arise from field-specific characteristics.
The project focuses on both bibliometric indicators (based on the written output of research) and non-bibliometric indicators.
The study does not attempt to identify a single ‘best-practice’ list of performance measures for use in the higher education sector. A basic premise of the project is that the ideal measures to be applied in any context will vary according to institutional settings, management priorities, and the basic purpose of the exercise in which they are being deployed. For example, measures to be used in a formula to distribute funds between institutions may have little overlap with measures aimed at identifying the leading researchers in a university. However, in achieving its aims the project will provide analysts with rigorous data on which to make informed judgements on the employment of performance indicators in a variety of common situations.
The role of this workshop
An extensive survey of the literature has been completed, and a copy is provided in this information pack. We have extracted a comprehensive list of suggested quantitative indicators from the literature review. In this workshop, the intention was to focus on a representative selection of indicators, targeting the more novel indicators and/or indicators that are not routinely used in performance assessment.
We sought knowledge and expertise to help in assessing each performance measure. They were assessed in relation to:
- Validity: the extent to which it is an effective surrogate for the characteristics it purports to measure;
- Reliability: the consistency of different evaluation measures in ranking research performance;
- Fairness: the degree to which it accommodates field-specific and institution-specific characteristics;
- Transparency: the extent to which data used can be independently verified;
- Independence: the extent to which the measures are resistant to manipulation by researchers or institutions;
- Cost: how complex it is to obtain the required data, and the expected compliance cost to institutions and government; and
- Behavioural impact: the likely effect it will have on the practice of research units and individual researchers, and whether that impact is in line with desired policy outcomes.
The workshop participants broke into groups in the afternoon session, and each group was given a list of quantitative indicators to assess on the above criteria. Where possible, examples were provided of data that might be used in a performance evaluation, drawing on the experimental database that has been developed for the project.
Assumptions
We would like to state at the outset that it is assumed that no quantitative indicator can be used as a stand-alone measure of research performance. The indicators are to be assessed on the assumption that they will be used alongside peer review.
For this workshop, we focussed specifically on the efficacy of the indicators at the institutional or academic organisational unit level, and did not seek to assess them as measures of individual performance.
What we will not be assessing
There are a number of quantitative indicators that are routinely used in performance assessment and/or have been the subject of considerable debate in the literature. We did not seek to assess these indicators, but instead focussed on those about which our knowledge is less comprehensive.
Next steps
The final phase in this project is to undertake a number of case studies to assess the indicators using actual data and a sample of institutions in a particular discipline. We have in train a case study in economics in which we are developing the methodology to undertake this phase. Once the process has been refined, we plan to extend our assessment to encompass as many as possible of those disciplines not well served by standard measures.
The indictors considered in the work groups were:
Indicator 1: Citations to non-source publications
Indicator 2: Journal publications classifed by impact quartiles
Indicator 3: Identification of highly cited publications
Indicator 4: Book publications weighted by publisher prestige
Indicator 5: Honours, awards and prizes
Indicator 6: Election to learned academies and professional academic associations
Indicator 7: Conferences
Indicator 8: Service to journals
Indicators 9a and 9b: Visiting Fellowships
Indicator 10: Office Bearers in learned academies and professional academic associations
Indicator 11: Copyright Agency Limited (CAL) data
Indicator 12: Membership of grants committees
For further information please contact:
Linda Butler
Research Evaluation and Policy Project
Research School of Social Sciences
The Australian National University
Canberra ACT 0200
T: (02) 6125 2154
F: (02) 6125 9767
E: linda.butler@anu.edu.au
Back to top of page
Back to Home Page
URL: repp.anu.edu.au