- Discovery Project
The aim of this study is to establish whether researchers adapt to funding conditions and, if so, with what consequences for the content of their research. This will be achieved by combining bibliometric analyses at various levels, interview-based case studies of research groups from all major disciplines at various universities, and four in-depth studies applying ethnographic observations of researchers’ adaptative behaviours. While anecdotal evidence strongly suggests that evaluation-based funding mechanisms affect the content of scientific research, it provides no systematic account of how this occurs. There is no conclusive evidence about when (under what circumstances), how (by what processes) and with what consequences researchers adapt to these instruments.
The project’s basic approach is to conduct a series of comparative case studies which are guided by a conceptual framework that combines insights from the new institutionalism and the sociology of scientific knowledge. In order to investigate the causal relationship between institutional mechanisms and the cognitive content of scientific research, we will employ an ‘actor-centred’ neoinstitutionalist analytical framework (Mayntz and Scharpf 1995; Scharpf 1997) This framework, which has been successfully adapted for science studies by one of the CIs (Laudel 1999), allows inclusion of non-institutional factors into explanations, which is of special importance for science studies where any empirical investigation must take into account field-specific characteristics of causation.
The underlying question is one of causes (institutional mechanisms of evaluation-based funding) and effects (content of scientific knowledge). One of the basic lessons learned from laboratory studies, however, is that investigations must be open to non-anticipated factors and patterns of causation. This is especially true for this project, which builds on slender prior knowledge about interactions between institutional conditions of action and the content of scientific research. It is nevertheless useful to record prior assumptions about the field under study and to develop a framework for comparative analyses (Miles and Huberman 1984). The following specification (diagram 1) contains what is assumed about influencing factors and causal relationships.
Institutional arrangements that link the characteristics of research output to the amount of funding for future research exist on at least two levels of the Australian science system. Universities receive that part of their operational grant earmarked for research on the basis of a formula which measures a university’s performance through data on research degrees, external funding and the number of refereed publications. While it is inevitable that scientists are judged and judge themselves by their performance with respect to these criteria, the link between individual performance and government funding is only an indirect one. There is no direct financial feedback from the government’s funding of universities to the individual researcher.
Diagram 1 Assumptions about variables and causal relationships
The impact of the universities’ institutions is the most powerful because it is the university that distributes the funds to researchers. Since the late 1980s, many universities have adopted internal procedures for the allocation of funds that are based on performance evaluation. These internal funding procedures provide a direct link between attributes of a researcher’s results and the money he or she gets for further research. Therefore, internal evaluation-based funding procedures will have a stronger impact on knowledge production, while the funding of universities may add to this impact to varying degrees.
In addition, researchers are funded from a variety of external sources, the most important being the Australian Research Council and the National Health and Medical Research Council. Preliminary results from interviews conducted by one of the CI’s (Grit Laudel) indicate that researcher’s dependence on these sources is rapidly increasing. It has now reached a degree that often makes it impossible to conduct research solely with university funding. This implies that the evaluation procedures adopted by the Research Councils and other funding bodies are also likely to have a significant impact on knowledge production.
The institutions of evaluation-based funding overlap with other institutions, among them general societal institutions and the (mostly informal) institutions of scientific communities. One important intervening institutional condition relates to the diversity of funding sources. Researchers do not depend solely on evaluation-based funding, as some recurrent funding will usually be distributed without evaluation. The scientists’ actual dependence on evaluation-based funding may vary greatly and is one of the crucial influences on possible changes in behaviour.
Another important intervening institutional influence relates to the scientific community. The scientific community’s informal institutions, such as standards for empirical conduct, institutionalised goals, and preferences, exert a strong influence on researchers because they are the prime social environments in which their results are judged. Any evaluation of researchers rests on their performance as perceived by the scientific community. The extent to which criteria of evaluation-based funding mechanisms correspond to these informal institutions determines whether a researcher faces conflicting pressures when seeking funds.
Any empirical investigation of science must take account of non-institutional field-specific characteristics of causation. For example, the cost of research varies greatly between fields and the capacity of researchers to conduct projects solely with untied recurrent funding will fluctuate accordingly. Other field-specific variables that are important in this context include the field’s publication practices and citation rates, its application-orientation (and thus the extent to which funds can be obtained from industry), or simpler criteria such as its size.
The analysis will also take into account important characteristics of actors such as researchers’ goals and interests, as well as their prior scientific success and reputation. These variables affect the degree that researchers are willing and able to adapt to evaluation-based funding mechanisms, or the extent to which they try to, and can, avoid changing their plans while still obtaining the necessary funds.
Mediating actions can be expected to take place on two different levels. Firstly, universities are currently adapting to the evaluation procedures applied to them. Increasing pressure on scientists to apply for grants is accompanied by increasing administrative support for such activities, and many universities are introducing internal mechanisms for evaluation and evaluation-based funding.
On the level of the individual scientists, researchers’ actions, and especially their strategic decisions, are influenced by their anticipation of evaluation procedures and funding. This anticipation can be assumed to be the crucial mediating process. Ethnographic observations of scientific practice have identified the processes that mediate between institutional conditions of research and the knowledge that is produced. Scientists adapt their strategic choices (such as decisions on research problems, objects, methods and instruments, collaborators and publication strategies) to the institutional environment, thus changing their knowledge producing actions. These decisions can only be discerned analytically. Laboratory studies have shown that they are embedded in a stream of choices made by scientists in their everyday work, and that at least some of these are made without consciously considering their impact on knowledge production. Many scientists conduct more than one project at a time and shift resources from several sources between projects. All the research processes must be included in the investigation. Only then will it be possible to consider the role that the different ‘mixes’ of recurrent funding and evaluation-based funding play.
The investigation’s dependent variables cover the content of knowledge produced and the channels through which it is communicated. Several cognitive characteristics of knowledge relating to content will be included:
In exploring the channels of communication of that scientific knowledge, we will look closely at the publication strategies chosen by researchers. We will investigate whether researchers follow consistent publication practices that affect strategic decisions about their research (e.g. do they select projects whose results are quickly and easily publishable?).
The project is designed as a series of comparative case studies, which will apply a combination of qualitative and quantitative methods to explore institutional settings, researchers’ perceptions of these settings and their reactions to them.
A series of case studies will explore the institutional environments and responses to them for approximately 160 researchers on the basis of bibliometric analyses, qualitative interviews and analyses of contextual information. This approach will achieve an overview of the variation of important conditions and researchers’ responses. The number of case studies is sufficient to provide a broad picture of perceptions of and conscious responses to evaluation-based funding across different types of universities and scientific fields.
Based on these case studies, four in-depth studies will be conducted that are aimed at exploring the full range of impacts of evaluation-based funding as mediated by scientists’ conscious and unconscious responses. Four ethnographic observations will be conducted at two different universities and in laboratories belonging to two different fields, thus enabling a comparison of different institutional settings and responses from different fields. These four case studies will be prepared and accompanied by bibliometric analyses of the research groups’ positions in world science.
Following qualitative methodology (e.g. Patton 1990: 169-186), the selection of cases for empirical investigation will depend on variations in important variables. In order to achieve a sufficient variation of evaluation-based funding mechanisms, a review will be undertaken of the research funding strategies currently in place in Australia’s major research institutions. This will include a detailed description of current and past mechanisms for distributing research funds, and a history of their introduction at the organisation and/or sector level. Time series data will be assembled covering staff and students, finance, research expenditures, and other relevant characteristics. On the basis of this overview, and a bibliometric analysis undertaken to identify any universities that appear to exhibit distinctive trends in measures linked to adaptive practices, at least five organisations with different institutional characteristics will be included. Ideally, only organisations where current strategies have been in place for several years will be selected. This is to ensure that any adaptive practices we seek to identify by bibliometric techniques have been in place long enough to be visible in the data.
Since researchers’ adaptation to institutional conditions of action is field-specific, a second selection will be made of the fields to be investigated. A number of different fields will be selected (including one in social and behavioural research).
Based on contextual and bibliometric analyses, about 160 researchers from at least five universities and six different fields will be selected for case studies. From the fields and organisations selected for interview-based case studies, two fields and two organizations will be selected for ethnographic observations.
Internet sources and research reports will be used to provide information about the selected research groups’ “research trails” (Chubin and Connolly 1982) and their current projects.
A detailed bibliometric analysis of the publication of all research groups selected for the detailed case studies will be conducted in order to identify trends in publication behaviour and in the development of the research groups’ international impact. The individual publication and citation profiles constructed from this data will be used as input to the guided interviews. Additional bibliometric analysis of research groups will be undertaken in the light of details obtained from interviews.
Bibliometric analyses for the study will use the comprehensive REPP database of Australian research publications. The database contains all Australian articles published in the major international journal literature (and indexed by the Institute for Scientific Information) between 1981 and 2000, and will be updated to 2002 in time for the commencement of the project. The results of the analyses will be used to assist in identifying the universities to be included in the empirical investigation, and in identifying the research fields to be targeted. The techniques used will focus on publication trends, specifically changes in aggregate output over time, changes in the types of journal in which articles are placed (high versus low impact, applied versus basic), changes in the impact achieved by the publications (as measured by citations), and changes in productivity (publications per staff).
Approximately 160 semi-structured interviews will be conducted with researchers in the selected fields and universities. The interviews will cover the following main themes:
In each university, research managers will be interviewed about current and historical funding mechanisms to ensure the identification of all relevant processes.
Ethnographic observations will be conducted in four different laboratories. It is difficult to plan ethnographic observations neatly in advance, and ethnographic observations in science vary greatly in the time spent on site and in the practices applied. Each observation will last 5 months, and will aim to cover periods in which decisions about new projects and funding are likely to be made (for example, the time in which ARC grant proposals are being prepared). The aim of these observations is to investigate adaptation processes by tracking the selections researchers make in their everyday work. By observing these selections, and investigating the reasons for them, it will be possible to reveal subtle and even unconscious adaptation processes.
The ethnographic observations will have to manage an inherent tension: The classical ethnographic approach in science studies emphasizes the necessity "not just to understand, but to let speak" (Knorr-Cetina 1981: 18), i.e. to give the field under study as much control of the information construed as possible. This approach requires working on minimal a priori assumptions, while in the context of this project a priori aims and categories are predefined. Careful preparation and supervision of the ethnographic observation will be necessary in order to maintain a balance between the focus that is determined by the project’s aims and the openness necessary for successful ethnography.
Similar case studies will be conducted in Germany, where the internal governance of universities is still characterised by academic self-organisation rather than hierarchies, and where the introduction of evaluation-based funding of universities and their subunits is just beginning to be introduced. Research groups belonging to the same fields as the ones in the Australian sample will be selected. Interview-based and ethnographic case studies will be conducted following the methodology that has been developed for (and tested in) the Australian study. Thus, a comparison will be possible that enables the identification of adaptive behaviour and its causes.
The empirical data from interviews and ethnographic observations will be analysed by a qualitative content analysis (Gläser and Laudel 2004). The initial categorical framework (diagram1) will be regularly revisited in the course of data collection in order to adapt it to observational data that contradict the scheme. Similarly, prior to and during data analysis the suitability of the categorical framework will be under constant surveillance. These revisions will secure the correspondence of analytical categories with what has actually been observed.
Data analysis will be aimed at the construction of types of institutional environments which will capture the variation of evaluation-based funding mechanisms to which researchers are subject. In order to include the bibliometric data in the case studies, a typology of publication and citation profiles will be developed. While it would be beneficial to construct types of scientists’ responses, too, this step depends on what will be actually observed.
In a second step, the adaptation processes and causal mechanisms of each case will be identified. The third step consists of a comparative analysis of the cases that will lead to general causal patterns as well as to an evaluation of the impact of intervening variables. In a fourth step, a comparison of ethnographic and interview-based case studies will lead to conclusions about the relationship between conscious und unconscious adaptation processes.
Chubin, Daryl E., and Terence Connolly, 1982. Research Trails and Science Policies. Norbert Elias, Herminio Martins and Richard Whitley (eds.), Scientific Establishments and Hierarchies. Dordrecht: Reidel, 293-311.
Gläser, Jochen, and Grit Laudel, 2004. Experteninterviews and qualitative Inhaltsanalyse: Instrumente rekonstruierender Untersuchungen. Wiesbaden: VS Verlag für Sozialwissenschaften.
Knorr-Cetina, Karin, 1981. The Manufacture of Knowledge: An Essay on the Constructivist and Contextual Nature of Science. Oxford: Pergamon Press.
Laudel, Grit, 1999. Interdisziplinäre Forschungskooperation: Erfolgsbedingungen der Institution ‘Sonderforschungsbereich’. Berlin: Edition Sigma.
Mayntz, Renate, and Fritz W. Scharpf, 1995. Der Ansatz des akteurzentrierten Institutionalismus. Renate Mayntz and Fritz W. Scharpf (eds.), Gesellschaftliche Selbstregelung und politische Steuerung. Frankfurt a. M.: Campus, 39-72.
Miles, Matthew B., and A. Michael Huberman, 1984. Qualitative data analysis: a sourcebook of new methods. Beverly Hills: Sage.
Patton, Michael Quinn, 1990. Qualitative Evaluation and Research Methods. Newbury Park: Sage.
Scharpf, Fritz W., 1997. Games Real Actors Play. Actor-Centered Institutionalism in Policy Research. Boulder: Westview Press.
