The impact of the Research Assessment Exercise on serial publication

The Research Assessment Exercise (RAE) has had a huge impact on universities in the UK. Academic staff research output appears to be the key driver in personal and institutional status and survival because significant, and increasingly selective, funding flows from the results of the RAE. This affects funding for the following five to six years and departments and individuals can rise or fall as a result of a good or bad rating. Inevitably this has had a major impact on both the drive to publish and on the quality of publications.This paper will focus on the impact on serial publications, taken from the personal viewpoint of the Chair of the 2001 RAE Panel 61: Library and Information Management. It will look in some detail at how quality has been challenged by the pressure on academics to publish and how this appears to have affected the quality of journal publication. The presenter will explore, in discussion, the implications for serial publication in the future.


Background
To set the context for this paper it is important to recognize that this is a personal view of the Research Assessment Exercise (RAE).I was Chair of the RAE 2001 Library and Information Management Panel Unit of Assessment 61.As such, I shall try to look back and give my impressions, particularly of serial publications, in the context of the whole RAE.I shall also look ahead to the next Research Assessment Exercise, which is scheduled for 2008.
For those of you who are not from the UK, or are not deeply embedded in academia in the UK, I will give a bit of background.The RAE has run five times over the last twenty years or so.It is an exercise organized by what are now the Higher Education Funding Councils for England, Scotland, Northern Ireland and Wales.The results of the exercise inform research funding in the universities for the following four, five or six years.So in the UK it is hugely important.All of the higher education institutions, including institutions like my own, which is a University College, have the opportunity to participate in this by submitting their 'research active' staff.

The importance of RAE
Departments and academics rise or fall as a result of the RAE.Since the 2001 results, quite a number of the departments that did badly, or did not get the highest possible research ratings, have subsequently closed.The RAE, as probably with any sort of funding mechanism, is a game to be played and you play it well, or you play it badly and you suffer as a result.It has to be remembered that there are rules and you have to try to interpret those rules.I will summarize some of the issues I think are particularly relevant to serial publications.
The assessment exercise is just a snapshot picture.There are no quality assurance visits to any institutions; it is purely evidence-based.Quite clearly, there is enormous pressure on academics to publish.Published criteria and working methods

JUDITH ELKIN Vice Principal University College Worcester
were binding on the panels in 2001 and I believe the situation will be exactly the same in 2008, subject to consultation still being carried out during 2004.

How assessments are made
A holistic approach was taken by all panels to the submissions.The final ratings relied on the expert judgements of the individual panels and the chairs of the panels.In other words the context was a disciplinary-based context.This was the RAE definition of research for 2001: "Original investigation undertaken in order to gain knowledge and understanding.It includes work of direct relevance to the needs of commerce and industry, as well as to the public and voluntary sectors; scholarship; the invention and generation of ideas, images, performances and artefacts, including design, where these lead to new or substantially improved insights; and the use of existing knowledge in experimental developments to produce new or substantially improved materials, devices, products and processes, including design and construction.It excludes routine testing and analysis of materials." The 2001 definition varied slightly from the 1996 definition because it had slightly more practicebased, application-based research in there, which is intended to be part of the whole research exercise.It is assumed that this definition will be revisited for 2008, but I doubt whether it will change very significantly.
The narrative which accompanied all of the different elements of the submission looked for a clearly articulated research strategy and research culture, supported by the rest of the submission.As I said at the beginning, all higher education institutions and all subject areas within them have the opportunity to submit, provided they feel that they have sufficient quality research work being undertaken there.
The individual submissions have a number of different sections.RAE2 is the section for returning 'research outputs', which may mean practicebased evidence as well serial publications and monographs, i.e. any research-based output.Separate returns are made on research students and the number of completions in the previous four or five years and the research income received over the previous period.
In addition, there is a section for 'esteem indicators'.There is an assumption that research active staff will have a fairly wide spread of esteem indicators, which includes such things as: national or international esteem; keynote papers; chairs of committees; members of prestigious research or government committees; journal editorship; advisory positions; and awards.All of these are spelled out in the published criteria and guidelines some two years before the actual submission dates.Prestige of individual research academics is seen as an important part of the whole research culture being assessed.
It is worth remembering, when we are focusing on serials, that serials are not the only published outputs.The individual panels published their own working methods and criteria, as well as publishing how they would judge research outputs -in other words, what they would prioritize.In some units of assessment, for instance particularly in the humanities, monographs were counted as more important than serials.In others, for example, music and performing arts, presentations -which could be an orchestral presentation or perhaps an exhibition -were significantly more important again than serial publications.All of these are classed as part of research, provided they are innovative and original and contribute to knowledge.

Quality of published work
A maximum of four 'research outputs' could be submitted by each research active member of staff.There is enormous pressure -particularly in some subject areas where refereed journal articles are seen as the main output of serious research publication -on individual academics to seek publication in what they would see as the most highly rated refereed journals in their subject areas.
There are a number of general issues here.This presentation focuses specifically on the library and information management panel, but it should be recognized that many of the issues were common across a number of other areas.It was made quite clear to the panels that quality of research was what was being looked for, not the source of publication.For example, electronic journals were counted as highly as refereed print journals and high quality work was found in different formats.
High quality research work was found in professional journals as well as in more highly rated refereed research journals.We found that the definition of research was frequently referred back to by the panel.It is one of those definitions which you almost take for granted as being a definition you can understand and makes some sort of sense, but I was quite surprised how often we referred back to confirm that a piece of work, paper, or journal article had no research content, no originality, added nothing new and was purely descriptive.This happened, regrettably, over and over again.
Because of the pressure on academics to produce four highly rated publications in the period, which was four years for most subjects and six years for humanities subjects, there was a tendency to submit rewrites of the same research four times over.There were far too many overly descriptive outputs and outputs with little or no research content, even in that broad definition of research.The poor overall quality of conference proceedings surprised the panel; there were a large number of brief, lightweight articles, with no references, claiming to be "refereed".It is entirely possible that a lot of conference papers have refereed abstracts, rather than refereed final papers.From the large number of conference proceedings looked at, it was assumed that this was largely the case.There appeared to be very little editorial influence in the final published proceedings.
Also surprising, and again this applied in other panels, was that there were some highly regarded refereed journals that carried very poor quality papers, on the basis of quality of research content.The opposite was that some relatively poorly regarded publications contained some very good original pieces, so it was not possible to rely on journal rating.No rating of journals actually took place, but I think all academics have some understanding of what are regarded as the best journals in a particular subject area.
On the whole, reviews, brief interim research reports and shallow overviews were not rated very highly by the panel, and there were huge numbers of those.Output where there was original thought and where there was some analysis or new methodologies was rated quite highly in terms of their research content.There were far too many joint authors submitted.There is pressure, particularly on new researchers, to publish with more experienced researchers, but very slight pieces with five or so authors were submitted.One had to wonder about the individual contributions from each of the individual members of that authorship.Electronic journals were particularly disappointing, with some very poorly edited, skimpy, very poor quality electronic journal publications.We expected to rate them equally with paper-based journals but found that very difficult because we were looking for quality research.Overall we felt this situation demonstrated lack of research management in many of the universities, as well as lack of editorial control.
The importance of monographs again depended slightly on the subject area of the panels, but they were well considered if they were research based On the whole, descriptive text-books were not rated as research worthy although some text-books, clearly based on research, were looked at on an individual basis.
The highest research ratings recognize research at international level.In all of the panels, international excellence was the gold standard, with regard to the best being carried out anywhere in the world.It did not matter where a conference was held.It could be in the UK or in Timbuktu.It was the international research content that was significant, not the place of publication.There were huge numbers of international conferences and conference proceedings, but very few of real quality internationally.

Dangers of pressure to publish
For all the reasons given, there is huge pressure on academics to publish but there is an additional pressure on academics to publish 'RAE-worthy' publications.This is to the detriment of other types of publication such as research results to peers, research results to the professional community, research feeding into teaching.There is a significant dominance of the RAE in publications.Thus there is considerable danger for the overall quality of serial publications.There is also a tendency to publish in journals which individual academics know will be well rated in terms of citation indices.
Overall there was a feeling that the RAE puts pressure on academics to publish unfinished work and speculative research in the guise of real research.There is also a danger, which many of you will recognize, that academics publish rather than be involved in some less research-friendly areas, such as working on editorial boards or being editors of journals.That puts a pressure on serial publication too, with the added temptation to recycle, to divide a single piece of research into several bits, so that you have three publications, rather than one high quality, significant publication.There is also, as we have seen, the attraction of publishing in joint names, irrespective of contribution.

The future of RAE
The consultation exercise on the research assessment exercise has recently resulted in the RAE 2008 publication and, despite the fact that Dr Gareth Roberts, who led the review, wanted to change the exercise significantly, it does look very similar to previous exercises.So it will be broadly the same as in 2001, with a census date of October 2007 for submission in November 2007, and the exercise itself taking place in 2008.There will be a six years publication date-span for all subject areas, rather than the previous four years, with a maximum of four publications (although individual panels will be allowed to set a lower number).It is anticipated that some of the humanities panels may well set a maximum of two publications to try to encourage higher quality research publications, rather than increasing the pressure by concentrating on the number of articles.There will be emphasis, yet again, that applied and practicebased research will not be disadvantaged.There is still a lot of work to do on the definitions of applied and practice-based research through a consultation during 2004.
The panels will change slightly in that there will be 15 to 20 main panels, with 70 sub-panels, but I am not sure that looks hugely different from the 67 sub-panels in 2001, with something like five umbrella panels.The disciplines of those main panels and sub-panels have yet to be decided.The most significant change will be that results will be published on a continuously graded quality profile for each submission, replacing the current sevenpoint rating, where the most highly rated research departments were looking at five-star research ratings.This will be done on an indiidual profile which will be criterion-referenced and will identify the proportions of work in each submission, reaching each of four defined star ratings, largely equivalent to the current national and international esteem.
There are still a lot of unanswered questions and there is still a lot of discussion to be had about the 2008 exercise.Although 2007 sounds a long way away, of course it is not very long in this context.Most of the highly rated research departments will have been working towards the next RAE for at least the last couple of years, so 2008 is unlikely to put much less pressure on academics to publish.There are clearly pressures on the markets, both in terms of serial publications and library provision of serials.I realize that I have been quite scathing here about a lot of the material that my panel looked at.When I shared this with a number of panel chairs, they also agreed that they had read vast quantities of publications which were returned as supposedly quality research, but really were very poor quality indeed.

Conclusion
There is pressure on academics to publish.There is pressure on journals to publish and I do not know where the balance comes in terms of how that should actually be moderated.As discussed, there is much consultation still to be had this year and all of you, certainly all of you in the UK, are in a position to respond to the consultations that will be coming out over the forthcoming months.