Electronic information services evaluation : current activity and issues in UK academic libraries

Electronic information services (EIS) are rapidly becoming embedded in UK library and information services and have revolutionised our work. While there is evidence of individual projects and exercises to conduct potential evaluation of EIS in a number of higher education (HE) libraries, practitioners need guidance in methods to evaluate this rapidly evolving area. eVALUEd is a HEFCE-funded project which is developing an online toolkit for evaluating EIS and to increase awareness of existing materials to help library managers in HE to evaluate such services efficiently and effectively. An initial survey of library managers and subsequent interviews informed the current picture of EIS evaluation and identified the main challenges. The main challenge cited was suppliers’ data. Lack of comparability, and unreliability, present difficulties for EIS evaluation. Some institutions in the UK are developing their own in-house solutions to overcome poor statistics. Finally, the survey provided suggestions for inclusion in the toolkit.


Introduction
Electronic information services (EIS) are rapidly becoming embedded in UK library and information services and have revolutionised the way in which we now work.
While the professional literature demonstrates considerable interest in the evaluation of such services, there is no over-arching system to assess the effectiveness, value and importance of EIS.
While there is evidence of individual projects and exercises to conduct potential evaluation of EIS in a number of higher education libraries, practitioners need guidance in methods to evaluate this rapidly evolving area of their service provision.
eVALUEd is a project funded by the Higher Education Funding Council of England (HEFCE) which is developing an online toolkit for evaluating electronic information services.This will include resources as well as increasing awareness of materials already published to help library managers in higher education institutions to evaluate such services.In order to inform the style of the toolkit and to establish a picture of evaluation activity in higher education libraries, a survey was conducted in 2002.

Survey methodology
The survey was an eight-page questionnaire to 196 library managers in UK higher education institutions in the Spring of 2002.The survey was intended to identify some of the issues faced by library professionals working with EIS, for example the level of evaluation activity and wider evaluation of institutional electronic services.[The questions are included in the appendix].

Survey and interview responses
There were 112 responses to the survey from 194 institutions (28 of which were completed online).The overall response rate was 58%.43% of institutions responding to this survey were operating converged services, where IT/IS services were merged, while 55% were not.(The remaining 2% did not answer this question).
Follow-up interviews were also undertaken with 20 respondents to the survey.These were intended to provide clarification of some of the issues raised in questionnaire responses.
The selection of interviewees was based on: library and information services (LIS) staff who had identified themselves as willing to participate in follow-up from the initial eVALUEd questionnaire and those who had indicated some level of evaluation activity.
Full analyses of the survey results and of the interviews are available as project papers on the project website. [1]

Analysis of the data collected
The data collected came from a representative number of HE colleges, universities established before 1992 and those established since.This provided a picture of the level of evaluation activity across the board as well as allowing for individuals or one-off exercises to be recorded.

Evaluation undertaken
Almost three-quarters (72%) of institutions responding to the questionnaire confirmed that they carried out evaluation of the electronic information services within their library and information service.The types of evaluation cited, and incidence, are listed as follows: The most commonly used method of evaluation cited was management information systems data collection.This includes use of supplier statistics.
The survey has confirmed trends in a review of the literature that there is no comprehensive, clear and straightforward method of evaluating electronic information services currently being used in the academic library community.The responses show a limited range of evaluation activity being undertaken.Such activities tend to be informal and unstructured.For example, the fourth most popular method was observation.This was often so informal (for example, evidence was based on 'hunches' gained from day-to-day contact with students) that its relevance could be considered negligible.Many respondents to the questionnaire cite management information statistics as informing their evaluative effort.However, these statistics are rarely produced in-house, rather, library services have to rely on the (often flawed) statistics provided by EIS suppliers.
Some librarians cited cost-benefit analysis as a method to evaluate electronic services.However, it appears that this is usually a crude cost equated to use based on supplier statistics.None of the interviewees referred to any benefit drawn from student responses to resources or the impact on learning, teaching or research.
In detailed analysis, it is clear that evaluation efforts are predominantly low-level.Generally, those practitioners that are undertaking evaluation are doing so with small parts of the service or with discrete, clearly defined collections, using whatever statistical data are available.
Very few institutions (19%) from the survey results had reached the point where they had a policy statement specifying the need for evaluation of electronic information services within their library and information service.

Barriers to evaluation
Despite a general will to evaluate services and the understanding that evaluation can provide useful evidence of the effectiveness, efficiency and value of electronic information services, library managers still experience major challenges to their ability to take evaluation through from conception to service improvement.
Some respondents referred to existing frameworks, for example the Society of Colleges, National and University Libraries (SCONUL) Annual library statistics. [2]These are national statistics about general service provision which are provided voluntarily by UK academic libraries.The data seemed to prompt libraries to collect some data concerning EIS.

Time and money
Although listed separately by practitioners when describing obstacles to evaluation, the two issues of time and money are interlinked and in fact pertain to the implementation of any new service or process.While some would argue along lines that money and time can always be found for the development of new processes where there is a collective will, nevertheless there are implications for cash-strapped organisations and those with small staff numbers, who may well have to consider the expensive option of hiring consultants to undertake evaluation for them.
If institutions can somehow find the initial time to undertake a feasibility study (this need not be on a grand scale), it might help them determine the following kinds of issue: collections/services which are in particular need of evaluation areas where data collection is already taking place (or could easily be done) level of analysis required appropriate staff add-on costs (printing, postage, dissemination) sources of funding -internal or external.
It should then be possible to make a start on a discrete area of digital library evaluation, however small.Assuming that in the first instance costs might have to be borne from existing budgets, nevertheless, managers could find it worthwhile to have the evidence of a successful evaluation exercise to hand in order to attract funds for future exercises from other sources.The skills required would be transferable and could apply to library services in general.In the same way, existing skills and knowledge of evaluation techniques could be employed to apply to EIS.

Skills
It is clear that library managers and other individuals who are called upon to undertake evaluation feel inadequately equipped for the task, both in terms of general evaluation skills and statistical expertise.Many of these skills gaps can be overcome by training, but this is an expensive option and usually only helpful if undertaken at the time it is needed, which cannot always be arranged.

Benchmarking
The issue of benchmarking also causes anxiety both because practitioners can feel inadequately equipped to make meaningful comparisons with peer institutions, and because they distrust a system which highlights differences (whether good or bad) between them.There is little evidence of benchmarking conducted in relation to EIS.Benchmarking initiatives in the traditional library setting, however, are becoming increasingly well developed in the UK.Several academic libraries are part of benchmarking consortia to develop evaluation initiatives and allow comparison of results.Such frameworks could be extended to EIS.

Fast-moving environment
The pace of change in electronic information services creates difficulty in two ways: 1. changes in service delivery, whereby print and electronic resources are increasingly offered in a seamless environment.This creates artificial divisions when trying to evaluate by separating the impact of the two kinds of resources.It also raises the question of definition of the 'digital library' as a distinct and self-standing concept.
2. opportunities to undertake longitudinal studies.In a climate of change it is difficult to replicate evaluation activities year on year, both because of changing electronic resources and changing financial circumstances.This in turn makes it difficult to analyse long-term effects of electronic information services.

Inadequate data
A key difficulty is the inadequacy, unreliability and incomparable nature of statistical data being supplied by e-publishers and vendors.This is compounded by libraries' own inadequate management information systems which do not allow accurate statistics of use to be drawn off.
Several interviewees commented that usage is a major issue and one noted:

"Need to know how to measure usage -we need a quick and easy way of analysing usage of journals and databases."
As the questionnaire highlighted, supplier statistics have presented a number of problems to libraries.Fifteen interviewees referred to obstacles created by supplier statistics.The main problem cited was lack of comparability, as statistical data and presentation vary widely between vendors.For example, two respondents noted: "Can't compare like with like.""Lack of statistical base between suppliers." Another issue is the unreliability of data collected.Technical issues can impact on the statistical data collected:

"Double clicks can count as two uses."
The responses to the questionnaire and followup interviews reflect themes in the literature.The capability offered by statistics for electronic products appears promising, particularly when one considers the potential problems which the collection of comparable statistics for printed journals might present.Statistics also offer the opportunity to break down usage further, for example by user type. [3]hile usage was considered to be one of the main areas where useful data could be gained swiftly if suitable methods and arrangements were in place, in reality, the resultant data varies widely between vendors as does the timing of any updates.These factors obviously have a major impact on the time required to analyse any statistical data. [4]ifferent journal hosts monitor different statistical information measures (searches performed, PDF documents downloaded, abstracts viewed, etc.), with little consistency between them.' [5] Many libraries lack control over their own management information systems which do not allow accurate statistics of use to be drawn off.In non-converged services, there seems to exist a lack of collaboration between the IT and Library and Information Services in order to develop inhouse data.
Several HEIs are tackling this problem and developing their own solutions in-house.These include questionnaires asking users about their patterns of usage for specific databases [6] and the development of a systems infrastructure to capture links by users to each electronic information service. [7] lead is being taken by publishers and librarians to address the problems of inadequate supplier statistics.Project COUNTER (set up in March 2002) is an international initiative which aims to ensure some standardisation of vendor statistics. [8]The first phase has established a code of practice to support librarians and publishers.This attempts to standardise the data provided by suppliers and the format in which it is presented.This should help to respond to the needs of the various stakeholders.

Existing frameworks
One of the main frameworks cited by questionnaire respondents and interviewees as influential in provoking the need to evaluate is that of the Society of Colleges, National and University Libraries (SCONUL) Annual library statistics. [9]hree interviewees mentioned problems presented by collecting data to support the SCONUL framework.In particular, comments were made about the new section on EIS.For example: "Difficult to separate out individual titles, e.g.unique electronic journal titles" Due to vendor arrangements and the inclusion of some electronic titles as 'bundles' it is often difficult to identify titles held: "Difficult to even count number of e-journals held."Some freely available electronic titles are stored in the same way as those purchased and this can cause problems in calculating numbers.
The framework has been adapted over time and it is likely that it will change again to accommodate the rapidly-changing environment.

Next steps and outcomes
Support required for evaluation 81 institutions (72%) said they would like support with their evaluation of electronic information services, while 17 (15%) said they would not and 14 (13%) did not respond.Some who did not want support suggested that they currently lacked the time to conduct evaluation.
Importantly, 95% (106 institutions) said that they would use a free evaluation toolkit if one were available, and 88% (99 institutions) said they would welcome training opportunities in the evaluation of electronic information services.This supports the premise on which the eVALUEd project has been built.
Respondents to the eVALUEd survey spoke of some positive results of evaluation exercises, as the following examples show: "Won bid for extra funding for more NESLI deals" "Extra facilities provided" "Higher profile given to the services" "Investment/ upgrade of hardware and software." "Our current priority is to build a broader electronic infrastructure.Evaluation is important but requires analysis over time.The most important issue is to have data which can be used to influence the external market." This further highlights the need for, and potential benefits of, an evaluation toolkit.

Toolkit development
Some of the main aspects which interviewees would like to see included in a toolkit are outlined: usage -the use and usage of services benchmarking -establishing a benchmarking framework for comparison of data between libraries impact analysis -assessing the value and impact of EIS cost-benefit analysis -monitoring the value and cost-effectiveness of EIS general evaluation -help with general evaluation skills and techniques performance indicators -creating measures and indicators and adapting existing ones.

"Anything that saves time and effort and provides a consistent evaluation framework would be welcome." (project interviewee)
The eVALUEd project is developing the online toolkit and has established links with the Society of Colleges, National and University Libraries (SCONUL).Work concerning the development of the toolkit will continue with the SCONUL subcommittee, Advisory Committee on Performance Improvement (ACPI).
The toolkit will attempt to cover a comprehensive set of evaluation issues and it will be piloted during 2003 among selected institutions.Case studies are being undertaken in a small number of institutions whose evaluation practice could usefully be disseminated to the rest of the sector.

Conclusion
The survey provided a useful means of assessing general information on current evaluation practices on EIS.It was a relatively straightforward means of collecting and analysing data across the sector and the interviews provided the opportunity to seek clarification and amplification of the important issues.
The main findings were that evaluation is certainly considered to be important.This was proved by both the high response rate to the questionnaire and the willingness of so many to take part in subsequent interviews.Furthermore, there was considerable interest in evaluation of EIS, despite the fact that little is being done at present.
In addition, the survey provided practical information and more examples of evaluation activities to supplement the information collected in the literature.A series of interviews with 'experts' in the wider evaluation field are continuing as a complement to those conducted with library and information professionals, and these will inform the development of the toolkit.