Perceptions of Value and Value Beyond Perceptions: Measuring the Quality and Value of Journal Article Readings

Based on a paper presented by Carol Tenopir at the UKSG seminar ‘Measure for Measure, or Much Ado About Nothing? Measuring the quality and value of online journals’, London, Thursday 14 June 2007 When measuring quality and value of journals, what is being measured, to whom does the value accrue, and why the measurement is being done must be considered. Both implicit and explicit measures of quality and value are possible. Preliminary results of ongoing reader- ship research show that academic library e-journal collections have both purchase and use value to academic faculty. They spend their time reading, read many articles each year, and reading benefits their work in many ways.


Introduction: what is value?
Economists refer to two aspects of value. In the information context, Machlup 1 described 1) purchase or exchange value or what one is willing to pay for information found in journals in money and/or time, and 2) use value or the favorable consequences derived from reading and using the information.
Quality, on the other hand, is an attribute of information that readers are willing to pay for (in time, if not in actual money) and which leads to favorable consequences of use. Quality derives from the reputation of a journal in which articles are published and is due to editorial processes including peer reviewing, editing, and distribution platforms. 2,3 Quality of information is also enhanced through 'value added' processes, such as indexing, abstracting and formatting. 4 A higher quality article is assumed to have a higher value to the readers, although this may not always be the case. Various and differing quantitative and qualitative measures of both quality and value are described here.

What, who, and why of measuring quality and value
The quality and value of online journals can be measured using a variety of methods that range from analyzing usage logs to capturing users' perceptions. To really understand quality and value of journals, we first must understand for what unit we are measuring value, the value to whom, and why value matters. The most appropriate measures of value depend on the context of these questions.
The question of what, or what is the unit of measure, takes on special meaning in a changing online environment. At its largest scope, the unit of what is being measured may be a publisher's journal system or an entire library's journal collection. Measuring an entire collection typically examines how the collection as a whole meets the needs or wants of the body of users in general. Not every title may be of equal value or experience equal amounts of use, but it is the total picture that matters. 5

Perceptions of value and value beyond perceptions: measuring the quality and value of journal article readings
Based on a paper presented by Carol Tenopir at the UKSG seminar 'Measure for Measure, or Much Ado About Nothing? Measuring the quality and value of online journals', London, Thursday 14 June 2007 When measuring quality and value of journals, what is being measured, to whom does the value accrue, and why the measurement is being done must be considered. Both implicit and explicit measures of quality and value are possible. Preliminary results of ongoing readership research show that academic library e-journal collections have both purchase and use value to academic faculty. They spend their time reading, read many articles each year, and reading benefits their work in many ways.
What may also refer to each specific journal title, in which case measures such as journal impact factor or lists of top journals in a field are applied to demonstrate the relative quality or value of each title. Focusing on value of entire journals may be for collection development decisions, such as adding or deleting a title, or is of importance to journal editors or publishers to enable them to see where their journals fit in comparison to other titles on the same topic. 6 Sometimes individual articles or readings are the more important unit for which value is measured, in which case citation rankings are the traditional bibliometric technique to measure quality and value. It can be argued that the value of the individual pieces of a journal (articles) can be measured quite independently of the whole. An important article may appear alongside articles of low importance or dubious value from one reader to another. Measuring individual article value is often done to measure relative contributions of an author or an academic department, as well as being one component of the journal title-level journal impact measure. 7 Finally, in the digital environment, what may even apply to parts of individual articles such as a specific graph or table or a piece of an article such as a paragraph or section. Certainly some readers need only a part of an article to answer their questions or meet their needs. Not much attention has yet been paid to measuring the quality and value of article parts, but such measures may be useful in the future to help in design of future products, services, or metadata and to better understand user behavior. 8 To whom value accrues may refer to individuals or to libraries or institutions. Individuals, of course, may include all readers of the unit of interest or be further subdivided into constituent groups such as academic staff, students or others, and each group may be even further divided into their tasks as readers or authors. Past research has shown that a person's work-place, work role, and purpose of reading, make a difference in reading patterns and corresponding indicators of value. 9,10 Why quality and value are being measured may be just because it is interesting to a researcher, but from a library perspective why may also be to assist librarians with collection decisions, help improve information products and services, and demonstrate return on investment (ROI) to funders. Publishers might make editorial and funding decisions based on their knowledge of the value of their journals to readers and library subscribers.
The context and perspective of this paper -in terms of to whom -is the value to readers, that is, academic staff or students, rather than to the institution (although, of course, libraries should extrapolate upwards, for the academic library's value to the institution lies in how well it provides valuable resources and services to the academic staff and students). We focus mostly on individual journal articles, although, again, libraries may extrapolate upwards to journal titles as a whole or categories of journals by examining where articles that are read most often are published. Why is to quantify the value of journal article reading to provide data useful for libraries, publishers, and researchers.
There are two broad categories of measures of value: 1) implicit measures that imply value, but do not directly measure it, and 2) explicit measures that directly describe purchase or use values.

Implicit measures of value
Implicit values in this context are those values that are implied by answers to other questions, for example those relating to amount of reading or use, or implied by usage logs. A reading of an article is only an indicator of use value, since the consequences of the reading are only implied. Logs present data on how often specific articles (or articles from a specific journal title) are downloaded and then we equate amount of use with value to the user community. Those articles or journal titles that are downloaded frequently are assumed to be used more and implicitly assumed further to be of high quality and therefore high use value. We do not really know that, of course; all downloads may not have been read or they may have been read and dismissed as low quality or not valuable, but we equate higher amounts of use with higher use value to readers because it is a convenient metric. Implicit measures of electronic resources are easier to collect, do not rely on direct involvement by users, are relative easy to quantify, and are particularly useful for measuring changes over time, such as increased use of e-journals. Showing the increase in overall downloads of articles provides a powerful argument for the e-collection. 11 Counting citations (and derived measures values from that such as journal impact factor) is another implicit measure of value. Citation counting implies more closely the relationship between the amount of use of articles by authors and the use value to them for research and writing. 12 Implicit values can also be gathered in surveys that ask questions about amount of use. In our surveys we have several questions that help us derive implicit measures of value. As can be seen in Figure 1, over time the reported number of article readings has increased on average per faculty member. In our latest surveys, faculty members report reading on average 21 articles per month. Projecting to a year (12 months) shows an increase in reading of approximately 68% since 1977 (from 150 to 252).
There are many possible reasons for an increase in readings, for example having electronic access broadens the scope of potential articles or there are more articles and journals published now than in the past, so scholars have to read more just to keep up with developments in their field. Still, more readings imply a continued and increasing use value.

Explicit measures of value
Explicit measures of value in this context are things that are readily identifiable as either purchase or use value by the readers. For example, in the surveys of university faculty and students, we explicitly ask respondents for the last scholarly article they read (believed to be random in time and therefore a random sample of readings), how much time they spent identifying, obtaining and reading the article. This is an explicit component of the purchase value to readers and can be converted into a direct expenditure based on the dollar value of their time. The other component of money expended is that borne by personal subscribers or libraries on readers' behalf (with the exception of author-side payment and other open access models). Incidentally, readers tend to expend five to ten times as much in their time as what is paid directly in personal and library subscriptions.
Regardless of how obtained, readers would not spend their valuable time if the information contained in the article readings was not perceived to have commensurate use value to them. Faculty members spend an average of 8 to 17 minutes per reading in identifying and obtaining articles (depending on methods used) and 34 minutes Students, not surprisingly, read for different purposes. Over 3,500 student responses to our seven surveys in Australia and the US report that they read most often: ■ to help complete a course assignment or required reading in a course (46-50% of readings) ■ for thesis/dissertation (33-37% of readings) ■ to keep up with the literature (7-8% of readings) ■ for personal interest (2-4% of readings).
There are, of course, differences between undergraduates and graduate/post-graduate readers.
Undergraduates read most often for their course assignments, while graduate students read for courses, but read heavily when completing their dissertation or thesis. Faculty were also asked to rate the importance of their last reading to achieving the principal purpose. When responding to the question of whether the last article read was 'not at all important' to the principal purpose, or was 'somewhat important' or 'absolutely essential', most readings for any purpose were rated as 'somewhat important' or 'essential'. Those readings for writing proposals or reports and research are more often 'absolutely essential' (See Table 1.) Another measure of explicit use value is how the article reading helped readers in their work. Article readings are reported by readers to help them in many ways to accomplish their tasks. Table 2 shows that reading journal articles most often helps faculty by inspiring new ideas or new (Since this is self-reported data from those faculty members who chose to complete the survey, the percentage of readings that wasted time may actually be higher. Those readers whose last reading was not helpful might be more reluctant to complete the survey.) Other observations of use value in general, and specifically from library collection, include: ■ 'productivity' measures are correlated to the amount of reading ■ faculty whose work has been recognized through award or special recognition are found to read more than others and they tend to use the library collections more.
Clearly, scholarly articles have many explicit use values to readers. The best way to gather data on explicit value is by interview or surveys. We prefer the critical incident technique so we can get information at the individual reading level, but other explicit value surveys such as the LibQual, widely used in libraries, ask respondents to rate expected value of services in general and then to say how the offerings at their library meet those expectations. (See <http://www.libqual.org/>).
Explicit values can also be qualitatively measured through stories or open-ended comments. These can make compelling arguments, especially to funding agencies. In our surveys we asked: "How has your use of scholarly materials changed in the last few years?" Most of the comments were positive reactions to electronic journal collections. The comments below from faculty are typical: How did we ever get along without electronic journals?
The ability to obtain articles online has made [

Some typical student comments include:
Finding articles online is so much easier and faster than finding articles in dusty journals in musty corners of the library.

I have found electronic journals an invaluable aide as it means I do not have to travel to the Uni for every little article (which takes AT LEAST 1 hour.)
Comments from both faculty and students speak to the value of collections in terms of time saved, increased productivity in work, and convenience. (For a full set of comments by both faculty and students see the individual university reports at<web.utk.edu/~tenopir/research/survey_ instruments/html> Explicit values can be influenced by perceptions, however. For example, if someone thinks of a journal as high quality, he may perceive that a specific article is also valuable. Librarians are no doubt faced with these perceptions when trying to eliminate certain journal titles. A title that a faculty member remembers as being useful in the past may be perceived as valuable even if it has not been read for years. Perceptions are powerful, but must be coupled with actual or reported measures of use to get the whole picture of value. Implicit measures of value are less prone to influence by perceptions.

Contingent valuation and return on investment in library journal collections
Contingent valuation -estimating the time or cost of not having a service and comparing that with the time or cost of the service -can also be used to calculate an implied value of journal collections. Aerni and King state: 'Contingent valuation is an economic method used to assess the benefits of non-priced goods and services (e.g., libraries or specific library services) by examining the implication of not having the product or service.' 13 After asking questions about the last article they read in our surveys of faculty, we ask several questions that address contingent valuation. First we ask participants to indicate what they would do if the source they used for their last reading (e.g. library collection) was not available to them. Then we ask them to speculate what it would cost in terms of time or money to get the same information if the source they used was not available to them. The question actually contains multiple parts, with the wording: " Eighty-one percent (889 of 1098) of faculty respondents who answered this question indicated they would obtain the information from another source. A wide range of sources were given, with libraries or library services frequently mentioned (including another library, a library print or electronic collection and inter-library loan as some most frequently mentioned). Even for those who obtained their most recent reading from a library source, the number one alternative source was another print or electronic library resource (See Table 3). Based on one study 14  This data leads to an estimate of the ROI in the journal collection made by the university. The investment or cost to the university is $3.43 million; that is, $1.56 million in faculty time and $1.87 million in library expenditures. 15 The cost of alternatives is $13.48 million, so that the net benefit of the collection is $10.05 million (i.e., $13.48 minus $3.43 million). The ROI is 2.9 to 1 (i.e., $10.05/$3.43 million). Other survey questions that address aspects of contingent valuation get at the role of personal subscriptions, library collections and other sources. We have found that the number of personal subscriptions on average has steadily declined over time. Researchers in all work-places received, on average, over 5 subscriptions in 1977, down to under 2 by 2003. The number of subscriptions held by faculty members is slightly higher (just over 3), but also has decreased over time. Medical faculty continues to hold the most personal subscriptions (between 5 and 6).

Changing reading patterns
Readings from personal subscriptions, libraryprovided journals, or other sources (such as the free web) demonstrates the explicit value of the library collections based on increased use over time. Figure 4 shows that as readings from personal subscriptions have declined, readings from library collections and other sources has increased. 'Other' includes readings from colleagues, listservs, from the free web, and so forth. Actually, some of these other readings are likely to be from the library e-collection even though the faculty member may not realize it. Transparent linking from web searches or database searches through the library portal provide affiliated users with full text access that is not always branded by the library. They may underestimate the library's contribution to their journal article reading. So, at least (and probably more than) 52% of scholarly article readings by faculty in our latest surveys come from the library collection.
The percent of readings from articles older than one year are also increasing over time. A majority of older articles come from the library collection. (Figure 5). This increase likely shows that electronic journal back-files are being used. Articles that are read for research, that are older, and that come from the library are more likely to be rated as essential.

Conclusion
In conclusion, journal article reading has many explicit and implicit values to readers. It is important for libraries and publishers to capture those values using such techniques as surveys, Older articles are judged more valuable and are more likely to come from libraries.
usage log analysis and citation analysis. Use and value to readers can be measured in multiple ways, including perceptions, explicit expressions of value and usage that implies value.
To keep with our Shakespearean influence for the UKSG Conference 'Measure for Measure, or Much Ado about Nothing? Measuring the quality and value of online journals', measure quality and value in multiple ways 'As You Like It' and as you need it, and by collecting good evidence of the value that e-journal collections bring, when it comes to budget time, 'All's well that end's well'…