The Norwegian Electronic Health Library (NEHL) is a national online library service for Norwegian health care personnel. Through the NEHL portal (www.helsebiblioteket.no), users have access to approximately 2,500 medical journals, bibliographic indexes, point-of-care reference tools and other medical information resources. The service is funded by the Norwegian government, and consequently must comply with laws and regulation concerning public procurement. This article will describe purchasing procedures used by NEHL to acquire access to electronic resources.

Background

NEHL is part of the Norwegian Knowledge Centre for the Health Services (NOKC). The NOKC is a centre working with summarizing research and the promotion of evidence-based medicine in the Norwegian Health Services. It is organized as a publication led by an editor-in-chief and is professionally independent when it comes to making decisions about content and the presentation of content through the library portal. Started as a project in August 2004, NEHL was officially opened on 6 June 2006 by the Minister of Health.

Public acquisitions

Laws and regulation on public acquisitions are more or less the same all over the European Union. Norwegian laws were recently changed in order to be more in harmony with the EU. The regulations state when public tenders should be used, and also describe in detail how a tender should be organized and what kind of process to use for different kinds of products and/or projects.

Any purchase with a total contract value of NOK 500,000 (£55,000) or more must be put to tender in Norway at the very least. Purchases exceeding NOK 1,100,000 (£120,000) must be published internationally through the European Tenders Electronic Daily (TED) database. There are, however, a few exceptions to these rules, and one exception in particular is relevant for NEHL. It is legal to negotiate directly if the product in question is truly unique and without competition in the market. This means that NEHL can negotiate directly with publishers for specific journals or journal packages. One example is the JAMA and their package of ten archive journals.

Tender process

NEHL have from the beginning in 2004 through December 2010 undertaken six international tenders. Five of these were for content resources and one tender was for technology. All the tenders for content used an open tender competition. The technology project was done with a negotiated tender.

Open tender

An open tender is advertised nationally through the Norwegian Doffin database and when the value is above NOK 1,100,000 in the European TED database. Interested parties download the tender documentation by registering a profile with Doffin. Participants in the competition can make one bid for the contract and there can be no negotiations on pricing. Vendors are allowed to ask questions to NEHL during the tender period. All questions and answers are anonymized, answered in writing and distributed equally to all parties who have registered for the competition. Participants are invited to Oslo to present their products and services during the tender process. These meetings also provide a good arena for them to ask questions directly of NEHL representatives. Vendors are encouraged to challenge any inconsistencies or things that may seem unclear. In some cases they may have suggestions on how specific parts of the requirements can be altered to accommodate a better solution than NEHL have asked for. Changes are, however, only possible within certain limits. It is, for example, not legal to change requirements in a way that will benefit one vendor.

The tender documentation should contain formal information about the process and practical information about how to participate in the competition. A typical tender for content from NEHL will consist of the following documents:

  • publication on Doffin and TED: mostly formal information about the process and a reference to further documentation for details
  • tender document: information about NOKC and NEHL and more details on the formalities of the process. The tender document also specifies the decision criteria for the purchase and schedule for the entire process
  • core requirements: specifications for the products and/or services NEHL want to acquire through the tender process.

In most cases templates will also be provided for the vendors to use when responding to the tender:

  • cost specifications: this is usually a spreadsheet where the pricing is supposed to be entered according to a specific pricing model. This is a very good way to make sure that the prices from different vendors are comparable. To ensure stability and predictability, vendors must price their products in Norwegian Kroner
  • requirements template: the template is based on the core requirements document. Participants will need to add information about their product according to each paragraph in the core requirements document.

Negotiated tender

The negotiated tender is not very different from the open tender. The biggest differences are that it is possible to negotiate on pricing and it requires more time to accommodate the negotiation process. This process is supposed to be used for large and complicated acquisitions like technology or construction projects. Legal counsel has advised NELH against using negotiated tender for content acquisitions.

Case study: tender for point-of-care reference tools

During the first half of 2009, a decision was made to acquire a point-of-care reference tool for Norwegian physicians. The tender process was led by NEHL staff in collaboration with representatives for the users. Several organizations were invited to participate in a reference group. The Norwegian Association for general practitioners, the hospital regions and medical libraries provided personnel for the task. Each person was asked to represent his or her profession and was encouraged to recruit others to help.

There were two main tasks for the reference group:

  • NEHL wanted help to decide on the decision criteria for the tender. The list of criteria is the single most important part of the tender documentation. The group was asked for input on criteria concerning content, quality of content and user-friendliness
  • the reference group would assess to what degree products presented in the tender applied to the decision criteria.

Weighted decision criteria

Decision criteria must according to Norwegian law be weighted according to importance prior to publication of the tender. This is to ensure predictability and a level playing field for the contenders.

NEHL staff members created a table of criteria and distributed them to the reference group for feedback and adjustments twice. The final list of criteria is shown in Table 1.

Table 1

Table of decision criteria produced by NEHL to aid the tender process


Criteria Weight Description

pricing/content 10% Pricing was important, but not more important than the overall quality and functionality of the products. NEHL reserved the right to consider how content was reflected in the price
access levels 4% NEHL have two access levels; full national access and limited access. Full national access means that all Norwegian Internet Protocol address (IPs) have access. Limited access means that users must come from a known institutional IP or log on to Helsebiblioteket.no with username and password
authentication 4% Support for different kinds of authentication
Helsebiblioteket.no branding 6% It is important for NEHL that users are aware of how access to content is provided
enterprise search 6% The NEHL portal is using enterprise search technology to provide advanced search features through content. Vendors and publishers must support this technology
technical requirements 3% Support for browsers and web standards
linking and integration 6% Support for deep linking to content from topic pages in the library portal
HHC support 2% Support for hand-held devices
rights to reference and re-use 4% NEHL wish to make sure that personnel working on the production of medical guidelines and procedures can use references to the products. It could also be interesting to translate selected content to Norwegian
agreements 4% Guidelines about licence agreements
full version 2% NEHL did not want quotes for partial products, just to be offered add-ons at a later stage for an additional fee
accessibility and availability 2% Support for usability standards
language 2% Support for languages and adaptation to Norwegian context
search features 2% Search features in the product
user interface 6% Usability
administrative tools 3% Support for statistics report and customization
scope and purpose 2% Information about scope and the disciplines covered by the products
coverage and comprehensiveness 8% The amount of information provided
stakeholder involvement 2% To what degree are representatives for patient groups consulted in production?
rigour of development 6% Compliance with evidence medicine methodology
clarity and presentation 2% Clarity does in this case not refer to usability and design, but rather to language
applicability 2% Concerns about cost and/or practical issues must be clearly stated where ever it is relevant
editorial independence 4% Transparency of authorship, conflicts of interest and funding should be documented and easily available
metadata 2% Support for medical ontologies like ICD10, ICPC2, MeSH and other relevant systems
training provision 2% Training programs provided by the vendor
operational services 2% Documentation on performance, disruptions and security systems
helpdesk services 2% Availability of support staff when needed

The case-study tender process

The tender was published through Doffin and TED in early July 2009 and the delivery deadline was on 18 September 2009 at 13:00. Any response delivered after 13:00 would be disqualified.

Two weeks after publication, ten vendors had registered and downloaded the tender documentation. Eight out of the ten vendors delivered a response within the deadline.

Product demonstrations were scheduled from mid-August through the first week of September. All vendors except one booked meetings and showed up to give demonstrations.

Selecting the winners

All deliveries were checked for any significant reservations. Then copies were distributed to the reference group for analysis. NEHL staff assessed the input from the reference group and used the information to score the quotes.

The highest score was for BMJ Best Practice from BMJ and the second highest score was for UpToDate from UpToDate Inc. NEHL chose to make agreements with both vendors for national access to their products.

Table 2

Scoring table


Product Vendor Final score

BMJ Best Practice BMJ Group 0,9604
UpToDate UpToDate 0,7092
Product 3 Vendor 3 0,6584
Product 4 Vendor 4 0,5990
Product 5 Vendor 5 0,4994
Product 6 Vendor 6 0,4594
Product 7 Vendor 7 0,3964
Product 8 Vendor 8 0,2476

Benefits of the tender process

The tender process is a very powerful tool for acquisitions. When NEHL started working on licence agreements, there were several conditions publishers and vendors did not even want to discuss. They would refuse to invoice in Norwegian Kroner and would not even listen when the subject of national access was presented in meetings. Permission to crawl publisher content for the Helsebiblioteket enterprise search platform was also a topic most companies tried to avoid.

The tender can be used to specify ‘must-have’, ‘should-have’ and ‘nice-to-have’ criteria, for example:

  • any company that wants to have a chance to win the tender must comply with the must- have criteria. These criteria are often answered with yes or no.
    Example: National access? Yes or no.
  • not complying with should-have criteria is not a valid reason to disqualify a contender. It is, however, wise to comply to have a chance to win the competition. Should-have criteria are often less precise and cannot be answered with yes or no.
    Example: Search features? The competing products have big variations in how they approach search and the presentation of search results. They all more or less comply with the requirements, but Best Practice got the best score from the users.
  • Nice-to-have criteria are less important, but can still be critical to win a competition. These criteria can range from yes or no to more vague questions.
    Example 1: Support for SNOMED ontology? Yes or no.
    Example 2: NEHL welcomes creative solutions that can enhance the user experience or the overall value of the product.

Today, NEHL have agreements with most publishers and vendors where national access and support for enterprise search are included. A significant number of agreements are also invoiced in Norwegian Kroner. None of these issues has been easy, but the tender process is probably the best way to communicate the needs of your company or organization.

Many comment that tenders are more time consuming than direct negotiations. This may be true for simple acquisitions where there are not too many difficult issues. When the picture is a bit more complicated, much time can be saved by issuing a tender where requirements are not up for discussion.