How the NCI-supported Resources are Evaluated

• Measures proposed for NCI evaluation of human specimen resources
• Criteria to assess the performance of a resource
• Criteria to measure the impact of a resource
• Criteria to measure the continuing need for a resource

Measures proposed for NCI evaluation of human specimen resources

In general, the NCI expects an evaluation to answer three critical questions:

• How effectively has the resource performed?
• What impact has the resource had on research?
• Is there a continuing need for the resource?

The objective of the evaluation must be clearly stated and well formulated evaluation questions developed, regardless of which evaluation method is chosen. An evaluation can be conducted by NCI staff, by academic or industry scientists, or by existing advisory boards or sub-committees of such boards. The process should be as objective as possible. This argues for including participants who are not directly involved with the resource.

Objective measures generally are used to assess the performance or effectiveness of a resource. While the use of fully objective measures is desirable, appropriate evaluation also requires the use of subjective measures. The subjective measures, while harder to quantitate, are required to measure impact and are essential to the evaluation. Impact measures are required to determine whether a resource is serving a critical need. For human specimen resources in support of cancer research, the primary impact measure is the importance of the research findings resulting from use of the resource. Key research findings can be major contributions to our understanding of cancer biology, development or validation of a critical technology or the development of a useful medical device such as a diagnostic assay or a predictor of therapeutic outcome. There are a variety of different types of specimen resources, each of which has unique characteristics that must be considered in order to properly evaluate the need for the resource, its impact, and its performance. For instance, a longitudinal study collection will require many years to become a useful resource, and should not be evaluated in its development stage for how it is contributing to critical research needs. In terms of performance, a resource focusing on a single organ system should not be evaluated for the range of tissue types it makes available. Although these are obvious examples, they underscore the central point that judicious selection of appropriate metrics is required in any evaluation. While each resource may require different measures of its effectiveness, the fundamental criterion for the evaluation of any resource remains whether or not it is effectively meeting a critical scientific need.

Criteria to assess the performance of a resource

Performance is by far the easiest item to evaluate. In general the factors listed below are numerical and can be easily tallied if identified in advance. As with the factors discussed below, the choice of specific evaluation factors will be made in the context of what the resource was designed to accomplish and how it was expected to operate.

Useful performance measures include:

• The number of specimens the resource has provided to researchers
• The number of researchers who have obtained specimens from the resource
• The number of different specimen types that the resource has provided
• The number of difficult to obtain specimens made available to researchers by the resource
• The cost per specimen
• The number of repeat requests
• The number of research papers published by resource users
• Feedback from users on resource performance (quality, access, availability, timeliness, usefulness, etc. )

Criteria to measure the impact of a resource

The impact of a resource is the most difficult characteristic to measure, although impact becomes easier to measure with time as more researchers are served. Time is required not only for a resource to become established, but also for the critical research findings resulting from use of the resource to be published and appreciated. It is also difficult to find quantitative metrics that clearly aid in the evaluation of impact.

Useful impact measures include:

• The importance of published studies using resource specimens
• The number of citations by other investigators of resource-related papers
• The role of the resource in FDA approval of a medical device
• The role of the resource in development of useful technologies
• The role of the resource in development of useful research techniques
• Feedback from users on the impact of the resource on their research

Criteria to measure the continuing need for a resource

To determine whether there is a continuing need for an existing resource it is important to consider the nature of the resource, its original objective and how long it has been in existence. Before a resource is created, it is important to clearly state how it is likely to be used, who the users will be and the expected number of users, and other critical expectations.

Questions to determine whether a resource is meeting a continuing research need include:

1. To what extent is the resource meeting its objectives? Key questions include:
   • Is the resource being used as anticipated?
   • Are high quality specimens (and data) being provided (collected)?
   • Is there evidence that the resource is facilitating scientific progress?
   • Has the resource improved access to specimens?
   • To what extent is the resource filling a unique need?
2. Is the resource still needed?
   • Are alternative specimen sources available to researchers?
3. Does the resource still need NCI support or can it support itself?

The NCI always considers the cost of a resource since funds used to support resources might otherwise be available to support investigator initiated research. Comparison of the costs of resources is only useful when comparing two essentially identical resources. In addition, the cost of resources is not a dollar for dollar tradeoff against research funding, since the availability of NCI resource support reduces the cost to investigators of obtaining tissue specimens. It is difficult to compare the relative cost of funding acquisition of specimens directly as part of individual research grant support with the cost of providing them through a central resource. While it is relatively easy to assess the cost per specimen of operating a centralized resource, it is difficult to assess the cost per specimen collected as part of a research study, since that includes monetary costs, staff time costs and loss of research productivity costs.

It is even more difficult to try to determine the cost per critical research finding. This is true whether evaluating centralized resources or evaluating the cost of collecting specimens as part of individual research projects. The value of a given research finding is difficult to estimate and the importance of any finding may take many years to become clear. Because of these considerations, we did not believe specific cost metrics can be used for the evaluation of tissue resources.

The approach presented above provides a philosophical framework which if applied thoughtfully should allow one to focus on the critical issues needed to decide whether or not to initiate, continue, or modify a human specimen resource.