Home   Performance Measurement Documents

It is recognized that the measurement of research and development activities presents several unique problems when attempting to apply more traditional techniques of performance measurement and requisite data collection. The following discussion will focus on a approach being used by the Army Research Laboratory (ARL) as highlighted in an article by Mr. Edward Brown entitled, "Measuring Performance at the Army Research Laboratory: The Performance Evaluation Construct."

R&D Performance Measurement Issues
Traditional performance measurement activities focus on the application of clearly defined performance measures to evaluate process or organizational performance outcomes. Inherent in the performance measures are the fact that they provide some explanation of what is acceptable performance and that the data collected and analyzed be useful in providing the user with timely information on the performance of the entity being measured. Measuring the performance of an R&D environment often presents problems in, both the meaning of performance, and the timeliness of the data.

One measure commonly used to evaluate R&D performance is the number of patents obtained. This measure of performance has no bearing on the quality of the work being performed nor on the impact the R&D activity will eventually have on the scientific community, or the world. The timeliness of this particular data typically lags three to five years behind the submission of the patent request. Thus, if patents obtained is used as a performance measure, an observed decrease in patents issued and the appropriate corrective actions taken could lag current performance by 5 years or more. Finally, what is the right number of patents to be obtained from an R&D activity? This question clearly has no concise answer, making it very difficult to define acceptable performance.

What makes R&D activities so difficult to measure using traditional performance measurement techniques? Two primary factors have been identified to answer this question:

1. The outcome of R&D activities often can not be quantified in advance; and
2. The outcome of R&D activities may lag the output of the activities by several decades.

Put more succinctly, the unknown cannot be measured.

One approach to overcoming measurement of the unknown is to measure the effects or outcomes of R&D activity retrospectively. In the competitive environment of obtaining grants for R&D activities, it is often very difficult convince a potential sponsor to fund a current R&D initiative based on performance from 20 years ago. The Army Research Laboratory (ARL) approach attempts to measure its’ R&D performance outside the bounds of the more traditional performance measurement techniques and in doing so presents several data collection challenges.

ARL Approach for Measuring R&D Performance
As previously discussed, the ability to measure process or organizational performance outcomes is very difficult, if not impossible to define. Thus, the ARL began developing its performance measurement approach by asking the question, "What information does the stakeholder really want to know from a performance evaluation system, beyond what the ultimate outcomes and impacts of the research will be?" ARL determined that its stakeholders want information that will aid them in answering three questions:

The key to the issues facing the ARL was, how best to answer these questions, given the limitation of performance measures in evaluating R&D performance.

Implementation of the ARL Approach to Measuring R&D Performance
To answer these question the ARL used a combination of peer review, customer evaluation, and performance measures.

Peer Review
The Organization of Economic Co-operation and Development states, "Peer Review is the name given to the judgement of scientific merit by other scientists working in, or close to the field of question. Peer review is premised upon the assumption that a judgement about certain aspects of science, for example its quality, is an expert decision capable of being make only by those who are sufficiently knowledgeable about the cognitive development of the field, its research agenda, and the practitioners within it."

The ARL established a peer review group called the ARL Technical Assessment Board (TAB). The TAB membership consists of 15 world-renowned scientists and engineers, and under the TAB are six panels, each with six to seven members. The three purposes of the TAB are first, to review the scientific and technical quality of ARL's program, secondly to make an assessment on the state of ARL's facilities and equipment, and finally, to appraise the preparedness of the technical staff. The TAB assesses one third of the ARL program each year with results forwarded to senior management within the Army and the Department of Defense. The primary focus of the peer review process is in answering the question, Is the work of the highest quality. Data collection and analysis is in the form of an annual report based on the TAB review. Issues relative to the qualitative nature of the review and the validation of the data are of concern. However, the independence of the TAB from the ARL program serve as an adequate basis to minimize these concerns.

Customer Evaluation
Dr. Edward B. Roberts of M.I.T.’s Sloan School of Management has developed a model of the stakeholders for R&D firms. In his model, Dr. Roberts defines three groups of stakeholders. This first group is comprised of the development and manufacturing groups which are directly dependent on the results of research. The second group is the company's customer for the finished product(s) or service. The third group of stakeholders is the senior management of the company.

Feedback from the first group of stakeholders is gathered via an annual questionnaire to determine if the products delivered by the ARL were what was agreed upon, if the delivery was timely, if the product worked, etc. The application of this questionnaire is more directed at applied research. Research involved with fundamental science is more problematic in that the stakeholder is not clearly defined. In this situation the laboratory director provides the needed feedback.

Feedback from senior management is not obtained through a questionnaire, since ARL does not deliver any tangible products to the Department of the Army senior management. A Stakeholders' Advisory Board (SAB) meets once a year to provide the ARL with the feedback data needed to evaluate its performance. The SAB, chaired by the Commanding General, determines the degree to which the ARL program is effective along several dimensions (e.g., mission-versus-customer funding, in-house work- versus-contractual program, near term emphasis-versus-far term emphasis).

Performance Measures
As previously discussed, performance measures have only limited or no utility in evaluating the outcomes of R&D activities. However, performance measures can provide useful information on the operational or functional health of an R&D organization. Measures such as maintenance backlog, workforce diversity, procurement cycle-time, papers published, patents received are just some examples by which performance measures can be used in the R&D environment.

The following graphic depicts the relative utility of peer review, customer evaluation, and performance measures in answering the three questions stakeholders want answered in evaluation R&D performance.

	Relevance	Productivity	Quality
Peer Review	-	*	+
Customer Evaluation	*	*	*
Performance Measures	+	+	*
+ = Very Useful     *  = Somewhat Useful     - = Less Useful

Copyright 2005 Oak Ridge Associated Universities