Peer Review and the Ex Ante Assessment of Societal Impacts Page: 3
The following text was automatically extracted from the image on this page using optical character recognition software:
5. In response to pressures from both the scientific community and society at large, agencies
continue to experiment to find better ways to include societal impacts considerations in ex ante
We conclude this introduction with a few points concerning methodology. Our research attends more to
the process than to the outcomes of assessment. It is natural to want to know whether the process of
peer review is successful at predicting the outcomes of research impact. But answering this question
would involve a study orders of magnitude larger than our own. It would require access to information
that funding agencies either generally do not make publicly available (for instance, the actual content of
research proposals, the reviews those proposals receive, and so forth) or information that many funding
agencies do not carefully track. Although the academic outputs of research, such as publications or
patents resulting from grant-funded research are often tracked by agencies, reporting requirements for
broader societal impacts are generally much less stringent, if they exist at all.
In any case, the review process itself, including both the procedures set up by the agencies and the
reactions of proposers and reviewers to impact considerations, reveals important underlying
philosophical commitments concerning the nature of research, expertise, and the relation between
science and society. It is these philosophical commitments that we see as most determinative of the
overall science-society relationship - and least attended to by the parties involved. These underlying
commitments factor into the proposal and review process, as well as into the ultimate funding decisions,
in important and generally unrecognized ways. In this paper, we hope to throw some light on why and
how these underlying philosophical commitments influence both the process of peer review and
attitudes toward the inclusion of impact considerations in the process.
In the following sections we summarize the peer review process as it functions at NSF and the EC. This is
followed by a discussion that draws points of comparison between the two agencies, and some final
concluding remarks. The descriptions of the peer review processes at both agencies are drawn from
administrative documents obtained from each agency, from presentations by and discussions with
participants at two CAPR workshops (the first held in Washington, DC in April, 2010 and a second held in
Brussels in December, 2010), site visits by CAPR team members to both agencies, and a survey of
stakeholders in the peer review processes at these agencies.3
US National Science Foundation - Broader Impacts Merit Review Criterion
NSF is the US federal agency that supports basic research across all fields of science and engineering
(with the exception of medical research, which is supported in the US by the National Institutes of
Health). With an annual budget of more than $7 billion, NSF is the funding source for approximately 20
percent of all federally supported basic research conducted by US colleges and universities. Over 95
percent of the funding proposals submitted to NSF (now routinely over 40,000 annually) undergo the
process of what NSF terms 'merit review' rather than 'peer review' - this to indicate that review by
peers is merely part of the larger funding decision-making process, which also includes NSF staff. In
other words, at NSF, peer reviewers do not make funding decisions.
Peer review takes place in one of three ways at NSF: via mail-only, panel-only, or combination (mail and
panel) methods. In mail-only reviews, proposals are distributed to reviewers (typically via email) who
then submit their comments to NSF without having had any contact with other reviewers. In panel-only
reviews, NSF convenes a panel of reviewers at NSF headquarters who meet to discuss a group of
Here’s what’s next.
This article can be searched. Note: Results may vary based on the legibility of text within the document.
Tools / Downloads
Get a copy of this page or view the extracted text.
Citing and Sharing
Basic information for referencing this web page. We also provide extended guidance on usage rights, references, copying or embedding.
Reference the current page of this Article.
Holbrook, J. Britt & Frodeman, Robert. Peer Review and the Ex Ante Assessment of Societal Impacts, article, 2011; [Guildford, United Kingdom]. (https://digital.library.unt.edu/ark:/67531/metadc38888/m1/3/: accessed April 24, 2019), University of North Texas Libraries, Digital Library, https://digital.library.unt.edu; crediting UNT College of Arts and Sciences.