The following is a special contribution to this blog by Michael E. Gorman, a Professor in the department of science, technology, and society (STS) at the University of Virginia. Mike recently completed a rotation as a program director at the National Science Foundation, and co-funded a workshop on transformative research that took place in Washington, DC, last month.
During my two-year stint as a rotator at NSF, I looked for places where I could add value. There was a lot of discussion about transformative research and even some special funds that could be used for projects deemed transformative. In September 2007, the National Science Board (NSB) “unanimously approved a motion to enhance support of transformative research at the NSF.” The Board noted:
“The term ‘transformative research’ is being used to describe a range of endeavors which promise extraordinary outcomes, such as: revolutionizing entire disciplines; creating entirely new fields; or disrupting accepted theories and perspectives — in other words, those endeavors which have the potential to change the way we address challenges in science, engineering, and innovation. Supporting more transformative research is of critical importance in the fast-paced, science and technology-intensive world of the 21st Century.”
And it recommended the following addition to the NSF’s merit review criteria: “To what extent does the proposed activity suggest and explore creative, original, or potentially transformative concepts?”
In a January 4, 2007, talk, Transformative Research: The Artistry and Alchemy of the 21st Century, then-Director of NSF Ardent Bement, Jr., emphasized the way in which NSF works on the frontier where transformations are most likely to occur, but also recognized that we will:
“continue to quibble among ourselves about the meaning of ‘transformative research’, which as yet has no universally accepted definition. That is just as it should be. When concepts as complex as ‘transformative research’ are still emerging, we need to practice a kind of ‘constructive ambiguity’. Doing so will give us the flexibility to incorporate new knowledge and fresh perspectives as they arise; in other words, leave room for discovery. In that way, we can make course corrections along the way, adapt to changing circumstances, and remain open to diverse suggestions about the issues.”
I am a social psychologist of science and engineering who worked in NSF’s Science Technology & Society (STS) program and with the Science of Science and Information Policy (SciSIP) program. I knew these programs had expertise that could be brought to bear on this problem, so I catalyzed a workshop organized by Robert Frodeman and Britt Holbrook of the University of North Texas, co-funded by STS and SciSIP. [Many thanks to Julia Lane of NSF’s SciSIP program for her help vetting the proposal and co-funding the workshop (NSF award #SES-1129067). Any opinions, conclusions, and recommendations expressed here are mine, and do not necessarily reflect the views of NSF or any of its employees.]
Workshop participants included 25 invited practitioners and scholars from a wide range of disciplines: engineers, historians, philosophers, and science policy. Former and current NSF officials and representatives of other government agencies attended parts of the workshop.
The discussion was wide-ranging and deep; no simple summary will do it justice, nor was there a consensus. Alternative courses of actions included:
- Convince the NSF drop the transformative criterion. As a basic science agency, the best way for the NSF to ensure transformations is to fund what Kuhn called normal science, and wait for anomalies to appear. Then perhaps the NSF could target research towards anomaly resolution, which might lead to the kind of revolution Kuhn talked about, creating a new paradigm.
- Liberate the NSF from worrying too much about the definition of transformative. Keep it flexible and a bit vague while making the benefits of transformative research clear. The NSF could provide exemplars of previously funded work that turned out to be transformative.
The context of use should be included in transformation — a discovery can be transformative in terms of science but not use, and a discovery that is not transformative scientifically can be a catalyst for transformative innovations. Perhaps it would be better to substitute innovation for transformation.
Transformation occurs across an ecosystem — or parts of an ecosystem that many have broader or lesser impacts on other parts over time. Transformative research is part of a reframing of the scientific ecosystem, including research practices, research frontiers and potential applications.
Peer review may hinder transformation, because peers tend to reflect the existing paradigm. One indicator of a potentially transformative project may be a bi-modal distribution in peer reviews, where some see the transformative potential in the work and others regard it with horror — not on grounds of expertise, but because the new idea is incommensurable with existing thinking and practice.
One alternative to peer review is a sandpit process which was used by the NSF and the U.K.’s Engineering and Physical Sciences Research Council (EPSRC) to catalyze and fund transformative ideas in synthetic biology. Participants from multiple disciplines evolved ideas for transformative research over several days. Program officers picked several ideas from among those that emerged and invited proposals on them, with the understanding that the proposals were likely to be funded.
In the peer process, reviewers that typically come from the same research community are critically reading and evaluating proposals. In the sandpit, researchers from multiple communities share ideas and look for possible collaborators — and are told that transformative ideas are a priority. The sandpit process looks like a better bet for producing potentially transformative work.
The fact that the sandpit is deliberately interdisciplinary is one factor that increases its transformative potential. Combining two or more disciplinary communities on a new project is likely to produce a result that will appear transformative from the standpoint of any of the disciplines of origin. The further apart the disciplines, the more likely a sandpit discussion will produce work that is potentially transformative. Consider, for example, combining participants from social sciences, ethics, environmental science, computer science and civil engineering to develop new ideas on sustainability.
Interdisciplinary review panels are often formed at the NSF, but a proposal then has to satisfy all of the disciplinary reviewers — work that builds off all of the disciplines in the panel but transcends them — may fare worse. It would be good to do empirical work comparing sandpits and review panels and varying whether each was done within a single research community or across several. Which approach would be most likely to lead to transformative work?
The kinds of centers created by the NSF and other organizations could be catalysts for transformative research, especially if the right administrative infrastructure were put in place, one that encourages and supports radically interdisciplinary collaboration based on a solid foundation of disciplinary expertise. Again, empirical work could be done on the right sorts of structures.
The NSF has an important education mission as well, both in schools and universities and in informal settings like museums and social media. Given the emphasis on evaluation by disciplinary standards, it is harder to promote transformative thinking in formal education than informal. But there are lots of interesting options within formal education, e.g., interdisciplinary capstone projects and curricula.
The kinds of new scientific and engineering instruments placed in such centers and in national labs can also lead to transformative work by making it possible to explore and manipulate new aspects of the universe. Information technology has enabled many of these instruments and allowed them to be connected globally. IT has also enabled collaborations that have a global reach, and even virtual centers. Computer scientists not only do transformative research, but they also enable it in multiple fields of endeavor.
This is a very interesting post; sounds like a productive workshop. I like the original tone that was set by Director Bement’s comments, which you spotlight. Perhaps the most important result of NSF’s criterion for POTENTIALLY transformative research is the discussion by scientists and engineers, at NSF and the PI community, on the potential consequences of one’s own research and the research of others (and some of this applies to “broader impacts” criteria as well). I see this as particularly vital in Computer Science, which is transforming (intervening/pervasive in) society, but everyone, technologists included, seem more or less “along for the ride,” with problems of unanticipated consequences (often because there is really little/no effort to anticipate). As part of the transformative research discussion, I also like the attendant research into recognizing and characterizing transformations stemming from past research, for purposes of communicating science and its importance to the public and government, and to inform discussion/projections on possible future transformations.
I think that there are problems with the implementation of the POTENTIALLY transformative criteria, some of which you touch on.
* Some dismiss “potentially transformative” criteria entirely because of uncertainty as to what the future holds, while others dismiss the uncertainty and therefore drop the all important “POTENTIALLY” descriptor.
* For those that take the criteria seriously, there is often little time spent on it, at least with respect to individual proposals, largely because of other time commitments. Thus, the analysis in reviews may be quite cursory, with failure to recognize that research that is “incremental” on some dimensions may be (later) transformative on others — you nicely point out the multidimensionality of the incremental/transformative space. I think that analyses on potentially transformative consequences can also be cursory at NSF, largely because program directors are stretched to the limits with respect to workload. Also, the characterizations (by all concerned) rarely get into the possibility of negative transformations, which are therefore unanticipated, but not necessarily “unanticipatable.”
* There seems to be a lack of appreciation for the fact that there is a large diversity of researchers, some who will be moved to and good at addressing the potentially transformative (and broader impact) aspects of their research, and others who probably couldn’t do so very well even if they were inclined to, but whose outstanding research may very likely contribute in some ways to later transformations. This unfortunate feeling that each individual be a “balanced” researcher, rather than balance stemming from cognitive diversity and community, may be a root cause of the problems you note with interdisciplinary review panels; ideally, program directors compensate for some review panel biases and limitations because of their often broader perspectives, but again, PDs are often extraordinarily overworked, at least those who are biased to take these criteria seriously. Apropos some of this, I very much like the ideas concerning sandboxes. I also like initiatives that encourage larger teams that exploit cognitive diversity and community.
I sympathize with the view that NSF remove the “potentially transformative” criterion as a formal part of the review, because at least early on it seemed to diminish the perceived importance of basic research (normal science) as leading to transformations, but I see the “potentially transformative” formal criteria as important for transforming the scientific culture, including helping to drive interdisciplinary research, which you have very nicely spotlighted as vital for transformation. The cultural transformation need not be particularly “pretty” to be worthwhile, so I am somewhat accepting of the perceived imperfections in implementation. Hopefully your workshop will contribute to some of the course corrections that Dr. Bement spoke of.
Doug:
Thanks for your thoughtful comments. I especially like the reminder to emphasize potentially transformative. One great research study would be too look backwards from research now recognized as transformative and see how it was reviewed early on. Multiple case studies of this sort might help us see whether transformative work needs to be flagged early, or whether it just emerges.
I also like your remarks about CS, which has contributed to pervasive technologies. So will nano, an area I have been involved in. Both will lead to transformations. The question is, how can we anticipate and manage those transformations in ways that are beneficial to users? I think service science is offering promising suggestions, and the human centered computing program at NSF may also come up with good ideas.
If you have any more thoughts on this, I would be glad to hear them.
I’m very glad to see NSF digging deeper into transformative research sponsorship. As an unfunded PI whose project is interdisciplinary (Algorithmic Geometry — integrated Math-Computer Science education for high school), and as someone who has served on review panels, I definitely agree that monoculture panels are inhospitable to transformative projects, and at best, will produce a bimodal evaluation.
Another unfair obstacle for transformative projects is the 15 page written proposal. Written language is a system of symbols that appeals to shared experience to get across meaning. But a new groundbreaking paradigm, by definition, is NOT a shared experience, so language cannot be counted on to convey it. To get across a new paradigm, we need to open the communication channel up to primary sensory experience, so that the reviewer(s) can experience the phenomena that has gotten the proponents excited. This gives the proponents the ability to firmly tie down the meaning of new jargon and concepts.
It would be entirely feasible to allow interactive multimedia presentations, where the PI interacts with the reviewers. The only necessary restriction NSF should place is in the time devoted to the review of a proposal. In one hour of unfettered communication using multimedia bandwidth, I would have just enough time to get across the revolution that has swept through geometry the past 25 years, and how pilot students don’t find it any harder than the obsolete, pre-computational geometry currently passes for the Common Core curriculum. And, I could show how we are effectively imparting an “inventor prep” learning experience, where creative problem-solving from 1st principles on “wicked difficult” challenges is a skill that is both transferable and measurable. Without multimedia and interaction with the reviewers, I’m dead in the water, as NONE of them have done any Algorithmic Geometry, and probably won’t believe what we claim our students are doing is possible. And that shoots down any hope for getting across transformative impact.
NSF needs to dwell on this question: What constitutes a remarkable new breakthrough, what obstacles do the proponents face in sharing it, and what audience will be the best judge of societal impact if the project gains traction? I have recommended in the past that transformative panels be made up of inventors and other unconventional thinkers, as you want global thinkers with many interests who are good at “connecting the dots”.
One idea I think needs to be considered is involving a wide swath of the scientific community in these reviews, in effect harnessing the wisdom of crowds to spot the most transformative projects. Combining this approach with multimedia (and non-reliance on techno-jargon) strips away most of the obstacles current thrown in the path of highly transformative research in its early stages.