A friend of mine from Field X once served as a program officer at a major research funding agency. (Names changed to protect the innocent.) As part of a quality assurance scheme, he was asked to review the proposal process for Field Y. He was surprised that every proposal he looked at, whether funded or not, was rated very high. He asked the program officer for Field Y how proposals could be ranked if they were all rated so high. He was told to pay no attention to the rating, but to look at what the reviewer said. So my friend looked at a number of highly-rated proposals. He found one where the reviewer said the proposed research had already been done and the results published by a different investigator, concluding, “This is not a good proposal, but this is no time to reduce funding to Field Y.” (Field Y receives considerably more funding than most fields, and has for a long time.)

This story contains a lesson about the ecology of review processes. Reviewers rate proposals to determine which proposals to support, but that’s not the only use for their ratings. Leaders of funding agencies do not allocate funding to fields by reading all the agency’s proposals and reviews. They use summary measures. One of these is “proposal pressure,” meaning the number of highly-rated proposals within a field that cannot be funded because the field’s budget is too small. A field with more highly-rated proposals than it can support is “under-funded.” Right?

We in the computing research field often eviscerate the proposals of our colleagues during proposal reviews. Why are we so fratricidal? Is it to demonstrate how tough we are? If so, we’re hurting ourselves. People from other fields are happy to have fratricidal computing researchers in competition for interdisciplinary grants because there will be more funding for everyone else!

There are two kinds of responsibility in proposal review. One separates good proposals from weak proposals to ensure that good proposals are funded. The other ensures that computing research holds its own in funding with other fields. The computing research field gets better when we criticize weak proposals and recommend improvements. The field does not get better when our criticisms of each other are so harsh that that computing researchers get less of the pie.

(Contributed by John Leslie King, University of Michigan)

No related posts.