Does the present grant system encourage mediocre science

views updated

Does the present grant system encourage mediocre science?

Viewpoint: Yes, by attenuating peer review mechanisms, grant evaluation systems encourage mediocre science.

Viewpoint: No, although far from perfect, the present grant system acts to promote science.

In economic terms, the knowledge obtained through scientific research usually falls in the category of "public good," that is, a product of labor that benefits the population as a whole and for which there is no particular ownership. Although some scientific discoveries are treated as "intellectual property," through the use of patents or trade secrets, at least for a time, since the scientific revolution of the seventeenth and eighteenth centuries, scientific societies and academic institutions have placed a premium on the open publication of results. The questions of who should pay for the expense of producing research results and how to allocate resources among the putative producers of them has no obvious or easy answers.

Centuries ago, scientific work was often limited to those who could find patrons in among the ruling class or the very wealthy. As the potential of systematic research to benefit industry, commerce, health, and agriculture became apparent in the eighteenth century, governments established bureaus of standards, geographic surveys, and technical schools at which research could be carried out. In nineteenth and early twentieth century United States, the accumulation of massive wealth by a few families led to the endowment of major universities (Vanderbilt, Duke, Canegie-Mellon, Stanford) and Foundations (Ford, Rockefeller, Sloan). Both the government agencies and private foundations sometimes solicited proposals for research projects or new facilities, at times calling on external experts to decide between meritorious proposals. Thus the grant system was born.

The United States government played a relatively minor role in making grants to academic institutions prior to the World War II. During the war, the massive mobilization of scientific talent and rapid development of the atomic bomb, radar, and other military hardware made a strong impression on the national leadership. At the same time, the scientists and academic managers who had worked shoulder to shoulder with high ranking military officials and other government bureaucrats gained substantial political savvy and clout. The outcome was the establishment of a government commitment to fostering research and the education of research scientists at United States universities. A National Science Foundation (NSF) was established to fund basic science research, and the role of the National Institutes of Health (NIH) in funding medical research, both on its own campus and at universities, was greatly expanded. Other agencies also began or expanded their own grant programs. The federal government commitment to support research and increase the pool of scientists and engineers was strengthened in 1957 when the Soviet launch of Sputnik I, the first earth satellite, reminded Americans that technical feats were not their exclusive preserve.

Among the most remarkable features of the postwar grant programs was the understanding that the government would pay the full cost of the research. For a successful researcher on a university faculty, this might include: full salary and fringe benefits for the summer months, partial salary during the academic year if the work involved "release time" from teaching duties, stipends and tuition payments for graduate students working as research assistants, full salaries for postdoctoral fellows and technicians, the cost of supplies and equipment, the cost of travel to related professional meetings, and a subsidy (page charges) of the cost of publishing the results in a scientific journal. To this the university was permitted to add a percentage for "indirect costs" such as building maintenance and depreciation and the purchase of library materials. In the mid 1970s the rate of indirect cost charged by some elite private universities exceeded 100%.

The availability of such large sums of money was quickly factored into the planning process at many universities. New doctoral programs were added, new faculty hired, and new buildings built, often with federal assistance. There is little debate today that the system did increase the supply of scientists and produce significant new discoveries. But the system was bound to suffer some strain from its own success, as rapidly increasing numbers of scientists had to compete for at best slowly increasing funds. By the end of the 1970s new questions were being asked about the grant system. Is it fair? Is it efficient? While it produces much good science, does it actually encourage mediocre science?

A case could be made that the system is not only fair, but benevolent. Even if a proposal is not funded, the submitter is afforded the benefit of review by a number of experts at no cost, and is welcome to submit a revised proposal addressing any objections raised. On the other hand, new investigators will be at a disadvantage compared to established scientists, since the expert reviewers are more likely to have confidence in the ideas of an investigator with a proven track record. Further, there is also a danger of grants being made on the basis of style more than substance. Many books and short courses on "grantsmanship" writing proposals and securing grants, are now offered to teach scientists how to write proposals most likely to be rated highly by reviewers. As the No essay that follows notes, even the benefit of expert review is no longer guaranteed by agencies such as the NIH, where a triage system of ranking proposals to decide which proposals are worthy of funding allows some to be discarded without the deeper study that others will receive.

The grant system is certainly inefficient, in that a great many scientists will spend from several days to several weeks out of each year reviewing the proposals of other scientists and several weeks more writing their own proposals. When, as at present, the vast majority of proposals are not funded, some scientists will feel obliged to generate a much larger number of proposals so that at least some funding will be obtained.

Does the grant system favor mediocre science over risk taking, truly innovative research? The likelihood of funding is certainly higher for a proposal for which the reviewers can see a plausible outcome. Also more funds will be available for research obviously connected to a desirable social or medical goal, such as environmental remediation or a cure for breast cancer. Thus the setters of the political agenda often steer the direction of research towards certain applications. On the other hand, the recipients of research grants have considerable latitude in the actual implementation of their program. Frequently, some more exploratory work is done along with that outlined in the proposal. Program officers, the people who make the actual funding decisions and monitor the recipients' reports, generally have no interest in micromanaging the day-to-day course of research and allow their grantees considerable latitude as long as some progress is reported toward the goals of the original proposal.

Another way in which the grant system could be said to encourage mediocre science is the way it limits access to academic careers of scientists whose interests lie outside "popular" areas of research. Candidates for teaching positions at major universities are often told that they will be expected to bring in a preset number of dollars in research grant support in the first five or six years or they will not be recommended for tenure. Once tenure is achieved, the investigator's freedom to change direction in research, is principle, guaranteed. The rewards accompanying bringing in more funding, however, work against major changes in research direction, however, unless the new area has also become "fashionable."

That there are clearly aspects of the grant system that are counterproductive, the vast majority of scientists would agree. Whether there is a better system that can in fact be implemented in United States institutions without disrupting many highly productive researchers is another matter. Hopefully adoption of a new system will not occur without consideration of the pros and cons of the present one.

—DONALD R. FRANCESCHETTI

Viewpoint: Yes, by attenuating peer review mechanisms, grant evaluation systems encourage mediocre science.

Although grants are designed to promote "good" science, the process has become so cumbersome, clogged, and confused that, despite noble intent, the grant process increasingly encourages mediocre or "safe" science.

Science research, especially basic science research, is heavily dependent on grants from public and private institutions. By the year 2000, universities and colleges depended on almost $20 billion annually to fund their research programs, and most of this money was invested in science research. Grant-derived funding paid a range of research costs, from test tubes to the salaries of technicians and professional investigators.

Foundation funding in the private sector is directly related to economic growth. Private foundations are required by law to annually distribute 5% of the valuation on foundation assets. Accordingly, during times of economic growth, foundation giving, which is in many cases dependant on underlying investments (e.g., stocks) must increase. During economic recessions, giving usually decreases. Federal grant funds, although not legally tied to economic growth, also historically mirror economic trends. Accordingly, funding for science research derived from both private and federal grants increased throughout the 1990s. The rate of growth in the competition for those grant dollars, however, vastly exceeded that rate of real growth in funds available. Increasing competition and dependence on grants to fund increasingly complex and expensive research programs exacerbated pre-existing weaknesses in strained grant evaluation systems. Moreover, specific reforms such as triage and electronic submissions designed to cope with increasing numbers of grant applications are proving to have the unintended side effect of profoundly shaping the kinds of science research funded.

Grant awards are rapidly becoming a contest of grantsmanship, the ability to write proposals and secure grants, rather than being decided on scientific merit. This emphasis on the form and procedures of the grant evaluation process, rather than on the substance of the science proposed, continually forces researchers away from the lab and into seminars on the craft of grant writing. More ominously for science, the investigators are forced in many cases to develop research proposals specially designed to please grant review committees. When such an emphasis is placed upon politics over scientific merit, science research loses in several significant ways.

First, there is a loss of scientific diversity, as grant evaluation committees view proposals that have predictable outcomes as less of a risk to precious investment capital. Critics of the current grant process consider this trend a hidden drive toward safe science, away from the more adventurous research that throughout history has been the path to spectacular insights and advances in science.

Second, as grantsmanship becomes increasingly important, new investigators fight an uphill battle to gain funding and build labs. Already several steps behind seasoned principal investigators who know how to craft strong proposals, new researchers often struggle along on grants designed for new scientists. These grants, although of generous intent, are paltry on the pocketbook and unrealistically low with regard to the real costs of research. Although there is some funding of dissertation research, organizations such as the National Science Foundation (NSF) actively discourage graduate students from submitting grant proposals. Although often highly touted, early education and grants to scientists starting out on research programs are often insufficient. In fact, only about one of four researchers seeking initial National Institutes of Health (NIH) funding actually apply for the "easier-to-obtain" grants designed for researchers making their first application for funding as a principal investigator. More confining and debilitating to new researchers are early development grants, which carry restrictive clauses that prohibit researchers from seeking other types of funding.

Third, revisions in the grant review process—no matter how well intended—both entrench established lines of research and disproportionately fund research into politically "in vogue," publicity driven topics.

Despite the lofty rhetoric of federal programs charged with funding science research, the numbers regarding actual funding reflect an increasingly brutal reality for investigators at all levels. The grant process is extremely competitive. Most grant proposals are not funded, and the percentage of proposed projects funded has steadily declined since the mid-1980s to current levels at which only 10% of proposals are ultimately funded. In this environment, some scientists and their sponsoring institutions become proposal mills—putting out a shotgun pattern of tens of proposals in hope that one or two may get funded. The time cost is a staggering drain on scientists and scientific research. For many months of the year, investigators may spend more time on the grant application process than on actual research.

It would be unreasonable to expect that any evaluation system dependent upon human judgment could be free of bias and prejudice. Regardless, recent attempts to reform and streamline the grant evaluation process actually make the system more fallible and less reliable. For example, the NIH—the single largest source of grant funds for research in the biomedical sciences—has moved to a triage system to decide which proposals are worthy of funding that essentially labels some proposals as unfundable, without even carrying out a full peer review of the merits of the proposal. NIH grants and funding have unquestionably led to significant and revolutionary advances in biomedicine and healthcare. The NIH deserves ample credit for these, just as the NSF deserves praise for past success. Regardless, it remains debatable whether the current system, beset by a crush of applications, can continue to promote good science—and aggressive science—when faced with a grant crisis precipitated by a crush of applications and threatened declines in funds available to researchers in less robust economic times.

Under the current grant review process, researchers or their sponsoring institutions, the proposers, submit their research ideas and strategy, as in the case of NIH proposals that are directed to various study sections composed of members with expertise in the area of research. Other foundations and institutes have similar procedures, often based upon NIH or NSF models. In the specific case of NIH, the study section members score the proposals and then rank them using a priority score. NIH priority scores range from 1 to 5, with 1 being the highest priority, thus the higher the priority score, the lower the chances of funding. Funding eventually establishes cutoffs, priority scores over which no funding will be given. In recent years, no proposals over 2.1 were funded.

Regardless of the exact procedures, protocol, and terminology, similar models of evaluation often work against more open-ended, basic-science oriented research proposals. Basic science proposals usually contain a wider range of possible outcomes than do more narrowly focused goal-oriented projects (e.g., projects regarding a specific clinical application). Reviewers tend to regard this unpredictability as a negative quality, and the emphasis on lowered risk results in a higher priority score, reflecting a lower project priority, for the proposal under review.

It is certainly difficult to evaluate and assign a priority to an unknown outcome, but the essential and long-cherished concept of scientific adventurism lies at the heart of empirical science. To slant research toward a particular goal usually casts a pall on the interpretation of data. In a very real sense, it upends the classic scientific method of probing for fallibility. Moreover, it is a well-known axiom of science that researchers, regardless of discipline, often find the results they are looking for. With a specific goal in mind, even the most intellectually honest of researchers is prone to shade and interpret data that corresponds with and conforms to expected results.

The present grant system encourages mediocre science because it encourages predictability. Regardless of the terminology of the particular grant foundation, researchers who fail to predict all possible outcomes for a project receive worse priority scores. As a result, the project becomes less a scientific inquiry and more an exercise designed to validate predicted results.

Political factors also influence funding. While including nonscientists on grant evaluation teams is a popular trend at many foundations, the inclusion of nonspecialists often means that evaluations of proposals are swayed and influenced by trendy, fashionable, or "politically correct" factors. Hot topics in the news or issues of special social concern often receive elevated evaluations that result in a lack of funding to more scientifically worthy projects. Much as financial stock prices may soar on naked speculation about any company with the word "biotech" in its title or prospectus, speculation—in the form of a prospective judgment on the priority or worthiness of a planned research project—can also soar because a project is related to several other already funded research projects. Funding in this way simply entrenches science.

If a proposal falls outside the interests of the evaluating committees or study group members, it is inevitable that an evaluation of the project's potential, particularly when graded by a priority score, will diminish. The fact that there is bias in evaluating groups is evidenced by the fact that substantially identical proposals can receive a wide range of evaluative scores. In an effort to counter this and reach the most target-specific evaluating groups, some researchers now attempt to limit the scope of their research and avoid interdisciplinary proposals.

In an increasingly global research environment, restrictive and burdensome clauses with regard to foreign versus domestic organizations are at best cumbersome. Public sources such as NSF will usually support only those portions of internationally collaborative research projects that are considered to be conducted by United States citizens.

Inundated with grant requests, larger granting institutions such as the NIH have instituted various triage-type procedures to handle the influx of grants. Whether triage-based policies and procedures work to alleviate problems of bias and prejudice remains a highly contentious issue. Many investigators contend that the attenuated review process hampers funding for innovative projects that might benefit from fuller consideration by the entire review process.

The difficulty in obtaining grants has created a vicious cycle. Because of the large number of grant applications, grants that are deemed "noncompetitive" at NIH are returned without full review. At a minimum, this eliminates much of the potential benefit traditionally derived from an investigator's ability to refine and resubmit improved proposals responsive to full peer criticism. Triage systems attenuate the peer review process and place an increased dependence on assessments of anticipated benefits or results. Some critics of the present systems argue that, in spite of the merits of peer review, the system is ripe for abuse in a modern era of intensely competitive science.

Stripped of full review, basic science proposals increasingly suffer, in part due to the trend of emphasizing clinical, or applied science. This trend fundamentally reshapes the intent of research, and results in mediocre science and a weak foundation upon which to build future "applied" research. The American astronomer Carl Sagan (1934-1996) often asserted, "There are many hypotheses in science which are wrong. That's perfectly all right; they're the aperture to finding out what's right. To be accepted, new ideas must survive the most rigorous standards of evidence and scrutiny."

—K. LEE LERNER

Viewpoint: No, although far from perfect, the present grant system acts to promote science.

Although there is no question that the grant application system can become a fickle procedural minefield for scientists, it is quite another matter to contend that the existing grant system encourages mediocre science. To analyze such a contention, it is important to clearly distinguish between the process of obtaining a grant and the scientific outcome produced by grant-funded research. And it is also important not to portray a diversity of funding sources as a one-eyed, one-source monolithic monster.

Foundations such as nonprofit organizations (NPOs), for-profit organizations (FPOs), and government entities (e.g., the NIH and NSF), provide billions of dollars to facilitate research in basic and applied sciences. A wide variety of other governmental agencies and private foundations also supply grants to advance scientific research. There is no question that there are problems in the methodologies of large granting agencies such as the National Institutes of Health (NIH) and the National Science Foundation (NSF). Review process problems, for example, may lead to grants being easier to obtain for studies involving the application of research rather than for basic science research. However, arguments that assert the existing grant system encourages mediocre science are contradicted by the only true measure of the current system—the steady output of scientific research, innovation, and advances.

Criticism of a process that is based upon anticipated results can be, at best, uncertain. In the worst cases, it represents a misguided attempt to change an already functioning system without any real data to support the proposed modification. There are admittedly often maddening, time-consuming hurdles for investigators to vault, and the grant allocation process is far from perfect. Regardless, as with all areas of politics—that being the appropriate term for a debate on the allocation of resources—interpretations regarding the current grant system are often "self-serving" rather than "science serving."

Science research grants fund proposals submitted by individual researchers or organizations, such as specialized research institutes, that submit proposals on behalf of individual researchers or research groups. The costs of doing research have spiraled upward in the past decade. With some notable exceptions in fields such as pharmaceuticals or human genetics where companies expect to make eventual profits, the process and progress of research is often dependent upon a mixture of public and private grant money. In the basic sciences, the dependence on research grant money is almost absolute, with little funding to be obtained from private sector sources.

The grant notices in any of the major journals or professional science publications of broad scope, such as The Scientist, reveal the typical spread of grant awards. On the monetarily modest end of the scale, a basic science research project in paleontology might receive a NSF grant of a few thousand dollars, while a botanical research project on the genetics of plants that might eventually prove beneficial in improved crop yields garners more than a $100,000 in multiyear funding from the same foundation.

Critics of the grant system often cite such examples in support of a contention that basic science often suffers at the benefit of research tied to potential applications. Such assertions often overlook the fact that studies with a wide gap in funding may significantly vary in duration and anticipated expense. More importantly, with finite resources, it is fair to argue that it is often prudent to place a value on the potential outcome of a research project.

When research has a direct link to biomedical issues, grant amounts usually soar. A recent study on the genetic basis of skin cancer announced simultaneously with the grants cited above provided approximately $1 million dollars to researchers. Again, elevated grant awards may reflect some combination of cost and an estimation of potential benefit but, as in the case of the grant for skin cancer research, the million-dollar grant came from a private foundation that makes awards in many areas of science and biomedicine.

Critics of the current grant process often focus on the problems related to obtaining federal grants. To focus criticism on the grant process, and to inflate the rhetoric of criticism to claim that the present system promotes mediocre science, denies the reality that there are a wide variety of funding sources available to researchers. Furthermore, alleged defects in the federal-review process must be taken in the context of the advantages offered by the system and the fact that there are other sources of funding for research projects.

Grants may be relatively narrow in focus, but of broad humanitarian and international application. For example, a research program seeking to establish a program to combat the transmission of parasites by developing programs to control insect vectors—the insects responsible for the transmission of particular diseases—garnered multiyear support from the private John D. and Catherine T. MacArthur Foundation of Chicago. This type of research is goal-oriented, with a specific target for investigators established at the outset. Such projects are often funded by grants from private foundations with broader humanitarian aims. In contrast, groups with narrow focus tend to selectively support the types or research most directly related to their specific interests. For example, a group dedicated to the study of paralysis, such as the American Paralysis Association, may provide hundreds of thousands of dollars toward finding a cure for paralysis caused by spinal cord trauma or stroke.

These types of awards, while goal oriented, may still foster basic medical research. For example, the grant for research related to paralysis specifically funded inclusive projects on topics of basic medical research related to in vitro analysis of spinal cord regeneration mechanisms, or the biochemical analysis of specific proteins found in high concentrations following spinal cord trauma.

Many grants awarded from public and private funds act to support good science at the most fundamental level. For example, some grants are designed solely to prepare undergraduate students for graduate education in science. Although certainly not basic science research, these types of grants are important in the training of future scientists. In a sense, they are the most basic and most fundamental investments in science.

The present system also contains checks and balances that promote good science by ensuring separation between the research lab and the marketplace. Programs designed strictly for the marketplace are the antithesis of rigorous scientific endeavor, in which, as American physicist and Nobel Prize winner Richard Feynman (1918-1988) once asserted, "one must be as ready to publish one's failures as one's successes…." Recognizing this truth, many grant agencies, including the NSF, specifically refrain from funding research designed to develop products for commercial purposes or markets.

There has certainly been a shift away from basic science research toward applied science research within granting agencies such as the NIH. Regardless, this trend is balanced by the actions of other agencies to specifically encourage rigorous "high" science, or the pursuit of basic science knowledge. For example, the NSF specifically discourages proposals involving particular medical goals in which the aim of the project may be the diagnosis or treatment of a particular disease or disease process.

It is also unfair to assert that requiring potential researchers to evaluate the potential merits of their work encourages mediocre or goal-oriented "safe" science. Being able to clearly state the objectives and potential of science research is neither imprudent nor constraining. Scientific serendipity has always been a factor in the advancement of science—but is, by definition, something that takes place along a path intended initially lead elsewhere. In almost all cases, it is difficult to evaluate the potential of a true unknown.

Arguments against the current grant system also generally ignore the fact that part of the grant review system is designed to assess the suitability of the methods researchers intend to apply to test their hypotheses. Lacking such oversight, the grant proposal process breaks down into political infighting regarding the value of potential outcomes. With attention to such review, the grant process encourages the foresightful and prudent application of science data and techniques. It is easy to argue that the current system may discourage the brilliant undiscovered scientist. However, the resources and funds available to support science research are finite, and that some rational process must exist to allocate resources economically.

The NIH and the NSF often fund science education programs. As long as faculty oversight is provided, some granting foundations may even award grants to graduate students conducting research programs intended to culminate in their doctoral dissertation. The present grant system also seeks to provide special support for women, minority scientists, and scientists with disabilities. As with direct grants to students, grants to faculty at nonresearch institutions, primarily those teaching at undergraduate colleges, are designed solely to provide the most fundamental support of science in the development of the next generation of researchers. Grants can also be used to remedy a shortage of investigators in a particular area of research.

Far from being exclusive, the present system of public funding also provides a mechanism for unaffiliated scholars and scientists, especially those with a demonstrated facility and capacity to perform the type of research proposed, to apply for support. These types of grant awards are admittedly increasingly rare, but a procedural mechanism does remain whereby significant proposals can at least be reviewed.

The current grant system—for all of its faults and political infighting—adds a layer of protection to scientific research by weeding out proposals with little merit. In addition, daunting as the grant process may be, even a failed proposal can be valuable to the earnest researcher. Because experts in the particular area of inquiry often review grant proposals, researchers may discover facts or data that can better shape future proposals or even assist in research. Reviewers also often help shape proposals by critically evaluating theoretical or procedural defects.

Reflecting a variety of procedures, most grant review processes promote good science by allocating resources based upon the significance of the project (including its potential impact on science theory) and an evaluation of the capability and approach of the investigator or investigative team. In particular, evaluating committees, especially when staffed with experts and functioning as designed, can help fine-tune research proposals so that methodologies are well integrated and appropriate to the hypothesis advanced. In cases of clinical and potentially dangerous research, such as the genetic alteration of microorganisms, the grant review processes provides supervision of procedures to assure that research projects are conducted with due regard to ethical, legal, and safety considerations.

With regard to promoting science, the present grant system is apparently the worst possible—except for all others. Although far from perfect, the present system represents a workable framework of critically important peer review. Admittedly subject to all the human fallibilities, it requires refinement, rather than general reproach, because it provides a measure of safety, quality control, and needed economy to science research.

—BRENDA WILMOTH LERNER

KEY TERMS

FOR-PROFIT ORGANIZATIONS (FPOS):

Companies that seek corporate profit for the benefit of their organizers or shareholders. In most cases, public grant sources require such entities to provide their own funding. Under unusual circumstances where the proposed research is, as termed in NSF guidelines, "of special concern from a national point of view," or is "especially meritorious," such projects may receive public funds.

FOUNDATION:

Legal entity operating to promote research and/or development. Foundations may be general or specific in their distribution of grants. There are more than a 250,000 private foundations chartered to operate in the United States. A small number of these foundations, however, contribute over one-half of all grant monies.

GRANT PROPOSAL:

Written proposal that outlines a research topic or question. The proposal contains preliminary data related to the projected research (i.e., outlines what is already known) and formulates a hypothesis regarding the subject of the research.

GRANT PROPOSERS:

Individuals and organizations submitting proposals for grants. In many cases, academic institutions (universities and colleges) make such proposals on behalf of the investigators who will actually conduct the research. In most cases, the grant proposer is responsible for administering and accounting for grant expenditures.

NONPROFIT ORGANIZATIONS (NPOS):

Organizations that do not seek profit for the benefit of their organizers. NPOs range from independent museums with broad interests, to research institutes and laboratories focusing on very specific areas of science. NPOs usually do not include academic institutions. NPOs may also be grant proposers and grant recipients.

TRIAGE:

From the French meaning "to divide into three," the term has its origin in the treatment of battlefield wounds. Wounded soldiers are usually triaged, or divided into three groups: (1) those who should receive only minimal attention, (2) those with wounds not severe enough to demand immediate attention, and (3) those to whom prompt medical attention can mean the difference between life or death. Whenever medical resources are scarce and the number of patients is more than staffing or supplies can handle, triage procedures are instituted to most fairly and effectively allocate resources. As applied to the grant awarding system, triage refers to disposing of those proposals not likely to be recommended for funding prior to a full review.

Science in Dispute