|Jet in Carina WFC3 IR [Courtesy NASA]
What are the standards for peer-reviewed scholarship?
David H. Bailey
Updated 1 January 2020 (c) 2020
Creationist and intelligent design writers insist that their writings constitute full-fledged scientific research. Creationists, for instance, hold that their notion that the earth and its living things (or even the entire universe) were created out of nothing a few thousand years ago is a scientific theory, every bit as much as evolution is a scientific theory. Similarly, leading spokespersons of the intelligent design movement have asserted that their movement is primarily a scientific movement, not a religious movement, and that "intelligent design theory" deserves a place in public school classrooms [Jones2005, pg. 24-35].
And in the same vein, many of the "new atheist" writers also consider their work to be important new scholarship that deserves to be taken seriously. See Atheists.
But numerous scholars argue otherwise, pointing out that little of this literature, from either camp, has passed peer review. Here is some background.
Peer review in scientific research
For centuries the process of peer review has been recognized as an integral part of both scientific research and the much larger world of scholarly research and communication. In the world of science today, a technical finding is not considered a bona fide scientific result unless and until it has passed peer review -- there is no such thing as non-peer-reviewed science. And many other fields of research hold similar standards -- work must be peer-reviewed before it can be taken seriously in the field in question. Along this line, note that whenever issues are "debated" in any other forum -- blogs, discussion forums, news columns, political campaigns, legislative bodies, television and radio, etc. -- such discussions are not to be taken seriously, particularly when the writers or speakers are not highly qualified researchers.
It is also keep important to keep in mind a principle popularized by Carl Sagan: extraordinary claims require extraordinary evidence [Sagan1998, pg. 60]. Thus manuscripts that make strong claims, such as that some long-standing theory is fundamentally faulty, or that a long-standing mystery has been resolved, or that long-sought experimental evidence has been found, then such manuscripts are scrutinized particularly carefully, and the authors are expected to provide exceptionally convincing reasoning and documentation.
The peer review process
The peer review system is remarkably similar across many disciplines, including all disciplines of science. It operates as follows. When a researcher (or team of researchers) completes a research project, the authors document their methodology, results and analysis in a manuscript, which is then submitted to a journal or refereed conference for peer review. The editor (if sent to a journal) or papers chair (if sent to a conference) then privately distributes the manuscript, electronically in most cases, to at least three other persons, chosen because of their knowledge and expertise in the manuscript's particular topic. In selecting referees, editors typically take pains to avoid persons who are in the same organization as one of the authors, or who otherwise might have a significant conflict of interest, professionally or financially, with the manuscript's authors. These anonymous referees rate the paper on criteria such as:
- Relevance to the journal or conference's charter. If the topic of the manuscript is judged not appropriate for the venue to which it was submitted, it may be rejected without any further review of its contents.
- Clarity of exposition. Most international scholarly research papers, and virtually 100% of scientific papers, are now written in English, which is the de facto universal language for research. A manuscript submitted to a journal should be written with the highest quality English prose, which often is a challenge, particularly for the growing community of international, non-native-English-speaking scientists. If the manuscript is murkily written, or if it has too many English errors, it may be rejected for this reason alone.
- Objectivity of style. Research papers must be written in a highly objective, modest, almost self-effacing style, openly acknowledging both sides of underlying issues. If editors or referees of a submitted manuscript find any bluster, hyperbole, chest-thumping, prejudice, unprofessional criticisms of other researchers or their work, or any other indications that the authors are not approaching their material with an entirely objective mindset, the manuscript will be rejected immediately.
- Acknowledgement of prior work. It is essential that the authors of the manuscript exhibit that they are fully familiar with the state-of-the-art research in the field, and have properly credited related work. Indeed, deficient documentation of prior work on the authors' topic is one of the most common reasons for rejection. Authors may choose to differ with the conclusions of earlier works, but if so they must explain in detail why they believe the earlier works were faulty or incomplete. Sweeping dismissals of previous research, especially in the absence of very compelling data and analysis, are usually seen as evidence that the authors are not really qualified to be addressing the issue in question.
- Freedom from plagiarism. The usage of other researchers' text or ideas without explicit citation is considered a serious breach of ethics in all fields of research, and, in most cases, is immediate cause for rejection. In sufficiently egregious cases, the authors may be permanently banished from the journal or conference to which their manuscript was submitted. Nowadays leading journals and conferences employ sophisticated plagiarism-detecting software, which often can detect overlap of even a few consecutive words of text with previously published papers. Along this line, it is generally considered inappropriate for authors to include more than a modest amount of text from their own earlier papers, as in introductory material. The submitted paper must include significant new material.
- Theoretical background. The authors must lay a firm theoretical foundation for their work. What precisely is the theory or principle that is to be tested or analyzed? What is the proposed methodology to test the theory?
- Experimental procedures and data analysis. In scientific research papers, this portion of the manuscript is typically scrutinized the most carefully of all. If the mathematical arguments and/or experimental procedures are sloppy, poorly documented or insufficiently focused on testing the hypothesis in question, the manuscript may be rejected regardless of its other merits.
- Statistical methods. Given the importance of quantitative data in modern science, it is essential that the researchers use the most appropriate statistical methods appropriate for their research. For example, most researchers nowadays take rigorous courses in statistical methods as part of their training. Even researchers in nonscientific fields of study such as history, religion, psychology, art or literature, are, increasingly, employing statistical methods in their work.
- Conclusions. Given all of the above procedures and results, are the conclusions truly justified, firmly based on the results presented? If the authors have read too much into their results, or if there are more prosaic explanations of their results, or if there is any indication that the authors are overstating their results, the manuscript will be rejected.
- Originality and importance. Even if all of the above items are satisfactory, if the manuscript's results are simply judged as not particularly useful or important to the field, the manuscript may be rejected.
When the editor (or chair) receives these reports, he/she decides whether: (a) to accept the manuscript as-is, (b) to accept the manuscript, provided that some relatively minor items identified by the referees are corrected or revised, (c) not to accept the paper as-is, but to reconsider if some relatively significant items are corrected or revised, or (d) to reject the manuscript outright. For many scientific journals and conferences, fewer than 25% of submissions are initially accepted or accepted pending minor revision. Others are rejected, require major revision (sometimes more than once), are subsequently submitted to another journal or conference, or are never formally published.
In spite of these daunting obstacles, hundreds of thousands of peer-reviewed articles are published each year in many thousands of journals and conference proceedings. This massive and rapidly growing body of work constitutes the core of what is properly referred to as modern scientific and secular scholarship.
Note that the process of anonymous peer review helps to keep the world of research free from "cronyism" and related ills. A manuscript submitted by a Nobel Laureate may be reviewed by a lowly postdoctoral researcher, and, if found wanting for one of the reasons listed above, the manuscript may well be rejected. It is important to point out that it is not appropriate for a referee or editor to reject a manuscript merely because he or she does not like the results -- solid, defensible reasons must be provided. And there are safeguards to ensure that there is no "conspiracy" in the research specialty to block papers with certain types of results or conclusions (see Conspiracy).
Is the process of peer review foolproof?
Obviously the process of peer review is not foolproof -- no human endeavor could ever be foolproof. Some highly questionable papers have been published, and some important papers were initially rejected. As a single example, there is growing concern that research funded by pharmaceutical companies is often not fully objective, and, in any case, that the medical research community must be much more careful in peer review of such work [Seife2012].
Indeed, the need for the biomedical arena in particular to tighten its standards was underscored rather dramatically by a recent Sokal-like hoax, as described in a October 2013 Science article [Bohannon2013]. Science journalist John Bohannon constructed a spoof paper describing the anticancer properties of a substance extracted from lichen. The article had major flaws, from a scientific standpoint -- for example, the effect of the substance was modest and essentially identical over five orders of magnitude of concentration, which to an experienced reviewer would have been immediate grounds to suspect the methodology and conclusions. What's more, there was no indication that the trials had been done with proper experimental safeguards for a human trial. The text, written by Bohannon, had been translated by Google to French, and then back to English, to give it the feel of a third-world author.
Bohannon then sent 304 variations of his paper to open-access journals, where publication costs are to be covered by the authors of accepted papers, but which are otherwise publicly available. By the time his article documenting the spoof was published in October 2013, 157 of the 304 papers had accepted, and only 98 had been rejected. Many of the journals that accepted the papers were based in third-world countries, with possibly less rigorous peer review standards, but several of these journals were affiliated with top-tier publishing houses [Bohannon2013]. This episode has sparked many journals and publishing houses to take a hard look at their peer review procedures. Several marginal journals have already been shut down, and other reforms are in the works.
Have creationist and/or intelligent design writers subjected their work to peer review?
While creationist and intelligent design writers have published their arguments dissenting from conventional science in various books and online articles, they have not, as far as anyone can determine, even seriously submitted these writings, much less have them published, in any reputable peer-reviewed scientific journal. For additional details, see Creationism and Intelligent-design .
This lack of peer-reviewed publications, or even serious attempts at submitting material for peer review, presents a severe obstacle for creationism and intelligent design to be taken seriously in the scientific world. If creationist and intelligent design writers (individually or collectively) believe that any of their technical issues have significant merit on purely scientific grounds, why do they not compose them into well-researched and well-analyzed articles and submit these articles to recognized peer-reviewed scientific journals?
After all, as emphasized in a recent Science letter signed by numerous prominent scientists (after brief mention of the prevailing theories of geology, big bang cosmology and evolution), "Even as these are overwhelmingly accepted by the scientific community, fame still awaits anyone who could show these theories to be wrong." [Gleick2010]. The only reasonable inference from the lack of publications is that these writers themselves recognize that the arguments and data that they have presented to date would not meet the rigorous standards required of serious peer-reviewed scientific literature.
Some prominent journals in the field of geology and evolution include the Journal of Evolutionary Biology, Evolution, the Journal of Geophysical Research, Science and Nature. Some prominent journals in the field of physics and cosmology include the Journal of Physics A: Mathematical and General, the Astrophysical Journal, the Journal of Astrophysics and Astronomy and the Journal of Cosmology and Astroparticle Physics. Each of these websites includes a facility for submitting papers.
Have the "new atheist" writers subjected their work to peer review?
But lest scientists and secular scholars get too smug in their assessment of creationist-intelligent design literature, it must be pointed out that the same criticism can be leveled at the "new atheist" literature.
It is a fact that in spite of the numerous widely read books and articles by the "new atheist" authors, hardly any of this material has been published in professional peer-reviewed academic journals. And, as with the creationist-intelligent design literature, the only reasonable conclusion one can draw is that these writers themselves recognize that their writings would likely not pass peer review. Indeed, this literature has numerous flaws, and published reviews have not been flattering. See Atheists for details.
Just as with the creationist and intelligent design writers, if any of the current crop of atheist authors believe that they have some new insights or arguments in religious studies, religious history, theology or science and religion, they are invited to submit manuscripts to a leading publication in the appropriate field. Otherwise, their writings will not be taken seriously by professional scholars in these fields.
Some prominent journals in the field of religious studies and religious history include Religious Studies: An International Journal for the Philosophy of Relgion, Religion and Theology and the Journal of Religious History. Some leading journals in the field of science and religion include Theology and Science and the Journal for Interdisciplinary Research on Religion and Science. Each of these websites includes a facility for submitting papers.
The process of peer review has been an essential part of both scientific research and the larger world of scholarly communication for centuries. It requires a huge amount of time and effort on the part of researchers, both to prepare manuscripts for peer review and also to review manuscripts written by other researchers. But the resulting peer-reviewed journals and conference proceedings are of significantly higher quality as a result. While the process is certainly not foolproof, and lapses have occurred of both types (accepting bad papers and rejecting good papers), it has demonstrated itself to be a highly effective means of uncovering truth.
With regards to both the creationist-intelligent design literature and also the "new atheist" literature, it is significant to note that, as far as anyone can determine, hardly any of these writers have even attempted, much less succeeded, in publishing their work in reputable peer-reviewed academic journals. This literature may be cast in scientific terminology, include references to history or philosophy, and have other trappings of serious scholarship, and may be persuasive to those who lack professional training in the particular fields of science, religious history or philosophy, but it is not real professional scholarship.
See also Atheists,
What is science?.