The Machine Plague

Artificial intelligence may be subtly polluting scientific discourse and impacting peer review processes.

Jonathan Ma

Apr 10, 2024 — 3 min read

AI systems like ChatGPT have demonstrated an ability to produce written work that can fool human experts. In the research community, some have turned to AI tools for assistance with literature reviews, paper drafting, and even peer reviewing their peers' work. On the surface, this may seem like a convenient way to speed up certain scholarly tasks. However, three new studies indicated that unchecked usage of AI in academic settings could undermine the integrity of the peer review system and homogenize scientific dialogue over time.

ChatGPT Leaves Lingering Signatures in Peer Reviews

One study, published on arXiv in March 2024, took an innovative approach to analyzing over 146,000 peer reviews submitted to top AI conferences before and after the release of ChatGPT.[1] By focusing on adjective usage patterns, the researchers were able to determine that between 6.5% to 16.9% of review passages displayed statistical signs of significant modification by language models.

Reviews judged more likely to contain AI-generated content tended to be those submitted closer to submission deadlines, lacking scholarly citations, and from reviewers less engaged in post-review discussions. While not conclusively proving any reviews were entirely computer-written, the study authors argue this level of machine involvement still poses risks to peer review standards. ChatGPT and other AI tools may be homogenizing feedback and skewing it toward model biases rather than meaningful scholarly critique.

AI Usage Spikes in Machine Learning Peer Review

A concurrent study in Nature found that while peer reviews for papers published across Nature journals showed no spike in potentially AI-modified language, reviews submitted to prominent ML conferences did.[2] By analyzing adjective frequencies before and after ChatGPT, telltale words like "commendable", "meticulous", and "versatile" that correlated more strongly with AI text stood out more prominently in conference reviews from late 2022 onward.

This suggests AI use for peer reviewing within the machine learning field has grown significantly more common since ChatGPT debuted, though other disciplines may not have followed suit as quickly. Both studies call for developing transparent guidelines on declaring AI assistance in scholarly work to maintain the integrity of scientific discourse.

AI Peers Review AI Peers

In a recent report from The Register, computer scientists have even begun using AI directly to review each others' machine learning papers.[3] While timesaving, this practice poses new conundrums. How can researchers verify a bot hasn't been naively fed confidential information or hallucinated nonsensical opinions? How might this homogenize feedback quality over time as algorithms influence each other?

As generative AI continues advancing, ensuring the scientific peer review process prioritizes diverse human judgment will grow more challenging. While AI tools offer writing conveniences, unchecked integration into knowledge-sharing networks risks polluting information ecosystems in subtle, hard-to-measure ways. Maintaining the epistemic diversity that drives new scientific ideas may require limiting machine roles to advisory rather than authoritative.

Implications for Learners

The effects of AI pollution reach beyond just academic research. Students learning from scientific papers could ingest diluted or misguided perspectives if review quality declines. And societal knowledge gains hinge on robust yet efficient circulation of facts between experts. If discourse becomes subtly skewed over time, will we understand complex real-world problems like climate change as well as if diverse points of view were given full consideration?

Going forward, interdisciplinary collaboration will be key to navigate these challenges. Computational frameworks like those developed in the arXiv and Nature studies provide an important starting point, but sociological analysis of evolving scholarly cultures is also needed. Researchers should reflect carefully on how and when generative AI can best augment versus displace human roles throughout scientific networks and knowledge transfer.

As tools evolve, transparency on machine involvement and consideration of societal impacts will be important for supporting open inquiry and maximizing knowledge gains.