I posted on LinkedIn the other day a light rant about AI in science, and especially within my field of climate science. In this post I want to flesh out some more thoughts related to scientific papers and peer review, I'll cover some of the other topics in future posts.
I think it's important to start by stressing that even before the recent AI explosion, peer review and the publication process has not exactly been thriving. Recent papers I've been involved with are, on average, crawling through the peer review system. Editors I know have all emphasized that finding reviewers keeps getting harder, which I really do sympathize with. But it's not just the peer review process. Even finding an editor/posting the preprint/typesetting stages seem to be struggling. I had a paper that spent over a month at the typesetting phase last year, and I've been involved with papers that have taken weeks to even get an editor assigned. Not great. I am very pro non-profit community journals that aren't part of the crazy profit-making publication complex, but editors are volunteering their time in these roles and so it's perfectly reasonable that helping to push your paper through this system might not be their top priority.
So, (again pre-AI explosion) I was already thinking a lot about other avenues of sharing research outside of the peer review system, and have been trying to question more if/when my work actually needs to be peer reviewed. I've been quite inspired in this thinking by other research fields (and now the AI frontier labs) that brazenly post preprints and just...move along. The general idea is that the field is moving too fast to wait for peer review, people just want to know what we did. Genius. I do really think this is a concept more research fields should embrace - non peer-reviewed outlets for sharing updates using existing methods (e.g. new data updates, field reports, algorithm documentation). With co-authors involved, especially those judged heavily in academia by publication track record, inertia and the traditional route still seem to win out 99% of the time, but I think people are a little more open to the conversation than they used to be. At the very least I have radically increased focus on ensuring that preprint options are available and prioritized, even for papers I'm not lead on, and I feel way more comfortable sharing these and not waiting for a final paper release than I used to be. Obviously there are downsides with unvetted claims circulating, but my feeling is that there is a significant fraction of our peer review output that does not include anything that controversial and in need of significant peer review; freeing up resources for the papers that do truly need it. A risk, but a risk we really need to start taking in my view.
There are good examples that make clear to me a shift is happening. The Climate Brink Substack by Andrew Dessler and Zeke Hausfather is a non-peer reviewed blog that provides data-driven insights on timely climate science issues and, in my view, provides an extremely valuable contribution to the scientific discourse. I think it helps a lot that these guys are very actively engaged with the scientific community and contribute peer reviewed science often too, they aren't rogue bloggers who's science is never vetted. That's the balance that seems key to me. In the journals there are some signs of progress too. I recently submitted a paper to Earth System Science Data and noticed they now provide a "living data" path for describing datasets that are regularly updated/extended, linking things to prior papers and focussing efforts on describing what has changed. This path still involves formal peer review but at least removes some of the redundancy issues plaguing a lot of papers. I'm still looking for good outlets for sharing faster research updates that provide me with some enhanced visibility and credibility, but maybe that's something I need to develop myself.
So, that was where I was at before the recent AI explosion made things even more crazy. Now we're facing both a deluge of AI-generated/assisted papers to review, and deeper philosophical questions about the role of peer review and what a research paper should even look like now. As a reviewer, I appreciate the higher quality of output (really no more painful editing of language at all..!) but am getting pretty tired of reviewing lengthy papers that seem to be not quite AI-slop but often unoriginal, poorly framed and boringly executed papers. As a creator of research papers my feelings are quite mixed, I'm generally really enjoying the AI-help across virtually all parts of my research workflows, but there is a (increasingly smaller) part of me that feels a bit sad that what I considered my quite strong ability to draft decent quality scientific papers has quickly become nearly meaningless.
In response to this truly wild moment in this history of the scientific endeavor, the most visible change I've observed is a requirement from journals to add 'AI disclosures', usually tucked away at the end of the paper. I won't use the exact quotes as I don't want to name and shame, but after doing a quick review of recent papers in a couple of journals I engage with, it's quite universally statements like: 'Claude used to help with data processing and manuscript editing' or 'ChatGPT used for figure formatting' which seem pretty random and also a little hard to believe in some cases too. You've opened the box to this extremely powerful analysis and writing tool, and all you're using it for is making your figures look a bit nicer? Regardless of honesty, the disclosures are always extremely brief and not a helpful way of increasing understanding of how you've benefitted (or not) from this new incredible tool, and how others might be able to do the same. As I said in my original post, it really just feels like a box ticking exercise with minimal inherent value and I don't think they are going to age well. At the very least (and while us humans are still in charge of the paper writing process) they should really involve much clearer accounts of how AI was used.
I'll be honest that I'm struggling with a good set of thoughts on what researchers and journals should do about all this - I think there are short-term fixes to help reviewers & editors, but beyond that, it's hard to make an accurate prediction about this fast-evolving landscape. I think at the very least what I want to see is more open discussion and creativity in exploring alternative research output sharing, and much clearer communication (guides!) on how people are using AI to accelerate their science. That's my biggest current frustration. How papers are used to extract needed information needs to be considered much more deeply, but I appreciate we want to maintain some level of humanity and not just give the robots everything they need to take over. Journals and editorial boards do need to be prioritizing this challenge above anything else right now though in my view, testing out ideas and taking some risks. I think people will be keen to engage in the process if it's treated as a joint exploration of research dissemination, rather than a mandatory disclosure. There's a lot more to be said about all this, but I'll just end with some semi-random predictions/hopes:
I think the current system of peer review is not going to last. It will be replaced by a hybrid AI-human review system that can better and more efficiently: 1. test reproducibility (peer reviewers rarely do this end-to-end, despite what people think), 2. vet scientific claims. I hope and am somewhat confident that the idea of assessing papers based on how suitable they are for fancy (Nature/Science) journals will die a death, but it may take a while. In the meantime, I think journals will incorporate AI tools to help screen for AI slop and to help editors/peer reviewers evaluate manuscripts. I'm amazed it's taking so long to do this though, which again makes me think that new AI-forward journals that spring up will fare better.
Related, we will (eventually) see an explosion of new publishing models, driven by AI-savvy editors, who can help enhance the focus on AI science explanation/guides and accessibility for both humans and AI agents. Do we really need your beautifully crafted introduction that only a few people will actually read? In a growing number of cases, I think it will make more sense to have our chosen AI generate an efficient and concise introduction/summary/abstract for us - letting humans focus our energy where it's needed. Better prioritization and flexibility to the types of information and data we need to share, less emphasis on every paper following the same convention.
We will see a surge in other methods of sharing research output, led I'd guess by the growing contribution of researchers operating outside academia who are less constrained by outdated promotion/assessment metrics. Frontier labs are already showing the way (a mix of website updates, documentation, preprints and some core peer reviewed papers) that focus on what's most useful to the end-user. I am interested to learn more about the ESSD "Living Data" process and pleased to see there is progress here, even within the journals.
There is a very good chance I'm not thinking big enough here, and that AI capabilities are going to accelerate in ways way more terrifying/exciting (?) than I am predicting here. Very keen to hear what others think about all this though and what I'm missing.
How I used AI for this post: So here is my attempt to highlight how AI helped me craft this post (not something I consider mandatory!). I enjoy writing and the craft of distilling ideas into a format like this, and I want to maintain that skill as long as I can. But after finishing the first draft, I asked ChatGPT (Astra Light) to crtique what I wrote, check for grammer/spelling issues, and suggest any edits. I incorporated some of the suggested edits and removed some sections I agreed didn't add much to the conversation. Overall I really appreciate having the help and instant feedback.