AI unleashes a Pandora's box: ChatGPT generates convincingly fake scientific article

A new study published in the Journal of Medical Internet Research on May 31, 2023, by Dr Martin Májovský and colleagues has revealed that artificial intelligence (AI) language models such as ChatGPT (Chat Generative Pre-trained Transformer) can generate fraudulent scientific articles that appear remarkably authentic. This discovery raises critical concerns about the integrity of scientific research and the trustworthiness of published papers.

Researchers from Charles University, Czech Republic, aimed to investigate the capabilities of current AI language models in creating high-quality fraudulent medical articles. The team used the popular AI chatbot ChatGPT, which runs on the GPT-3 language model developed by OpenAI, to generate a completely fabricated scientific article in the field of neurosurgery. Questions and prompts were refined as ChatGPT generated responses, allowing the quality of the output to be iteratively improved.

The results of this proof-of-concept study were striking—the AI language model successfully produced a fraudulent article that closely resembled a genuine scientific paper in terms of word usage, sentence structure, and overall composition. The article included standard sections such as an abstract, introduction, methods, results, and discussion, as well as tables and other data. Surprisingly, the entire process of article creation took just 1 hour without any special training of the human user.

While the AI-generated article appeared sophisticated and flawless, upon closer examination expert readers were able to identify semantic inaccuracies and errors particularly in the references—some references were incorrect, while others were non-existent. This underscores the need for increased vigilance and enhanced detection methods to combat the potential misuse of AI in scientific research.

This study’s findings emphasize the importance of developing ethical guidelines and best practices for the use of AI language models in genuine scientific writing and research. Models like ChatGPT have the potential to enhance the efficiency and accuracy of document creation, result analysis, and language editing. By using these tools with care and responsibility, researchers can harness their power while minimizing the risk of misuse or abuse.

In a commentary on Dr Májovský’s article, published here, Dr Pedro Ballester discusses the need to prioritize the reproducibility and visibility of scientific works, as they serve as essential safeguards against the flourishing of fraudulent research.

As AI continues to advance, it becomes crucial for the scientific community to verify the accuracy and authenticity of content generated by these tools and to implement mechanisms for detecting and preventing fraud and misconduct. While both articles agree that there needs to be a better way to verify the accuracy and authenticity of AI-generated content, how this could be achieved is less clear. “We should at least declare the extent to which AI has assisted the writing and analysis of a paper,” suggests Dr Ballester as a starting point. Another possible solution proposed by Majovsky and colleagues is making the submission of data sets mandatory.

The article “Artificial Intelligence Can Generate Fraudulent but Authentic-Looking Scientific Medical Articles: Pandora's Box Has Been Opened” was published by JMIR Publications in its flagship title, the Journal of Medical Internet Research and can be accessed here.

Please cite as:

Májovský M, Černý M, Kasal M, Komarc M, Netuka D Artificial Intelligence Can Generate Fraudulent but Authentic-Looking Scientific Medical Articles: Pandora’s Box Has Been Opened J Med Internet Res 2023;25:e46924 

URL: https://www.jmir.org/2023/1/e46924/

doi: 10.2196/46924

###

About JMIR Publications

JMIR Publications is a leading, born-digital, open access publisher of 30+ academic journals and other innovative scientific communication products that focus on the intersection of health, and technology. Its flagship journal, the Journal of Medical Internet Research, is the leading digital health journal globally in content breadth and visibility, and is the largest journal in the medical informatics field.

To learn more about JMIR Publications, please visit jmirpublications.com or connect with us via Twitter, LinkedIn, YouTube, Facebook, and Instagram.

Head office: 130 Queens Quay East, Unit 1100, Toronto, ON, M5A 0P6 Canada

Media contact: communications@jmir.org

The content of this communication is licensed under the terms of the Creative Commons Attribution License (https://creativecommons.org/licenses/by/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work, published by JMIR Publications, is properly cited.