Articles - Journaltalk

I helped expose or correct errors in four published papers in mathematical economics: Tirole (1985), Kocherlakota (1992), Miao and Wang (2018), and Stachurski and Toda (2019). Knowing where each error lies, I asked four chatbots—Gemini, Refine, Claude, and ChatGPT—whether the results in those original papers are correct. The chatbots generated false positives—initially endorsing flawed proofs—and varied in how much guidance they needed before finding the error. ChatGPT Pro performed best, Claude was middling, and Gemini performed worst. A competent human paired with a frontier model can outperform typical referees, but chatbots alone cannot yet be trusted to validate mathematical economics. One aspect of my little experiment is that corrections of the original errors had already been published at the time of the experiment, and so a chatbot's success in catching the error might reflect its canvassing for published commentary, as opposed to its prowess in immanent mathematical inspection. Since the chatbots generated false positives, however, we know that they are fallible one way or another.