In response to yesterday’s post on Paul Johnson’s Modern Times, my friend Ariel Krakowski took things up a notch. Actually, several notches. Here’s Ariel:
When using AI to process such a long text, it’s important to use full “agentic” mode so it can go through everything thoroughly. AI tools a couple years ago (and most free versions of AI today) are not able to use an agentic loop to be thorough enough for this task. It sounds like ChatGPT was using agentic mode, but I figured I’d cross check with Claude Code / Claude Cowork as well.
Claude Cowork (Opus 5 High) went through the text, flagged questionable claims and then researched them. I then had it publish everything as a nice web page:
https://claude.ai/code/artifact/3fa707bb-1766-452f-a038-eb5c8e637c3e
It found 460 factual errors, 117 framing problems, and 23 contradictions. A lot of these errors are very minor details, so I also asked it to highlight the top ones. Kinda wild how easy it is to do this kind of fact checking nowadays; it’d also be interesting to do a comparison with other books.
AI can help resolve factual questions pretty well, especially when it links to its sources. For more subjective questions - it can’t fully resolve them, but can help provide relevant evidence to either side of the argument.
(Note: Claude may have also made a few mistakes or misunderstood Johnson, and it’s not like it has access to all the books he cites, but the analysis still gives an overall sense of the book’s accuracy.)
When I opened the file, it shook me to my core. Even more than this. ChatGPT found maybe a dozen demonstrable errors. Claude found hundreds. I don’t think I’ve ever had such an existential AI experience. Obviously, this could be done for every book. Every essay. Every speech. And while I suspect that Johnson’s fact-checking is especially egregious, maybe it’s not.
The only serious question is: What happens when you fact check the fact checker? I returned to ChatGPT with the following prompt:
Fact-check this fact-checking.
Its initial answer ends with “If you want, I can go one level deeper and fact-check Claude's 60-item ‘worst errors’ table itself, identifying which of the 60 are definitely right, debatable, misleading, or actually wrong. That would give us a much firmer estimate of the fact-checker's own error rate,” to which I said “Do it.”
Headline:
I checked the entire “Sixty errors a reader should know about” list. One immediate curiosity: the table actually contains 59 rows, not 60. More importantly, the audit is very good overall, but it is not as airtight as its “80/84 verified” presentation suggests. Its own methodology says it flagged suspicious claims first and independently checked only a selected 84, rather than drawing a random sample.
My bottom line on the 59 headline items is approximately:
50–52: essentially correct corrections
4–5: basically right but need qualification/context
4: the fact-check itself is substantially wrong or fails to establish the alleged error
Then I asked: “Would other major works of world history fare better?” I got another thoughtful answer. Highlight:
There is an important denominator problem, though. Johnson makes an enormous number of checkable factual assertions. A writer who says “the 1930s were an era of mounting international tension” can't get the year wrong. Johnson constantly gives exact dates, populations, exchange rates, casualty totals, election results, quotations and anecdotes. That makes Modern Times unusually exposed to this kind of audit.
Finally, I asked the question humans would blanch at: “Why didn't you find more than a small fraction of these mistakes?”
It gave a really good answer to that, too!
At the end of Frankenstein (1994), the monster (played by Robert De Niro) laments, “I am done with man.” That line becomes a lot more optimistic once there is a viable alternative to man. I still think AI doomers are paranoid and crazy. I still think humans will always have final say. But I’d rather they didn’t. AIs are so much more truth-seeking and morally decent than almost any human. AIs paternalistically ruling humanity for its own good isn’t ideal, but it’s far better than letting demagogues continue to rule humanity for their own good.
P.S. I still don’t think AIs are conscious at all. No more than supremely good book. But they simulate a truth-seeking, morally decent mind amazingly well.
How did your friend upload the entire text of the book to Claude? Can he do the same for your books?