Note that because of sycophancy bias, it is quite easy to nudge ChatGPT or similar LLMs into adopting pretty much any desired position, even with follow-up questions that seem quite neutral on their face. Some of Curtis Yarvin's recent LLM discussions have this issue as well.
Not an expert on anything here, but it seems to me that simultaneously:
a) Johnson was factually wrong in claiming Marxist roots for most of the antisemitism that existed in the 20th century, BUT…
b) he was spectacularly correct in effectively predicting the dominant, virulent strain of antisemitism in the West today: that coming from effectively Marxist leftist oppressor-oppressed ideologues.
This is a really interesting mode of interacting with ChatGPT, thanks for the idea!
Recently I was discussing Timothy Snyder's work and got the sense that historians are very negative on his work, but I started to suspect it was because of a faction who disagreed with his ideology. I started digging through with ChatGPT inspired by this post and it is quite interesting.
When using AI to to process such a long text, it's important to use full "agentic" mode so it can go through everything thoroughly. AI tools a couple years ago (and most free versions of AI today) are not able to use an agentic loop to be thorough enough for this task. It sounds like ChatGPT was using agentic mode, but I figured I'd cross check with Claude Code / Claude Cowork as well.
Claude Cowork (Opus 5 High) went through the text, flagged questionable claims and then researched them. I then had it publish everything as a nice web page:
It found 460 factual errors, 117 framing problems, and 23 contradictions. A lot of these errors are very minor details, so I also asked it to highlight the top ones. Kinda wild how easy it is to do this kind of fact checking nowadays; it'd also be interesting to do a comparison with other books.
AI can help resolve factual questions pretty well, especially when it links to its sources. For more subjective questions - it can't fully resolve them, but can help provide relevant evidence to either side of the argument.
(Note: Claude may have also made a few mistakes or misunderstood Johnson, and it's not like it has access to all the books he cites, but the analysis still gives an overall sense of the book's accuracy.)
ChatGPT is not a >>>reliable<<< source of information about anything. It's prone to hallucinations as we all know, and it It gives you indicators, but you have to check. The neural network of a historian's brain beats silicon.
I'm loathe to buy into the micro-gotchas game it loves to play but here's an early example.
It whirs: "After the Russo-Japanese War, Johnson says Japan took “the Sakhalin islands. Japan received southern Sakhalin, south of the 50th parallel—not Sakhalin as a whole. ... Getting 'southern Sakhalin' versus 'Sakhalin' wrong is a factual error."
Here's what Johnson wrote, in a full actual quote I can provide because ChatGPT can't for all sorts of legal and tech reasons, which is a BIG limitation:
"Immediately [Japan] issued an ultimatum to Russia, took Port Arthur and won the devastating naval battle of Tsushima in May 1905, assuring herself commercial supremacy in Manchuria, and taking the Sakhalin (KARAFUTO) islands as part of the settlement. [caps mine]"
"Per the terms of the Treaty of Portsmouth ending the Russo-Japanese War, the southern half of Sakhalin was ceded to Japan, with the 50th parallel north as the boundary line, becoming KARAFUTO Prefecture. [again, caps mine]"
HE'S NOT WRONG. [caps mine]
But even if a quibble continues into a slap fight, this being the internet, who cares, NO respectable historian would assess another's work this way. Yet this is the only analysis ChatGPT can do, because ultimately it doesn't understand anything. It can only evaluate truth based on patterns, which is totally superficial, misses context and lacks judgement.
Oh wait, I spoke too soon.
ChatGPT has the cyber-gall to say "[Johnson's] problem is different: he writes with exactly the same supreme confidence when he has the facts nailed down, when he is advancing a controversial causal interpretation, and when he has simply gotten something wrong."
FFS. ChatGPT is talking about itself.
Pull the plug. Because if you don't, and the power goes out for us for longer than a few hours, we are f**ked.
I'm sorry, but do you not understand how completely stupid this is? Paul Johnson is not a historian. He's a journalist and a noted conservative ideologue who deliberately exaggerates and misstates facts in all of his works. He is an absolutely terrible example to use to try to fact check how historians write about history.
If you want to tear into a real work of history and see what ChatGPT has to say about it try something like John Dower's Embracing Defeat, which won both the Pulitzer Prize and the National Book Award. Any mistakes you find there would be much more interesting.
I haven't read the book, but a lot of these errors seem not particularly important. Getting the date wrong at which Japan took Korea would matter in a book about Japanese history, but does it matter here for his central thesis?
Interesting how AI can systematically audit a book like this—something a human reader rarely has time for.
My work with retirement planning often hits a similar wall: clients trust a financial product's reputation until I show them the errors in the fine print. The method matters more than the verdict.
Caplan's 'show your work' challenge is one more example of how the right question, not the right answer, unlocks the real insight.
"One of his greatest strengths. Johnson is unusually willing to apply ugly empirical facts against movements with morally attractive stated goals."
So, like all neo-cons, he (rightly) bashes communism (that many would argue has morally attractive stated goals). Being a neo-con isn't a "strength" for anyone, much less a "great" strength.
Note that because of sycophancy bias, it is quite easy to nudge ChatGPT or similar LLMs into adopting pretty much any desired position, even with follow-up questions that seem quite neutral on their face. Some of Curtis Yarvin's recent LLM discussions have this issue as well.
Not an expert on anything here, but it seems to me that simultaneously:
a) Johnson was factually wrong in claiming Marxist roots for most of the antisemitism that existed in the 20th century, BUT…
b) he was spectacularly correct in effectively predicting the dominant, virulent strain of antisemitism in the West today: that coming from effectively Marxist leftist oppressor-oppressed ideologues.
This is a really interesting mode of interacting with ChatGPT, thanks for the idea!
Recently I was discussing Timothy Snyder's work and got the sense that historians are very negative on his work, but I started to suspect it was because of a faction who disagreed with his ideology. I started digging through with ChatGPT inspired by this post and it is quite interesting.
When using AI to to process such a long text, it's important to use full "agentic" mode so it can go through everything thoroughly. AI tools a couple years ago (and most free versions of AI today) are not able to use an agentic loop to be thorough enough for this task. It sounds like ChatGPT was using agentic mode, but I figured I'd cross check with Claude Code / Claude Cowork as well.
Claude Cowork (Opus 5 High) went through the text, flagged questionable claims and then researched them. I then had it publish everything as a nice web page:
https://claude.ai/code/artifact/3fa707bb-1766-452f-a038-eb5c8e637c3e
It found 460 factual errors, 117 framing problems, and 23 contradictions. A lot of these errors are very minor details, so I also asked it to highlight the top ones. Kinda wild how easy it is to do this kind of fact checking nowadays; it'd also be interesting to do a comparison with other books.
AI can help resolve factual questions pretty well, especially when it links to its sources. For more subjective questions - it can't fully resolve them, but can help provide relevant evidence to either side of the argument.
(Note: Claude may have also made a few mistakes or misunderstood Johnson, and it's not like it has access to all the books he cites, but the analysis still gives an overall sense of the book's accuracy.)
ChatGPT is not a >>>reliable<<< source of information about anything. It's prone to hallucinations as we all know, and it It gives you indicators, but you have to check. The neural network of a historian's brain beats silicon.
I'm loathe to buy into the micro-gotchas game it loves to play but here's an early example.
It whirs: "After the Russo-Japanese War, Johnson says Japan took “the Sakhalin islands. Japan received southern Sakhalin, south of the 50th parallel—not Sakhalin as a whole. ... Getting 'southern Sakhalin' versus 'Sakhalin' wrong is a factual error."
Here's what Johnson wrote, in a full actual quote I can provide because ChatGPT can't for all sorts of legal and tech reasons, which is a BIG limitation:
"Immediately [Japan] issued an ultimatum to Russia, took Port Arthur and won the devastating naval battle of Tsushima in May 1905, assuring herself commercial supremacy in Manchuria, and taking the Sakhalin (KARAFUTO) islands as part of the settlement. [caps mine]"
Of course I know nothing about Sakhalin, so I just went to Wikipedia - https://en.wikipedia.org/wiki/Japanese_invasion_of_Sakhalin:
"Per the terms of the Treaty of Portsmouth ending the Russo-Japanese War, the southern half of Sakhalin was ceded to Japan, with the 50th parallel north as the boundary line, becoming KARAFUTO Prefecture. [again, caps mine]"
HE'S NOT WRONG. [caps mine]
But even if a quibble continues into a slap fight, this being the internet, who cares, NO respectable historian would assess another's work this way. Yet this is the only analysis ChatGPT can do, because ultimately it doesn't understand anything. It can only evaluate truth based on patterns, which is totally superficial, misses context and lacks judgement.
Oh wait, I spoke too soon.
ChatGPT has the cyber-gall to say "[Johnson's] problem is different: he writes with exactly the same supreme confidence when he has the facts nailed down, when he is advancing a controversial causal interpretation, and when he has simply gotten something wrong."
FFS. ChatGPT is talking about itself.
Pull the plug. Because if you don't, and the power goes out for us for longer than a few hours, we are f**ked.
I'm sorry, but do you not understand how completely stupid this is? Paul Johnson is not a historian. He's a journalist and a noted conservative ideologue who deliberately exaggerates and misstates facts in all of his works. He is an absolutely terrible example to use to try to fact check how historians write about history.
If you want to tear into a real work of history and see what ChatGPT has to say about it try something like John Dower's Embracing Defeat, which won both the Pulitzer Prize and the National Book Award. Any mistakes you find there would be much more interesting.
I haven't read the book, but a lot of these errors seem not particularly important. Getting the date wrong at which Japan took Korea would matter in a book about Japanese history, but does it matter here for his central thesis?
Interesting how AI can systematically audit a book like this—something a human reader rarely has time for.
My work with retirement planning often hits a similar wall: clients trust a financial product's reputation until I show them the errors in the fine print. The method matters more than the verdict.
Caplan's 'show your work' challenge is one more example of how the right question, not the right answer, unlocks the real insight.
"One of his greatest strengths. Johnson is unusually willing to apply ugly empirical facts against movements with morally attractive stated goals."
So, like all neo-cons, he (rightly) bashes communism (that many would argue has morally attractive stated goals). Being a neo-con isn't a "strength" for anyone, much less a "great" strength.
ChatGPT says ...
" Unfortunately, the old Amazon review is now difficult to retrieve reliably"
I followed that link no problem. If the complaint is because Amazon hides reviews from ChatGPT, would ChatGPT accept copy/paste?