If he had the EPUB (or .pdf) it would have been easy. The entire book would have fit into Claude's context. However, in my experience the context would have been too huge and confused the AI (perhaps the latest models don't do this) and it would have gone much better to have fed the AI one chapter at a time. The output suggests that he did indeed feed the book one chapter at a time.
How are you legally obtaining the epub and/or pdf and uploading it onto Claude? That's my question. Is Caplan now advocating piracy on his website?
For your context related worries, the post explicitly suggests that the author used an agent as opposed to a chatbot to bypass the worries about context window.
I agree, an agent (or a little script) would have fed the AI one chapter at a time.
As for copyright, I have hundreds of EPUBs for books that I paid for. I would have no qualms putting any of them into an AI for analysis. Most (all?) AIs would show no reluctance to do that analysis. If an AI decided to refuse my request on the grounds of copyright, it would have just lost me as a customer. If the copyright owners decided to come after me for using AI to analyze a book they’d sold me, we’d have to see how that goes. ‘molon labe’ as Leonidas responded when Xerxes asked the Greeks to hand over their weapons before Marathon.
For the analysis itself, nothing that either Bryan Caplan or Ariel posted would be a violation of copyright. It is totally fair to analyze a book and share that analysis.
I think there are two separate points; what is the legality of uploading books into a Claude/ChatGPT conversation and what is the morality of it? Legally, I'm not sure but I wouldn't be surprised if this went against certain copyright laws. Morally though, if one has already bought the book/PDF and made it available to Claude I don't think that should be considered wrong. Is it that different from letting a friend borrow your book? I'm sure Caplan feels similarly; if it's against the law, it's probably a law he disagrees with and doesn't see as a big deal breaking (see also: immigration).
I would think there would be a legal difference between using AI for analysis and loading the book into the AI for training. I am not a lawyer, but the first should pass copyright, but the latter wouldnt.
What Ariel Krakowski did requires advanced knowledge of how to use AI. It would be very interesting were he to comment on his process. I doubt it was a one shot query with an answer. In my experience, AIs go off the rails pretty quickly if under supervised.
While I would be delighted for Ariel to show how this is wrong, for his process I would imagine:
- ask for a chapter by chapter list of verifiable claims, output in some kind of machine legible format. The completion of this step might take a while.
- with that output, ask the AI to verify each claim and classify the result as 'verified', 'factual error', 'framing problem', 'contradiction', or whatever categorization you think best; the results go in an updated output machine readable file that adds the results of the search to each verifiable claim in step one. The completion of this step will definitely take 'a while' (depending on your plan, how much you want to pay extra for speed etc etc).
- with that output, ask it to analyze the results and present them as a single-file interactive HTML dashboard displaying summary cards, category filters, expandable details, and citations.
In Claude Opus, that would cost ~$20 (a rough guess). In an open weights model, perhaps $0.20 but more babysitting might be required through the steps.
To improve process, before you executed the plan you'd first give it to the AI and ask it to critique and refine. For instance, perhaps the AI has a better idea about how to categorize the errors. Frequently, the AI will tell me something I'd not have thought of that changes the plan ('clever!), or reminds me of something stupid and obvious that I'd overlooked (slaps forehead). With that critique, you update the plan and then execute.
As another improvement, you'd take the file of claims, errors and results from step 2 and give it to another AI to ask it to verify and critique the results. You'd then decide how to change the error file before proceeding to step 3, producing the HTML dashboard.
This is not to say that Ariel should have done more than he did. What he did was exceptionally impressive for someone who has not been deeply working with AI. However, once one is at Ariel's level of competence at working with AIs, then you always know how you could spend more money/time to get even better results.
I have found the "one more thing" aspect of LLMs to be extremely irritating when editing with them. When asked they WILL find an opinion to have or an error to flag. Much like, I suppose, human referees. But still, the process does not seem to naturally whittle itself down to a "good enough" product, especially if you hand things off between sessions/models. Each new request for feedback gives the same gross amount of feedback. It goes nowhere.
I've been fact-checking public school history textbooks with Claude. The textbooks contain hundreds of errors. They give the wrong dates for historical events. There are basic proofreading errors.
Would love to see you do this for your forthcoming book on free markets and capitalism. From what I see online, people say that deep expertise in certain areas are still beating the AI models (at least, according to their own judgement). Maybe you should try this on The Case Against Education too, or some other books you feel very confident in.
This is amazing. I hope he or someone else will do it for influential one-volume histories written by left-wing historians (Howard Zinn, Eric Hobsbawm).
Also of note, at the moment this seems primarily focused on the kinds of factual questions that can be checked in a “balls and strikes” kind of way. I assume it’s trickier to engage with questions like “what is modernity?”
Look, I think how specific and well-posed a question is matters a lot to what should and shouldn't be considered "scientific" or "rational". It used to be you claimed everything you say is literally true, and I felt most of it was ill-posed.
Today you are telling me the book has exactly 460 factual errors, 117 "Framing problems tendentious or contested", 83 self-contradictions,
AND
"80/4 verified as errors, 1 proven correct"
You can't show your work, you can't tell me what sort of standard is being applied, I don't understand what "80/4 verified as errors, 1 proven correct" means if anything. Do you?
What claim if any are you willing to stand behind?
I don't have a copy of the book, and a quick google search of the very first claim shows the information was communicated privately to Einstein in September, and without a copy of the book I can't really get any sense on whether Claude is being fair here or not.
> AIs are so much more truth-seeking and morally decent than almost any human.
Using the latest state-of-the-art AIs in real-world software engineering, this is, currently, definitely not always true. Even with expert prompting, my experience is a high rate of errors, some catastrophic. In the hands of an expert, I find a minor productivity boost. In the hands of a junior colleague of mine, it is definitely a net negative. They're still impressive, and maybe they'll improve.
There's a forest/trees issue in the fact checking process. Let's look at the forest for a moment. Do the myriad errors seriously undermine Johnson's various theses?
How did your friend upload the entire text of the book to Claude? Can he do the same for your books?
If he had the EPUB (or .pdf) it would have been easy. The entire book would have fit into Claude's context. However, in my experience the context would have been too huge and confused the AI (perhaps the latest models don't do this) and it would have gone much better to have fed the AI one chapter at a time. The output suggests that he did indeed feed the book one chapter at a time.
How are you legally obtaining the epub and/or pdf and uploading it onto Claude? That's my question. Is Caplan now advocating piracy on his website?
For your context related worries, the post explicitly suggests that the author used an agent as opposed to a chatbot to bypass the worries about context window.
I agree, an agent (or a little script) would have fed the AI one chapter at a time.
As for copyright, I have hundreds of EPUBs for books that I paid for. I would have no qualms putting any of them into an AI for analysis. Most (all?) AIs would show no reluctance to do that analysis. If an AI decided to refuse my request on the grounds of copyright, it would have just lost me as a customer. If the copyright owners decided to come after me for using AI to analyze a book they’d sold me, we’d have to see how that goes. ‘molon labe’ as Leonidas responded when Xerxes asked the Greeks to hand over their weapons before Marathon.
For the analysis itself, nothing that either Bryan Caplan or Ariel posted would be a violation of copyright. It is totally fair to analyze a book and share that analysis.
I am not sure paying for a book gives you the license to own the EPUB.
Well I'm hoping Caplan can do similar anlyses on his own books.
>Is Caplan now advocating piracy on his website?
I think there are two separate points; what is the legality of uploading books into a Claude/ChatGPT conversation and what is the morality of it? Legally, I'm not sure but I wouldn't be surprised if this went against certain copyright laws. Morally though, if one has already bought the book/PDF and made it available to Claude I don't think that should be considered wrong. Is it that different from letting a friend borrow your book? I'm sure Caplan feels similarly; if it's against the law, it's probably a law he disagrees with and doesn't see as a big deal breaking (see also: immigration).
I would think there would be a legal difference between using AI for analysis and loading the book into the AI for training. I am not a lawyer, but the first should pass copyright, but the latter wouldnt.
What Ariel Krakowski did requires advanced knowledge of how to use AI. It would be very interesting were he to comment on his process. I doubt it was a one shot query with an answer. In my experience, AIs go off the rails pretty quickly if under supervised.
While I would be delighted for Ariel to show how this is wrong, for his process I would imagine:
- ask for a chapter by chapter list of verifiable claims, output in some kind of machine legible format. The completion of this step might take a while.
- with that output, ask the AI to verify each claim and classify the result as 'verified', 'factual error', 'framing problem', 'contradiction', or whatever categorization you think best; the results go in an updated output machine readable file that adds the results of the search to each verifiable claim in step one. The completion of this step will definitely take 'a while' (depending on your plan, how much you want to pay extra for speed etc etc).
- with that output, ask it to analyze the results and present them as a single-file interactive HTML dashboard displaying summary cards, category filters, expandable details, and citations.
In Claude Opus, that would cost ~$20 (a rough guess). In an open weights model, perhaps $0.20 but more babysitting might be required through the steps.
To improve process, before you executed the plan you'd first give it to the AI and ask it to critique and refine. For instance, perhaps the AI has a better idea about how to categorize the errors. Frequently, the AI will tell me something I'd not have thought of that changes the plan ('clever!), or reminds me of something stupid and obvious that I'd overlooked (slaps forehead). With that critique, you update the plan and then execute.
As another improvement, you'd take the file of claims, errors and results from step 2 and give it to another AI to ask it to verify and critique the results. You'd then decide how to change the error file before proceeding to step 3, producing the HTML dashboard.
This is not to say that Ariel should have done more than he did. What he did was exceptionally impressive for someone who has not been deeply working with AI. However, once one is at Ariel's level of competence at working with AIs, then you always know how you could spend more money/time to get even better results.
I have found the "one more thing" aspect of LLMs to be extremely irritating when editing with them. When asked they WILL find an opinion to have or an error to flag. Much like, I suppose, human referees. But still, the process does not seem to naturally whittle itself down to a "good enough" product, especially if you hand things off between sessions/models. Each new request for feedback gives the same gross amount of feedback. It goes nowhere.
I've been fact-checking public school history textbooks with Claude. The textbooks contain hundreds of errors. They give the wrong dates for historical events. There are basic proofreading errors.
Would love to see you do this for your forthcoming book on free markets and capitalism. From what I see online, people say that deep expertise in certain areas are still beating the AI models (at least, according to their own judgement). Maybe you should try this on The Case Against Education too, or some other books you feel very confident in.
This is amazing. I hope he or someone else will do it for influential one-volume histories written by left-wing historians (Howard Zinn, Eric Hobsbawm).
Also of note, at the moment this seems primarily focused on the kinds of factual questions that can be checked in a “balls and strikes” kind of way. I assume it’s trickier to engage with questions like “what is modernity?”
Fact-checking ideological claims will just reimplement Politico. One piece of far-left dogma will be fact-checked against another.
What I want to know is why the ChatGPT answer about why it caught fewer mistakes than Claude is titled “Costco Fairfax Soft Drinks”!
I started in an old chat, and couldn't make it port the output to a new URL.
Look, I think how specific and well-posed a question is matters a lot to what should and shouldn't be considered "scientific" or "rational". It used to be you claimed everything you say is literally true, and I felt most of it was ill-posed.
Today you are telling me the book has exactly 460 factual errors, 117 "Framing problems tendentious or contested", 83 self-contradictions,
AND
"80/4 verified as errors, 1 proven correct"
You can't show your work, you can't tell me what sort of standard is being applied, I don't understand what "80/4 verified as errors, 1 proven correct" means if anything. Do you?
What claim if any are you willing to stand behind?
I don't have a copy of the book, and a quick google search of the very first claim shows the information was communicated privately to Einstein in September, and without a copy of the book I can't really get any sense on whether Claude is being fair here or not.
Good thoughtful essay and demonstrates a strong command of the subject matter.
Wouldn't it be fun to feed it one of Howard Zinn's works? Or a Lysenko screed in the original Russian?
> AIs are so much more truth-seeking and morally decent than almost any human.
Using the latest state-of-the-art AIs in real-world software engineering, this is, currently, definitely not always true. Even with expert prompting, my experience is a high rate of errors, some catastrophic. In the hands of an expert, I find a minor productivity boost. In the hands of a junior colleague of mine, it is definitely a net negative. They're still impressive, and maybe they'll improve.
There's a forest/trees issue in the fact checking process. Let's look at the forest for a moment. Do the myriad errors seriously undermine Johnson's various theses?
Chat basically says no, if you read the fact check of the fact check.