Yes, not sure if it is possible to do it with enough precision to avoid the biggest violator squirming out of evidence that they are systematically incorrect or biased, but I think that it is a great idea.
I am a big believer in transparent public indexes to force groups to compete against each other.
Thanks for sharing this paper! This looks interesting. We will incorporate it into the market-based projects. For the AI grading, I tried to make clear that we are attempting to avoid the authority/expert problem by grading past predictions where the outcome is not in dispute and by identifying logical fallacies. Even with cherry-picking, which relies most on a source of ground truth, the data used to assess cherrypicking is not typically in dispute (and when it is, we account for it.) But you are correct that the trust (and indeed the trust-worthiness) of the grading sources is the key problem, especially if people are motivated for social or psychological reasons to just dismiss any outcome that doesn't agree with their priors no matter what.
You might be interested to look at how the world of football/soccer transfer news and rumours works on X. Essentially over time fans created a tier list of journalists and at this point 2 figures, David Ornstein and Fabrizio Romano have established themselves as the top tier reliable standard for transfer news.
This is great! We were already noticing that sports and predictions about AI performance are the topics starting to launch prediction markets into the mainstream.
This is the only good idea I have heard that actually might revive the “marketplace of ideas” framework from the epidemic cesspool that online information has become.
Excellent. Factual checking of actual results of prior forecasts, many made without fear of being held accountable, will be a major win for public discourse.
There should be an additional category for repetition. I've noticed recently that information is repeated several times in the same article. They all seem to do it, liberal or conservative outlets. It isn't merely trying to establish something being a "fact" through repetition. It's also repetition of the same information of an earlier paragraph to fill space. Maybe photons are cheap.
At least I know it's not just a figment of my imagination. Thank you for that!
If you know anyone is the business ask them. See if you get an explanation. Normally news stories are written in the Inverse Pyramid format. This seems to contradict that or be some type of odd variation. I've also noticed there is a delay in naming names.
A Congressman from ...... Who? Repeat. A Congressman from....
Not: Joe Blow the Congressman from Kokomo. said.....
Have to keep scrolling to get that information. I did give some thought that maybe it's to keep you scrolling so you have to look at the adverts on the side or in-between paragraphs. But????
I've noticed this on Substacks that let everyone read the first few paragraphs for free and charge a subscription fee for the rest. The first few paragraphs are all of the form, "shocking new report shows <outrageous behavior by prominent figure|our team was right once again|surprising new finding that will change the world>", and deliberately omits details so readers can't just look up the "shocking new report" for themselves. The motivation here is obviously to get people to subscribe.
You mean it's all about the money? I'm shocked, just shocked! One thing of note is both Left and Right do it. At least in these days of divisiveness the country is united about one thing.
In a slightly more serious vein TV News has been doing this for decades. I'm dating myself, but back in the days of B&W TV the national news was a 15 minute program (Douglas Edwards I believe). That was expanded to 30 minutes about the time Walter Cronkite came along. You had 30 minutes of local news followed by 30 minutes of national news. Local stations realized they had a gold mine and expanded local news to an hour and then two. Largely repetitious. The half hour late news at 11:00 Pm before Carson (Steve Allen first and then Jack Parr, told you I was dating myself) became News at 10:00, and so on. I could go on about how (I think anyway) West Coast Newspaper switching from afternoon to morning papers kind of committed suicide but probably bored you enough.
Awesome! Have you guys thought of doing something like this for TV news? You could use a transcript or AI to record what they say and use similar methods.
I would love to see this applied to the "great vaccine debate," where the very term "misinformation" is effectively owned by "~intellectual elites" —who decide when and where it applies, yet its deployment can itself be a form of misinformation.
And those "~intellectual elites" who are so very sensitive to any downgrading of their intellectual grandeur, are sure to provide a "measurable" effect.
Propositions (1) and (2) make a lot of sense - there's is an enormous demand for misinformation and a deep market willing to supply it. I'm less sanguine about (3), trying to tackle it.
Humans have always had answers to every conceivable question. How did the Universe begin? We always has an answer to that question. There's never been a time in history when humans didn't have an answer, a story, for everything.
We are uncomfortable with "I don't know, and no else does either." As long as humans demand answers, they'll be lots of people willing to supply wrong ones. I don't think it will be possible to shame humanity out of its desire for "answers".
Many years ago I suggested to Paul Gigot (then and now editor of the WSJ editorial page) a regular OP-ED reporting on past predictions. All media gleefully report on alarming predictions made by politicians and scientists almost every day. Almost all political and religious dogma rests on predictions triggering fear, envy and guilt. Yet the media never follow up. Neither did he.
Isn't the problem here that she assumes her conclusions and is looking for "evidence" to support it? That cant be good. She may very well be right about 1) through 3) but "research" this ain't
Oh man, I love this. I hope she gets traction.
Yes, not sure if it is possible to do it with enough precision to avoid the biggest violator squirming out of evidence that they are systematically incorrect or biased, but I think that it is a great idea.
I am a big believer in transparent public indexes to force groups to compete against each other.
The wiggling is one reason we want to keep track of "ungradable predictions" and other vagueries and make sure they are counted against the speaker.
Sounds like a neat project; wishing all the luck to Ms. Bambauer & her team.
The problem with all these proposals is that they require trusted "experts". Why not a "retrodiction market" instead? See: https://papers.ssrn.com/sol3/papers.cfm?abstract_id=4880722
Thanks for sharing this paper! This looks interesting. We will incorporate it into the market-based projects. For the AI grading, I tried to make clear that we are attempting to avoid the authority/expert problem by grading past predictions where the outcome is not in dispute and by identifying logical fallacies. Even with cherry-picking, which relies most on a source of ground truth, the data used to assess cherrypicking is not typically in dispute (and when it is, we account for it.) But you are correct that the trust (and indeed the trust-worthiness) of the grading sources is the key problem, especially if people are motivated for social or psychological reasons to just dismiss any outcome that doesn't agree with their priors no matter what.
You might be interested to look at how the world of football/soccer transfer news and rumours works on X. Essentially over time fans created a tier list of journalists and at this point 2 figures, David Ornstein and Fabrizio Romano have established themselves as the top tier reliable standard for transfer news.
This is great! We were already noticing that sports and predictions about AI performance are the topics starting to launch prediction markets into the mainstream.
This is the only good idea I have heard that actually might revive the “marketplace of ideas” framework from the epidemic cesspool that online information has become.
*epistemic cesspool
This reminds me of Arnold Kling's Fantasy Intellectual Teams.
Excellent. Factual checking of actual results of prior forecasts, many made without fear of being held accountable, will be a major win for public discourse.
If it wasn't for lack of context, there would be no news.
There should be an additional category for repetition. I've noticed recently that information is repeated several times in the same article. They all seem to do it, liberal or conservative outlets. It isn't merely trying to establish something being a "fact" through repetition. It's also repetition of the same information of an earlier paragraph to fill space. Maybe photons are cheap.
I have noticed that, too. Not sure what to make of it.
At least I know it's not just a figment of my imagination. Thank you for that!
If you know anyone is the business ask them. See if you get an explanation. Normally news stories are written in the Inverse Pyramid format. This seems to contradict that or be some type of odd variation. I've also noticed there is a delay in naming names.
A Congressman from ...... Who? Repeat. A Congressman from....
Not: Joe Blow the Congressman from Kokomo. said.....
Have to keep scrolling to get that information. I did give some thought that maybe it's to keep you scrolling so you have to look at the adverts on the side or in-between paragraphs. But????
I've noticed this on Substacks that let everyone read the first few paragraphs for free and charge a subscription fee for the rest. The first few paragraphs are all of the form, "shocking new report shows <outrageous behavior by prominent figure|our team was right once again|surprising new finding that will change the world>", and deliberately omits details so readers can't just look up the "shocking new report" for themselves. The motivation here is obviously to get people to subscribe.
You mean it's all about the money? I'm shocked, just shocked! One thing of note is both Left and Right do it. At least in these days of divisiveness the country is united about one thing.
In a slightly more serious vein TV News has been doing this for decades. I'm dating myself, but back in the days of B&W TV the national news was a 15 minute program (Douglas Edwards I believe). That was expanded to 30 minutes about the time Walter Cronkite came along. You had 30 minutes of local news followed by 30 minutes of national news. Local stations realized they had a gold mine and expanded local news to an hour and then two. Largely repetitious. The half hour late news at 11:00 Pm before Carson (Steve Allen first and then Jack Parr, told you I was dating myself) became News at 10:00, and so on. I could go on about how (I think anyway) West Coast Newspaper switching from afternoon to morning papers kind of committed suicide but probably bored you enough.
Awesome! Have you guys thought of doing something like this for TV news? You could use a transcript or AI to record what they say and use similar methods.
Hopefully yes, some day.
Totally brilliant, JANE BAMBAUER!!!!!!!!!!!
I would love to see this applied to the "great vaccine debate," where the very term "misinformation" is effectively owned by "~intellectual elites" —who decide when and where it applies, yet its deployment can itself be a form of misinformation.
And those "~intellectual elites" who are so very sensitive to any downgrading of their intellectual grandeur, are sure to provide a "measurable" effect.
How can I help?
Propositions (1) and (2) make a lot of sense - there's is an enormous demand for misinformation and a deep market willing to supply it. I'm less sanguine about (3), trying to tackle it.
Humans have always had answers to every conceivable question. How did the Universe begin? We always has an answer to that question. There's never been a time in history when humans didn't have an answer, a story, for everything.
We are uncomfortable with "I don't know, and no else does either." As long as humans demand answers, they'll be lots of people willing to supply wrong ones. I don't think it will be possible to shame humanity out of its desire for "answers".
Many years ago I suggested to Paul Gigot (then and now editor of the WSJ editorial page) a regular OP-ED reporting on past predictions. All media gleefully report on alarming predictions made by politicians and scientists almost every day. Almost all political and religious dogma rests on predictions triggering fear, envy and guilt. Yet the media never follow up. Neither did he.
Isn't the problem here that she assumes her conclusions and is looking for "evidence" to support it? That cant be good. She may very well be right about 1) through 3) but "research" this ain't