SIGNAL QUALITY
Some time in 2020 (summer, possibly later), I was on a phone call with one of my best friends - someone I grew up with from about the ninth grade on. Someone I spent a lot of time with, from ages 15 - 30. For some period of time, this is someone for whom I could finish his sentences, and he mine.
On this phone call, my friend explained to me how he quickly recovered from COVID thanks to taking Ivermectin. This surprised me. Everything I had been reading/hearing about Ivermectin as a treatment for COVID suggested that it was not effective at all.
Ever since that conversation, I’ve been pondering how it could be that someone with whom I had been so close for so long could come to such a different conclusion about an important matter.
In today’s information environment, we have many challenges. The speed and ease with which information travels is one. High-quality and low-quality information moves equally well around the globe in the blink of an eye. The advertising-based revenue model of most media outlets creates a basic alignment problem, where they are incentivized to cover news and write stories that are more likely to spur engagement (which rhymes with enragement), leaving many wonky or less engaging, detail-rich stories unseen or unnoticed. And as individuals we all have cognitive biases - Tim Urban’s deep dive into how tribal, sports fan-type thinking has driven a lot of politics in the last decade or so is highly informative on this front.
Setting all of this aside, the question I want to examine is this: How does a person seeking objective truth in today’s world assign a quality score to information? Going back to that conversation with my friend, he clearly was assigning different scores than I was to the various information available to us at the time.
I’m not suggesting that we should always land on the same answers to important questions. But understanding why we believe what we believe is important, as it is the only way we might improve on how we collect and process information in the future.
Since we can’t personally witness every consequential event that happens in the world each and every day - we must have trusted proxies. So, how does one sort out higher quality information sources from amongst all of the options?
In attempting to wrap my brain around this challenge, I reached out to someone I follow on Twitter - Jay Rosen, professor of journalism at New York University. Jay’s critiques of news outlets’ coverage of politics in recent years has been revelatory. I wondered if he knew of any tools to help consumers of media to assign “quality scores” to news outlets or reporters. He kindly made me aware of a service called NewsGuard.
NewsGuard is a service that assigns “trust scores” to more than 8,500 news and information websites. Subscribers to this service download an internet browser plug-in that makes the scores available as they browse the web in real-time. This service was created by a team of journalists, who continuously assess the credibility and transparency of news and information websites based on the following nine criteria:
1. Does not repeatedly publish false content.
Factual errors are generally minor.
Major mistakes are quickly and transparently corrected.
The site doesn’t quote other news sources that frequently make false claims.
2. Gathers fair and accurate information responsibly.
They reference multiple sources.
Sources present direct firsthand information on subjects or events.
They do not egregiously distort or misrepresent information.
3. Regularly corrects and clarifies errors.
Effective practices for identifying errors.
Transparent acknowledgement of errors when they occur.
Does not leave significant false content uncorrected.
4. Handles the difference between news and opinion responsibly.
Clearly distinguish News reporting from Opinion writing.
When reporting news, does not “cherry pick” facts to advance opinions.
Content providers who advance a particular point of view disclose that point of view.
5. Avoids deceptive headlines.
No false info in headlines.
No significant sensationalization, otherwise not reflective of the underlying story.
6. Discloses ownership and financing.
User-friendly disclosure of ownership and/or financing of the website/new outlet.
Notable political affiliations made clear.
Readers know who is funding the content and any relevant interests the owner or funder might have in the content.
7. Clearly labels advertising.
The site makes clear which content is paid for and which is not.
8. Reveals who is in charge, including potential conflicts of interest.
Those in charge of the content are identified on the site.
There is a way for readers to contact the site about editorial issues.
9. Provides names of reporters/content creators, along with either contact or biographical information.
Information about who is producing content for the site is made accessible to readers in a clear fashion.
The sites evaluated by NewsGuard are assigned a score ranging from 0 - 100, resulting in the following credibility tiers:
100 High Credibility
75-99 Generally Credible
60-74 Credible With Exceptions
40-59 Proceed With Caution
0-39 Proceed With Maximum Caution
As someone looking for something close to objectivity, these criteria seem reasonable to me.
Just as we must trust reporters to act as our proxies when reporting on the events of the day, so too must we trust the people at NewsGuard to fairly use these criteria when evaluating news sources, if we are to use their ratings when consuming news online.
NewsGuard is led by two veteran journalists and entrepreneurs: Steven Brill founded The American Lawyer, Court TV, and the Yale Journalism Initiative. Gordon Crovitz was publisher of The Wall Street Journal and a columnist for the paper.
Curious to see how my default news sites fared against these criteria, I subscribed to NewsGuard so that I could view the ratings. My primary news sources: The Wall Street Journal, Washington Post, New York Times, NPR, PBS, and the BBC - all score very high (most receive 100s). Sources like FOX News and OAN fall well short of these scores. (FYI - I am not sharing specific sites’ scores because the NewsGuard user agreement prohibits this.)
It’s important to acknowledge that this scoring system is not perfect, nor is NewsGuards implementation of it. In fact, nothing is perfect - reporters sometimes make mistakes, warranting retractions. I’m mindful to not let the notion of perfect be the enemy of good.
After the 2020 Presidential election, numerous news sources gave credence to unfounded theories of election fraud. Even if they didn’t outright report that such fraud had in fact occurred, the amount of airtime that they gave to proponents of these baseless claims created (at a minimum) a sense for many Americans that “If there’s smoke there must be fire.” Some people were so alarmed and outraged that they stormed the U.S. Capitol on January 6, 2021. This behavior was not unreasonable for those who trusted that a major fraud had actually occurred. Similarly, if doctors say that Ivermectin is a valid, effective way to treat COVID-19, some people will certainly rely on that guidance when making their own personal health care decisions.
We live in an age where virtually everyone has a high-quality camera and microphone, and are able to produce competent looking videos of all types. We are flooded with “content,” and if I look hard enough I can find someone saying just about anything. Even though such content has the look and feel of something that is trustworthy, it is apparent to me that a great deal of it is factually or contextually false. Whether due to a lack of reporting rigor, a lack of understanding, or simply a function of bad-faith actors pumping out falsehoods, when it comes to information our mindset should be one of Buyer Beware.
It’s a damn wonder that humanity keeps moving forward. Despite everything - biased reporting, misaligned incentives, individual cognitive biases, somehow we tend to get a little better every day. On a macro level, humanity is getting collectively smarter, it seems to me.
Paying $5/month to subscribe to NewsGuard will not ensure that you receive perfect information. But it might remind you to be aware of your news sourcing, and mindful that all sources of information have flaws. Even without subscribing, we can all benefit from considering NewsGuard’s nine apolitical criteria for evaluating information sources.
The world has always been chaotic. Perhaps slightly less so today, in some key ways. But when it comes to sourcing and processing information, it feels like the Wild West to me. It’s not rocket science - it’s much harder than that. Rocket science relies exclusively on proven principles of physics. We have to navigate our lives using information that contains so many flaws, both in it’s provenance and also in how we process it using our fallible critical thinking skills.
As adults, we have to own our experiences and our decisions. The people who have been arrested and charged with crimes related to the January 6th storming of the U.S. Capitol are facing real consequences, and the fact that they were relying on bad information has not been much help to their defense.
We can’t fix all the many flaws in the information environment, but we can improve how we evaluate the information to which we expose ourselves.