Reality Team Conversations: What is a Credible Source?
This week, Reality Team connects with Danny Rogers, Co-Founder and CTO of the Global Disinformation Index. Did disinfo play a role in the Capitol attack? We’ll be discussing what makes for a credible source of news, information or data: whether we need a definitive definition of “true,” and how to at least tell the difference between news outlets that are trying to report reliable, credible information… and those that aren’t.
Deb: Hi everyone. I’m Deb Lavoy. This is Reality Team Conversations. This week we’re going to talk to Danny Rogers about how to look at some unreliable sources of information and walk through how to tell the difference between real and not real. Danny has been on this show with us before and talked about the work they do to identify credible sources at the Global Disinformation Index.
Today we are going to walk through a few sites that share disinformation and misinformation around vaccines to give you a sense of what things you can look for that are fairly obvious and why this sometimes gets a little complicated.
Danny, how are you?
Danny: Hanging in there Deb. How are you?
Deb: Good. I think this is going to be a pretty interesting demonstration. Do you want to kind of set us up and give us a little context about you know what it means to be a credible source, a not credible source, and why you chose some of the examples you chose today?
Danny: Yeah. This came out of a conversation, Deb, that you and I had before about some of the pernicious nature of some of these sites that they look really deceptively, like credible – they look like government websites or they look like credible non-profits, when in fact they have ulterior motives.
The whole art of assessing credibility is a challenge that our whole organization kind of is committed to it and works on and iterates on and it is a never-ending kind of cat-and-mouse game, because ultimately what you’re trying to do is kind of glean the intent of the person who put the content up and that’s really hard to do through a website.
Deb: So to the extent that you can, you’re trying to create rules that are relatively objective for doing this, and just in our conversation right before this, you showed how that is harder than it seems.
Danny: Yeah. I’ve had this conversation a lot with people in the industry. There’s sort of this nostalgia for the time when printing a newspaper required a printing press and a distribution network and you know you had physical paper and there was some barrier to entry to making something that was printed and distributed to a lot of people. Even that is actually not as hard, but it was sort of very obvious that the newspapers were the ones with the big printing facilities and the trucks and the delivery and the person typing up the newsletter in their basement on a typewriter and mailing it to their friends was what it was. It was very easy to tell the difference.
An Example of How Disinformation Mimics Journalism: Bloomberg News vs Infowars
Now, we were just noting that the Infowars homepage looks deceptively like the Bloomberg News homepage. I can tell you that those are very very different organizations from a journalistic perspective, but if you just look at the style – the look and feel of the website – they look similar. If you didn’t know any better, it would look credible.
Deb: Do you want to share your screen? We’ve talked about a lot of this stuff – I think it’s really interesting to actually walk through it with you. Do you want to share your screen and show us Bloomberg and Infowars? We can start there and then walk through some of the vaccine stuff we were talking about.
Danny: Yeah, absolutely.
A lot of us are news junkies in some form or another, and this is the current Bloomberg News homepage and this is very familiar to all of us. One of the big things that stands out is this giant banner ad for some fund or something like that. Some investment thing, whatever that is, but you keep going down and now you have everything we know to be the Bloomberg News homepage.
Look how similar the Infowars page looks. This is not an accident: it looks very similar and undoubtedly on purpose, so that when you go to Infowars it looks like a thing you’re used to trusting.
Deb: It’s something that signals to you “Look – we look like a real newspaper, therefore you should believe us.”
Danny: Right, exactly, and it has the same look. It’s uncanny to the point of where it’s clearly intentional.
Deb: Right, so for those of you watching this, it’s fairly well-known that Infowars is really a front of a huge fountain of disinformation mostly-created to sell very questionable nutritional supplements, and by questionable, I mean completely fake and probably dangerous.
Danny: Well, this is where… so we’re in the business of I have my opinion, you have your opinion, everyone has their opinion, but how do we actually try to objectively assess the journalistic integrity of these two outlets?
Deb: Yes, let’s.
Danny: We have methodologies that do things like look at their journalistic practices, so we go to Bloomberg, and whether they’re listed on the page or whether they’re published somewhere, we look for their journalistic integrity indicators – things like their editorial policies, their corrections policies, and their editorial independence.
Deb: Is there a current statement of that on their site? (re: Bloomberg)
Danny: There are statements, there is transparency about who owns them, there is a separation of powers – Mike Bloomberg is not also the Editor-in-Chief of Bloomberg News, that’s a different person.
Even doing this is a bit of an investigatory challenge depending on who it is and what they choose to share and where in the world you are.
It’s pretty obvious who owns Bloomberg because it’s literally the name of the publication, but it’s also if you dig you find out who is in charge of editorial versus who owns it, whereas if you dig in the same thing here (Infowars), Alex Jones owns the owns the website and writes all the articles and is in charge of all of the editorial content.
Deb: Should we look into one of these articles?
Danny: We can, but I don’t even want to get into that level yet, because that’s kind of getting into the sort of nitty-gritty of “What do you think of this article or that article?” There’s definitely content elements like how often is a piece of content targeting a particular individual or how much of the content is biased.
There are definitely a lot of different signals that we use to assess both the content and what we call the operations pillars and things like that to look at credibility. I don’t expect everyone to do that. I think of this conversation, for example, more as in lieu of doing what we do professionally, which is investigate these organizations and score their journalistic integrity metrics and whatnot.
One of the things we notice a lot is this. I’ve now scrolled down on both of these where I’m blocking that top-level banner ad, so they look very similar. Let’s scroll back up and there’s something very obvious here.
They both have these big banner ads on top and they both look the same, but this banner ad is for some regulated investment asset and this one is for some DNA Force supplement pill. One of the things we see a lot on a lot of the sort of junkier sites is there are a ton of ads specifically for supplements – like supplements ads are what fills the space when there are no big expensive ads like what you see on Bloomberg. If I refresh this enough times, I’ll get other big brands or something like that, whereas here these are house ads for Infowars branded supplements.
Deb: The thing there is that it’s not just a supplement, but it’s actually a supplement sold by the owner of the site.
Danny: I think I mentioned this before we started, but NBC News has their own – they sell stuff too…
Deb: …but if you go to their home page, it’s not on the homepage.
Danny: Right. You can buy Dateline socks, but I don’t think NBC exists to sell Dateline socks, and if you go to nbcnews.com, there are no ads for buying Dateline socks at the top of nbcnews.com. The purpose of NBC News is not to sell Dateline socks.
Deb: That’s something that can be super obvious or it can be more subtle is what you’re saying.
Danny: Yeah. They’re selling sweatpants, but they’re not selling NBC News sweatpants at the top of the NBC News homepage.
Deb: That brand is going after the guys in my household pretty hard as well.
Danny: It’s the ad for the times, let’s put it that way. I will neither confirm nor deny what kind of pants I’m currently wearing.
This is something that you see a lot, so it’s a pretty obvious signal. If we go to some other sites – we were talking about switching gears to the sort of anti-vax universe of stuff, especially front of mind now, with the COVID vaccine rollout – these sites are… there’s sort of a universe of sites where you can just kind of look and see there is something up with this.
For starters, this is like a natural news sort of health website.
Deb: What’s interesting is I’m noticing the tagline “Defending Health, Life and Liberty” – that’s a very political interpretation of health news.
Danny: Exactly. That’s what is interesting about this particular site is that ostensibly and the roots of this site are in kind of the sort of bunk cure anti-vax universe; a huge amount of this site is political and a huge amount of it is aligned with a lot of the other sort of more politicized and far-right kind of stuff too.
Deb: Yeah, so the top-line headline there is about Biden-Obama pushing America toward war with Russia, which I don’t think has much to do with vaccines.
Danny: Exactly. But if you go down here, like this one – “The silver MIRACLE explained: It’s money, medicine and freedom all in one” – it’s driving…
[A pop-up abruptly appears]
…and this guy who runs it [in the pop-up], he has a podcast that was hosting the head of the Oath Keepers before the capital riots.
Deb: The Oath Keepers being a white supremacist…
Danny: …being a group that was recently indicted for seditious conspiracy. So there’s a lot of crossover here between the sort of libertarian-cryptocurrency world and – the alternative medicine does not do it justice, like… – the bunk cure world.
Deb: This is actually a thing where I think it’s really interesting. There are a lot of people who are really interested in health and wellness and diet and nutrition and alternative medicine and some of that is very well-intentioned. There is acupuncture, there is meditation, there is yoga, there is vitamin C – those sorts of things, and then there is whatever the heck this is.
Danny: Right – there is a line at which point it switches over from harmless to harmful. The yoga-meditation stuff is somewhere between beneficial and harmless, but when you’re talking about something from “use lemon juice to cure your cancer” to “don’t get the COVID vaccine”, etc, then it becomes increasingly harmful.
Deb: Right, and one of the things – this is a little bit off the topic of this – I really am interested in pursuing over these next few months is how do we engage a very honest conversation with people who are truly well-intentioned and curious and exploring in that harmless-to-beneficial space, but who are quite vulnerable to being drawn into the in-between space and then ultimately into a more truly dangerous space.
You can’t go to that health and wellness crowd and say “Only listen to the CDC”, because they’re just going to say “Look, that’s obviously not a thing”, and so we’re just going to have
to find in good faith a way to have the conversation that helps delineate the “this is useful and healthful” and “this is dangerous and toxic”. That’s a little bit off-target, but for anyone who wants to engage in that conversation, I’m super interested in the various people and philosophies and experts who might be able to help untangle that.
Anyway, go ahead.
An Example of How Disinformation Mimics Nonprofits and Government Websites: National Vaccine Information Center
Danny: Yeah, I totally agree, and that’s a whole separate challenge. From our perspective, just in the assessing the credibility part, you see some of the challenges in assessing credibility when everything looks the same.
As we were talking about and the impetus for this conversation, this particular page – which is particularly challenging – is it looks very official and it’s not just “it looks very official”, I mean it looks a lot like an almost-governmental home page.
Deb: If you look at that – just to point out to people who might not be able to see it as easily as you can – the name of this website is called the National Vaccine Information Center…
Danny: …which sounds very official. It sounds like a government function.
Deb: Right. It’s a dot org URL and some people will think that that means it’s legitimate.
Danny: And here’s the really the real kicker: they’re actually an approved 501c3 organization, so they actually have IRS approval to let people deduct their donations from their taxes and it gives them access to things like Amazon Smile (right here at the bottom) and Guidestar Seal of Transparency, which means that they are a top-rated non-profit.
Danny: Yeah, and Greatnonprofits’ 2020 Top-Rated Nonprofit.
Deb: Are those legitimate?
Danny: Well, Guidestar basically rates non-profits on how much of their donations go – based on their form 990s, what’s called their Form 990 with the IRS – like how they spend the money that’s donated to them. So as long as they spend some percentage of the money donated to them on the actual thing that they’re supposed to do and not just on salaries to executives sort of thing, then they get the Guidestar Seal of Transparency, which is some approval that says the money that you donate actually goes to the thing that they say they do.
Deb: Has anyone had a conversion with Guidestar?
Danny: Guidestar has a very simple rubric – like it’s an approved 501c3 and they say they do this and they get their money and their forms as they spend it on that, and all of that is probably true and kosher – it’s just the thing that they say they do and they spend their money on can be hugely harmful. As we’ve talked about, the Oath Keepers are also an approved 501c3 organization and they’ve just been indicted for seditious conspiracy.
Deb: Does that mean they lose their status?
Danny: No – that’s the thing. This is an open issue, but what happens when you have one part of the government saying “Go ahead and deduct your donations to this group from your taxes” and the other part of the government indicting them for trying to overthrow Congress.
Deb: So here we are at this site that has a legit-sounding name; it’s on a dot org URL; it’s got these seals of approval; it’s got an Amazon Smile account – which is by the way I saw that and that’s what made me call you last week if you remember.
Danny: Amazon Smile is just like Guidestar – they just take all the non-profits approved by the IRS. Except I should give them credit, because as of last week the Oath Keepers were finally kicked off.
Deb: Thanks Amazon. So how do you look at this then and understand whether or not you should be skeptical?
Danny: Well, that’s really a bigger question. For me, it’s a different story because it’s literally my job to make that assessment. The two things I do is look at what exactly are they advocating for and what they’re advocating for is vaccine skepticism, which goes against science, and therefore that’s a red flag to me, number one. But aside from the fact-checking part of it, our goal is to ascertain who’s behind this site and what is their intent, what is their motivation?
Deb: One of the things that I saw is that they have this search vaccine reactions.
Danny: There we go – searchable reaction database.
Deb: I got to this page and I got to one of the one of the reports. I was looking at this and I was fairly appalled. So imagine, if you will, someone sees a report on social media – because this is what’s going to happen to all of us, by the way – and if you have ever been fooled by one of these sites, just to be clear, welcome to the club. Everybody gets fooled by disinformation at some point or another, and we just got to get comfortable with the fact that we will get fooled, but we’ll learn from it and go forward. Disinformation isn’t for dumb people; disinformation is sophisticated and clever, that’s kind of what we’re showing you, and it will fool you sometimes.
If you go to the vaccine symptoms… I got to one of these pages, and this, by the way, looks like scientific data, and I don’t know what it was, but somewhere in one of these pages it said so-and-so died and the symptoms were that they were frothing at the mouth and they were having tremors, and because I spend all day looking at this stuff and have seen multiple debunks of the fact that there can be side effects and reactions, and none of them are those things.
Danny: This is sort of like throwing lots of science words at enough things to make it sound like you know there’s some connection, sort of a lot of scientific innuendo or whatever you want to call it, but like there’s no actual – this aside – science.
You ask me what I do – besides assessing the scientific integrity, which is not what we’re here to talk about, I want to know who’s behind it. This makes it very easy. There’s this guy Mercola behind it – it’s literally at the bottom. One of the things I do a lot is scroll to the very bottom of a lot of websites, down to the fine print of all these things. Whether it’s looking at the contact us or in this case very simply this guy Mercola is a known actor; he’s a known person in terms of someone who goes around funding and publishing all kinds of anti-vax and bunk cure stuff.
So you can go very quickly from a website that looks like credible and relatively benign with one click to the guy behind it and now you’re seeing all of the crazy stuff, like Bill Gates and Vitamin D and things like that and now you’re getting down to “lemon juice cures cancer” level type stuff very quickly. Now is where it is a very quick kind of jump and the fact that that link is so clear, this tells me…
Deb: What is the disclaimer down there saying? I’m curious. “…The entire contents of this website are based upon the opinions of Dr. Mercola, unless otherwise noted.”
Danny: Right, not intended as medical advice.
Deb: Well, none of them are allowed to give medical advice.
Danny: Right, but no one reads the bottom of this, right, and this is as much a copyright statement as anything else, but my point is this guy Mercola is a known sort of disinformation actor.
Deb: But I had never heard of him before you mentioned him last week.
Danny: Well, but that’s the challenge. I live neck-deep in this world to keep track of it, and I know all about this, and the people I work with know all about this, but I don’t know how other than looking for the ads for the supplements that you’d spot this, which was, by the way, one more click away and now we’re at Dr. Mercola’s Herring Caviar Oil.
Deb: So not only is it selling supplements, it’s selling supplements with the same name.
Danny: With his name on it. That’s what the whole purpose of this is and I will tell you I don’t think I’d want to spend a lot of time near anyone taking herring caviar oil, let’s just put it that way.
Deb: I won’t comment on that.
One thing I did last week when I was, you know, super… every once in a while I get triggered by something, I was super triggered by how legitimate that site looked at-first glance, it was very upsetting to me. One thing I did was I actually googled National Vaccine Information Center and it took me to a wiki page.
Danny: Right, that’s another thing that’s very easy to do…
Deb: …but only if you know to be skeptical and the reason I was skeptical is I knew that the content of those articles – I had enough knowledge of the science because I’ve been studying this stuff – that I knew it wasn’t true. So if you go here, you’ll see a very detailed takedown and history of this website.
Deb: So one thing you might want to do is if you get somewhere and you see something and you’re not – look, we’re all going to be worried and stressed out about vaccine information because it’s stressful and worrying to be in the middle of a pandemic, right?
Deb: You’re gonna end up on sites, the thing to do is if it’s not something you absolutely know or if it’s showing you information that you aren’t just like you’re not really sure, one thing is to google it and see what other people say about the site.
Danny: Right. This pretty much says it all – that it looks and sounds like a federal agency, but actually its entire existence is to oppose federal efforts to vaccinate children. It’s very simple.
Deb: So as you click around, you said that you look at journalistic integrity, but there are several other things you also look at.
This is interesting. So Danny just scrolled down on the Wikipedia page and this is a very carefully researched and very well-referenced.
Danny: It’s also quoting some of our research, actually.
Deb: Oh, that must be satisfying. We have a new person on our team who said “Oh, you guys should definitely create a little thing and it automatically tags a site as legit or not.” If only.
Danny: There are tools out there for that of varying quality. Newsguard has a browser extension that is mostly reliable. I take exception to a couple of their assessments, but that’s minor details.
Deb: So what that means is if you go to Newsguard, I think it’s newsguard.com, right?
Danny: I think it’s Newsguard Tech, but I don’t know actually. You’ll have to look it up. They have a browser extension you can install that will label links on your browser as to their assessment of them.
Deb: Right, and even if there is some debate around the edges, it is certainly better than nothing. We were talking a while ago that the stuff that is clearly a best effort to do good work that’s well-resourced and has the right talent, etc, is fairly easy to recognize, and the very clear bad guys are pretty easy to recognize, but the gray space is a lot harder. Something like a Newsguard extension can help.
Danny: For sure, there are a lot of tools out there that can help, but you have to install them and pay attention to them, which i hope more people do.
Deb: Absolutely. Talk to us about some of the other pillars that you look at when you’re evaluating a site. Not just journalistic integrity, you have several other pillars.
Danny: We have a number of tools that we throw at the problem. We also have some automated tools that look for what we call adversarial narrative content that lines up with other topics that we’re tracking and training on.
Deb: What is adversarial narrative content?
Denny: We identify certain common topics, content related to QAnon or content related to 5G conspiracy theories, things like that, and we identify content that’s similar in no two training sets that we develop. We also look at samplings of content from a whole variety of publications in media markets around the world and we assess them for things like polarization bias and targeting and click-baitiness of the headlines and all sorts of other indicators and we put everything together into a big sort of weighted hopper and use that to make our determinations. It’s relatively objective and relatively observable and we work with a lot of partners, especially in various media markets around the world, like we recently assessed the media market in Argentina or in Georgia or Latvia and we work with partners locally there who have local expert knowledge to do some of those content assessments and whatnot.
Deb: You’re also part of the international fact checking…
Danny: Well, we’re not. We’re not fact checkers. We’re not going in fact checking news, but we work with a lot of partners in these countries that are part of that network. Ultimately we’re not fact checking the articles; we’re looking at the meta characteristics of them or the operational characteristics of the publication as well as some element of the perceptions. We also do public opinion surveys in various media markets of the perceptions of different outlets: what do people in that country who read that outlet think of it? When was the last time they remember it issuing a correction or something like that? It’s a whole host of data from both automated platforms and manual analysis that goes into making a determination. We try to be as comprehensive as possible, but it’s not the kind of thing I would expect a casual internet user to do. That’s why we try to do it and build it into the tech fabric of the internet to abstract that away.
Deb: Interesting. So I guess the message to sort of normal people who are not as geeky as you are about examining newspapers, if you’re not nerdy into this whole field like we are, if something is important and particularly when it’s part of a really hot topic, such as the new COVID vaccine or a variety of political issues that are going around these days, you can do a few really simple things.
The first thing is if something makes you gasp, if something makes you say “Aha”, that’s a sign that this information has had some sort of an emotional impact on you and since that’s one of the prime goals of disinformation, be wary.
So sometimes I’ll read something in the New York Times and I’ll have that response, but if I have that response and it brings me somewhere totally unfamiliar – we call it take a sec to check – don’t react. If it’s important, it will eventually show up in one of the major news outlets or on the CDC website or on Johns Hopkins website. If it’s on one of these sites that you’re not really sure, just check out to see if the other guys are covering that story or take a second to google it. If it’s that important, you don’t necessarily need to share it before you know whether it’s real.
Danny: Yeah, definitely seeing who else confirms things is useful. The challenges come either when we’re walking about a subject matter or a part of the world that the New York Times doesn’t pay close attention to, and we can get into a whole conversation about different news outlets and what they cover, but there’s a lot that the western media does miss in parts of the world that we don’t pay attention to. Like, can you tell me the outcome and significance of the recent Kyrgyzstan presidential election? The answer’s probably not if you’re watching this in the US.
Deb: I cannot tell you.
Danny: But it happened and there were valid implications for people I happen to know who are worried about their future now. It also doesn’t work if your entire universe of “does someone else cover it well”, well, that guy who runs the natural news site, the content he produces is syndicated on a whole bunch of other sites too, so if I’m in this echo chamber where I’m like “Well, it got covered on Natural News, but I’m not going to believe it till it gets covered on Infowars also”, well that’ll be confirmed pretty quickly because they all share content for that purpose. There is a larger problem of the information bubble of the fact that it’s possible now with social media to basically exist in your own unique information universe where you’ll find confirmation of basically whatever you want to find confirmation of. There will be a piece of content somewhere that will confirm it for you in one way or another.
Deb: So there’s certainly a world of people who live in a disinformation bubble, but there is a, I believe, larger world of people who do not live deeply in the information space at all and they know that there are people who are lying to them, but they do not know which people are lying and which are telling the truth. This makes people really uneasy. I have a 21 year old son and I hear this from him a lot when we get into some of these subjects, and he says you know if he doesn’t like my argument, he’ll say “Well, everybody’s lying”, and he doesn’t want to engage the topic any further and I think that that’s the thing we need to work on.
Danny: That’s the problem. To me, that’s the real tragedy of the whole thing. Yeah, you can approach the internet with some healthy skepticism, but eventually everyone ends up with “I don’t trust anything I read online” and you give up, and then you leave this giant vacuum for anyone to fill in with whatever truth they want to create and that’s when to me is when you give up on a shared reality. That opens a door to all of the worst authoritarianism tendencies and whatnot – that creates the cynical blank slate where people get away with whatever they want because everyone’s lying, I give up, I don’t try, I don’t care, and eventually the rule of law goes too. To me, that’s the tragedy of the whole thing. Often, the goal, the ultimate goal of people, is not that these people really care personally about whether you get the vaccine or not, it’s that this is another way for them to break down your trust in the things that are getting in their way of making their money, which sums up the last four years in many ways. If somebody owes seven billion dollars in back taxes to the government, then the best way to never have to pay that is to just destroy the government, and the way you destroy the government is that you get nobody to believe in it anymore. That to me, that breakdown of trust is not about the subject, it’s about breaking down the overall trust fabric…
Deb: …about breaking down the fabric of society so that you don’t have to play by its rules anymore, I get that.
Deb: But I think the task that lays out for us quite clearly, us in the disinformation nerdy world, is how do we help people regain or get away from that feeling of helplessness and overcome it and feel that they know how to direct their skepticism and how to just sort of bring them back into the idea that yes some people are lying to you, but you should generally be able to figure out who is who. There may be places where it’s harder to do that, but as someone else I was talking to recently said, if it’s if it’s a really technical issue and someone over here is saying X and all these other people over here are saying Y, the experts are probably better able to evaluate the nonsense more than you are because it’s very easy to tell lies with technical information.
Deb: I got a little complicated there, but you know I think our job is to make it easier for people to tell who’s telling the truth and who’s telling lies. I don’t think it’s going to be easy, but I think it’s a good thing to do.
Danny: Yeah, I completely agree, and one of the things that I come back to is another thing it means is that the organizations and the people who are trustworthy, who do have some degree of altruistic motivation need to double down on that trustworthiness.
Danny: And the way you fight nihilism is not with more nihilism.
Deb: Right, exactly, that is true. So I’m reading… Do you know Ethan Zuckerman? He’s good at being optimistic about this sort of stuff.
Danny: Good, we need that.
Deb: I do admire that, by the way. That skill to look this in the face and say “Why losing faith in institutions provides the tools to transform them” – it’s a book called Mistrust. I haven’t read it yet, I just got it yesterday. So I’m looking for people who are finding ways to be optimistic about this.
Danny: All in all, there are things to be optimistic about. The rule of law held in the last election and some people maybe don’t believe that, but I think the overwhelming majority of people believe that. The system bubbles on and Congress’s certification of the election was delayed by a mere six hours and I think that is something that is worth being optimistic and celebrating about.
Deb: Agreed. On that note, let’s wrap up for today. I hope our audience found the walkthrough of those websites interesting. I did. The more I get into that, the more I understand how important but also how subtle some of the challenges are there for you. I look forward to following the progress there.
Danny: Thanks, it’s a long journey.
Deb: Thanks, Danny.
Danny: Thanks for having me on.
Deb: Take care.
See an error in our work? Please let us know and include the source of your info. We’ll update or correct as appropriate.