Peer Review and the Quest for Truth: Crash Course Scientific Thinking #4

“Alien life may have been discovered – right here on Earth!”

Or at least, that’s what some headlines seemed to indicate in 2010 after a

NASA-backed study was released.

But that was over a decade ago, so where are all the aliens?!

Well, the truth is out there, but as always, it’s more complex.

Hi, I’m Hank Green, and this is Crash Course Scientific Thinking.

Now, I’m not saying I don’t believe in aliens.

The universe is vast and full of opportunities for life.

Also, I would never say that, because they might be listening.

But let's clear up a big misconception right at the top: this study,

which was published in the prestigious journal Science in 2010

and reported by lots of news outlets… never actually claimed to find aliens.

That was an unfortunate miscommunication on the part of the media.

What the study actually claimed to find was arsenic-based life.

Which admittedly sounds a lot more boring than aliens.

But if it were true, it would be very revolutionary.

Unfortunately… the claim did not hold up under closer examination of the evidence.

After the original announcement, new headlines ran stating that actually,

maybe we didn’t find anything all that strange.

As a Science Person on the internet, but also like,

a regular person trying to navigate my own media feed: Blech.

It’s frustrating when we are given contradictory science stories.

But there are a lot of reasons why stories like this one are more

nuanced than they come off in the news.

We’ll talk a lot more about how science gets communicated in the news in

a later episode.

But today I want to tell you about how this study was received by

the scientific community,

and why it serves

as a great example of one of science’s biggest super-powers:

self-correction.

So what happened?

The research was done in California’s salty, arsenic-filled Mono Lake, which,

for a poison-loving microorganism, is like the Amalfi Coast.

Researchers found this bacterium, GFAJ-1,

which seemed like it might replace phosphorus with arsenic in its molecular makeup.

Which would be super weird

and cool because the fundamental building blocks of life

(as we know it) are carbon,

hydrogen, nitrogen, oxygen, sulphur, and phosphorus but definitely not arsenic.

This would be like making a smoothie with yogurt, berries, greens, and glass.

So the team set up an experiment,

where they put that little GFAJ-1 bacteria in a salty-lake-water environment

that had very little phosphorus

and lots of arsenic.

That arsenic-laced water shouldn’t have supported life.

But the bacteria did not die.

It grew and grew, seemingly on arsenic alone.

After all their hypothesizing, observing, and experimentation,

the team felt like they had made a breakthrough.

So they wrote about their findings in a super exciting study called “A Bacterium

That Can Grow by Using Arsenic Instead of Phosphorus.”

A true banger.

But here’s the thing: the scientific process doesn’t stop with the writing.

In order to get published in a real, academic journal,

articles typically go through peer review,

a system where experts check one another’s work to make sure it meets the

standards for scientific quality.

Reviewers assess whether a study’s methods are sound

and if the findings actually support the conclusions.

The peer review process makes good work better by bringing up ideas

and concerns researchers might never think of on their own.

But what actually happens when a paper is undergoing peer review?

Like, let’s say that I did my own experiment

and wrote a paper about it.

In my secret laboratory in my basement,

I tested out a new kind of eco-friendly sunscreen

and found that it works better than all the other sunscreens on the market.

How do I get

that paper into a scientific journal

and share my legitimized findings with the wider world?

I think that it might be time for some Sage Advice!

I can help you there, Hank.

Also, I loved your title, “Awesome Sunscreen Club: It’s the Best, Seriously,

Subscribe Now.”

I mean, I tell it like it is, Sage.

So, when do I get published?

Well, there are a few things that need to happen before that.

First, you have to submit your paper.

That’s, like, step 1.

Then, the journal will send your article to a team of scientists who work

in your same field.

For your study, they’ll probably be people like photobiologists and dermatologists,

people who know sunscreen and skin disease.

So those are the “peers” in peer review.

It wouldn't make sense to send a paper about sunscreen to an astronomer.

Exactly.

Your peers' job is to answer a big but basic question:

“Is what I’m reading actually supported by the evidence provided?”

They'll also ask questions like, “Is this study relevant to the field?

Does it effectively build on the current state of research,

or does it overlook known sunscreen-related issues?”

And the questions don’t end there!

They’ll also want to find out if you used sound methods.

Did you measure the results in trustworthy ways?

Did you analyze the results correctly?

And do the results really imply what you say they do?

So like, if, hypothetically, my sunscreen wasn’t quality controlled,

or my sample size was too small, or my statistical analysis was whack

which of course it never would be!

But if it was, then that's the reviewers’ job to catch those mistakes.

Yes, but, again, only

as it relates to the very specific field of sunscreen research.

Like, if you mixed up two very common sunscreen ingredients in your tests,

they’d want you to know about it.

Because they’re experts in sunscreen.

And peer review is all about field-specific experts reviewing each other's work.

So, how long does peer review last?

The whole process can take some time.

So the world may have to wait for the latest by Hank Green et

al.

And even then, there will be more peer review

that happens after it’s published.

And that's been today's Sage Advice!

Thanks, Sage!

So it’s hard to get past that initial peer review.

Scientists have to make sure they’ve checked all the right boxes

and that their research is solid.

Which matters, because the big collaborative effort of science depends on trust

and really good scrutiny.

New researchers base their work on earlier discoveries,

and peer review helps make sure those discoveries are supported by evidence.

But sometimes, despite our best efforts,

papers do make it into publication with flawed data.

Which is what happened with the arsenic-based life study!

As Sage mentioned, peer review doesn’t end with journals.

And in fact, it’s only after a study is published

that the most intense peer review begins.

A published study is one of the main ways scientists present their work to

other scientists in their field.

Those other scientists then approach the same question from different angles in order to

test the findings themselves.

After a study is published,

scientists might try to replicate its results using the same data

as the original study,

to see if they arrive at the same conclusions.

Very basically, they re-run the numbers.

Others might try a different experiment using the same methodology,

but with all new data.

Basically asking, “If someone else does this study,

will they come to the same conclusion?”

And in the case of groundbreaking arsenic-based life…they didn’t.

Other scientists tested that same strain of GFAJ-1, and tried to grow it

just like the original experiment on arsenic instead of phosphorus.

And it didn’t work.

The new studies found

that while the bacteria could grow in a low-phosphorus environment,

it couldn’t with no phosphorus at all, which, while interesting, was not revolutionary.

Dr.

Rosie Redfield, a lead researcher on one of those new studies,

explained that the environment the original experiment used to grow the original bacteria probably

contained just enough phosphorus to support its growth.

So this arsenic-based alien… wasn’t an alien after all.

It actually required phosphorus, just like everybody else.

The original study wasn’t repeatable.

After the news broke, the journal, Science,

accepted corrections from other researchers and attached them to the original study,

explaining where the experiment had gone wrong.

Some scientists called for the journal to retract the study completely.

And in 2025, the paper was officially retracted.

When a publication retracts a study,

they usually slap a big warning label at the top of it to let

other folks know it’s been retracted,

but they leave it up as a form of transparency.

So it could still be read,

but with the knowledge that it had been retracted for error.

Ultimately, the saga of this study is a valuable tool

that shows us how powerful the scientific process actually is

when it comes to self-correction.

Failed repetitions, corrections, niche drama about really specific bacteria

all those things are good for science as a whole.

Even if it might not seem that way from the outside.

Because while the paper’s conclusion may have been ultimately flawed,

it sparked new discussions about how we would know

if we really did find an alien-ish lifeform.

All of this underscores a big point about science:

breakthrough knowledge almost never comes out of one study.

Which is a real problem with how science is often covered in the news.

The most newsworthy papers are going to be the ones

that are the most surprising,

but when a study introduces a shocking new truth,

it’s going to need way more than one paper before it becomes settled science.

For now, it’s important to understand

that all one study really shows us is what happened in

that one study.

New scientific knowledge comes when a community of scientists validate and repeat research.

Which takes time.

But again, it’s worth it because, otherwise,

scientists can waste time wandering off in the wrong direction.

So, where does that leave us on aliens and arsenic?

Well, at the end of the day, scientists didn’t stop looking.

Because that’s the work of science:

to understand the universe through the collection of evidence and the testing of ideas.

It builds on itself.

Both its successes and its failures.

And both are absolutely integral to learning about the universe.

So we don’t have proof of any arsenic-based life (alien or otherwise).

But there are still plenty of things we do have proof for,

like how cancer forms, or how evolution happens.

And scientists will continue to test the limits of what we know,

and how we can know it.

Something we’ll talk about more in our next episode,

when we discuss the nature of uncertainty and scientific consensus.

I’ll see you then.

This episode of Crash Course Scientific Thinking was produced in partnership with HHMI BioInteractive,

bringing real science stories to thousands of high school and undergrad life science classrooms.

If you’re a teacher,

visit their website for resources

that explore the topics we discussed in the video today.

Thanks for watching this episode of Crash Course Scientific Thinking,

which was filmed in Missoula, Montana,

and was made with the help of all these nice people.

If you want to help keep Crash Course free for everyone, forever,

you can join our community on Patreon.