How Not to Run an Experiment

Nicholas Senz

A study published in the journal Cognitive Science has led its authors to the conclusion that children “exposed” to religious ideas have greater difficulty discerning fact from fiction than their counterparts who have not been “exposed” to religious ideas.

I have placed the word “exposed” in quotation marks to draw attention to it. It is one of many flaws in this study, and a prime example of my chief objection to practices of the social sciences.

In the ideal scientific experiment, the scientist sets up two scenarios identical in every way except one, the one element under investigation. If there is a significant difference between the two scenarios, we know that the different element is the causal factor, because every other factor has been accounted for an controlled.

Now, there are approximately fifty bajillion elements that contribute to a human person’s mental makeup: every experience and genetic predisposition is a factor in the way people think and perceive things.

Does any social scientist have the ability to control all of those factors except one? Can any social scientist come anywhere within 10 billion parsecs of creating a scenario where a true experiment can take place? Of course not! The human mind is far too complex for that, and human experience is far too variable for that.

Our researchers in this study have tried to distinguish two groups: those who have been “exposed” to religious ideas and those who have not. Their determining factors are church attendance and type of school attended–if a child does not attend church regularly and does not attend a parochial school, the researchers deem that the child has not been “exposed” to religious ideas.


Maybe the family doesn’t attend church regularly but the parents still maintain a belief in God which they have passed on to their children–the “spiritual but not religious” bunch we hear so much about. Maybe the children have irreligious parents and religious friends. Maybe they’ve seen TV shows and movies that depict God as existing and acting in the real world and they’ve found them credible and intriguing. There are all sorts of ways a child can be significantly “exposed” to religious ideas beyond what the researchers have accounted for.

The researchers’ method also leaves much to be desired. They told the children stories with different levels of “realism” and different levels of “magical” or “non-magical” elements, sometimes making them similar to stories from the Bible (and thus “familiar”), and asked the children whether the characters were real or pretend. For instance, they gave four scenarios like this:

Familiar/Magical: John waved his magic stick and parted the red sea.

Familiar/non-Magical: John waved his stick and parted the red sea.

Unfamiliar/Magical: John waved his magic stick and parted the mountain.

Unfamiliar/non-Magical: John waved his stick and parted the mountain.

Any of these should be identified as not likely to be real, said the researchers, because these sorts of things don’t happen in the real world. They also found that the “secular” children were more likely to identify these as not real, except that the secular children were mostly likely to believe the last statement (31% for the last compared to 9% for the second and third; 59% of the religious children identified that as real). How often does it happen in the real world that a man waves a stick and parts a mountain? That’s a bit odd, isn’t it?

I would be very curious to see what would have been the result if the researchers had told these five- and six-year old children fantastic-sounding things that have and do actually happen in the real world, such as “Neil walked on the moon” or “Linda lives on a ship in the sky” (International Space Station). What nonsense! Who could believe such a thing?

Ah, but if you’ve been told that it is true that a man has walked on the moon or a woman lives on a ship in the sky, then it’s no longer so fantastic, is it? Just as if you’ve been told that it’s true that God parted a sea or that God made the sun dance in front of thousands of people and dried their wet clothes, you’ll believe that such things can happen.

Here’s the point: all knowledge begins with experience, either direct observation or the experience of knowledge gained from those whom we trust, such as parents and teachers. It may seem absurd to a child that the Earth goes around the Sun when they look at the sky and see just the opposite; or that the Sun is 93 million miles away when it seems like they could throw a baseball and hit it; or that caterpillars and butterflies are the same animal; or any of a number of things. Children don’t have the experience or knowledge to always be able to distinguish what’s possible from what’s impossible in the world. They have to gain that experience and be taught those things.

If a child is taught that everyone not in their own family is mean and untrustworthy, they’ll see the world as such and judge people accordingly. If a child is taught that all of space is filled with an invisible ether through which all things move and which imposes friction on things so as to slow them down (as they would have been taught in the early modern period), they’ll operate based on that assumption. If a child is taught that there is a God who is all-powerful and who has and does intervene in the affairs of the world He created, they will abstract from that principle and judge accordingly.

The authors of this study have made big fat assumptions about what is possible and what is not possible. To them I would say with Hamlet, “There are more things in heaven and earth, Horatio, than are dreamt of in your philosophy.”

It should be no surprise that children who have been exposed to religious ideas are more likely to believe that fantastic-sounding things could be real. They’re simply more open-minded.

Share on facebook
Share on google
Share on twitter
Share on linkedin
Share on pinterest

9 thoughts on “How Not to Run an Experiment”

  1. Pingback: Judgments About Fact And Fiction By Confused Researchers | William M. Briggs

  2. Just as if you’ve been told that it’s true that God parted a sea or that God made the sun dance in front of thousands of people and dried their wet clothes, you’ll believe that such things can happen.

    That is not a good comparison. The earth does go around the sun. People do live on the ISS. Men did walk on the moon.

    The parting of the Red Sea and the Fatima Sun Miracle are not true. One is a story told to the Jews and the other is a case of mass hysteria. Children need to be taught real truths not imaginary ones.

  3. Pingback: The Church Endures by Diane Moczar -

  4. Nicholas, great article. Thank you for making sense out of this. I so appreciate Catholic writers, and their views!

  5. Could you please provide a bit if detail about your reference from Cognitive Science? Publication date, volume, actual name of article and authors as I would like to review the actual study. It’s hard to believe that the Cognitive Science Society’s editorial and peer review boards would publish a obviously seriously “flawed” study. I would like to read it…thanks!

    1. Phil, the article is Corriveau, K.H.; Chen, E.E.; & Harris, P.L. (2014, July 3). “Judgments About Fact and Fiction by Children From Religious and Nonreligious Backgrounds”. Cognitive Science, Vol. 38 Issue 5 (DOI: 10.1111/cogs.12138). If you have a subscription to Wiley Online Library, you can find it there. While you’re at it, you might check up on the “Bogdanov affair” on Wikipedia. Problems in peer review may be more prevalent among the social sciences than among the hard sciences; nevertheless, they do occur even there. That’s because the only flaw in the peer review process is that scientists are humans, prone to error and bias like anyone else. It works very well, spectacularly well … but not all the time, every time.

Leave a Comment

Your email address will not be published. Required fields are marked *

This site uses Akismet to reduce spam. Learn how your comment data is processed.

%d bloggers like this: