Wednesday, February 21, 2018

More falsehoods?

Continuing on the previous two posts...Bem's data, which Dr. R made available here, shows that the data for experiment 7 was collected in 2005. I, and others, had made the assumption that the data for experiment 7 was collected along with the data for experiments 5 and 6, because the description of the experiment is the same as the description of the 300 series experiments in the 2003 report (which includes the data later used to form experiments 5 and 6). It turns out that this was not the case.

The 2003 report describes a set of experiments using supraliminal exposures and low affect trials. Two groups are compared on the basis of "boredom proneness" (in this case "openness to experience"). A t-test of the difference gives the degrees of freedom at 92, which indicates there were 94 subjects in the experiment. This is different from the description of the other experiments in the 300 supraliminal series which were testing variations in the precognitive habituation hypotheses, both in the description of the targets used and in the numbers of subjects.

Then, as we now know from the data newly made available to us, Bem later performs another set of experiments using supraliminal exposures and low affect trials on 200 subjects, in 2005. Again, two groups are compared on the basis of "boredom proneness" (in this case, "stimulus seeking"). This experiment is written up in "Feeling the Future" as experiment 7, but no mention is made of his previous experiment which also tested the idea.

Bem has denied that there is a file-drawer of relevant experiments/trials which he failed to mention in "Feeling the Future". Yet here we have a relevant experiment, testing the same hypothesis in the same way, as a later experiment which does make its way into "Feeling the Future". And he makes no mention of the prior experiment, let alone including its results. It has been suggested (here, for example) that Bem's passing reference to a couple of file-drawer studies on precognitive habituation can be taken as including the file-drawer on the induction of boredom study. However, why would this be the case? The supraliminal PH studies and the boredom study were performed using different targets and different subjects, and were testing different hypotheses. And while Bem found a way to excuse the supraliminal PH studies from consideration, by regarding them as conceptual replications which didn't work, that same excuse doesn't hold for the boredom studies, since there is no difference in concept between the earlier and later studies.

So to sum it up, we find unequivocal evidence, from the 2003 report, that Bem did at least two things which he denies. He set up an experiment which tested multiple hypotheses, then presented the experiment as though it was set up to test a single hypothesis; one which was chosen post hoc based on where he was able to tease out statistically significant results (http://naturalismisuseful.blogspot.com/2018/01/). And he ran multiple tests of the same hypothesis, yet failed to mention or include one of those tests, leaving them in a file-drawer. He has had ample opportunity to own up to doing this. At what point does this become deliberate deception and a violation of the standards of practice in place in 2011?