Shifting goalposts?

I commented on another blog about a study which had some similarities to the Sheldrake staring experiments.

http://thinkingdeeper.wordpress.com/2014/03/09/sheldrake-vs-ubc-the-same-experiment/

http://hct.ece.ubc.ca/publications/pdf/gauchou-rensink-cac2012.pdf

http://www.sheldrake.org/files/pdfs/papers/sensoryclues.pdf

While I was reading the UBC paper, I was aware that I felt less critical about the paper than I would be if it had been a parapsychology paper.  Considering Dean Radin's criticisms from my previous blog post, and my criticisms of Radin's presentation of the blessed tea study, is it fair for me to be any less critical of the UBC paper (or alternatively, more critical of parapsychology papers)?  After all, like Sheldrake's and Radin's papers, there were multiple ways offered to analyze the results, the findings were post hoc, and novel outcome measures were offered.

Or were they?

An important design choice in the UBC paper is highlighted by contrasting it with Radin's paper.  Radin gave two groups of people tea which had been blessed or not, and measured change in mood and the subject's belief that they were in the intervention group.  The authors of the UBC study asked people general knowledge questions explicitly and implicitly (through the use of a Ouija board), and measured accuracy and the subject's belief that they were guessing at the answers.  In both cases, the significant finding was an interaction between the intervention and the belief condition.  Amongst those who believed they received the blessed tea, those who actually received the blessed tea had more improvement than those who did not.  Amongst those who believed they were guessing, those who were asked general knowledge questions implicitly (via the Ouija board) performed more accurately than when asked those questions explicitly.

Why are Radin's findings likely false, while the UBC study findings may be true?  The biggest difference is that Radin's findings are post hoc, while those in the UBC study were pre-planned.  Post-hoc testing violates the assumptions which underlie statistical significance testing, which reduces the validity of the results.

How can we tell whether a finding is pre-planned vs. post hoc?  It is not sufficient for the researcher to state a comparison was pre-planned.  And merely choosing to measure a number of different variables does not qualify as pre-planning.  So we can look at other factors, such as experimental manipulation, descriptions of the planning, and the analysis of the results.

The UBC group deliberately manipulated the belief condition by selecting questions which the subject identified as guesses.  They were identified as "guesses" independently of the accuracy of the answer and independently of their use in the "Ouija" board condition.  This experimental manipulation must be pre-planned.  There was no equivalent in Radin's study.  To be equivalent, Radin would also need to manipulate the belief condition (in this case, by manipulating what information was given to the subjects).  Unlike the UBC study, "belief" was a dependent variable in Radin's study, so it wouldn't be possible to form groups on the basis of "belief" prior to the drinking of the tea.

Another way to tell whether a comparison was pre-planned is to look at which comparisons were used in the sample size calculations (if reported).  In the UBC study, there are no sample size calculations reported.  In Radin's study, he reports that the sample size was assumed to be adequate based on his intentional chocolate study.  In that study, mood level (not change in mood) on each day was compared between conditions and "belief" was not a reported variable.  Had "belief" been a pre-planned condition in the tea study, it should have been accounted for, in some way, in the sample size assessments.

Finally, a quick way to check whether a comparison was pre-planned is to look at whether all the subjects are included in the analysis and whether the reasons for any exclusions are independent of the outcome.  In the UBC study, the analysis included 21/27 of the subjects who participated in the study.  Exclusions were based on a lack of success (i.e. movement of the planchette without conscious interference) in the use of the Ouija board and were unrelated to the outcome.  Radin included 40% of the subjects in his analysis, excluding more than half of the participants.  Thirty-two out of 221 were dropped for reasons unrelated to the outcome.  The remainder (101/221) were dropped for reasons which were strongly related to the outcome.  It would be very unlikely that a researcher would pre-plan a comparison which would so dramatically violate the significance testing.

To be fair, there is a good chance that the UBC study results are also false.  The sample size was small and it was somewhat exploratory, even if it was well-designed in comparison to Radin's study.  It will be interesting to see whether the findings hold up under attempted replications.

Linda

Gorilla video

A poster at Skeptiko recently brought up an article from Dean Radin talking about the disconnect between his perception of the strength of the evidence for psi, and how that evidence seems to be ignored by scientists in general.  I wrote a response, not realizing that the article was 5 years old, mostly because the mis-use of the gorilla video is a pet peeve of mine.  (By the way, this particular book is invaluable for releasing pet-peeve-tension.) :-)

https://realitysandwich.com/7283/what_gorilla/

Radin:
"Imagine you're watching a basketball game. Your favorite team is wearing white and the other team is in black. In the midst of the action, someone in a dark gorilla suit calmly walks to the center of the court, waves to the crowd, then walks off the court. Do you think you would notice this peculiar event? Most people might say yes. Most people would be wrong."
...
"Because of these blind spots, some common aspects of human experience literally cannot be seen by those who've spent decades embedded within the Western scientific worldview. That worldview, like any set of cultural beliefs inculcated from childhood, acts like the blinders they put on skittish horses to keep them calm. Between the blinders we see with exceptional clarity, but seeing beyond the blinders is not only exceedingly difficult, after a while it's easy to forget that your vision is restricted.

An important class of human experience that these blinders exclude is psychic phenomena, those commonly reported spooky experiences, such as telepathy and clairvoyance, that suggest we are deeply interconnected in ways that transcend the ordinary senses and our everyday notions of space and time."

My response:
"A bit of a nitpick (because the gorilla video experiment has been over-used and abused)...what you describe isn't related to the inattentional blindness demonstrated in the gorilla video.  Merely having a preference for one or the other team, while watching a game, does not lead to missing the the gorilla.  Most people notice the gorilla under those conditions.  You have to give people a different task which fully occupies their attention, if you want them to fail to notice the gorilla.  And even then,  half the people will still notice the gorilla.

What you are describing, with respect to the perception of psi among scientists in general, is the effect of how our prejudices tend to influence our judgement.  In this case, fans of the white team will see the gorilla.  But whether they see it as disruptive vs. helpful to their team may depend upon its colour.

I agree that discussions which take place in the media tend to misrepresent what is happening at the scientific level.  I propose that the way to draw other scientists into taking the research seriously is to follow the path of evidence-based medicine, with respect to practices which reduce the risk of bias and the production of false-positive results.  As it is, research performed under conditions where problems in design, implementation and/or analysis can grossly inflate the number of falsely positive studies, can be easily dismissed as reflecting the result of bias, rather than a true effect."

Linda

Outcomes, Part 2

Hand in hand with flexibility in outcomes is "selective outcome reporting".  Selective outcome reporting refers to failing to report outcomes which should be of interest.  It isn't just a matter of having a variety of ways in which the outcome could be measured, and only reporting the results of some.  Sometimes authors fail to report the only outcome of interest, or fail to report the details (such as the results of a significance test or the size of the effect).

We had one example in the previous blog post.  Robertson and Roy did not report on the result of a comparison between the recipient group and the control group in any of the experimental conditions.  In particular, they did not report on the double-blind condition, which was the purported reason for doing the experiment in the first place (according to Robertson and Roy 2001).  Instead we were given the results of a complicated set of analyses which broke up the experimental groups and recombined the subjects into new groups.  It is reasonable to offer exploratory analyses after the fact, but not at the expense of failing to report on the main outcome.  

Another recent example comes from Dean Radin.  In his study of blessed tea, he measures the mood in those consuming blessed tea and those who receive tea which is identical except that it has not been blessed, under blind conditions.  Yet he makes no mention of the main outcome proposed for the study - was there any difference in mood (in this case, improvement in mood from baseline) between those drinking blessed tea and those drinking tea which hadn't been blessed.  When asked for those results, it turns out that there wasn't a significant difference between the control and the intervention group.  Yet he presents the study as though it was positive.  And as far as I can tell, it is accepted as though it is a positive study by proponents.  This is accomplished by selectively reporting on a result which is not a valid and reliable type of outcome (post-hoc sub-group analysis) and substituting it for the actual outcome.

It is to be expected that the author of a study will be most interested in presenting the study in a positive light, and in a way which confirms what they hoped it would confirm.  Even in a field where there is a culture of publishing results, regardless of whether they are positive or negative (parapsychology), it's still preferable to be the researcher who publishes positive results.  But the more useful and interesting approach is to look at whether the results are likely to be true-positives, rather than false-positives.

Linda

Robertson, T. J. and Roy, A. E. (2004) Results of the application of the Robertson-Roy Protocol to a series of experiments with mediums and participants. JSPR 68.1

Roy, A. E. and Robertson, T. J. (2001) A double-blind procedure for assessing the relevance of a medium’s statements to a recipient. JSPR 65.3

http://deanradin.com/evidence/Shiah2013.pdf