Outcomes, Part 2

Hand in hand with flexibility in outcomes is "selective outcome reporting".  Selective outcome reporting refers to failing to report outcomes which should be of interest.  It isn't just a matter of having a variety of ways in which the outcome could be measured, and only reporting the results of some.  Sometimes authors fail to report the only outcome of interest, or fail to report the details (such as the results of a significance test or the size of the effect).

We had one example in the previous blog post.  Robertson and Roy did not report on the result of a comparison between the recipient group and the control group in any of the experimental conditions.  In particular, they did not report on the double-blind condition, which was the purported reason for doing the experiment in the first place (according to Robertson and Roy 2001).  Instead we were given the results of a complicated set of analyses which broke up the experimental groups and recombined the subjects into new groups.  It is reasonable to offer exploratory analyses after the fact, but not at the expense of failing to report on the main outcome.  

Another recent example comes from Dean Radin.  In his study of blessed tea, he measures the mood in those consuming blessed tea and those who receive tea which is identical except that it has not been blessed, under blind conditions.  Yet he makes no mention of the main outcome proposed for the study - was there any difference in mood (in this case, improvement in mood from baseline) between those drinking blessed tea and those drinking tea which hadn't been blessed.  When asked for those results, it turns out that there wasn't a significant difference between the control and the intervention group.  Yet he presents the study as though it was positive.  And as far as I can tell, it is accepted as though it is a positive study by proponents.  This is accomplished by selectively reporting on a result which is not a valid and reliable type of outcome (post-hoc sub-group analysis) and substituting it for the actual outcome.

It is to be expected that the author of a study will be most interested in presenting the study in a positive light, and in a way which confirms what they hoped it would confirm.  Even in a field where there is a culture of publishing results, regardless of whether they are positive or negative (parapsychology), it's still preferable to be the researcher who publishes positive results.  But the more useful and interesting approach is to look at whether the results are likely to be true-positives, rather than false-positives.

Linda

Robertson, T. J. and Roy, A. E. (2004) Results of the application of the Robertson-Roy Protocol to a series of experiments with mediums and participants. JSPR 68.1

Roy, A. E. and Robertson, T. J. (2001) A double-blind procedure for assessing the relevance of a medium’s statements to a recipient. JSPR 65.3

http://deanradin.com/evidence/Shiah2013.pdf