The dangers of double dipping

Analyses of data from functional magnetic resonance imaging (fMRI) and studies tracking electrical activity in the brain can generate misleading results, suggests an article online this week in Nature Neuroscience.

Techniques such as fMRI, which provides images of the brain, and electrophysiology, which record electrical activity from many neurons at once, generate a large amount of data. Some of this data is intrinsically 'noisy', being accompanied by other background activity, like listening to a radio station with poor signal quality. In order to enhance the signal of interest, the raw data is substantially analyzed, often using complex transformations.

Nikolaus Kriegeskorte and Chris Baker analysed some artificially generated 'noise' data, unrelated to experimental variables, but still obtained results This shows that in many cases, analysis can generate spurious results entirely unconnected with real experimental results.

These spurious results arise from a practice the authors call 'double dipping'. For example, scientists may hypothesize that a brain region responds more strongly to one particular stimulus compared to another. An incorrect, 'double dipping' analysis would look for areas that were more active with the original stimulus and selectively analyze only these areas to test the hypothesis. However, some areas may activate more to one stimulus than another purely by chance, and restricting further analysis only to these areas would yield a misleading result.