Home | Third Culture | Digerati | Reality Club

THE FILE DRAWER PROBLEM

A confirmation rate of 29 percent is generally considered impressive by meta-analytic standards. This high a proportion of confirming results would arise by chance considerably less than once in a million times. This strong degree of statistical support creates a problem for Harris's thesis. She has attempted to reconcile this difficulty by claiming that these meta-analytic results cannot be analyzed statistically. (3) Because some studies report more than one finding, Harris notes, multiple findings from the same population are not statistically independent. This assertion is true, but Harris fails to point out that it is true only in a limited sense. About a third of the studies in both of our meta-analytic surveys report only one finding, and the likelihood of these findings arising by chance is also less than one in a million. In other words, the trend toward significant birth-order findings is sufficiently pronounced that it remains even if two-thirds of the data are thrown away.

Harris has raised a second objection to the use of statistical testing. Birth-order researchers, she claims, have employed a "divide-and-conquer" strategy by testing for birth-order effects in subsamples. Because it is not generally known how many of these subsamples may have been tested, Harris maintains that one cannot apply the normal rules of probability to the resulting findings. This claim is a variant of the so-called "file drawer" problem, which involves the tendency for investigators to publish their significant findings but for these same investigators to leave their nonsignificant findings lying around in file drawers. In my 1995 article, as well as in my 1996 book, I discussed this important issue, a point that Harris fails to bring to the attention of her readers (who are thereby encouraged to think that I did not consider it). In this particular context, moreover, Harris's claims about the file drawer problem are misleading. In both Harris's and my own meta-analytic totals, confirming findings involving interaction effects do not occur more often than expected by chance, compared with other kinds of significant findings, whereas the opposite outcome ought to be the case if Harris's speculative assertion has any merit. (4)

By this point, we can perceive a characteristic style to Harris's mode of argument. She begins by casting doubt on the validity of birth-order results, but she invariably fails to test her claims by using readily available data that might either confirm or refute her position. Here is one more example of a formal test that refutes Harris's own claims and findings. Let us grant, as most researchers do, that file drawers contain a higher proportion of null findings than does the published literature. This tendency would presumably not affect the publication of significant findings that refute other researcher's claims about birth order. In my own meta-analysis, the number of significant confirming findings exceeds the number of significant refuting findings by a 5-to-1 ratio, which would occur by chance less than once in a million times (and less than once in ten thousand times if we throw away two-thirds of the data by ignoring all studies reporting more than one finding). For Harris's own totals, the ratio of confirming to negating findings is 4 to 1, which would occur by chance less than once in a hundred thousand times (and less than once in a thousand times if we again choose to ignore studies reporting more than one finding). In short, it is not only possible, but appropriate, to apply statistical tests to these birth-order results. For obvious reasons, Harris would rather not do so.

SELF-REPORT DATA VERSUS REAL-LIFE DATA

In Born to Rebel I noted that real-life studies--those based on observable behavior--yield a significantly higher proportion of confirming studies than do self-report personality measures. (5) Radical revolutions in history provide a dramatic case in point, which leads Harris to engage in what appears to be another misrepresentation. For example, she claims that my assessment of attitudes toward revolutionary change in history involved only "a single question," remarking: "How well can we judge someone's personality by his answer to a single question?" In actual fact, my historical survey of 6,556 individuals involved responses to more than a hundred different historical events. None of these events involved a single question. Consider the case of Darwinism, which - at minimum--required Darwin's contemporaries to make judgments about organic evolution in general, the evolution of mankind, and Darwin's own controversial theory of natural selection. The number of questions that I and my 110 expert raters assessed in this historical study varied by event. For example, my analysis of attitudes toward the Reign of Terror among the 893 deputies of the French National Convention (1792-94) involved six different votes that took place within the Convention; the decision to sign, or not to sign, a document protesting illegal actions within the Convention; and 19 different measures of political activity, which I combined into an overall scale of tough-mindedness. (6) In assessing openness to experience among the 3,890 scientists who participated in 28 scientific revolutions, I employed five different measures: religious and political attitudes (as rated by 94 expert historians), world travel, breadth of intellectual interests, and openness to radical innovations (as judged by 110 expert historians). (7) In conclusion, it is an outrageous misrepresentation of the facts for Harris to say that the historical data presented in Born to Rebel reflect responses to only "a single question."


Previous | Page 1 2 3 4 5 | Next