Familywise error rate

From Wikipedia, the free encyclopedia

In statistics, familywise error rate (FWER) is the probability of making one or more false discoveries, or type I errors among all the hypotheses when performing multiple pairwise tests^[1]^[2].

1 Classification of m hypothesis tests
2 What constitutes a family
3 See also
4 References

[edit] Classification of m hypothesis tests

	# declared non-significant	# declared significant	Total
# true null hypotheses	$U$	$V$	$m 0$
# non-true null hypotheses	$T$	$S$	$m - m 0$
Total	$m - R$	$R$	$m$

The m specific hypotheses of interest are assumed to be known in advance, but the numbers of true null hypotheses m₀, and alternative hypotheses, m₁, are unknown. R is an observable random variable, while S, T , U, and V are all unobservable random variables. V is the number of Type I errors, (hypotheses declared significant when they are actually from the null distribution), and T is the number of Type II errors, (hypotheses declared not significant when they are actually from the alternative distribution).

In terms of random variables,

$\mathrm{FWER} = \Pr(V \ge 1), \,$

or equivalently,

$\mathrm{FWER} = 1 -\Pr(V = 0).$

[edit] What constitutes a family

In confirmatory studies (i.e., where one specifies a finite number of a priori inferences), families of hypotheses are defined by which conclusions need to be jointly accurate or by which hypotheses are similar in content/purpose. As noted by Hochberg and Tamrane (1987), "If these inferences are unrelated in terms of their content or intended use (although they may be statistically dependent), then they should be treated separately and not jointly" (p. 6). For example, one might conduct a randomized clinical trial for a new antidepressant drug using three groups: existing drug, new drug, and placebo. In such a design, one might be interested in whether depressive symptoms (measured, for example, by a Beck Depression Inventory score) decreased to a greater extent for those using the new drug compared to the old drug. Further, one might be interested in whether any side effects (e.g., hypersomnia, decreased sex drive, and dry mouth) were observed. In such a case, two families would likely be identified: 1) effect of drug on depressive symptoms, 2) occurrence of any side effects.

Thus, one would assign an acceptable Type I error rate, alpha, (usually .05) to each family and control for familywise error using appropriate multiple comparison procedures. In the case of the first family, effect of antidepressant on depressive symptoms, pairwise comparisons among groups (here, there would be three possible comparisons) would be jointly controlled using techniques such as Tukey's Honestly Significant Difference (HSD) comparison procedure or a Bonferroni correction. In terms of the side effect profile, one would likely be interested in controlling for Type I error in terms of all side effects considered jointly so that decisions about the side effect profile would not be erroneously inflated by allowing each side effect and each pairwise comparison among groups to receive its own uncorrected alpha. By the Bonferroni inequality, allowing each side effect and comparison its own alpha would result in a Type I error of .05 * 3 side effects * 3 pairwise comparisons per side effect = 0.45 (i.e., 45% chance of making a Type I error). Thus, a more appropriate control for side effect familywise error might divide alpha by three (.05/3 = .0167) and allocate .0167 to each side effect multiple comparison procedure. In the case of Tukey's HSD (a strong control multiple comparison procedure), one would determine the critical value of Q, the studentized range statistic, based on the alpha of .0167.

[edit] See also

False discovery rate

[edit] References

Hochberg, Y., & Tamhane, A. C. (1987). Multiple comparison procedures. New York: Wiley.

^ Shaffer J. P. Multiple Hypothesis Testing, Annual Review of Psychology, January 1995, Vol. 46, Pages 561-584 http://dx.doi.org/10.1146/annurev.ps.46.020195.003021
^ Benjamini, Y., and Hochberg Y. (1995). "Controlling the false discovery rate: a practical and powerful approach to multiple testing". Journal of the Royal Statistical Society. Series B (Methodological) 57 (1), 289–300. [1]