Post hoc tests

Post-hoc in latin means "after this". Post-hoc tests are used to analyse the result of the experimental data. They are based on family-wise error rate. 

Family-wise error rate: It is the probability of making at least one Type I Error, when we are performing multiple simultaneous tests. It is also known as alpha inflation or Cumulative Type I error. 

The most common post-hoc tests are: 

• Bonferroni procedure 
• Duncan's new multiple range test 
• Dunn's multiple comparison test 
• Fisher's least significant difference 
• Holm-Bonferroni procedure 
• Newman-Keuls 
• Rodger's method 
• Scheffe's Method 
• Tukey's test 
• Dunnett's correction 
• Benjamin-Hochberg procedure

Bonferroni Procedure

This multiple-comparison post-hoc correction is used when you are performing many independent or dependent statistical tests at the same time. The problem with running many simultaneous tests is that the probability of a significant result increases with each test run. This post-hoc test sets the significance cut off at α/n. 

Imagine looking for the Ace of Clubs in a deck of cards: if you pull one card from the deck, the odds are pretty low (1/52) that you’ll get the Ace of Clubs. Try again (and try perhaps 50 times), you’ll probably end up getting the Ace. The same principal works with hypothesis testing: the more simultaneous tests you run, the more likely you’ll get a “significant” result. Let’s say you were running 50 tests simultaneously with an alpha level of 0.05. The probability of observing at least one significant event due to chance alone is:
P (significant event) = 1 – P(no significant event)
= 1 – (1-0.05)50 = 0.92.
That’s almost certain (92%) that you’ll get at least one significant result.

Holm-Bonferroni Method
The ordinary Bonferroni method is sometimes viewed as too conservative. Holm’s sequential Bonferroni post-hoc test is a less strict correction for multiple comparisons. 

Duncan’s new multiple range test (MRT)
When you run Analysis of Variance (ANOVA), the results will tell you if there is a difference in means. However, it won’t pinpoint the pairs of means that are different. Duncan’s Multiple Range Test will identify the pairs of means (from at least three) that differ. The MRT is similar to the LSD, but instead of a t-value, a Q Value is used.

Fisher’s Least Significant Difference (LSD)
A tool to identify which pairs of means are statistically different. Essentially the same as Duncan’s MRT, but with t-values instead of Q values. 

Newman-Keuls
Like Tukey’s, this post-hoc test identifies sample means that are different from each other. Newman-Keuls uses different critical values for comparing pairs of means. Therefore, it is more likely to find significant differences.

Rodger’s Method
Considered by some to be the most powerful post-hoc test for detecting differences among groups. This test protects against loss of statistical power as the degrees of freedom increase.

Scheffé’s Method
Used when you want to look at post-hoc comparisons in general (as opposed to just pairwise comparisons). Scheffe’s controls for the overall confidence level. It is customarily used with unequal sample sizes.

Tukey’s Test
The purpose of Tukey’s test is to figure out which groups in your sample differ. It uses the “Honest Significant Difference,” a number that represents the distance between groups, to compare every mean with every other mean.

Dunnett’s correction
Like Tukey’s this post-hoc test is used to compare means. Unlike Tukey’s, it compares every mean to a control mean. 

Benjamin-Hochberg (BH) procedure
If you perform a very large amount of tests, one or more of the tests will have a significant result purely by chance alone.

The Benjamini-Hochberg Procedure is a powerful tool that decreases the false discovery rate.

The false discovery rate (FDR) is the expected proportion of type I errors. 

Adjusting the rate helps to control for the fact that sometimes small p-values (less than 5%) happen by chance, which could lead you to incorrectly reject the true null hypotheses. In other words, the B-H Procedure helps you to avoid Type I errors (false positives).

A p-value of 5% means that there’s only a 5% chance that you would get your observed result if the null hypothesis were true. In other words, if you get a p-value of 5%, it’s highly unlikely that your null hypothesis is not true and should be thrown out. But it’s only a probability–many times, true null hypotheses are thrown out just because of the randomness of results.

example: Let’s say you have a group of 100 patients who you know are free of a certain disease. Your null hypothesis is that the patients are free of disease and your alternate is that they do have the disease. If you ran 100 statistical tests at the 5% alpha level, roughly 5% of results would report as false positives.

There’s not a lot you can do to avoid this: when you run statistical tests, a fraction will always be false positives.However, running the B-H procedure will decrease the number of false positives.

1. Put the individual p-values in ascending order.
2. Assign ranks to the p-values. For example, the smallest has a rank of 1, the second smallest has a rank of 2.
3. Calculate each individual p-value’s Benjamini-Hochberg critical value, using the formula (i/m)Q, where:
   • i = the individual p-value’s rank,
   • m = total number of tests,
   • Q = the false discovery rate (a percentage, chosen by you).
4. Compare your original p-values to the critical B-H from Step 3; find the largest p value that is smaller than the critical value.

PSM / COMMUNITY MEDICINE by Dr Abhishek Jaiswal is licensed under a Creative Commons Attribution-NonCommercial-ShareAlike 4.0 International License.
Based on a work at learnpsm@blogspot.com.
Permissions beyond the scope of this license may be available at jaiswal.fph@gmail.com.