|LETTER TO EDITOR
|Year : 2011 | Volume
| Issue : 2 | Page : 121-122
Is statistical significance a relevant tool for assessing clinical significance?
Syed Wasif Gillani
Discipline of Clinical Pharmacy, School of Pharmaceutical Sciences, Universiti Sains Malaysia, Penang, Malaysia
|Date of Web Publication||25-Nov-2011|
Syed Wasif Gillani
Discipline of Clinical Pharmacy, School of Pharmaceutical Sciences, Universiti Sains Malaysia 11800, Penang
Source of Support: None, Conflict of Interest: None
|How to cite this article:|
Gillani SW. Is statistical significance a relevant tool for assessing clinical significance?. J Pharm Negative Results 2011;2:121-2
In medicine and psychology, clinical significance refers to either of two related but slightly dissimilar concepts, whereby certain findings or differences, even if measurable or statistically confirmed, either may or may not have additional significance, either by  being of a magnitude that conveys practical relevance (a usage that conflates practical and clinical significance interchangeably) or  more technically and restrictively addresses whether an intervention or treatment may or may not fully correct the finding. Commentators who utilize the second, more restrictive usage designate the broader usage as linguistically imprecise and thus erroneous. Now a days, there is an element of statistics in the clinical assessment of therapeutic or outcome analysis.
During the period 1978-79, a total of 760 articles were published in NEJM. Statistical proportion was divided with 40% of descriptive statistics (%, mean and SD) and 56% element of t-test for continuous variables. The contingency tables (chi-square, Fisher's exact) increased to about 66%.  Similar study by another author describes the statistical patterns during the period 2003-2004. 311 articles were published in NEJM. There was a major shift of analysis with high increase in survival analysis (61%), multiple regression (51%), power analysis (39%) and multiple comparisons (23%). There was moderate increases in contingency tables (53%), epidemiologic stats (35%), and nonparametric tests (35%), ANOVA (16%). Steady level with descriptive or no stats, transformation, and simple linear regression. In contrast, there was a moderate decrease in t-test (26%), and significant decrease in Pearson correlation (3%). The author reported increase from 1.9 to 4.2 statistical methods per article. 
A minor change in response or treatment outcome will not produce statistical significance results but is considered as clinical significance in terms of pharmacovigilance issues. Similar conclusion was made by Van et al. in 2007, "Many conclusions of studies of exercise therapy for chronic low back pain have been based on statistical significance of results rather than on clinical importance and, consequently, may have been too positive. Authors of trials should report not only statistical significance of results but also clinical importance". 
The point of concern here is statistical analysis is needed to justify the objectives of the study and not to make the findings complicated with bulk amount of statistics. The current available research material seems to be statistical research with misleading significances. Statistical significance tends to be used in the context of null hypothesis significance testing (NHST). NHST answers the question: If a hypothesis that an effect is zero in the population is true (the null hypothesis), what is the probability of obtaining data that indicate the effect is not zero?  NHST is often misunderstood in several ways: that the P-value is the probability that the null hypothesis is false; that it is related to probability of replication; and that if the null hypothesis is rejected, the proposed alternative hypothesis should be accepted [Table 1].
Given the nature of NHST, and its common misuse, statistical significance does not yield information about magnitude of effect, practical significance, or clinical significance.  NHST only yields information about whether results are statistically likely, given some assumption about the population.  In terms of testing clinical treatments, statistical significance can only provide an answer to the question: If a treatment is actually ineffective, how likely it is that the statistical test of the treatment would erroneously indicate that the treatment is effective?
In terms of clinical treatment studies, clinical significance answers the question: Is a treatment effective enough to cause the patient to be normal? For example, a treatment might significantly change depressive symptoms (statistical significance), the change could be a large decrease in depressive symptoms (practical significance-effect size), and 40% of the patients no longer met the diagnostic criteria for depression (clinical significance). It is very possible to have a treatment that yields a significant difference and medium or large effect sizes, but does not move a patient from dysfunctional to functional. Clinical significance, first proposed by Jacobson, Follette, and Revenstorf  as a way to answer the question, is a therapy or treatment effective enough such that a client does not meet the criteria for a diagnosis.
Jacobson and Truax later defined clinical significance as "the extent to which therapy moves someone outside the range of the dysfunctional population or within the range of the functional population."  Later, they proposed two components of this index of change: the status of a patient or client after therapy has been completed and how much change has occurred during the course of therapy.  Differences that are common in the population are also unlikely to be clinically significant because they may simply reflect a level of normal human variation. Additionally, clinicians look for information in the assessment data and the client's history that corroborates the relevance of the statistical difference, to establish the connection between performance on the specific test and the individual's more general functioning. ,
At this point of discussion, majority of researchers think what we will do if statistical concerns are consider as biased in clinical studies. So, there are many ways to calculate statistical significance and practical significance, and there are various ways to calculate clinical significance. Five common methods are the Jacobson-Truax method, the Gulliksen-Lord-Novick method, the Edwards-Nunnally method, the Hageman-Arrindell method, and hierarchical linear modeling. 
| References|| |
|1.||Cohen J. The earth is round (P< .05). Am Psychol 1997;49:997-1003. |
|2.||Haase RF, Ellis MV, Ladany N. Multiple criteria for evaluating the magnitude of experimental effects. J Couns Psychol 1989;36:511-6. |
|3.||Emerson JD, Colditz GA. Use of statistical analysis in the New England Journal of Medicine. N Engl J Med 1983;309:709-13. |
|4.||Horton NJ, Switzer SS. Statistical methods in the journal. N Engl J Med 2005;353:1977-9. |
|5.||Van Tulder M, Malmivaara A, Hayden J, Koes B. Statistical significance versus clinical importance: trials on exercise therapy for chronic low back pain as example. Spine (Phila Pa 1976) 2007;32:1785- 90. |
|6.||Clinical Significance: "Clinical" Significance and "Practical" Significance are NOT the Same Things. Online Submission, Paper presented at the Annual Meeting of the Southwest Educational Research Association (New Orleans, LA, Feb 7, 2008). |
|7.||Jacobson NS, Follette WC, Revenstorf D. Psychotherapy outcome research: Methods for reporting variability and evaluating clinical significance. Behav Ther 1984;15:336-52. |
|8.||Jacobson NS, Truax, P. Clinical significance: a statistical approach to defining meaningful change in psychotherapy research. J Consult Clin Psychol 1991;59:12-9. |
|9.||Sattler JM. Assessment of children: Cognitive foundations (5/e). San Diego: Sattler Publications. ISBN 978-0-9702671-6-0; 2008. |
|10.||Kaufman AS. Lichtenberger E.Assessing Adolescent and Adult Intelligence 3 rd ed. Lay summary on 22 Aug 2010. Hoboken (NJ): Wiley. ISBN 978-0-471-73553-3; 2006. |
|11.||Peterson L. "Clinical" Significance: "Clinical" Significance and "Practical" Significance are NOT the Same Things. Online Submission, Paper presented at the Annual Meeting of the Southwest Educational Research Association. New Orleans, LA, Feb 7, 2008. |