Journal of Pharmaceutical Negative Results

ORIGINAL ARTICLE
Year
: 2011  |  Volume : 2  |  Issue : 1  |  Page : 39--41

Quality of reporting of descriptive and inferential statistics in negative studies published in Indian medical journals


Jaykaran1, Deepak Saxena2, Preeti Yadav1, ND Kantharia1, Paresh Solanki1,  
1 Department of Pharmacology, Govt. Medical College, Surat, India
2 Department of Community Medicine, Govt. Medical College, Surat, India

Correspondence Address:
Jaykaran
Department of Pharmacology, Govt. Medical College, Surat
India

Abstract

Aims and Objectives : It is observed that negative studies published in medical journals are underpowered to detect the actual difference between the groups, but no data are available on the quality of the statistics reported in these studies. Therefore, this study was carried out with aim of evaluating negative studies published in Indian medical journals for adequacy of reporting of descriptive and inferential statistics. Materials and Methods : All the original articles published in 14 Pubmed-indexed Indian medical journals were analyzed to determine whether the study was negative or positive. All the negative studies were analyzed for correctness of the descriptive statistics and inferential statistics. The types of data and statistical methods were also noted down. Descriptive statistics was used and values were expressed as frequency, percentages and confidence interval. Results : Incorrect descriptive statistics was mentioned in 15 (28.8%, 95% CI 18.3-42.2%) studies. Information related to assumptions of statistical tests were mentioned in only two (3.8%, 95% CI 1.0-12.9%) articles. Inappropriate/incorrect statistical tests was used in 22 (42.3%, 95% CI 29.8-55.8%) studies. The most common reason for inappropriate reporting of descriptive statistics was use of mean and SD for description of ordinal data. The most common reason for incorrect statistical test was use of parametric test for ordinal data. The most common statistical test was the t-test. Conclusion : Negative studies published in prominent Indian medical journals are statistically weak, and readers critically analyze these studies before making any opinion based on them.



How to cite this article:
Jaykaran, Saxena D, Yadav P, Kantharia N D, Solanki P. Quality of reporting of descriptive and inferential statistics in negative studies published in Indian medical journals.J Pharm Negative Results 2011;2:39-41


How to cite this URL:
Jaykaran, Saxena D, Yadav P, Kantharia N D, Solanki P. Quality of reporting of descriptive and inferential statistics in negative studies published in Indian medical journals. J Pharm Negative Results [serial online] 2011 [cited 2020 Apr 1 ];2:39-41
Available from: http://www.pnrjournal.com/text.asp?2011/2/1/39/82981


Full Text

 Introduction



Studies that failed to disprove the null hypothesis are called as "negative studies." In an editorial written in the Journal of Pharmaceutical Negative Results (JPNR), it is mentioned that the "main aim of the journal is to publish negative results so that newer generation of researchers should not waste their time and money repeating the same studies and finding the same unpublishable results." [1] Thus, labeling a study as a negative study is a work of great responsibility as once the negative study is published, it will discourage other researchers to work for the same study question. Here, it is very important to understand that to conclude a hypothesis beyond doubt, the study should be methodologically strong. [2] Therefore, when a study concludes that there is "no significant difference," than this study should be strong enough to conclude the same. These studies should have enough power/sample size to find the worthwhile difference if it exists. [3] But, it is observed that most of the negative studies published in medical journals are underpowered to find the difference between the groups. [4],[5] Hence, conclusions made in these studies cannot be considered valid. It is also important to report the power/sample size, confidence interval, etc. in published articles so that a reader can evaluate the study for its validity by using these parameters, but it is also observed that most of the negative studies published in medical journals are poor in adequate reporting of these parameters. [6] Not only should the power/sample size calculation be done adequately but also the correct statistics should be used in the analysis of data in these studies. Various studies carried out for the statistics evaluation in the published literature concluded that incorrect statistics is not an uncommon phenomenon. [7] But, no such data are available specifically for negative studies published in Indian medical journals and, thus, this study was designed with the aim of evaluation of negative studies published in Indian medical journals for correctness of inferential and descriptive statistics reported in them.

 Materials and Methods



All the Pubmed-indexed journals subscribed by the central library of our institute were taken into consideration. These journals were Annals of Indian Academy of Neurology (AIAN), Indian Journal of Orthopedics (IJ Orhto), Indian Journal of Critical Care Medicine (IJCCM), Indian Journal of Dermatology, Venereology and Leprology (IJDVL), Indian Journal of Nephrology (IJN), Indian Journal of Dermatology (IJD), Indian Journal of Ophthalmology (IJO), Indian Journal of Urology (IJU), Indian Journal of Anesthesia (IJA), Indian Journal of Psychiatry (IJPsy), Indian Pediatrics (IP), Indian Journal of Medical Research (IJMR), Indian Journal of Medical Science (IJMS) and Indian Journal of Community Medicine (IJCM). All the original articles published in 2009 from these journals were evaluated by the first author (J.K.) to sort out the negative studies from them. Studies were considered negative studies on the basis of these criteria: one primary outcome is not statistically significance, second most important outcome is not statistically significant (if primary outcome is not reported). The most important outcome was considered to be that outcome for which either sample size calculation was done or which came first in the abstract. [7] The second author reassessed the 50 studies and, of them, the kappa statistics was calculated (k = 0.87). All these negative studies were evaluated for the incorrect reporting of descriptive statistics and inferential statistics. Types of data and statistical methods used in the studies were also noted in the proforma.

Statistics

Descriptive statistics was used and values were reported as frequency (percentages) and 95% confidence interval (CI).

 Results



Of the 276 articles published in 2009 in these 14 journals, 52 (18.8%, 95% CI 14.6-23.8%) were negative studies. More than one type of data was used in most of the studies. It was observed that ratio data was used in 49 (94.2%) studies, nominal data were used in 35 (67.3%) studies and ordinal data was used in 19 (36.5%) studies.

It was observed that incorrect reporting of descriptive statistics was observed in 15 (28.8%, 95% CI 18.3-42.2%) studies. The most common reason (14 studies, 93.3%) for incorrect reporting of descriptive statistics was use of mean and SD for description of ordinal data, as for ordinal data, the median and range should be used.

The most common statistical method was the t-test (50%) [Table 1]. It was observed that many of the statistical methods were not written in proper name, like out of 26 t-tests, whether the test is paired or unpaired was mentioned in only 10 articles and in the remaining 16 articles, only the "t-test" or "Students t test" was mentioned.{Table 1}

Information related to fulfillment of assumptions of selected statistical test was mentioned in only two (3.8%, 95% CI 1.0-12.9%) articles.

Inappropriate/incorrect statistical tests was used in 22 (42.3%, 95% CI 29.8-55.8%) studies. The most common cause of incorrect statistical tests was use of parametric test for nonparametric data (27.2%) [Table 2].{Table 2}

 Discussion



This study shows that many of the negative studies published in Indian medical journals are analyzed by inappropriate statistical tests, and reporting of descriptive statistics is also incorrect in many studies. In the light of these findings, the results shown by these studies are questionable and invalid. Thus, readers should critically analyze these kinds of negative studies before making any opinion based on them. It is been observed in studies carried out for the western medical journals that the quality of statistics is poor in the published literature. These surveys show that the statistical errors are very common, with the error rate ranging from 60 to 90%. [8],[9],[10],[11] These statistical errors raise ethical issues like what is the justification of exposing patients to the new intervention in the study, which any way is too faulty to show any difference? A researcher working in the same field may be misguided by the "no significance difference" concluded in the study, while the fact is that the study is too weak to conclude the same. Therefore, not only the authors of these studies but also the editors of the journals publishing these kinds of studies should understand the great responsibility they have.

There may be various reasons for having these statistical errors in published studies, like investigators not having enough knowledge of research methodology and biostatistics, insufficient review of study protocol by institutional ethics committee, inadequate knowledge of statistics in peer reviewers and editors, etc. [12] All these issues should be taken care of and researchers, peer reviewers and editors should be trained in various statistical methods. The role of the Institutional ethics committee is very important, and a statistician should be included in the ethics committee.

This study has some limitations, like negative studies published in 2009 only were analyzed and only those journals were considered that were Pubmed indexed and subscribed by our library. A similar study having more journals and more negative studies can explore this issue further. Another limitation is that we only evaluate the descriptive and inferential statistics. Many statistical parameters like adjustment of multiple endpoints, confidence interval, exact P-value, post hoc power, etc. were not analyzed in this study. This study could have been more meaningful with the analysis of these parameters. As far as our analysis goes, many studies are performed to measure the adequacy of power/sample size calculation and reporting of these parameters in negative studies, but we could not find a single study evaluating misuse of statistics in negative studies published in Indian medical journals.

References

1Kundoor V, Mueen Ahmed KK. Uncovering negative results: Introducing an open access journal "Journal of Pharmaceutical Negative Results". J Pharm Negative Results 2010;1:1-3.
2Jaykaran, Yadav P, Chavda N, Kantharia ND. Some issue related to the reporting of statistics in clinical trials published in Indian medical journals: A Survey. Int J Pharmacol 2010;6:354-9.
3Jaykaran, Yadav P, Bhardwaj P, Goyal J. Problems in reporting of statistics: Comparison between journal related to basic science with journal related to clinical practice. Int J Epidemiol 2009;7:1.
4Williams HC, Seed P. Inadequate size of 'negative' clinical trials in dermatology. Br J Dermatol 1993;128:317-26.
5Keen HI, Pile K, Hill CL. The prevalence of under-powered randomized clinical trials in rheumatology. J Rheumatol 2005;32:2083-8.
6Hebert RS, Wright SM, Dittus RS, Elasy TA. Prominent medical journals often provide insufficient information to assess the validity of studies with negative results. J Negat Results Biomed 2002;1:1.
7Karan J, Kantharia ND, Yadav P, Bhardwaj P. Reporting statistics in clinical trials published in Indian journals: A survey. Pak J Med Sci 2010;26:212-6.
8Altman DG. Statistics in medical journals: developments in the 1980s. Stat Med 1991;10:1897-913.
9Gore SM, Jones IG, Rytter EC. Misuse of statistical methods: Critical assessment of articles in the BMJ from January to March 1976. BMJ 1977;1:85-7.
10Pocock SJ, Hughes MD, Lee RJ. Statistical problems in the reporting of clinical trials: A survey of three medical journals. N Engl J Med 1987;317:426-32.
11MacArthur RD, Jackson GG. An evaluation of the use of statistical methodology in the Journal of Infectious Diseases. J Infect Dis 1984;149:349-54.
12Altman DG, Goodman SN, Schroter S. How statistical expertise is used in medical research. JAMA 2002;287:2817-20.