Abstract
All articles published in the British Journal of Oral and Maxillofacial Surgery (BJOMS) and the International Journal of Oral and Maxillofacial Surgery (IJOMS) between January 1999 and December 2009 were classified by study design and evaluated to find the evidence base in oral and maxillofacial surgery (OMFS). Those in related specialties, and the impact factor of related dental journals were also compared. From a total of 3294 articles (1715 (52%) BJOMS; and 1579 (48%) IJOMS) most of the studies were observational or descriptive (36% BJOMS; and 31% IJOMS). Review articles constituted 5% in the British Journal and 6% in the International Journal. Analytical (non-controlled) studies made up 6% and 7% of the studies in the British Journal and the International Journal, respectively. There were 28 randomised controlled trials (RCT) (2%) in the British Journal and 40 (3%) in the International Journal. One meta-analysis was recorded in the International Journal, and one closed loop audit was recorded in the British Journal. Forty percent of the papers in both journals were non-clinical, scientific, or animal studies. The number of RCTs published in OMFS is low and is comparable with the related specialties of ear, nose, and throat (ENT) (1%) and plastic surgery (4%). Greater effort is required to carry out quality research if we are to provide the best possible evidence to patients for our interventions.
Keywords
Introduction
A widely quoted definition of evidence-based medicine is “the conscientious, explicit, and judicious use of current best evidence in making decisions about the care of individual patients”.
1
An editorial published in the Lancet in 1996 argued that there was a lack of scientific rigor in surgical research.2
, 3
Doctors should use both clinical expertise and evidence from external research; neither of these alone is enough, and without current best evidence, practice risks becoming unjustified and entirely subjective, to the detriment of patients.4
, 5
, 6
, 7
There is a well established hierarchy of levels of evidence, and the medical community considers that meta-analyses and randomised controlled trials (RCTs) are the most scientifically stringent means of investigating the efficacy of one intervention against another. Other grades of evidence (in increasing weakness of level of evidence) are case controlled studies, comparative studies, case series, correlation studies and expert committee reports, and the clinical experiences of respected authorities. Levels of evidence or hierarchy of study designs from the US Agency for Health Care Policy are listed in Table 1.
8
US Agency for Healthcare Research and Quality. Available from: http://www.ahrq.gov./clinic/epcix.htm.
Table 1Levels of evidence.
Source: US Agency for Health Care Policy and Research.
| Level | Type of evidence |
|---|---|
| 1a | Evidence obtained from meta-analysis of randomised controlled trials |
| 1b | Evidence obtained from at least one randomised controlled trial |
| 2a | Evidence obtained from at least one well-designed controlled study without randomisation |
| 2b | Evidence obtained from at least one other type of well-designed quasi-experimental study |
| 3 | Evidence obtained from well-designed, non-experimental descriptive studies such as comparative, correlation, and case-controlled studies |
| 4 | Evidence based on expert committee reports or opinions, or clinical experience of respected authorities, or both |
This aim of this retrospective review was to find out the amount and the quality of the evidence base in oral and maxillofacial surgery (OMFS) by analysing the study design of articles published over the last 10 years in two widely read journals within the specialty, and particularly to discover the number of RCTs and meta-analyses published. Comparisons have been made with related specialties, and difficulties involved in carrying out RCTs in surgery, and the impact factor of journals in general have been discussed.
Method
All articles published in the British Journal of Oral and Maxillofacial Surgery (BJOMS) and the International Journal of Oral and Maxillofacial Surgery (IJOMS) between January 1999 and December 2009 were evaluated. These journals were chosen because of their wide readership and because they are likely sources for up-to-date information for clinicians.
The classification system for articles used by Maran et al.
9
was adopted in this study as there is no internationally agreed system (Table 2).Table 2Categorisation of articles used in this study.
| Observational study |
| Descriptive |
| Case series (including single case reports) |
| Review articles |
| Analytical |
| Cross-sectional design |
| Retrospective |
| Prospective |
| Case controlled study, non-randomised |
| Randomised controlled trial (single and meta-analyses) |
| Audit |
| Non-clinical |
| Animal studies |
| Scientific |
| Technical – new instruments or techniques |
| Educational – historical vignettes, quizzes |
| Editorial |
| Letters |
The abstracts of all articles were scrutinised, and any that could not be classified from the abstract alone were read in full. This was done entirely by the sole author. Book reviews, abstracts from other journals, and obituaries were not included.
As in Maran's methodology, observational studies were categorised as being either descriptive or analytical. Descriptive studies consisted of case reports or series with no a priori hypothesis stated, and review articles. Studies in which a clear hypothesis was tested were classified as analytical.
Three types of non-controlled analytical studies were included. Cross-sectional studies examine the relation between a condition and a variable of interest in a defined population at one period of time – for example, is blood glucose level abnormal in patients with dentofacial abscesses on admission to hospital? Questionnaire surveys were included in this category. Retrospective studies look back in time to examine the relation between previous conditions and current disease – for example, do patients admitted with a fractured mandible have a history of excessive alcohol intake? Prospective studies follow a cohort of patients to look forward in time to a defined outcome – for example, the incidence of temporary or permanent paraesthesia after removal of third molars.
Controlled studies are self-explanatory and those that are randomised are more powerful than those that are not. Studies were only classified as audits if the audit loop was completed. Letters, editorials, animal studies, technical notes, and scientific and educational articles were all classified as non-clinical.
Results
The results of the comparison between the two journals are shown in Table 3.
Table 3Comparison of the types of papers published in the British Journal of Oral and Maxillofacial Surgery (BJOMS) and the International Journal of Oral and Maxillofacial Surgery (IJOMS).
| Type of paper | No. (%) | |
|---|---|---|
| BJOMS (n = 1715) | IJOMS (n = 1579) | |
| Observational study | ||
| Case series | 618 (36) | 481 (31) |
| Review | 87 (5) | 91 (6) |
| Analytical study | ||
| Cross sectional design | 110 (6) | 110 (7) |
| Retrospective | 87 (5) | 139 (9) |
| Prospective | 71 (4) | 57 (4) |
| Case controlled trial (non-randomised) | 34 (2) | 31 (2) |
| Randomised controlled trial | 28 (2) | 40 (3) |
| Meta-analysis | 0 | 1 (<1) |
| Audit | 1 (<1) | 0 |
| Non-clinical | ||
| Educational | 21 (1) | 3 (<1) |
| Scientific | 57 (3) | 267 (17) |
| Animal | 78 (5) | 104 (7) |
| Technical | 264 (15) | 205 (13) |
| Editorial | 28 (2) | 5 (<1) |
| Letters | 231 (14) | 45 (3) |
Discussion
Of the 3294 articles included in this study only one meta-analysis (level 1a evidence) and 68 (2%) RCTs were recorded. Articles were not appraised critically as this would have been difficult. For reference, all RCTs published in the British Journal in the last two years are listed.
10
, 11
, 12
, 13
The numbers of RCTs and meta-analyses are comparable with those found in a similar study by Maran et al. in the ear, nose, and throat (ENT) specialty over a four-year period between 1990 and 1994 (one meta-analysis and 1% RCTs). Taghinia et al.14
reported that over a 20-year period between 1986 and 2006 only 163 RCTs had been published in three major plastic surgery journals, and of the 1000 most highly cited articles in each journal during the period only 40 (4%) were RCTs compared with 118 in the Annals of Surgery. In a survey of selected OMFS journals from 2004 to 2006, Kyzas15
concluded that OMFS publications suffer from a relative shortage of high quality evidence, and that more, larger, adequately powered, and better reported RCTs are warranted.Stirrat
16
said that there are four ethical imperatives that medical practitioners must consider. Firstly, all practitioners must make the interests of patients paramount. Secondly, any recommendation made to a patient must be supported by the best evidence available. Thirdly, all new interventions and procedures must be properly compared with currently accepted methods, and finally, those who do not fulfil the first three must be held to account. Daya17
wrote that “RCTs provide the most secure basis for valid inferences about the effects of treatments but pose several unique challenges”. Millat et al.18
surveyed 152 general surgeons in France and concluded that they acquired most of their information by reading and attending scientific meetings, and they attached more importance to the fame of the author than to the conduct of the study. The overall impact of the RCT was weak among the surgeons questioned. Meakins- Millat B.
- Fingerhut A.
- Flamant Y.
- Hay J.M.
- Fagniez P.L.
- Farah A.
- et al.
Survey of the impact of randomised clinical trials on surgical practice in France. French Associations for Research in Surgery (AURC and ACAPEM). Association Universitaire de Recherche en Chirurgie. Association des Chirurgiens de l’Assistance Publique pour l’Evaluation Médicale.
Eur J Surg. 1999; 165: 87-94
19
argued that the “dogmatism of the hierarchy of evidence suggests that there is no other way of defining a recommendation” and questioned whether these hard rules of evidence should be universally applicable to surgery.One potential problem with RCTs in surgery is the issue of equipoise. To recommend involvement in a trial there must be genuine uncertainty about the benefit or harm from an intervention, or about the relative merits of alternative treatments. Both patients and surgeon must share this. Also, with any new procedure, at what point in the learning curve should studies be carried out? The issue of sham surgery that is used as a control also has ethical implications.
In England and Wales responsibility for the evaluation of interventional procedures has been devolved by the Department of Health to the National Institute of Health and Clinical Excellence (NICE). Guidelines related to dentistry and OMFS are published on the NICE website.
20
Relatively few topics related to OMFS are being considered and this may need to be addressed.National Institute of Health and Clinical Excellence. Available from: http://guidance.nice.org.uk/Topic/MouthDental.
Stirrat concluded that “all surgical procedures and other interventional procedures must be subjected to rigorous, objective, and prospective evaluation. The contribution that EBM can make is acknowledged, but its simplistic and uncritical application to surgery is ultimately not beneficial to the individual”.
The Cochrane Collaboration, an international, independent, not-for-profit organisation of more than 28,000 contributors from more than 100 countries, is dedicated to collecting up-to-date, accurate information about the effects of health care that is readily available worldwide.
21
Cochrane reviews are the most comprehensive, reliable, and relevant sources of evidence, and are intended to help providers, practitioners, and patients make informed decisions about health care. As of July 2010, in a survey by the author of the 71 systematic reviews considered to be relevant to OMFS, only 29 (41%) concluded that there was enough evidence on which to make recommendations. In all cases the evidence was considered weak.The Cochrane Collaboration. Available from: http://www2.cochrane.org/reviews/en/subtopics/84.html.
The Institute for Scientific Information (ISI) produce the Impact Factor (IF) Index for journals published in the scientific community to provide clinicians with access to current research information of the highest quality.
22
It is a measure of the frequency an “average article” in a journal has been cited in a three-year period (the ratio between citations and the recent citable items published), and can be considered to be the average number of times published papers are cited up to two years after publication. For example, the 2010 IF for a journal would be calculated by dividing the number of times articles published in 2008–2009 were cited in indexed journals during 2010 by the number of articles, reviews, proceedings or notes published in 2008–2009 (note that the IF for 2009 will actually be published in 2010, because it could not be calculated until all the 2009 publications had been received; the IF for 2010 will be published in 2011).Institute for Scientific Information. ISI impact factor description. Available from: http://thomsonreuters.com/products_services/science_products/a-z/journal_citation_reports.
Lau and Samman
23
reviewed all the 932 articles related to the calculation of the IF in 2004 in four OMFS journals and concluded that none were classed as level 1 evidence, 2% were level 2, 8% were level 3, and 40% were level 4. A total of 465 (50%) articles were classified as non-evidence of which 47% were case reports.The IF is misunderstood by many who think that it is used to measure the impact of a particular journal, and some researchers think that the term should be abolished.
24
The best research evidence is from good quality research that has had careful measures taken to minimise bias. The quality of any published research should not be related to the citation rate of that particular journal, so the IF needs to be interpreted with care. The latest IFs for the International Journal and the British Journal (2009) are 1.444 (2008: 1.487), and 1.327 (2008: 0.787), respectively. For comparison, the five medical journals in 2009 with the highest IFs were the New England Journal of Medicine (IF 50.017), the Journal of the American Medical Association (IF 31.171), The Lancet (IF 28.409), Annals of Internal Medicine (IF 17.457), and the British Medical Journal (IF 12.827).Although meta-analyses and RCTs are considered the best forms of evidence with which to guide treatment, many busy clinicians find that reviews of current evidence, technical notes, and case series are helpful, and they can still change practice. However, in an editorial in BJOMS in 2005 McGurk urged OMFS practitioners to improve the standard of research in the specialty.
25
To my knowledge this study is the first attempt to analyse all articles in two major OMFS journals over a 10-year period. Although the number of RCTs is comparable with other related specialties, in common with other surgical disciplines more effort is required to carry out better quality, ethical research if we are to provide patients with the best possible evidence for our interventions, given the recognised difficulties in carrying out such research.
Conflict of interest
None stated.
References
- Evidence based medicine: what it is and what it isn’t.BMJ. 1996; 312: 71-72
- Surgical research or comic opera: questions, but few answers.Lancet. 1996; 347: 984-985
- Summing up evidence: one answer is not always enough.Lancet. 1998; 351: 123-127
- From best evidence to best practice: effective implementation of change in patients’ care.Lancet. 2003; 362: 1225-1230
- Inpatient general medicine is evidence based. A-Team, Nuffield Department of Clinical Medicine.Lancet. 1995; 346: 407-410
- The inhumanity of medicine.BMJ. 1994; 309: 1671-1672
- Practitioners of evidence based care. Not all clinicians need to appraise evidence from scratch but all need some skills.BMJ. 2000; 320: 954-955
US Agency for Healthcare Research and Quality. Available from: http://www.ahrq.gov./clinic/epcix.htm.
- Is there an evidence base for the practice of ENT surgery?.Clin Otolaryngol Allied Sci. 1997; 22: 152-157
- Comparison between the use of an ultrasonic tip and a microhead handpiece in periradicular surgery: a prospective randomised trial.Br J Oral Maxillofac Surg. 2009; 47: 386-388
- Random control trial of dermis-fat graft and interposition of temporalis fascia in the management of temporomandibular ankylosis in children.Br J Oral Maxillofac Surg. 2008; 46: 521-526
- Propofol alone and combined with dexamethasone for the prevention of postoperative nausea and vomiting in adult Japanese patients having third molars extracted.Br J Oral Maxillofac Surg. 2008; 46: 207-210
- Nurse-delivered brief interventions for hazardous drinkers with alcohol-related facial trauma: a prospective randomised controlled trial.Br J Oral Maxillofac Surg. 2008; 46: 96-101
- Randomised controlled trials in plastic surgery: a 20-year review of reporting standards, methodologic quality, and impact.Plast Reconstr Surg. 2008; 122: 1253-1263
- Evidence-based oral and maxillofacial surgery.J Oral Maxillofac Surg. 2008; 66: 973-986
- Ethics and evidence based surgery.J Med Ethics. 2004; 30: 160-165
- The randomized controlled trial and challenges for the new millennium.Evid Based Obstetrics Gynecology. 1999; 1: 46
- Survey of the impact of randomised clinical trials on surgical practice in France. French Associations for Research in Surgery (AURC and ACAPEM). Association Universitaire de Recherche en Chirurgie. Association des Chirurgiens de l’Assistance Publique pour l’Evaluation Médicale.Eur J Surg. 1999; 165: 87-94
- Innovation in surgery: the rules of evidence.Am J Surg. 2002; 183: 399-405
National Institute of Health and Clinical Excellence. Available from: http://guidance.nice.org.uk/Topic/MouthDental.
The Cochrane Collaboration. Available from: http://www2.cochrane.org/reviews/en/subtopics/84.html.
Institute for Scientific Information. ISI impact factor description. Available from: http://thomsonreuters.com/products_services/science_products/a-z/journal_citation_reports.
- Levels of evidence and journal impact factor in oral and maxillofacial surgery.Int J Oral Maxillofac Surg. 2007; 36: 1-5
- The journal “impact factor”: a misnamed, misleading, misused measure.Cancer Genet Cytogenet. 1998; 104: 77-81
- Editorial.Br J Oral Maxillofac Surg. 2005; 43: 371
Article info
Publication history
Published online: December 14, 2010
Accepted:
November 20,
2010
Identification
Copyright
© 2010 The British Association of Oral and Maxillofacial Surgeons. Published by Elsevier Inc. All rights reserved.
