The Average Level Of Evidence In Orthopedic Journals

Hi, my name's Doctor Don Buford. I'm an orthopedic surgeon and incoming president of the Interventional Ortho Biologics Foundation. I'm based in Dallas, Texas. I'm also the founder and director of the Texas Ortho Biologics Institute, and you can see our website here on the bottom of this slide.

Uh, the title of this presentation is The A Level of Evidence in orthopedic Journals.

These are my surgeon disclosures. Our AAOS Academy requires us to disclose, um, any potentially relevant uh conflicts of interest or, or sources of bias. And so, on the left is my most recent quarterly disclosure statement. Uh, the Other, uh, thing I'd like to disclose because it may be relevant to this talk for some of you, is that I am the editor of the Biologic Orthopedics Journal along with Alan Mishra and Bill Murrow. And, um, this is something that we've been, uh, putting together over the past 2 or 3 years, and

I'm happy to see it, uh, finally come to fruition over the past year. It's an online open access journal, and we accept all, uh, orthobiologic Research, whether basic science or clinical, and uh you can go to the website there if you're interested in potentially submitting or reading articles in the journal.

These are my Social Media disclosures. I am active on Instagram, Facebook, and Twitter. I am one of the founding members of the Ortho Biologic Ethics Consortium, which is a multidisciplinary, uh, group of clinicians and salespeople and distributors who are all trying to, uh, work in this, uh, Regenerative Medicine field in an ethical and evidence-based way, um, active on LinkedIn and YouTube. Our YouTube channel was recently named the number 6 orthopedic channel on the internet.

So have a look at that if you're interested. And um now let's get into the uh the substance of this lecture.

I would now like to discuss the Levels of Evidence for clinical studies. A level one study is a high quality randomized controlled trial.

It can also be considered a meta-analysis, where all of the studies included are level one randomized controlled trials.

Um, According to this 2004, um, review from the American Academy of Orthopedic Surgeons, a level two study would be a lesser quality randomized controlled trial, uh, perhaps a controlled trial where there was, uh, significant patient loss to follow up, for example, or some Other defect in the design or execution of the trial. A prospective comparative study would be considered level two evidence.

Um, a meta-analysis of multiple level 2 studies would be considered a level 2 study as well.

Um, the mid-range of Levels of Evidence would be level 3. that would be a typical case control study where one group has an intervention and the Other group does not, but this is not randomized necessarily. It is also not necessarily prospective.

A retrospective comparative study would also typically fall under the level 3 category of evidence.

The most commonly done study that's level 4 would be a case series where there's multiple patients that all receive the same treatment. There's not necessarily a control, it's not randomized, it's not prospective, um, typically done by one, researcher or clinician, but is a lower level of evidence because of the lack of a control and blinding and prospective analysis.

The level 5 study, uh, or the level 5 of evidence would be typically uh expert opinion. So an expert in a particular field gives a lecture and says, in my experience, I've treated X number of patients and noticed this to be the best result. So that would be considered level 5 evidence. Useful, but not necessarily without bias.

I'd like to talk about the Google Scholar H5 index. Most of us have heard about the impact factor. The impact factor relates to a specific journal, and is really more of a measure of journal prestige. It's usually a number that covers the last two years of publications of a journal, and refers to how many times articles in a particular journal have been cited across um uh the internet or or across academia.

The Google Scholar H5 index is a little bit different. It covers the last five years of publications, and it's defined as the largest number H, such that 8 articles published in a journal in the last five years have at least 8 citations each. So just as a quick example, let's say we have the biologic Orthopedics journal.

And over the last 5 years, we have 45 articles that have each been published, um, and cited at least 45 times. So that would give us, um, our particular Google Scholar H5 index of 45.

This is the H5 index rank of top orthopedic journals as of August 1st, 2019 at the time we did this study.

The number 1 ranked orthopedic journal was the American Journal of Sports Medicine. The H5 index number was 90. And so what that means is over the prior 5 years, there were 90 articles that had at least 90 citations in the American Journal of Sports Medicine. Uh, the number 2 journal was Clinical Orthopedics and Related Research, which was also included in, in our study.

Uh, with an H5 index of 71, uh, JBJS or the Journal of Bone and Joint Surgery, the American edition, uh, was the number 3 journal at that time, was also included in our study. Uh, the, the first journal in the top 10 that we did not include was the Journal of Arthroplasty. Uh, the next two journals, number 5 and 6, we did include.

And those were the journals of Knee Surgery, sports, traumatology, and arthroscopy, and also the, the Green Journal of arthroscopy, um, now renamed arthroscopy, the Journal of Arthroscopic and Related Surgery. And then the sixth and final journal that we included in this analysis was the Journal of Shoulder and Elbow Surgery.

Um,

So these are 6 of the top 10 journals. These are all uh well recognized journals and Orthopedics, well read, and all um considered very impactful in terms of developing new literature and, and new science, both clinical and basic science in Orthopedics.

So, what is the absolute gold standard for evidence? Well, I would say that that's a prospective, randomized, triple-blinded controlled trial. So what triple-blinded means is that not only does the researcher not know the intervention that's being given, the patient doesn't know the intervention they're receiving, and the person that's evaluating the outcomes doesn't know which patient got which intervention. So that's triple blinded.

That would truly be the gold standard where nobody really knows what's going on until it's decoded at the end of the, the, uh, the study.

So the question that we had sitting around one day was how close do we actually get to the gold standard in clinical Orthopedics in these top 10 journals.

It was a really a thought question at the time. And uh nowadays, uh in, in 2021, when you sit around and, and think, you don't uh sit on a piece of stone, you, you sit in a chair and you have a Starbucks, and you think about how could we answer that question. And this study is how we came up with a solution to try and decide what is the average level of evidence in, in orthopedic surgical journals.

And so this is our study design. We use 6 of those top 10 journals, based on the Google Scholar H5 index.

We looked at 1 year um time interval for all of the journals, which actually was 13 months since each journal was published once a month. We only looked at human clinical studies, so we eliminated basic science or drug interventions.

We looked at the study level of evidence as determined by the submitting author of the paper, or as determined by the editor of the journal. Um, most of these journals now require a level of evidence determination by the submitting author, so if that was present, we use that. Absent those two, sources, then the authors of this paper, each separately came up with a level of evidence, and then those were averaged.

How did we select those 6 journals? Well, as the senior author, I, I made it my prerogative, uh, primarily because these are 6 journals that I already subscribed to, and they were readily accessible, um, to complete our study.

And so these are the results. Starting with the American Journal of Sports Medicine, which was the number one ranked journal in 2019, we looked at 241 papers over those 13 months.

The average level of evidence was 2.96, so just under 3. And you can see the 95% confidence interval was pretty tight, ranged from 2.8 to 3.12.

The number 2 ranked journal in 2019 was Clinical Orthopedics and Related Research. In that journal, over 13 months, we, we evaluated 155 papers. We found the average level of evidence was 2.83. And remember here, the lower the level of evidence, um, uh, the closer we are to the randomized prospective double-blinded trial, to the, to the randomized controlled trial.

So, uh, COR, or clinical Orthopedics and related Research, had an average LOE across 155 papers of 2.83, uh, with pretty tight confidence intervals around the 95%, um, confidence level.

JBJS, perhaps the most well-known by name, orthopedic journal, uh, was the number 3 ranked journal in 2019. The average level of evidence across 164 papers was 3.04, with a 95% confidence interval of 2.91 to 3.17. Again, very narrow confidence intervals. Most of these papers hovered right around the level three evidence in that journal.

The next journal we used was Knee Surgery, sports, traumatology, and arthroscopy.

You can guess where we're going here. The average level of evidence across 402 papers over 13 months in this journal was 3.05.

The 95% confidence interval, 2.89 to 3.21.

The next journal, one of my favorites, Journal of arthroscopy and Related Surgery.

We looked at 226 papers. The average LOE was 3.19.

The final journal was the Journal of Shoulder and Elbow Surgery, also one of my favorites.

We looked at 237 papers and the average level of evidence was 3.25.

So when you put all of those averages together across these 6 journals and look at it graphically, this is what it looks like.

Remember, uh, on the left side, the Y axis is level of evidence. So, the, the taller, the taller the bar, the lower the level of evidence, um, as we see in this graphic. So, although it looks like there's a lot of variation amongst these six journals, when you look at the Y axis, you can see that the low range is about 2.8, up to a high of about 3.2. So, the actual range is pretty tight.

Um, across 1,425 cumulative papers looked at, the average level of evidence was 3, 3.05, but essentially 3, with a relatively narrow 95% confidence interval of 2.89 to 3.21. So with that, with that information, we concluded that the average level of evidence across these top six journals this year or that year, 2019 through 2018, was 3.

Now, in terms of some clinical questions that this brings up, does the study's level of evidence relate to the clinical importance of its conclusion? The answer is, of course not. It, it, it doesn't always follow.

That just because a study has a high level of evidence, that the conclusion is clinically relevant at all. And by the same token, an expert opinion, which is level 5 evidence, relatively low, may actually be very clinically relevant because that expert um has more experience than anyone else and is well respected and looked at as an ethical and evidence-based um researcher. So just because a level of evidence, uh, reaches a certain level, it doesn't really correlate with the, uh, clinical importance of the conclusion of the study.

What this really means for us in Orthopedics is that the average published orthopedic Research trial is truly a case control study. And this is borne out time and time again. We looked at different journals, we looked within a certain month of a journal, we looked um across different time intervals, and really, no matter how we sliced and diced it up, the average level of evidence for any journal, for any month at any time was about 3. It means that the average published orthopedic Research trial is a case control study.

It's not blinded.

It's not randomized.

And yet, we still quite often get very clinically useful information from these level 3 studies when they're well designed and well executed. And, and certainly when they're repeated across different centers with different clinicians.

So closing thoughts, the average level of evidence of published Orthobiologics Research trials should ideally match the average level of evidence of published orthopedic Research trials, and again, that's the level of evidence of 3.

Clinically relevant data can still be obtained from studies regardless of the level of evidence, and that is where the art comes in. It becomes the clinician's responsibility to read critically, um, not only the techniques and the methods, but also the conclusions of these studies regardless of level of evidence. But the idea that the gold standard for clinical change and, and, um, for publication should be level one is clearly not the case in this study.

I appreciate your time and and attention, and I hope you enjoy the rest of IOF Max. We're looking forward to a new and much better year in 2021.