What makes a scientific article credible? A look at peer review and impact factors

Imagine you want to eat out tonight and decide to try out a new restaurant in your town. How do you choose which restaurant to go to? Do you check online for reviews or ask your friends for recommendations? Now, imagine that you narrow your choices down to two restaurants. One of them has five-star reviews from the top chefs of your country and the other has no reviews. Which restaurant are you more likely to choose?

You’re probably even more selective when choosing which journal articles to rely on for evidence-based medicine. A vast amount of research is being published regularly and sifting through the numerous articles can be daunting. Here are three questions that will help you quickly determine an article’s credibility:

1. Is it published in a peer-reviewed journal?
2. Is it published in a journal with a high impact factor?
3. Is it cited by other authors in their papers?

Peer-reviewed journals are considered the gold standard of scientific research publications. Reputable journals have subject matter experts who volunteer their time to review submitted articles and evaluate their credibility. Think of the peer-review process as a team of experts reviewing and approving the work before you see it.

How does the peer-review process work?

During the submission process, experts raise any concerns they may have with an article to the authors. An article is evaluated on its originality, significant findings, research methodology, and writing. The reviewers usually come back to the authors with comments and suggestions on how to make the article (e.g., study) better. The authors are then given a specific amount of time to respond back with their revisions. Once an article meets the standards of the editorial board, it is cleared for publication. If it doesn’t meet the standards, it will be rejected, and the authors will usually submit the article to another (usually less prestigious) journal.

How does the peer-review process differ across journal publications?

The length of time that the peer-review process takes differs across publications. As well, some journals do not share the authors’ names with the reviewers, and the authors are unaware of who reviewed their paper. Other journals fully disclose the authors’ names and affiliated institutions.

What are some problems with the peer-review process?

The peer-review process is not perfect. Faulty scientific papers do still get published due to potential loopholes. For example, a journal relies on the integrity of its editorial board. Experts are not paid for their work. They may be working with tight deadlines making it difficult for them to critically evaluate all the research that comes their way.

Experts also shouldn’t have any personal affiliations with the authors of the study. Would you trust a chef’s review of a restaurant if you found out he owned it? Or that the owner was his daughter? Finally, peer-reviewers only see the manuscript that is in front of them. They don’t get to see the raw data that the researchers used. So, any errors in the data may not be picked up by them.

Determining where an article is published is an important step for determining its credibility. The peer-review process is the first test of a scientific article’s credibility. Ideally, experts in the specific field will be best equipped to identify potential concerns with a paper’s methodology and findings. Rigorous journal standards should filter out dodgy scientific papers before they are released to the public. But, remember that the peer-review process does not guarantee a journal article’s validity.

A journal publication that claims to be peer-reviewed may still be unreliable. One way to ensure its credibility is to examine its impact factor. The impact factor, or IF, of a journal publication is the number of times an average paper published in the journal is cited by other articles. It gives you an idea of how reputable the journal’s articles are. Ideally, an impact factor gives you an impression of a journal publication’s impact on the scientific community.

In general, credible journals have high impact factors. Conversely, a low impact factor may indicate that a journal is predatory and unreliable. However, the actual value of a specific impact factor may differ across disciplines. An impact factor of three may be considered low for a wide-ranging specialty (such as internal medicine), but considered high for a specific discipline such as physics. Impact factors are best used when compared between journals within your target field.

Where can you find a journal’s impact factor?

A journal’s impact factor can usually be found on the journal’s website. However, it may be tricky to locate. Sometimes, the impact factor listed may even be fake! One sign of a predatory journal is that they may list a made-up impact factor on their website to fake credibility. It’s important to verify the impact factor on a journal’s website with an online database that lists this type of information. The best-known site, Web of Science Journal Citation Reports (JCR), is a great online resource to find the latest impact factor for a journal.

What are some problems with impact factors?

As we already mentioned, predatory journals may just make up an impact factor on their website. So, you’ll need to cross-reference what they state with credible databases.

Younger journals that may be credible won’t be able to have an impact factor for the first two years since impact factors aren’t calculated for journals that are less than two years old.

Also, impact factors are calculated using the average number of citations in a publication. This means that a journal with a few highly cited articles will have a high impact factor although most of the articles are not cited. So, a journal’s impact factor isn’t always an accurate reflection of how an individual paper is cited.

A great source to directly evaluate an article’s credibility is Google Scholar, which allows you to see how many times an article has been cited.

If an article is cited by other papers, this usually means that the authors’ citing it think that it’s legitimate and valuable research. Overall, articles with many citations are deemed valuable by many researchers.

Just as positive reviews of a product can be a good indication of the product’s quality, a well-received article with many citations can give you a good idea about the article’s quality.

The problem with citations

However, multiple citations do not always equal quality research. For example, researchers may cite their own work in other articles. These articles will then appear as highly cited on Google Scholar. As well, if there are thousands of citations it doesn’t necessarily mean that all of those citations are from credible authors. Think about the reviews you may find for a popular Thai restaurant promising authentic cuisine. How many of those reviewers have experience with authentic Thai cuisine in the first place?

So, now you know how to check an article’s credibility by looking for peer reviews, citations, and the journal’s impact factor. But what about the people writing the papers? Check out the next article in this guide to learn how to assess whether an author has the experience and credentials to write a credible research article.