Critical Evaluation: Overview

The critical evaluation model consists of combining multiple techniques : evaluating the processes used in source creation AND evaluating the context, source, and content.

Diagram: Critical evaluation techniques - adapted from Meriam Library (2019), Forzani (2019) and Hart (2018). Click image for PDF version.

Processes used in source creation

You start with evaluating the Credibility, Transparency, Corroboration, Defined variables, Validity, Reliability of the processes used in creating the source.

Context, Source and Content

Then you apply the CRAAP checklist evaluation approach to the source in this order:

Context

• Currency of information?
• How good does the source look: is the presentation accurate?

Source

• Evaluating authority of author/creator/publisher.
• Purpose of writing the text.

Content

• Relevance

Difficulty of applying the model

Using these questions, it is comparatively easier to evaluate context and source... 

In practice, internet users "scored highest on the actions that are easiest to perform and only require their opinion...and lowest on the recommendations that are more time consuming and that require effort" (Metzger, 2007: 2080).

... than evaluating the content.

"Expertise on the topic at hand is required to evaluate sources and to distinguish between relevant and irrelevant content. This expertise... typically evolves over the years when studying a subject" (Weber et al., 2018: 923).

Be aware that you bring your own biases to the source evaluation process.

The third most used behaviour when evaluating sources in Silva et al.'s 2018 study of 84 first year undergraduates at Brigham Young University was judgment based on their own political, social, economic or religious beliefs. Students examined whether the source did or did not fit with their own beliefs and evaluated the source accordingly.

Similarly, in an eye-tracking study of 79 students at a major German university (van Strien et al., 2016), participants with strong prior attitudes ranked websites that supported those attitudes as more credible than those websites that did not.

These findings support the tendency for students to evaluate sources as credible if the information confirms their pre-existing beliefs and not credible if the information does not, regardless of how well-argued, well researched and referenced the information is (Metzger and Flanagin, 2013: 215).

There is also extensive evidence that shows that political beliefs bias your perception of facts (Van Bavel and Pereira, 2018).

Prior knowledge also has a bearing on source evaluation. Students with little knowledge of the topic are more likely to trust less trustworthy sources and fail to differentiate between relevant and irrelevant criteria when deciding how trustworthy the sources are (Bouali and Kolinsky, 2023). However, the way that prior knowledge is measured may influence the relationship between prior knowledge and sourcing in previously unknown ways.(Anmarkrud et al., 2022).

The five truth criteria

When thoughts flow smoothly - the message is compatible with what you know - the more likely you are to accept the message, regardless of its truth.

This concept can be applied to five truth criteria (Schwartz, 2015):

• Compatibility, as evidence by these studies with students.
• Credibility -is the source trustworthy, are you familiar with it?,
• Social consensus - do others believe it is true?
• Coherent - do the conclusions follow from the argument?
• Evidence - is there supporting evidence in peer-reviewed articles or in trustworthy news sources?

Each of these truth criteria can be assessed on the basis of either: 

• Relevant knowledge, which is slower because it takes more effort and time. Applying the evaluation criteria outlined in this guide might require extensive searching.

or

• Intuitive assessment, which is faster and less trouble.

When the intuitive assessment suggests the message does not ring true, then you are more likely to apply the slower route of trying to confirm accuracy if you have the time and it is worthwhile.

"Familiar, frequently encountered information and information that is coherent and compatible with one’s knowledge is .. easier to process than information that is not...ease of processing provides heuristically useful – but fallible – information for assessing how well a claim meets major truth criteria" (Schwarz and Jalbert, 2020: 77).

Claims can feel true - regardless of whether they are - because they are easy to process which allows for manipulation. One of the key variables that allow for ease of processing is repetition - repeating a few key messages, especially on social media, brings exposure and familiarity that increases popularity and truth.

"Interpreting sourced material is difficult and time-intensive...students need to engage in more critical thinking about what kinds of sourced material lend authority" (Silva et al., 2018: 37). 

In a survey of 8,353 college students in the U.S., evaluating sources was "time-consuming and arduous, though it was perceived as essential. The process is fundamental to building a sound argument, but also selecting valid and reliable sources for an assignment was important to their credibility" (Head and Eisenberg, 2010: 18). 

In Project Information Literacy's October 2018 News study report (Head et al., 2018) surveying over 5,800 college students, 62% checked the currency of an item. Over half also read the complete story before sharing, checked the URL to see where the story originated, checked to see who it was that posted the news item, read any comments about the news post and compared and fact-checked the news story against other sources.

One of their recommendations was to equip students with a healthy skepticism about news and the dangers of misinformation alongside "a more affirmative set of tools for acknowledging well-grounded information to steer them away from cynicism, and the distrust of all news" (Head et al., 2018: 32).

This aligns with the UK Government's Online Media Literacy Strategy which supports the education and empowerment of all internet users with the key skills and knowledge they need to be safe online.  "We want users to be able to critically evaluate the content they consume" (UK Government, 2021: 2).