Critical Evaluation: Criteria Checklist Approach: CRAAP
There are different models and criteria that you can use to help you to make an informed decision on whether it is a reliable source to which you can refer in your academic work.
One of these models, the CRAAP test (Meriam Library, 2019) can help you to systematically consider different characteristics of any source of information: Currency, Relevance, Authority, Accuracy, and Purpose.
Whichever model you choose, be aware of the limitations of using "evaluation checklists".
Currency
- When was the source of information published?
- Are you required to only refer to sources of information published within a certain time-period?
- Is there more current data available?
- Has the information been challenged, revised or superseded by more current data or a more recent edition?
- Check the date of publication for your source of information, whether this is a book, journal article or other format.
- Refer to study/research guidelines for what is considered a current or acceptable publication date for your sources.
- Ensure you have searched widely enough to find more recent editions or publications, where these might exist.
Of the 8,353 respondents to a survey of college students on 25 campuses distributed across the U.S. in the spring of 2010, the most used (77%) proxy for information quality was whether web content was up-to-date (Head and Eisenberg, 2010: 11).
Relevance
- Is the information relevant to your topic? Does it answer your question?
- Is the source of information aimed at a particular audience or is it pitched at a level that is appropriate for academic purposes?
- Revisit your assignment title, project brief or research outline to check the information for relevance.
- When searching for sources of information, use the advanced search from the Library search at https://www.bcu.ac.uk/library and limit your search results to peer-reviewed sources. This guarantees that they are suitable for academic purposes.
- Check the keywords in the sources you find to see whether they match the keywords in your search. Checking the title is usually sufficient to act as a quick filter. If searching for journal articles, once you have filtered by title, check the abstract to determine whether the source is relevant.
Determining whether the source material is relevant to your objectives based upon keyword matching was the most common basis for evaluation among 1650 undergraduates in two American state universities (Wiley et al., 2009).
The keywords that research physicists were looking for when evaluating sources related to three aspects: the topic itself, the research methods used and the names of key authors and key research groups (Bazerman, 1985: 7).
Note: Silva et al. (2018) found that students' reliance on their previous experience was not a help in distinguishing highly-biased sources from fact-based sources. Their conclusion was that "previous experience must be strong and domain-specific in order to help make valuable information-evaluation judgements" (Silva et al., 2018: 38).
Authority
- Who is the author/publisher/sponsor for the source of information?
- What are the author's credentials and qualifications?; are they suitably qualified to write such a piece?
- Does the publisher have a good reputation for academic publications?
- Is there a conflict of interest or potential bias arising from the sponsor, author's employer or owner of the source?
- Do you need information from a specific country? US law, for example, is different from UK law. Scottish law may well be different to English law.
- Experts ask these questions to make inferences about a journal:
- Is this a peer-reviewed journal?
- Is this journal well-respected in the subject field? There are lists that rank academic journals.
- For how long has it been published?
- Is this the publication of a professional institution or society?
- Does it have an editorial board and advisers who have a publishing background?
- Specialist search tools/resources available through BCU Library will often allow you to search for further work by a particular author, in order to gain a better understanding of the author's body of work
- Peer-reviewed articles provide a level of assurance that the information published is considered sound according to the current level of knowledge available to experts in the area.
- When searching for sources of information using specific search tools available through BCU Library, you have the option to limit your search results to only peer-reviewed, which means they will be suitable for academic purposes.
- Publications should reveal details of any sponsors but you may have to search the 'small print' for this information.
- If the source is a webpage, does the URL suggest that the source has a reputation to uphold, such as a government or university, for example:.gov or .ac.uk
"Attention to the source seems to be an efficient strategy to determine the reliability of multiple documents and to select more reliable documents in the first place" (von der Muhlen et al., 2016: 1695)
This finding was a result of comparisons between psychology staff and students and confirmed previous work by Wineburg (1991) in the field of history and Lundeberg (1987) in the field of law.
Are the author(s) credentials or affiliations listed? Have you heard of the author or read any of their other work? What is the author's track record? This is the "best source of evidence for making credibility choices" (Goldman, 2001: 106).
"There are three things which... induce us to believe a thing apart from any proof of it: good sense, good moral character, and goodwill" (Aristotle, 350 BCE).
"when choosing between sources, people are likely to believe a source whose name they recognize as credible compared to an unfamiliar source, even with little inspection of the actual content of the site or source credentials" (Metzger and Flanagin, 2013: 214).
This has been confirmed by Pilditch et al. (2020: 10) who found that "sources believed to be very trustworthy are seen as more credible even when they provide erroneous advice whilst sources perceived to be of low trustworthiness are distrusted further – even in cases where they actually provide accurate advice".
Accuracy
- Is the information supported by evidence, for example: verifiable references which you can check and/or research data?
- Has the article been peer-reviewed?
- Can you verify any of the information in a different, reputable source?
-
Can you trust the source? Questions that you can ask are: how good does the source look? are there any factual errors or out of date information? how good is the quality of the writing - are there mistakes in the spelling or grammar? can you communicate with the author and with whom is the author affiliated?
- Peer-reviewed articles provide a level of assurance that the information published has been checked by another expert in the field.
- When searching for sources of information using specific search tools available through BCU Library, you have the option to limit your search results to only peer-reviewed, which means they will be suitable for academic purposes.
- If you carry out wide enough searches, using effective search tools available through BCU Library, you should be able to identify articles which could offer verification.
Of the 8,353 college students in Head and Eisenberg's (2010) survey, many respondents (71%) mentioned checking a site for typos, misspellings and a poor use of graphics and navigational systems as a basis of evaluating design and the credibility of content.
Purpose
- What is the purpose of the information? Is it to inform, entertain, sell, or persuade?
- Is there an obvious financial benefactor or sponsor associated with the source of information?
- Is the information substantiated with a high standard of evidence or is it simply opinion or propaganda?
- Is there discussion of the theoretical background?
- Are there potential political, ideological, cultural, religious, institutional or personal biases?
- Scrutinise the source for any clues about possible sponsors or other parties which stand to gain financially or reputationally from the information.
- Check references and review any available research data to check that this an academic source of information.
- Go back to the original source of the information and check what was stated rather than using the blog, social media post or online news article.
- Ensure that you search widely enough to discover reputable sources of information which can corroborate or refute the data in the source you have found.
- When searching for sources of information using specific search tools available through BCU Library, you have the option to limit your search results to only peer-reviewed, which means they will be suitable for academic purposes.
The presence of advertising, especially pop-ups, often leads people to think that a site is more interested in commercial interests and therefore not reliable (Fogg et al., 2003).
Limitations of the "evaluation checklists"
Students are often recommended to evaluating webpages with a checklist approach such as:
- The CRAAP test (Meriam Library, 2019): Currency, Relevance, Authority, Accuracy and Purpose.
- RADAR (Mandalios, 2013, PDF): Relevance, Authority, Date, Appearance, Reason for writing.
- The SIFT method (Hapgood, 2019): Stop, Investigate the source, Find better coverage and Trace claims to the original context.
- The Trust in Online Health Information (TOHI) scale, used in a health context, that features brand, content, credibility, ease of use, recommendation, style, usefulness and verification (Rowley et al., 2015).
Using these features is a "fairly reliable process for evaluating sources" (Chinn and Rinehart, 2017: 1714). The checklist approach is an efficient way of evaluating sources, can easily be applied and uses memorable acronyms.
However, this efficiency encourages surface evaluation, can exclude sources that may be relevant and ignores the actual claims made in the source (Kim and Hannafin, 2016).
The checklist approach therefore needs to be augmented by emphasising the processes used by the source in its creation.
"Knowing the processes used by sources provides a more accurate evaluation of the credibility of the source's claims than relying solely on the source feature" (Chinn and Rinehart, 2017: 1711).