Critical Evaluation: Processes in source creation
Learn how to critically evaluate the sources you are using so that you can confidently provide these as evidence in your work.
Source credibility is " a function of the reliability of the processes used by the source" (Chinn and Rinehart, 2017: 1710).
Whether an expert is credible in what they write will depend on the processes used to generate the evidence. The processes used to generate the evidence will vary depending upon the subject.
The research methods used and their validity is a more accurate evaluation of the credibility of the source than relying just on source features.
Initial evaluations of quantitative and qualitative studies
Use the probing questions listed below based on Tables 3.6 and 3.7 of Hart (2018: 83).
Credibility
- Is there sufficient primary data to warrant the author's interpretation?
- Is the interpretation plausible?
- Does the evidence support the claims?
- Are the claims framed against the existing literature?
- Does the interpretation explain existing as well as new evidence?
Transparency
- What research methods were used to create the source?
- Are the methods suited to the aims of the study?
Corroboration
- Are there multiple sources of evidence?
- How reliable are these?
- Are there alternative sources of data that could have been used?
- Does the author refer to other sources?
- Have you checked these sources to corroborate the evidence?
Checking that other sources corroborate the information presented takes more effort than other strategies but remains "a fast and frugal means of arriving at a credibility judgment" (Metzger and Flanagin, 2013: 215). In the survey by Wineburg and McGrew (2019), professional fact-checkers quickly scanned a web site and then opened up a new tab to check the credibility of the original site. This lateral scanning proved far quicker at ascertaining the accuracy of a website than the vertical scanning used by undergraduates and academic historians at Stanford University. The vertical scanning approach stayed within the original website meaning they were more likely to be victims of features such as domain names and logos that can easily be manipulated.
Out of 84 first year students in a study at Brigham Young University, 81 used sources in the article as a way to evaluate its reliability. This varied from commenting on what facts were sourced, whether a specific institution or organization was named as the source of the evidence or when an opinion had no supporting evidence (Silva et al., 2018).
Knowledge of the use of news syndicates (such as Reuters and Associated Press which produce and distribute current news stories and press releases to subscribing organizations) proved helpful to the first year undergraduates evaluating news stories in Evanson and Sponsel (2019) as it helped them to make an accurate evaluation of the news story.
Out of 84 first year students in a study at Brigham Young University, 81 used sources in the article as a way to evaluate its reliability. This varied from commenting on what facts were sourced, whether a specific institution or organization was named as the source of the evidence or when an opinion had no supporting evidence (Silva et al., 2018).
Knowledge of the use of news syndicates (such as Reuters and Associated Press which produce and distribute current news stories and press releases to subscribing organizations) proved helpful to the first year undergraduates evaluating news stories in Evanson and Sponsel (2019) as it helped them to make an accurate evaluation of the news story.
Defined and measurable outcome variables
- Have these outcome variables been isolated and defined?
- Can the output variables be measured?
- Is there a standard way of doing so and has it been used?
Validity
- Is the sample representative of the population being studied?
- Has the sample selection minimised the possibility of bias?
- What is the significance that the sample gives to the results?
- What is the sample size?
25 is usually considered the lowest reasonable group size in quantitative research.
- Are there any p-values greater than 0.10?
If so, there is little evidence of statistical significance. Any differences seen can be due to chance. The smaller the p-value, the stronger the case for statistical significance. You may find the advice contained in Philip Dee's introductory video Appraising Research (YouTube) to be helpful.
Reliability
- How were threats to the accuracy of the samples and the research methods used addressed?
- Are the research methods transparent?
- Has the reliability of the research methods been confirmed?
You are more likely to rate more highly sources that produce empirical evidence based on sound research methods.