Evidence Synthesis in the Social Sciences

Your last step before actually working with your evidence is to screen all included studies for risk of bias, and critically appraise them for quality. As with many steps in this process, there are many frameworks and formulations of appraisal that can help you systematically review your included studies for bias and quality. 

Guidance from Prisma (2020): 

"Specify the methods used to assess risk of bias in the included studies, including details of the tool(s) used, how many reviewers assessed each study and whether they worked independently, and, if applicable, details of automation tools used in the process." - p. 11

The following is a very comprehensive list of critical appraisal tools. Each tool corresponds to either a body of knowledge or a specific type of study or research method. Be sure to use a tool that matches the subject, method, and/or study type of the tool you're use to appraise. 

• AACN Levels of Evidence

  Help nurses determine strength of research studies, assess findings, and evaluate evidence

• AMSTAR: A Measurement Tool to Assess Systematic Reviews

  AMSTAR Checklist to assess the quality of a systematic review by helping users differentiate between systematic reviews, focusing on their methodological quality and expert consensus

• AOTA Critically Appraised Papers

  American Occupational Therapy Association Critically Appraised Papers has a CAP Worksheet and CAP Guidelines for researchers to complete before submission to Evidence Exchange. The worksheet is based on the McMaster quantitative critical appraisal form.

• Centre for Evidence Based Medicine (Oxford) CATmaker and EBM calculator

  CEBM's computer assisted critical appraisal tool CATmaker and the interactive evidence based medicine calculator. CATmaker requires a file to be downloaded for use.

• Centre for Evidence Based Medicine (Oxford) Critical Appraisal Tools

  CEBM's worksheets to assess systematic reviews, diagnostic, prognosis, and RCTs article types.

• Cochrane Assessing Risk of Bias in a Randomized Trial

  Cochrane Handbook chapter detailing the Cochrane risk of bias assessment for randomized trials

• Critical Appraisal Skills Programme

  The CASP provides worksheets to appraise randomized control trials, systematic reviews, cohort studies, case control studies, qualitative research, economic evaluations, diagnostic tests, and clinical prediction rules.

• Dartmouth Biomedical Libraries

  Dartmouth provides a series of worksheets designed to aid in formulating clinical questions, appraising the evidence, and applying the evidence to the patient.

• Diffusion of Innovations in Health Service Organisations

  Includes quality checklists for experimental studies, quasi-experimental studies, attribution studies, questionnaire surveys, qualitative studies, mixed methods, "real-world" implementation studies, and action research.

• EPHPP Quantitative Study Quality Assessment Tool

  Canada's Effective Public Health Panacea Project (EPHPP) Quality Assessment Tool for Quantitative Studies, which critically appraises quantitative studies on public health topics. The tool is a linked PDF and there are also links to articles explaining the tool.

• GRADE (The Grading of Recommendations Assessment, Development and Evaluation)

  GRADE working group guidelines and publications for grading the quality of evidence in healthcare research and policy

• HCPRDU Evaluation Tool for Mixed Methods Studies

  University of Salford Health Care Practice R&D Unit critical appraisal tool to assess the quality of mixed methods studies

• HCPRDU Evaluation Tool for Qualitative Studies

  University of Salford Health Care Practice R&D Unit critical appraisal tool to assess the quality of qualitative studies

• HCPRDU Evaluation Tool for Quantitative Studies

  University of Salford Health Care Practice R&D Unit critical appraisal tool to assess the quality of quantitative studies

• The Integrative Review by Whittemore and Knafl

  Benchmark article describing how to perform integrative reviews. Table 1 of the article which includes a scale to determine methodological rigor and data relevance for evidence.

• JAMA Evidence Critical Appraisal Worksheets

  Critical appraisal worksheets on topics including diagnosis, harm, therapy and qualitative research from JAMA Evidence

• JAMA Series on Step-By-Step Critical Appraisal

  Listing from Johns Hopkins of the JAMA User's Guide to Medical Literature with each step in critical appraisal process. 

• Joanna Briggs Institute Checklist for Assessing the Validity of Descriptive/Correlational Studies

  Article with study validity assessment tool from JBI's System for the Unified Management, Assessment and Review of Information (SUMARI) software. The software requires a subscription, but the article provides the validity checklist from the software.

• Joanna Briggs Institute Critical Tools

  JBI’s critical appraisal tools are generally used in nursing and assist in assessing the trustworthiness, relevance and results of published papers with downloadable worksheets to appraise for RCTs, systematic reviews and other studies.

• Johns Hopkins Nursing Evidence Based Practice

  A foundational book for evidenced-based practice. Check out the appendices of the ebook for evidence appraisal tools

• Knowledge Translation Program

  The Toronto based KTP has critical appraisal worksheets for systematic reviews, prognosis, diagnosis, harm and therapy

• McMaster Critical Review of Qualitative and Quantitative Studies

  Guidelines from McMaster University in Canada on assessing the strength of qualitative and quantitative studies, specifically focusing on Occupational Therapy research. The quantitative and qualitative worksheets and guidelines are found within the table on the page.

• Mixed Methods Appraisal Tool (MMAT) Version 2018

  The MMAT assesses the quality of qualitative, quantitative, and mixed methods studies.

• PEDro

  PEDro (Physiotherapy Evidence Database) Scale is an excellent webpage which provides access to a range of appraisal resources including a tutorial and appraisal tool.

• Quality Assessment Tool for Studies of Diverse Designs (QATSDD)

  This paper describes QATSDD, an interdisciplinary study quality assessment of tool that is particularly useful to health services researchers in psychology, sociology, and nursing

• Risk of Bias Toolkit 

            Validity assessment tools for evidence synthesis

• ROBIS

  University of Bristol tool for assessing the risk of bias in systematic reviews of interventions, diagnosis, prognosis and etiology.

• Scottish Intercollegiate Guidelines Network

  SIGN worksheets to critically appraise systematic reviews, RCT, cohort studies, case-controlled studies, diagnostic studies and economic studies

• Simulation Research Rubric

  The Simulation Research Rubric provides a reliable method for grading the quality of simulation based research reports

• SORT (Strength of Recommendation Taxonomy)

  The SORT scale addresses the quality, quantity, and consistency of evidence in individual studies or bodies of evidence

• STROBE (Strengthening the Reporting of OBservational studies in Epidemiology)

  Addresses cohort, case-control, and cross-sectional studies

Adapted from Duquesne University Libraries