Inter-Rater Consistency — Quality Control SOP
Ensures that different reviewers analyzing the same case produce materially consistent scores, conclusions, and exposure classifications.

Purpose

  • Eliminate reviewer-dependent variability
  • Ensure scoring consistency across all cases
  • Protect credibility of exposure classifications
  • Standardize interpretation of clinical findings
  • Create institutional-level reliability

What AI Extracts (Facts Only)

  • All scoring inputs used by each reviewer
  • Category-by-category score breakdown
  • Final exposure scores from multiple reviewers
  • Variances between reviewers
  • Flagged discrepancies in key domains (causation, damages, etc.)

What Clinician Must Confirm (Validation)

  • All reviewers used the same baseline assumptions
  • Causation interpretation is aligned
  • Damages inputs are based on the same validated data
  • No reviewer introduced unsupported assumptions
  • Scoring logic is applied consistently across categories
Consistency is not agreement—it is alignment based on the same facts and standards.

Critical Thinking Steps

  • Compare category-level scores, not just final totals
  • Identify where scoring diverges (causation, damages, documentation)
  • Trace variance back to underlying assumptions
  • Determine whether difference is:
    • Justified (reasonable interpretation difference)
    • Unjustified (error, omission, or bias)
  • Resolve discrepancies through evidence-based review
  • Re-align scores based on validated findings

Variance Thresholds

  • 0–5 point difference → Acceptable variance
  • 6–10 point difference → Review required
  • 11–15 point difference → Mandatory reconciliation
  • 16+ point difference → Scoring invalid until resolved
Large variance signals a breakdown in process—not a difference of opinion.

Root Cause Analysis for Variance

  • Baseline misunderstanding
  • Different causation interpretations
  • Inconsistent weighting of damages
  • Documentation overlooked or misinterpreted
  • Failure to apply rebuttal adjustments
  • Subjective bias or assumption creep

Stop Rules

  • STOP if reviewers used different source data
  • STOP if causation is not aligned across reviewers
  • STOP if scoring categories were applied inconsistently
  • STOP if variance exceeds acceptable thresholds without explanation
No final exposure score is released until variance is reconciled and documented.

Final Output Requirements

  • Final reconciled exposure score
  • Variance report between reviewers
  • Category-level comparison summary
  • Explanation of any resolved discrepancies
  • Confirmation of aligned assumptions
  • Documented rationale for final score
The final score must represent a unified, evidence-based position—not an individual opinion.