Skip to content

Paper Checker Guide

PaperChecker is a fact-checking system designed to validate research claims in medical abstracts by systematically searching for contradictory evidence across a document database.

Overview

The system performs seven key operations:

  1. Extracting statements from abstracts
  2. Generating semantic negations of claims
  3. Conducting multi-strategy searches
  4. Scoring document relevance
  5. Pulling supporting citations
  6. Synthesizing evidence reports
  7. Classifying findings

Verdict Classification

Results are categorized into three types:

Verdict Meaning
SUPPORTS Counter-evidence search found no contradictions
CONTRADICTS Evidence was found opposing the claim
UNDECIDED Insufficient or conflicting evidence

Search Methodology

PaperChecker employs three complementary search strategies:

Uses vector similarity to find related documents based on meaning.

2. HyDE (Hypothetical Document Embedding)

Generates hypothetical documents that would contradict the claim, then searches for similar real documents.

Traditional full-text search using extracted keywords.

Results are deduplicated across strategies. Higher multi-strategy matches typically indicate well-researched topics.

Usage

Basic Usage

uv run python paper_checker_cli.py abstract.json

Input Format

{
  "abstract": "This study demonstrates that...",
  "title": "Effects of Treatment X on Condition Y",
  "pmid": "12345678",
  "authors": "Smith J, et al.",
  "year": 2024,
  "journal": "Journal of Medicine",
  "doi": "10.1234/example"
}

Minimum Requirements

Abstracts require minimum 50 characters of text.

Command Line Options

# With output file
uv run python paper_checker_cli.py abstract.json -o results.json

# Adjust scoring threshold
uv run python paper_checker_cli.py abstract.json --score-threshold 3.0

# Quick mode
uv run python paper_checker_cli.py abstract.json --quick

Confidence Assessment

Level Description
High Strong, consistent evidence from quality sources
Medium Moderate evidence with limitations
Low Limited evidence or conflicting findings

Output Format

{
  "abstract_pmid": "12345678",
  "verdict": "SUPPORTS",
  "confidence": "high",
  "claims_analyzed": [
    {
      "claim": "Treatment X reduces symptom Y",
      "contradictory_evidence": [],
      "supporting_evidence": [
        {
          "pmid": "87654321",
          "title": "Confirming Study",
          "relevance": 0.92
        }
      ]
    }
  ],
  "overall_assessment": "No contradictory evidence found...",
  "search_strategies_used": ["semantic", "hyde", "keyword"],
  "documents_searched": 150
}

PaperChecker works well for:

  • Validating preprint claims
  • Auditing systematic reviews
  • Identifying controversial findings
  • Assessing training data quality
  • Understanding evidence landscapes

Limitations

Not suited for:

  • Extremely recent findings (database lag)
  • Highly specialized niche topics
  • Methodological statements
  • Non-biomedical content

Performance Considerations

Factor Typical Value
Processing time per abstract 2-5 minutes
Documents searched 50-200
Claims extracted 3-10 per abstract

Optimization Tips

  • Maintain adequate database coverage
  • Ensure document embeddings exist
  • Adjust scoring thresholds based on desired precision vs recall

Lab Interface

For interactive use, the Paper Checker Lab provides a GUI:

uv run python paper_checker_lab.py

Features:

  • Abstract input with metadata fields
  • Real-time progress tracking
  • Interactive results exploration
  • Export capabilities

Integration with Workflow

graph TD
    A[Input Abstract] --> B[Extract Claims]
    B --> C[Generate Negations]
    C --> D[Multi-Strategy Search]
    D --> E[Score Documents]
    E --> F[Extract Citations]
    F --> G[Synthesize Report]
    G --> H[Classify Verdict]

Database Requirements

PaperChecker requires:

  • PostgreSQL with pgvector extension
  • Populated document database
  • Pre-computed embeddings for semantic search

Best Practices

Abstract Selection

  • Choose abstracts with clear, testable claims
  • Ensure the topic is covered in your database
  • Consider publication date relative to database currency

Interpreting Results

  • SUPPORTS doesn't mean "proven true"
  • CONTRADICTS warrants further investigation
  • UNDECIDED may indicate novel research