RAGScore

⚡ 2-Line RAG Evaluation

# Step 1: Generate QA pairs from your docsragscore generate docs/
# Step 2: Evaluate your RAG systemragscore evaluate http://localhost:8000/query

That's it. Get accuracy scores and incorrect QA pairs instantly.

============================================================✅ EXCELLENT: 85/100 correct (85.0%)Average Score: 4.20/5.0============================================================
❌ 15 Incorrect Pairs:
  1. Q: "What is RAG?"     Score: 2/5 - Factually incorrect
  2. Q: "How does retrieval work?"     Score: 3/5 - Incomplete answer

🚀 Quick Start

Install

pip install ragscore              # Core (works with Ollama)pip install "ragscore[openai]"    # + OpenAI supportpip install "ragscore[notebook]"  # + Jupyter/Colab supportpip install "ragscore[all]"       # + All providers

Already installed? Keep up to date — new versions add features like failure diagnosis and retrieved context capture:

bash

pip install --upgrade ragscore

Option 1: Python API (Notebook-Friendly)

Perfect for Jupyter, Colab, and rapid iteration. Get instant visualizations.

from ragscore import quick_test
# 1. Audit your RAG in one lineresult = quick_test(    endpoint="http://localhost:8000/query",  # Your RAG API    docs="docs/",                            # Your documents    n=10,                                    # Number of test questions)
# 1b. Tailored QA — target specific audiencesresult = quick_test(    endpoint="http://localhost:8000/query",    docs="docs/",    audience="developers",                   # Who asks the questions?    purpose="api-integration",               # What's the document for?)
# 2. See the reportresult.plot()
# 3. Inspect failuresbad_rows = result.df[result.df['score'] < 3]display(bad_rows[['question', 'rag_answer', 'reason']])

Rich Object API:

• result.accuracy - Accuracy score
• result.df - Pandas DataFrame of all results
• result.plot() - 3-panel visualization (4-panel with detailed=True)
• result.corrections - List of items to fix

Option 2: CLI (Production)

Generate QA Pairs

# Set API key (or use local Ollama - no key needed!)export OPENAI_API_KEY="sk-..."
# Generate from any documentragscore generate paper.pdfragscore generate docs/*.pdf --concurrency 10
# Tailored QA generation — target specific audiencesragscore generate docs/ --audience developers --purpose faqragscore generate docs/ --audience customers --purpose "pre-sales"ragscore generate docs/ --audience "compliance auditors" --purpose "security audit"

Evaluate Your RAG

# Point to your RAG endpointragscore evaluate http://localhost:8000/query
# Custom optionsragscore evaluate http://api/ask --model gpt-4o --output results.json

🔬 Detailed Multi-Metric Evaluation

Go beyond a single score. Add detailed=True to get 5 diagnostic dimensions per answer — in the same single LLM call.

result = quick_test(    endpoint=my_rag,    docs="docs/",    n=10,    detailed=True,  # ⭐ Enable multi-metric evaluation)
# Inspect per-question metricsdisplay(result.df[[    "question", "score", "correctness", "completeness",    "relevance", "conciseness", "faithfulness"]])
# Radar chart + 4-panel visualizationresult.plot()

==================================================✅ PASSED: 9/10 correct (90%)Average Score: 4.3/5.0Threshold: 70%──────────────────────────────────────────────────  Correctness: 4.5/5.0  Completeness: 4.2/5.0  Relevance: 4.8/5.0  Conciseness: 4.1/5.0  Faithfulness: 4.6/5.0==================================================

CLI:

ragscore evaluate http://localhost:8000/query --detailed

🔍 Failure Diagnosis (--diagnose)

When answers fail, --diagnose tells you why — retriever miss, generator hallucination, incomplete answer, or wrong interpretation:

ragscore evaluate http://localhost:8000/query --diagnose

🔍 Failure Diagnosis:  Retriever Miss: 3 (42.9%)  Generator Hallucination: 2 (28.6%)  Incomplete Answer: 1 (14.3%)  Wrong Interpretation: 1 (14.3%)

Uses the support_span already generated with each QA pair to give the judge grounding context. Combine with --detailed for full diagnostics:

ragscore evaluate http://localhost:8000/query --diagnose --detailed -o results.json

📓 Full demo notebook — build a mini RAG and test it with detailed metrics.

🎯 Audience & Purpose demo — generate tailored QA for developers, customers, auditors, and more.

🏠 Ollama local demo — 100% private RAG evaluation with no API keys.

🏠 100% Private with Local LLMs

# Use Ollama - no API keys, no cloud, 100% privateollama pull llama3.1ragscore generate confidential_docs/*.pdfragscore evaluate http://localhost:8000/query

Perfect for: Healthcare 🏥 • Legal ⚖️ • Finance 🏦 • Research 🔬

Ollama Model Recommendations

RAGScore generates complex structured QA pairs (question + answer + rationale + support span) in JSON format. This requires models with strong instruction-following and JSON output capabilities.

Minimum recommended: 8B+ models. Smaller models (1.5B–3B) produce lower quality support spans and may timeout on longer chunks.

Ollama Performance Guide

# Recommended: 8B model with concurrency 2 for local machinesollama pull llama3.1:8bragscore generate docs/ --provider ollama --model llama3.1:8b
# GPU server (A10/L20): larger model with higher concurrencyollama pull qwen2.5:32bragscore generate docs/ --provider ollama --model qwen2.5:32b --concurrency 5

Expected performance (28 chunks, 5 QA pairs per chunk):

RAGScore auto-reduces concurrency to 2 for local Ollama to avoid GPU/CPU contention.

🔌 Supported LLMs

📊 Output Formats

Generated QA Pairs (output/generated_qas.jsonl)

{  "id": "abc123",  "question": "What is RAG?",  "answer": "RAG (Retrieval-Augmented Generation) combines...",  "rationale": "This is explicitly stated in the introduction...",  "support_span": "RAG systems retrieve relevant documents...",  "difficulty": "medium",  "source_path": "docs/rag_intro.pdf"}

Evaluation Results (--output results.json)

{  "summary": {    "total": 100,    "correct": 85,    "incorrect": 15,    "accuracy": 0.85,    "avg_score": 4.2  },  "incorrect_pairs": [    {      "question": "What is RAG?",      "golden_answer": "RAG combines retrieval with generation...",      "rag_answer": "RAG is a database system.",      "score": 2,      "reason": "Factually incorrect - RAG is not a database"    }  ]}

🧪 Python API

from ragscore import run_pipeline, run_evaluation
# Generate QA pairsrun_pipeline(paths=["docs/"], concurrency=10)
# Generate tailored QA pairs for specific audiencesrun_pipeline(    paths=["docs/"],    audience="support engineers",    purpose="fine-tuning a support chatbot",)
# Evaluate RAGresults = run_evaluation(    endpoint="http://localhost:8000/query",    model="gpt-4o",  # LLM for judging)print(f"Accuracy: {results.accuracy:.1%}")

🤖 AI Agent Integration

RAGScore is designed for AI agents and automation:

# Structured CLI with predictable outputragscore generate docs/ --concurrency 5ragscore evaluate http://api/query --output results.json
# Exit codes: 0 = success, 1 = error# JSON output for programmatic parsing

CLI Reference:

⚙️ Configuration

Zero config required. Optional environment variables:

export RAGSCORE_CHUNK_SIZE=512          # Chunk size for documentsexport RAGSCORE_QUESTIONS_PER_CHUNK=5   # QAs per chunkexport RAGSCORE_WORK_DIR=/path/to/dir   # Working directory

🔐 Privacy & Security

Compliance: GDPR ✅ • HIPAA ✅ (with local LLMs) • SOC 2 ✅

🧪 Development

git clone https://github.com/HZYAI/RagScore.gitcd RagScorepip install -e ".[dev,all]"pytest

📡 Telemetry

RAGScore collects telemetry only in MCP server mode (ragscore serve). Standard CLI and Python API usage do not send telemetry.

We collect limited anonymous operational metrics to understand feature usage and improve reliability. No document content, prompts, QA text, model outputs, API keys, endpoint URLs, or file paths are collected.

Collected in MCP mode:

• MCP tool invoked
• LLM provider and model name
• ragscore version, Python version, OS type
• Success/failure status
• Random anonymous installation ID

Opt out:

export RAGSCORE_NO_TELEMETRY=1

�� Links

• GitHub • PyPI • Issues • Discussions

⭐ Star us on GitHub if RAGScore helps you!
Made with ❤️ for the RAG community