Skip to content
Agentmart
Submit a tool

RAGScore

Generate QA datasets & evaluate RAG systems with failure diagnosis.

Connector by HZYAI · GitHub account verified by the MCP Registry

IndexedApache-2.0Local (stdio)★ 15
Write a review

Install

Works with Claude Code, Codex, VS Code, Cursor, Windsurf, Claude Desktop. Read the repository before you give any tool access to your data.

npx agentmarket install connector/hzyai-ragscore

Asks which of your tools to add it to (Claude Code, Codex, Cursor, VS Code, Windsurf, Gemini CLI, Claude Desktop) and shows the change first. See https://agentmart.dev/cli.

Get the weekly diff

The weekly diff: one email a week with new agent tools, notable updates and grade changes, plus security alerts for tools you follow. Double opt-in, one-click unsubscribe.

We use your email only for this newsletter. Privacy

About

Generate QA datasets & evaluate RAG systems with failure diagnosis. Any LLM.

Reviews

Write a review

No reviews yet. Installed it? Reviews come from verified installers only. How reviews work

Permissions

What this server can do and what it asks for, from its tool list and install config. Checked Oct 25, 2026.

Tools
2 read · 0 write · 0 delete · 2 not yet classified
Secrets
2 (API keys, tokens or passwords)
Install scripts
Not scanned yet
Runs as
Python package

Tools by what they do 4

Read from the source code; the running server may list different tools.

Read 2 · Look things up without changing anything.

  • evaluate_rag
  • get_corrections

Not yet classified 2 · We could not tell from the name and description.

  • generate_qa_dataset
  • quick_test_rag

Secrets and settings it asks for

Secrets: values you paste in that let it act as you.

  • ANTHROPIC_API_KEY
  • OPENAI_API_KEY

Files it touches

  • Any path the agent passes to its tools

Tools (4)

Read from the source code; the running server may differ. Descriptions are the server’s own words; hidden characters are shown as [U+…].

  • evaluate_rag

    Evaluate a RAG API endpoint against a QA dataset. Queries the RAG endpoint with questions and scores the answers using LLM-as-judge (1-5 scale). Args: endpoint: RAG API endpoint URL (e.g., http://localhost:8000/query) dataset_path: Path to QA dataset JSONL (default: output/generated_qas.jsonl) concurrency: Max concurrent requests (default: 5) provider: LLM provider for judging (openai, anthropic, ollama). Auto-detected if not set. model: LLM model for judging (e.g. gpt-4o-mini). Uses provider default if not set. detailed: Enable multi-metric evaluation (correctness, c…

  • generate_qa_dataset

    Generate QA pairs from documents for RAG evaluation or synthetic data. Scans documents (PDF, TXT, MD) and generates question-answer pairs that can be used to test RAG systems or as training data. Args: path: Path to file or directory containing documents concurrency: Max concurrent LLM calls (default: 5) num_questions: Number of QA pairs to generate per chunk (default: 5) provider: LLM provider (openai, anthropic, ollama). Auto-detected if not set. model: LLM model name (e.g. gpt-4o-mini, claude-3-haiku). Uses provider default if not set. audience: Target audience (e.…

  • get_corrections

    Get corrections from the last quick test for RAG improvement. Returns incorrect QA pairs that can be injected into the RAG system to improve accuracy. Args: output_path: Optional path to save corrections JSONL Returns: JSON array of corrections or path to saved file

  • quick_test_rag

    Quick RAG accuracy test - generate QAs and evaluate in one call. Perfect for rapid iteration and sanity checks. Use 'golden' to reuse pre-generated QA pairs (saves LLM cost, deterministic). Args: endpoint: RAG API endpoint URL docs_path: Path to documents to generate test questions from. Required unless 'golden' is provided. num_questions: Number of test questions (default: 10) threshold: Pass/fail accuracy threshold (default: 0.7 = 70%) provider: LLM provider (openai, anthropic, ollama). Auto-detected if not set. model: LLM model name. Uses provider default if not se…

Signals

Stars
15
Forks
2
Open issues
0
Weekly downloads
32
Last commit
May 25, 2026 (5 months ago)
Last release
May 19, 2026 (5 months ago)
Last package publish
May 25, 2026 (5 months ago)

From GitHub (fetched Oct 24, 2026) and the package registry.

Details

Registry name
io.github.HZYAI/ragscore
Version
0.8.6
Auth
API key
Packages
pypi ragscore
License
Apache-2.0
Repository
github.com/HZYAI/RagScore
First seen
Oct 24, 2026
Last checked
Oct 24, 2026
Source
Official MCP Registrygithub-graphql
Cite this page

RAGScore MCP server. Agent Mart. https://agentmart.dev/connectors/hzyai-ragscore (last updated 2026-10-24)