EvalCore

What is EvalCore

EvalCore is an open-source, offline snapshot testing tool for AI behavior. It records how your LLM app or agent behaves during a live run, then replays that recording in CI on every change. The replay is deterministic, requires no network or API keys, and costs $0. It is a single binary that works with any language and supports OpenAI-compatible models, REST APIs, shell commands, and OTel/OpenInference traces.

How to use EvalCore

1. Describe the suite as data: Create a YAML file (e.g., evals.yaml) that defines targets (your app), datasets (JSONL files with test cases), and scorers (what "good" means, e.g., contains or judge).
2. Run it once, live: Execute evalcore run evals.yaml against the real model. This records every request and response to a local SQLite cassette, keyed on a hash of the canonical request.
3. Replay in CI: On every PR, run evalcore run evals.yaml --cache replay --baseline main. The cassette is replayed offline, producing identical verdicts. CI gates on the exit code.

Features of EvalCore

• Record / replay: Model calls are recorded to a local SQLite cassette; CI replays byte-for-byte with no network, no keys, no flaky judges.
• Trials: Run multiple trials per case for statistical significance.
• Compare models: Easily compare different models or configurations.
• Agent traces: Support for agent traces via OTel/OpenInference.
• Gates & baselines: Use --baseline main to fail only on regressions; gates enforce pass rate thresholds.
• Reports & history: Generate reports and track history over time.
• Cost tracking: See token usage and cost per run.
• Any language: Works with any language that speaks HTTP or shell.
• No telemetry, no signup, no server: Fully offline, privacy-respecting.

Use Cases of EvalCore

• CI/CD for AI: Automatically test every prompt change, model swap, or dependency update in CI to catch regressions before users do.
• Offline testing: Run evaluations without network access or API keys, making tests fast and deterministic.
• Model comparison: Compare different models or configurations side-by-side using the same dataset.
• Agent evaluation: Record and replay agent traces to ensure consistent behavior.

Pricing

EvalCore is free and open source under the Apache-2.0 license. It costs $0 in CI and has no telemetry, no signup, and no server.