OpenEval: self-hosted LLM observability
A self-hosted LangSmith/Helicone alternative: trace every LLM call, version prompts and datasets, run evals against any provider, and compare runs to catch regressions. One docker compose up, no vendor lock-in.
Problem
- Hosted LLM observability tools tie traces, prompts and evals to one vendor and one bill.
- An eval platform that quietly bills your OpenAI account on docker compose up is a bad first impression.
Approach
- LiteLLM wherever a model is called (evals, SDK tracing, a zero-code proxy): the provider is a config string, not a code path.
- The judge model defaults to a local Ollama model; paid providers only when you supply your own key.
- Eval runs on Celery, not a request-scoped asyncio task: hundreds of rows against a real LLM API take minutes and have to survive an API restart and scale independently.
- SSE for eval progress, not WebSockets: progress only flows server to client.
- Every resource (traces, datasets, prompts, eval runs, experiments, annotations) is scoped organisation → project → role from the first migration, because multi-tenancy retrofits are where authorization bugs live.
What failed, and the tradeoffs
- RAGAS was the original metric library and did not survive: ragas 0.4.x hard-imports a module removed from langchain-community 0.4, while litellm/instructor need openai>=2.20, and no combination of versions resolves both. DeepEval covers the same ground, including RAGAS-equivalent contextual precision/recall.
- The two metrics with no DeepEval equivalent (context_entity_recall, noise_robustness) are GEval rubrics matching RAGAS’s published definitions, rather than being skipped.
- Stubbed, not production-hardened: the Kubernetes manifests (secrets are a template; the worker HPA needs KEDA) and a dependency-free line diff and JSON forms instead of Monaco.
- The GitHub Action posts scores on a pull request but has no failure threshold, so it reports regressions rather than blocking them. In this repo it is skipped until a backend is configured.
- Live on a free tier: an idle backend took between 1.5 and 4 minutes to answer when measured.
Result
- 24 built-in metrics, several backed by real RAGAS-equivalent / DeepEval implementations, plus a custom-metric hook.
- Comparing two runs returns metric deltas, per-row diffs and Welch’s t-test significance, and webhooks can fire on a detected regression.
- Experiments with significance testing, an annotation queue with Cohen’s kappa, scoped API keys, Prometheus + Grafana self-monitoring.
Stack
- Python
- FastAPI
- Celery
- PostgreSQL
- Redis
- Next.js
- LiteLLM
- DeepEval
- Docker