Siddharth Gaur

OpenEval — Self-Hosted LLM Observability

Tradeoff · Eval jobs run on Celery, not a request-scoped task: scoring hundreds of rows against a real LLM API takes minutes and has to survive a restart.

A self-hosted LangSmith/Helicone alternative: trace every LLM call, version prompts and datasets, run 24 built-in evaluators (RAGAS-equivalent and DeepEval-backed) against any provider, and compare runs to catch regressions — one docker compose up, no vendor lock-in. Eval jobs run on Celery, not a request-scoped async task, because scoring hundreds of rows against a real LLM API takes minutes and needs to survive a process restart. Every resource — traces, datasets, prompts, eval runs — is scoped to an organisation → project → role from the first migration, because multi-tenancy retrofits are where authorization bugs live. Live on a free tier: the first request after idle can take minutes.

• Python
• FastAPI
• Celery
• PostgreSQL
• Next.js
• Redis
• Docker

DeepResearch — Multi-Agent Research System

Tradeoff · The critic can send gaps back to the planner, but a retry-depth limit stops it researching forever.

Give it a research question and nine cooperating agents — planner, researcher, browser, summariser, fact-checker, critic, code executor, writer, citation validator — plan sub-questions, search the web in parallel, cross-check every claim against at least two sources, and hand back a cited report with a per-claim HIGH/MEDIUM/LOW/UNVERIFIED confidence tag. The fact-checker/critic loop can send gaps back to the planner, bounded by a retry-depth limit so it can’t research forever. LangGraph checkpoints to Postgres after every node, and a crashed worker’s job is redelivered to another replica instead of being lost. Runs end to end on Ollama + DuckDuckGo for $0.

• Python
• LangGraph
• Celery
• FastAPI
• Next.js
• Postgres / pgvector
• Docker

indic-reg-bench — A Benchmark for Indian Regulatory Documents

Tradeoff · The 2,000-order benchmark sample (seed 42) follows the year distribution of all 11,957 listed orders, so no era dominates the score.

An open benchmark for document understanding on SEBI enforcement orders: five tasks, a harness, and a HuggingFace dataset. Everything else here is a system I built; this is an instrument other people measure their systems with. The hard part is designing tasks a regex can’t win — SEBI orders quote the noticee’s own settlement pleas in phrasing identical to the ruling, so the first currency amount disagrees with the regex-located operative paragraph in 48.6% of comparable orders.

• Python
• HuggingFace Datasets
• PDF parsing
• Evaluation
• GitHub Actions

QueryPilot — Natural Language to SQL

Tradeoff · Safety never depends on parsing the SQL: the connection is read-only at the engine and an authorizer denies writes at prepare time.

Ask a database questions in plain English — and it is structurally impossible for the answer to write to, alter, or escape that database. An LLM writes the SQL, but nothing trusts it: every query runs on a connection that is read-only at the SQLite engine level and behind a SQLite authorizer that denies any non-read operation at prepare time, so the guarantee never depends on parsing the SQL text (the part an attacker controls). The linked code is v2 — FastAPI, schema-aware RAG, Postgres, Docker — whose injection benchmark gates every PR in CI.

• Python
• FastAPI
• SQLite
• ChromaDB
• Postgres
• Docker
• Streamlit

ML Platform — Feature Store + Drift Monitor + Fraud Scoring

Tradeoff · Merging three repos replaced three stubs with real calls; one had silently swallowed an ImportError.

A feature store, a drift monitor, and a real-time fraud-scoring service, merged from three separate repos into one that can actually import across the seam. Each stub was real before the merge: the feature store’s drift detection was a documented no-op waiting on a "not-yet-built sibling," the monitor duck-typed its feature-store adapter instead of importing it, and the fraud service wrapped its monitoring hook in a try/except that silently swallowed ImportError. The merge replaces all three with a real call, verified by a cross-subsystem test that could not have been written before — 113 tests total, throughput measured at 29.7 req/s from a committed locustfile, not extrapolated.

• Python
• Kafka (Redpanda)
• Redis
• XGBoost
• IsolationForest
• Postgres
• FastAPI