← Siddharth GaurResume

ML Platform: feature store, drift monitor and fraud scoring

A feature store, a drift monitor and a real-time fraud-scoring service, merged from three repositories into one that can import across the seam.

Terminal output of the ml-platform drift demo: no drift on in-distribution data, then injected drift on amount and merchant risk score flagged as critical.
Drift demo: injected drift caught and flagged critical.

Problem

  • The features you train on, the features you serve and the distribution you compare live traffic against have to agree. In separate codebases they drift apart silently.
  • The seams showed the cost: drift detection was a documented stub, the monitor duck-typed its feature-store adapter, and the fraud service wrapped its monitoring import in a try/except that returned None, so a whole backend could silently do nothing.

Approach

  • Merging replaced all three: the stub is a real call, the adapter a real import, and the try/except is gone.
  • The feature schema is imported from the fraud subsystem rather than restated, because a copied schema is where training/serving skew starts.
  • One end-to-end test crosses all three subsystem boundaries. It could not have been written before the merge.

What failed, and the tradeoffs

  • The merge eliminated almost no duplicate code: file similarity across the repos was 0.01–0.21. The case for merging was the interfaces, not deduplication.
  • The fraud data is synthetic, and the injected fraud is what the features are built to detect. Its metrics measure the pipeline, not the modelling.
  • The fraud service is not deployed and has no public endpoint. Managed Kafka is out of budget. Paths in deploy/fraud still reference the absorbed layout, so docker compose up doesn’t work unchanged.
  • The drift detector has no multiple-comparison correction: with 11 features at α=0.05, a clean run flags at least one about 4 times in 10.
  • Inherited lint findings are ignored in CI rather than fixed, so CI reports regressions instead of staying permanently red.

Result

  • 113 tests passing: 25 monitoring, 23 features, 60 fraud and 5 new cross-subsystem.
  • 29.7 req/s at a median of 1400 ms and a p95 of 2300 ms, measured on a laptop from a committed locustfile, with the bottleneck identified: uvicorn running without --workers.

Stack

  • Python
  • Kafka (Redpanda)
  • Redis
  • XGBoost
  • IsolationForest
  • Postgres
  • FastAPI

Links

Next: OpenEval: self-hosted LLM observability →