ML Platform: feature store, drift monitor and fraud scoring
A feature store, a drift monitor and a real-time fraud-scoring service, merged from three repositories into one that can import across the seam.
Problem
- The features you train on, the features you serve and the distribution you compare live traffic against have to agree. In separate codebases they drift apart silently.
- The seams showed the cost: drift detection was a documented stub, the monitor duck-typed its feature-store adapter, and the fraud service wrapped its monitoring import in a try/except that returned None, so a whole backend could silently do nothing.
Approach
- Merging replaced all three: the stub is a real call, the adapter a real import, and the try/except is gone.
- The feature schema is imported from the fraud subsystem rather than restated, because a copied schema is where training/serving skew starts.
- One end-to-end test crosses all three subsystem boundaries. It could not have been written before the merge.
What failed, and the tradeoffs
- The merge eliminated almost no duplicate code: file similarity across the repos was 0.01–0.21. The case for merging was the interfaces, not deduplication.
- The fraud data is synthetic, and the injected fraud is what the features are built to detect. Its metrics measure the pipeline, not the modelling.
- The fraud service is not deployed and has no public endpoint. Managed Kafka is out of budget. Paths in deploy/fraud still reference the absorbed layout, so docker compose up doesn’t work unchanged.
- The drift detector has no multiple-comparison correction: with 11 features at α=0.05, a clean run flags at least one about 4 times in 10.
- Inherited lint findings are ignored in CI rather than fixed, so CI reports regressions instead of staying permanently red.
Result
- 113 tests passing: 25 monitoring, 23 features, 60 fraud and 5 new cross-subsystem.
- 29.7 req/s at a median of 1400 ms and a p95 of 2300 ms, measured on a laptop from a committed locustfile, with the bottleneck identified: uvicorn running without --workers.
Stack
- Python
- Kafka (Redpanda)
- Redis
- XGBoost
- IsolationForest
- Postgres
- FastAPI