Add model performance monitoring and drift detection

[stabrea]

Summary

Implement monitoring to track model performance degradation over time and detect data drift that signals when retraining is needed.

Motivation

• Fraud patterns evolve constantly; a model trained today degrades within weeks
• Data drift (changing transaction distributions) is the Add support for real-world datasets (Kaggle IEEE-CIS, Credit Card Fraud) #1 cause of silent model failure
• Production ML requires automated alerting when performance drops below thresholds

Proposed Approach

1. Performance tracking:

   • Log prediction outcomes and compare against confirmed labels
   • Track precision, recall, F1, and AUC-PR over rolling windows
   • Configurable alert thresholds (e.g., recall drops below 0.85)

2. Data drift detection:

   • Population Stability Index (PSI) for feature distribution shifts
   • Kolmogorov-Smirnov test for continuous features
   • Chi-squared test for categorical features
   • Dashboard showing drift scores per feature over time

3. Concept drift detection:

   • Track prediction confidence distribution changes
   • Monitor false positive/negative rate trends
   • ADWIN or Page-Hinkley drift detection algorithms

4. Alerting and reporting:

   • Weekly performance summary report (markdown/HTML)
   • Configurable webhook alerts for drift above threshold
   • Retraining recommendation with suggested data window

Acceptance Criteria

• Performance metrics tracked over configurable time windows
• PSI-based data drift detection for all input features
• Automated drift alerts with configurable thresholds
• Summary report generation with visualizations