# Evaluation Protocol

## For Research (Current)

All evaluation is **offline** using historical data:

1. **Static diagnostic**: NDCG@3, Hit@3 on validation queries
2. **Rollout simulation**: Single-agent and multi-agent simulators
3. **Historical replay**: Compare against actual trip outcomes
4. **Shadow evaluation**: Record recommendations without execution
5. **A/B testing**: Statistical comparison (simulation only)

All results are clearly labeled with their source:
- `SIMULATION` — Multi-agent simulator
- `HISTORICAL_REPLAY` — Historical data replay
- `SHADOW` — Shadow evaluation
- `REAL A/B` — Not yet available

## For Pilot Deployment (Future)

### Phase 1: Shadow Evaluation

1. Deploy recommendation API in shadow mode
2. Record vehicle positions and current zone from telemetry
3. Generate recommendations WITHOUT showing them to drivers
4. Compare AI recommendations against actual driver decisions
5. Evaluate after 2-4 weeks of data collection

### Phase 2: Controlled A/B Test

1. Randomly assign drivers to control (Hot Zone) and treatment (Two-Step) groups
2. Show recommendations only to treatment group
3. Track: revenue, utilization, empty distance, trip completion rate
4. Run for minimum 4 weeks (to capture day-of-week and weather effects)
5. Use bootstrap paired comparison for statistical analysis
6. Monitor for spillover effects between groups

### Phase 3: Full Rollout

1. Deploy to all consenting drivers
2. Continuous shadow evaluation against non-participating drivers
3. Monitor for policy degradation as adoption increases
4. A/B test policy variants (different models, different constraints)

## Metrics

### Primary

| Metric | Definition |
|---|---|
| Revenue per vehicle-hour | Total fare / total vehicle-hours |
| Utilization | Time with passenger / total time |
| Empty distance | Distance driven without passenger |

### Secondary

| Metric | Definition |
|---|---|
| Trip completion rate | Completed trips / trip opportunities |
| Recommendation acceptance | Times driver followed AI recommendation |
| Zone exposure | Distribution of recommended zones |
| Market saturation | % of time a zone has more drivers than demand |

## Statistical Rigor

- Bootstrap confidence intervals (2000 resamples, 95% CI)
- Paired comparison (same seeds, same demand realizations)
- Effect size (Cohen's d)
- Multiple comparison correction (Bonferroni for >2 metrics)

## Important Caveats

1. **No real-world validation exists** — All current metrics are simulation-based
2. **Simulator omits key dynamics** — No congestion, airport queues, driver adaptation
3. **Recommendation ≠ execution** — Drivers may ignore AI recommendations
4. **Market equilibrium unknown** — Mass adoption may degrade policy performance