Benchmark Protocol v2.0

Dataset

Source: NYC TLC Yellow Taxi Trip Records (2022-2025)
Zones: 263 official NYC taxi zones
Splits: Strict chronological (train: 2022-2023, val: 2024, test: 2025)
Format: Parquet files with cleaned trip records

Task Definition

Given a taxi driver’”’“‘s current zone and time of day, recommend the top-3 zones to maximize expected revenue.

Forecasting Sub-task

Predict zone-level pickup demand 30 minutes ahead.

Decision Making Sub-task

Generate ranked zone recommendations.

Offline RL Sub-task

Learn a policy from fixed trajectories to maximize cumulative reward.

Metrics

Forecasting

Metric	Definition	Higher is Better
MAE	Mean Absolute Error of predicted vs actual pickups	No
RMSE	Root Mean Squared Error	No
SMAPE	Symmetric Mean Absolute Percentage Error	No

Decision Making

Metric	Definition	Higher is Better
Revenue/driver	Average daily fare revenue per driver	Yes
Utilization	Fraction of time with passengers	Yes
Demand Coverage	Fraction of demand served	Yes

Offline RL

Metric	Definition	Higher is Better
Episode Return	Cumulative reward per episode	Yes
Return Std	Standard deviation across seeds	No
Stability CV	Coefficient of variation across seeds	No

Baselines

Forecasting

Historical Average (heuristic)
LightGBM (gradient boosting)
XGBoost (gradient boosting)
GraphSAGE (graph neural network)
Ensemble (weighted combination)

Decision Making

Hot Zone (always recommend highest-demand zone)
Single-Step (greedy utility maximization)

RL

DQN (deep Q-network)
Double DQN (double deep Q-network)
IQL (implicit Q-learning, offline)

Evaluation Procedure

Load pre-computed statistics or trained models
Run benchmark-specific evaluation
Compute metrics with bootstrap confidence intervals (2000 resamples)
Generate standardized report

Reproducibility Requirements

All random seeds must be fixed and documented
Config files must be included (see configs/)
Python version and package versions recorded
Experiment manifest generated (see scripts/create_experiment_manifest.py)