Leaderboard

Open Urban Mobility Benchmark — Public Leaderboard

Evaluation type: SIMULATOR / HISTORICAL-REPLAY only. These are not production revenue estimates and no real-world A/B results are reported.

Regenerated: 2026-08-08 09:26 UTC by python benchmark/run.py --leaderboard

Scope & Honesty Statement

All numbers below are extracted from checked-in benchmark artifacts in outputs/.
Endpoints are not comparable across rows: NDCG@3/Hit@3/Utility@1 come from a 3,360-query static diagnostic; Daily Fare from a 100-seed legacy single-driver rollout; Revenue/Driver + Utilization from the finite-demand multi-agent simulator. Do not rank across endpoints.
No production revenue, deployment, or real-world A/B evidence exists in this repository.

Model	Endpoint	NDCG@3	Hit@3	Utility@1	Daily Fare	Revenue/Driver	Utilization
Hot Zone (`hot_zone`)	reference_metrics (static + 100-seed rollout)	0.7846	0.5842	19.43	$431.21	—	—
Single-Step (`single_step`)	reference_metrics (static + 100-seed rollout)	0.9024	0.8804	25.06	$548.77	—	—
Two-Step Horizon (`two_step`)	reference_metrics (static + 100-seed rollout)	0.9565	0.9714	27.59	$570.61	—	—
Hot Zone (`hot_zone`)	multi_agent_benchmark (finite-demand, 30 runs)	—	—	—	—	$1233.41	0.0731
Single-Step (`single_step`)	multi_agent_benchmark (finite-demand, 30 runs)	—	—	—	—	$1764.56	0.1111
Two-Step Horizon (`two_step`)	multi_agent_benchmark (finite-demand, 30 runs)	—	—	—	—	$1508.71	0.0951
DQN (`dqn`)	rl_benchmark (multi-seed)	—	—	—	—	$1821.77	0.1121
Double DQN (`double_dqn`)	rl_benchmark (multi-seed)	—	—	—	—	$1742.77	0.1069
Finite Horizon (`finite_horizon`)	rl_benchmark (multi-seed)	—	—	—	—	$1511.16	0.0952

Model	MAE	RMSE
Historical Average (`historical_average`)	1.7273	5.9237
LightGBM (`lightgbm`)	1.5114	5.0707
Ensemble (LGB+XGB) (`ensemble`)	1.4868	—
GraphSAGE (`graphsage`)	1.5037	—
GAT (`gat`)	1.5058	—
OD Messages (`od_messages`)	1.5024	—

No external submissions yet. See docs/external_contribution.md to submit a model.

python benchmark/run.py --leaderboard