Testing World Models on Actions They Have Never Seen: Held-Out Plans, Forecast Accuracy and Price Rules in Retail
Better forecasts did not reliably identify which world models could handle unfamiliar price and promotion plans.
Across 2,853 fitted models, forecast accuracy helped compare model types but was a poor guide between versions of the same world model; when merchants planned into high-demand weeks, the relationship reversed. Training aimed at learning action effects left errors on unseen plans unchanged. Adding regular-price information and three price rules helped in generated retail worlds, though a simpler log-linear model won when it could match the true response curve. On real retail data, the rules reduced errors on unusually deep price cuts in one dataset, delivered a change within chance in another, and did not help with lasting price moves.
ArXiv · AI/CL/LG's note
Across 2,853 fitted models, forecast accuracy helped compare model types but was a poor guide between versions of the same world model; when merchants planned into high-demand weeks, the relationship reversed. Training aimed at learning action effects left errors on unseen plans unchanged. Adding regular-price information and three price rules helped in generated retail worlds, though a simpler log-linear model won when it could match the true response curve. On real retail data, the rules reduced errors on unusually deep price cuts in one dataset, delivered a change within chance in another, and did not help with lasting price moves.
ArXiv · AI/CL/LG's note
score 5