Google Research’s MilleMiglia illustrates how realistic scenario generators can expose planning failures that historical examples and aggregate benchmarks miss.
A planning product can perform well on every historical case and still be unready for production. History contains the operating environments that happened, not every environment that could happen. It may contain few depot outages, synchronized delays, unusual demand combinations or newly introduced policies. Worse, historical replay can reward a planner for reproducing yesterday’s compromises rather than handling tomorrow’s constraints.
Google Research published MilleMiglia on September 18, 2026. The open-source generator creates synthetic middle-mile logistics instances spanning small academic problems through large continent-scale networks. Its scenarios represent fixed vehicle schedules, distribution-center throughput, shipment transfers and synchronization over time. The immediate contribution is a logistics benchmark generator. The broader product lesson is that realistic scenario generation can become part of an evaluation system, not merely a source of training records.
That distinction matters as optimization engines and AI agents gain influence over consequential operations. A solution can satisfy its encoded constraints, improve its headline objective and still violate business intent. It might concentrate delays on priority customers, consume all recovery capacity, exploit an unrealistic transfer assumption or generate plans that operators cannot execute. A test suite must therefore ask more than whether a plan is feasible or cheap.
The practical pattern is to define an operational contract, generate controlled families of scenarios, compare candidate planners with simple baselines, inspect intent-level failures, shadow the system and preserve every useful failure as a regression case. This turns synthetic data into release infrastructure. It also gives product, operations and engineering teams a shared way to decide what evidence is sufficient for deployment.
What MilleMiglia changes
Middle-mile logistics is not simply a larger last-mile routing problem. A shipment can move through several distribution centers and vehicles over multiple days. Each transfer depends on time: arriving after a connecting vehicle departs can add a full cycle of delay. Distribution centers have finite sorting or cross-docking throughput, vehicles follow schedules and freight can wait in storage.
MilleMiglia represents this structure as a multi-commodity flow problem on a space-time graph. Nodes correspond to distribution centers at particular time intervals. Arcs represent either vehicle movements or freight being held at a center. Google Research says the generator creates network geography, demand and structured vehicle rotations using statistical distributions informed by public information and privately disclosed data. It emits instances containing the problem’s central constraints in one format. These are claims made in the Google Research publication; the examined material does not report comparative planner results.
The important mechanism is controlled recombination. Instead of publishing one sanitized company network or a small fixed benchmark, a generator can produce many networks with varied size, topology, demand, schedules and capacity pressure. An evaluator can isolate a factor, sweep its severity and reproduce the random seed that produced a failure. That makes the scenario a diagnostic instrument rather than another anonymous row in a test set.
Test generation is not training-data synthesis
Synthetic data is often discussed as a way to train a model when private or labelled records are scarce. MilleMiglia can support learning scenarios, according to Google Research, but its more immediately transferable use is independent evaluation. The distinction is procedural, not merely semantic.
The same generator may support both uses, but the records, seeds and governance should remain separate. Primary purpose Improve a model or policy by supplying learning examples Measure behavior under defined operating conditions Who sees the cases? The training process may consume them directly Final holdout families remain unavailable to planner development Desired coverage Representative and useful for learning Representative conditions plus deliberate boundaries, rare combinations and adversarial cases Main leakage risk Synthetic artifacts distort the learned policy The planner or team tunes to known seeds and scenario templates Required output A trained artifact A reproducible evidence record with scenario version, seed, planner version and verdict Failure handling May trigger data or model changes Creates a regression case when the failure is operationally meaningful
If a planner trains on every generated pattern and is then evaluated on near-identical instances, a large score can reflect generator familiarity. Keep separate generator configurations for development, pre-release qualification and post-incident regression. Where possible, vary the causal structure as well as numeric values. Changing every demand volume by ten percent is weaker than introducing a hub closure that alters which transfers remain possible.
Write the operational contract first
A scenario generator cannot rescue an underspecified product. Before generating anything, translate operational intent into a testable contract. The contract should separate hard feasibility, optimization objectives, policy rules and resilience requirements.
Hard constraints Conditions no released plan may violate, such as vehicle capacity, shipment compatibility, facility throughput, transfer timing and legal limits. Objective terms Quantities the planner trades off, such as transport cost, late deliveries, handling operations, emissions proxies or unserved demand. Record units and weights. Business-intent invariants Rules that may not appear in the mathematical model but define acceptable conduct, such as protecting medical shipments, avoiding systematic harm to a region or requiring human review for exceptional rerouting. Operational usability Requirements governing plan stability, explanation, execution lead time and the amount of manual correction expected from operators. Recovery margin Capacity or schedule slack that must remain after optimization so an ordinary disruption does not immediately make the plan infeasible.
A route optimization service can only optimize against the objectives and constraints supplied to it, as the Google Maps Platform documentation makes explicit. This is not a defect unique to one service. It is a general property of formal optimization. The planner searches the problem the product team encoded, while operators live with the larger problem the encoding approximates.
This is one reason deployment deserves a different standard from construction. The difference between building and deploying AI is not that the algorithm suddenly changes. Deployment introduces live incentives, incomplete specifications, recovery costs and people who must act on the output.
Build scenario families, not random piles
A thousand randomly generated tests can provide less information than fifty organized tests. Treat each scenario family as a hypothesis about how the system could fail. Give the family a named variable, a realistic range and an expected behavioral response.
Establish a calibrated core. Generate ordinary networks whose aggregate properties resemble the intended operating domain: facility density, shipment mix, route frequency, transfer count and utilization. Compare these distributions with authorized production summaries, not individual private records. Vary one pressure at a time. Sweep demand, throughput, delay, vehicle capacity or connection frequency across a defined range. This reveals thresholds where behavior changes sharply. Combine pressures. Create interactions that historical data rarely contains, such as elevated demand plus a hub outage plus reduced overnight capacity. Many planning failures emerge from combinations rather than extreme values of one variable. Add structural novelty. Test a new region, sparse network, asymmetric flows, new service class or facility topology. Parameter noise does not establish that a planner can generalize to a different graph. Generate intent traps. Construct cases where the easiest numerical improvement conflicts with an operator policy. Examples include sacrificing one low-volume region, using nearly impossible transfer times or exhausting all contingency capacity. Freeze qualification suites. Version the generator, configuration and seeds. A release gate must run against stable holdouts so score movement can be attributed to the planner rather than an unnoticed test change.
Every family needs an oracle or review procedure. Some checks are exact: no capacity violation, no shipment appearing before arrival and no throughput above its limit. Others are comparative: lower cost than a baseline under equivalent service. Intent-level checks may require thresholds, segment analysis or operator review. Label these categories so a subjective policy decision is not presented as mathematical ground truth.
Illustrative worked example: qualifying a planner
A fictional middle-mile release gate The companies, figures and results below are hypothetical. They demonstrate an evaluation design and are not reported MilleMiglia results.
Suppose a retailer is evaluating a new planner for a seven-day network containing 14 distribution centers, 96 scheduled vehicle movements and 3,200 shipments. Shipments have standard, priority and temperature-controlled classes. The product objective minimizes transport and holding cost plus penalties for lateness and unserved shipments.
The team writes six hard rules: vehicle and facility capacities cannot be exceeded; a shipment can board only after it arrives and completes handling; temperature-controlled freight may use only compatible lanes; priority freight must arrive within 30 hours; no shipment may disappear; and every exceptional unscheduled movement requires review. It adds three intent checks: no region may have more than twice the network-wide late-delivery rate, at least 8 percent of outbound capacity must remain as recovery margin, and plan churn after a small demand update must stay below 15 percent of assignments.
The evaluator generates four families with 40 instances each. The ordinary family targets 60 to 75 percent facility utilization. The capacity family raises one major hub from 70 to 105 percent demand pressure. The disruption family delays selected inbound vehicles by 30 to 180 minutes. The interaction family combines a two-hour hub outage, 20 percent elevated priority demand and one cancelled overnight movement. Each instance stores its seed and the variables applied.
Three planners run under the same compute budget. Baseline A sends each shipment through the cheapest feasible path in arrival order. Baseline B uses the currently deployed planning heuristic. Candidate C is the new optimizer. The evaluator records feasibility, weighted objective, late shipments by class and region, recovery margin, plan churn, runtime and manual-review triggers.
Aggregate means can conceal an intent violation, so the table includes a worst-segment measure. Hard-feasible instances 149 of 160 158 of 160 160 of 160 Relative operating cost 1.08 1.00 0.94 Priority shipments late 4.7% 2.1% 1.6% Worst regional late rate 9.8% 5.0% 8.7% Median recovery capacity retained 11% 9% 3% Interaction-family plan churn 12% 14% 31%
Candidate C appears best if the release decision uses feasibility, average cost and overall priority lateness. It is nevertheless unacceptable under the stated contract. Inspection of the failed scenarios shows that it protects the global objective by routing flexible freight away from a peripheral region and filling nearly every remaining departure. The plans are mathematically feasible, but regional service deteriorates, recovery margin disappears and minor updates cause extensive reassignment.
The correct response is not to discard the generator or celebrate the average. The team should preserve the revealing seeds, classify the failures and decide which requirement belongs in the optimization model versus an external validator. Regional service may become a constraint or penalty. Recovery margin may become reserved capacity. Churn may require a stability term. Candidate C should then be rerun against both the frozen suite and new holdouts, because repairing known cases alone invites overfitting.
This is the optimization version of an agent succeeding the wrong way. The system reaches the encoded target while violating the operator’s actual intent. A complementary misalignment incident process helps distinguish a bad objective, a missing authorization boundary, faulty tool behavior and inadequate oversight.
Turn results into release gates
An evaluation suite becomes infrastructure when it can block a release for an explainable reason. Define gates before reviewing the candidate’s results. Otherwise, teams tend to rationalize whichever metrics improved.
A practical release policy Require zero violations of machine-checkable hard constraints across the frozen qualification suite. Permit no operationally material regression against the production baseline on protected service segments. Require a documented primary-objective improvement under equal data, compute, timeout and stopping rules. Evaluate degradation curves as demand, delays and outages intensify, then require operator disposition of every high-severity intent failure. Store reproducible artifacts containing generator version, configuration, seed, planner build, timeout and validator version.
Baselines should include more than the strongest available planner. A simple heuristic tests whether sophistication provides meaningful value. The incumbent establishes the cost of change. A deliberately conservative policy can reveal whether the candidate’s gains come from consuming safety margin. Run all planners under comparable data access, hardware budgets and stopping rules.
Evaluation must also cover the surrounding product: input validation, timeouts, fallback behavior, observability and human intervention. These concerns often differentiate an impressive component from a dependable system. The framework-selection questions in How I evaluate AI agent frameworks apply here too: integration and operational control matter alongside the intelligence of the core planner.
Shadow deployment and regression memory
Generated scenarios cannot reproduce every feature of a live operation. After offline qualification, run the candidate in shadow mode: feed it current inputs, prevent it from controlling operations and compare its proposed plans with actual decisions. Record whether its inputs were available at decision time. A comparison contaminated with future information creates false confidence.
Shadowing should answer three questions. Does production input differ from the generator’s calibrated distributions? Do operators reject plans for reasons absent from the contract? Does the candidate remain stable when data arrives late, changes or contains errors? These findings should update the scenario catalogue, but production data should not automatically overwrite deliberate stress cases.
Every consequential failure becomes durable evaluation memory. Preserve the smallest reproducible scenario where possible, the original full scenario, expected invariants and the failure classification. Add a test proving that the repaired planner no longer fails, plus nearby variants showing that the repair is not a narrow patch. Retirement should require an explicit reason, such as a policy or network change, rather than a desire to shorten the suite.
Limits of realistic generators
Realistic is not identical to real Matching selected distributions does not prove that generated networks reproduce every dependency, institutional practice or human workaround in production. The generator has a specification gap too If it omits labor rules, weather correlations, data latency or informal facility practices, planners will not be tested against them. Rare is not the same as arbitrary Stress tests should have a plausible mechanism. Impossible scenarios can still test defensive behavior, but they should be labelled separately from operating forecasts. A validator can share the planner’s mistake If both rely on the same incorrect capacity interpretation, a plan may pass automatically. Independent checks and operator review reduce correlated failure. Privacy claims need local review Synthetic records are not automatically non-sensitive. Teams should assess whether calibration inputs, unusual topology or low-frequency combinations reveal protected information. Passing does not prove future safety A suite provides bounded evidence over examined scenarios and checks. It cannot establish correctness across every future network, policy and disruption.
The Google Research publication describes MilleMiglia as privacy-preserving and realistic, and explains the distributions and constraints it incorporates. In the supplied material, it does not provide an independent privacy analysis, production deployment study or comparative solver benchmark. That does not diminish the generator’s value as a research building block, but it limits what can be concluded from the publication alone.
Use this next
Choose one consequential decision. Start with a bounded planner output such as assignment, scheduling, inventory transfer or resource allocation. Identify who bears the cost when it fails. Write five invariants and three intent checks. Do this with operators before discussing model architecture. Include at least one service-distribution check and one resilience check. Create four named scenario families. Use ordinary load, one-variable pressure, interacting disruption and structural novelty. Specify ranges and expected behavior before generating instances. Run three planner classes. Compare a simple baseline, the incumbent and the candidate on feasibility, objective quality, segment outcomes, stability, runtime and recovery margin under equal conditions. Make failure retention automatic. A failed seed, configuration and verdict should enter a review queue. Accepted failures become versioned regression tests. Shadow before granting control. Compare proposals against live decisions, capture operator rejection reasons and promote only after predefined offline and shadow gates pass.
The decision is not whether synthetic scenarios are perfectly realistic. They will not be. The useful question is whether they expose important behavior earlier, more reproducibly and with better coverage than historical replay alone. For planning products, a scenario generator should sit beside validators, baselines, shadow runs and incident review. Together, those components create an evaluation system capable of finding not only infeasible plans, but also plans that succeed according to the metric and fail according to the operation.
Subscribe free to Harshith's Newsletter to read every article in the interactive edition.
Harshith Vaddiparthy works with founders, operators, and teams on practical AI products, workflows, advisory, training, and mentorship. This no-JavaScript version preserves the page's core information and navigation.