SaaB - Simulation as a Benchmark

AI benchmarking is a scoreboard industry without any solid base. A model gains two more "points", the chart changes and a leap toward intelligence is born.

Static tests face documented problems with contamination and saturation. But the deeper failure is conceptual, we keep treating success on isolated static questions as evidence of competence in situations that unfold. A correct answer in a static, predictable environment cannot establish anything about the capabilities of an LLM. So, I can’t find any meaning in this kind of leaderboard.

SaaB (Simulation as a Benchmark) makes consequences part of the test. Place a model inside a world, give it limited resources, incomplete information, competing demands, and other actors, and let its decisions change what happens next. Each turn inherits the damage, the opportunity, and the obligations created by the previous one.

UNIMATRIx explores this idea through simulated societies, environments where agents make complex decisions that can preserve or destroy the legacy. The ambition is to evaluate competence across a sustained sequence of actions, under defined and comparable conditions. Simulation needs a scientific approach to find good recipes and compare them. A badly designed world gives a bad benchmark. UNIMATRIx’s initial benchmarks are experimental; their validity must be demonstrated. The standard should include varied scenarios, inspectable behavior, and evidence that measured success means something beyond exploiting the simulation.