No. 1 Game Experimentation Mistake
The number one mistake I see when teams evaluate A/B tests is failing to measure the realized treatment. They jump straight to the top-line KPI and call the test a success or failure before asking whether the player experience changed enough to move it.
Suppose a Match-3 puzzle game adds two moves across every level to make the game easier. That is the assignment, not yet the realized treatment! The hypothesis is that an easier game, with fewer fail screens per level (but more levels played), will increase LTV. The realized treatment is how much easier the game became. Did attempts per success fall from 1.8 to 1.6, or from 1.8 to 1.79? Those are entirely different experiments, even if the config diff says "+2 moves" in both. A flat LTV read after a 0.01 change in attempts per success does not mean difficulty is irrelevant.
The same problem appears when a team introduces a new mode. Before interpreting retention or LTV, measure adoption, the mode's share of matches, where that play came from, and the net additional play after cannibalization. If players barely touched the mode, or merely switched from another one, the test says little about the underlying idea.
High-impact experiments are a surface-area problem: realized treatment strength × player reach × duration. Changing one level affects only players who reach it, and only for the tiny share of their lifetime spent there. Moving the move limit across every level changes the experience for nearly every player, repeatedly. The second treatment has far more leverage over a player's lifecycle, and therefore a plausible path to moving LTV. That does not automatically make it the right experiment. It is the right way to scope the effect size before committing resources.
Strength matters too. Every system value sits on an LTV response curve. There may be an optimal move count, progression speed, reward amount, or mode allocation, but the curve can be tall and steep or low and almost flat. Optimizing save progression probably has a larger effect than optimizing the order of a settings menu. A team can find the optimal configuration of a system that barely changes Game LTV.
This is why an A/B test's success is determined before it runs. The team should define the intended change in player experience, measure the realized change, and estimate the intervention's surface area. Only then should it interpret the top-line KPI.
Positive, negative, and flat results are all useful when they determine whether a system deserves more investment. The job of a capital allocator, or a game systems designer, is to direct effort where each dollar produces the highest return. Running the test is a victory when it replaces an argument with evidence that compounds across future decisions. If the result does not become an important piece of evidence, the test was not worth the resources.