No. 1 Game Experimentation Mistake
The number one mistake I see when teams evaluate A/B tests is failing to measure the realized treatment size. They jump straight to the top-line KPIs and call the test a success or failure before asking whether the player experience effectively changed via the treatment.
For example, suppose a Match-3 puzzle game adds two moves across every level to make the game easier. That is the assignment, not yet the realized treatment! The hypothesis is that an easier game, with fewer fail screens per level (but more levels played), will increase LTV. The realized treatment is how much easier the game became. Did attempts per success fall from 1.8 to 1.6, or from 1.8 to 1.79? A flat LTV read after a +/- 0.01 change in attempts per success does not mean difficulty is irrelevant.
The same problem appears when a team introduces a new mode. Before interpreting retention or LTV, teams should measure adoption, the mode's share of matches, where that play came from, and the net additional play after cannibalization. If players barely touched the mode, or merely switched from another one, the test says little about the underlying idea.
Model high-impact experiments are a surface-area problem: realized treatment strength × player reach × duration. Changing one level affects only players who reach it, and only for the tiny share of their lifetime spent there. Moving the move limit across every level changes the experience for nearly every player, every time they play. The second treatment has far more leverage over a player's lifecycle, and therefore a plausible path to moving LTV. While that does not automatically make it the right experiment, it is the right way to scope the effect size before committing resources.
Strength matters too. Every system value sits on an LTV response curve. There may be an optimal move count, progression speed, reward amount, or mode allocation, but the curve might be tall & steep or low & flat. Optimizing progression speed probably has a larger effect than optimizing the order of a settings menu. Teams can (and regularly do) find the optimal configuration of a system that barely changes Game LTV.
This is why an A/B test's success is determined before it runs. The team should define the intended change in player experience, measure the realized change, and estimate the intervention's surface area. Only then should it interpret the top-line KPI.
Positive, negative, and flat results are all useful when they determine whether a system deserves more investment. The job of a capital allocator, or a game systems designer, is to direct effort where each dollar produces the highest return. Running the test is a victory when it replaces an argument with evidence that compounds across future decisions. If the result does not become an important piece of evidence, the test was not worth the resources.