level 6Testing & Reliability
Flaky test
also: flaky tests, flaky, flakiness
A test that sometimes passes and sometimes fails even though nothing in the code changed.
a bit more technical
A non-deterministic test whose result varies between runs on the same code, typically due to timing, test order, shared data, real network calls or dates. Flaky tests erode trust in the whole test suite.
picture it
A smoke alarm that goes off at random. After a few false alarms, everyone ignores it, including the time there's a real fire.
why it matters
Agents sometimes "fix" a flaky test by rerunning it until it passes, adding long waits, or deleting it. Knowing the term lets you ask for the real cause instead.
what your agent might say
The checkout test is flaky: about 20% of the time it clicks Pay before the button is enabled. I replaced the fixed half-second wait with waiting for the button to be enabled.
how you might use it
The login test fails randomly. Find out why it's flaky and fix the cause. Don't just add retries or a longer wait.
common mistake
Thinking a flaky test is always the test's fault. Sometimes the randomness reveals a real timing bug in the app that users hit too.
builds on