What they're testing
Whether you treat suite health as something to be managed, and know the common causes of flakiness.
The short answer~30 seconds
Measure first: tag every run and compute a per-test failure rate, so you have a ranked list rather than an impression. Then quarantine the worst offenders out of the PR-blocking path — not to ignore them but so that a red CI means something again. Then fix in order, with a deadline: a flaky test not fixed within two weeks gets deleted. A test nobody believes has cost and no value.
The long answer
Flakiness clusters into four causes. Timing: a fixed sleep instead of waiting for a condition, or a test that depends on when it runs. Ordering: tests sharing state, so they pass alone and fail together — randomising the run order exposes this immediately. Concurrency: two tests using the same database row. And environment: network dependence, timezone, the runner's locale.
The timing group deserves its own note because the fix is nearly always the same: replace sleep(500) with a bounded conditional wait (waitFor until the element appears, capped at five seconds). It's both faster in the common case and more stable when the CI machine is slow — and CI machines are always slow exactly when you don't expect it.
The hard part isn't fixing them but keeping them fixed. If nobody owns suite health, the flake rate creeps back within months. What I've seen work is making it a visible metric alongside CI duration, and treating a newly flaky test as a bug to fix rather than an annoyance to tolerate.
What they'll ask next
?Is auto-retrying tests ever acceptable?
Only as a recorded stopgap, never a solution. The problem is that retries make a flaky test invisible, so you lose the signal that would fix it. If you enable retries, log and report the retry count — otherwise you're just hiding it.
These lose points
- Adding a longer
sleepuntil it stops flaking. You've made CI slower and the problem is still there. - Accepting flakiness as normal. From then on every red is ignored, including the real one.