StackOverflow2013

Note that there are some explanatory texts on larger screens.

plurals

POHow do you handle unit/regression tests which are expected to fail during development?
text
Body
copied!<p>During software development, there may be bugs in the codebase which are known issues. These bugs will cause the regression/unit tests to fail, if the tests have been written well.</p> <p>There is constant debate in our teams about how failing tests should be managed:</p> <ol> <li><p>Comment out failing test cases with a REVISIT or TODO comment.</p> <ul> <li><strong>Advantage</strong>: We will always know when a <em>new</em> defect has been introduced, and not one we are already aware of.</li> <li><strong>Disadvantage</strong>: May forget to REVISIT the commented-out test case, meaning that the defect could slip through the cracks.</li> </ul></li> <li><p>Leave the test cases failing.</p> <ul> <li><strong>Advantage</strong>: Will not forget to fix the defects, as the script failures will constantly reminding you that a defect is present.</li> <li><strong>Disadvantage</strong>: Difficult to detect when a <em>new</em> defect is introduced, due to failure noise.</li> </ul></li> </ol> <p>I'd like to explore what the best practices are in this regard. Personally, I think a tri-state solution is the best for determining whether a script is passing. For example when you run a script, you could see the following:</p> <ul> <li>Percentage passed: 75%</li> <li>Percentage failed (expected): 20%</li> <li>Percentage failed (unexpected): 5%</li> </ul> <p>You would basically mark any test cases which you <em>expect</em> to fail (due to some defect) with some metadata. This ensures you still see the failure result at the end of the test, but immediately know if there is a <em>new</em> failure which you weren't expecting. This appears to take the best parts of the 2 proposals above.</p> <p>Does anyone have any best practices for managing this?</p>

Querying!

Guidance

An individual column

Larger individual text columns get their own page to allow for proper reading.

SQuiL has stopped working due to an internal error.

If you are curious you may find further information in the browser console, which is accessible through the devtools (F12).

Reload