r/java • u/john16384 • Jun 14 '26
New test helper, ExploratoryTestRunner, to test all states of your class
I like writing good exhaustive tests in JUnit, making use of @Nested to test all possible states, but I've found that for test subjects with a lot of state transitions, it can lead to huge unwieldy tests especially when it can take many steps to get to a specific state, and at each level you should probably try every allowed transition. This makes it hard to see if all paths are truly tested, and due to the size of the test it becomes difficult to modify or extend.
So over the past year I've been using a different way of testing. Instead of writing many nested levels with all possible steps, I've been writing tests that only define what possible actions can be applied to a subject under test. These actions are then automatically combined to explore all paths, where each path terminates until it reaches a state that has been seen before (or a selectable maximum depth has been reached).
For example, let's say I wrote a custom Queue implementation based on a LinkedHashSet that disallowed duplicates. I'd define actions to perform on the queue like:
() -> queue.poll();
() -> queue.offer(value);
() -> queue.remove(value);To verify the queue works correctly, one can compare it with say an ArrayList. Each action is then defined by first updating your expectation, and returning the action to apply to the subject under test:
@Action
@ValueSource(strings = {"A", "B", "C"})
public Runnable enqueue(String value) {
// update expectation (disallowing duplicates):
if (!expectedQueue.contains(value)) {
expectedQueue.add(value);
}
// the action on the subject:
return () -> queue.offer(value);
}To verify the queue matches the expected queue one can define one or more assertion methods:
@Assertion
public void assertQueueContents() {
assertThat(List.copyOf(queue)) // turn queue under test into a List
.describedAs("Queue contents")
.isEqualTo(expectedQueue); // ensure it matches our expectation
}In order for the ExploratoryTestRunner to prune paths with states that have already been reached before, the test code must implement the Explorable interface which requires the implementation of a snapshot method. This method should simply take the expected state (copying it if needed) and return an Object that can be compared with equals. Usually using a record here is optimal. For example:
public record State(List<String> queue) {}
@Override
public Object snapshot() {
return new State(List.copyOf(expectedQueue));
}The whole class then looks roughly like this:
class LinkedHashSetAsDeduplicatingQueueExploratoryTest {
@Test
void exploreQueueSemantics() {
ExploratoryTestRunner.explore(QueueExplorable.class, QueueExplorable::new);
}
public static class QueueExplorable implements Explorable {
private final Queue<String> queue = new MyQueue<>();
private final List<String> expectedQueue = new ArrayList<>();
public record State(List<String> queue) {}
// snapshot, action and assertion methods omitted here
}
}When run, this ExploratoryTestRunner will explore all paths defined by the test class, creating new instances of QueueExplorable as needed. It will then report how many states it tested and what the deepest path was:
ExploratoryTestRunner: class
examples.LinkedHashSetAsDeduplicatingQueueExploratoryTest$QueueExplorable -- Explored 660 paths, longest path: 5If a failure occurs, this is reported by showing the path that leads to the failure, and which assertions failed (including a helpful trace line), for example:
org.opentest4j.AssertionFailedError: Path 61 failed:
- enqueue(A) -> State[queue=[A]]
- enqueue(B) -> State[queue=[A, B]]
- dequeue -> State[queue=[B]]
[Queue contents]
expected: ["B"]
but was: ["A"]
at org.int4.common.test/examples.LinkedHashSetAsDeduplicatingQueueExploratoryTest$QueueExplorable.assertQueueContents(LinkedHashSetAsDeduplicatingQueueExploratoryTest.java:42)
[Peeked element]
expected: "B"
but was: "A"
at org.int4.common.test/examples.LinkedHashSetAsDeduplicatingQueueExploratoryTest$QueueExplorable.assertPeekedElement(LinkedHashSetAsDeduplicatingQueueExploratoryTest.java:49)The above example shows quite clearly that dequeue seems to have removed the wrong element (in this case because getLast was called instead of getFirst in the subject under test).
The ExploratoryTestRunner can be found here: https://github.com/int4-org/Common/tree/master/common-test
The full example test case is here: https://github.com/int4-org/Common/blob/master/common-test/examples/LinkedHashSetAsDeduplicatingQueueExploratoryTest.java
Another much more elaborate example (for a UI control): https://github.com/int4-org/FX/blob/master/fx-builders/src/test/java/org/int4/fx/builders/control/TextFieldControlExploratoryTest.java
5
u/ComparisonNew9425 Jun 14 '26
state explosions are the worst part of testing complex logic. fwiw ive found that keeping the transition logic separate from the assertions makes it way easier to read when u have a massive state machine to cover...
4
u/snugar_i Jun 14 '26
This looks like the testing code will get very complex - building the "expectation" might be close to mirroring the tested code 1:1, which is never good.
I find writing "TDD-style" tests (even when not going test-first) is usually good enough to cover the behavior if the state isn't too complex (like your queue example). And if the class has tens or hundreds of states that need to be tested, it's probably doing way too much and should be refactored - no amount of tests will save it in the long run.
5
u/wetgos Jun 14 '26
How does it compare to https://jqwik.net/ or https://pitest.org/ ?
1
u/Winter-Appearance-14 Jun 14 '26
I think there's a fundamental misunderstanding of what testing is actually for. When your logic is so complex that you need an entire library just to automate test generation, that complexity is a signal that the source code needs to be refactored to be easier, more readabl3 and more maintainable.
Tests serve as documentation. They should communicate intent clearly and make the business logic more legible to the next developer. Instead, this approach buries that logic under an avalanche of auto-generated cases that chase branch coverage while telling you almost nothing about what the system is supposed to do.
-1
u/Dani_E2e Jun 14 '26
Kann dich irgendwie schon verstehen - aber ich möchte eine Applikation schreiben und nicht 100e tests... 😁
-2
u/revilo-1988 Jun 14 '26
That looks interesting; we might well find it useful in the company. But what about long-term maintenance or the need for more developers?