View on GitHub

jqwik

Property-Based Testing in Java

The jqwik User Guide

The user guide is still rough and incomplete. Volunteers for polishing and extending it are more than welcome.

Table of Contents

How to Use

jqwik is an alternative test engine for the JUnit 5 platform. That means that you can use it either stand-alone or combine it with any other JUnit 5 engine, e.g. Jupiter (the standard engine) or Vintage (aka JUnit 4). All you have to do is add all needed engines to your testCompile dependencies as shown in the gradle file below.

jqwik is currently not deployed to Maven Central but JitPack is being used to provide the latest release(s). That’s why you have to add the JitPack-Repository to your list of maven repositories.

Gradle

Add the following stuff to your build.gradle file:

buildscript {
	dependencies {
	    ...
		classpath 'org.junit.platform:junit-platform-gradle-plugin:1.0.2'
	}
}

apply plugin: 'org.junit.platform.gradle.plugin'

repositories {
    ...
    mavenCentral()
    maven { url "https://jitpack.io" }
}

ext.junitPlatformVersion = '1.0.2'
ext.junitJupiterVersion = '5.0.2'
ext.jqwikVersion = '0.7.2'

junitPlatform {
	filters {
		includeClassNamePattern '.*Test'
		includeClassNamePattern '.*Tests'
		includeClassNamePattern '.*Properties'
	}
	enableStandardTestTask true
}

dependencies {
    ...

    // Needed to enable the platform to run tests at all
    testCompile("org.junit.platform:junit-platform-launcher:${junitPlatformVersion}")
    
    // jqwik dependency
    testCompile "com.github.jlink:jqwik:${jqwikVersion}"
    
    // Add if you want to also use the Jupiter engine
    // Also add if you use IntelliJ 2017.2 or older to enable JUnit-5 support
    testCompile("org.junit.jupiter:junit-jupiter-engine:${junitJupiterVersion}")
    
    // You'll probably need some assertions
    testCompile("org.assertj:assertj-core:3.8.0")

}

See the Gradle section in JUnit 5’s user guide for more details on how to configure test execution.

Maven

Add the following repository and dependency to your pom.xml file:

<repositories>
    ...
    <repository>
        <id>jitpack.io</id>
        <url>https://jitpack.io</url>
    </repository>
</repositories>

<dependencies>
    ...
    <dependency>
        <groupId>com.github.jlink</groupId>
        <artifactId>jqwik</artifactId>
        <version>0.7.2</version>
        <scope>test</scope>
    </dependency>
</dependencies>

See the Maven section in JUnit 5’s user guide for details on how to configure the surefire plugin and other dependencies.

Project without Build Tool

I never tried it but using jqwik without gradle or some other tool to manage dependencies should also work. You will have to add at least the following jars:

Creating an Example-based Test

Just annotate a public, protected or package-scoped method with @Example. Example-based tests work just like plain JUnit-style test cases and are not supposed to take any parameters.

A test case method must

Here is a test class with two example-based tests:

import static org.assertj.core.api.Assertions.*;

import net.jqwik.api.*;
import org.assertj.core.data.*;

class ExampleBasedTests {
	
	@Example
	void squareRootOf16is4() { 
		assertThat(Math.sqrt(16)).isCloseTo(4.0, Offset.offset(0.01));
	}

	@Example
	boolean add1plu3is4() {
		return (1 + 3) == 4;
	}
}

Creating a Property

You create a Property by annotating a public, protected or package-scoped method with @Property. In contrast to examples a property method is supposed to have one or more parameters, all of which must be annotated with @ForAll.

At test runtime the exact parameter values of the property method will be filled in by jqwik.

Just like an example test a property method has to

If not specified differently, jqwik will run 1000 tries, i.e. a 1000 different sets of parameter values and execute the property method with each of those parameter sets. The first failed execution will stop value generation and be reported as failure - usually followed by an attempt to shrink the falsified parameter set.

Here are two properties whose failures might surprise you:

import net.jqwik.api.*;
import org.assertj.core.api.*;

class PropertyBasedTests {

	@Property
	boolean absoluteValueOfAllNumbersIsPositive(@ForAll int anInteger) {
		return Math.abs(anInteger) >= 0;
	}

	@Property
	void lengthOfConcatenatedStringIsGreaterThanLengthOfEach(
		@ForAll String string1, @ForAll String string2
	) {
		String conc = string1 + string2;
		Assertions.assertThat(conc.length()).isGreaterThan(string1.length());
		Assertions.assertThat(conc.length()).isGreaterThan(string2.length());
	}
}

Currently jqwik cannot deal with parameters that are not annotated with ‘@ForAll’. However, this might change in future versions.

Optional @Property Parameters

The @Property annotation has a few optional values:

Assertions

jqwik does not come with any assertions, so you have to use one of the third-party assertion libraries, e.g. Hamcrest or AssertJ.

If you have Jupiter in your test dependencies anyway, you can also use the static methods in org.junit.jupiter.api.Assertions.

Lifecycle

Method Lifecycle

The current lifecycle of jqwik test methods is rather simple:

import net.jqwik.api.*;

class TestsWithLifecycle implements AutoCloseable {

	TestsWithLifecycle() {
		System.out.println("Before each");
	}

	@Example void anExample() {
		System.out.println("anExample");
	}

	@Property(tries = 5)
	void aProperty(@ForAll String aString) {
		System.out.println("anProperty: " + aString);
	}

	@Override
	public void close() throws Exception {
		System.out.println("After each");
	}
}

In this example both the constructor and close() will be called 6 times: Once for anExample() and 5 times for aProperty(...).

Other Lifecycles

Currently jqwik does not have special support for a lifecycle per test container, test try or even package. Later versions of jqwik might possible bring more features in that field. Create an issue on github with your concrete needs.

Grouping Tests

Within a containing test class you can group other containers by embedding another non-static and non-private inner class and annotating it with @Group. Grouping examples and properties is a means to improve the organization and maintainability of your tests.

Groups can be nested and there lifecycle is also nested, that means that the lifecycle of a test class is also applied to inner groups of that container.

import net.jqwik.api.*;

class TestsWithGroups {

	@Property
	void outer(@ForAll String aString) {
	}

	@Group
	class Group1 {
		@Property
		void group1Property(@ForAll String aString) {
		}

		@Group
		class Subgroup {
			@Property
			void subgroupProperty(@ForAll String aString) {
			}
		}
	}

	@Group
	class Group2 {
		@Property
		void group2Property(@ForAll String aString) {
		}
	}
}

Default Parameter Generation

jqwik tries to generate values for those property method parameters that are annotated with @ForAll. If the annotation does not have a value parameter, jqwik will use default generation for the following types:

If you use @ForAll with a value, e.g. @ForAll("aMethodName"), the method referenced by "aMethodName" will be called to provide an Arbitrary of the required type (see Customized Parameter Generation).

Constraining Default Generation

Default parameter generation can be influenced and constrained by additional annotations, depending on the requested parameter type.

All types:

Strings:

If Strings are not constrained a standard set of alphanumeric characters and a few other chars is used.

The following constraints can be combined with each other:

Characters:

If Characters are not constrained any char between '\u0000' and '\uffff' might be created.

The following constraints can be combined with each other:

List, Set, Stream and Arrays:

Integer and int:

Long, long and BigInteger:

Float and float:

Double, double and BigDecimal:

Constraining contained types

In case of collections, arrays and Optional the constraining annotations are also applied to the contained type, e.g.:

@Property
void aProperty(@ForAll @StringLength(max=10) List<String> listOfStrings) {
}

will generate lists of Strings that have 10 characters max.

Side Note

In future versions of jqwik constraints for contained types might have to be added to the type itself, like: @ForAll List<@StringLength(max=10) String> listOfStrings . Currently, though, not all Java 8 implementations support the retrieval of type parameter annotations through reflection.

Self-Made Annotations

You can make your own annotations instead of using jqwik’s built-in ones. BTW, ‘@Example’ is nothing but a plain annotation using @Property as “meta”-annotation.

The following example provides an annotation to constrain String or Character generation to German letters only:

@Target({ ElementType.ANNOTATION_TYPE, ElementType.PARAMETER })
@Retention(RetentionPolicy.RUNTIME)
@Digits
@AlphaChars
@Chars({'ä', 'ö', 'ü', 'Ä', 'Ö', 'Ü', 'ß'})
@Chars({' ', '.', ',', ';', '?', '!'})
@StringLength(min = 10, max = 100)
public @interface GermanText { }

@Property(tries = 10, reporting = ReportingMode.GENERATED)
void aGermanText(@ForAll @GermanText String aText) {}

The drawback of self-made annotations is that they do not forward their parameters to meta-annotations, which constrains their applicability to simple cases.

Customized Parameter Generation

Sometimes the possibilities of adjusting default parameter generation through annotations is not enough. In that case you can delegate parameter provision to another method. Look at the following example:

@Property
boolean concatenatingStringWithInt(
    @ForAll("shortStrings") String aShortString,
    @ForAll("10 to 99") int aNumber
) {
    String concatenated = aShortString + aNumber;
    return concatenated.length() > 2 && concatenated.length() < 11;
}

@Provide
Arbitrary<String> shortStrings() {
    return Arbitraries.strings('a', 'z', 1, 8);
}

@Provide("10 to 99")
Arbitrary<Integer> numbers() {
    return Arbitraries.integers(10, 99);
}

The String value of the @ForAll annotation serves as a reference to a method within the same class (or one of its superclasses or owning classes). This reference refers to either the method’s name or the String value of the method’s @Provide annotation.

The providing method has to return an object of type @Arbitrary<T> where T is the static type of the parameter to be provided.

Parameter provision usually starts with a static method call to Arbitraries, maybe followed by one or more filtering, mapping or combining actions.

For types that have no default generation at all, jqwik will use any provider method returning the correct type even if there is no explicit reference value in @ForAll. If provision is ambiguous jqwik will complain and throw an exception at runtime.

Static Arbitraries methods

The starting point for generation usually is a static method call on class Arbitraries.

Generate values yourself

Select values randomly

Integers

Decimals

Characters and Strings

Collections, Streams, Arrays and Optional

Generating types who have generic type parameters, requires to hand in an Arbitrary instance for the generic types.

Generate null values

Predefined generators will never create null values. If you want to allow that, call Arbitrary.injectNull(double probability). The following provider method creates an arbitrary that will return a null String in about 1 of 20 generated values.

@Provide 
Arbitrary<String> stringsWithNull() {
  return Arbitraries.strings(0, 10).injectNull(0.05);
}

Filtering

If you want to include only part of all the values generated by an arbitrary, use Arbitrary.filter(Predicate<T> filterPredicate). The following arbitrary will filter out all even numbers from the stream of generated integers:

@Provide 
Arbitrary<Integer> oddNumbers() {
  return Arbitraries.integers().filter(aNumber -> aNumber % 2 != 0);
}

Keep in mind that your filter condition should not be too restrictive. Otherwise the generation of suitable values might take very long or even never succeed, resulting in an endless loop.

Mapping

Sometimes it’s easier to start with an existing arbitrary and use its generated values to build other objects from them. In that case, use Arbitrary.map(Function<T, U> mapper). The following example uses generated integers to create numerical Strings:

@Provide 
Arbitrary<String> fiveDigitStrings() {
  return Arbitraries.integers(10000, 99999).map(aNumber -> String.valueOf(aNumber));
}

You could generate the same kind of values by constraining and filtering a generated String. However, the shrinking target would probably be different. In the example above, shrinking will move towards the lowest allowed number, that is 10000.

Using generated values to create another Arbitrary

Similar as in the case of Arbitrary.map(..) there are situations in which you want to use a generated value in order to create another Arbitrary from it. Sounds complicated? Have a look at the following example:

@Property
boolean fixedSizedStrings(@ForAll("listsOfEqualSizedStrings")List<String> lists) {
    return lists.stream().distinct().count() == 1;
}

@Provide
Arbitrary<List<String>> listsOfEqualSizedStrings() {
    Arbitrary<Integer> integers2to5 = Arbitraries.integers(2, 5);
    return integers2to5.flatMap(stringSize -> {
        Arbitrary<String> strings = Arbitraries.strings('a', 'z', stringSize, stringSize);
        return Arbitraries.listOf(strings);
    });
}

The provider method will create random lists of strings, but in each list the size of the contained strings will always be the same - between 2 and 5.

Combining Arbitraries

Sometimes just mapping a single stream of generated values is not enough to generate a more complicated domain object. In those cases you can combine several arbitraries to a single result arbitrary using Combinators.combine() with up to four arbitraries. Create an issue on github if you need more than four.

The following example generates Person instances from three arbitraries as inputs.

@Property
void validPeopleHaveIDs(@ForAll Person aPerson) {
    Assertions.assertThat(aPerson.getID()).contains("-");
    Assertions.assertThat(aPerson.getID().length()).isBetween(5, 24);
}

@Provide
Arbitrary<Person> validPeople() {
    Arbitrary<Character> initials = Arbitraries.chars('A', 'Z');
    Arbitrary<String> names = Arbitraries.strings('a', 'z', 2, 20);
    Arbitrary<Integer> ages = Arbitraries.integers(0, 130);
    return Combinators.combine(initials, names, ages)
        .as((initial, name, age) -> new Person(initial + name, age));
}

class Person {
    private final String name;
    private final int age;

    public Person(String name, int age) {
        this.name = name;
        this.age = age;
    }

    public String getID() {
        return name + "-" + age;
    }

    @Override
    public String toString() {
        return String.format("%s:%s", name, age);
    }
}

The property should fail, thereby shrinking the falsified Person instance to [Aaaaaaaaaaaaaaaaaaaaa:100].

Assumptions

If you want to constrain the set of generated values in a way that embraces more than one parameter, filtering does not work. What you can do instead is putting one or more assumptions at the beginning of your property.

The following property works only on strings that are not equal:

@Property
boolean comparingUnequalStrings( //
        @ForAll @StringLength(min = 1, max = 10) String string1, //
        @ForAll @StringLength(min = 1, max = 10) String string2 //
) {
    Assume.that(!string1.equals(string2));

    return string1.compareTo(string2) != 0;
}

This is a reasonable use of Assume.that(boolean condition) because most generated value sets will pass through.

Have a look at a seemingly similar example:

@Property
boolean findingContainedStrings( //
        @ForAll @StringLength(min = 1, max = 10) String container, //
        @ForAll @StringLength(min = 1, max = 5) String contained //
) {
    Assume.that(container.contains(contained));

    return container.indexOf(contained) >= 0;
}

Despite the fact that the property condition itself is correct, the property will most likely fail with the following message:

timestamp = 2017-11-06T14:36:15.134, 
    seed = 1066117555581106850
    tries = 1000, 
    checks = 20, 

org.opentest4j.AssertionFailedError: 
    Property [findingContainedStrings] exhausted after [1000] tries and [980] rejections

The problem is that - given a random generation of two strings - only in very few cases one string will be contained in the other. jqwik will report a property as exhausted if the ratio between generated and accepted parameters is higher than 5. You can change the maximum discard ratio by specifying a parameter maxDiscardRatio in the @Property annotation. That’s why changing to @Property(maxDiscardRatio = 100) in the previous example will probably result in a successful property run, even though only a handful cases - of 1000 generated - will actually be checked.

In many cases turning up the accepted discard ration is a bad idea. With some creativity we can often avoid the problem by generating out test data a bit differently. Look at this variant of the above property, which also uses Assume.that() but with a much lower discard ratio:

@Property
boolean findingContainedStrings_variant( //
        @ForAll @StringLength(min = 5, max = 10) String container, //
        @ForAll @IntRange(min = 1, max = 5) int length, //
        @ForAll @IntRange(min = 0, max = 9) int startIndex //
) {
    Assume.that((length + startIndex) <= container.length());

    String contained = container.substring(startIndex, startIndex + length);
    return container.indexOf(contained) >= 0;
}

Result Shrinking

If a property could be falsified with a generated set of values, jqwik will try to “shrink” this sample in order to find a “smaller” sample that also falsifies the property.

Try this property:

@Property
boolean stringShouldBeShrunkToAA(@ForAll @AlphaChars String aString) {
    return aString.length() > 5 || aString.length() < 2;
}

The test run result should look something like:

timestamp = 2017-11-04T16:42:25.859, 
    seed = -633877439388930932, 
    tries = 38, 
    checks = 38, 
    originalSample = ["LVtyB"], 
    sample = ["AA"]

AssertionFailedError: Property [stringShouldBeShrunkToAA] falsified with sample ["AA"]

In this case the originalSample could be any string between 2 and 5 chars, whereas the final sample should be exactly AA since this is the shortest failing string and A has the lowest numeric value of all allowed characters.

Integrated Shrinking

jqwik’s shrinking approach is called integrated shrinking, as opposed to type-based shrinking which most property-based testing tools use. The general idea and its advantages are explained here.

Consider a somewhat more complicated examples:

@Property
boolean shrinkingCanTakeLong(@ForAll("first") String first, @ForAll("second") String second) {
    String aString = first + second;
    return aString.length() > 5 || aString.length() < 2;
}

@Provide
Arbitrary<String> first() {
    return Arbitraries.strings('a', 'z', 1, 10).filter(string -> string.endsWith("h"));
}

@Provide
Arbitrary<String> second() {
    return Arbitraries.strings('0', '9', 0, 10).filter(string -> string.length() >= 1);
}

Shrinking still works, although there’s quite a bit of filtering and string concatenation happening:

timestamp = 2017-11-04T16:58:45.431, 
    seed = -5596810132893895291, 
    checks = 20, 
    tries = 20, 
    originalSample = ["gh", "774"], 
    sample = ["h", "0"]

AssertionFailedError: Property [shrinkingCanTakeLong] falsified with sample ["h", "0"]

Switch Shrinking Off

Sometimes shrinking takes a really long time or won’t finish at all (usually a jqwik bug!). In those cases you can switch shrinking off for an individual property:

@Property(shrinking = ShrinkingMode.OFF)
void aPropertyWithLongShrinkingTimes(
	@ForAll List<Set<String>> list1, 
	@ForAll List<Set<String>> list2
) {	... }

Running and Configuration

When running jqwik tests (through your IDE or your build tool) you might notice that - once a property has been falsified - it will always be tried with the same seed to enhance the reproducibility of a bug. This requires that jqwik will persist some runtime data across test runs.

You can configure this and other default behaviour in jqwik’s configuration.

jqwik Configuration

jqwik will look for a file jqwik.properties in your classpath in which you can configure a few basic parameters:

database = .jqwik-database
rerunFailuresWithSameSeed = true

This type of configuration is preliminary and will likely be replaced by JUnit 5’s platform configuration mechanism soon. Moreover, there will probably be many more default parameters to change.

Program your own Generators and Arbitraries

This topic will probably need a page of its own.

Register default Generators and Arbitraries

The API for providing Arbitraries and Generators by default is not public yet.

Glossary

Arbitrary

The fundamental type that is used to generate values. The name was first chosen by QuickCheck and later adopted by most property-based-testing libraries.

Under the hood, most instances of type Arbitrary use RandomGenerators to do the actual generation for them.

Property

A property is a test method that has one or more parameters annotated with @ForAll.

RandomGenerator

Instances of type RandomGenerator take care of actually generating a specific parameter instance.

Test Method

A test method is a a public, protected or package-scoped method, annotated with @Example or @Property.

Try

A try is the attempt at running a test method a single time with a specific set of parameters.