4 min read

0x009 - Property-based Testing 📐

0x009 - Property-based Testing 📐

Hi, y’all! Welcome to unzip.dev, a newsletter dedicated to developer trends through unpacking trending dev concepts. My name is Agam More, and I’m a developer generalist. I’ll be posting every few weeks, so if you haven’t subscribed yet, go for it!

Sneak peek to the EOF: check out "A byte of coding".

Property-based Testing

“Really test your functions.”

Terms: PBT, QuickCheck.

TL;DR:

  • Problem: It’s hard to really test functions with only examples.
  • Solution: Write rules your function should abide by, then automatically generate examples and test them to find failing edge cases.
  • In Sum: Property-based testing requires a bit more work, but, in return, it will thoroughly test your functions. Example-based testing should still be used in conjunction with property-based testing.

How does it work? 💡

  1. Think and write some input-independent rules for the function you are testing.
  2. The Property-based testing framework will automatically generate many test case examples.
  3. It will then check those test case examples against your rules.
  4. If it finds any test failures, it will “shrink” the failed input to the minimum failing test case example so you can see the most simplistic input that makes it fail.

For example:

Let’s imagine we want to test a sorting algorithm that we wrote.

In property-based testing, we would start by defining some general rules that should be true all the time:

  1. Check that the output is the same size as the input (Invariantability).
  2. Sorting multiple times will produce the same result (Idempotentency).
  3. Check that every 2 numbers comply with num0 < num1 (Easy-to-verify).
  4. Check that the output is one of the permutations of the input (Invariantability).

Note that all the rules are independent of the input and the sorting implementation. This way, the framework can throw many examples against it to find edge case failures.

Now for the shrinking phase. Let’s assume our implementation of sort sucks. It sorts by taking the absolute value of a number. The generator found a failing test case of 100 numbers where one was negative. Now the shrinker comes into play, after a few runs it finds better and better failing examples until it lands on: [1,-3,2] which is way easier to reason about than the original 100 numbers array.

For those asking, what is the difference between fuzzing and property testing, my friend ben eloquently said: “The difference is mostly on guidance and expectations; fuzzers are mostly coverage-based, and property testing is expectation-based.”

Use cases ✅

  • Bug-sensitive applications.

Why? 🤔

  • Robustness: Because of the nature of the generator (kind of a fuzzer) you are actually testing for obscure edge cases you probably wouldn't think of yourself. You are really testing the essence of your function.
  • Write fewer tests: Property testing tests against many automatically-generated examples, thus reducing the need for many example-based tests.
  • Deeply understand your code: Having to formalize the core rules of your function makes you think more deeply about your code and how it should behave.

Why not? 🙅

  • Time and work: Coming up with the right properties to test is hard, and in most cases, the implementation could take a bit more time to implement than a simple example test.
  • Less readable: Property-based tests are less readable than regular example-based tests (which act as documentation and happy-path tests). In my opinion, I would mix property-based with example-based tests, so you get the best of both worlds.
  • Randomness != completeness: Because PBT picks random examples to test it might have a difficult time finding some single-value edge cases (see example).

Tools & players 🛠️

🤠
My opinion: Start asking if you can incorporate property-based testing for every function you write. I would still combine them with normal “example-based” tests for readability and documenting my code. Specifically I would probably use hypothesis.

Forecast 🧞

  • Market adoption: Unit tests are all over the place, but I can totally see PBT being used more commonly. We just need to educate more developers on it. I think that most languages will have somekind of builtin property-based framework out-of-the-box.
  • Auto-generate tests: I suspect that many of the boilerplate code needed for property-based tests can be automated and inferred from the code (maybe from something like GPT-codex?). Perhaps even some strategies can be automated, making PBT a no-brainer concept for almost any project. Hypothesis is already doing something like it!

Examples ⚗️

Who uses it? 🎡

Extra ✨

Some extra information that is related to the subject matter:

Thanks 🙏

I want to thank @TomGranot (the best DevRel I know), @Benstav (An amazing programmer and the person who introduced me to property-based testing).

EOF

I recently got introduced to Alex, the creator of "A byte of coding", a curated newsletter about technical deep-dives. I usually don't like to subscribe to curated links, as I just have too many things on my plate, but Alex's newsletter is quite unique. He summarizes the links and has a decent amount of quality content (which isn't trivial at all!). Go ahead and check the last issues out here.

Any questions, feedback, or suggestions are welcome 🙏

Simply reply to this e-mail or tweet at me @agammore - I promise to respond!

Agam More

Agam More