How we test Dynamic Segmentation at Atomic

At Atomic.io we developed a user segmentation feature. In this blog post we’ll delve into a few interesting aspects of the feature’s implementation and how we use property based testing to ensure we have excellent test coverage across our duration-based filter criteria.

What are Segments?

In short Segments are dynamic groups of users that match a chosen criteria. They are dynamic because changes in membership can be caused not only by changes to filter criteria, but by changes to user properties, and the passing of time.

Segment are made up of filter criteria combined using boolean logical operators (AND / OR) which are tested against a user record to determine membership.

We support two types of time filter criteria, absolute date times, for example before '1:59 AM, June 4th 2024' and relative periods, for example in the last month. This article focusses on the implementation and testing of relative filters.

For the most part we compare and convert dates using the fantastic open source date-fns library.

How do filters work?

To explain how date filters behave against a user record, let's take a concrete example of a filter which matches users active in the last 7 days:

Filter: lastActive (date property) is within the last 7 Days

Given a user record, we can check each property (in this case only lastActive) against this filter criteria and determine whether they’re currently in the segment or not. However because the segment is dynamic, we also need to determine if and when this could change.

Luckily for us, evaluating these conditions against a user record, is a perfect example of what functional programmers call a Pure Function.

Our pure evaluation function takes the following form:

Function arguments:

• UserRecord: e.g. {"name": "Alice", "lastActive": "2024-05-27T03:53:15.992Z", ... }

• FilterConditions: e.g. {"operator": "AND", "conditions": [...]}

• CurrentTime: Date

Function result, a record/tuple containing:

• IsMember: true | false

• ReEvaluateAt: null | Date (the future time at which to re-evaluate this record)

Traditional unit testing

For segments, we have many unit tests. Tests for individual conditions, and tests which compose various conditions together in interesting ways. This kind of testing is critical. They’re fast to run, give us confidence in our implementation and also help to document the intent of the system.

However, each of these tests only use discrete points in time, but time is continuous (has infinitely many values) and as such, testing each moment in time with a unit test would be impossible, and also a waste of…time. We asked ourselves if there are other ways we could explore a range of different dates?

One approach we considered, was to introduce randomness into our unit tests, testing a different value each time. This is a great starting point, and gets us thinking about the behaviour of the system. However, this method still only tests a set number of values per run, and each one is completely random.

How can we take a more systematic and comprehensive approach?

Enter property based testing!

There are other excellent articles and guides on when to use property based tests, however broadly speaking we’re checking ‘invariants’ - things that don’t change across a range of inputs. Some generic examples might be:

• adding two numbers in a different order should give the same result add(x, y) === add(y, x) (commutativity)

• or that applying a pure function more than once gives the same result toLowerCase(str) === toLowerCase(toLowerCase(str)) (idempotence)

Segment re-evaluation property

There are many invariant properties of our segment evaluator we can consider, however one which we felt was a good match for property testing is that we ‘always re-evaluate a users segment membership at the right time’.

To illustrate this, lets again take a concrete scenario:

• Filter: lastActive within the last 7 days

• User record: lastActive is 1pm on 0ct. 8

• Current time (T): 2pm on Oct. 8 (one hour after last-active)

After some head scratching we can see that the result will be:

• The user is currently a member of the segment

• Assuming the user record doesn’t change, they will no longer be a member 7 days after the lastActive date (the moment after ’1pm on Oct. 15’)

As a general property, it might be described in the following way:

• Given segment membership of a user at time T (under usual system operation T is just ‘now’)

• If the segment membership is expected to change in the future (T2)

• We expect the membership to be unchanged the instant before T2 (T1)

• We expect the membership to be different at T2

A test like this gives us confidence that the predicted segment reevaluation time is correct (not too early and not too late) across a broad range of dates.

Written in pseudo code that might look something like the following:

Writing a property test

In order to keep our test focussed on a single property, we’re going to only test a single filter condition rather than a fully composed filter (containing multiple conditions).

Translating this into a generalised form in Typescript would be a fair amount of work. Luckily for us there is a fantastic open source library fast-check which handles all of the grunt work for us.

First we need to construct a user record generator. This generator will be used by fast-check to generate many different user records during the test. To construct it, we need a date property with an arbitrary date value. We’ll use fast-check’s in-built Date primitive and record composite:

Next, we’ll construct our filter condition generator. First we need to write an arbitrary duration generator:

We can then use that to construct our filter condition:

Next we bring the generators together in a property (where the technique gets its name). To quote directly from fast-check’s documentation:

And that’s it! We now have the a way to explore a wide range of conditions, dates and durations!

Fast check has the ability to ‘shrink’ it’s arbitrary values, which is fantastic for debugging problems. A duration which fails our property check can often be shrunk down from a complicated form containing many elements, to something that still causes the issue but only has one of those problematic elements (e.g. P1Y2MT1D1S → PT1D). The same goes for dates being simplified down to the exact minute or hour.

What did we find?

We found a few very tricky bugs with our initial implementation with this method. For example, not handling future dates correctly or making assumptions about how durations work. Most notably, we learned the perhaps obvious fact that there is not an pure inverse relationship between duration addition and subtraction!

We also learned that our property based tests sometimes did not find all the issues we had written our own tests for. This might be due to our use of the tool or the time we allowed it to run for but in general is not surprising given the infinite input space!

Final thoughts

Property based testing, and fast-check in particular is a fantastic tool for testing code where the operational domain is very large. We were amazed at how fast-check was able to sniff out bugs ‘like a bloodhound’. That said, it’s definitely no substitute for a deep understanding of the domain and writing targetted unit tests!

Hopefully you enjoyed this article, please reach out with any feedback or corrections. If you’re interested in learning more about Atomic, please check out our website or contact our sales team!