Feature update: Easier validator setup for improved data quality

TL;DR

The Validio 4.5 release includes updates to incident commenting, inherited ownership and an updated validator workflow - check the changelog for all details. For this blog post, we’ll dive deeper into the updated validator workflow. In short, this is what we’ve added:

• Use case based validator setup: The validator wizard has been simplified and redesigned to be based around data quality use cases and categories of validations, such as pipeline health, data consistency, and completeness.
• Duplicate and iterate on existing validators: Easily edit already configured validators, or use existing validators as a template for new data observability validators without having to recreate all of the work.
• Editable debug query: Expand and edit data issue debug queries directly to make root-cause analysis easier.
• Basic validator setup: Set up basic data quality validations like null filters and referential integrity checks without having to write everything with custom SQL.

If you’re ready to get started, this is all rolled out in the latest release. Or, read on for a deep dive on how we’ve made the validator workflow better.

Use case-based validator library

When setting up monitoring for your data there’s generally two angles to consider: technical validations, such as null%, row count and freshness, and business validations, looking at the actual value of data. A good starting point is to think about your validator setup in terms of different use cases. To simplify this, we’ve restructured the library of Validio validators to align with this. Now, you can easily browse validators based on categories like pipeline health, data consistency, and completeness and set up the validators to support your needs.

Duplicate and edit existing validators

We know things can change, and so can validators. That’s why we’ve made it possible to edit, duplicate, and iterate on existing validators. In addition to using this as a way to edit existing validators, you can also save yourself double work by using existing validators as a template when setting up new ones. Simply click a validator to open a new wizard, primed with the validator configuration. Edit based on the desired new validator and hit save.

Easier root-cause analysis with editable debug query

We’ve made the debug query editable, so you can easily pull in more information when doing root-cause analysis. For example, if you are monitoring a ratio of something you might want to be able to look at both the numerator and denominator to see how it behaves and now you can do that for better root cause analysis.

Simplified setup for basic validators

Sometimes you just want the simple data observability checks. Previously, anyone comfortable with writing custom SQL could easily do that, but with data consumers increasingly wanting to take control of data quality we needed to make this more straightforward. Now, you can set up “basic” validator use cases like null filters and referential integrity checks without having to write everything with custom SQL.

Related reads

• 4.5 Changelog
• A guide to setting up your first Data Quality and Observability rules
• Introducing Segment Filtering: Effortless application of custom SQL rules across datasets
• Introducing notification muting