Validio's IaC: unique capabilities through real code

If you work at a modern company, chances are you want to work with modern tools. At Validio, providing a world-class developer experience is an integral part of our Deep Data Observability platform offering. To live up to this promise, we have built a developer toolkit for engineers in modern data teams to fully harness the benefits of the platform. You can use the Validio developer toolkit to automate functionality or interact with the Validio platform, without having to use the web interface. 

This article focuses on Validio’s Infrastructure as Code (IaC) in particular, showcasing how it can empower you to unlock the full potential of the Validio platform. Read this if you are:

• Anyone who is looking for a data observability tool complemented  by a developer-friendly toolkit  
• Data engineers or analysts who want to dynamically build your sources or validators for data quality validation
• DevOps engineers who manage installations of software and multiple environments: testing, staging, production
• System owners who manage a growing number of resources—spreading across teams, functions, or applications

Introducing Validio’s Infrastructure as Code

Companies that adopted modern data stacks usually work with a large collection of  tools developed by a wide range of providers. For these companies, it has become more and more important to both automate and revision control any setup and configuration. Recognizing the need for efficient usage of our platform, we developed 'Validio code' for robust, automated control. 

Validio’s Infrastructure as Code (IaC) is a way to allow you to configure, maintain, and provision any of Validio’s resources such as sources and validators via code.

Unique capabilities of Validio’s IaC

Let’s dive into what makes Validio’s IaC special for you as a developer.

When we say code we actually mean code

When we started working on our IaC solution, it was clear that all our engineers had preferences based on their previous experience with other IaC solutions. Many had run into issues and limitations by having to do “programming” in formats such as YAML, JSON, or drilling down the documentation of HCL. Therefore, the Validio team chose an approach that would add simplicity and flexibility to our system: Python as our configuration language.

This unique approach brings multiple values to Validio’s technical users: 

Full expressiveness of a programming language

Writing loops in a YAML configuration file (and other markup languages such as JSON, HCL) is tedious. It doesn’t have to be that way with Python. 

One example of using Python’s loops is to create repeating configuration, making it easy to manage multiple environments.If you maintain testing or staging environments, a common problem is their tendency to diverge from the production environment, making them useless. By using the same exact configuration and resource definition, you can configure all your environments the same way without duplicating much of your code.

It's easy to dynamically build your setup with the full expressiveness of a programming language. Overall, the Python ecosystem comes with a lot of tools and SDKs, making it convenient to reach external resources for data needed in your setup. This could be secrets stored in a cloud service, validators based on each row in a table, or user creation based on your active directory (a database of who’s employed at the company). 

Built-in documentation and autocompletion

Even though we opted to use Python, you don’t need to learn a new full SDK or figure out in what order you need to do things. Instead, you use Python to create resource objects in a typesafe way with inline IDE documentation, syntax highlighting and autocompletion and save them to a .py -file. Validio code will take care of the rest by reading the file and automatically figure out what should be created, updated or deleted based on what’s already provisioned.

Bidirectional interaction between IaC and GUI

When developing Validio’s IaC, we knew that most engineers tend to play around in the GUI until they’re happy with the setup and lock that setup in code. If the interaction is only from IaC to GUI, you will have to start over and manually convert your setup to code. With Validio’s bidirectional support that uniquely offers the interaction GUI to IaC, our system will generate the code for you. With a simple command, you can move all (or selected) resources from one namespace to another, while generating the code for the resources. 

The GUI to IaC support also makes it easy to opt in or out of using IaC at any time, whether you are a new user setting up everything from scratch with IaC, or an existing user who has been using Validio through the GUI for years. You can easily decide where you want to manage your resources. Validio’s approach allows technical users to benefit from IaC regarding backups and revision control, while simultaneously enabling non-technical users to work within the GUI to build their validators.

Getting the full benefits of Validio’s platform through IaC

Not only does IaC make Validio fit better in a tech stack that might already be provisioned by tools like Terraform or Pulumi, it also opens up to more features that might not be available through the web GUI.

Add security by handling human errors

IaC enhances security by allowing for quick restoration of validation rules if deleted, since they are defined in code. If you accidentally remove validators, you can add them back easily. 

Sensitive data management is improved with IaC, as it can securely fetch credentials like database passwords on the fly without human exposure. Moreover, integration with vaults or secret storages within CI/CD pipelines during automated provisioning will be very easy by using Python. 

Separating your resources

The introduction of namespaces in IaC helps manage and track an increasing number of resources by organizing them by team, function, or application. We added support for namespaces to make it extra easy for you to keep track when the number of resources grows. Namespaces also enable the creation of specific validators for various purposes, like separating sales data from application logs.

How Validio’s IaC interacts with the rest of Validio’s developer toolkit

We built our IaC solution with all the building blocks from our developer toolkit. At its core, the API is called to create, update, or remove any resources. Next, the SDK makes interaction with the API easy. Lastly, the CLI is used to interact with the SDK.

This means that we’re not only creating tools for the specific need of IaC, we’re giving you the full developer toolbox including API, SDK and CLI to build your workflow exactly the way you want it. For more information, visit Validio's documentation of developer tooling.

Parting thoughts

Validio’s IaC has been built with unique capabilities through our Python approach and the bidirectional IaC and GUI interaction. With IaC, you can add security by handling human errors, as well as separate a potentially growing number of resources through using namespaces. 

Validio’s developer toolkit, including IaC, has been an important enabler for our customers to fully harness the power of data quality validation in real time. 

Are you part of a modern data team looking to get a Deep Data Observability platform, with a state of art developer toolkit? Reach out to us for a demo.