Improved Dynamic Thresholds: We made our anomaly detection even more precise

Detecting anomalies in seasonality patterns is crucial for businesses to make accurate assessments and informed decisions. 

Seasonality patterns refer to predictable changes that occur within specific time periods, such as calendar or commercial seasons. These patterns can provide valuable insights into various aspects of a business, including sales, customer behavior, and market trends. However, irregularities within these patterns can have a significant impact on data accuracy and reliability. 

For instance, retail sales typically surge during the fourth quarter of the calendar year. As such, accurately detecting anomalies becomes crucial for retail businesses to optimize revenue during peak consumer demand.

TABLE OF CONTENTS

• 1. Enhanced anomaly detection in seasonality patterns
• 2. The puzzle of irregular seasonality
• 3. How Validio solves it
• Conclusion

Validio enhances its anomaly detection in seasonality patterns

Enhanced Dynamic Thresholds is our latest improvement for Validio’s anomaly detection model. To address the complexities of irregular seasonality, we incorporated two key components in our model: calendric and fuzzy seasonality. 

This technology seamlessly adapts to irregular seasonal patterns by leveraging information on past incidents, Dynamic Threshold status, and configured validator windows.

And here's the best part - as a user, you don't need to configure anything manually. The enhanced Dynamic Thresholds trigger automatically when sufficient evidence is detected. 

Here’s how it works:

The puzzle of irregular seasonality

When defining anomaly detection algorithms, incorporating seasonality patterns is essential to ensure that the monitored data represents the underlying reality of the business operations. But seasonality patterns are often irregular, making it challenging to detect deviations accurately. Undetected anomalies, a villain we call silent data issues, can hurt your business without you knowing.

On the other hand, alert fatigue can quickly develop if your anomaly detection model picks up on too many signals. We’ll revisit how Validio mitigates alert fatigue later in this post.

To accurately detect anomalies in irregular data patterns, there are generally two types of seasonality patterns you have to consider: calendric or fuzzy. 

Let’s walk through each of these and their potential business impact.

Calendric seasonality

Calendric seasonality corresponds to regular fluctuations in data values within specific time periods and are often tied to business cycles. For instance, businesses plan their work on a monthly basis, set quarterly goals, and review costs annually during budget planning. It is natural for these patterns to be reflected in your data.

To illustrate this, let's consider an example. 

Suppose you have an Airflow job that runs on either the first or last day of the month to collect data from third-party vendors. This data is used to calculate important performance metrics for your business. This doesn't mean that you are not interested in tracking your KPIs daily or weekly. It's simply a consequence of relying on external data sources. 

In Validio, you can set up volume validators to monitor the incoming row counts of your datasets. These usually returns a value of 0 on all days except when your pipeline runs to ingest data. These specific days are when you actually receive the data, but they don’t show how the underlying data changes in reality—between pipeline runs. With the improved Dynamic Thresholds, you will be able to account for these unique behaviors in your data.

By accurately detecting anomalies even in complex data patterns, you can ensure that your metrics reflect the true nature of your business operations.

Fuzzy seasonality

Fuzzy seasonality, a slightly more elusive phenomenon, also possesses the power to disrupt your operations.

Fuzzy seasonality refers to unpredictable patterns that can occur in various forms. While we may understand and explain these patterns in hindsight, it is challenging to predict when they will happen on a macro scale. However, by analyzing sufficient historical data, we can learn and account for the fact that there will be some deviation from the usual behavior in the data.

To better understand this concept, let's examine the example of paydays. In Sweden, payday typically falls on the 25th of the month, unless it falls on a weekend. In such cases, it is moved to the last weekday prior to the 25th. This consistent event can impact your metrics, but it is important to recognize that its influence may vary across different segments or regions. For instance, the impact of payday in India might differ from that in Sweden. Therefore, your monitoring approach should be flexible enough to detect and adapt to these variations in real-time.

How Validio solves it

Validio's improved anomaly detection model effectively groups atypical data points in time, allowing for the identification of fuzzy seasonal patterns with no precisely defined repeat frequencies. 

This approach also involves analyzing fixed patterns within a month or a week, such as occurrences every 2nd day of the month or every Monday. 

As a result, our models can recognize seasonality patterns based solely on the data or in relation to a calendar pattern. This capability allows us to uncover various repeating patterns in the data, including detecting overlapping seasonalities. 

For example, our model can identify a weekly sales pattern that experiences additional spikes at the end of the month due to paydays, which is typical for our retail and e-commerce customers. Our initial testing shows a significant increase in anomaly detection accuracy, reducing false positives by over 60%!

In conclusion: From noise to relevant alerts

Validio's enhanced Dynamic Thresholds improve anomaly detection by automatically adapting to complex irregular seasonality patterns. This feature includes calendric and fuzzy seasonality without requiring manual configuration. So, using dynamic thresholds improves the precision and accuracy of anomaly detection, making it easier to detect deviations in your business metrics, even in the presence of irregular seasonality. 

By providing more precise anomaly detection, Validio helps you surface insights that are truly valuable to your business, significantly reducing the number of irrelevant alerts.