Anomaly Detection on Time Series with MSET-SPRT in Python

In the world of anomaly detection, especially for complex systems like industrial machinery, nuclear reactors, and cybersecurity applications, traditional methods often fall short in handling high-dimensional and correlated data streams. This is where Multivariate State Estimation Technique (MSET) combined with the Sequential Probability Ratio Test (SPRT) shines.

MSET-SPRT is a powerful hybrid technique that uses machine learning for state estimation and statistical hypothesis testing for anomaly detection. It is widely used in mission-critical systems due to its accuracy and robustness.

Understanding MSET-SPRT

1. What is MSET?

MSET is a nonlinear regression technique that estimates the expected state of a system using a database of past observations. It works as follows:

• It maintains a memory matrix of past normal system states.
• It learns a weighted combination of historical states to estimate the current state.
• It detects deviations between actual and estimated values.

2. What is SPRT?

SPRT is a sequential hypothesis testing technique used to detect whether a deviation from normal behavior is statistically significant. It is used to:

• Continuously evaluate residual errors (differences between actual and estimated values).
• Raise an alarm when an anomaly is detected with statistical confidence.

Together, MSET-SPRT provides an effective mechanism for detecting anomalies in multivariate data.

MSET-SPRT in Python

Let’s implement MSET-SPRT with a simple dataset.

Step 1: Import Necessary Libraries

import numpy as np
import scipy.stats as stats
import matplotlib.pyplot as plt

Step 2: Generate Synthetic Data

We create a multivariate normal dataset representing a system operating under normal conditions.

# Simulating normal system behavior (3 correlated sensors)
np.random.seed(42)
mean =…