How to Detect and Handle Multivariate Outliers in EDA

In exploratory data analysis (EDA), detecting and handling multivariate outliers is crucial for building robust machine learning models and ensuring data quality. Unlike univariate outliers, which involve extreme values in a single variable, multivariate outliers are observations that deviate significantly from the pattern of multiple variables considered together. These outliers can distort statistical analyses, affect model performance, and mislead insights if not addressed properly.

Understanding Multivariate Outliers

Multivariate outliers are data points that do not conform to the general distribution of the dataset when multiple variables are taken into account. An observation might seem normal when viewed through one variable but appear anomalous when the interaction among several features is analyzed.

For example, in a dataset containing height and weight, an individual with a high weight might not be an outlier univariately. However, if that person has an unusually low height for the given weight, it may be considered a multivariate outlier.

Why Multivariate Outliers Matter

Bias in Models: Outliers can skew model parameters, leading to overfitting or underfitting.
Inaccurate Insights: They may distort the correlation between variables.
Poor Generalization: If not treated, models trained on such data may perform poorly on unseen data.
Invalid Assumptions: Many statistical methods assume normality or homoscedasticity, both of which can be violated by outliers.

Methods for Detecting Multivariate Outliers

1. Mahalanobis Distance

Mahalanobis distance measures the distance of a point from the mean of a multivariate distribution, taking into account the correlations among variables.

Formula:

D^2 = (x – mu)^T Sigma^{-1} (x – mu)

$x$ : Observation vector
$mu$ : Mean vector
$Sigma$ : Covariance matrix

A higher Mahalanobis distance indicates a greater likelihood of being an outlier. This method works well for normally distributed data.

Steps:

Standardize the dataset.
Compute the mean vector and covariance matrix.
Calculate Mahalanobis distances.
Use a Chi-square distribution to determine threshold.

2. Isolation Forest

Isolation Forest is a tree-based anomaly detection method designed for high-dimensional data. It isolates observations by randomly selecting a feature and then randomly selecting a split value. Outliers are easier to isolate and require fewer splits.

Advantages:

Handles high-dimensional data well.
No need to assume data distribution.
Efficient and scalable.

3. Local Outlier Factor (LOF)

LOF compares the local density of a point to that of its neighbors. If a point has a substantially lower density than its neighbors, it is likely an outlier.

Use case: Especially useful in datasets with clusters of varying density.

4. DBSCAN (Density-Based Spatial Clustering of Applications with Noise)

DBSCAN groups points that are closely packed and marks points that lie alone in low-density regions as outliers.

Pros:

Does not require specifying the number of clusters.
Can detect outliers as points not belonging to any cluster.

5. Principal Component Analysis (PCA)

PCA can help visualize high-dimensional data and identify multivariate outliers by reducing dimensionality. Outliers may be more apparent in the lower-dimensional space.

Steps:

Apply PCA to transform the data.
Plot the principal components.
Observe the points that deviate significantly from the cluster.

6. Elliptic Envelope

Assumes data follows a Gaussian distribution and fits an ellipse around the central data points. Points outside the ellipse are considered outliers.

Best suited for: Data with a roughly elliptical distribution.

Visualizing Multivariate Outliers

Pair Plot

Using seaborn.pairplot, you can plot all combinations of variables to see how relationships unfold in 2D. Anomalies will appear isolated from clusters.

3D Scatter Plots

For three-variable cases, 3D scatter plots can be insightful. Outliers will visibly deviate from the main cluster.

Heatmaps

Correlation heatmaps can indicate multicollinearity, which might help explain or identify outliers in context.

Handling Multivariate Outliers

1. Remove Outliers

If the outliers are due to data entry errors or are not relevant to the modeling objective, removing them might be best.

Example:

python
df_clean = df[mahalanobis_distances < threshold]

2. Transform Variables

Apply transformations like log, square root, or Box-Cox to normalize distributions and reduce the impact of outliers.

3. Cap Outliers

Winsorization replaces extreme values with a specific percentile value.

python
from scipy.stats.mstats import winsorize
df['feature'] = winsorize(df['feature'], limits=[0.05, 0.05])

4. Use Robust Models

Models such as Decision Trees, Random Forests, and XGBoost are less sensitive to outliers.

5. Create Binary Features

Sometimes, keeping outliers and flagging them with an additional binary variable can preserve information while allowing models to adjust accordingly.

6. Clustering for Segmentation

Before modeling, cluster the data (e.g., via K-Means) and assess which clusters contain outliers. You can handle them separately or treat them as different cohorts.

Real-World Example Using Python

python
import numpy as np
import pandas as pd
from scipy.stats import chi2
from sklearn.covariance import MinCovDet

# Sample multivariate dataset
df = pd.DataFrame({
    'Feature1': np.random.normal(0, 1, 100),
    'Feature2': np.random.normal(0, 1, 100)
})

# Injecting outliers
df.loc[0] = [10, 10]
df.loc[1] = [-10, -10]

# Compute Mahalanobis distance
def mahalanobis(x=None, data=None, cov=None):
    x_mu = x - np.mean(data)
    inv_covmat = np.linalg.inv(cov)
    left = np.dot(x_mu, inv_covmat)
    mahal = np.dot(left, x_mu.T)
    return mahal.diagonal()

cov_matrix = np.cov(df.values.T)
df['Mahalanobis'] = mahalanobis(x=df.values, data=df.values, cov=cov_matrix)
threshold = chi2.ppf((1-0.01), df=2)
df['Outlier'] = df['Mahalanobis'] > threshold

Tips for Effective Multivariate Outlier Handling

Always normalize/standardize features before applying distance-based methods.
Validate outliers manually or with domain knowledge.
Don’t over-clean: Sometimes, what appears to be an outlier is actually a valuable rare event.
Document your outlier detection and treatment steps for reproducibility.

Conclusion

Detecting and handling multivariate outliers is a vital part of the data preprocessing pipeline in EDA. With the right techniques—ranging from statistical methods like Mahalanobis distance to machine learning-based methods like Isolation Forest—you can ensure that your data is clean, consistent, and ready for accurate analysis. How you handle outliers should align with your data context and analysis goals, always balancing between model performance and data integrity.

Share This Page: