How to Visualize the Effect of Data Scaling on Model Performance

Data scaling plays a crucial role in the performance of machine learning models, particularly those that are sensitive to feature magnitudes. Visualizing the effect of scaling not only aids in understanding its impact but also helps in choosing the right preprocessing strategy. In this article, we explore various visualization techniques and comparative analyses to understand how data scaling affects model performance.

Understanding Data Scaling

Data scaling is a preprocessing technique used to standardize or normalize data features to a specific range or distribution. Common scaling methods include:

• Min-Max Scaling: Transforms features to a fixed range, usually [0, 1].

• Standardization (Z-score Normalization): Centers the data around zero with a standard deviation of one.

• Robust Scaling: Uses the median and interquartile range, making it less sensitive to outliers.

• MaxAbs Scaling: Scales each feature by its maximum absolute value.

Scaling is essential for distance-based models like K-Nearest Neighbors (KNN), gradient descent-based models like logistic regression, and algorithms sensitive to feature variance like SVMs and neural networks.

Why Visualization Matters

Visualizing the effect of scaling helps:

• Identify model sensitivity to feature magnitude.

• Understand how scaling transforms the feature space.

• Compare model accuracy and decision boundaries.

• Communicate model behavior effectively to stakeholders.

Dataset Selection for Visualization

To demonstrate the effect of data scaling, we use datasets that are easy to visualize, such as:

• Iris Dataset

• Breast Cancer Dataset

• Digits Dataset

• Synthetic 2D classification datasets (like make_classification or make_moons)

These datasets allow visual inspection of decision boundaries, feature distributions, and classification results.

Visualizing Raw vs. Scaled Features

A straightforward method is plotting raw feature distributions versus scaled ones using histograms and boxplots.

1. Histograms

Plot histograms for each feature before and after scaling to observe:

• Range compression (for Min-Max Scaling)

• Centering and variance (for Standardization)

python
CopyEdit
import matplotlib.pyplot as plt
from sklearn.preprocessing import StandardScaler
import seaborn as sns
import pandas as pd
from sklearn.datasets import load_iris

iris = load_iris()
df = pd.DataFrame(iris.data, columns=iris.feature_names)

scaler = StandardScaler()
df_scaled = pd.DataFrame(scaler.fit_transform(df), columns=iris.feature_names)

for col in df.columns:
    fig, ax = plt.subplots(1, 2, figsize=(12, 4))
    sns.histplot(df[col], kde=True, ax=ax[0])
    ax[0].set_title(f'Original - {col}')
    sns.histplot(df_scaled[col], kde=True, ax=ax[1])
    ax[1].set_title(f'Scaled - {col}')
    plt.show()

2. Boxplots

Boxplots reveal how scaling affects the spread and central tendency of features.

PCA Visualization

Principal Component Analysis (PCA) helps visualize high-dimensional data in two or three dimensions. By applying PCA before and after scaling, one can observe how scaling affects the variance captured in each component.

python
CopyEdit
from sklearn.decomposition import PCA

pca = PCA(n_components=2)
components_raw = pca.fit_transform(df)
components_scaled = pca.fit_transform(df_scaled)

plt.figure(figsize=(12, 5))
plt.subplot(1, 2, 1)
plt.scatter(components_raw[:, 0], components_raw[:, 1], c=iris.target, cmap='viridis')
plt.title('PCA on Raw Data')

plt.subplot(1, 2, 2)
plt.scatter(components_scaled[:, 0], components_scaled[:, 1], c=iris.target, cmap='viridis')
plt.title('PCA on Scaled Data')
plt.show()

This visualization shows how scaling can redistribute the variance, leading to more informative principal components.

Visualizing Decision Boundaries

For classification models, decision boundary plots are one of the most direct ways to see the effect of scaling.

Example: Logistic Regression with and without Scaling

python
CopyEdit
from sklearn.linear_model import LogisticRegression
from sklearn.model_selection import train_test_split
from sklearn.preprocessing import StandardScaler
from sklearn.datasets import make_classification
import numpy as np

X, y = make_classification(n_features=2, n_redundant=0, n_informative=2, n_clusters_per_class=1, n_samples=500, random_state=4)
X_train, X_test, y_train, y_test = train_test_split(X, y, stratify=y, random_state=42)

model_raw = LogisticRegression().fit(X_train, y_train)

scaler = StandardScaler()
X_train_scaled = scaler.fit_transform(X_train)
model_scaled = LogisticRegression().fit(X_train_scaled, y_train)

def plot_decision_boundary(model, X, y, title):
    h = .02
    x_min, x_max = X[:, 0].min() - .5, X[:, 0].max() + .5
    y_min, y_max = X[:, 1].min() - .5, X[:, 1].max() + .5
    xx, yy = np.meshgrid(np.arange(x_min, x_max, h),
                         np.arange(y_min, y_max, h))
    Z = model.predict(np.c_[xx.ravel(), yy.ravel()])
    Z = Z.reshape(xx.shape)

    plt.figure(figsize=(6, 4))
    plt.contourf(xx, yy, Z, alpha=0.8)
    plt.scatter(X[:, 0], X[:, 1], c=y, edgecolors='k')
    plt.title(title)
    plt.show()

plot_decision_boundary(model_raw, X_train, y_train, 'Raw Data')
plot_decision_boundary(model_scaled, X_train_scaled, y_train, 'Scaled Data')

These plots show how scaling affects the decision boundaries and convergence of the classifier.

Comparing Model Metrics

Another visualization involves comparing evaluation metrics (accuracy, precision, recall, F1 score) before and after scaling using bar charts or tables.

python
CopyEdit
from sklearn.svm import SVC
from sklearn.metrics import accuracy_score

model_unscaled = SVC()
model_unscaled.fit(X_train, y_train)
y_pred_unscaled = model_unscaled.predict(X_test)

X_test_scaled = scaler.transform(X_test)
model_scaled = SVC()
model_scaled.fit(X_train_scaled, y_train)
y_pred_scaled = model_scaled.predict(X_test_scaled)

accuracy_scores = {
    'Unscaled': accuracy_score(y_test, y_pred_unscaled),
    'Scaled': accuracy_score(y_test, y_pred_scaled)
}

plt.bar(accuracy_scores.keys(), accuracy_scores.values())
plt.title('Model Accuracy Comparison')
plt.ylabel('Accuracy')
plt.show()

This clearly displays the performance improvement post scaling, especially for models like SVM or KNN.

Heatmaps for Feature Correlation Before and After Scaling

Heatmaps visually communicate the change in feature correlation matrices after scaling.

python
CopyEdit
sns.heatmap(df.corr(), annot=True, cmap='coolwarm')
plt.title("Correlation Matrix - Before Scaling")
plt.show()

sns.heatmap(df_scaled.corr(), annot=True, cmap='coolwarm')
plt.title("Correlation Matrix - After Scaling")
plt.show()

This helps in understanding if scaling impacts inter-feature relationships (usually doesn’t, but visual confirmation is valuable).

Conclusion

Visualizing the effect of data scaling provides tangible insights into how model behavior and performance can change dramatically with appropriate preprocessing. From altering decision boundaries to improving convergence rates and model accuracy, scaling is a crucial step. By employing histograms, PCA plots, decision boundaries, and metric comparisons, practitioners can make informed choices and communicate the value of scaling effectively. These visualizations not only justify preprocessing steps but also enhance interpretability and trust in the machine learning pipeline.