CS5720 - Week 2
Slide 35 of 40

Regularization Techniques Introduction

The Overfitting Problem

Overfitting occurs when a neural network learns the training data too well, including noise and specific details that don't generalize to new data.
Signs of Overfitting:
  • Training accuracy is very high (95%+)
  • Validation accuracy is much lower
  • Large gap between training and validation loss
  • Model performs poorly on new data
  • Training loss continues decreasing while validation loss increases

Regularization Solutions

Regularization techniques prevent overfitting by constraining the model's complexity.
  • ⚖️
    Weight Decay (L2)
    Penalizes large weights to keep the model simple
  • 🎲
    Dropout
    Randomly removes neurons during training
  • ⏰
    Early Stopping
    Stops training before overfitting occurs
  • 🔄
    Data Augmentation
    Increases training data variety artificially

Regularization Impact Comparison

Overfitting Model
Training: 98% accuracy
Validation: 75% accuracy
Problem: Memorizes training data
Well-Regularized Model
Training: 89% accuracy
Validation: 87% accuracy
Success: Good generalization
Under-regularized Model
Training: 95% accuracy
Validation: 72% accuracy
Issue: High variance
Goal: Find the sweet spot where training and validation performance are close!
← Previous Next →
Prepared by Dr. Gorkem Kar