CS5720 - Weight Initialization Strategies

The Foundation of Training

🎯 The Goldilocks Principle

Weight initialization must be "just right" - not too small (vanishing gradients), not too large (exploding gradients), but perfectly balanced to maintain stable signal flow through the network.

Why Initialization Matters:

Determines initial loss landscape position
Affects gradient flow from the start
Can make or break deep network training
Influences convergence speed dramatically

Forward Signal

Var[a] = 1.0

Backward Gradient

Var[∇] = 1.0

Weight Distribution Visualization

Xavier initialization for balanced variance

Major Initialization Strategies

⚖️

Xavier/Glorot

σ = √(2/(fan_in + fan_out))

Best for: Sigmoid, Tanh activations. Maintains variance across layers for symmetric activations.

🔥

He/Kaiming

σ = √(2/fan_in)

Best for: ReLU and variants. Accounts for ReLU killing half the neurons.

📐

LeCun

σ = √(1/fan_in)

Best for: SELU activation. Older method, still useful for specific cases.

🔲

Orthogonal

Q from QR decomposition

Best for: RNNs, preserves norm during forward/backward pass.

🎯

Sparse

Fixed sparsity pattern

Best for: Large networks where you want to enforce structure.

🔄

LSUV

Layer-Sequential Unit-Variance

Best for: Very deep networks, data-dependent initialization.

Modal Title

Weight Initialization Strategies

The Foundation of Training

Why Initialization Matters:

Weight Distribution Visualization

Major Initialization Strategies