CS5720 - What Makes a Network "Deep"?

Understanding Depth

A Deep Neural Network is a neural network with multiple hidden layers between the input and output layers. But it's not just about the number - it's about what happens at each layer.

🎯 The Magic Number

There's no official threshold, but networks with 3+ hidden layers are typically considered "deep". Modern networks can have hundreds or even thousands of layers!

Why does depth matter?

Each layer learns increasingly abstract representations:
• Early layers: Simple features (edges, colors)
• Middle layers: Complex patterns (shapes, textures)
• Later layers: High-level concepts (objects, categories)

Network Architecture

Input Layer

Hidden Layer 1

Hidden Layer 2

H10

H11

Hidden Layer 3

Output Layer

Shallow vs Deep Networks

Shallow Networks

(1-2 hidden layers)

✓ Faster training
✓ Less prone to overfitting
✓ Easier to interpret
✗ Limited representation power
✗ May need more neurons

Deep Networks
(3+ hidden layers)
✓ Learn hierarchical features
✓ More parameter efficient
✓ Better for complex tasks
✗ Harder to train
✗ Require more data

Very Deep Networks

(50+ layers)

✓ State-of-the-art performance
✓ Can learn very complex patterns
✗ Vanishing gradients
✗ Need special techniques
✗ Computationally expensive

Modal Title

What Makes a Network "Deep"?

Understanding Depth

Network Architecture

Shallow vs Deep Networks