CS5720 - Week 7
Slide 124 of 140

LSTM Forward Pass: Step-by-Step

🎯 Forward Pass Steps
1Input Preparation
2Gate Computation
3Candidate Generation
4Cell State Update
5Hidden State Output
h_{t-1}
0.2
-0.1
0.5
+
x_t
0.8
0.3
→
Concatenate
[h_{t-1}, x_t]
Combined Input
0.2
-0.1
0.5
0.8
0.3
→
Forget Gate
0.65
Input Gate
0.78
Output Gate
0.82
Candidate Values
C̃_t = tanh(W_C·[h_{t-1}, x_t] + b_C)
0.42
C_{t-1}
→
Cell Update
C_t = f_t ⊙ C_{t-1} + i_t ⊙ C̃_t
→
C_t
Hidden State
h_t = o_t ⊙ tanh(C_t)
→
h_t (Output)
Complete LSTM Forward Pass Equations
Complete Forward Pass
f_t = σ(W_f · [h_{t-1}, x_t] + b_f)
i_t = σ(W_i · [h_{t-1}, x_t] + b_i)
C̃_t = tanh(W_C · [h_{t-1}, x_t] + b_C)
C_t = f_t ⊙ C_{t-1} + i_t ⊙ C̃_t
o_t = σ(W_o · [h_{t-1}, x_t] + b_o)
h_t = o_t ⊙ tanh(C_t)
Computational Complexity
Time: O(d² + d·n)
Space: O(d)
Parameters: 4d(d + n + 1)
Where d = hidden_size, n = input_size
Implementation Notes
• Use stable sigmoid/tanh implementations
• Initialize forget bias to 1.0
• Clip gradients during training
• Consider layer normalization
← Previous Next →
Prepared by Dr. Gorkem Kar