Why plain RNNs forget everything
A standard Recurrent Neural Network (RNN) processes a sequence one step at a time, passing a hidden state forward like a note handed down a line of people. In theory this hidden state can carry information from many steps ago. In practice, it can't — not for long. Every time the note is copied through a tanh or sigmoid activation and multiplied by a weight matrix, the gradient used to update earlier weights shrinks a little more. Over dozens of time steps it shrinks to almost nothing. This is the vanishing gradient problem, and it means vanilla RNNs are effectively short-sighted: they can relate a word to the one before it, but not to one from three paragraphs earlier.
Long Short-Term Memory (LSTM) networks, introduced by Sepp Hochreiter and Jürgen Schmidhuber in 1997, fix this with a simple but powerful idea: separate what the network remembers from what the network outputs, and give it learned gates that decide what to keep, what to add, and what to expose at every single time step.
The cell state: a conveyor belt for memory
The core of an LSTM is the cell state (often written Cₜ). Picture
it as a conveyor belt that runs straight through every time step with only minor, deliberate
edits made to it — no heavy matrix multiplications, no repeated squashing through activation
functions. Because the path is so direct, gradients can travel back through many time steps
without shrinking to zero. This is the single architectural choice that makes long-range memory possible.
Alongside the cell state, the LSTM also maintains a hidden state (hₜ),
which is the short-term, filtered version of the cell state that gets passed to the next layer
and used as the network's output at that time step. Three gates — forget,
input, and output — control the traffic between these two states.
How LSTM decides what to remember
Every gate is a small neural layer: a sigmoid function squashes its output between 0 and 1, acting like a dial rather than an on/off switch — 0 means "block everything," 1 means "let everything through," and anything in between is a partial pass.
1. Forget Gate
Looks at hₜ₋₁ and xₜ and decides what fraction of the old cell state to keep. A value near 0 erases that piece of memory; near 1 preserves it fully.
2. Input Gate
Decides which new values to write into the cell state. A candidate layer proposes new content; the gate decides how much of it actually gets added.
C̃ₜ = tanh(W_C · [hₜ₋₁, xₜ] + b_C)
3. Output Gate
Decides what part of the (now updated) cell state should be exposed as the hidden state — the value passed forward and used for predictions.
hₜ = oₜ * tanh(Cₜ)
One time step, from input to output
Combine the new input with the previous hidden state
hₜ₋₁ and xₜ are concatenated and fed into all three gates plus the candidate layer, each with its own learned weight matrix.
Forget the irrelevant parts of memory
The old cell state is scaled by the forget gate's output: Cₜ₋₁ * fₜ. This removes information that is no longer useful for the task.
Add new, relevant information
The candidate values C̃ₜ are scaled by the input gate and added in: Cₜ = (Cₜ₋₁ * fₜ) + (iₜ * C̃ₜ).
Produce the filtered output
The updated cell state is squashed through tanh and scaled by the output gate to produce hₜ, which becomes both this step's output and the next step's short-term memory.
LSTM vs. GRU vs. vanilla RNN
Gated Recurrent Units (GRUs) are a popular, lighter alternative to LSTM. Here's how the three architectures stack up.
| Property | Vanilla RNN | LSTM | GRU |
|---|---|---|---|
| Gates | None | 3 (forget, input, output) | 2 (reset, update) |
| Separate cell state | No | Yes | No — merged into hidden state |
| Long-range memory | Poor | Strong | Strong, slightly shorter horizon |
| Parameters | Fewest | Most | ~25% fewer than LSTM |
| Training speed | Fastest | Slower | Faster than LSTM |
| Best for | Very short sequences | Long, complex dependencies | Smaller datasets, faster iteration |
Where LSTMs are used in production
Even with Transformers dominating headlines, LSTMs remain a strong, resource-efficient choice for sequential data — especially when sequences are long, data is limited, or low-latency inference matters.
Language Modeling
Next-word prediction, text generation, and early machine translation systems before attention-based models.
Time Series Forecasting
Stock price trends, energy demand, weather, and sensor data where order and long-term seasonality matter.
Speech Recognition
Converting audio waveforms into text by modeling phoneme sequences over time.
Anomaly Detection
Flagging unusual patterns in server logs, network traffic, or industrial sensor readings.
Music Generation
Learning melodic and rhythmic structure to generate new musical sequences note by note.
Handwriting Recognition
Modeling pen-stroke sequences to transcribe handwritten text.
Video Activity Recognition
Combining CNN features per frame with an LSTM to model motion over time.
Predictive Text & Autocomplete
On-device keyboards that suggest the next word based on typing history.
Advantages and limitations
Advantages
- Handles long-range dependencies far better than vanilla RNNs
- Gating mechanism gives fine-grained control over what's remembered
- Works well with limited training data compared to large Transformers
- Lower memory and compute footprint at inference time — good for edge devices
Limitations
- Sequential computation — can't parallelize across time steps like Transformers
- Still struggles with extremely long sequences (thousands of steps)
- More parameters and slower to train than GRUs
- Largely outperformed by attention-based models on large-scale NLP tasks
A minimal LSTM in Keras
Here's a compact, practical example of building an LSTM-based sequence classifier using TensorFlow/Keras.
Frequently asked questions
Is LSTM still relevant with Transformers around?
Yes, particularly for on-device, low-latency, or resource-constrained settings, and for tasks like time-series forecasting where sequence length is manageable and data is limited.
How many LSTM units should I use?
There's no universal number — start with 32–128 units for small-to-medium tasks and tune based on validation performance and overfitting.
What's the difference between "cell state" and "hidden state"?
The cell state is the long-term memory carried across time steps with minimal transformation. The hidden state is the filtered, short-term output derived from it at each step.
Can LSTM handle bidirectional context?
Yes — a Bidirectional LSTM runs two LSTMs (forward and backward) and combines their outputs, which is common in NLP tasks like named entity recognition.
Launch your Graphy