h_t = tanh(W_hh · h_{t-1} + W_xh · x_t + b_h)

y_t = W_hy · h_t + b_y

Where:

• h_t is the hidden state at time t
• x_t is the input at time t
• W_hh, W_xh, W_hy are weight matrices
• b_h, b_y are bias vectors
• tanh is the activation function

Implementation Example

import numpy as np

class SimpleRNN:

def __init__(self, input_size, hidden_size, output_size):

# Initialize weights

self.W_xh = np.random.randn(input_size, hidden_size) * 0.01

self.W_hh = np.random.randn(hidden_size, hidden_size) * 0.01

self.W_hy = np.random.randn(hidden_size, output_size) * 0.01

self.b_h = np.zeros((1, hidden_size))

self.b_y = np.zeros((1, output_size))

def forward(self, inputs):

"""

inputs: list of input vectors, one per time step

returns: outputs and hidden states

"""

h = np.zeros((1, self.W_hh.shape[0])) # Initial hidden state

hidden_states = []

outputs = []

for x in inputs:

h = np.tanh(np.dot(x, self.W_xh) + np.dot(h, self.W_hh) + self.b_h)

y = np.dot(h, self.W_hy) + self.b_y

hidden_states.append(h)

outputs.append(y)

return outputs, hidden_states

RNN Architectures for Different Tasks

One-to-Many: Single input produces sequence output

• Example: Image captioning (image → sequence of words)

Many-to-One: Sequence input produces single output

• Example: Sentiment analysis (text → positive/negative)

Many-to-Many (Synchronized): Outputs at each time step

• Example: Part-of-speech tagging, video classification

Many-to-Many (Encoder-Decoder): Input sequence → output sequence

• Example: Machine translation, summarization

The Vanishing Gradient Problem

While RNNs are theoretically powerful, basic RNNs struggle to learn long-range dependencies. This is due to the vanishing gradient problem.

Why Gradients Vanish

During backpropagation through time (BPTT), gradients are multiplied at each time step. If these multiplications consistently result in values less than 1, gradients shrink exponentially:

gradient ∝ ∏(t=1 to T) |W_hh · tanh'(h_t)|

With tanh derivatives bounded by 1 and typical weight matrices, gradients can become negligibly small after just 10-20 steps. The network effectively "forgets" early inputs.

Consequences

• Difficulty learning long-range dependencies
• Bias toward recent information
• Poor performance on tasks requiring memory of distant past
• Slow training and unstable optimization

Long Short-Term Memory (LSTM)

LSTM networks, introduced by Hochreiter and Schmidhuber in 1997, solve the vanishing gradient problem through a carefully designed architecture with gates.

The Cell State

The key innovation is the cell state (C_t), a pathway that allows information to flow through time with minimal modification. Think of it as a conveyor belt running through the entire sequence—information can be added or removed, but the main flow is preserved.

Gate Mechanisms

LSTM uses three gates to control information flow:

Forget Gate (f_t): Decides what information to discard from the cell state

f_t = σ(W_f · [h_{t-1}, x_t] + b_f)

Input Gate (i_t): Decides what new information to store

i_t = σ(W_i · [h_{t-1}, x_t] + b_i)

Output Gate (o_t): Decides what to output based on the cell state

o_t = σ(W_o · [h_{t-1}, x_t] + b_o)

Complete LSTM Equations

f_t = σ(W_f · [h_{t-1}, x_t] + b_f) # Forget gate

i_t = σ(W_i · [h_{t-1}, x_t] + b_i) # Input gate

C̃_t = tanh(W_C · [h_{t-1}, x_t] + b_C) # Candidate cell state

C_t = f_t ⊙ C_{t-1} + i_t ⊙ C̃_t # New cell state

o_t = σ(W_o · [h_{t-1}, x_t] + b_o) # Output gate

h_t = o_t ⊙ tanh(C_t) # Hidden state

Where ⊙ denotes element-wise multiplication.

Why LSTM Works

1. Additive updates: Cell state is updated additively, not multiplicatively
2. Gradient highway: Gradients can flow unchanged through the cell state
3. Selective memory: Gates learn what to remember and forget
4. Protected information: Important information can persist indefinitely

LSTM Implementation

import torch

import torch.nn as nn

class LSTMCell(nn.Module):

def __init__(self, input_size, hidden_size):

super(LSTMCell, self).__init__()

self.hidden_size = hidden_size

# Combined gate computation for efficiency

self.gates = nn.Linear(input_size + hidden_size, 4 * hidden_size)

def forward(self, x, states):

h_prev, c_prev = states

# Concatenate input and previous hidden state

combined = torch.cat([x, h_prev], dim=1)

# Compute all gates at once

gates = self.gates(combined)

# Split into individual gates

i, f, g, o = gates.chunk(4, dim=1)

# Apply activations

i = torch.sigmoid(i) # Input gate

f = torch.sigmoid(f) # Forget gate

g = torch.tanh(g) # Candidate cell state

o = torch.sigmoid(o) # Output gate

# Update cell state

c = f * c_prev + i * g

# Compute hidden state

h = o * torch.tanh(c)

return h, c

class LSTM(nn.Module):

def __init__(self, input_size, hidden_size, num_layers=1):

super(LSTM, self).__init__()

self.hidden_size = hidden_size

self.num_layers = num_layers

self.cells = nn.ModuleList([

LSTMCell(input_size if i == 0 else hidden_size, hidden_size)

for i in range(num_layers)

])

def forward(self, x, initial_states=None):

batch_size, seq_len, _ = x.size()

if initial_states is None:

h = [torch.zeros(batch_size, self.hidden_size, device=x.device)

for _ in range(self.num_layers)]

c = [torch.zeros(batch_size, self.hidden_size, device=x.device)

for _ in range(self.num_layers)]

else:

h, c = initial_states

outputs = []

for t in range(seq_len):

inp = x[:, t, :]

for layer, cell in enumerate(self.cells):

h[layer], c[layer] = cell(inp, (h[layer], c[layer]))

inp = h[layer]

outputs.append(h[-1])

return torch.stack(outputs, dim=1), (h, c)

Gated Recurrent Unit (GRU)

GRU, introduced by Cho et al. in 2014, is a simplified version of LSTM that often performs comparably with fewer parameters.

GRU Architecture

GRU combines the forget and input gates into a single "update gate" and merges the cell state with the hidden state:

z_t = σ(W_z · [h_{t-1}, x_t]) # Update gate

r_t = σ(W_r · [h_{t-1}, x_t]) # Reset gate

h̃_t = tanh(W · [r_t ⊙ h_{t-1}, x_t]) # Candidate hidden state

h_t = (1 - z_t) ⊙ h_{t-1} + z_t ⊙ h̃_t # New hidden state

LSTM vs GRU

| Aspect | LSTM | GRU |

|--------|------|-----|

| Parameters | More (3 gates + cell state) | Fewer (2 gates) |

| Training Speed | Slower | Faster |

| Performance | Often better on complex tasks | Comparable on many tasks |

| Memory Usage | Higher | Lower |

| Explainability | Separate cell state provides interpretability | Simpler but less interpretable |

Choose GRU for faster training or when data is limited; choose LSTM when modeling complex long-range dependencies.

Bidirectional RNNs

Standard RNNs only process sequences forward, but many tasks benefit from both past and future context.

How Bidirectional RNNs Work

A bidirectional RNN runs two separate hidden layers:

• Forward layer: processes sequence from start to end
• Backward layer: processes sequence from end to start

The outputs are then concatenated:

class BidirectionalLSTM(nn.Module):

def __init__(self, input_size, hidden_size):

super().__init__()

self.forward_lstm = nn.LSTM(input_size, hidden_size, batch_first=True)

self.backward_lstm = nn.LSTM(input_size, hidden_size, batch_first=True)

def forward(self, x):

# Forward pass

forward_out, _ = self.forward_lstm(x)

# Backward pass (reverse, process, reverse back)

x_reversed = torch.flip(x, [1])

backward_out, _ = self.backward_lstm(x_reversed)

backward_out = torch.flip(backward_out, [1])

# Concatenate

return torch.cat([forward_out, backward_out], dim=-1)

Applications

• Named Entity Recognition
• Machine Translation
• Speech Recognition
• Text Classification

Note: Bidirectional models require the full sequence upfront, making them unsuitable for real-time streaming applications.

Deep RNNs

Stacking multiple RNN layers creates deep recurrent networks:

# PyTorch makes this easy

lstm = nn.LSTM(

input_size=128,

hidden_size=256,

num_layers=3,

batch_first=True,

dropout=0.2, # Dropout between layers

bidirectional=True

)

Deep RNNs can learn hierarchical representations:

• Lower layers: low-level patterns
• Higher layers: abstract features

Training Techniques

Backpropagation Through Time (BPTT)

BPTT unfolds the RNN across time steps and applies standard backpropagation:

1. Forward pass through all time steps
2. Compute loss (sum of losses at each step or final step only)
3. Backward pass, accumulating gradients
4. Update weights

Truncated BPTT

For long sequences, full BPTT is memory-intensive. Truncated BPTT limits the backward pass to a fixed number of steps:

def truncated_bptt(model, data, k1=50, k2=50):

"""

k1: forward steps before backprop

k2: backward steps for gradient computation

"""

hidden = None

for i in range(0, len(data) - k1, k1):

batch = data[i:i+k1]

# Detach hidden state from graph

if hidden is not None:

hidden = (hidden[0].detach(), hidden[1].detach())

output, hidden = model(batch, hidden)

loss = compute_loss(output)

loss.backward()

optimizer.step()

optimizer.zero_grad()

Gradient Clipping

RNNs are susceptible to exploding gradients. Gradient clipping constrains gradient norms:

# Clip gradients to maximum norm of 5

torch.nn.utils.clip_grad_norm_(model.parameters(), max_norm=5.0)

Dropout in RNNs

Standard dropout applied to recurrent connections hurts performance. Use:

Variational Dropout: Same dropout mask across time steps

Dropout between layers: Apply dropout to outputs, not recurrent connections

# Built-in dropout in PyTorch LSTM

lstm = nn.LSTM(input_size, hidden_size, num_layers=2, dropout=0.3)

Learning Rate Scheduling

RNNs often benefit from learning rate decay:

scheduler = torch.optim.lr_scheduler.ReduceLROnPlateau(

optimizer, mode='min', factor=0.5, patience=5

)

Practical Applications

Language Modeling

Predict the next word given previous words:

class LanguageModel(nn.Module):

def __init__(self, vocab_size, embed_size, hidden_size):

super().__init__()

self.embedding = nn.Embedding(vocab_size, embed_size)

self.lstm = nn.LSTM(embed_size, hidden_size, batch_first=True)

self.fc = nn.Linear(hidden_size, vocab_size)

def forward(self, x, hidden=None):

embeds = self.embedding(x)

output, hidden = self.lstm(embeds, hidden)

logits = self.fc(output)

return logits, hidden

Sequence-to-Sequence (Seq2Seq)

Encoder-decoder architecture for translation, summarization:

class Encoder(nn.Module):

def __init__(self, input_size, hidden_size):

super().__init__()

self.lstm = nn.LSTM(input_size, hidden_size, batch_first=True)

def forward(self, x):

_, (hidden, cell) = self.lstm(x)

return hidden, cell

class Decoder(nn.Module):

def __init__(self, output_size, hidden_size):

super().__init__()

self.lstm = nn.LSTM(output_size, hidden_size, batch_first=True)

self.fc = nn.Linear(hidden_size, output_size)

def forward(self, x, hidden):

output, hidden = self.lstm(x, hidden)

prediction = self.fc(output)

return prediction, hidden

Time Series Forecasting

Predict future values from historical data:

class TimeSeriesLSTM(nn.Module):

def __init__(self, input_features, hidden_size, num_layers, output_size):

super().__init__()

self.lstm = nn.LSTM(input_features, hidden_size,

num_layers=num_layers, batch_first=True)

self.fc = nn.Linear(hidden_size, output_size)

def forward(self, x):

lstm_out, _ = self.lstm(x)

last_output = lstm_out[:, -1, :] # Take last time step

prediction = self.fc(last_output)

return prediction

Sentiment Analysis

Classify text sentiment:

class SentimentLSTM(nn.Module):

def __init__(self, vocab_size, embed_dim, hidden_dim, output_dim):

super().__init__()

self.embedding = nn.Embedding(vocab_size, embed_dim)

self.lstm = nn.LSTM(embed_dim, hidden_dim,

bidirectional=True, batch_first=True)

self.fc = nn.Linear(hidden_dim * 2, output_dim)

self.dropout = nn.Dropout(0.5)

def forward(self, x):

embedded = self.dropout(self.embedding(x))

output, (hidden, cell) = self.lstm(embedded)

# Concatenate final hidden states from both directions

hidden = torch.cat([hidden[-2,:,:], hidden[-1,:,:]], dim=1)

return self.fc(self.dropout(hidden))

Modern Alternatives and Extensions

Attention Mechanisms

Attention allows the decoder to focus on relevant parts of the input sequence:

class Attention(nn.Module):

def __init__(self, hidden_size):

super().__init__()

self.attention = nn.Linear(hidden_size * 2, hidden_size)

self.v = nn.Parameter(torch.rand(hidden_size))

def forward(self, hidden, encoder_outputs):

# hidden: [batch, hidden_size]

# encoder_outputs: [batch, seq_len, hidden_size]

seq_len = encoder_outputs.size(1)

hidden = hidden.unsqueeze(1).repeat(1, seq_len, 1)

energy = torch.tanh(self.attention(

torch.cat([hidden, encoder_outputs], dim=2)

))

attention_weights = torch.softmax(

torch.sum(self.v * energy, dim=2), dim=1

)

context = torch.bmm(attention_weights.unsqueeze(1), encoder_outputs)

return context, attention_weights

Transformer Architecture

Transformers, introduced in 2017, have largely superseded RNNs for many NLP tasks:

Advantages over RNNs:

• Parallel computation (no sequential dependency)
• Better at capturing long-range dependencies
• Easier to scale

When RNNs Still Make Sense:

• Streaming/online processing
• Memory-constrained environments
• Very long sequences (transformers have quadratic complexity)
• When explicit sequential modeling is beneficial

Temporal Convolutional Networks (TCN)

TCNs use dilated convolutions for sequence modeling:

class TCN(nn.Module):

def __init__(self, input_size, hidden_size, kernel_size, dilation):

super().__init__()

self.conv = nn.Conv1d(

input_size, hidden_size,

kernel_size, padding=(kernel_size-1)*dilation,

dilation=dilation

)

def forward(self, x):

return F.relu(self.conv(x))

Best Practices and Tips

Data Preparation

1. Sequence padding: Pad sequences to equal length or use pack_padded_sequence
2. Bucketing: Group similar-length sequences to minimize padding
3. Normalization: Scale numerical inputs appropriately

from torch.nn.utils.rnn import pack_padded_sequence, pad_packed_sequence

# Pack sequences for efficiency

packed = pack_padded_sequence(embeddings, lengths, batch_first=True)

packed_output, hidden = self.lstm(packed)

output, _ = pad_packed_sequence(packed_output, batch_first=True)