# Simplified 2D convolution operation

def convolve2d(image, kernel):

output_height = image.shape[0] - kernel.shape[0] + 1

output_width = image.shape[1] - kernel.shape[1] + 1

output = np.zeros((output_height, output_width))

for i in range(output_height):

for j in range(output_width):

region = image[i:i+kernel.shape[0], j:j+kernel.shape[1]]

output[i, j] = np.sum(region * kernel)

return output

Key Parameters:

Filter Size: Common sizes are 3×3, 5×5, or 7×7. Smaller filters are computationally efficient and can capture fine-grained patterns. Larger filters have a wider receptive field but more parameters.

Number of Filters: Each filter learns to detect a different pattern. Early layers might have 32-64 filters, while deeper layers often have 256-512 or more.

Stride: The step size when sliding the filter. A stride of 1 means moving one pixel at a time; stride 2 reduces the output size by half.

Padding: Adding zeros around the input border. "Same" padding maintains the spatial dimensions; "valid" padding produces smaller outputs.

Feature Maps and Learned Filters

Each convolutional layer produces multiple feature maps—one per filter. These maps highlight where specific patterns appear in the input.

What Do Filters Learn?

Early layers learn low-level features:

• Edge detectors (vertical, horizontal, diagonal)
• Color gradients
• Simple textures

Middle layers combine these into mid-level features:

• Corners and junctions
• Simple shapes
• Parts of objects (eyes, wheels, leaves)

Deep layers recognize high-level concepts:

• Faces, cars, animals
• Complex textures
• Object configurations

This hierarchical feature learning is what makes CNNs so powerful—they automatically discover the relevant features for a task.

Pooling Layers

Pooling layers reduce the spatial dimensions of feature maps, decreasing computational requirements and providing translation invariance.

Max Pooling: Takes the maximum value within each pooling window. This preserves the strongest activations while reducing size.

def max_pool(feature_map, pool_size=2):

h, w = feature_map.shape

output = np.zeros((h // pool_size, w // pool_size))

for i in range(0, h, pool_size):

for j in range(0, w, pool_size):

output[i//pool_size, j//pool_size] = np.max(

feature_map[i:i+pool_size, j:j+pool_size]

)

return output

Average Pooling: Takes the average value, providing smoother downsampling.

Global Average Pooling: Reduces each feature map to a single value by averaging. Often used before the final classification layer.

Activation Functions

After each convolutional operation, an activation function introduces non-linearity.

ReLU (Rectified Linear Unit): The standard choice for CNNs. ReLU zeros out negative values while passing positive values unchanged. It's computationally efficient and helps with the vanishing gradient problem.

Leaky ReLU: Allows a small gradient for negative values, preventing "dying" neurons.

GELU and Swish: More recent alternatives that provide smooth non-linearities and sometimes improve performance.

Fully Connected Layers

After several convolutional and pooling layers, the extracted features are flattened and passed through fully connected (dense) layers for the final classification or prediction.

Modern CNN Architectures

LeNet-5 (1998)

The pioneering CNN architecture for digit recognition:

• Two convolutional layers with average pooling
• Three fully connected layers
• ~60,000 parameters

AlexNet (2012)

The architecture that started the deep learning revolution:

• Five convolutional layers, three fully connected layers
• First to use ReLU activation
• Introduced dropout for regularization
• ~60 million parameters

VGGNet (2014)

Demonstrated the power of depth:

• 16-19 layers using only 3×3 convolutions
• Simple, uniform architecture
• Showed that depth matters
• ~138 million parameters

GoogLeNet/Inception (2014)

Introduced the Inception module:

• Multiple filter sizes (1×1, 3×3, 5×5) applied in parallel
• 1×1 convolutions for dimensionality reduction
• 22 layers with only ~6.8 million parameters
• Auxiliary classifiers for training stability

ResNet (2015)

Revolutionary skip connections enabling extremely deep networks:

• Residual blocks: y = F(x) + x
• Enables training networks with 50, 101, even 152+ layers
• Addresses vanishing gradient problem
• Won ImageNet 2015 with top-5 error of 3.57%

# ResNet basic block

class ResidualBlock(nn.Module):

def __init__(self, channels):

super().__init__()

self.conv1 = nn.Conv2d(channels, channels, 3, padding=1)

self.bn1 = nn.BatchNorm2d(channels)

self.conv2 = nn.Conv2d(channels, channels, 3, padding=1)

self.bn2 = nn.BatchNorm2d(channels)

def forward(self, x):

residual = x

out = F.relu(self.bn1(self.conv1(x)))

out = self.bn2(self.conv2(out))

out += residual # Skip connection

return F.relu(out)

DenseNet (2017)

Takes skip connections further by connecting every layer to every other layer:

• Feature reuse across the network
• Fewer parameters than ResNet for similar performance
• Alleviates vanishing gradients

EfficientNet (2019)

Systematic scaling of network dimensions:

• Compound scaling: depth, width, and resolution scaled together
• Highly efficient with state-of-the-art accuracy
• EfficientNet-B7 achieves 84.3% top-1 ImageNet accuracy

Vision Transformer (ViT) (2020)

While not strictly a CNN, ViT showed that pure transformer architectures can excel at vision:

• Images split into patches treated as tokens
• Self-attention instead of convolutions
• Requires large datasets or pretraining
• Now often outperforms CNNs on major benchmarks

Techniques for Training CNNs

Data Augmentation

Artificially expanding the training dataset through transformations:

Geometric Transformations:

• Random horizontal/vertical flips
• Rotation by random angles
• Random cropping and scaling
• Affine transformations

Color Transformations:

• Brightness and contrast adjustment
• Saturation and hue shifts
• Color jittering

Advanced Augmentation:

• Cutout/Random erasing: Removing random patches
• Mixup: Blending two images and their labels
• CutMix: Cutting and pasting patches between images
• AutoAugment: Learning optimal augmentation policies

# PyTorch data augmentation example

transform = transforms.Compose([

transforms.RandomResizedCrop(224),

transforms.RandomHorizontalFlip(),

transforms.ColorJitter(brightness=0.4, contrast=0.4),

transforms.ToTensor(),

transforms.Normalize(mean=[0.485, 0.456, 0.406],

std=[0.229, 0.224, 0.225])

])

Batch Normalization

Normalizes layer inputs during training:

• Reduces internal covariate shift
• Enables higher learning rates
• Provides regularization effect
• Speeds up convergence significantly

# Batch normalization in a conv block

self.conv = nn.Conv2d(in_channels, out_channels, 3, padding=1)

self.bn = nn.BatchNorm2d(out_channels)

self.relu = nn.ReLU()

# Forward pass

x = self.relu(self.bn(self.conv(x)))

Transfer Learning

Leveraging pretrained models for new tasks:

Feature Extraction: Use pretrained layers as fixed feature extractors, training only the final classifier.

Fine-Tuning: Start from pretrained weights but allow gradual updates during training on the new task.

# Transfer learning with PyTorch

model = torchvision.models.resnet50(pretrained=True)

# Freeze early layers

for param in model.parameters():

param.requires_grad = False

# Replace final layer for new task

model.fc = nn.Linear(model.fc.in_features, num_classes)

# Only train the new layer initially

optimizer = optim.Adam(model.fc.parameters(), lr=0.001)

Learning Rate Scheduling

Adjusting the learning rate during training:

Step Decay: Reduce by a factor every N epochs

Cosine Annealing: Smooth cosine curve from initial to minimum LR

Warm Restarts: Periodic increases followed by decay

One Cycle Policy: Increase then decrease over training

Regularization Techniques

Preventing overfitting in CNNs:

Dropout: Randomly zero activations during training (typically 0.2-0.5 probability)

Weight Decay (L2 Regularization): Penalize large weights in the loss function

Label Smoothing: Replace hard labels with soft distributions

Stochastic Depth: Randomly skip layers during training (in ResNets)

CNN Applications

Image Classification

The foundational CNN task:

• ImageNet: 1000 categories, 1.2 million images
• CIFAR-10/100: Small images with 10/100 classes
• Fine-grained classification: Distinguishing bird species, car models, etc.

Object Detection

Locating and classifying multiple objects in images:

R-CNN Family: Region-based approaches using CNN for classification

• R-CNN: Slow, processes each region separately
• Fast R-CNN: Shares computation across regions
• Faster R-CNN: Adds Region Proposal Network (RPN)

YOLO (You Only Look Once): Single-pass detection

• Divides image into grid cells
• Predicts bounding boxes and class probabilities
• Extremely fast, suitable for real-time applications

SSD (Single Shot Detector): Multi-scale feature maps for detection at different sizes

Semantic Segmentation

Classifying every pixel in an image:

FCN (Fully Convolutional Networks): Replaces fully connected layers with convolutional layers

U-Net: Encoder-decoder architecture with skip connections

• Popular for medical image segmentation
• Precise localization through skip connections

DeepLab: Uses atrous (dilated) convolutions for multi-scale processing

Instance Segmentation

Combining object detection with semantic segmentation:

Mask R-CNN: Extends Faster R-CNN with a mask prediction branch

Face Recognition

Identity verification and identification:

• Feature embedding extraction
• Triplet loss or softmax-based training
• Applications in security, authentication, photo organization

Medical Imaging

CNNs have transformed medical image analysis:

• Tumor detection in CT/MRI scans
• Retinal disease diagnosis
• Skin cancer classification
• X-ray analysis for COVID-19 detection

Implementation Best Practices

Architecture Design Guidelines

1. Start Simple: Begin with proven architectures like ResNet before experimenting
2. Use 3×3 Convolutions: Stack multiple 3×3 layers instead of larger filters
3. Double Channels When Halving Dimensions: Common practice when using stride-2 convolutions
4. Add Batch Normalization: Place after convolution, before activation
5. Global Average Pooling: Replace fully connected layers where possible

Training Guidelines

1. Use Pretrained Weights: Almost always beneficial, even for different domains
2. Start with Lower Learning Rate for Fine-Tuning: Typically 1/10 of training from scratch
3. Monitor Validation Performance: Watch for overfitting
4. Use Mixed Precision Training: Faster and more memory efficient
5. Gradient Clipping: Prevents exploding gradients

Common Pitfalls to Avoid

Data Leakage: Ensure validation/test data isn't used in training or augmentation fitting

Class Imbalance: Use weighted loss, oversampling, or focal loss

Incorrect Preprocessing: Ensure test data uses same normalization as training

GPU Memory Issues: Reduce batch size, use gradient checkpointing, or mixed precision

Building a CNN from Scratch

Here's a complete example of building and training a CNN for image classification:

import torch

import torch.nn as nn

import torch.nn.functional as F

import torch.optim as optim

from torchvision import datasets, transforms

from torch.utils.data import DataLoader

class SimpleCNN(nn.Module):

def __init__(self, num_classes=10):

super(SimpleCNN, self).__init__()

# Convolutional layers

self.conv1 = nn.Conv2d(3, 32, kernel_size=3, padding=1)

self.bn1 = nn.BatchNorm2d(32)

self.conv2 = nn.Conv2d(32, 64, kernel_size=3, padding=1)

self.bn2 = nn.BatchNorm2d(64)

self.conv3 = nn.Conv2d(64, 128, kernel_size=3, padding=1)

self.bn3 = nn.BatchNorm2d(128)

# Pooling and dropout

self.pool = nn.MaxPool2d(2, 2)

self.dropout = nn.Dropout(0.25)

# Fully connected layers

self.fc1 = nn.Linear(128 * 4 * 4, 512)

self.fc2 = nn.Linear(512, num_classes)

def forward(self, x):

# Block 1

x = self.pool(F.relu(self.bn1(self.conv1(x))))

x = self.dropout(x)

# Block 2

x = self.pool(F.relu(self.bn2(self.conv2(x))))

x = self.dropout(x)

# Block 3

x = self.pool(F.relu(self.bn3(self.conv3(x))))

x = self.dropout(x)

# Flatten and fully connected

x = x.view(x.size(0), -1)

x = F.relu(self.fc1(x))

x = self.dropout(x)

x = self.fc2(x)

return x

# Training setup

model = SimpleCNN(num_classes=10)

criterion = nn.CrossEntropyLoss()

optimizer = optim.Adam(model.parameters(), lr=0.001)

scheduler = optim.lr_scheduler.StepLR(optimizer, step_size=10, gamma=0.1)

# Training loop

def train_epoch(model, loader, criterion, optimizer, device):

model.train()

total_loss = 0

correct = 0

total = 0

for images, labels in loader:

images, labels = images.to(device), labels.to(device)

optimizer.zero_grad()

outputs = model(images)

loss = criterion(outputs, labels)

loss.backward()

optimizer.step()

total_loss += loss.item()

_, predicted = outputs.max(1)

total += labels.size(0)

correct += predicted.eq(labels).sum().item()

return total_loss / len(loader), 100. * correct / total