def federated_averaging(global_model, client_updates, client_sizes):

"""

Aggregate client model updates using weighted averaging.

Args:

global_model: Current global model parameters

client_updates: List of updated parameters from each client

client_sizes: Number of samples each client trained on

Returns:

Updated global model parameters

"""

total_samples = sum(client_sizes)

# Initialize aggregated parameters

aggregated = {key: torch.zeros_like(val)

for key, val in global_model.items()}

# Weighted sum of client updates

for update, size in zip(client_updates, client_sizes):

weight = size / total_samples

for key in aggregated:

aggregated[key] += weight * update[key]

return aggregated

Clients with more data contribute more heavily to the global model, reflecting their larger information contribution.

Cross-Device vs. Cross-Silo Federated Learning

Federated learning takes different forms depending on the setting:

Cross-device federated learning involves massive numbers of edge devices—smartphones, tablets, IoT devices—each with small amounts of data. This is the original setting that motivated federated learning, exemplified by Google's keyboard prediction. Challenges include device heterogeneity, intermittent connectivity, and coordinating millions of participants.

Cross-silo federated learning involves smaller numbers of organizations—hospitals, banks, enterprises—each with substantial datasets. Participants are more reliable but may have competing interests. This setting enables collaboration between organizations that cannot share data directly.

The technical challenges differ significantly between settings, though the core principle remains the same.

Challenges and Solutions

Federated learning introduces unique challenges beyond traditional distributed training.

Data heterogeneity: Client data is not independently and identically distributed (non-IID). A keyboard user typing primarily in Spanish has different data than one typing in English. This heterogeneity can cause model divergence and slow convergence.

Solutions include:

• Personalization layers that adapt to local distributions
• Data augmentation to balance distributions
• Regularization to prevent excessive divergence

Communication efficiency: Transmitting model updates consumes bandwidth, which may be limited and expensive. A modern transformer has billions of parameters; sending full updates is impractical.

Solutions include:

• Gradient compression: Send sparse or quantized updates
• Structured updates: Update only certain layers locally
• Communication scheduling: Update less frequently

System heterogeneity: Devices vary enormously in computational capability, network connectivity, and availability. Some devices may drop out mid-training.

Solutions include:

• Asynchronous aggregation that doesn't wait for all clients
• Adaptive selection of participating clients
• Robust aggregation that handles partial participation

Security against adversaries: Malicious clients might send corrupted updates to poison the model or infer information about other clients.

Solutions include:

• Byzantine-robust aggregation methods
• Secure aggregation protocols
• Differential privacy (discussed below)

Differential Privacy: Mathematical Privacy Guarantees

While federated learning keeps raw data decentralized, model updates can still leak information. A determined adversary might infer properties of training data from observed updates. Differential privacy provides mathematical guarantees that limit this information leakage.

The Definition

Differential privacy is defined formally: A randomized mechanism M satisfies (ε, δ)-differential privacy if for any two adjacent datasets D and D' (differing in one record) and any set of outcomes S:

$$P[M(D) \in S] \leq e^\epsilon \cdot P[M(D') \in S] + \delta$$

In plain terms: an observer seeing the mechanism's output cannot reliably determine whether any particular individual's data was included. The parameters ε (epsilon) and δ (delta) quantify the privacy guarantee—smaller values mean stronger privacy.

Intuition and Implications

Differential privacy ensures that participation in a dataset doesn't substantially change the output distribution. An adversary knowing the mechanism's output, the algorithm, and all data except one person's cannot confidently infer that person's data.

This provides several important properties:

Composability: Privacy guarantees compose across multiple analyses. Using the same data multiple times degrades privacy according to precise rules.

Post-processing immunity: Any analysis of differentially private outputs remains differentially private. You cannot accidentally weaken the guarantee by additional computation.

Resistance to auxiliary information: Even an adversary with substantial external information about an individual cannot use differentially private outputs to learn more.

Mechanisms for Achieving Differential Privacy

Several mechanisms add the randomness necessary for differential privacy:

Laplace mechanism: Add noise drawn from a Laplace distribution with scale calibrated to the function's sensitivity (maximum change from one record).

import numpy as np

def laplace_mechanism(value, sensitivity, epsilon):

"""

Apply Laplace mechanism for differential privacy.

Args:

value: True query answer

sensitivity: Maximum change from adding/removing one record

epsilon: Privacy parameter

Returns:

Noised answer satisfying epsilon-differential privacy

"""

scale = sensitivity / epsilon

noise = np.random.laplace(0, scale)

return value + noise

Gaussian mechanism: Add Gaussian noise, trading pure ε-differential privacy for (ε, δ)-differential privacy with often more acceptable noise magnitudes.

Exponential mechanism: For non-numeric outputs, select from possible outputs with probability exponential in a quality score, with calibration ensuring differential privacy.

Privacy Budget and Composition

Each differentially private operation consumes some of the privacy budget. The total privacy loss across multiple operations follows composition theorems:

Basic composition: For k analyses, each satisfying (ε, δ)-DP, the composition satisfies (kε, kδ)-DP.

Advanced composition: Tighter bounds show composition grows roughly as √k rather than k for small ε.

Organizations must track and manage their privacy budget, making decisions about which analyses to prioritize.

Differentially Private Machine Learning

Combining differential privacy with machine learning enables training models that provably protect training data.

DP-SGD: Private Gradient Descent

The most common approach, DP-SGD, modifies stochastic gradient descent:

1. Compute per-sample gradients: Rather than batch gradients, compute gradients for each sample individually.

1. Clip gradients: Bound each gradient's norm to a maximum value C, limiting any sample's influence.

1. Add noise: Add calibrated Gaussian noise to the sum of clipped gradients.

1. Update parameters: Use the noisy gradients for parameter updates.

import torch

def dp_sgd_step(model, data, labels, loss_fn,

max_grad_norm, noise_multiplier, lr):

"""

Perform one step of DP-SGD.

"""

model.zero_grad()

# Compute per-sample gradients

per_sample_grads = []

for x, y in zip(data, labels):

output = model(x.unsqueeze(0))

loss = loss_fn(output, y.unsqueeze(0))

loss.backward()

grads = {name: param.grad.clone()

for name, param in model.named_parameters()}

per_sample_grads.append(grads)

model.zero_grad()

# Clip gradients

for grads in per_sample_grads:

total_norm = torch.sqrt(sum(

g.norm()**2 for g in grads.values()

))

clip_coef = min(1, max_grad_norm / (total_norm + 1e-6))

for name in grads:

grads[name] *= clip_coef

# Sum clipped gradients

summed_grads = {}

for name in per_sample_grads[0]:

summed_grads[name] = sum(

grads[name] for grads in per_sample_grads

)

# Add noise

for name in summed_grads:

noise = torch.normal(

mean=0,

std=noise_multiplier * max_grad_norm,

size=summed_grads[name].shape

)

summed_grads[name] += noise

# Average and update

for name, param in model.named_parameters():

param.data -= lr * summed_grads[name] / len(data)

Privacy Accounting

Tracking privacy expenditure during training requires careful accounting. The moments accountant and Rényi differential privacy provide tighter bounds than basic composition:

from opacus.accountants import RDPAccountant

accountant = RDPAccountant()

for epoch in range(num_epochs):

for batch in dataloader:

# Training step

train_step(model, batch)

# Account for privacy

accountant.step(

noise_multiplier=noise_multiplier,

sample_rate=batch_size / len(dataset)

)

# Get final privacy guarantee

epsilon = accountant.get_epsilon(delta=1e-5)

print(f"Training achieved ({epsilon:.2f}, 1e-5)-differential privacy")

The Opacus Library

Facebook's Opacus library simplifies differentially private training in PyTorch:

from opacus import PrivacyEngine

model = create_model()

optimizer = torch.optim.SGD(model.parameters(), lr=0.01)

dataloader = create_dataloader()

privacy_engine = PrivacyEngine()

model, optimizer, dataloader = privacy_engine.make_private(

module=model,

optimizer=optimizer,

data_loader=dataloader,

noise_multiplier=1.0,

max_grad_norm=1.0,

)

for epoch in range(epochs):

train(model, dataloader, optimizer)

epsilon = privacy_engine.get_epsilon(delta=1e-5)

Privacy-Utility Trade-offs

Differential privacy inevitably degrades model utility. More privacy (lower ε) requires more noise, which harms accuracy. The challenge is achieving acceptable privacy at acceptable utility cost.

Factors affecting this trade-off:

• Dataset size: Larger datasets tolerate more noise per sample
• Model architecture: Some architectures are more amenable to DP training
• Training procedure: Careful hyperparameter tuning improves results
• Privacy budget allocation: Strategic use of privacy budget matters

Research continues on techniques to improve the privacy-utility trade-off.

Combining Federated Learning and Differential Privacy

Federated learning and differential privacy are complementary. Federated learning reduces exposure by keeping data local; differential privacy limits inference from what is exposed (model updates).

Secure Aggregation

Before even adding differential privacy, secure aggregation prevents the server from seeing individual updates:

Conceptually:

• Clients encrypt their updates
• Server aggregates encrypted updates
• Decryption reveals only the sum
• Individual updates remain hidden

Cryptographic protocols (like secure multi-party computation) enable this without trusting the server. Combined with differential privacy in the aggregated updates, this provides layered protection.

Local vs. Central Differential Privacy

Privacy guarantees can be applied at different points:

Local differential privacy: Each client adds noise before sending updates. Provides the strongest guarantees—even a compromised server learns limited information—but requires substantial noise.

Central differential privacy: The server adds noise to aggregated updates. Requires trusting the server but permits much less noise for the same privacy guarantee.

Hybrid approaches are possible, with moderate local noise combined with additional central noise.

Practical Implementation

Combining federated learning with differential privacy in practice:

class FederatedPrivateTrainer:

def __init__(self, model, clients, epsilon_per_round, delta):

self.global_model = model

self.clients = clients

self.epsilon = epsilon_per_round

self.delta = delta

def train_round(self):

# Send model to clients

client_updates = []

for client in self.clients:

# Client performs local DP-SGD training

update = client.train_with_dp(

self.global_model,

epsilon=self.epsilon,

delta=self.delta

)

client_updates.append(update)

# Secure aggregation (simplified)

aggregated = self.secure_aggregate(client_updates)

# Update global model

self.update_global_model(aggregated)

def secure_aggregate(self, updates):

# In practice, use cryptographic protocols

return sum(updates) / len(updates)