Reinforcement Learning Guide

Understand and implement reinforcement learning algorithms from tabular methods through deep RL, including policy gradients, actor-critic, and model-based approaches.

RL Fundamentals

The RL Framework

An agent interacts with an environment to maximize cumulative reward:

Agent                     Environment
  |                           |
  |--- action a_t ---------->|
  |                           |--- next state s_{t+1}
  |<-- reward r_t, state s_t |--- reward r_{t+1}
  |                           |

Concept	Symbol	Definition
State	s	Observation of the environment
Action	a	Decision made by the agent
Reward	r	Scalar feedback signal
Policy	pi(a|s)	Mapping from states to actions
Value function	V(s)	Expected cumulative reward from state s
Q-function	Q(s, a)	Expected cumulative reward from (s, a)
Discount factor	gamma	Weight of future vs. immediate rewards (0-1)
Return	G_t	Sum of discounted future rewards from time t

Key Equations

# Return (discounted cumulative reward)
G_t = r_t + gamma * r_{t+1} + gamma^2 * r_{t+2} + ...

# Bellman equation for V
V(s) = E[r + gamma * V(s') | s]

# Bellman equation for Q
Q(s, a) = E[r + gamma * max_a' Q(s', a') | s, a]

# Policy gradient theorem
gradient J(theta) = E[gradient log pi_theta(a|s) * Q(s, a)]

Algorithm Taxonomy

Category	Algorithm	Key Idea	On/Off Policy
Value-based	Q-Learning	Learn Q(s,a), act greedily	Off-policy
	DQN	Q-Learning + neural net + replay buffer	Off-policy
	Double DQN	Two networks to reduce overestimation	Off-policy
	Dueling DQN	Separate value and advantage streams	Off-policy
Policy gradient	REINFORCE	Monte Carlo policy gradient	On-policy
	PPO	Clipped surrogate objective	On-policy
	TRPO	Trust region constraint	On-policy
Actor-Critic	A2C/A3C	Advantage actor-critic (parallel)	On-policy
	SAC	Maximum entropy + off-policy AC	Off-policy
	TD3	Twin delayed DDPG	Off-policy
Model-based	Dreamer	World model + imagination	On-policy
	MBPO	Model-based policy optimization	Off-policy
	MuZero	Learned model + planning (MCTS)	Off-policy

Implementation: DQN

import torch
import torch.nn as nn
import torch.optim as optim
import numpy as np
from collections import deque
import random

class QNetwork(nn.Module):
    def __init__(self, state_dim, action_dim, hidden_dim=128):
        super().__init__()
        self.net = nn.Sequential(
            nn.Linear(state_dim, hidden_dim),
            nn.ReLU(),
            nn.Linear(hidden_dim, hidden_dim),
            nn.ReLU(),
            nn.Linear(hidden_dim, action_dim)
        )

    def forward(self, x):
        return self.net(x)

class DQNAgent:
    def __init__(self, state_dim, action_dim, lr=1e-3, gamma=0.99,
                 epsilon=1.0, epsilon_decay=0.995, epsilon_min=0.01,
                 buffer_size=10000, batch_size=64):
        self.action_dim = action_dim
        self.gamma = gamma
        self.epsilon = epsilon
        self.epsilon_decay = epsilon_decay
        self.epsilon_min = epsilon_min
        self.batch_size = batch_size

        self.q_network = QNetwork(state_dim, action_dim)
        self.target_network = QNetwork(state_dim, action_dim)
        self.target_network.load_state_dict(self.q_network.state_dict())
        self.optimizer = optim.Adam(self.q_network.parameters(), lr=lr)

        self.replay_buffer = deque(maxlen=buffer_size)

    def select_action(self, state):
        if random.random() < self.epsilon:
            return random.randint(0, self.action_dim - 1)
        with torch.no_grad():
            q_values = self.q_network(torch.FloatTensor(state))
            return q_values.argmax().item()

    def store_transition(self, state, action, reward, next_state, done):
        self.replay_buffer.append((state, action, reward, next_state, done))

    def train_step(self):
        if len(self.replay_buffer) < self.batch_size:
            return 0.0

        batch = random.sample(self.replay_buffer, self.batch_size)
        states, actions, rewards, next_states, dones = zip(*batch)

        states = torch.FloatTensor(np.array(states))
        actions = torch.LongTensor(actions)
        rewards = torch.FloatTensor(rewards)
        next_states = torch.FloatTensor(np.array(next_states))
        dones = torch.FloatTensor(dones)

        # Current Q values
        q_values = self.q_network(states).gather(1, actions.unsqueeze(1)).squeeze()

        # Target Q values (Double DQN variant)
        with torch.no_grad():
            best_actions = self.q_network(next_states).argmax(1)
            next_q = self.target_network(next_states).gather(1, best_actions.unsqueeze(1)).squeeze()
            targets = rewards + self.gamma * next_q * (1 - dones)

        loss = nn.MSELoss()(q_values, targets)
        self.optimizer.zero_grad()
        loss.backward()
        self.optimizer.step()

        self.epsilon = max(self.epsilon_min, self.epsilon * self.epsilon_decay)
        return loss.item()

    def update_target(self):
        self.target_network.load_state_dict(self.q_network.state_dict())

Implementation: PPO

class PPOAgent:
    def __init__(self, state_dim, action_dim, lr=3e-4, gamma=0.99,
                 lam=0.95, clip_ratio=0.2, epochs=10):
        self.gamma = gamma
        self.lam = lam
        self.clip_ratio = clip_ratio
        self.epochs = epochs

        self.actor = nn.Sequential(
            nn.Linear(state_dim, 64), nn.Tanh(),
            nn.Linear(64, 64), nn.Tanh(),
            nn.Linear(64, action_dim), nn.Softmax(dim=-1)
        )
        self.critic = nn.Sequential(
            nn.Linear(state_dim, 64), nn.Tanh(),
            nn.Linear(64, 64), nn.Tanh(),
            nn.Linear(64, 1)
        )
        self.optimizer = optim.Adam(
            list(self.actor.parameters()) + list(self.critic.parameters()), lr=lr
        )

    def compute_gae(self, rewards, values, dones):
        """Generalized Advantage Estimation."""
        advantages = []
        gae = 0
        for t in reversed(range(len(rewards))):
            next_value = values[t + 1] if t + 1 < len(values) else 0
            delta = rewards[t] + self.gamma * next_value * (1 - dones[t]) - values[t]
            gae = delta + self.gamma * self.lam * (1 - dones[t]) * gae
            advantages.insert(0, gae)
        return torch.FloatTensor(advantages)

    def update(self, states, actions, old_log_probs, rewards, dones):
        values = self.critic(states).squeeze().detach().numpy()
        advantages = self.compute_gae(rewards, values, dones)
        returns = advantages + torch.FloatTensor(values[:len(advantages)])
        advantages = (advantages - advantages.mean()) / (advantages.std() + 1e-8)

        for _ in range(self.epochs):
            probs = self.actor(states)
            dist = torch.distributions.Categorical(probs)
            new_log_probs = dist.log_prob(actions)
            entropy = dist.entropy().mean()

            ratio = (new_log_probs - old_log_probs).exp()
            clipped = torch.clamp(ratio, 1 - self.clip_ratio, 1 + self.clip_ratio)
            actor_loss = -torch.min(ratio * advantages, clipped * advantages).mean()

            critic_loss = nn.MSELoss()(self.critic(states).squeeze(), returns)

            loss = actor_loss + 0.5 * critic_loss - 0.01 * entropy
            self.optimizer.zero_grad()
            loss.backward()
            self.optimizer.step()

Research Environments

Environment	Domain	Complexity	Key Paper
Gymnasium (ex-Gym)	Classic control, Atari	Low-High	Brockman et al., 2016
MuJoCo	Continuous control, robotics	Medium-High	Todorov et al., 2012
DMControl	Continuous control from pixels	High	Tassa et al., 2018
ProcGen	Procedurally generated games	High (generalization)	Cobbe et al., 2020
Minigrid	Grid-world navigation	Low-Medium	Chevalier-Boisvert et al.
Isaac Gym	GPU-accelerated physics sim	High	Makoviychuk et al., 2021
NetHack	Complex roguelike game	Very High	Kuttler et al., 2020

Top Venues

Venue	Type	Focus
NeurIPS	Conference	Broad ML including RL
ICML	Conference	Broad ML including RL
ICLR	Conference	Representation learning, deep RL
AAAI	Conference	Broad AI
CoRL	Conference	Robot learning
JMLR	Journal	Broad ML (open access)
L4DC	Conference	Learning for dynamics and control

Key Research Directions (2024-2025)

1. RLHF / RLAIF: RL from human or AI feedback for LLM alignment
2. Offline RL: Learning from pre-collected datasets without environment interaction
3. Foundation models for control: Using pre-trained LLMs/VLMs as world models or planners
4. Multi-agent RL: Cooperative and competitive settings with communication
5. Safe RL: Constrained optimization to ensure safety during training and deployment
6. Sample-efficient RL: Reducing the gap between model-free and model-based sample complexity