Reinforcement Learning Basics: A Guide for University Students

Reinforcement learning (RL) is a fundamental pillar of modern machine learning, distinct from supervised and unsupervised paradigms. Unlike supervised learning, where an agent is provided with a labelled dataset of correct answers, an RL agent learns through trial and error by interacting with an environment. This article explores the core mechanics of RL, providing the theoretical foundation you need to excel in your university assessments.

By the end of this guide, you will understand how an agent perceives its environment, how it makes decisions via a policy, and how it uses scalar rewards to optimise its behaviour over time. Mastering these concepts is essential for understanding advanced topics like Deep Q-Networks and Policy Gradients.

The Agent-Environment Loop

The core of reinforcement learning is the interaction loop between an agent and its environment. At each discrete time step $t$, the agent observes the current state $S_t$ of the environment. Based on this observation, the agent selects an action $A_t$ according to its policy. The environment then transitions to a new state $S_{t+1}$ and provides a scalar reward $R_{t+1}$ to the agent. This cycle repeats, creating a trajectory of experience: $(S_0, A_0, R_1, S_1, A_1, R_2, S_2, \dots)$.

Defining the Policy

The policy, denoted as $\pi$, is the strategy or mapping function that the agent uses to determine its next action based on the current state. Mathematically, a deterministic policy is defined as $\pi(s) = a$, while a stochastic policy is defined as $\pi(a|s) = P(A_t = a | S_t = s)$. The goal of the agent is to find an optimal policy $\pi^*$ that maximises the expected cumulative reward over time.

The Reward Hypothesis

In RL, the reward $R_t$ is a scalar feedback signal that indicates how well the agent is performing at step $t$. The Reward Hypothesis states that all goals can be described by the maximisation of the expected cumulative reward. This is often expressed as the return $G_t$, which is the sum of future rewards: $G_t = \sum_{k=0}^{\infty} \gamma^k R_{t+k+1}$, where $\gamma \in [0, 1]$ is the discount factor. The discount factor determines the importance of future rewards; a lower $\gamma$ makes the agent short-sighted, while a $\gamma$ close to 1 encourages long-term planning.

Worked Example 1: Simple Grid World

Imagine an agent in a 1x3 grid. The goal is to reach the rightmost cell (State 3).

• State 1: Start
• State 2: Middle
• State 3: Goal
• Rewards: -1 for each step, +10 for reaching the goal.

If the agent is in State 2 and chooses action 'Right', it moves to State 3. The reward $R_3 = 10$. If it chooses 'Left', it moves to State 1, $R_1 = -1$. The agent learns to prefer 'Right' because it leads to the terminal state with a high reward.

Worked Example 2: Calculating Cumulative Return

Consider an agent receiving a sequence of rewards: $R_1=0, R_2=0, R_3=10$. Let the discount factor $\gamma = 0.9$. We calculate the return $G_0$ from time $t=0$:

$$G_0 = \gamma^0 R_1 + \gamma^1 R_2 + \gamma^2 R_3$$ $$G_0 = 1(0) + 0.9(0) + 0.81(10) = 8.1$$

If $\gamma$ were 0.5, the return would be $0.5^2(10) = 2.5$. This demonstrates how the discount factor heavily weights immediate versus delayed gratification.

Common Mistakes

1. Confusing State and Observation: Students often treat the observation as the full state. In many environments, the agent only sees a partial observation, not the complete environment state.
2. Ignoring the Discount Factor: Forgetting that $\gamma$ is applied to future rewards, not the immediate reward $R_{t+1}$, is a frequent error in exam calculations.
3. Misunderstanding Exploration vs. Exploitation: Failing to balance trying new actions (exploration) with choosing known high-reward actions (exploitation) leads to sub-optimal policies.

Frequently Asked Questions

What is the difference between a model-based and model-free agent? Model-based agents learn or are given a model of the environment's dynamics (transitions and rewards), whereas model-free agents learn directly from experience without modelling the environment.

Why do we use a discount factor? It ensures mathematical convergence in infinite horizons and models uncertainty about the future.

What is a Markov Decision Process (MDP)? An MDP is a mathematical framework for modelling decision-making where the future state depends only on the current state and action, satisfying the Markov property.

Conclusion

Reinforcement learning is a powerful paradigm that bridges the gap between simple decision-making and complex autonomous behaviour. By understanding the agent-environment loop, the role of the policy, and the mechanics of reward maximisation, you are well-prepared for your studies. To see these concepts in action, generate a free animated lesson on this topic at MathInstructor AI.