Understanding Generative Adversarial Networks: A Mathematical Perspective

Introduction to GANs

Generative Adversarial Networks (GANs) represent one of the most significant breakthroughs in deep learning over the last decade. First introduced by Ian Goodfellow and his colleagues in 2014, GANs provide a framework for training generative models through an adversarial process. For undergraduate students, understanding GANs is essential, as they appear frequently in advanced machine learning modules and research projects involving synthetic data generation.

At its core, a GAN consists of two neural networks: the generator ($G$) and the discriminator ($D$). These networks are locked in a zero-sum game, where the generator attempts to create data that is indistinguishable from the training set, while the discriminator attempts to identify whether a given sample is real or synthetic. This article will guide you through the mathematical mechanics of this competition and how it leads to the generation of high-quality data.

The Architecture of the Adversarial Game

The GAN framework is modelled as a two-player minimax game. The generator $G$ takes a random noise vector $z$ from a prior distribution $p_z(z)$ and maps it to the data space, producing a synthetic sample $G(z)$. Simultaneously, the discriminator $D$ takes an input $x$ and outputs a scalar $D(x)$, representing the probability that $x$ came from the real data distribution $p_{data}(x)$ rather than the generator.

The objective function for this game is defined as:

$$\min_G \max_D V(D, G) = \mathbb{E}{x \sim p{data}(x)}[\log D(x)] + \mathbb{E}_{z \sim p_z(z)}[\log(1 - D(G(z)))]$$

In this equation, the discriminator aims to maximise the probability of assigning the correct label to both real and fake samples, while the generator aims to minimise the probability that the discriminator correctly identifies its output as fake.

Training the Discriminator: A Worked Example

To understand the discriminator's role, consider a simplified scenario where we have one real data point $x=1$ and one generated point $G(z)=0$. Suppose the discriminator is a simple function $D(x) = \sigma(wx + b)$, where $\sigma$ is the sigmoid function.

If we want to update the discriminator to correctly classify these, we look at the loss function for $D$:

$$L_D = -[\log(D(1)) + \log(1 - D(0))]$$

If $D(1) = 0.8$ and $D(0) = 0.3$, the loss is:

$$L_D = -[\log(0.8) + \log(1 - 0.3)] = -[\log(0.8) + \log(0.7)] \approx -[-0.223 - 0.357] = 0.58$$

By performing gradient descent on $L_D$, the discriminator adjusts its weights $w$ and $b$ to push $D(1)$ closer to 1 and $D(0)$ closer to 0, effectively learning the boundary between real and fake data.

Training the Generator: The Minimax Objective

The generator does not have direct access to the real data. It learns solely through the gradient signal provided by the discriminator. When the generator produces a sample $G(z)$, it wants the discriminator to output $D(G(z)) = 1$.

Consider the generator's loss component: $L_G = \log(1 - D(G(z)))$. In practice, researchers often use the non-saturating loss $-\log(D(G(z)))$ to provide stronger gradients early in training.

Worked Example: Suppose the discriminator is currently very confident that the generator's output is fake, such that $D(G(z)) = 0.1$.

Using the non-saturating loss:

$$L_G = -\log(0.1) \approx 2.30$$

If the generator updates its parameters such that $D(G(z))$ improves to $0.4$, the new loss is:

$$L_G = -\log(0.4) \approx 0.92$$

The reduction in loss from 2.30 to 0.92 demonstrates how the generator successfully 'fools' the discriminator more effectively, moving its output distribution closer to the real data distribution.

Common Mistakes

1. Mode Collapse: This occurs when the generator finds a small subset of the data that fools the discriminator and produces only those samples, ignoring the diversity of the training set.
2. Vanishing Gradients: If the discriminator becomes too perfect too quickly, the gradient for the generator can become zero, causing training to stall. This is why balancing the training of both networks is crucial.
3. Non-Convergence: Because GANs are a minimax game rather than a standard minimisation problem, they do not always converge to a stable equilibrium and can oscillate indefinitely.

Frequently Asked Questions

What is the difference between a GAN and a VAE? GANs use an adversarial process to generate data, whereas Variational Autoencoders (VAEs) use probabilistic graphical models and reconstruction loss to learn the underlying data distribution.

Why is the discriminator called an adversary? It is called an adversary because it is in direct competition with the generator; its success is the generator's failure, and vice versa.

Can GANs be used for non-image data? Yes, GANs are versatile and can be applied to audio synthesis, text generation, and time-series forecasting, provided the architecture is adapted to the data type.

Conclusion

Generative Adversarial Networks are a powerful tool in the machine learning toolkit, bridging the gap between simple discriminative models and complex generative systems. By mastering the minimax game and the interplay between the generator and discriminator, you are well-equipped to tackle advanced topics in deep learning. To see these concepts in action with interactive visualisations, generate a free animated lesson on this topic at MathInstructor AI.