Backpropagation Without Computational Graphs

I often find myself encountering two kinds of explanations for backpropagation, one of the fundamental algorithms behind deep learning.

The first is what I would call the Karpathy style , which relies heavily on computational graphs and visualizations. The second is the more rigorous mathematical approach found in university textbooks.

Both are useful, but neither works particularly well for me. I find it difficult to follow computational graphs, while textbook explanations can introduce too much mathematical formalism at once.

I wanted to try something different.

If, like me, you're not particularly visual, perhaps working through a succession of equations can provide a useful middle ground.

Let's start with some definitions.

Definitions

Backpropagation (short for backward propagation of errors) is an algorithm for computing the derivatives of a loss function with respect to the parameters of a neural network.

A gradient is a vector containing the partial derivatives of a scalar-valued function with respect to its inputs.

OK, and?

These definitions aren't particularly helpful on their own.

The idea is relatively simple.

A derivative tells us how a function changes when we change one variable. When a function depends on several variables, its gradient collects these derivatives into a vector.

The gradient points in the direction of the steepest local increase of the function.

This is useful in deep learning because we want to minimize a loss function. By moving the model's parameters in the opposite direction of the gradient, we can try to reduce that loss.

There is an important distinction here: backpropagation computes the gradients, while an optimization algorithm, such as gradient descent, uses those gradients to update the model's parameters.

But how do we calculate these gradients when a function contains many operations?

We can break the process into four steps:

1. Decompose the expression into smaller equations.
2. Perform the forward pass.
3. Calculate the local derivatives.
4. Backpropagate the gradients using the chain rule.

We don't need to draw a computational graph to do this.

Example 1: Two Independent Variables

Let's take a simple expression:

\[ x=3,\quad y=5 \]

\[ L=x^2\times10y+3 \]

Our objective is to calculate:

\[ \frac{dL}{dx}\quad\text{and}\quad\frac{dL}{dy} \]

1. Decompose the expression

Instead of working with the complete expression, we break it into smaller equations.

\[ \begin{aligned} a&=x^2 \\[0.35em] b&=10y \\[0.35em] c&=a\times b \\[0.35em] L&=c+3 \end{aligned} \]

Each equation represents a simple operation whose derivative we can calculate independently.

This gives us the same mathematical dependencies that a computational graph would represent, without having to visualize one.

2. Forward pass

Now we calculate the value of each intermediate variable.

\[ \begin{aligned} a&=3^2=9 \\[0.35em] b&=10\times5=50 \\[0.35em] c&=9\times50=450 \\[0.35em] L&=450+3=453 \end{aligned} \]

Our final result is:

\[ L=453 \]

We also know the values of all the intermediate variables, which we will need when calculating derivatives.

3. Calculate the local derivatives

Now we differentiate each equation separately, using the usual derivative rules.

\[ \begin{aligned} \frac{da}{dx}&=2x=6 \\[0.35em] \frac{db}{dy}&=10 \\[0.35em] \frac{\partial c}{\partial a}&=b=50 \\[0.35em] \frac{\partial c}{\partial b}&=a=9 \\[0.35em] \frac{dL}{dc}&=1 \end{aligned} \]

Notice that these are local derivatives.

For example:

\[ \frac{\partial c}{\partial a}=50 \]

This tells us how \(c\) changes when \(a\) changes, with \(b\) held constant.

It doesn't yet tell us how the final loss changes when \(x\) changes.

To calculate that, we need the chain rule.

4. Backpropagate the gradients

We start from the final result:

\[ \frac{dL}{dL}=1 \]

Since \(L=c+3\), we have:

\[ \frac{dL}{dc}=1 \]

Using the chain rule, we multiply the incoming gradient by the local derivative.

First:

\[ \frac{dL}{da}=\frac{dL}{dc}\times\frac{\partial c}{\partial a}=1\times50=50 \]

Then:

\[ \frac{dL}{db}=\frac{dL}{dc}\times\frac{\partial c}{\partial b}=1\times9=9 \]

Finally, we reach our original variables.

\[ \frac{dL}{dx}=\frac{dL}{da}\times\frac{da}{dx}=50\times6=300 \]

\[ \frac{dL}{dy}=\frac{dL}{db}\times\frac{db}{dy}=9\times10=90 \]

We have calculated both derivatives:

\[ \boxed{\frac{dL}{dx}=300,\quad\frac{dL}{dy}=90} \]

And we did it without drawing a graph.

Example 2: When a Variable Appears More Than Once

Let's try something slightly harder.

This time, we'll use the same variable twice in our expression.

\[ x=5 \]

\[ L=x^2\times10x+3 \]

The four steps remain the same.

1. Decomposition

\[ \begin{aligned} a&=x^2 \\[0.35em] b&=10x \\[0.35em] c&=a\times b \\[0.35em] L&=c+3 \end{aligned} \]

Notice that both \(a\) and \(b\) depend on \(x\).

This will become important during backpropagation.

2. Forward pass

\[ \begin{aligned} a&=5^2=25 \\[0.35em] b&=10\times5=50 \\[0.35em] c&=25\times50=1250 \\[0.35em] L&=1250+3=1253 \end{aligned} \]

3. Local derivatives

Just as before, we calculate the derivative of each operation.

\[ \begin{aligned} \frac{da}{dx}&=2x=10 \\[0.35em] \frac{db}{dx}&=10 \\[0.35em] \frac{\partial c}{\partial a}&=b=50 \\[0.35em] \frac{\partial c}{\partial b}&=a=25 \\[0.35em] \frac{dL}{dc}&=1 \end{aligned} \]

4. Backpropagation

We initialize:

\[ \frac{dL}{dL}=1 \]

Since \(L=c+3\):

\[ \frac{dL}{dc}=1 \]

Then:

\[ \frac{dL}{da}=\frac{dL}{dc}\times\frac{\partial c}{\partial a}=1\times50=50 \]

\[ \frac{dL}{db}=\frac{dL}{dc}\times\frac{\partial c}{\partial b}=1\times25=25 \]

Now we encounter something different.

Both \(a\) and \(b\) depend on \(x\).

When \(x\) changes, it changes both intermediate variables, and each of these changes affects the final loss.

Therefore, we must account for both contributions.

Using the chain rule:

\[ \frac{dL}{dx}=\frac{dL}{da}\times\frac{da}{dx}+\frac{dL}{db}\times\frac{db}{dx} \]

Substituting our values:

\[ \frac{dL}{dx}=50\times10+25\times10 \]

\[ \boxed{\frac{dL}{dx}=750} \]

This is gradient accumulation.

When a variable contributes to the final result through multiple operations, its total gradient is the sum of the contributions from those operations.

In this example, we cannot calculate the gradient with respect to \(x\) by considering only \(a=x^2\) or only \(b=10x\).

We must include both because changing \(x\) affects the loss through both intermediate variables.

Checking our result

We can verify our calculation by simplifying the original expression.

\[ L=x^2\times10x+3 \]

\[ L=10x^3+3 \]

Differentiating directly:

\[ \frac{dL}{dx}=30x^2 \]

For \(x=5\):

\[ \frac{dL}{dx}=30\times25=750 \]

We obtain the same result.

Conclusion

Backpropagation doesn't require us to think in terms of computational graphs.

We can work through a succession of equations:

1. Decompose a complex expression into simpler operations.
2. Evaluate each operation during the forward pass.
3. Calculate its local derivatives.
4. Apply the chain rule backward, accumulating gradients when a variable contributes through multiple operations.

The same principles apply to neural networks, although the operations become more complex and the variables are often tensors rather than scalars.

In practice, frameworks such as PyTorch automate this process using automatic differentiation. They keep track of the operations performed during the forward pass and use the chain rule to compute gradients during the backward pass.

For me, thinking in equations makes backpropagation easier to follow. Instead of trying to visualize how gradients move through a graph, I can focus on one derivative at a time.