“Let’s look at just one final example because it shows us how to handle variables which are independent of other variables.
If we have a function
where x , y and z are independent of each other. What do we mean by independent? We mean that x , y and z can be any value and don’t care what the other variables are – they aren’t affected by changes in the other variables. This wasn’t the case in previous example where y was x 3 + x , so y was dependent on x .
What is ∂f / ∂x ? Let’s look at each part of that long expression. The first bit is 2 xy , so the derivative is 2 y . Why is this so simple? It’s simple because y is not dependent on x . What we’re asking when we say ∂f / ∂x is how does f change when x changes. If y doesn’t depend on x , we can treat it like a constant. That y might as well be another number like 2 or 3 or 10.
Let’s carry on. The next bit is 3 x 2 z . We can apply the power reduction rule to get 2*3 xz or 6 xz . We treat z as just a boring constant number like 2 or 4 or maybe 100, because x and z are independent of each other. A change in z doesn’t affect x .
The final bit 4 z has no x in it at all. So it vanishes completely, because we treat it like a plain constant number like 2 or 4.
The final answer is
The important thing in this last example is having the confidence to ignore variables that you know are independent. It makes doing calculus on quite complex expressions drastically simpler, and it is an insight we’ll need lots when looking at neural networks.
You can do Calculus!
If you got this far, well done!
You have a genuine insight into what calculus really is, and how it was invented using approximations that get better and better. You can always try these methods on other tough problems that resist normal ways for solving them.
The two techniques we learned, reducing powers and the chain rule, allows us to do quite a lot of calculus, including understanding how neural networks really work and why.
Enjoy your new powers!
Appendix B: Do It with a Raspberry Pi
In this section we will aim to get IPython set up on a Raspberry Pi.
There are several good reasons for doing this:
Raspberry Pis are fairly inexpensive and accessible to many more people than expensive laptops.
Raspberry Pis are very open – they run the free and open source Linux operating system, together with lots of free and open source software, including Python. Open source is important because it is important to understand how things work, to be able to share your work and enable others to build on your work. Education should be about learning how things work, and making your own, and not be about learning to buy closed proprietary software.
For these and other reasons, they are wildly popular in schools and at home for children who are learning about computing, whether it is software or building hardware projects.
Raspberry Pis are not as powerful as expensive computers and laptops. So it is an interesting and worthy challenge to be prove that you can still implement a useful neural network with Python on a Raspberry Pi.
I will use a Raspberry Pi Zero because it is even cheaper and smaller than the normal Raspberry Pis, and the challenge to get a neural network running is even more worthy! It costs about £4 UK pounds, or $5 US dollars. That wasn’t a typo!
Here’s mine, shown next to a 2 penny coin. It’s tiny!
We’ll assume you have a Raspberry Pi powered up and a keyboard, mouse, display and access to the internet working.
There are several options for an operating system, but we’ll stick with the most popular which is the officially supported Raspian , a version of the popular Debian Linux distribution designed to work well with Raspberry Pis. Your Raspberry Pi probably came with it already installed. If not install it using the instructions at that link. You can even buy an SD memory card with it already installed, if you’re not confident about installing operating systems.”