March 2019
# Neural Networks

## Hello Sigmoid

Welcome to my **Hello World** of **Backpropagation**. 

This is my tiny and straightforward Haskell implementation of a basic 
neural net using **gradient descent**. I coded this from scratch, along
reading the first two chapters of _Neural Networks and Deep Learning_ [1].

Be warned that the following implementation aims at clarity and readability,
but not performance! In another article I will probably discuss, how to 
optimize it heavily, utilizing _Parallel Programming_ / _Tensor Flow (CUDA)_. 
We can even run it on a cluster someday...

The source code below was auto-fetched from: 
<https://gitweb.softwarefools.com/?p=miguel/haskell.git;a=blob;f=mnist/Neuronet.hs>

~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ {.haskell .numberLines}
{BEGIN:EMBED}
https://gitweb.softwarefools.com/?p=miguel/haskell.git;a=blob_plain;f=mnist/Neuronet.hs
{END:EMBED}
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

## MNIST

Even this simple _vanilla_ network can already give us some tangible 
results, when run on the well-known MNIST Database [2] of handwritten digits,
which contains a set of 60.000 training samples and another set of 
10.000 test samples.

![](digits.png){}

The following screencast shows the success rates of the simple network for
the first ten epochs. (The speed of the cast was increased quite a bit)
 
<asciinema-player loop="1" preload="true" theme="solarized-dark" autoplay="true" src="mnist.cast"></asciinema-player>

I created a little javascript version with 100 random test samples where 
you can test the trained neuronal network inside your browser, if it
supports HTML5. 

* [link to neuroticus.html](neuroticus.html)

## Technical Vocublary

A loose collection of notes and some important terms, you should make 
yourself familiar with, when learning about neural networks.

* sigmoid neurons vs. perceptrons
* recurrent neural networks
* cost / loss / objective function
* required assumptions about the cost function
* quadratic cost function / mean squared error
* cross-entropy cost function (recheck proof)
* binary entropy
* gradient descent 
* gradient (vector of partial derivatives)
* stochastic gradient descent 
* on-line / incremental learning
* deep neural networks
* mini-batches
* backpropagation (based on 4 fudnamental equations / check proofs)
* weighted input
* saturated neuron
* softmax
* log-likehood
* overfitting
* hyper-parameters
* validation data
* regularization (l1,l2,dropout)

## Git Repo

* <https://gitweb.softwarefools.com/?p=miguel/haskell.git;a=tree;f=mnist>

## Ref

* [1] <http://neuralnetworksanddeeplearning.com/>
* [2] <http://yann.lecun.com/exdb/mnist/>
* [3] <http://www.deeplearningbook.org/>
* [4] <https://medium.com/tebs-lab/how-to-classify-mnist-digits-with-different-neural-network-architectures-39c75a0f03e3>
* [5] <https://adeshpande3.github.io/adeshpande3.github.io/The-9-Deep-Learning-Papers-You-Need-To-Know-About.html>