blob: e298046bab557f1bbea22073e26b4720f441e8d4 (
plain)
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
|
March 2019
# Neural Networks
## Hello Sigmoid
Welcome to my **Hello World** of **Backpropagation**.
This is my tiny and straightforward Haskell implementation of a basic
neural net using **gradient descent**. I coded this from scratch, along
reading the first two chapters of _Neural Networks and Deep Learning_ [1].
Be warned that the following implementation aims at clarity and readability,
but not performance! In another article I will probably discuss, how to
optimize it heavily, utilizing _Parallel Programming_ / _Tensor Flow (CUDA)_.
We can even run it on a cluster someday...
The source code below was auto-fetched from:
<https://gitweb.softwarefools.com/?p=miguel/haskell.git;a=blob;f=mnist/Neuronet.hs>
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ {.haskell .numberLines}
{BEGIN:EMBED}
https://gitweb.softwarefools.com/?p=miguel/haskell.git;a=blob_plain;f=mnist/Neuronet.hs
{END:EMBED}
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
## MNIST
Even this simple _vanilla_ network can already give us some tangible
results, when run on the well-known MNIST Database [2] of handwritten digits,
which contains a set of 60.000 training samples and another set of
10.000 test samples.
{}
The following screencast shows the success rates of the simple network for
the first ten epochs. (The speed of the cast was increased quite a bit)
<asciinema-player loop="1" preload="true" theme="solarized-dark" autoplay="true" src="mnist.cast"></asciinema-player>
I created a little javascript version with 100 random test samples where
you can test the trained neuronal network inside your browser, if it
supports HTML5.
* [link to neuroticus.html](neuroticus.html)
## Technical Vocublary
A loose collection of notes and some important terms, you should make
yourself familiar with, when learning about neural networks.
* sigmoid neurons vs. perceptrons
* recurrent neural networks
* cost / loss / objective function
* required assumptions about the cost function
* quadratic cost function / mean squared error
* cross-entropy cost function (recheck proof)
* binary entropy
* gradient descent
* gradient (vector of partial derivatives)
* stochastic gradient descent
* on-line / incremental learning
* deep neural networks
* mini-batches
* backpropagation (based on 4 fudnamental equations / check proofs)
* weighted input
* saturated neuron
* softmax
* log-likehood
* overfitting
* hyper-parameters
* validation data
* regularization (l1,l2,dropout)
## Git Repo
* <https://gitweb.softwarefools.com/?p=miguel/haskell.git;a=tree;f=mnist>
## Ref
* [1] <http://neuralnetworksanddeeplearning.com/>
* [2] <http://yann.lecun.com/exdb/mnist/>
* [3] <http://www.deeplearningbook.org/>
* [4] <https://medium.com/tebs-lab/how-to-classify-mnist-digits-with-different-neural-network-architectures-39c75a0f03e3>
* [5] <https://adeshpande3.github.io/adeshpande3.github.io/The-9-Deep-Learning-Papers-You-Need-To-Know-About.html>
|
