Merge branch 'rnn_exercise' into 'main'

Finished Rnn exercise See merge request !93

Merge branch 'rnn_exercise' into 'main'
d2cdb396 · Daniel Müller · 84a67ea5 · 6414d156 · d2cdb396 · d2cdb396
Commit d2cdb396 authored 3 years ago by Daniel Müller
--- a/notebooks/source/exercises/exercise_recurrent_neural_network.ipynb
+++ b/notebooks/source/exercises/exercise_recurrent_neural_network.ipynb
+{
+ "cells": [
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {
+    "collapsed": true
+   },
+   "outputs": [],
+   "source": [
+    "import numpy as np\n",
+    "import tensorflow as tf\n",
+    "import matplotlib.pyplot as plt\n",
+    "from tensorflow.keras.models import Sequential\n",
+    "from tensorflow.keras.layers import Dense, Lambda, SimpleRNN"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "outputs": [],
+   "source": [
+    "(X_train, y_train), (X_test, y_test) = tf.keras.datasets.fashion_mnist.load_data()"
+   ],
+   "metadata": {
+    "collapsed": false,
+    "pycharm": {
+     "name": "#%%\n"
+    }
+   }
+  },
+  {
+   "cell_type": "markdown",
+   "source": [
+    "# Task:\n",
+    "1. Preprocess dataset\n",
+    "2. Create model\n",
+    "3. Fine tune model to get maximum accuracy possible"
+   ],
+   "metadata": {
+    "collapsed": false
+   }
+  }
+ ],
+ "metadata": {
+  "kernelspec": {
+   "display_name": "Python 3",
+   "language": "python",
+   "name": "python3"
+  },
+  "language_info": {
+   "codemirror_mode": {
+    "name": "ipython",
+    "version": 2
+   },
+   "file_extension": ".py",
+   "mimetype": "text/x-python",
+   "name": "python",
+   "nbconvert_exporter": "python",
+   "pygments_lexer": "ipython2",
+   "version": "2.7.6"
+  }
+ },
+ "nbformat": 4,
+ "nbformat_minor": 0
+}
\ No newline at end of file
+%% Cell type:code id: tags:
+
+``` python
+import numpy as np
+import tensorflow as tf
+import matplotlib.pyplot as plt
+from tensorflow.keras.models import Sequential
+from tensorflow.keras.layers import Dense, Lambda, SimpleRNN
+```
+
+%% Cell type:code id: tags:
+
+``` python
+(X_train, y_train), (X_test, y_test) = tf.keras.datasets.fashion_mnist.load_data()
+```
+
+%% Cell type:markdown id: tags:
+
+# Task:
+1. Preprocess dataset
+2. Create model
+3. Fine tune model to get maximum accuracy possible
--- a/notebooks/source/exercises/solution_recurrent_neural_network.ipynb
+++ b/notebooks/source/exercises/solution_recurrent_neural_network.ipynb
--- a/notebooks/source/recurrent_neural_network.ipynb
+++ b/notebooks/source/recurrent_neural_network.ipynb
@@ -4,9 +4,7 @@
   "cell_type": "markdown",
   "id": "d421e819-8e21-4033-9e21-cdfbe6ff5b48",
   "metadata": {},
-   "source": [
-    "## Creating a Recurrent Neural Netowrk"
-   ]
+   "source": []
  },
  {
   "cell_type": "code",
@@ -17,12 +15,7 @@
    {
     "name": "stderr",
     "output_type": "stream",
-     "text": [
-      "c:\\users\\nikla\\appdata\\local\\programs\\python\\python38\\lib\\site-packages\\numpy\\_distributor_init.py:30: UserWarning: loaded more than 1 DLL from .libs:\n",
-      "c:\\users\\nikla\\appdata\\local\\programs\\python\\python38\\lib\\site-packages\\numpy\\.libs\\libopenblas.GK7GX5KEQ4F6UYO3P26ULGBQYHGQO7J4.gfortran-win_amd64.dll\n",
-      "c:\\users\\nikla\\appdata\\local\\programs\\python\\python38\\lib\\site-packages\\numpy\\.libs\\libopenblas.XWYDX2IKJW2NMTWSFYNGFUWKQU3LYTCZ.gfortran-win_amd64.dll\n",
-      "  warnings.warn(\"loaded more than 1 DLL from .libs:\"\n"
-     ]
+     "text": []
    }
   ],
   "source": [

 %% Cell type:markdown id:d421e819-8e21-4033-9e21-cdfbe6ff5b48 tags:

-## Creating a Recurrent Neural Netowrk

 %% Cell type:code id:24d9924a-f948-4ccf-b8ae-30674dc5fdb7 tags:

 ``` python
 import numpy as np
 import tensorflow as tf
 import pandas as pd
 import matplotlib.pyplot as plt
 from tensorflow.keras.models import Sequential
 from tensorflow.keras.layers import Dense, Dropout, Lambda, SimpleRNN

 tf.random.set_seed(42)
 ```

 %% Output

-    c:\users\nikla\appdata\local\programs\python\python38\lib\site-packages\numpy\_distributor_init.py:30: UserWarning: loaded more than 1 DLL from .libs:
-    c:\users\nikla\appdata\local\programs\python\python38\lib\site-packages\numpy\.libs\libopenblas.GK7GX5KEQ4F6UYO3P26ULGBQYHGQO7J4.gfortran-win_amd64.dll
-    c:\users\nikla\appdata\local\programs\python\python38\lib\site-packages\numpy\.libs\libopenblas.XWYDX2IKJW2NMTWSFYNGFUWKQU3LYTCZ.gfortran-win_amd64.dll
-      warnings.warn("loaded more than 1 DLL from .libs:"

 %% Cell type:markdown id:175f3b1a-04a0-43d7-bb70-987c78e3af54 tags:

 ### Loading Data
 The first step for this Neural Network is getting our Data that we want to learn from.
 We can see that we have 60000 images of size 28x28 in the Train Dataset and

 %% Cell type:code id:2ee80de8-59a4-417e-b83a-f8ce407b7f05 tags:

 ``` python
 # load data and split into train and test sets
 (X_train, y_train), (X_test, y_test) = tf.keras.datasets.mnist.load_data()

 # if the raspberrypi only has 1GB ram it is necessary to reduce the dataset using the following commands
 #data_slice_train = 6000
 #data_slice_test = 1000
 #X_train = X_train[:data_slice_train]
 #y_train = y_train[:data_slice_train]
 #X_test = X_test[:data_slice_test:]
 #y_test = y_test[:data_slice_test:]

 # list of features
 class_names = ["Zero", "One", "Two", "Three", "Four", "Five", "Six", "Seven", "Eight", "Nine"]

 print(f"Training Data Size: {X_train.shape}")
 print(f"Training Data Image shape: {X_train[0].shape}")
 print(f"\nTest Data Size: {X_test.shape}")
 print(f"Test Data Image shape: {X_test[0].shape}")
 ```

 %% Output

    Downloading data from https://storage.googleapis.com/tensorflow/tf-keras-datasets/mnist.npz
    11493376/11490434 [==============================] - 1s 0us/step
    11501568/11490434 [==============================] - 1s 0us/step
    Training Data Size: (60000, 28, 28)
    Training Data Image shape: (28, 28)
    
    Test Data Size: (10000, 28, 28)
    Test Data Image shape: (28, 28)

 %% Cell type:markdown id:f0fee4bd-38e4-4be9-850b-811381115c80 tags:

 Here we plot the first image to see how the first 25 pictures looks

 %% Cell type:code id:fb8539ca-4dcd-4ebf-918d-6e47e53bb713 tags:

 ``` python
 plt.figure(figsize=(10,10))
 for i in range(25):
    plt.subplot(5,5,i+1)
    plt.xticks([])
    plt.yticks([])
    plt.grid(False)
    plt.imshow(X_train[i], cmap=plt.cm.binary)
 plt.show()
 ```

 %% Output



 %% Cell type:markdown id:732f9dd7-1046-4a89-96a6-3f10ef12c00d tags:

 ### Preprocessing
 In order to save memory usage istead of Preprocessing and saving a new copy of the data we created this function.
 This function will normalize the input data in will be incorperated into the Neural Network.

 %% Cell type:code id:e58a989f-dd64-4fd7-8340-f7f7d3e267d4 tags:

 ``` python
 # function to convert the range form 0-255 to 0-1.
 # if this step was applied normaly the new values would be saved and memory would be wasted
 def normalize_img(img):
    return  img / np.float32(255.0)
 ```

 %% Cell type:markdown id:c8e41404-dcfb-4ccb-88c4-2074a263acca tags:

 ### Creating the Model
 It is now time to create our Recurrent Neural Network.
 The first layer is a so called Lambda Layer which applies the normalizing function we defined in the previous step on the input data.

 Afterwards we add a SimpleRNN Layer which is one of 3 built-in RNN layers in tensorflow. The other 2 options are: LSTM and GRU. If you habe a NVIDIA Graphics card you may also use CuDNNLSTM or CuDNNGRU for faster performance.

 Next a Dropout Layer is used to prevent overfitting.

 %% Cell type:code id:6e9a8e54-9091-43a9-81c0-7197a3a8d092 tags:

 ``` python
 # neural network is a sequence of layers
 model = Sequential()

 model.add(Lambda(normalize_img))    # add our "preprocessing" as a layer to save memory
 model.add(SimpleRNN(128, activation='relu', return_sequences=True)) # add the SimpleRNN layer. set return_sequences to true so more rnn layers can be used later. (dont enable if you only use one)
 model.add(Dropout(0.2)) # Dropout-layer to prevent overfitting
 ```

 %% Cell type:markdown id:6aae772d-317a-40bb-9bb1-5ac756e72946 tags:

 Now we Add another SimpleRNN. This second RNN layer can be added to get a deeper neural network and allows the network to learn more complex sequences. this step is entirely optional for the simple training data that we are using and can therefore be skipped.

 %% Cell type:code id:12dfbb02-e42c-470a-9ebf-b4dc6b756abe tags:

 ``` python
 model.add(SimpleRNN(128, activation='relu'))    # Second SimpleRNN layer just to show it works. As it is the last layer return_sequences is flase (default value)
 model.add(Dropout(0.2)) # Another Dropout-layer
 ```

 %% Cell type:markdown id:f978a8d3-73e0-4c64-a600-ace041ab98db tags:

 Finally we add a Dense Layers to convert the output from our RNN layers to a result.

 %% Cell type:code id:60a82154-52b8-4a82-a021-954d87c67acd tags:

 ``` python
 model.add(Dense(10, activation='softmax'))  # Dense Layer to get output of 10 values for the 10 classes
 ```

 %% Cell type:markdown id:0c83b207-e159-4da4-a55d-f0e185758ebc tags:

 ### Compiling the Model

 Here we have to use the loss function "sparse_categorical_crossentropy". Using other loss functions like the mean squared error will give bad results of our network.

 %% Cell type:code id:4717068a-0250-4b21-b709-cd43ca184df5 tags:

 ``` python
 loss = tf.keras.losses.SparseCategoricalCrossentropy()  # loss function for our network
 opt = tf.keras.optimizers.Adam()    # set optimizer to Adam which is the most used optimizer in general (produces good results)
 metrics = ['accuracy']  # our scoring metric is accuracy

 model.compile(loss=loss, optimizer=opt, metrics=metrics)    #compiling the model
 ```

 %% Cell type:markdown id:87fdb35f-9d15-4f1c-8ad3-5060771377c1 tags:

 ### Fit the model


 %% Cell type:code id:00779d5b-ac58-4acd-a75b-e2008f9a45d5 tags:

 ``` python
 model.fit(X_train, y_train, epochs=10, validation_data=(X_test,y_test)) # training for 10 epochs use "test" data as validation
 ```

 %% Output

    Epoch 1/10
    1875/1875 [==============================] - 23s 11ms/step - loss: 0.3977 - accuracy: 0.8751 - val_loss: 0.2014 - val_accuracy: 0.9447
    Epoch 2/10
    1875/1875 [==============================] - 22s 12ms/step - loss: 0.1692 - accuracy: 0.9534 - val_loss: 0.1315 - val_accuracy: 0.9623
    Epoch 3/10
    1875/1875 [==============================] - 21s 11ms/step - loss: 0.1384 - accuracy: 0.9628 - val_loss: 0.0941 - val_accuracy: 0.9738
    Epoch 4/10
    1875/1875 [==============================] - 21s 11ms/step - loss: 0.1206 - accuracy: 0.9678 - val_loss: 0.1151 - val_accuracy: 0.9699
    Epoch 5/10
    1875/1875 [==============================] - 20s 11ms/step - loss: 0.1092 - accuracy: 0.9705 - val_loss: 0.0911 - val_accuracy: 0.9749
    Epoch 6/10
    1875/1875 [==============================] - 20s 11ms/step - loss: 0.1077 - accuracy: 0.9711 - val_loss: 0.1056 - val_accuracy: 0.9711
    Epoch 7/10
    1875/1875 [==============================] - 20s 11ms/step - loss: 0.0972 - accuracy: 0.9736 - val_loss: 0.0965 - val_accuracy: 0.9726
    Epoch 8/10
    1875/1875 [==============================] - 21s 11ms/step - loss: 0.1004 - accuracy: 0.9732 - val_loss: 0.0737 - val_accuracy: 0.9809
    Epoch 9/10
    1875/1875 [==============================] - 21s 11ms/step - loss: 0.0881 - accuracy: 0.9769 - val_loss: 0.0779 - val_accuracy: 0.9799
    Epoch 10/10
    1875/1875 [==============================] - 21s 11ms/step - loss: 0.0862 - accuracy: 0.9768 - val_loss: 0.0762 - val_accuracy: 0.9793

    <keras.callbacks.History at 0x29ebda1cc10>

 %% Cell type:markdown id:ea3acc8d-33a1-4fed-bf46-651210ab70a2 tags:

 Here is the resulting summary of our neural network

 %% Cell type:code id:37171d2b-5490-457a-88ef-8311734669e9 tags:

 ``` python
 print(model.summary()) # summary of the network
 ```

 %% Output

    Model: "sequential"
    _________________________________________________________________
     Layer (type)                Output Shape              Param #
    =================================================================
     lambda (Lambda)             (None, 28, 28)            0
    
     simple_rnn (SimpleRNN)      (None, 28, 128)           20096
    
     dropout (Dropout)           (None, 28, 128)           0
    
     simple_rnn_1 (SimpleRNN)    (None, 128)               32896
    
     dropout_1 (Dropout)         (None, 128)               0
    
     dense (Dense)               (None, 10)                1290
    
    =================================================================
    Total params: 54,282
    Trainable params: 54,282
    Non-trainable params: 0
    _________________________________________________________________
    None

 %% Cell type:markdown id:699c24c7-2199-4d1e-a932-930e06ff0d90 tags:

 ### Testing the Model
 First we evaluate our model using the test data

 %% Cell type:code id:0033276c-8a64-4e56-99a4-a75c60722440 tags:

 ``` python
 test_loss, test_acc = model.evaluate(X_test,  y_test, verbose=2)

 print('\nTest accuracy:', test_acc) # higher is better
 print('Test loss:', test_loss) # lower is better
 ```

 %% Output

    313/313 - 1s - loss: 0.0762 - accuracy: 0.9793 - 1s/epoch - 4ms/step
    
    Test accuracy: 0.9793000221252441
    Test loss: 0.07616104185581207

 %% Cell type:markdown id:75c58982-ed4e-4964-a322-645b02f6e063 tags:

 Now we shall use the test data and save the predictions we get so we can analyze the data.
 Printing out 1 solution gives us a List of 10 Values. These Values represent the probability of each answer beeing correct.
 In this case our model predicts, that the correct answer is 7 with a 99.9% probability.

 %% Cell type:code id:c35c307e-faf8-47f1-93e2-e035a0fa091d tags:

 ``` python
 predictions = model.predict(X_test) # make predictions of the test data
 print(predictions[0])   # show first entry of the solution
 ```

 %% Output

    [1.2902734e-17 1.5439928e-06 5.9133182e-08 1.8422478e-06 1.7637525e-06
     3.2230691e-09 5.8177671e-23 9.9989605e-01 1.4390735e-11 9.8851175e-05]

 %% Cell type:markdown id:22cb363f-e481-4f2b-88b3-c4af642ef291 tags:

 Let's compare this solution with the actual solution:

 %% Cell type:code id:bdb951fb-7b06-4480-a0e7-c2f31fbb1090 tags:

 ``` python
 print("Solution of the Modells: ", np.argmax(predictions[0]))
 print("Correct Solution: ", y_test[0])
 ```

 %% Output

    Solution of the Modells:  7
    Correct Solution:  7

 %% Cell type:markdown id:2f4a4d98-7b4d-48ed-b073-4402cf9341da tags:

 As we can see the solution predicted by the model is correct, however only looking at one entry doesn't give us a good overview.
 I will now define some functions to plot out the data we are looking at.

 %% Cell type:code id:28403290-6b52-46ed-9eb6-a4438a1aae9d tags:

 ``` python
 def plot_image(i, predictions_array, true_label, img):
  true_label, img = true_label[i], img[i]
  plt.grid(False)
  plt.xticks([])
  plt.yticks([])

  plt.imshow(img, cmap=plt.cm.binary)

  predicted_label = np.argmax(predictions_array)
  if predicted_label == true_label:
    color = 'blue'
  else:
    color = 'red'

  plt.xlabel("{} {:2.0f}% ({})".format(class_names[predicted_label],
                                100*np.max(predictions_array),
                                class_names[true_label]),
                                color=color)

 def plot_value_array(i, predictions_array, true_label):
  true_label = true_label[i]
  plt.grid(False)
  plt.xticks(range(10))
  plt.yticks([])
  thisplot = plt.bar(range(10), predictions_array, color="#777777")
  plt.ylim([0, 1])
  predicted_label = np.argmax(predictions_array)

  thisplot[predicted_label].set_color('red')
  thisplot[true_label].set_color('blue')
 ```

 %% Cell type:markdown id:7678da36-0f0b-4514-b291-39def73a09a3 tags:

 Using those functions we can now show the following graphically.

 It will show you the Image that is being predicted. Below that it will show you What it predicted, aswell as the Percentage of certinty. In Brackets you will see the True Answer.
 To the right of the Image you will see a Graph showing the entire Answer of the Modell.

 %% Cell type:code id:93f0c451-d448-4e8e-8522-6f357d8ce036 tags:

 ``` python
 num_rows = 10
 num_cols = 4
 num_images = num_rows*num_cols
 plt.figure(figsize=(2*2*num_cols, 2*num_rows))
 for i in range(num_images):
  plt.subplot(num_rows, 2*num_cols, 2*i+1)
  plot_image(i, predictions[i], y_test, X_test)
  plt.subplot(num_rows, 2*num_cols, 2*i+2)
  plot_value_array(i, predictions[i], y_test)
 plt.tight_layout()
 plt.show()
 ```

 %% Output



 %% Cell type:markdown id: tags:

 Now i will load an image of a number i drew myself to see if the network can identify it correctly.

 %% Cell type:code id:254fbca7-c9ca-4d12-b10b-2d52a4fa7dcd tags:

 ``` python
 from PIL import Image
 imframe = Image.open('../data/RNN_own_numbers/eight.png')
 npframe = np.array(imframe.getdata())
 reshapeframe = np.reshape(npframe, (28, 28))
 array = []
 array.append(reshapeframe)
 test_data = np.asarray(array)

 imgplot=plt.imshow(test_data[0])
 plt.show()
 ```

 %% Output



 %% Cell type:markdown id: tags:

 Now we test this image. You can use your own numbers to test the network. Just make a 28*28 Image and imported above

 %% Cell type:code id:acdeca6b-6164-4eea-b477-1d7192e1b8ff tags:

 ``` python
 test_prediction = model.predict(test_data)
 print(test_prediction)
 print(test_prediction.shape)
 print("Solution of the Modells: ", np.argmax(test_prediction[0]))
 print("Correct Solution: ", 8)
 ```

 %% Output

    [[0.08659085 0.11173349 0.21166202 0.06632368 0.1174572  0.05388441
      0.13200265 0.03325519 0.14504877 0.04204175]]
    (1, 10)
    Solution of the Modells:  2
    Correct Solution:  8