{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Classification problems with Keras"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Classification is the task of reconize some item from a set of candidate targets. This is a very common task for Deep Learning technique, which is very well performed for example with image recogniction.\n",
    "\n",
    "Yet, image recognition (which is a type of classification problem) takes as input images, which are composed of pixels distributed on a 2D dimension. We cannot simply unroll 2D image data into a 1D array of input, because spatial structure of the images is very important, information is in the structure of the data.\n",
    "\n",
    "That's why when dealing with 2D problems, and particularly with images, we turn from simple Dense layers (which treat input as a 1D stream of data) to Convolutional layers."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Convolutional Neural Network (CNNs)\n",
    "\n",
    "(material from: <a href=\"http://neuralnetworksanddeeplearning.com/chap6.html#introducing_convolutional_networks\">Neural Networks and Deep Learning - Chapter 6</a>)\n",
    "\n",
    "Convolutional neural networks are biologically inspired variants of multilayer perceptrons, designed to emulate the behaviour of a visual cortex. CNNs are multi-layered feed-forward neural networks that are able to learn task-specific invariant features in a hierarchical manner. These models mitigate the challenges posed by the classic Multi-Layer Perceptron (MLP) architecture by exploiting the strong spatially local correlation present in natural images.\n",
    "\n",
    "Convolutional neural networks (CNNs) use three basic ideas:\n",
    "- local receptive fields\n",
    "- shared weights\n",
    "- pooling."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Local Receptive Fields\n",
    "we'll connect the input pixels to a layer of hidden neurons. But we won't connect every input pixel to every hidden neuron. Instead, we only make connections in small, localized regions of the input image. each neuron in the first hidden layer will be connected to a small region of the input neurons, say, for example, a 5×5 region, corresponding to 25 input pixels. So, for a particular hidden neuron, we might have connections that look like this:\n",
    "<img src=\"imgs/convolution-stencil.png\" width=\"50%\">\n",
    "That region in the input image is called the local receptive field for the hidden neuron. It's a little window on the input pixels. Each connection learns a weight. And the hidden neuron learns an overall bias as well. You can think of that particular hidden neuron as learning to analyze its particular local receptive field.\n",
    "\n",
    "We then slide the local receptive field across the entire input image. For each local receptive field, there is a different hidden neuron in the first hidden layer. To illustrate this concretely, let's start with a local receptive field in the top-left corner:\n",
    "<img src=\"imgs/convolution-stencil-sliding.png\" width=50%>\n",
    "the starting input image is a 28x28 pixel, using a convolution kernel of size 5x5, we map into an 24x24 output pixel image"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Shared weights\n",
    "\n",
    "Each hidden neuron share the same weights and bias. This means that all the neurons in the first hidden layer detect exactly the same feature. Think of the feature detected by a hidden neuron as the kind of input pattern that will cause the neuron to activate: it might be an edge in the image, for instance, or maybe some other type of shape, just at different locations in the input image.\n",
    "\n",
    "Yes, convolutional networks are well adapted to the translation invariance of images.\n",
    "\n",
    "To do image recognition we'll need more than one feature map. And so a complete convolutional layer consists of several different feature maps. For example, if we use 3 different convolutional layers, than we can map 3 different features. \n",
    "<img src=\"imgs/convolution-features.png\" width=50%>\n",
    "Each feature map is defined by a set of shared weights, and a single shared bias. The result is that the network can detect 3 different kinds of features, with each feature being detectable across the entire image.\n",
    "\n",
    "A big advantage of sharing weights and biases is that it greatly reduces the number of parameters involved in a convolutional network. For each feature map of our previous example we need 25=5×5 shared weights, plus a single shared bias. So each feature map requires 26 parameters. If we have 20 feature maps that's a total of 20×26=520 parameters defining the convolutional layer. By comparison, suppose we had a fully connected first layer, with 784=28×28 input neurons, and a relatively modest 30 hidden neurons. That's a total of 784×30 weights, plus an extra 30 biases, for a total of 23,550 parameters. The fully-connected layer would have more than 40 times as many parameters as the convolutional layer."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Convolutional layers\n",
    "\n",
    "Kesar provides <code><font color='red'>layers.Conv{<em>N</em>}D</font></code>  layer objects for convolution, where <em>N</em> is the number of dimension of the features for each input (1D, 2D, 3D). \n",
    "\n",
    "Important parameters are:\n",
    "- <code>filters</code>: number of output filters (features to catch)\n",
    "- <code>kernel_size</code>: size of the convolution window\n",
    "- <code>strides</code>: define window shifting step (default 1)\n",
    "- <code>padding</code>: One of \"valid\" or \"same\" (case-insensitive)\n",
    "  - valid: no padding, stop at last valid element in window\n",
    "  - same: use padding (default)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Pooling Layers\n",
    "Pooling is used in convolutional neural networks to make the detection of certain features invariant to scale and orientation changes. Pooling generalise over lower level, more complex information.\n",
    "\n",
    "Pooling layers are usually used immediately after convolutional layers. \n",
    "\n",
    "Kesar provides <code><font color='red'>layers.Pooling2D</font></code>  layer objects for this purpose. Important parameters are:\n",
    "- <code>pool_size</code>: size of the max pooling windows.\n",
    "- <code>strides</code>: downscaling factor. If None, it will default to pool_size.\n",
    "- <code>padding</code>: One of \"valid\" or \"same\" (case-insensitive).\n",
    "  - valid: valid: no padding, stop at last valid element in window\n",
    "  - same: use padding (default)\n",
    "  \n",
    "For example, a max-pooling layer with <code>pool_size=(2,2)</code> and <code>stride=None</code> will reduce an input 81x81 to an 41x41 output. Using <code>padding='valid'</code> will reduce output to 40x40, because no extra padding will be applied on original input shape."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "\n",
    "<img src=\"imgs/Max-pooling.jpg\" width=400pt>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Dropout Layers\n",
    "Recently the pooling approach has been often substituted or combined with a dropout. The drop out layer acts during training only. It simply turn off random fraction of input into the layer below, which helps prevent overfitting.\n",
    "\n",
    "Kesar provides <code><font color='red'>layers.Dropout</font></code>  layer objects for this purpose. Important parameters are:\n",
    "- <code>rate</code>: fraction of input unit to drop.\n",
    "- <code>noise_shape</code>: shape of the binary dropout mask.\n",
    "- <code>seed</code>: for random initialization."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## MNIST Benchmark\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The MNIST data set is a standard set of handwritten numerical digits from 0 to 9 which is commonly used as the \"Hello World\" test for Deep Learning classification problem.\n",
    "\n",
    "Keras comes with many dataset built in and automatically splits the data into a training and validation set."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "# python module party :-)\n",
    "%matplotlib inline\n",
    "\n",
    "import numpy as np\n",
    "\n",
    "import tensorflow as tf\n",
    "from tensorflow import keras\n",
    "\n",
    "from tensorflow.keras.models import Sequential\n",
    "from tensorflow.keras.layers import Dense, Activation\n",
    "from tensorflow.keras.layers import Dropout, Flatten\n",
    "from tensorflow.keras.layers import Conv2D, MaxPooling2D\n",
    "from tensorflow.keras import backend as K\n",
    "\n",
    "# here we get access to the Keras MNIST dataset\n",
    "from tensorflow.keras.datasets import mnist\n",
    "\n",
    "# fix random seed for reproducibility\n",
    "seed = 7\n",
    "np.random.seed(seed)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "# the data, shuffled and split between train and test sets\n",
    "(x_train, y_train), (x_test, y_test) = mnist.load_data()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can rapidly develop a convolutional neural network in order to experiment with our image classification task. The first step will be to pre-process the data into a form that can be fed into a keras model"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Input dataset has shape (60000, 28, 28) and type uint8\n"
     ]
    }
   ],
   "source": [
    "# let's have a look at the shape of the input training set\n",
    "print(\"Input dataset has shape\", x_train.shape, \"and type\", x_train.dtype)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Input data are numpy array of 60K samples each composed by a 28x28 pixel grayscale images."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAXIAAAD1CAYAAABN5n7dAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDMuMC4xLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvDW2N/gAAIABJREFUeJzt3WecVEXaxuH/iKuiiCAiYsKIGAEVRZcXEUFdM2JiMYAZVwwrRsw55wBrjqsYURTMiHkx4KpgFl0jigkB87wf/N1d02cy0336VM99fZmh+3R39aG75jlVTz1VUVlZiZmZxWu+UjfAzMyaxh25mVnk3JGbmUXOHbmZWeTckZuZRc4duZlZ5NyRm5lFzh25mVnk3JGbmUVu/jRfrKKiolksI62srKxo6LE+J9X5nNTM56U6n5M/OSI3M4ucO3Izs8i5Izczi5w7cjOzyLkjNzOLnDtyM7PIpZp+aNmy3nrrAXDwwQcDsOeeewJw0003AXDZZZcB8Morr5SgdWbWUI7IzcwiV5HmVm9pJO+3aNECgMUWW6zG+xV9LrzwwgCsttpqAPzjH/8A4Pzzzwdg0KBBucf89NNPAJx99tkAnHLKKXW2IesLGrp16wbAE088AUDr1q1rPO77778HoF27dk1+zayfk8babLPNALj11ltzt22yySYAvP322w16jnJYEHT88ccD4Tsx33x/xoZ9+vTJHfPUU0816jnL7bNSCF4QZGZW5qIbI19++eUBWGCBBQDYeOONAejVqxcAbdq0AWDgwIENer5PPvkEgEsvvRSAAQMGADBr1qzcMa+99hrQ+MgiazbYYAMA7r77biBcteiqTO/5l19+AUIk3rNnTyB/rFzHlELv3r2B0L5777039Tb06NEDgMmTJ6f+2lkwZMgQAI4++mgA/vjjj7z7val7uhyRm5lFLoqIXGO6EMZ1axsDbyhFEBrj+/HHH4Ew5vn555/njv3222+Bho99ZoXmAdZdd10AbrnlFgA6duxY4/HvvvsuAOeeey4At99+OwDPPvssEM4VwFlnnVWEFjeMxl9XXXVVIN2IXGPAK664IgCdOnXK3VdR0agh76jpfS+00EIlbknxbbjhhgDsvvvuQJgLWXPNNfOOGzFiBACfffYZEEYJ9L178cUXi9ZGR+RmZpFzR25mFrkohlY+/vjj3O8zZ84EGj60osuZ7777DoBNN90UCJN1N998c8HamTWjR48G8lMp66IhmFatWgFhcldDGeuss06BWzhvtHDp+eefT/21NSy13377AeGyGeCtt95KvT1p69evHwDDhw/Pu13vfZtttgHgyy+/TLdhRbDrrrsCcMkllwCwxBJLAGEIbeLEiQC0b98egPPOOy/v8TpO9++2225Fa6sjcjOzyEURkX/zzTe534888kgg/OV/9dVXgZA+KFOmTAGgf//+AMyePRsIExSHHnpoEVtcWlp6v/XWWwPVJ+EUaT/wwANAWASlSRqdU03y9u3bt8bnKRVNOJbCNddck/dvTRCXO03cXX/99UD1K2JFox999FG6DSug+ef/sztcf/31Abj66quBkDQwadIkAE477TQAnnnmGQAWXHBBAMaMGQPA5ptvnve8L730UjGbDTgiNzOLXhQReVX33XcfENIQtYila9euAOyzzz5AiDIVicubb74JwP7771/8xqZMaZqPPvooEJbea3HG+PHjgTBmrjQqpRUq2vzqq6+AsBBKqZqK8CGMp6dZUEtj9B06dEjtNZOSkajOdbnba6+9AFh66aXzbtc4sQqtxUzphcmrLv0fa8z8hx9+yLtftycjcS02vPHGGwvf2ARH5GZmkYsuIpfkX0UVeBJlFdxxxx1A9SXE5aRz585AmD9Q1Pj1118DYXGTIgMtfnrwwQfzftanZcuWud+POOIIAAYPHtyktjfGVlttVa0dadFVgBYCyaeffpp6W9KkTI29994bCN8jZYGdfvrppWlYAWnM+7jjjgPCFeyVV14JhCvWZJ8jI0eOrPH2Qw45BAhXuMXkiNzMLHLRRuRJJ598MhAyNjT+q7zXRx55pCTtKhbNlEOYD1DEqnkD5Vtr1ryQkayKl6VJJYdF8x1p0DlWZP7OO+8A+cXVyskKK6wAhAJrSdp05Mknn0yrSQV14okn5n5XJK61JQ8//DAQCoLNnTs377EqS6AxcX0XlNWlq5SxY8cWpe01cURuZha5sonIlZ2isXFlUygXVJGDotMrrrgCiLfcZvfu3XO/KxKX7bffHoi/7G59ilFCVpk+W265JRAyGZIZCRpX1VhxudH7T67mffzxx4Gw2jE2KnN90EEH5W5TH6BIfIcddqjxsaussgoQCuvp6l/uuusuIBSdS5MjcjOzyJVNRC7vv/8+EArfayXaHnvskfdzkUUWAUL+a9WytTG48MILc79rbE4ReKEjca2kzFrmz+KLL17vMVpfoHOkOZNll10WCBuUKPtG71XjoqrV8/PPPwNh9d/LL7/c9DeQQYpGta2haBWj8smTWWKx0P+3snGqUpbJkksuCcDQoUMB2G677QBYa621gFCLSJG8fqruTnLtShockZuZRa7sInLRZgOqhaEIVpvmnnnmmUAokH/GGWcA2c8LVo2ZqpttKCK4//77i/KaisSrzieolk2aFCWrHaNGjQJC1kFNNMariPy3334DYM6cOQBMnToVgOuuuw4Icyi6qlEVP63SU+ZPuVU6rC9L5YMPPgDir2qozJSqud2qTvjhhx8Ctc+bqRaR8slVCVPrNVS7qBQckZuZRa5sI3J54403ANhll10A2HbbbYEwdn7AAQcAYdswVUvMKkWEGusDmDFjBhBWsTaVctSVmy+qbwNw7LHHFuS1GkOZBqqwp42366Ja9qrRM23aNABeeOGFBr2mavIoalNkWm5q20RZkmPmsVKWUdXMlHHjxgFhzkXzbMoDv+GGG4BQhVVbICoi179LyRG5mVnkyj4iF/0l1o5AqnCmLITevXsDYTccVXWLgTIqmpp5o0hctSVUu0XjwxdccEHuWNVrKYVzzjkntdfSnIrUNoYcK821JPPkRVFpbBuP16fqRsi62qqP+gitGtfVSxau0hyRm5lFruwjcmUt7LTTTgD06NEDCJG4KHtBu4DEpKnZKorKFIGrvrKisYEDBzbp+cuJsqHKhWoQtW3bNu92zSFoPYaF+alkFpfHyM3MrMnKLiJXhbyDDz4YgB133BGApZZaqsbjf//9dyCML2dt9WKS8qGr7p+pGfjG7kN6+OGHA3DCCScAoY65akmoeqKVr3bt2gHVP/eqxV3KuZCsUS2WLHJEbmYWuegjckXa2odSkbhWqtVGK/i0orNYqyILLVnfAcI5uPTSS4GwSnHmzJkA9OzZEwh1ZlR/RPVGlGutiEPRmAW6AtJuTA3NQ88qraNQbZmk5557Ls3mRGGLLbYodRNq5YjczCxy0UXk2qFljTXWAODyyy8HoEuXLnU+Tnmj5513HhAyMrI+Jt4QLVq0AMLKR2WZqCaEVq0mKepSrfaqu6ZYPl0B1RbBxkIZSqoCqc+/apCoTn/sNVWKYaWVVip1E2oV96fSzMzckZuZxS7TQysqYjN69Ojcbbo0rO8yR8MGWlauibzkRqqxef7554H8bc60yEk0+alhKNHkpxYwNDZd0WCjjTYCQiGl2Girs2Q6rso3jxgxIvU2xeLpp58GsrnRiiNyM7PIZSoi33DDDYGwVHyDDTYAYJlllqn3sdooQCl42jiiFNsuFZMKWGmhE4RSvCp2laSNcq+66ioA3nvvvWI2sSxVXYBlzZNKYmuzGo0KrLzyykD+ZhVpc0RuZha5TEXkAwYMyPtZExW3UjF4bd2lsXCVqy13VUvWagOI5EYQ1nTjx48HYOeddy5xSwpDW9RpDqlXr16lbE6UdLWvUthaVDh8+HAg9FFpckRuZha5ito2Gi3Ki1VUpPdiJVRZWdngAVWfk+p8Tmrm81JdKc5J69atARgzZgwQFlfdc889AAwdOhQo7PxcfefEEbmZWeQckRdB1iOKUvA5qc4Rec1i+awoMtcY+bBhw4CwmU0hx8odkZuZlTlH5EUQS0SRJp+T6hyR18yfleockZuZlblUI3IzMys8R+RmZpFzR25mFjl35GZmkXNHbmYWOXfkZmaRc0duZhY5d+RmZpFzR25mFjl35GZmkXNHbmYWOXfkZmaRc0duZhY5d+RmZpFzR25mFjl35GZmkXNHbmYWOXfkZmaRc0duZha5+dN8MW+UWp3PSXU+JzXzeanO5+RPjsjNzCLnjtzMLHLuyM3MIueO3Mwscu7Izcwi547czCxy7sjNzCKXah65lcYll1wCwCGHHALAG2+8AcA222wDwEcffVSahpk1U48//jgAFRV/pof37du3Sc/niNzMLHLNJiJfdNFFAWjVqhUAW2+9NQDt27cH4MILLwTg559/LkHrimOFFVYAYPfddwfgjz/+AGD11VcHoEuXLkDzisg7d+4MwF/+8hcAevfuDcCVV14JhHNUn7FjxwKw22675W775ZdfCtbOUtF52XjjjQE488wzAfjrX/9asjaVi4suuij3u87vTTfdVJDndkRuZha5so3IFY0effTRAGy00UYArLXWWjUe37FjRyCMI5eDr776CoBJkyYBsN1225WyOSWx5pprAjBkyBAAdt55ZwDmm+/PGGbppZcGQiReWdmw0h06l6NGjcrddthhhwHwww8/NLHVpbPYYosB8OSTTwLwxRdfALDUUkvl/dsa7uyzzwbgwAMPzN3266+/AmGsvKkckZuZRa5sInKN9yoqGjx4MAAtW7YEwuzw//73PwBmzZoFhPHiXXbZBQhjpW+99VYazS6q2bNnA81rDDzprLPOAmCrrbYqyvPvueeeud+vvfZaAJ599tmivFYpKBJ3RD7vevbsCYT5B4BnnnkGgDFjxhTkNRyRm5lFzh25mVnkoh1a0aTMOeecA8Cuu+4KhDTDpHfffReALbbYAgiXORpCWWKJJfJ+loM2bdoA0LVr1xK3pHQeffRRoPrQyowZM4AwHKLJz2T6odLENtlkk6K2M6s0JNmcKUV15MiRAAwaNAiAb775ps7H6TglWLz//vu5+0aMGFHQNjoiNzOLXLQR+YABAwDYd9996zxOfwX79+8PhMnOVVZZpYity4aFF14YgOWXX77G+3v06AGEq5JynBS96qqrALjvvvvyblf6V32Td61btwZCWQOlK0rV533ppZea1tgMUjrmQgstVOKWlM6//vUvAFZddVUA1lhjDSBMWNbmuOOOA6Bdu3YA7Lfffrn7XnvttYK20RG5mVnkoo3ItbAjafr06QBMnjwZCAuCFImL0g7L2WeffQbADTfcAMDJJ5+cd7/+/d133wFw+eWXp9W01Pz2229A9f//htKcStu2bWu8/5NPPsn9Xk7lHZLWX399AF544YUStyR9c+bMARp+ddKtWzcAOnXqBIR5l2Je1TgiNzOLXLQRucab9t9/fwAeeeQRAN577z0gZCXUpkOHDkVsXbacdtppQPWI3GqnYlj6nGlhWdKJJ56YWpvSoCuY77//HgjZYSuvvHLJ2lQq+t6svfbaAEybNg2ofXx7kUUWAcIogOaodBVz1113Fa2tjsjNzCIXbUSu8d95jTJVRKs5qS1X2kJJh2OOOQYIWU1Vl1VXNWXKFCBkv5QLzZc8/fTTQNh8pDlZbrnlgHA1pquUgw8+GAjF6JJUClvzd+qj0igB7IjczCxy0Ubk9VE5Wo1bJWncS5577jkAnn/++eI2rIQaW6q1HKic8R577AFAv379ajyuV69eQO3nRqVpFbE/9NBDAMydO7dgbbXS0grMe++9FwirvC+77DIAnnrqqRofp1WaKpUsZ5xxRjGaWSNH5GZmkYs+ItfMsFZbnXTSSUD12hq1jQ9rHGvo0KEA/P7778VrrKVG0dX9998P1L66taE0ZqxVfs2NVieWi/nnD12ftkKsre6O5tOOPfZYIIyFL7744kAYE1ddGm3fNnr06OK9gQRH5GZmkYsuIlcWQffu3QG4++67gbBVm8YsFWlrzHvLLbcEQgQv+su84447AnDJJZcA5bGRroUoqb4qfvVl9Ch7429/+xsA48ePL1QTo1Bu2wRW3TT7mmuuAcL8iD4DWpOiVa36uf322wOwzDLLAKHvUTbL3nvvXdS218QRuZlZ5KKIyBdYYIHc74qs77nnnrxjTjnlFACeeOIJIGy3pXEs3Z7cfLl9+/ZA2BLs448/BvKr2pVLDY3aok7VWy6nWiuqVtinTx8gjIM+/PDDAPz00091Pn6fffYBYPjw4UVqYbZp8+VyyyPXvgXXX3997jatBVAO/d///ncAvv32WwAuuOACINSkV2SuqzxF8spyUV0fffaq1iEvFkfkZmaRq0gzp7iioqJRL6bx8FNPPTV325FHHpl3jMYqlSesv6qKtJXvu+666wJh7Pvcc88FQoSucS957LHHcr9rFyL9hRat7kuqrKxs8LYqjT0nTaGMnNr+z9dZZx0Apk6dWvDXzuo5qY1qjMycOTPv9m233RYozBh5Y84JpHteBg4cCMCdd94JhLknZYcVs3Z9MT8rujJXZUKA008/HciP0qvSe1YWirJYkhG53HbbbUD+xtxNVd85cURuZha5TI6Rt2jRAgjVx6rubzd79mwgrLC7/fbbgRCJa/xK473KbtGencOGDQPCGKB2gNHejKq5UXWWXvs+isbAVlxxxXl+j6UwatQoAA444IAa71clycMOOyy1NmWV6pA3V6ovIoo+F1xwwVI0p2DGjh0L5M+x1VerXmPfyfk17cmp+RipWqM+LY7Izcwil8mIXJGhInHt0AEhmlT98Z49ewJhZabyfFU/WuPrGv9K/vVVDY0JEybk/dRfWwiz2HL44YfP4zsrLe3NWW40l7L55pvnbtNYaGNroehzpPUEzZUiV31munTpAoSrtYMOOqg0DWuixvy/ap5EKzd19a4slDFjxhS4dfPOEbmZWeQymbXy+eefAyHzpGoetyIEVTVU3egk1SlXfniaNVSynqHxzjvvANV3fVGeuc5pIfNfi3FOVLFw5MiRAPTv3z93n+Yv6hv/1DoD1eZRpbtFF1007zhF9po70RxLU2Q5a0UuvvhiIFypaGet+vLwmyIr3x/VVtFcnVZu9ujRA0h3LNxZK2ZmZS6TY+RffPEFECLyqjPlXbt2zTtWeeKTJk0CworM6dOnA65mWJM333wTgJVWWinv9th2DlJmUjKbAOCoo44CYNasWXU+h6J4rTNIXqFOnDgRgKuuugooTCQeI52X5lCDSDnm++67LxDeuypfliIrpT6OyM3MIueO3MwscpkcWlERpx122AEIl70AM2bMAOC6664DwrL55nDJVyi6RNRy83KkhV+Npc/XAw88AMChhx4KFHdyLwZKvVMpC22HVo60AFBDLLfccgsQNq3JIkfkZmaRy2T6Yeyykj5VG0Ua48aNA2D11VdXWwDo3LkzkP30w27dugGh1Oxee+3V4PbovWmxWXIrt+Sy62KIIf1QG7S0bdsWCCUvirm4rNTfn2TaoRYElfIqxOmHZmZlzhF5EZQ6osiiYp4TpacOGTIkd5tKkyqSVFqqxj+1BF2prqUQQ0SuonS6atOCqFjL2MbKEbmZWZlzRF4Ejiiq8zmpLoaIvBT8WanOEbmZWZlzR25mFjl35GZmkXNHbmYWOXfkZmaRSzVrxczMCs8RuZlZ5NyRm5lFzh25mVnk3JGbmUXOHbmZWeTckZuZRc4duZlZ5NyRm5lFzh25mVnk3JGbmUXOHbmZWeTckZuZRc4duZlZ5NyRm5lFzh25mVnk3JGbmUXOHbmZWeTckZuZRW7+NF+soqKiWewrV1lZWdHQY31OqvM5qZnPS3U+J39yRG5mFjl35GZmkXNHbmYWuVTHyM0sbp07dwZgwoQJALRo0QKATp06laxN5ojczCx6jsjNrF6XXXYZALvuuisAiy++OADjxo0rWZsscERuZha5sonI11hjDQC22WYbAPbff38AJk+eDMCrr76ad/zFF18MwC+//JJWE82i0aFDBwDuueceAHr27AlAZeWfadtvvPEGAPvss08JWmdJjsjNzCJXob+wqbxYEVZhHXDAAQCcf/75ALRq1apBj+vbty8ATz75ZKGb5JVpNSjEOdH/rcZpf/rpJwDWW289ABZddNHcsYMHDwZg4sSJAHz66ad1vuYXX3wBwNixYwF46aWXGtrceZbFlZ3KStH3aautttJrA3DMMccA4fyU8/dH7/nf//43EM6Frv4/+eSTYr10NV7ZaWZW5qKPyDV7Pm3aNACWXHLJBj3uu+++A0J098gjjxSsTVmJKLKkEOfk3HPPBWDEiBEFalV1f/zxBwBTp04FQjSmn9OnTy/Ya2UxItdY+DPPPJN8bQB23313IJyPYsjK92fhhRcG4O233wZgmWWWAcL82zXXXFOsl67GEbmZWZmLPmvlm2++AeCkk04C4IILLgDCX9OPP/4YgOWXXz7vcW3atAFgyy23BAobkZcbrdpr2bJl7rZBgwYBMGzYsLxjH3zwQQCGDh1a8HbsuOOOdd4/c+bM3O///e9/6zxWUdZqq60GhM9D9+7dAVhrrbUAOOOMM/Ker5AReZZobPy2224DQgQuOveaQ2gO5syZA8C7774LhIi8ffv2JWtTbRyRm5lFzh25mVnkoh9akVGjRgFw4IEHAtC1a1cAfvjhhzofd/nllxe3YRHq168fEC6nNYyy2GKL5Y6pbZJck2XFsMUWWwBhGOCdd97Ju1+XwgCff/55o55bqYuvv/46UH0obrvttgPC0FG52WOPPYDwvh966CEgfJ/qS98sZ1dccQUAffr0AWD11VcvYWtq5ojczCxy0acfJu20004AjBw5EoBu3brVebz+ur711lsFa0NW0qcaSmlUa6+9NgA9evSo8bhZs2blfr/11luBUAJB6WhapJOU9XOiqw69L/n5558B+L//+z+gsAuFspB++NxzzwHhe/LZZ58BIQngvffeK/RL1itrn5XlllsOgI8++ggIZT1WXHFFoPFXf/PC6YdmZmWubMbI5a677gLCggalFSraTDr99NOBEMk3B+3atQPgrLPOAmDvvfcGQirnyy+/DMDZZ58NhAJJc+fOzT2H0jpjtcACCwBw6aWXArDnnnvWeNxGG20EwJQpU9JpWEq23357ADbccEMgzHnceeedQO1XVs2ZUjL12dG8yejRo0vWJnFEbmYWubKLyFUsSVkrWthRm+RS5ObghBNOAEIJUm0aoHmFH3/8sTQNS8Gmm24KhCyNIUOG5N3/66+/AnDIIYcAhZ07yQItfNKYf9K3334L1F8Q6tBDDwXC+LEUs3xCqSXnExWZZ4EjcjOzyEUfkXfp0gWAe++9F4BVVlkFgPnnb9hbu//++4vTsAxQmYKjjz4aCFHoYYcdBoQSpA8//DBQ3uOiG2ywARDmTLRpcJKiLs0B/P777ym0Lj16Pyr9O998f8ZyKhY2adKkGh93+OGH5/17+PDhQPVNl4844ggAll122dxtzTkHPS2OyM3MIhd9RK48cOV0NjQSF0UaijDKyfHHHw+EiHzMmDFAiErLOQJP2mWXXYDaI3HRuKdWcCpv/IEHHgDClZ8yeWKzySabAGGMXJG4rkC+/vrrvOOVX67jlakhs2fPBsKYuoqQKXsMYLfddgNCHrYVniNyM7PIRR+RK0I66qijADjnnHMAWGihhRr0+I4dOxanYRlw7LHHAmHct77Vl+VMmwjrCk6rV5dYYok6H7f++uvn/VS5ZG3erc0uAGbMmFHAFhdO1S3wdOUqWsl58803A2Elp+rZHHnkkUDIO1fErqs6lY1WHZ4nnngi79/lRHnkaa6GbyhH5GZmkYs+Ihet0FMReOXLisbOVe2wdevWKbauNP7zn/8AIZrUe9cKzUcffbQ0DSsB1RTZeuutgVDlTxF5hw4dgFDxUatdkxssKMvjn//8JxCyPwA222wzIIw7Z0WvXr1yv1900UV591199dUAnHrqqUA4D8nNl1VnR/MsyhdfddVVgVB9VMc9/vjjudcol7HxLEbi4ojczCxyZVf9sI7XBuDkk08G4MQTTwTg/fffB0I0VYjoIe3qbaqX8eqrrwKhOps2ptYqRa3o1MpNPS6N1YtZq2hXH60QVjaT8tDrcswxxwD54+Z1Sav6obKWIGxdJ8ksr2effRYInw3R9+Opp54Cat+kWXMHTVnhmbXPSrL6oWiVsM5JMbn6oZlZmSubMfL6KD9YkbiotkYsK/iUZTNu3LjcbRrvVU78LbfcAoRqhhobV0TeqlUrIETsVp3qkt9xxx0APPbYYwD07t271sdoVXHWVJ0v0pVpchNl5YuvsMIKecdppaaizto2adZxisibA13NZ4EjcjOzyDWbiFx1x5OuvfZaoP5qb1nxyiuvAPlZNxoDVSSepEp1ougy1tWJafrtt9+AUKO9rog8uYdoFmlOrLa5MWXc6P511lkHCCs/tT7jww8/BMKKz++//75ILbaGcERuZha5TGetaCeb66+/PnebVifqZ300pqzMjGT++MorrwzABx980Jim1amYs+5arak6KgAtW7as8Vjl1CvXV7PuAwcOBEJ0n4a0MxH0/77ffvsB4f9fedANpdosqhDZt2/fvPsVsVe9r6E17tPKWlGGCVRvm3LMNUauXaE0j1LltYGwslN13MePHz8vTapTLFkr+l6lMVburBUzszKX6TFyrdbcdtttc7dp1lw1IlTrWDUitNJOx6kGSzISV40IPU8stM+msm0AunfvDkC/fv3yjm3bti0QKvkpt7cUO6OnYamllsr9PmHCBCDs1apz0VBa4agVnMlIXKZNm5b7Pau7TVX9rMyZMwcIteqVN17flXlyZWcxIvHYaNWrdtgqJUfkZmaRy3RErr90VSu2aVfziRMnAjB9+nQApk6dCoRZ9KoV3yBEHBorVRW7WCsBqhaGBVVzmBWJiz5Db7/9NhDqzYjmGXQFp0g8+TnSWLEiVK2azTJl3AAMGjQICO+vT58+NT7mxhtvBOD1118HwqrhNFYxZs2XX34JwJtvvgkjhukdAAABtElEQVTAmmuuWcrm1MgRuZlZ5NyRm5lFLtPph6KJSQgTdVdeeWWjnkPL1ZXSWExZS5/KgjTOiVINAUaPHl3jMRoiSC5g0UYImjiujQqODRgwAMgv19pYaaUfxiar35/JkycDIaFCZTKS298Vg9MPzczKXKYnO0UFeQAWXHBBoPqCBUVSmswRRV79+/cvZhMtA6pulHH77bcDYeNfqS/iTtKCH02k3n333QC8+OKL89xOi9OUKVOAEJEn+6BSckRuZha5KMbIY5PVMb5SSvuc6MpNY9la0KPCVslxzeTmGtpEWLcrGiskj5HXLKvfH5X4VXkQpWhqm7ti8hi5mVmZc0ReBFmNKErJ56Q6R+Q182elOkfkZmZlzh25mVnk3JGbmUXOHbmZWeTckZuZRS7VrBUzMys8R+RmZpFzR25mFjl35GZmkXNHbmYWOXfkZmaRc0duZhY5d+RmZpFzR25mFjl35GZmkXNHbmYWOXfkZmaRc0duZhY5d+RmZpFzR25mFjl35GZmkXNHbmYWOXfkZmaRc0duZhY5d+RmZpFzR25mFjl35GZmkXNHbmYWOXfkZmaR+39o9KhSUQhR6QAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<Figure size 432x288 with 15 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "import matplotlib.pyplot as plt\n",
    "plot_rows, plot_cols = 3 , 5\n",
    "for n in range(plot_rows * plot_cols):\n",
    "    plt.subplot(plot_rows, plot_cols, n+1)\n",
    "    plt.imshow(x_train[n], cmap='gray'); plt.axis('off')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Dataset normalization"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "When dealing with images as input, we must preprocess input training data in the manner the Keras backend describe an image. Images can be described in the following two ways:\n",
    "- <b>channel_first:</b> (channels, width, height)\n",
    "- <b>channel_last:</b> (width, height, channels)\n",
    "\n",
    "The MNIST input images have a depth of 1, so channels is 1 because they are simple greyscale images."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We need to reshape the input with a channel information:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [],
   "source": [
    "# input image dimensions\n",
    "img_rows, img_cols = 28, 28\n",
    "if K.image_data_format() == 'channels_first':\n",
    "    x_train = x_train.reshape(x_train.shape[0], 1, img_rows, img_cols)\n",
    "    x_test = x_test.reshape(x_test.shape[0], 1, img_rows, img_cols)\n",
    "    input_shape = (1, img_rows, img_cols)\n",
    "else:\n",
    "    x_train = x_train.reshape(x_train.shape[0], img_rows, img_cols, 1)\n",
    "    x_test = x_test.reshape(x_test.shape[0], img_rows, img_cols, 1)\n",
    "    input_shape = (img_rows, img_cols, 1)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In order to decrease waiting time for training process, let's reduce the number of samples to use in this tutorial."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [],
   "source": [
    "#sub-sample of test data to improve training speed. Comment out\n",
    "#if you want to train on full dataset.\n",
    "x_train_full = x_train # for later use\n",
    "y_train_full = y_train # for later use\n",
    "x_train = x_train[:20000,:,:,:]\n",
    "y_train = y_train[:20000]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "As we already observed, input samples are integers. Neural network layers work on float. We need to cast (change the type) of the input to float32 type.\n",
    "\n",
    "It is also convenient, for convergency reasons, to normalize the input range values between [0,1]."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "x_train shape: (20000, 28, 28, 1)\n",
      "20000 train samples of type float32\n",
      "10000 test  samples of type float32\n"
     ]
    }
   ],
   "source": [
    "#normalise the images and double check the shape and size of the image data\n",
    "x_train = x_train.astype('float32')\n",
    "x_test = x_test.astype('float32')\n",
    "x_train /= 255\n",
    "x_test /= 255\n",
    "print('x_train shape:', x_train.shape)\n",
    "print(x_train.shape[0], 'train samples of type', x_train.dtype)\n",
    "print(x_test.shape[0],  'test  samples of type', x_test.dtype)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Build categorical matrices"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The y_train is of course just a 60K sequence of numbers from 0 to 9, which represent the set in the category."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(20000,)"
      ]
     },
     "execution_count": 8,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# what about the y_tain data?\n",
    "y_train.shape"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We should have 10 different classes, one for each digit, but it looks like we only have a 1-dimensional array. We should represent them into a categorical matrix, so that for each index we have an array of 10 element with 1 on the right label and 0 otherwise."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [],
   "source": [
    "import tensorflow.keras.utils as np_utils\n",
    "# Convert 1-dimensional class arrays to 10-dimensional class matrices\n",
    "y_train_indexes = y_train.copy() # for later use\n",
    "y_test_indexes = y_test.copy()   # for later use\n",
    "y_train = np_utils.to_categorical(y_train_indexes, 10)\n",
    "y_test = np_utils.to_categorical(y_test_indexes, 10)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([[1., 0., 0., 0., 0., 0., 0., 0., 0., 0.],\n",
       "       [0., 0., 0., 0., 1., 0., 0., 0., 0., 0.],\n",
       "       [0., 1., 0., 0., 0., 0., 0., 0., 0., 0.],\n",
       "       [0., 0., 0., 0., 0., 0., 0., 0., 0., 1.]], dtype=float32)"
      ]
     },
     "execution_count": 10,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# let's have a look at the new format\n",
    "y_train[1:5]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Build the model"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "_________________________________________________________________\n",
      "Layer (type)                 Output Shape              Param #   \n",
      "=================================================================\n",
      "conv2d (Conv2D)              (None, 26, 26, 32)        320       \n",
      "_________________________________________________________________\n",
      "max_pooling2d (MaxPooling2D) (None, 13, 13, 32)        0         \n",
      "_________________________________________________________________\n",
      "dropout (Dropout)            (None, 13, 13, 32)        0         \n",
      "_________________________________________________________________\n",
      "flatten (Flatten)            (None, 5408)              0         \n",
      "_________________________________________________________________\n",
      "dropout_1 (Dropout)          (None, 5408)              0         \n",
      "_________________________________________________________________\n",
      "dense (Dense)                (None, 10)                54090     \n",
      "=================================================================\n",
      "Total params: 54,410\n",
      "Trainable params: 54,410\n",
      "Non-trainable params: 0\n",
      "_________________________________________________________________\n"
     ]
    }
   ],
   "source": [
    "# Create a convolutional multi-layer using max-pooling and dropout\n",
    "# uncomment layers to produce more accurate training\n",
    "\n",
    "# number of classes\n",
    "nb_classes = 10\n",
    "# number of convolutional filters to use\n",
    "nb_filters = 32\n",
    "# size of pooling area for max pooling\n",
    "pool_size = (2, 2)\n",
    "# convolution kernel size\n",
    "kernel_size = (3, 3)\n",
    "\n",
    "\n",
    "model = Sequential()\n",
    "model.add(Conv2D(nb_filters, kernel_size=kernel_size,\n",
    "                 activation='relu',\n",
    "                 input_shape=input_shape))\n",
    "#model.add(Conv2D(64, kernel_size, activation='relu'))\n",
    "model.add(MaxPooling2D(pool_size=pool_size))\n",
    "model.add(Dropout(0.25)) # try out SpatialDropout2D\n",
    "\n",
    "# now let's add a fully connected layer ...\n",
    "model.add(Flatten()) # transform 2D output into a 1D to feed the Dense layer\n",
    "#model.add(Dense(256, input_shape=input_shape, activation='relu'))\n",
    "#model.add(Dense(128, activation='relu'))\n",
    "model.add(Dropout(0.5))\n",
    "\n",
    "# ... and the final layer which has an output size of nb_classes\n",
    "# corresponding to the size of our category items to match\n",
    "model.add(Dense(nb_classes, activation='softmax'))\n",
    "\n",
    "model.compile(loss=keras.losses.categorical_crossentropy,\n",
    "              optimizer=keras.optimizers.Adam(),\n",
    "              metrics=['accuracy'])\n",
    "\n",
    "model.summary()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Model Fitting"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now we are going to train out model (or uncomment the load_model code to restore a previous one without going through the optimization process)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Train on 20000 samples, validate on 10000 samples\n",
      "Epoch 1/10\n",
      "20000/20000 [==============================] - 12s 589us/step - loss: 0.6775 - acc: 0.8120 - val_loss: 0.3051 - val_acc: 0.9112\n",
      "Epoch 2/10\n",
      "20000/20000 [==============================] - 11s 533us/step - loss: 0.3152 - acc: 0.9085 - val_loss: 0.2249 - val_acc: 0.9345\n",
      "Epoch 3/10\n",
      "20000/20000 [==============================] - 10s 524us/step - loss: 0.2507 - acc: 0.9270 - val_loss: 0.1895 - val_acc: 0.9454\n",
      "Epoch 4/10\n",
      "20000/20000 [==============================] - 11s 553us/step - loss: 0.2111 - acc: 0.9380 - val_loss: 0.1592 - val_acc: 0.9549\n",
      "Epoch 5/10\n",
      "20000/20000 [==============================] - 10s 479us/step - loss: 0.1876 - acc: 0.9449 - val_loss: 0.1418 - val_acc: 0.9600\n",
      "Epoch 6/10\n",
      "20000/20000 [==============================] - 10s 478us/step - loss: 0.1671 - acc: 0.9505 - val_loss: 0.1311 - val_acc: 0.9623\n",
      "Epoch 7/10\n",
      "20000/20000 [==============================] - 10s 525us/step - loss: 0.1556 - acc: 0.9541 - val_loss: 0.1222 - val_acc: 0.9655\n",
      "Epoch 8/10\n",
      "20000/20000 [==============================] - 11s 561us/step - loss: 0.1430 - acc: 0.9580 - val_loss: 0.1097 - val_acc: 0.9685\n",
      "Epoch 9/10\n",
      "20000/20000 [==============================] - 10s 524us/step - loss: 0.1346 - acc: 0.9591 - val_loss: 0.1018 - val_acc: 0.9706\n",
      "Epoch 10/10\n",
      "20000/20000 [==============================] - 10s 511us/step - loss: 0.1287 - acc: 0.9612 - val_loss: 0.0977 - val_acc: 0.9716\n"
     ]
    }
   ],
   "source": [
    "batch_size = 128\n",
    "epochs = 10\n",
    "restore_model=False\n",
    "\n",
    "if restore_model:\n",
    "    model = keras.models.load_model(\"saved/mnist_model\")\n",
    "else:\n",
    "    model.fit(x_train, y_train, \n",
    "              batch_size=batch_size, \n",
    "              epochs=epochs,\n",
    "              validation_data=(x_test, y_test))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Once your model has been trained, let's evaluate its score on data not used during traing (also known as the test data set). The evaluate method returns a list of two element: the first is the <font color='red'>score</font>, the second is the <font color='red'>accuracy</font>."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Accuracy: 97.16%\n"
     ]
    }
   ],
   "source": [
    "score = model.evaluate(x_test, y_test, verbose=0)\n",
    "print(\"Accuracy: %.2f%%\" % (score[1]*100))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Test the model"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's inspect the output of our model. We expect to be a probability distribution over the set of possible category elements of our dataset [0-9]. For example, let's select a random input from the x_test data and see how the output of our model looks like. We use the <font color='red'>predict_proba</font> model method. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [],
   "source": [
    "def plot_digit_and_probability(image, matrix_probs, text=None):\n",
    "    plt.figure(figsize=(8,4));\n",
    "    plt.subplot(1, 2, 1);\n",
    "    if text is not None:\n",
    "        plt.title('Example of digit: {}'.format(text));\n",
    "    plt.imshow(image, cmap='gray'); plt.axis('off');\n",
    "    plt.subplot(1, 2, 2);\n",
    "    plt.title('Probabilities for each digit class');\n",
    "    plt.bar(np.arange(10), matrix_probs.reshape(10), align='center'); \n",
    "    plt.xticks(np.arange(10), np.arange(10).astype(str));"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAeQAAAEICAYAAACOKIcAAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDMuMC4xLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvDW2N/gAAHelJREFUeJzt3Xu8XGV97/HPl0QEJIKaaCGJRI8IUqoVU7T1UgTxgJVLrSKoVC2V1iMoR6wiWi8oLbRWa1tsD4KCglC8g40FrZdWESQIXgCpkXJJQAl3EG/g7/yx1sbJZs/sndnZsxc7n/frNa/MzO+ZZ/1mzc78Zj3reWZSVUiSpNm1yWwnIEmSLMiSJHWCBVmSpA6wIEuS1AEWZEmSOsCCLElSB1iQJXVWknckOW3Ix74iydcGxD+f5OUTtU1yV5LHDnjsZUl2GyavAX3ukOTSJHcmee2G7HtDSbIsSSWZP+Tj12s/j3vslNtO0s/Qf1MzbaidKkn9JLkaeBRwL/AT4PPAYVV112zmNV5V7T0gtuXY9SSnAKur6q098d+cgZTeCHy5qn57BvrupN79vD5tJ3pN5gKPkCXNhH3aN9BdgOXA/d440/A96Ne2Ay4b5oHDHrGqW/zPIGnGVNUamiPknQGSfCXJsUm+DtwNPDbJtknOTnJLklVJXjWum82S/Gs7lPutJE8aCyQ5KskP29jlSf5w3GOT5J+S3J7k+0n26Al8JcmfTpR3Oyz7uCSHAi8F3tgOmZ7Txq9O8pz2+iY9edyc5KwkD29jmyU5rb3/tiQXJXnUBNv7EvBs4J/a7Tw+yVZJPpJkbZJrkrx17ANMO/T79STvS3Iz8I4J+uybVxv/eJIftfvmP5P8Zk9s8yR/12739iRfS7J5T/cvTXJtkpuSvGWifdj284j2tb0jyTeB/zXRfu5pe07b9qIk7x43vD3wNZlg27+Z5Avt39WPkxzdp92g/fC89u/qziRrkryhvX9hks+1r+ktSf5rQ3y4tCBLmjFJlgLPAy7puftg4FBgAXANcCawGtgWeCHwV0l272m/H/Bx4OHAx4DPJHlQG/sh8ExgK+CdwGlJtul57FPbNguBtwOf6i1Kk6mqE4HTgb+pqi2rap8Jmh0O7A/8fvscbgVOaGMvb3NbCjwC+HPgpxNsZ3fgv2iG9resqv8G/rF97GPbvv8YeOW453YVzemBY9czL2g+KG0PPBL4Vvs8x7wHeArwezT7/Y3Ar3rizwB2APYA3pbkCRNsn3Z7PwO2Af6kvfRzAs0pjt+g2W8vn6jRVF6TJAuALwL/TvPcHwf8R5/tDtoPJwN/VlULaD5Ufqm9/0iav9lFNPv/aGD630NdVV68ePGywS7A1cBdwG00BfcDwOZt7CvAMT1tl9Kca17Qc99fA6e0198BXNAT2wS4AXhmn21fCuzXXn8FcD2Qnvg3gYN7cvnTnrZf62lXwOPa66cA757gOT6nvX4FsEdPbBvglzRzdP4EOB944hT2W28+84BfADv1xP8M+EpPvtdO0l/fvCZou3X7nLdq9/FPgSdN0G5Z227JuH164ARt57Xb27Hnvr+aaD/3tN2hJ/bu9XlNxm37IOCSPrF3AKf1id23H9rb17b7/aHj2h0DfHYsnw118QhZ0kzYv6q2rqrtqur/VFXvUeF1Pde3BW6pqjt77rsGWDxR+6r6Fb8+mibJH6eZmXxbkttojmIW9jx2TbXvoD19bzutZ3Z/2wGf7snhCpoPGY8CPgqcC5yZ5Pokf9NzdD/IQuBBbb5j+u6X9c0rybwkx7XD2XfQfMAY2+5CYDOakYV+ftRz/W5goslZi2g+lPTmec0E7fq1nez5DbKUwfkDMMl+APgjmhGea5J8Ncnvtvf/LbAKOC/JVUmOmkau97EgzxGZZInHNPpNkg8nubU9BzRZ+3WWRaRnackUHjvltnpA6y2Q1wMPb4cYxzwaWNNze+nYlfY83RLg+iTbAR8EDgMeUVVbA98D0vPYxUl6bz+63eaw+U7kOmDv9gPI2GWzqlpTVb+sqndW1U40w7/Ppxl6nsxNNEeM243LvXe/DJ0X8BKaUwHPoTkqXtY+Ju22f8a4871DWAvcQ8/r1z6HQW2X9Ny3tE9bmNpzn8oSqUH7gaq6qKr2oxnO/gxwVnv/nVV1ZFU9FtgXeH165icMy4I8BWkmcPy0nUAwdvmn2c5rRJ4B7EkzRLXr+j64qvauqlPXt+0wHzDao4/r2kkh1/SbxKHuqKrraIZ0/zrNBKgnAocAvetEn5LkBe2HvCOAnwMXAA+heWNeC5DklbSTx3o8EnhtkgcleRHwBGDFeqb5Ywa/uf8LcGz7AYEki5Ls115/dpLfSjIPuIOmyP6qf1eNqrqX5s3/2CQL2r5fz7r7ZTJ986I5f/9z4GZgC5qh5LFt/wr4EPDeNBPu5iX53SQPXo9tjz2HTwHvSLJFkp3of154fNsdGfzBZbLX5HPANkmOSPLgdh8+dYJ2ffdDkk2TvDTJVlX1S5rX71dt7PntBLMAt9OMPEz6uk7Ggjx1+1QzgWDscthsJzQi2wFXV9VPZjuRKTiZ5nzVQ2mORl6a5AWznJMmdxDNkcn1wKeBt1fVF3vinwVeTDMp6WDgBe2R5+XA3wHfoHmD/i3g6+P6vpBmws5NNBOfXlhVN69nficDO7VDv5+ZIP5+4Gya4cs7aT4sjL35/wbwCZo38yuAr9IMY0/F4TSTnK4CvkYzoe1D65H3oLw+QjN8vAa4vI31egPwXeAi4BbgeIarF4fRDGf/iOa874cnabtV2/ajwBk0xXIiA1+T9hTInsA+bX8/oJnFPt5k++Fg4Op2OPvPaWZ3Q/M39UWauRLfAD5QVV8e8NymZkOekJ6rF3omcEwQ+2fgkz23j6eZzRfgYTSf1NbSvJl8jnUnQ3yFZuLC+e0Lew7NTMzTaf4DXwQs62lfwGtp/oPeRHMeY5M29grWnQCxI/AFmv9MVwIHDHh+29L8x72F5rzIq9r7D6EZurq3ze+dEzx2Hs2MzJvavF7T5jm/5zn2TlT5u7bt/9D8B7xfW5qjmN7t3jbEa7aY5g3ljbP99+PFi5f1v7TvpafOdh6jvHiEPH1HAr/VDrE+k6aIvbyav6hNaD4Rbkdz7uSnwPih7gNpPoUtpjln8432MQ+n+UT99nHt/5DmixZ2oTn3cb9lBEkeQlOMP0YzZHcg8IF2yGgiEy47qaqTaT4VfqOaUYHxuQC8iua82JPbvF7YZxtjbfcGfrvNf/+JGlXVFeO2u3X7vF6S5DsD+h9bl3pX+3weQrMPJHVckh2TPLGdt7IrzXvpp2c7r1GyIE/dZ8ZmK7aXVwFU1d00BfW9NOd3Dq+q1W3s5qr6ZFXdXc0QyrE0awJ7fbiqflhVt9Osh/thVX2xqu6hWXv55HHtj6+qW6rqWuDvaYb7xns+zTDzh6vqnqq6BPgk8KLxDdOsE3068Kaq+llVXQqcxNQmngAcAPx9VV1XVbfQLFkZ1Pb9VbW6qm4FjpviNgCoqo9V1RMnaXMczXmhXWiGvW5fn21ImjULaM4j/wT4V5rRtM/OakYj5tetTd3+te55rftU1YVJrqI5Gj1r7P4kWwDvA/aiGb4GWJBkXjWTGKA59zXmpxPcHr+cYPwSgomWcGwHPLVd6jBmPhOfu+q37GT5BG0nsu0EOU217XSWNfTVjk5ckuR/03xZxOtnYjuSNpyquohmTfJGyyPkDSDJa4AH00xKeWNP6Eiab7N5ajUTjZ419pBpbG78EoKJlnBcB3y11l3usGVVvXqCtlNZdjLIDRPkNKjthlrWMBXzmf7SDUkaCY+QpynJ42kmZu1Gs0D+m0k+3w79LqA5yr0tzdf1TXQOdn39RZILaY6cX0czVD7e54DjkhxMc34YmvO2d7XnZ+9TVdclGVt28gbg8TTnbl7K1JxFs6zkczRDTYMWyJ8FvC7Jv7Vt3zSg7Y+BJUk2rapfTJZEuz71Ve02bgN+h2aC2aAhdA1p4cKFtWzZstlOQ+q8iy+++KaqWjSVthbkqTsnyb09t79Ac072NJrzut8GaNe+fjTJcppzvB+jmVV8Pc05kQknMq2HzwIX0ywPOIVm+v86qurOJM+lKdbvpRkJ+Tb9h24PolmzeD3NbPC39xuen8AHaYr4t2lmhr8H2H2Stt9p2/4DzQeZeydo+yWaX775UZJfVdXCJC8Fjq7+P333hzQFeNP2ufxje9EGtmzZMlauXDnbaUidl2TQabx127bTy/UAkKSA7atq1WznsiEk2Rv4l6rabtLG6pTly5eXBVmaXJKLq2pKc3I8h6yRSfOTbs9LMj/JYpoh/I1qWYMk9WNB1iiFZtbzrTQ/x3cF8LZZzUiSOsJzyA8gVTWd2dmzrl2z/TuznYckdZFHyJIkdYAFWZrjknwoyY1JvtcnniT/kGRVku8k2WXUOUoa8ZB1O0tY0iQ28OmJU2i+Q/0jfeJ70/x6zfY0vwb0z/z6V4EkjYhHyNIcV1X/SfNLXv3sB3ykGhcAWyfZZjTZSRpjQZa0mHW/V3x1e986khyaZGWSlWvXrh1ZctLGwlnWkqakqk4EToTmi0FmOZ1OWHbUvw392KuP+4MNmInmAo+QJa1h3R/6WMLUf1xE0gZiQZZ0NvDH7WzrpwG3V9UNs52UtLFxyFqa45KcQfMjHguTrKb5ytIHAVTVvwArgOcBq2h+seyVs5OptHGzIEtzXFUdNEm8aH6qUtIscshakqQOsCBLktQBFmRJkjrAgixJUgdYkCVJ6gALsiRJHWBBliSpAyzIkiR1gAVZkqQOsCBLktQBFmRJkjrAgixJUgdYkCVJ6gALsiRJHWBBliSpAyzIkiR1gAVZkqQOsCBLktQBFmRJkjrAgixJUgdYkCVJ6gALsiRJHWBBliSpAyzIkiR1gAVZkqQOsCBLktQBFmRJkjrAgixJUgfMn+0E5qJDDjmkb+ykk04aYSaNH/zgB31jk+Vz9tln9419//vfHzonSdK6PEKWJKkDLMiSJHWABVmSpA6wIEuS1AEWZGmOS7JXkiuTrEpy1ATxRyf5cpJLknwnyfNmI09pY2dBluawJPOAE4C9gZ2Ag5LsNK7ZW4GzqurJwIHAB0abpSRw2dPQzjnnnL6xPffcs2+sqmYinYEe97jH9Y0dd9xxAx/7qEc9qm/syCOPHDonjcyuwKqqugogyZnAfsDlPW0KeGh7fSvg+pFmKAnwCFma6xYD1/XcXt3e1+sdwMuSrAZWAIdP1FGSQ5OsTLJy7dq1M5GrtFGzIEs6CDilqpYAzwM+muR+7w1VdWJVLa+q5YsWLRp5ktJcZ0GW5rY1wNKe20va+3odApwFUFXfADYDFo4kO0n3sSBLc9tFwPZJHpNkU5pJW+O/D/VaYA+AJE+gKciOSUsjZkGW5rCqugc4DDgXuIJmNvVlSY5Jsm/b7EjgVUm+DZwBvKJmY/ahtJFzlrU0x1XVCprJWr33va3n+uXA00edl6R1eYQsSVIHeITcx2677TYwvvvuu/eNbbrppn1j3/3ud/vGrrnmmknz6ufYY4/tG9t55537xj74wQ8O7PfwwydcAQPAJZdc0jd22mmnDexXkrQuj5AlSeoAC7IkSR1gQZYkqQMsyJIkdYAFWZKkDrAgS5LUAS576mPQ8iSAz372s31jCxYs6Bs79NBD+8ZuuOGGyRMbwkMf+tDJG/Uxf37/P5GHPexhQ/crSVqXR8iSJHWABVmSpA6wIEuS1AEWZEmSOsCCLElSB1iQJUnqAJc99XHzzTcPjL/kJS8ZUSaz65577ukbu+OOO0aYiSTNbR4hS5LUARZkSZI6wIIsSVIHWJAlSeoAC7IkSR1gQZYkqQNc9jRHDPqFqRe96EVD9/uBD3ygb+zUU08dul9J0ro8QpYkqQMsyJIkdYAFWZKkDrAgS5LUARZkSZI6wIIsSVIHWJAlSeoA1yE/gDztaU/rGzv33HP7xgatUZ7MhRdeOPRjJUlT5xGyJEkdYEGW5rgkeyW5MsmqJEf1aXNAksuTXJbkY6POUZJD1tKclmQecAKwJ7AauCjJ2VV1eU+b7YE3A0+vqluTPHJ2spU2bh4hS3PbrsCqqrqqqn4BnAnsN67Nq4ATqupWgKq6ccQ5SsKCLM11i4Hrem6vbu/r9Xjg8Um+nuSCJHtN1FGSQ5OsTLJy7dq1M5SutPGyIEuaD2wP7AYcBHwwydbjG1XViVW1vKqWL1q0aMQpSnOf55BHbNNNN+0be/WrXz3wsccff/xQ/Q6yZs2agfFLLrlkqH7VGWuApT23l7T39VoNXFhVvwT+J8l/0xToi0aToiTwCFma6y4Ctk/ymCSbAgcCZ49r8xmao2OSLKQZwr5qlElKsiBLc1pV3QMcBpwLXAGcVVWXJTkmyb5ts3OBm5NcDnwZ+Iuqunl2MpY2Xg5ZS3NcVa0AVoy772091wt4fXuRNEs8QpYkqQMsyJIkdYAFWZKkDvAc8gzYbrvt+sbOP//8vrFtttlmJtIZaPHi8d8Rsa4VK1b0jR199NF9Y2eeeebQOUnSxsgjZEmSOsCCLElSB1iQJUnqAAuyJEkdYEGWJKkDLMiSJHWAy55mwPz5/XfrbCxtmo5ly5b1jZ1++ul9Y29605v6xl75ylcO3Oall146aV6SNNd4hCxJUgdYkCVJ6gALsiRJHWBBliSpAyzIkiR1gAVZkqQOsCBLktQBrkOeAT//+c/7xmZqje273vWuvrE777xz6H7f/OY39409+9nP7ht70pOe1Dd2zjnnDNzm/vvv3zd28cUXD3ysJD1QeYQsSVIHWJAlSeoAC7IkSR1gQZYkqQMsyJIkdYAFWZKkDnDZ0wxYvXp139guu+wywkym7+677+4be+1rX9s3dsABB/SNLV68eOA2P/WpT/WN7bHHHn1jq1atGtivJHWZR8iSJHWABVmSpA6wIEuS1AEWZEmSOsCCLElSB1iQJUnqAJc9aaDzzz+/b+zCCy/sG9t88837xvbZZ5+B21y6dGnf2CMf+ci+MZc9SXog8whZmuOS7JXkyiSrkhw1oN0fJakky0eZn6SGBVmaw5LMA04A9gZ2Ag5KstME7RYArwP6D3tImlEWZGlu2xVYVVVXVdUvgDOB/SZo9y7geOBno0xO0q9ZkKW5bTFwXc/t1e1990myC7C0qv5tUEdJDk2yMsnKtWvXbvhMpY2cBVnaiCXZBHgvcORkbavqxKpaXlXLFy1aNPPJSRsZC7I0t60BeqetL2nvG7MA2Bn4SpKrgacBZzuxSxo9lz1paPfee2/f2H77TXSasjHo15wA9t9//76x0047rW/suc99bt/YRrwk6iJg+ySPoSnEBwIvGQtW1e3AwrHbSb4CvKGqVo44T2mj5xGyNIdV1T3AYcC5wBXAWVV1WZJjkuw7u9lJ6uURsjTHVdUKYMW4+97Wp+1uo8hJ0v15hCxJUgdYkCVJ6gALsiRJHWBBliSpAyzIkiR1gLOsNXLnnXfewPigdcjLli3rG9thhx36xjbidciSHiA8QpYkqQMsyJIkdYAFWZKkDrAgS5LUARZkSZI6wIIsSVIHuOxJM2LHHXfsG3vLW94ywkwk6YHBI2RJkjrAgixJUgdYkCVJ6gALsiRJHWBBliSpAyzIkiR1gMueNnJbbrnlwPhTnvKUvrF99923b+yAAw7oG1u8ePHkifVx++23943dfPPNQ/crSbPNI2RJkjrAgixJUgdYkCVJ6gALsiRJHWBBliSpAyzIkiR1gMueHkAG/YLSJpv0/2x1xBFHDNUnwDOe8YzJE9vAVq1a1Tf21re+tW/sggsumIl0JGkkPEKWJKkDLMiSJHWABVmSpA6wIEuS1AEWZEmSOsCCLM1xSfZKcmWSVUmOmiD++iSXJ/lOkv9Ist1s5Clt7CzI0hyWZB5wArA3sBNwUJKdxjW7BFheVU8EPgH8zWizlASuQ54R8+bN6xvbdttt+8aOOeaYgf0efPDBfWOD1iHPhrVr1/aN/eVf/uXAx55xxhl9Y3feeefQOW2kdgVWVdVVAEnOBPYDLh9rUFVf7ml/AfCykWYoCfAIWZrrFgPX9dxe3d7XzyHA5ycKJDk0ycokKwd94JI0HAuyJACSvAxYDvztRPGqOrGqllfV8kWLFo02OWkj4JC1NLetAZb23F7S3reOJM8B3gL8flX9fES5SerhEbI0t10EbJ/kMUk2BQ4Ezu5tkOTJwP8D9q2qG2chR0lYkKU5raruAQ4DzgWuAM6qqsuSHJNk37bZ3wJbAh9PcmmSs/t0J2kGOWQtzXFVtQJYMe6+t/Vcf87Ik5J0PxbkGbBgwYK+sRe/+MV9Y0uWLBnY70wsbVqz5n6nE9fx/ve/v2/s3nvv7Rt73/veN3ROkrQxcshakqQOsCBLktQBFmRJkjrAgixJUgdYkCVJ6gALsiRJHeCypxlw22239Y295z3vGSomSZrbPEKWJKkDLMiSJHWABVmSpA6wIEuS1AEWZEmSOsCCLElSB1iQJUnqAAuyJEkdYEGWJKkDLMiSJHWABVmSpA6wIEuS1AEWZEmSOsCCLElSB1iQJUnqAAuyJEkdYEGWJKkDLMiSJHWABVmSpA6wIEuS1AEWZEmSOmD+KDdWVRnl9iRJeqDwCFmSpA6wIEuS1AEWZEmSOsCCLM1xSfZKcmWSVUmOmiD+4CT/2sYvTLJs9FlKsiBLc1iSecAJwN7ATsBBSXYa1+wQ4NaqehzwPuD40WYpCSzI0ly3K7Cqqq6qql8AZwL7jWuzH3Bqe/0TwB5JXBEhjdhIlz1JGrnFwHU9t1cDT+3XpqruSXI78Ajgpt5GSQ4FDm1v3pXkymnmtnD8NjamvrJ+4xAPuOdnX/fZbqoNLciSpqSqTgRO3FD9JVlZVcvt64GZk31teA5ZS3PbGmBpz+0l7X0TtkkyH9gKuHkk2Um6jwVZmtsuArZP8pgkmwIHAmePa3M28PL2+guBL1VVjTBHSThkLc1p7Tnhw4BzgXnAh6rqsiTHACur6mzgZOCjSVYBt9AU7VHYYMPfG0FfXczJvjaw+EFYkqTZ55C1JEkdYEGWJKkDLMiSRm6yr/Ncj34+lOTGJN+bZj5Lk3w5yeVJLkvyumn0tVmSbyb5dtvXO6eTW9vnvCSXJPncNPu5Osl3k1yaZOU0+9o6ySeSfD/JFUl+d8h+dmjzGbvckeSIIfv6v+0+/16SM5JsNkw/bV+va/u5bNh81nubnkOWNErt13n+N7AnzReVXAQcVFWXD9HXs4C7gI9U1c7TyGkbYJuq+laSBcDFwP5D5hTgIVV1V5IHAV8DXldVF0wjv9cDy4GHVtXzp9HP1cDyqpr2l2YkORX4r6o6qZ3Bv0VV3TbNPufRLMN7alVds56PXUyzr3eqqp8mOQtYUVWnDJHHzjTfarcr8Avg34E/r6pV69vX+vAIWdKoTeXrPKekqv6TZmb4tFTVDVX1rfb6ncAVNN9gNkxfVVV3tTcf1F6GPvJJsgT4A+CkYfvY0JJsBTyLZoY+VfWL6Rbj1h7AD9e3GPeYD2zerqffArh+yH6eAFxYVXdX1T3AV4EXDNnXlFmQJY3aRF/nOVTxmwntr109GbhwGn3MS3IpcCPwhaoaui/g74E3Ar+aRh9jCjgvycXtV6EO6zHAWuDD7VD6SUkesgHyOxA4Y5gHVtUa4D3AtcANwO1Vdd6QeXwPeGaSRyTZAnge637BzoywIEtSK8mWwCeBI6rqjmH7qap7q+q3ab4Zbdd2CHSYfJ4P3FhVFw+byzjPqKpdaH796zXtkP8w5gO7AP9cVU8GfgIMPRcAoB323hf4+JCPfxjNSMtjgG2BhyR52TB9VdUVNL96dh7NcPWlwL3D9LU+LMiSRm0qX+c5cu353k8Cp1fVpzZEn+0w7peBvYbs4unAvu253zOB3ZOcNo181rT/3gh8mub0wTBWA6t7jvw/QVOgp2Nv4FtV9eMhH/8c4H+qam1V/RL4FPB7wyZTVSdX1VOq6lnArTTzHmaUBVnSqE3l6zxHqp2IdTJwRVW9d5p9LUqydXt9c5rJa98fpq+qenNVLamqZTT76UtVNdRRX5KHtBPWaIeXn0szNDtMXj8CrkuyQ3vXHsB6T4Ab5yCGHK5uXQs8LckW7eu5B81cgKEkeWT776Npzh9/bBq5TYlfnSlppPp9necwfSU5A9gNWJhkNfD2qjp5iK6eDhwMfLc99wtwdFWtGKKvbYBT2xnDmwBnVdW0littII8CPt3+1PV84GNV9e/T6O9w4PT2Q9VVwCuH7aj9gLAn8GfD9lFVFyb5BPAt4B7gEqb3tZefTPII4JfAazbQpLWBXPYkSVIHOGQtSVIHWJAlSeoAC7IkSR1gQZYkqQMsyJIkdYAFWZKkDrAgS5LUAf8fHbHRaBmVMhcAAAAASUVORK5CYII=\n",
      "text/plain": [
       "<Figure size 576x288 with 2 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "#choose a random data from test set and show probabilities for each class.\n",
    "i = 32 # np.random.randint(0, len(x_test))\n",
    "digit = x_test[i].reshape(28,28)\n",
    "probs = model.predict_proba(digit.reshape(1,28,28,1),batch_size=1)\n",
    "plot_digit_and_probability(digit, probs, y_test_indexes[i])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "When the probability is sharp over a digit, the prediction is very accurate and the result is correct."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Inspect Wrong Predictions\n",
    "\n",
    "What about \"wrong results\"? Let's have a look ..."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "10000/10000 [==============================] - 1s 135us/step\n",
      "Class prediction array contains:  [7 2 1 ... 4 5 6]\n"
     ]
    }
   ],
   "source": [
    "# returns an array of class prediction for the input sample\n",
    "# that is an array of recognized digit for each input\n",
    "predictions = model.predict_classes(x_test, batch_size=32, verbose=1)\n",
    "\n",
    "print(\"Class prediction array contains: \", predictions)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "There are  284 wrong predictions over 10000\n"
     ]
    }
   ],
   "source": [
    "# construct a list of indexes\n",
    "predicted_indexes = np.arange(len(predictions))\n",
    "\n",
    "# for each predicted result select those which are not correct\n",
    "wrong_results = predicted_indexes[y_test_indexes!=predictions]\n",
    "\n",
    "\n",
    "print(\"There are \", len(wrong_results), \"wrong predictions over\", len(predictions))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "\n",
    "If we inspect the probability distribution of wrong result predictions we find that the higher probability is not the only sharp peak over a digit, but other stronger peaks exist, not always on the right result. Probabilities are in general lower and more spread between classes than for a correctly labelled digit."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "element 8\n",
      "element 92\n"
     ]
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAeQAAAEICAYAAACOKIcAAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDMuMC4xLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvDW2N/gAAIABJREFUeJzt3XucHFWd9/HPlwnhlghIBhdyVxI1eOEyBpXFRS6+Ei4JauRJUBZcTfRZgiC4GlyeEHjARUHUfYy6kZuCIURQGSECIqDCEsxAgpIE2DEGMkFhuIVbBJL8nj+qBiqdrp6ZnklPpef7fr36la46p079unrSvz6nTlUrIjAzM7O+tV1fB2BmZmZOyGZmZoXghGxmZlYATshmZmYF4IRsZmZWAE7IZmZmBeCEbGaFJWmOpKur3PZkSXdVKP+VpJPK1ZX0oqS3Vth2uaRDq4mrQptvl7RM0guSvtCbbfcWSaMkhaQBVW7freNcsm2X63bSTtV/U1tbVQfVzCyPpNXAW4CNwEvAr4CZEfFiX8ZVKiImVigb1PFc0pVAW0ScnSnfdyuE9GXgjojYbyu0XUjZ49yduuXek3rgHrKZbQ3Hph+gBwBNwBYfnEr4M+gNI4Hl1WxYbY/VisX/Gcxsq4mItSQ95HcBSLpT0gWS7gZeBt4qaW9JzZKekdQqaXpJMztKujYdyr1f0ns7CiTNkvTntGyFpI+WbCtJ35W0TtJDkg7PFNwp6bPl4k6HZfeRNAP4JPDldMj0l2n5aklHpM+3y8TxtKSFkt6clu0o6ep0/XOSlkh6S5n93Q58GPhuup+xknaV9GNJ7ZIelXR2xxeYdOj3bknfkvQ0MKdMm7lxpeU/lfS39Nj8TtK+mbKdJH0z3e86SXdJ2inT/CclPSbpKUn/Xu4Ypu3skb63z0v6A/C2csc5U/eXad0lks4vGd6u+J6U2fe+kn6d/l09IemrOfUqHYej0r+rFyStlfSldP0QSTem7+kzkn7fG18unZDNbKuRNBw4CliaWX0iMAMYDDwKLADagL2BKcDXJB2WqT8Z+CnwZmA+8AtJ26dlfwYOAXYFzgWulrRXZtuD0jpDgHOAn2WTUmciYh7wE+AbETEoIo4tU+1U4Djgn9LX8CwwNy07KY1tOLAH8HlgfZn9HAb8nmRof1BEPAL8v3Tbt6Zt/zPw6ZLXtork9MAF3YwLki9KY4A9gfvT19nhYuBA4IMkx/3LwKZM+T8CbwcOB2ZLemeZ/ZPu7+/AXsC/pI88c0lOcfwDyXE7qVylrrwnkgYDtwE3k7z2fYDf5Oy30nG4DPhcRAwm+VJ5e7r+TJK/2UaS4/9VoOf3oY4IP/zww49eewCrgReB50gS7veAndKyO4HzMnWHk5xrHpxZ9x/AlenzOcDiTNl2wF+BQ3L2vQyYnD4/GXgcUKb8D8CJmVg+m6l7V6ZeAPukz68Ezi/zGo9In68EDs+U7QW8RjJH51+A/wbe04Xjlo2nAXgVGJcp/xxwZybexzppLzeuMnV3S1/zrukxXg+8t0y9UWm9YSXHdGqZug3p/t6RWfe1csc5U/ftmbLzu/OelOx7GrA0p2wOcHVO2evHIV1+LD3ubyqpdx5wQ0c8vfVwD9nMtobjImK3iBgZEf8aEdle4ZrM872BZyLihcy6R4Gh5epHxCbe6E0j6Z+VzEx+TtJzJL2YIZlt10b6CZppe+8evbItjQR+nolhJcmXjLcAVwG3AAskPS7pG5nefSVDgO3TeDvkHpfuxiWpQdKF6XD28yRfMDr2OwTYkWRkIc/fMs9fBspNzmok+VKSjfPRMvXy6nb2+ioZTuX4AejkOAB8nGSE51FJv5X0gXT9RUArcKukVZJm9SDW1zkh1wl1colHD9qVpCskPZueA+qs/maXRShzaUkXtu1yXdumZRPk48Cb0yHGDiOAtZnl4R1P0vN0w4DHJY0EfgjMBPaIiN2ABwFlth0qKbs8It1ntfGWswaYmH4B6XjsGBFrI+K1iDg3IsaRDP8eQzL03JmnSHqMI0tizx6XquMCTiA5FXAESa94VLqN0n3/nZLzvVVoBzaQef/S11Cp7rDMuuE5daFrr70rl0hVOg5ExJKImEwynP0LYGG6/oWIODMi3gpMAs5QZn5CtZyQu0DJBI716QSCjsd3+zquGvlH4EiSIarx3d04IiZGxI+6W7eaLxiSrpT0asn71NDdmK12ImINyZDufyiZAPUe4DNA9jrRAyV9LP2SdzrwCrAY2IXkg7kdQNKnSSePZewJfEHS9pI+AbwTWNTNMJ+g8of7D4AL0i8ISGqUNDl9/mFJ707/Dp8nSbKb8ptKRMRGkg//CyQNTts+g82PS2dy4yI5f/8K8DSwM8lQcse+NwGXA5comXDXIOkDknboxr47XsPPgDmSdpY0jvzzwqV130HlLy6dvSc3AntJOl3SDukxPKhMvdzjIGmgpE9K2jUiXiN5/zalZcekE8wErCMZeej0fe2ME3LXHRvJBIKOx8y+DqhGRgKrI+Klvg6ki75R8j5t7OuArFPTSHomjwM/B86JiNsy5TcA/4tkUtKJwMfSnucK4JvAPSQf0O8G7i5p+16SCTtPkUx8mhIRT3czvsuAcenQ7y/KlH8HaCYZvnyB5MtCx4f/PwDXkXyYrwR+SzKM3RWnkkxyWgXcRTKh7fJuxF0prh+TDB+vBVakZVlfAv4ELAGeAb5OdfliJslw9t9Izvte0UndXdO6VwHXkCTLciq+J+kpkCOBY9P2/odkFnupzo7DicDqdDj78ySzuyH5m7qNZK7EPcD3IuKOCq+ta3rzhHS9PshM4ChT9n3g+szy10lm8wnYneSbWjvJh8mNbD4Z4k6SiQv/nb6xvySZifkTkv/AS4BRmfoBfIHkP+hTJOcxtkvLTmbzCRDvAH5N8p/pYeD4Cq9vb5L/uM+QnBeZnq7/DMnQ1cY0vnPLbNtAMiPzqTSuU9I4B2ReY3aiyjfTun8h+Q+4RV2SXkx2v8918X26kgoTPfzww49t55F+lv6or+Oo5cM95J47E3h3OsR6CEkSOymSv6jtSL4RjiQ5d7IeKB3qnkryLWwoyTmbe9Jt3kzyjfqckvofJbnRwgEk5z62uIxA0i4kyXg+yZDdVOB76ZBROWUvO4mIy0i+Fd4TSW+zNBaA6STnxfZP45qSs4+OuhOB/dL4jytXKSJWlux3t/R1nSDpjxXaB/jX9LrA+yR9vJO6ZlYQkt4h6T3pvJXxJJ+lP+/ruGrJCbnrftExWzF9TAeIiJdJEuolJOd3To2ItrTs6Yi4PiJejmQI5QKSawKzroiIP0fEOpLr4f4cEbdFxAaSay/3L6n/9Yh4JiIeA75NMtxX6hiSYeYrImJDRCwFrgc+UVpRyXWiBwNfiYi/R8Qy4FK6NvEE4Hjg2xGxJiKeIblkpVLd70REW0Q8C1zYxX0AEBHzI+I9Far8J29cT/h/gCslHdydfZhZnxlMch75JeBaktG0G/o0ohrz7da67rjY/LzW6yLiXkmrSBLBwo71knYGvgVMIBm+BhgsqSHeOLf5RKap9WWWSy8nKL2EoNwlHCOBg9JLHToMoPy5q7zLTprK1C1n7zIxdbVuTy5r2EJE3J9ZXCTpJ8DH2PK8opkVTEQsIbkmud9yD7kXSDoF2IFkUsqXM0VnktzN5qCIeBPwoY5NerC70ksIyl3CsQb4bWx+ucOgiPjfZep25bKTSv5aJqZKdXvrsoauCHp2rM3MasY95B6SNJZkYtahJBfI/0HSr9Kh38EkvdznlNyur9w52O76N0n3kvScTyMZKi91I3ChpBNJzg9Dct72xfT87OsiYo2kjstOvgSMJTl380m6ZiHJZSU3kgw1VbpAfiFwmqSb0rpfqVD3CWCYpIER8WpXApE0heRWeS+TXFf4KZJZltbLhgwZEqNGjerrMMwK77777nsqIhq7UtcJuet+KSl7Cc2vSc7JXk1yXvcBACU3ML9KUhPJOd75JLOKHyc5J1J2IlM33ADcR3J5wJUk0/83ExEvSPoISbK+hGQk5AGS6xjLmUZyzeLjJLPBz8kbni/jhyRJ/AGSmeEXA4d1UvePad3/JPkiU+7SpNtJfvnmb5I2RcQQSZ8Evhr5P313GsnxEMks7ukRcWcXX4d1w6hRo2hpaenrMMwKT1Kl03ib102nl9s2QFIAYyKita9j6Q2SJgI/iIiRnVa2QmlqagonZLPOSbovIro0J8fnkK1mlPyk21GSBkgaSjKE368uazAzy+OEbLUkkp/Ie5bk5/hWArP7NCIzs4LwOeRtSERs0zOG02u239fXcZiZFZF7yGZmZgXghGxmZlYANR2yTmcJm1kntvXTE2bWfe4hm5mZFYATspmZWQF4lrWZmeUaNeumqrddfeHRvRhJ/XMP2czMrACckM3MzArACdnMzKwAnJDNzMwKwAnZzMysAJyQzeqcpAmSHpbUKmlWmfIRku6QtFTSHyUd1RdxmvV3TshmdUxSAzAXmAiMA6ZJGldS7WxgYUTsD0wFvlfbKM0MnJDN6t14oDUiVkXEq8ACYHJJnQDelD7fFXi8hvGZWcoJ2ay+DQXWZJbb0nVZc4BPSWoDFgGnlmtI0gxJLZJa2tvbt0asZv2aE7KZTQOujIhhwFHAVZK2+GyIiHkR0RQRTY2NjTUP0qzeOSGb1be1wPDM8rB0XdZngIUAEXEPsCMwpCbRmdnrnJDN6tsSYIyk0ZIGkkzaai6p8xhwOICkd5IkZI9Jm9WYE7JZHYuIDcBM4BZgJcls6uWSzpM0Ka12JjBd0gPANcDJEeHfLjerMf/ak1mdi4hFJJO1sutmZ56vAA6udVxmtjn3kM3MzArACdnMzKwAnJDNzMwKwAnZzMysAJyQzczMCsAJ2czMrACckM3MzArACdnMzKwAnJDNzMwKwAnZzMysAJyQzczMCsD3sratYvfdd88tGzFixFbZ56OPPppb9sUvfrHitg8++GBu2SOPPJJb9sADD3QemJlZF7iHbGZmVgBOyGZmZgXghGxmZlYATshmZmYF4IRsVuckTZD0sKRWSbPKlH9L0rL08Yik5/oiTrP+zrOszeqYpAZgLnAk0AYskdQcESs66kTEFzP1TwX2r3mgZuaEbJUdffTRuWWTJk3KLTv00ENzy/bZZ5+ehJSr0uVJI0eOrLjtDjvsUNU+GxoaqtquhsYDrRGxCkDSAmAysCKn/jTgnBrFZmYZHrI2q29DgTWZ5bZ03RYkjQRGA7fnlM+Q1CKppb29vdcDNevvnJDNrMNU4LqI2FiuMCLmRURTRDQ1NjbWODSz+ueEbFbf1gLDM8vD0nXlTAWu2eoRmVlZTshm9W0JMEbSaEkDSZJuc2klSe8AdgfuqXF8ZpZyQjarYxGxAZgJ3AKsBBZGxHJJ50nKzsqbCiyIiOiLOM3Ms6zN6l5ELAIWlaybXbI8p5YxmdmWnJDrxNve9rbcslNOOSW3bPr06RXb3WmnnXLLJHUeWA2NHTu2r0MwM6uah6zNzMwKwAnZzMysAJyQzczMCsAJ2czMrACckM3MzArACdnMzKwAnJDNzMwKwNch14lhw4bllp122mk1jGTreuihh3LLli9fXsNIzMx6l3vIZmZmBeCEbGZmVgBOyGZmZgXghGxmZlYATshmZmYF4IRsZmZWAL7saSsYMmRIblmlS5Duvvvuiu3efPPNuWWvvPJKbtm6detyy1566aWK+9xll11yy2699dbcsgcffDC37N577624z6VLl+aWrV+/Press9fSX0maAHwHaAAujYgLy9Q5HpgDBPBARJxQ0yDNzAnZrJ5JagDmAkcCbcASSc0RsSJTZwxwFnBwRDwrac++idasf/OQtVl9Gw+0RsSqiHgVWABMLqkzHZgbEc8CRMSTNY7RzHBCNqt3Q4E1meW2dF3WWGCspLslLU6HuM2sxjxkbWYDgDHAocAw4HeS3h0Rz2UrSZoBzAAYMWJErWM0q3vuIZvVt7XA8MzysHRdVhvQHBGvRcRfgEdIEvRmImJeRDRFRFNjY+NWC9isv3JCNqtvS4AxkkZLGghMBZpL6vyCpHeMpCEkQ9irahmkmXnIumrVXg703ve+N7fsox/9aNXxLF68OLfsgAMOyC1bvXp1xXYrDU22tbXllm3atKliu1YbEbFB0kzgFpLLni6PiOWSzgNaIqI5LfuIpBXARuDfIuLpvovarH9yQjarcxGxCFhUsm525nkAZ6QPM+sjHrI2MzMrACdkMzOzAnBCNjMzKwAnZDMzswJwQjYzMysAz7LOMXDgwIrl8+fPzy2rdGnT1772tdyy2267rfPAqtDZpU2VPPbYY70XiJmZ5XIP2czMrACckM3MzArACdnMzKwAnJDNzMwKwAnZzMysAJyQzczMCsAJ2czMrAD69XXIgwYNyi0766yzKm57zDHH5JY99dRTuWUXX3xxbtnLL79ccZ9mZla/3EM2MzMrACdkMzOzAnBCNqtzkiZIelhSq6RZZcpPltQuaVn6+GxfxGnW3/Xrc8hm9U5SAzAXOBJoA5ZIao6IFSVVr42ImTUP0Mxe5x6yWX0bD7RGxKqIeBVYAEzu45jMrAwnZLP6NhRYk1luS9eV+rikP0q6TtLwcg1JmiGpRVJLe3v71ojVrF/r10PWxx13XG7ZrFlbnGrbTKWfJTzkkENyy9atW9d5YGa19Uvgmoh4RdLngB8Bh5VWioh5wDyApqamqG2IZvXPPWSz+rYWyPZ4h6XrXhcRT0fEK+nipcCBNYrNzDKckM3q2xJgjKTRkgYCU4HmbAVJe2UWJwEraxifmaX69ZC1Wb2LiA2SZgK3AA3A5RGxXNJ5QEtENANfkDQJ2AA8A5zcZwGb9WNOyGZ1LiIWAYtK1s3OPD8LqHyvWDPb6jxkbWZmVgBOyGZmZgXQr4esP/jBD1a97dKlS3PL2traqm7XzMz6J/eQzczMCsAJ2czMrACckM3MzArACdnMzKwAnJDNzMwKwAnZzMysAPr1ZU9TpkypetsJEybklp1zzjm5ZTfccENu2bJly6qOx8zMtm3uIZuZmRWAE7KZmVkBOCGbmZkVgBOymZlZATghm5mZFYATspmZWQE4IZvVOUkTJD0sqVXSrAr1Pi4pJDXVMj4zS/Tr65AbGxtzyzZt2lRx2x122CG3bPbs2bllZ599dm7ZD37wg4r7XLx4cW7ZiBEjcstaW1tzy5YvX15xn5Xsu+++uWX33HNPbpl/nrJ2JDUAc4EjgTZgiaTmiFhRUm8wcBpwb+2jNDNwD9ms3o0HWiNiVUS8CiwAJpep93+BrwN/r2VwZvYGJ2Sz+jYUWJNZbkvXvU7SAcDwiLipUkOSZkhqkdTS3t7e+5Ga9XNOyGb9mKTtgEuAMzurGxHzIqIpIpoqne4xs+o4IZvVt7XA8MzysHRdh8HAu4A7Ja0G3g80e2KXWe05IZvVtyXAGEmjJQ0EpgLNHYURsS4ihkTEqIgYBSwGJkVES9+Ea9Z/OSGb1bGI2ADMBG4BVgILI2K5pPMkTerb6MwsSxFRu51JtdtZF1x00UW5ZWeccUYNI6k/lSb93HnnnRW3nTp1ai9Hs+2JCPV1DJU0NTVFS4s70f3BqFkV5/pVtPrCo3sxkm2TpPsiokungNxDNjMzKwAnZDMzswJwQjYzMysAJ2QzM7MCcEI2MzMrACdkMzOzAujXv/Y0a1buL9Fx7bXXVtx2/vz5uWUDBuQf1uHDh+eWbbdd/Xw/qnRrxSlTplTcttIvYp1//vlVx2RmVmT1kwHMzMy2YU7IZmZmBeCEbGZmVgBOyGZmZgXghGxmZlYATshmZmYF0K8ve9q4cWNuWWe/ZDN27Niq9nn44Yfnlm2//fYVt50zZ05u2fve976q4ukLUuUfMjrwwANrFImZWXG4h2xmZlYATshmZmYF4IRsVuckTZD0sKRWSVvcnk7S5yX9SdIySXdJGtcXcZr1d07IZnVMUgMwF5gIjAOmlUm48yPi3RGxH/AN4JIah2lmOCGb1bvxQGtErIqIV4EFwORshYh4PrO4CxA1jM/MUv16lrVZPzAUWJNZbgMOKq0k6RTgDGAgcFi5hiTNAGYAjBgxotcDNevv3EM2MyJibkS8DfgKUPbntiJiXkQ0RURTpV/zMrPquIdcY7/5zW+q3na//fbLLat0HfKGDRtyy6644oqK+/zhD3+YW3b66afnlp1wwgkV27WaWQtkf/NzWLouzwLg+1s1IjMryz1ks/q2BBgjabSkgcBUoDlbQdKYzOLRwP/UMD4zS7mHbFbHImKDpJnALUADcHlELJd0HtASEc3ATElHAK8BzwIn9V3EZv2XE7JZnYuIRcCiknWzM89Pq3lQZrYFD1mbmZkVgBOymZlZATghm5mZFYDPIW9Dbr311tyyCy64ILdswID8t3n69OkV97nPPvvklh166KEVt61WW1vbVmnXzKzI3EM2MzMrACdkMzOzAnBCNjMzKwAnZDMzswJwQjYzMysAJ2QzM7MC8GVP25CVK1fmli1cuDC37Pjjj696nx/+8Ier2m7jxo25ZTfddFPFbWfNmlXVPs3MtmXuIZuZmRWAE7KZmVkBOCGbmZkVgBOymZlZATghm9U5SRMkPSypVdIWM+YknSFphaQ/SvqNpJF9EadZf+eEbFbHJDUAc4GJwDhgmqRxJdWWAk0R8R7gOuAbtY3SzMCXPW1T1q9fn1t2+umn55YNGjQot6ypqaniPvfcc8/cstWrV+eWXXXVVbllc+bMqbhP61XjgdaIWAUgaQEwGVjRUSEi7sjUXwx8qqYRmhngHrJZvRsKrMkst6Xr8nwG+FW5AkkzJLVIamlvb+/FEM0MnJDNLCXpU0ATcFG58oiYFxFNEdHU2NhY2+DM+gEPWZvVt7XA8MzysHTdZiQdAfw78E8R8UqNYjOzDPeQzerbEmCMpNGSBgJTgeZsBUn7A/8FTIqIJ/sgRjPDCdmsrkXEBmAmcAuwElgYEcslnSdpUlrtImAQ8FNJyyQ15zRnZluRh6zN6lxELAIWlaybnXl+RM2DMrMtuIdsZmZWAO4h14knnngit+zYY4/NLTvxxBMrtvv+978/t+zcc8/NLXvySZ+KNDPrDveQzczMCsAJ2czMrACckM3MzArA55DNzOrMqFk3Vb3t6guP7sVIrDvcQzYzMysAJ2QzM7MC8JB1P1fpZxK7Um5mZr3DPWQzM7MCcEI2MzMrACdkMzOzAnBCNjMzKwAnZDMzswJwQjYzMysAJ2QzM7MCcEI2q3OSJkh6WFKrpFllyj8k6X5JGyRN6YsYzcwJ2ayuSWoA5gITgXHANEnjSqo9BpwMzK9tdGaW5Tt1mdW38UBrRKwCkLQAmAys6KgQEavTsk19EaCZJdxDNqtvQ4E1meW2dF23SZohqUVSS3t7e68EZ2ZvcEI2sy6JiHkR0RQRTY2NjX0djlndcUI2q29rgeGZ5WHpOjMrGCdks/q2BBgjabSkgcBUoLmPYzKzMpyQzepYRGwAZgK3ACuBhRGxXNJ5kiYBSHqfpDbgE8B/SVredxGb9V+eZW1W5yJiEbCoZN3szPMlJEPZZtaH3EM2MzMrACdkMzOzAnBCNjMzKwAnZDMzswJwQjYzMysAJ2QzM7MCcEI2MzMrACdkMzOzAnBCNjMzKwAnZDMzswJwQjYzMysAJ2QzM7MCqOmPS0SEark/MzOzbYV7yGZmZgXghGxmZlYATshmZmYF4IRsVuckTZD0sKRWSbPKlO8g6dq0/F5Jo2ofpZk5IZvVMUkNwFxgIjAOmCZpXEm1zwDPRsQ+wLeAr9c2SjODGs+yNrOaGw+0RsQqAEkLgMnAikydycCc9Pl1wHclKSKipzsfNeumqrddfeHRPd291al6/btyQjarb0OBNZnlNuCgvDoRsUHSOmAP4KlsJUkzgBnp4ouSHu5hbENK97HZ/rrXT6/YVjcVsa2axdSbx72I72E3Y6rYVheN7GpFJ2Qz65KImAfM6632JLVERJPb2jZjclu9z+eQzerbWmB4ZnlYuq5sHUkDgF2Bp2sSnZm9zgnZrL4tAcZIGi1pIDAVaC6p0wyclD6fAtzeG+ePzax7PGRtVsfSc8IzgVuABuDyiFgu6TygJSKagcuAqyS1As+QJO1a6LXh737QVhFjclu9TP4ibGZm1vc8ZG1mZlYATshmZmYF4IRsZjXX2e08u9HO5ZKelPRgD+MZLukOSSskLZd0Wg/a2lHSHyQ9kLZ1bk9iS9tskLRU0o09bGe1pD9JWiappYdt7SbpOkkPSVop6QNVtvP2NJ6Ox/OSTq+yrS+mx/xBSddI2rGadtK2TkvbWV5tPN3ep88hm1ktpbfzfAQ4kuRGJUuAaRGxouKG5dv6EPAi8OOIeFcPYtoL2Csi7pc0GLgPOK7KmATsEhEvStoeuAs4LSIW9yC+M4Am4E0RcUwP2lkNNEVEj2/AIelHwO8j4tJ0Bv/OEfFcD9tsILkM76CIeLSb2w4lOdbjImK9pIXAooi4soo43gUsILnT3avAzcDnI6K1u211h3vIZlZrr9/OMyJeJfngm1xNQxHxO5KZ4T0SEX+NiPvT5y8AK0nuYFZNWxERL6aL26ePqns+koYBRwOXVttGb5O0K/Ahkhn6RMSrPU3GqcOBP3c3GWcMAHZKr6ffGXi8ynbeCdwbES9HxAbgt8DHqmyry5yQzazWyt3Os6rktzWkv3a1P3BvD9pokLQMeBL4dURU3RbwbeDLwKYetNEhgFsl3ZfeCrVao4F24Ip0KP1SSbv0QnxTgWuq2TAi1gIXA48BfwXWRcStVcbxIHCIpD0k7QwcxeY32NkqnJDNzFKSBgHXA6dHxPPVthMRGyNiP5I7o41Ph0CriecY4MmIuK/aWEr8Y0QcQPLrX6ekQ/7VGAAcAHw/IvYHXgKqngsAkA57TwJ+WuX2u5OMtIwG9gZ2kfSpatqKiJUkv3p2K8lw9TJgYzVtdYcTspnVWldu51lz6fne64GfRMTPeqPNdBj3DmBClU0cDExKz/0uAA6TdHUP4lmb/vsk8HOS0wfVaAPaMj3/60gSdE9MBO6PiCeq3P4I4C8R0R4RrwE/Az5YbTARcVlEHBgRHwJEKiFeAAABLUlEQVSeJZn3sFU5IZtZrXXldp41lU7EugxYGRGX9LCtRkm7pc93Ipm89lA1bUXEWRExLCJGkRyn2yOiql6fpF3SCWukw8sfIRmarSauvwFrJL09XXU4m/+kZzWmUeVwdeox4P2Sdk7fz8NJ5gJURdKe6b8jSM4fz+9BbF3iW2eaWU3l3c6zmrYkXQMcCgyR1AacExGXVdHUwcCJwJ/Sc78AX42IRVW0tRfwo3TG8HbAwojo0eVKveQtwM+TXMUAYH5E3NyD9k4FfpJ+qVoFfLrahtIvCEcCn6u2jYi4V9J1wP3ABmApPbvt5fWS9gBeA07ppUlrFfmyJzMzswLwkLWZmVkBOCGbmZkVgBOymZlZATghm5mZFYATspmZWQE4IZuZmRWAE7KZmVkB/H/8vjRBwZnEhQAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<Figure size 576x288 with 2 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAeQAAAEICAYAAACOKIcAAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDMuMC4xLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvDW2N/gAAHjpJREFUeJzt3Xu4XGV59/HvjwACisdEwSQQrBQbzzaF9m2lngsewFbbcqinWimvRjxVRduCYqUgFltf0ZYiaEUExFNELGrxUKxggqJyso00moBoOIgoWIjc7x9rbTrZzEx2Zu/MXux8P9c1V2bmftaz7pm9s+9Zz3qeNakqJEnS7NpmthOQJEkWZEmSOsGCLElSB1iQJUnqAAuyJEkdYEGWJKkDLMiSOivJW5KcPuK2L05y4ZD4Z5O8qF/bJD9L8rAh216e5Emj5DWkz72SXJrkliRHzGTfMyXJkiSVZNsRt9+s93nStlNuu4l+Rv6d2tJGelMlaZAka4CHAL8Efg58FlheVT+bzbwmq6r9h8TuM3E/yQeAdVX1Vz3xR26BlN4AfLGqHrcF+u6k3vd5c9r2+5nMBR4hS9oSntP+AX0CsAy42x/ONPwb9L92By4fZcNRj1jVLf5nkLTFVNU1NEfIjwJI8qUkb0/yVeBW4GFJHppkRZIbk6xO8rJJ3eyQ5Kx2KPcbSR47EUhyZJLvtbErkvz+pG2T5D1Jbk5yVZKn9gS+lOTP+uXdDss+PMlhwKHAG9oh00+38TVJntbe36YnjxuSnJ3kgW1shySnt8//JMnKJA/ps78LgCcD72n386tJ7pfkX5KsT/L9JH818QGmHfr9apJ3JbkBeEufPgfm1cY/muS69r35SpJH9sR2TPJ37X5vTnJhkh17uj80yQ+SXJ/kL/u9h20/D2p/tj9N8nXgV/q9zz1tP922XZnkbyYNbw/9mfTZ9yOTfL79vfpRkjcPaDfsfXhm+3t1S5JrkvxF+/z8JOe2P9Mbk/z7THy4tCBL2mKSLAaeCXyz5+kXAIcBOwPfB84E1gEPBZ4PHJvkKT3tDwQ+CjwQOAP4ZJLt2tj3gCcC9wPeCpyeZNeebfdp28wHjgY+3luUNqWqTgY+DLyjqu5TVc/p0+yVwHOB321fw03ASW3sRW1ui4EHAYcDt/XZz1OAf6cZ2r9PVf0n8P/abR/W9v1C4CWTXtvVNKcH3r6ZeUHzQWlP4MHAN9rXOeGdwK8D/4fmfX8DcGdP/HeAvYCnAkcl+bU++6fd3y+AXYE/bW+DnERzimMXmvftRf0aTeVnkmRn4AvAv9K89ocD/zZgv8Peh/cDf15VO9N8qLygff51NL+zC2je/zcD078OdVV58+bN24zdgDXAz4Cf0BTc9wI7trEvAcf0tF1Mc655557n/hb4QHv/LcBFPbFtgB8CTxyw70uBA9v7LwauBdIT/zrwgp5c/qyn7YU97Qp4eHv/A8Df9HmNT2vvXwk8tSe2K3AHzRydPwX+A3jMFN633nzmAbcDS3vifw58qSffH2yiv4F59Wl7//Y13699j28DHtun3ZK23aJJ7+lBfdrOa/f3iJ7nju33Pve03asn9jeb8zOZtO+DgW8OiL0FOH1A7K73oX38g/Z9v++kdscAn5rIZ6ZuHiFL2hKeW1X3r6rdq+rlVdV7VLi25/5DgRur6pae574PLOzXvqru5H+PpknywjQzk3+S5Cc0RzHze7a9ptq/oD19P3Rar+zudgc+0ZPDlTQfMh4CfAg4HzgzybVJ3tFzdD/MfGC7Nt8JA9+Xzc0rybwkx7XD2T+l+YAxsd/5wA40IwuDXNdz/1ag3+SsBTQfSnrz/H6fdoPabur1DbOY4fkDsIn3AeB5NCM830/y5SS/1T5/ArAa+FySq5McOY1c72JBniOyiSUe0+g3SU5LclN7DmhT7TdaFpGepSVT2HbKbXWP1lsgrwUe2A4xTtgNuKbn8eKJO+15ukXAtUl2B/4ZWA48qKruD1wGpGfbhUl6H+/W7nPUfPtZC+zffgCZuO1QVddU1R1V9daqWkoz/PtsmqHnTbme5ohx90m5974vI+cFHEJzKuBpNEfFS9pt0u77F0w63zuC9cAGen5+7WsY1nZRz3OLB7SFqb32qSyRGvY+UFUrq+pAmuHsTwJnt8/fUlWvq6qHAQcAr03P/IRRWZCnIM0EjtvaCQQTt/fMdl5j8jvA02mGqPbe3I2rav+q+uDmth3lA0aShUk+1U6yWJfk8M3NV+NVVWtphnT/Ns0EqMcALwV614n+epI/aD/kvRr4H+Ai4N40f5jXAyR5Ce3ksR4PBo5Isl2SPwR+DThvM9P8EcP/uP8j8Pb2AwJJFiQ5sL3/5CSPTjIP+ClNkb1zcFeNqvolzR//tyfZue37tWz8vmzKwLxozt//D3ADsBPNUPLEvu8ETgVOTDPhbl6S30pyr83Y98Rr+DjwliQ7JVnK4PPCk9s+guEfXDb1MzkX2DXJq5Pcq30P9+nTbuD7kGT7JIcmuV9V3UHz87uzjT27nWAW4GaakYdN/lw3xYI8dc+pZgLBxG35bCc0JrsDa6rq57OdyBScDvw3zVDhs2gmBz15dlPSFBxMc2RyLfAJ4Oiq+kJP/FPAH9NMSnoB8AftkecVwN8BX6P5A/1o4KuT+r6YZsLO9TQTn55fVTdsZn7vB5a2Q7+f7BP/B2AFzfDlLTQfFib++O8CnEPzx/xK4Ms0w9hT8UqaSU5XAxfSTGg7dTPyHpbXv9AMH18DXNHGev0F8B1gJXAjcDyj1YvlNMPZ19Gc9z1tE23v17b9EPARmmLZz9CfSXsK5OnAc9r+/otmFvtkm3ofXgCsaYezD6eZ3Q3N79QXaOZKfA14b1V9cchrm5qZPCE9V2/0TODoE3sf8LGex8fTzOYL8ACaT2rraf6YnMvGkyG+RDNx4T/aH+ynaWZifpjmP/BKYElP+wKOoPkPej3NeYxt2tiL2XgCxCOAz9P8Z/ou8EdDXt9Daf7j3khzXuRl7fMvpRm6+mWb31v7bDuPZkbm9W1er2jz3LbnNfZOVPm7tu1/0/wHvFtbmqOY3v3+ZAo/o/u0fS3oee5k4EOz/fvjzZu3zb+1f0s/ONt5jPPmEfL0vQ54dDvE+kSaIvaian6jtqH5RLg7zbmT24DJQ90H0XwKW0hzzuZr7TYPpPlEffSk9r9Pc6GFJ9Cc+7jbMoIk96YpxmfQDNkdBLy3HTLqp++yk6p6P82nwq9VMyowOReAl9GcF3t8m9fzB+xjou3+wOPa/J/br1FVXTlpv/dvX9chSb49oO9M+nfi/uQhTEkdlOQRSR7TzlvZm+Zv6SdmO69xsiBP3ScnZiu2t5cBVNWtNAX1RJoh01dW1bo2dkNVfayqbq1mCOXtNGsCe51WVd+rqptp1sN9r6q+UFUbaNZePn5S++Or6saq+gHw9zTDfZM9m2aY+bSq2lBV3wQ+Bvzh5IZp1on+NvDGqvpFVV0KnMLUJp4A/BHw91W1tqpupFmyMqztP1TVuqq6CThuivsAoKrOqKrHDIjdQjNc+dftucgn0MyQ3Glz9iFp1uxMcx7558BZNKNpn5rVjMbMy61N3XNr4/Nad6mqi5NcTXM0evbE80l2At4F7EczfA2wc5J51UxigObc14Tb+jyevJxg8hKCfks4dgf2aZc6TNiW/ueuBi07WdanbT8P7ZPTVNtOZ1lDP4fSXFxgLc3w+enAlrjmsKQZVlUradYkb7U8Qp4BSV4B3ItmUsobekKvo7mazT5VdV9g34lNprG7yUsI+i3hWAt8uTZe7nCfqvq/fdpOZdnJMD/sk9OwtjO1rOHuG1R9v6qeXVULqmofmrWEm1yqJUld4BHyNCX5VZqJWU+iWSD/9SSfbYd+d6Y5yv1Jmsv19TsHu7len+RimiPnV9EMlU92LnBckhfQnB+G5rztz9rzs3epqrVJJpad/AXwqzTnbg5las6mWVZyLs1Q07AF8mcDr0rymbbtG4e0/RGwKMn2VXX7VBJJc/m+dTQzM/8IeAbNBLGtWpL9aGbczgNOqarjJsVfTDNBcOJD2Huq6pRhfc6fP7+WLFky88lKc8wll1xyfVUtmEpbC/LUfTrJL3sef57mnOzpNOd1vwWQ5gLmH0qyjOYc7xk0s4qvpTkn0nci02b4FHAJzfKAD9BM/99IVd2S5Bk0xfpEmpGQb9GsY+znYJo1i9fSzAY/etDwfB//TFPEv0UzM/ydwFM20fbbbdt303yQ+WWfthfQfPPNdUnurKr5SQ4F3lyDv/ru94C/pDlv/E1gv6paP8XXMSe1619PolkCsg5YmWRFNUuGep1Vm7GUb8mSJaxatWoGM5XmpiTDTuNt3LadXq57gCQF7FlVq2c7l5mQZH/gH6tq90021kjSXOrvLVX1e+3jNwFU1d/2tHkxsGxzCvKyZcvKgixtWpJLqmpKc3I8h6yxSfOVbs9Msm2ShTRD+FvVsoZZsJCNJ8+tY+PrIU94XpJvJzmnnXl/N0kOS7Iqyar167fqgQdpi7Aga5xC8xV5N9EMKV8JHDWrGQmaC9IsaZeUfR7oe6nTqjq5qpZV1bIFC6Z0SkzSZvAc8j1IVU1ndvasa9ds/8Zs57GVuYaNZ7MvYtIM+tr4UpKnAO8YQ16SJvEIWZrbVgJ7JtkjyfY0V21b0dsgya49Dw+gGbmQNGYeIUtzWFVtSLKc5jt55wGnVtXlSY4BVlXVCpplawfQfP3djTTXRZc0ZmOdZd3OEpa0CV0/PeEsa2lqnGUtSdI9jAVZkqQO8ByyJHXAkiM/M/K2a4571gxmotniEbIkSR1gQZYkqQMsyJIkdYAFWZKkDrAgS5LUARZkSZI6wIIsSVIHWJAlSeoAC7IkSR1gQZYkqQMsyJIkdYAFWZKkDrAgS5LUARZkSZI6wIIsSVIHWJAlSeoAC7IkSR1gQZYkqQMsyJIkdYAFWZKkDrAgS5LUARZkSZI6wIIsSVIHWJAlSeoAC7IkSR1gQZYkqQMsyJIkdYAFWZKkDrAgS5LUARZkSZI6wIIsSVIHWJClOS7Jfkm+m2R1kiOHtHtekkqybJz5SWpYkKU5LMk84CRgf2ApcHCSpX3a7Qy8Crh4vBlKmmBBlua2vYHVVXV1Vd0OnAkc2Kfd24DjgV+MMzlJ/8uCLM1tC4G1PY/Xtc/dJckTgMVV9ZlhHSU5LMmqJKvWr18/85lKWzkLsrQVS7INcCLwuk21raqTq2pZVS1bsGDBlk9O2spYkKW57Rpgcc/jRe1zE3YGHgV8Kcka4DeBFU7sksbPgizNbSuBPZPskWR74CBgxUSwqm6uqvlVtaSqlgAXAQdU1arZSVfaelmQpTmsqjYAy4HzgSuBs6vq8iTHJDlgdrOT1Gvb2U5As2v77bcfGj/iiCMGxo4++uiBsRtuuGFg7MEPfvDQfT796U8fGPvqV786MLb77rsPjB1yyCFD93n88ccPjN15551Dt+26qjoPOG/Sc0cNaPukceQk6e48QpYkqQMsyJIkdYAFWZKkDrAgS5LUARZkSZI6wIIsSVIHWJAlSeoA1yFvBbbZZvDnrne9611Dt1269G7f1HeX5cuXD4ydddZZA2Pvfe97h+7z6quvHhh7wAMeMDD2hS98YWBshx12GLrPU045ZWDML1KQNA4WZEnSVmvJkUO/5GyoNcc9awYzcchakqROsCBLktQBFmRJkjrAgixJUgdYkCVJ6gBnWc8R973vfQfGTjvttIGx6667bmi/b3rTmwbGLrrook0n1sepp546NH799dcPjH35y18eGBu2tGn//fcfuk+XNkmabR4hS5LUARZkSZI6wIIsSVIHWJAlSeoAC7IkSR1gQZYkqQNc9jRH7LfffgNjCxcuHBgb9o1NAD/84Q9HzmmQCy+8cGj8yU9+8sDYdtttNzB2+OGHD4xddtllm05MkmaRR8iSJHWABVmSpA6wIEuS1AEWZEmSOsCCLElSB1iQJUnqAJc9zRF77bXXwNhXvvKVgbEtsaxpUx75yEcOja9YsWJg7JZbbhkYW7t27cg5SdJs8whZkqQOsCBLktQBFmRJkjrAgixJUgdYkCVJ6gALsjTHJdkvyXeTrE5yZJ/44Um+k+TSJBcmWTobeUpbOwuyNIclmQecBOwPLAUO7lNwz6iqR1fV44B3ACeOOU1JuA75HuWEE04YGNt3330Hxp74xCduiXRG9vKXv3xofMcddxwYe8lLXjIw9u1vf3vknOawvYHVVXU1QJIzgQOBKyYaVNVPe9rfG6ixZigJsCBLc91CoPeKKeuAfSY3SvIK4LXA9sBT+nWU5DDgMIDddtttxhOVtnYOWUuiqk6qql8B3gj81YA2J1fVsqpatmDBgvEmKG0FLMjS3HYNsLjn8aL2uUHOBJ67RTOS1JcFWZrbVgJ7JtkjyfbAQcBGFwtPsmfPw2cB/zXG/CS1PIcszWFVtSHJcuB8YB5walVdnuQYYFVVrQCWJ3kacAdwE/Ci2ctY2npZkKU5rqrOA86b9NxRPfdfNfakJN2NBblDDjrooJHjw75+8fbbbx85p2Ee+MAHDowde+yxA2Mve9nLhvZ74omDl8Gec845m05Mku6BPIcsSVIHWJAlSeoAC7IkSR1gQZYkqQMsyJIkdYAFWZKkDnDZ05htu+3gt/z4448fuu1f//VfD4zdeuutI+WzzTbDP5M95Sl9v2cAgKOOOmpgbNiXD7zmNa8Zus93v/vdQ+OSNBd5hCxJUgdYkCVJ6gALsiRJHWBBliSpAyzIkiR1gAVZkqQOcNnTmD3qUY8aGBu2JArg3HPPHRhbtGjRwNiSJUsGxg499NCh+zz88MMHxoZ9i9QBBxwwMHb++ecP3ackbY08QpYkqQMsyJIkdYAFWZKkDrAgS5LUARZkSZI6wIIsSVIHWJAlSeoA1yGP2S9+8YuBsfnz5w/d9nOf+9zA2GMf+9iBsTVr1gyM3XzzzUP3edtttw2MXXbZZQNjrjWWpM3jEbIkSR1gQZYkqQMsyJIkdYAFWZKkDrAgS5LUARZkSZI6wGVPY3bVVVcNjL361a8euu0hhxwyMHbUUUcNjJ100kkDY2984xuH7vMRj3jEwNiFF144dFtJ0tR5hCxJUgdYkCVJ6gALsiRJHWBBliSpAyzI0hyXZL8k302yOsmRfeKvTXJFkm8n+bcku89GntLWzoIszWFJ5gEnAfsDS4GDkyyd1OybwLKqegxwDvCO8WYpCVz21Cnve9/7phUf5IgjjhgY29Syp4suumjkbdUJewOrq+pqgCRnAgcCV0w0qKov9rS/CPiTsWYoCfAIWZrrFgJrex6va58b5KXAZ/sFkhyWZFWSVevXr5/BFCWBBVlSK8mfAMuAE/rFq+rkqlpWVcsWLFgw3uSkrYBD1tLcdg2wuOfxova5jSR5GvCXwO9W1f+MKTdJPTxClua2lcCeSfZIsj1wELCit0GSxwP/BBxQVT+ehRwlYUGW5rSq2gAsB84HrgTOrqrLkxyT5IC22QnAfYCPJrk0yYoB3Unaghyylua4qjoPOG/Sc0f13H/a2JOSdDcW5Dli8eLFA2PvfOc7B8Zuuummof0+73nPGxi74447Np2YJGlKHLKWJKkDLMiSJHWABVmSpA6wIEuS1AEWZEmSOsCCLElSB1iQJUnqANch34PMmzdvYOyCCy4YGLvqqqsGxp7xjGcM3ed111236cQkSdPmEbIkSR1gQZYkqQMsyJIkdYAFWZKkDrAgS5LUARZkSZI6wGVP9yCvfOUrB8Z22WWXgbGdd955S6QjSZpBHiFLktQBFmRJkjrAgixJUgdYkCVJ6gALsiRJHWBBliSpA1z21CFPetKThsbf9ra3DYwdf/zxM5yNJGmcPEKWJKkDLMiSJHWABVmSpA6wIEuS1AEWZEmSOsCCLElSB7jsacxe+MIXDoydcMIJQ7f95Cc/OTB23HHHjZyTJGn2eYQsSVIHWJAlSeoAC7IkSR1gQZYkqQMsyNIcl2S/JN9NsjrJkX3i+yb5RpINSZ4/GzlKsiBLc1qSecBJwP7AUuDgJEsnNfsB8GLgjPFmJ6mXy56kuW1vYHVVXQ2Q5EzgQOCKiQZVtaaN3TkbCUpqWJDH7PWvf/3A2M9//vOh2x522GEDYxs2bBg5J81pC4G1PY/XAfuM0lGSw4DDAHbbbbfpZyZpIw5ZS5qSqjq5qpZV1bIFCxbMdjrSnGNBlua2a4DFPY8Xtc9J6hgLsjS3rQT2TLJHku2Bg4AVs5yTpD4syNIcVlUbgOXA+cCVwNlVdXmSY5IcAJDkN5KsA/4Q+Kckl89extLWy0ld0hxXVecB50167qie+ytphrIlzSKPkCVJ6gCPkDvktNNOGxq/7bbbxpSJJGncPEKWJKkDLMiSJHWABVmSpA6wIEuS1AFO6pIkbXFLjvzMyNuuOe5ZM5hJd3mELElSB3iEvAXssssuA2PHHnvswNhZZ521JdKRJN0DeIQsSVIHWJAlSeoAC7IkSR3gOWRJ0j3KXJ2x7RGyJEkdYEGWJKkDHLLeAq677rqBsY985CNjzESSdE/hEbIkSR1gQZYkqQMsyJIkdYAFWZKkDrAgS5LUARZkSZI6wIIsSVIHWJAlSeoAC7IkSR1gQZYkqQMsyJIkdYAFWZKkDrAgS5LUARZkSZI6YKxfv1hVGef+JEm6p/AIWZKkDrAgS5LUAWMdspakuWTJkZ8Zeds1xz1rBjPRXOARsjTHJdkvyXeTrE5yZJ/4vZKc1cYvTrJk/FlK8ghZmsOSzANOAp4OrANWJllRVVf0NHspcFNVPTzJQcDxwB+PP9vx8KhWXWVBlua2vYHVVXU1QJIzgQOB3oJ8IPCW9v45wHuSpKpqnIlq5szkhw4/wIxP/D8nzV1Jng/sV1V/1j5+AbBPVS3vaXNZ22Zd+/h7bZvrJ/V1GHBY+3Av4LvTTG8+cP0mW9nXTPZjX+Pva/eqWjCVhh4hS5qSqjoZOHmm+kuyqqqW2dc9Myf7mnlO6pLmtmuAxT2PF7XP9W2TZFvgfsANY8lO0l0syNLcthLYM8keSbYHDgJWTGqzAnhRe//5wAWeP5bGzyFraQ6rqg1JlgPnA/OAU6vq8iTHAKuqagXwfuBDSVYDN9IU7XGYseHvraCvLuZkXzPMSV2SJHWAQ9aSJHWABVmSpA6wIEsau01dznMz+jk1yY/btdTTyWdxki8muSLJ5UleNY2+dkjy9STfavt663Rya/ucl+SbSc6dZj9rknwnyaVJVk2zr/snOSfJVUmuTPJbI/azV5vPxO2nSV49Yl+vad/zy5J8JMkOo/TT9vWqtp/LR81ns/fpOWRJ49RezvM/6bmcJ3DwpMt5TrWvfYGfAf9SVY+aRk67ArtW1TeS7AxcAjx3xJwC3LuqfpZkO+BC4FVVddE08nstsAy4b1U9exr9rAGWTb7oy4h9fRD496o6pZ3Bv1NV/WSafc6jWYa3T1V9fzO3XUjzXi+tqtuSnA2cV1UfGCGPRwFn0lzp7nbgX4HDq2r15va1OTxCljRud13Os6pup/nDd+AoHVXVV2hmhk9LVf2wqr7R3r8FuBJYOGJfVVU/ax9u195GPvJJsgh4FnDKqH3MtCT3A/almaFPVd0+3WLceirwvc0txj22BXZs19PvBFw7Yj+/BlxcVbdW1Qbgy8AfjNjXlFmQJY3bQmBtz+N1jFj8toT2264eD1w8jT7mJbkU+DHw+aoauS/g74E3AHdOo48JBXwuySXtpVBHtQewHjitHUo/Jcm9ZyC/g4CPjLJhVV0DvBP4AfBD4Oaq+tyIeVwGPDHJg5LsBDyTjS+ws0VYkCWpleQ+wMeAV1fVT0ftp6p+WVWPo7ky2t7tEOgo+Twb+HFVXTJqLpP8TlU9AdgfeEU75D+KbYEnAO+rqscDPwdGngsA0A57HwB8dMTtH0Az0rIH8FDg3kn+ZJS+qupKmm89+xzNcPWlwC9H6WtzWJAljdtULuc5du353o8BH66qj89En+0w7heB/Ubs4reBA9pzv2cCT0ly+jTyuab998fAJ2hOH4xiHbCu58j/HJoCPR37A9+oqh+NuP3TgP+uqvVVdQfwceD/jJpMVb2/qn69qvYFbqKZ97BFWZAljdtULuc5Vu1ErPcDV1bVidPsa0GS+7f3d6SZvHbVKH1V1ZuqalFVLaF5ny6oqpGO+pLcu52wRju8/AyaodlR8roOWJtkr/app7LxV3qO4mBGHK5u/QD4zSQ7tT/Pp9LMBRhJkge3/+5Gc/74jGnkNiVeOlPSWA26nOcofSX5CPAkYH6SdcDRVfX+Ebr6beAFwHfac78Ab66q80boa1fgg+2M4W2As6tqWsuVZshDgE80tYptgTOq6l+n0d8rgQ+3H6quBl4yakftB4SnA38+ah9VdXGSc4BvABuAbzK9y15+LMmDgDuAV8zQpLWhXPYkSVIHOGQtSVIHWJAlSeoAC7IkSR1gQZYkqQMsyJIkdYAFWZKkDrAgS5LUAf8f5d4QaCWREnMAAAAASUVORK5CYII=\n",
      "text/plain": [
       "<Figure size 576x288 with 2 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "wrong_samples=2\n",
    "\n",
    "for n in range(wrong_samples):\n",
    "    i = wrong_results[n] # np.random.randint(0, len(x_test))\n",
    "    print('element', i)\n",
    "    digit = x_test[i].reshape(28,28)\n",
    "    probs = model.predict_proba(digit.reshape(1,28,28,1),batch_size=1)\n",
    "    plot_digit_and_probability(digit, probs, y_test_indexes[i])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Visualize Convolution Weights"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In order to visualize what a convolution layer has learned, we can reconstruct a new neural network without the last catergorization layers, that is without the linearization and the softmax classification steps."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(10000, 26, 26, 32)"
      ]
     },
     "execution_count": 19,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "layer = K.function([model.layers[0].input], [model.layers[0].output])\n",
    "l = layer([x_test])[0]\n",
    "l.shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAr0AAAKtCAYAAAAq8tZDAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDMuMC4xLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvDW2N/gAAIABJREFUeJzs3XeclOX1//+zgKAU6b0XadJFiiCCKKjYC8QeW0yMfmKNfk3UaDTy0RiNJrHH3hvELlgQREBAEERBei+CFBVQYH9/fPK7POesM+wsM7sz17yef53rcbEzt3vvPXM593vOVVBYWCgAAABAzMqV9QEAAAAAmcaiFwAAANFj0QsAAIDosegFAABA9Fj0AgAAIHosegEAABA9Fr0AAACIXoXSfLKCggKaApeBwsLCgnQ/JueybKT7XHIeywbXZDw4l/HgXMYj0bnkk14AAABEj0UvAAAAoseiFwAAANFj0QsAAIDosegFAABA9Fj0AgAAIHql2rKsLDRt2tSMt2zZEuqNGzeW9uFgNxo3bhzqli1bmrl99tnHjBcuXBjqBQsWZPbAkJLatWuHuk2bNmauUqVKZjx//vxQr1y5MrMHhiIKCmxnn6pVq5pxq1atQt28eXMzN27cODPetGlTmo8OQKpatGhhxps3bw71hg0bSvlosguf9AIAACB6LHoBAAAQPRa9AAAAiF4UmV6dEezcubOZq1u3rhlXq1Yt1N99952ZmzBhQqjJpmXGvvvua8YDBgww48MPPzzhv/V5zz59+oR60aJFZm7ixImh1tlfpEeFCvalo3///mZ86KGHhrpXr15mrlmzZma8ZMmSUD///PNm7rXXXgv1unXrSnawKELneDt06GDmTj31VDPW15m+rkREXn311QwcHYDd0WuZo446yszNnDnTjLdt2xbqQw45xMx98sknof7+++/TeYhZiU96AQAAED0WvQAAAIheQWFhYek9WUFBWp6scuXKZjx06NBQ9+zZ08w1atTIjL/++utQL1682Mzpj/Y//fRTMzdjxowSHWs2KCwsLNj9v0pNKudSt0A64ogjzJy+DS5iW135W+g+pjBnzpyEz6l/9quvvjJzH3zwQfIDzmLpPpepnMdy5X76f+Tu3bubOR9v0P9Wt8sRKXrOjznmGH08Zu7FF18M9f3332/mJk2aVJzDzkplfU22bds21HfffbeZO+igg8z4iiuuCPWDDz5Y0sOLVlmfS6RPNp/LESNGmPEJJ5wQat+y85lnnjHj2bNnJ3xc3VLSv+d++eWXKR9ntkh0LvmkFwAAANFj0QsAAIDosegFAABA9HKyZZlvgXT99deHWmdCRYrmPr/99ttQt2/f3sxt3bo11FWqVDFz9erVM+N333031Dt37izOYect3VLOZ4b23ntvM65YsWKoJ0+ebOY+//xzM9aPpc+rfxy/nbFvhfbxxx+HmrZYie3atSvU06ZNM3N+nMzDDz+ccO6uu+4y44svvjjU+++/v5nzWdRXXnkl1L4dIay+ffuGWrckExG5+uqrzZgcb37o16+fGX/00UdldCQQEWncuLEZn3766Wasv//gv3O0Y8eOYj+P3gbeb0HeqVMnM06WDc4VfNILAACA6LHoBQAAQPRY9AIAACB6OZnpPe+888y4W7duofY9WH0uafXq1aH2W+7p/pQ68yYiUr16dTOuVatWqF9//XUzt2XLlkSHnpf222+/UA8bNszM1ahRw4yfeuqpUPuc0vLly824VatWofb5J50V3rhxo5nTGWMRkd69e4fab9+4bNkyQem59NJLzXj06NGhfu6558zcrbfeasb6b8DnhnV/btjMu+/xee+995b24aCU6PN+zTXXmLlzzz3XjHWG//bbbzdz/I1knu9p779XpNcdL7zwQlqe0383xmd49Vrrs88+M3P67yWb8UkvAAAAoseiFwAAANHLiXhD+fLlzbhHjx5mvGrVqlC//PLLZs5HD3Skwd9a163H/Fac+mN9EXurwbfdeumll0JN1MH+rrZt22bm1qxZY8b6dsrutkDU29v6dlZ6zrdh8S3mfvjhh1B37NjRzOm2MEuXLk16PEi/999/P9TNmzc3c2+++aYZ6+1y/TX5j3/8I9Tr169P5yHmpIMPPjjU/vUU8dKvdR9++KGZ89u867+Ryy67zMzpyNrll1+ezkPEfw0YMMCMdQxPpGirweLyLV/1e66PfHo6ctisWTMzt3LlylCn0jKttPFJLwAAAKLHohcAAADRY9ELAACA6OVEprdy5cpmXKdOHTOeO3duqF988UUzp/O+nm5fJmLzLH4b06OOOsqMu3fvHmrfSuSbb74JtW65lK90JttnhnRmU0Rk7dq1xX7cxYsXh9pvb6zzT3vttZeZq1atmhnXrVs31D7nps+tziyJZHduqazpvNmnn35q5rZv316ix9TbhIuIDBw40Izfe++9UPv2Szo7ft9995Xo+WNSv379UOvfG3KP3lbdX1vJrjXf3tMbM2ZMqC+88EIzd+KJJ4bab1v9v//7v0kfF8XTtGlTM/7LX/6SlsfVuW4RkUGDBoXafwcqGf8dF/8enK34pBcAAADRY9ELAACA6LHoBQAAQPRyIoSht04UKZql1NnOZBleEZHatWsnfFzdv3PRokVmbs6cOWas+xT269fPzG3YsCHUY8eONXM+K5wPfvzxx1DrLKGIyFdffWXGOtOre+SKiHTp0sWMGzVqFGrfc1n3afU9l32uWI/nz59v5saPHy/YvZ49e5qx3sb7iy++MHMtW7Y0Y91/9/PPPzdzfuvpZE499dRQ/+tf/zJzOmPsc2v5uNX0rFmzQu23Hk3F8OHDzXjFihWh9lvAN2jQINT+tZce2CXXvn37UPse5NOmTSvx4+oc/E033WTm9Nj3j/WvxX4beBSP3x7cryVSob+34l/vypX76bPPU045xczp7yeJiEyePDnU+n1dpGgP/mzFJ70AAACIHoteAAAARC8n4g1+21C9FV6q9C1yv1Wpjh74WzRNmjQx41q1aoXa36rTt1KPPPJIM+dbquUDfYt63bp1Zm6fffYx4ypVqoTat6Rq3bq1Gevbpb6tnY5R+G1W27Zta8b6lo2/Paj/9nwsQsdqRHLn9k4m+FvkDRs2DHWnTp3MnN8G8+yzzw61v5b0LfLf//73Zs7fetO3Y9944w0zd+yxx4Zat+gREXnyySfNeNeuXRI7HSvSt8d3p3///mb89NNPm/EhhxyS8Gf1defbCBJvKDnfgqq06ddPpM+UKVPMWK85dse3O9Pxs1GjRiX8tz56NmLECDNu1apVqJ944oliH0824ZNeAAAARI9FLwAAAKLHohcAAADRy4lMr+dbdyRrueOzubqtkd82T7fy0G08RIpuXauf02c9t2zZEmqfQ/UZGd0azWd4YqH/GydOnGjmpk6dmvDnfFbNtztr3LhxqH3rM53N9S1zFi5caMY1a9YMtc4Ji4h07do11P5vQOePRWwbKP83Gnve98svvzRj3SrJt67yY823l9NZ7UsvvdTM+UyZbjfns+Nt2rQJtd8m1bdbe+WVV0I9btw4MxdL3ldvDb158+Zi/9yQIUPM2LejS3Zu/RbfSA/9Hnf66aebOb2VsIjI888/XyrHhJLRr3eVKlUyc/77JsncfvvtZjx69OhQV69e3czpTO/q1avNnH9f1S1f/fdx/Dbxmn9d19+fmjlzZsKfywQ+6QUAAED0WPQCAAAgegWFhYWl92QFBWl5Mt8WR0cR3n//fTOnb2uKiFx33XWh1ruU+HG9evWKfTz+lqeOPnz22Wdmzrdb+/rrr0PtYxL+1r/foaW4CgsLC3b/r1JT0nPp24f5tl96Nzt/2/nAAw80Yx1b6Natm5nT589HFpLxu8wsWbIk1H5XPh+rqVOnTqj9znvTp08342SxjmTSfS7TdU1mGx9rOuecc0J90UUXmTn/GqHP+WOPPWbmnnvuOTP2fxPFVdbXpL51un379mI/h2/BOHfuXDP20aF8UNbnUreCu/LKK82cfz17/PHHQ33ooYeaOf36JSLyt7/9LdR6p72YlfW51I444ggz9jEAvfusbwH44IMPmvFVV10Vah9n0m3KUmmL5iN7s2fPDrWPJu67775mrI/Bt0H0a4JUduXUEp1LPukFAABA9Fj0AgAAIHosegEAABC9nMz0psLn9Y466qhQJ9sC18/5zIxuLeJzUzqDUrVqVTPXvXt3M9bZT9/+R7dQE7HtZ3zuJZlsyimlokWLFma8//77m7He/tJniJK1YfLnUmfC/eM0atQo1D5j7Lcz1m2ydFZbRGTHjh1mrM+fb/eVDJne9PM51XvuuSfUPtv/0ksvmfGdd94Zap/fTyZXr0mflfbtAPNRWZ9L3XrTv995+nzpnLuIyGWXXWbGuj2Vz2/q9lk6Ay9iW3aKiBQU/PTr8e+H+jX8hhtuMHNlkSMu63NZUu3atTPjE0880YwffvjhUPt2jvp9zJ9Lvw7S7c58izK9DXwqdMtQkaLvz/p5/N9WMmR6AQAAkLdY9AIAACB6LHoBAAAQvegzvZ7uE9usWTMzt/fee4fa9/DVuSQRmztJ1qfX5158RqZXr16h7tixo5lbuXKlGU+aNCnUTz75pJlLlqfJ1ZyS1759ezPWvf/8ttE6Z+175voMth77ral130J/nn3vwQ4dOoTab1Hs8546c+yzaz7LrZHpLV233XabGfttXl9++eVQ33HHHWYuWe4+V69Jf334rHo+ytVz6Xuy6n6tIiJdunQJdb9+/czcgAEDQu23y/WvxboPtN9KV/dQ9z/ntwu/+eabJdNy9Vx6/jrVv+eS9r1NRSpbFO9Oq1atQp1KH3AyvQAAAMhbLHoBAAAQvbyLN+j2RL79xYQJE9L+fP5jfr+Vrr4NPmjQIDPnbynp2IRuQSIi8vzzzyc8hly9ZeNbhA0cONCM9W0zvx3q+PHjQ70nt1bKly8fan/ufLxB3+bzrZ18W5a333471P6W3/z58xMeTy7GG3w7Hf93Xr9+/VD7bcQ//PDDzB1YCdx9991m3Ldv31D7a/K+++5L+Di5dE3qv91jjz3WzOn/fhGRG2+8MdR6m9SY5dK5zAQfBfTvefp12r8W67jheeedZ+b835p+j7vgggtKdrC7Eeu51L/bjz76yMyl0jKzuA466CAz/v777814xowZaX9Oj3gDAAAA8haLXgAAAESPRS8AAACiF32m1297eOaZZ4batyx77bXXQu1zL5mi28Z06tTJzPltIvv37x/qKVOmmLmrr7461L4lSS7llHTbuEMOOcTM+WyobhvnW4Lp7FgmstoiIo0bNzbjww8/PNT6v0NEpGLFimast4J87733Ev5b374sFzO9vtWcz+r95je/CfXq1avN3E033RTqN998MwNHt2dGjx4dav9aes0114Ta5+Zy6ZrU7al8btl/L0JnNPNFLp3LbOazwcOHDzfj0047LdT6vVpE5NZbb03LMcRyLvVrqoht9TV9+vSEP+e3KE4X3XZMxK57Zs+ebeb8ltclRaYXAAAAeYtFLwAAAKJXYff/JLc1adLEjA8++OBQd+/e3czpnbFKK96wYcOGUPv2TCeccIIZ69ZBfvcc3fapNHZcyZRktzZ0+zDP3z7+5ptv0nZMifid1D799NNQX3/99WbO/3fpf5tsN70Y+HPjxzNnzgz1LbfcYuZ0pOfdd981cz/88EO6DrHEXnnllVDr1xYRkYYNG4Y6E22BSsumTZtC/cQTT5g5H3fINt26dQu1b5V35513lvbhlLn9998/1IceeqiZ8+95yW6DZ4K/te5b/unjK81YZi7wf9t6x08R2yLT78Sn1yCZ4ndS02O9K62I3QlQROSTTz4JtX6vKCk+6QUAAED0WPQCAAAgeix6AQAAEL3oM73r168341mzZoX6gAMOMHN6i2LfKsrnbTORGfVZlnLl7P+T6C2LW7RoYeZ0dnnatGlpP7ayoLOEInabXxGbkx06dKiZ03lbnwX2+cpk2/5qukWaSNH2TCNGjAi130LZb825ZMmSUPtzuXjx4mIdTyx03sxn1U899dRQn3XWWWbujTfeMOOVK1eW6Pl17lNEpEGDBqF+6623kv6sbp3kc4Y//vhjiY4n2+gcnX9d1Fujl4VGjRqZsW4bKGLbHnbs2NHM5WOmV7dybN26tZnz7z/6ewllsaW03569NLauzVX++w3+d6W/85PsOyT+PbZy5cpmnIl1z4UXXmjGgwcPNuNLLrkk1GR6AQAAgGJg0QsAAIDosegFAABA9KLP9Pref0899VSoq1WrZuZ0PqxOnTpmrk+fPmass8LJ+oXubku9evXqhbpz585mTvdUFLFb206ePNnMxZLj1SZNmmTG/nzp7W2XLl1q5nQ2yW837fO/OsNZoULiS2Lfffc14++++86MdQZ50aJFZs7nivX5yrcMbzLvvPOOGetc4W233Wbmzj//fDOeN29eqL///vuEz+Gztr5v5caNG0Pt/67mzJljxvp5fN9mnduOhf+bb9OmjRn7309J6dc+32td032cRYr2Stavk8OGDUvLseUy3R/1kUceMXPXXXddwvFFF12U2QPDHvE9lk8++WQz1t9jSZbp3b59e9KxVrFixYRz/r16yJAhZqz3SPDvzz6PrN+f04FPegEAABA9Fr0AAACIXkFpbudXUFBQ5nsH6tZR/lZYhw4dQu1vZfut+nR7JN9aRfO3tX17Kv2x/3777ZfwcUTs7dIXX3zRzOntQH3corCwsCDpA5dAWZxLf8vkiCOOCPWCBQvMnI6f6O1gRURq165txvr2SvXq1RM+v/85Hz/Rfwe+DZq/bf/mm28mfJ5k0n0us+GaLK7f/va3Znz55ZebcePGjUOdbMvqsWPHmvHf//53M9bnateuXSkfZ3HEck16OoK1u2hXMvqWZo0aNRL+Ox9Hmj17domfs6Ry9Vz6390xxxxjxjfffHOofbvG0047LdRTp07NwNGlxscPmzZtGuopU6aYOf3e7aNOuXouPf9edfzxx4favzeNGzeu2I+rf88+lqA1b97cjH0LTx2T8i1E165da8aPPvposY9PS3Qu+aQXAAAA0WPRCwAAgOix6AUAAED08i7Tq/ms58CBA0PtW5Yly5W1bdvWjHWuzf9+/ePobTx9bthv+adzq6+++qqZ8+2StFhySl7NmjVDrc+diMhXX30Vap8R8lun6vygPnciNpvkz4/Pcs+dOzfUW7ZsSXboJZbPmd6YxHpN5qNYzqXPwestwPVWsCIivXr1CrX/Dol/L9LvVT5Pqlvc+dZUup2niEjfvn1D7dvY+ffKxx57LNT++xS6JaEXy7n0dDvW3r17mzndGtRnnHfs2GHG+nfXqVMnM7d69epQ622PRYqeS72ttc/0pguZXgAAAOQtFr0AAACIHoteAAAARC+vM73J+B6sPrers7m+J57uaej7fPrMqO5lqXMuIkW39NTb8/msTTKx5pSS0Vlcnf0VKfq70/0q/ZbA+lz67Jof60yczw2nC5neOOTjNRmrfDiXfrvpo446KtT++xTt2rUz42T9s/X3JKpWrWrm/Hcxli9fHmq/Ve2YMWPMWPfmTZbh9fLhXHp67wD/O0+2nXu5cvYz00z1NC8pMr0AAADIWyx6AQAAED3iDWlQpUoVM9Zb2W7fvt3M+ds7/nZCJuTjLZtU6AiD3+pY337z51lvpVhaiDfEgWsyHvl+Ln1kz2/l3rVr11D7lp36FnlBgf01Llq0yIwXLlwYat8+MpW4XzL5fi5jQrwBAAAAeYtFLwAAAKLHohcAAADRI9ObB8gpxYNMbxy4JuPBuYwH5zIeZHoBAACQt1j0AgAAIHosegEAABA9Fr0AAACIHoteAAAARI9FLwAAAKJXqi3LAAAAgLLAJ70AAACIHoteAAAARI9FLwAAAKLHohcAAADRY9ELAACA6LHoBQAAQPRY9AIAACB6LHoBAAAQPRa9AAAAiB6LXgAAAESPRS8AAACix6IXAAAA0WPRCwAAgOix6AUAAED0WPQCAAAgeix6AQAAED0WvQAAAIhehdJ8soKCgsLSfD78n8LCwoJ0Pybnsmyk+1xyHstGJq7JZs2acS7LwNKlS3l9jQTvlfFIdC75pBcAAADRY9ELAACA6LHoBQAAQPRY9AIAACB6LHoBAAAQvVLt3gDsifLly5vxzp07y+hIgHhs27bNjNevX2/Gu3btSvizffr0MeOmTZuGet68eWZuw4YNJT1EpEmVKlVC/d1335XhkWBPVa1aNdQtWrQwcytXrjRjrr2f8EkvAAAAoseiFwAAANFj0QsAAIDoFRQWlt5mIexMUjZyaZeZxo0bh9pneFesWGHG+ZjpZUe2OJT1jmw6x7tu3bp0H4qIiAwfPtyM99prr1BPnjzZzG3fvj0jx1AasmlHttq1a5txsqznqlWrSvIUUcvm98oBAwaYsT7X7du3N3MHHXSQGc+ZMyfUd999t5nz76uxYEc2AAAA5C0WvQAAAIgeLctQpnzLoyZNmoS6Zs2aZm7Hjh1mPG3atFB/9tlnGTg6IE6ZijRozz//fMK53/zmN2Y8ffp0M/Ytl5BY/fr1Q3300UebOd+qatmyZaVyTNhzzZo1M+PNmzeb8Q8//BBqfz3XqFHDjE855ZRQn3rqqWbupptuCvVDDz1UsoPNIXzSCwAAgOix6AUAAED0WPQCAAAgetG1LNPbYIqIDB482IyffPLJUPuMaKyyqQ2Lb6FzxRVXmHGvXr1C7c/Ppk2bzHj+/Pmh9lm1l156KdQLFy4syaFmpbJsWVa3bt1QV6tWzczpfJmISL169ULt2+focywi8vXXX4f66aefNnNTp04t7uHllLJuWZZt2c5hw4aZsW5X6PP62daqsKxblg0ZMiTUt9xyi5l79dVXzVjnN1FUNr1XppN+vX7kkUfM3EknnRTqxx57zMxdeeWVZqxfq7MdLcsAAACQt1j0AgAAIHosegEAABC9KDK9xxxzTKivueYaM+fzg7NmzQr19ddfb+Zee+21DBxd2cumnNLIkSPN+PzzzzdjvTXm22+/beZ0P0oRkX322SfUVapUMXNr1qwJ9bPPPmvm3nnnHTPetWvX7g47a5RlprdixYqhrlWrlpmrXLmyGffo0SPUPtPbsGFDM65Tp06oV69ebeb0Nfncc88V91CzXq5mevU24SIiHTt2NOP169eH2vfeTUWbNm1C3a1bNzP3ySefhDobrt2yzvTecccdof7tb39r5nTfc5HUMpmdOnUKdcuWLc2czwrHIpveK0vLpZdeGmqf+Z47d64ZX3DBBaGeMWNGZg9sD5HpBQAAQN5i0QsAAIDoRbENsb7V+sYbb5i5f/3rX2bcs2fPUJ9zzjlmTt8+f/jhh9N5iPivBg0amPE333xjxrpl1fjx45M+lm5Pd/DBB5s5fUvUn+fmzZub8csvvxzq0tieNVftvffeofbt4/xtZr2N7FtvvWXm5s2bZ8b6Fuxxxx1n5nQrq0aNGpm5O++8sziHjRQVFNi7gjoCV6GCfcvo3r27GY8YMSLUc+bMMXNnnnlmsY9BtyP0W+nqFl0TJ05MeKz5onPnzqH2r6d70mJKv6/67Y11JMm3wEJuueuuu0Kto0MiIvfdd58Z/+c//wm13874o48+ysDRpR+f9AIAACB6LHoBAAAQPRa9AAAAiF5Otizz7ZG+//77Yv+s3h71l7/8pZnr3bt3qH1OKZfbmWVTGxa9PbCIPR8iNve3ePHiYj+ub2uk80Y6xy0iUqNGDTPWOeLRo0ebOZ0tzAZl2bJMtx6rVKmSmfvyyy/N2Ocwte3bt5uxzon6Fli6HaH/W5kyZYoZP/XUUwmfM9uUdcuyzZs3h9rns1Pht31v27ZtqP/0pz+ZOd1G8OSTTy7xc9auXTvU7du3N3NLly4t8eOWVFm3LNO/V/9+7r9DkeIxhPqiiy4yc127dg31okWLzNytt95a4ucsa9n0Xul16dLFjPXr5uzZs82c3xa+pPz19eSTT4bar8OOOuqoUKfy3p0ptCwDAABA3mLRCwAAgOix6AUAAED0sjbTu++++4Za589KS82aNc3Y9z/MJWWdU6pbt26off9jn68dMGBAsY9B/+zGjRvNXJ8+fUI9fPhwM3fEEUeYsf778n2e/baMZa0sM70HHHBAqP1Wwh9++KEZZ+Ka9Zm21q1bm7HeAtcfT7Yp60xv//79Q/3MM8+k7Rj06/bgwYPN3KGHHhrqLVu2mLlrr722RM+n+8WK2K3JS0tZZ3qXLFkSav96Wr169WI/p8/Mr127NuG/Pfzww0N9xhlnmLn333/fjB999NFiH0NZK+v3ymQuu+wyM161alWo/XdldLZexGa9ff/sVOj31ZEjR5o5fd5vvPHGEj9HupDpBQAAQN5i0QsAAIDoZe02xOXLly/T58/lOEO20S2RZsyYYeb0LfPd8bdskrXF0q2LZs2aZeZ69Ohhxrr9zrJly8zc0KFDQ/32228X+1hjtHXr1lD7rWrT1SInmc8++yzpuCxubeeqbdu2ZeRxdazllVdeMXN+rO1JG0rNbzGut8P+8ccfS/SY2W7s2LGhPvfcc4v9c23atDHj/fff34x9+0ZtzJgxofZRJh8NRHr411jdLlBHCEWKvlf+4he/CLW/9j/++ONQ+22HvUmTJv3sz4nYlpa+1ZlvaVmW+KQXAAAA0WPRCwAAgOix6AUAAED0sjbT26xZs1DrPIpI0UzKE088EeodO3Zk9sCQMn1O/JaVfjvbZM455xwz1i2qdC0isnr16lD7v5dp06aZcblyP/2/X5UqVcyczkpVrFjRzO29995mXBat9UqTPo+dOnUyc759mM5R+3ZyPjet51PJBvscqN6e1p+LPdlqN0Y6D6i37BZJbwuz4ipphvfrr782Y7819a9+9atQ+2y/bvUlIrJz584SHUNZe+edd0KdSsvHK664wox9FjdZplebPHlysZ8TJfevf/3LjH/5y1+GWmfXRYrm5/V7lW47JmLPu38teO2118xYtxr0rc8uuOCCUI8aNcrM3XDDDWb83HPPSVnhk14AAABEj0UvAAAAope18YZvv/021P7Wqb8FnWznGN96+pIBAAAgAElEQVTKQ7fO0O03kDm7du0K9fLly82cjwy0a9cu1A0aNDBzLVu2NOOBAweG2rcjatSoUahT2XVQt+USsTsP+ZiEbhXk7bXXXmYcQ7ukFStWhNrfRvbXpD7nPXv2NHNnnnmmGev2hNOnTzdzCxcu/Nnn3x1/q1ZHWGhHaF8zf//735s5HQ0SsTstnX/++WbuoYceysDRlZyPx/zjH/8ItW/RpdsRioh89NFHoc6lOMwHH3wQ6qeeeqrYP+d3VaTlX3bz72OPPPJIsX9WR5Z8fEnvouj51pT6344bN87M3XrrraE+/vjjzdz9999vxr/+9a9D/ac//cnM+cdNNz7pBQAAQPRY9AIAACB6LHoBAAAQvYJU8o57/GQFBcV+Mp3z22+//RLOidh8mm9v5rdl1NtU+nZZOofoW5/pjLE/hqpVq5o5nUm5+eabzVxZtHcpLCws2P2/Sk0q57KkfFss3aJFRKRz586h9udH59N8WyPfakznPX2mVz9uixYtzJxvi6XbxPgti9OVI033uSzpefS/Q5/N1hlmndMWETnuuOPMWG9f2aVLFzNXo0aNhM/pM9b6mvX5RP371y2eREQeeOABM850pkwkM9dks2bNin0u9e/nwAMPNHO+/ZHe7vvOO+80c/6c6HzwhAkTins4WeHII48MdbLfgbd06dKseX2tXr26GSfLJl966aVmrFv+iYiMHDky1P47CZnYdnx313dpyNX3ymynWweKiFxyySWh9n+zfs3kX5+LK9G55JNeAAAARI9FLwAAAKLHohcAAADRy9pMb0n5/py+x6/uGdqrVy8zd/DBB4fa91n1/er0vN++UuedfG/RF154wYzvueceybRYcko+U6szeLovr4hIq1atQu37EFaoYNtTb9++PdQ61y1it7r1vWh9jlg/j+9j63sj+vniypZMb4rPYcb+96/zpU2aNDFzOs/vs1/+GtXnzmcQ9etAjx49zJzPI7/33nuhvuuuu8xcuvotl3Wmt6T0dSVS9DX0xBNPDHXv3r3NnL5efNbU5+71dqd16tQxc/o7FBdeeKGZe/PNNxMeeyoOO+wwM547d27Cf5tNmd5U+NfF7t27m7HOZPv3OJ2/3ZPsrX7dPuCAA8yc/hsQsf2IMyWW98psp/dPuOOOO8zckCFDzPiyyy4LdSpbpJPpBQAAQN5i0QsAAIDoRRdvSBe/fbG+zS1iW1v5Nlf6Ns3ZZ59t5nwbrsceeyzUesvMdIrllo2/HVevXr1Qz58/PyPPqW+9H3300WbOb7Woj69+/fpmTm/lKmLbm6WyHXYuxhvSxccZfOtCzbcc1Nevb4umb8mLiDRu3DjUflvk2267zYxL+vqZS/GGZcuWFfvf6qiI325atwHzLfx8u6qNGzeG2t8+P/nkk0Pt4w2LFy82Yx1ZS4VuYyhiYxw+spar8Qb/XuS3iNfnIFN0ZE1v+S5S9G9Cj2+//faMHE+uvlf6tpBdu3Y1Yx25HDVqVKYPJyU+0qa3MxYR6dChQ6jPOOMMM/fll18mfFziDQAAAMhbLHoBAAAQPRa9AAAAiB6Z3gzz2eDhw4ebsW6h9vDDD5u5dG3jmas5Ja9fv35mvGDBglD7lkc+05kJPkels4ZDhw41c3o7XRHbeuX+++83c8m2PM2VTK/O3/rXmNI4N8n4LLBvd6e3Mvdbs86cOdOMU2mho8Wa6U1GZ+L9Nr/r1q0r0WO2adPGjC+//HIzbtiwYaj1a22qdPs1f+y5lOnVv6+LLrrIzL311ltm7LfrLm3Dhg0zY70Nvc+E+21uSypX3yt9Bt1/T+Haa68N9YwZM8zcueeem7kDKwH/nQv9vafx48ebuf/5n/9J+DhkegEAAJC3WPQCAAAgelHEG3SLId/+YuHChWZc0tto6eJvreqWLf5Y03VucvWWTZ8+fcz4ggsuMGPdwmbs2LFm7pFHHsncgSWgIwxXX321mbvmmmvMWLcs+8tf/mLmpk6dmvA5ciXeoP+u9c55IiK1atUy44kTJ4bat3YrC3pXR90uR0Rk/fr1ZqxjNX4umVyKN+j2R3PmzDFz/jWruPzuer4dYUlVqlTJjPXOavrvTETks88+K9FzNG3a1IxzKd7QrVu3UD/99NNmbtasWWY8YsSITBxCid1yyy2h/n//7/+ZuYMOOsiMU2kDqeXqe+Xu6PdOHwm47rrrQp1t7cxEbFvCvn37mjkd2/CxI+INAAAAyFssegEAABA9Fr0AAACIXoWyPoB00Dku3yrK5zx0+4sff/wxswf2M3bu3GnGuu0WLJ/P+/77781YZ0X97/WTTz4J9VdffWXmtm/fnq5DNHTG2B+rbxPTunXrUPvcaLJMb67Q1928efPMnN8aVucMsyHTq9sh+Ryop/O/sdJbcT///PNmzrdgLG7GV//9ixRt/6fztps3b074OLtrp6a3J9c583y1aNGiUPsWZQMHDjTjwYMHh/rdd9/N6HGJ2K3CRYpeWy+88EKodSs6Ebv9NYp68MEHQ61z7iIiV1xxRaj134dI0RaNJeXbBerX2N2tgSZPnhxqf56rVauW8rHwSS8AAACix6IXAAAA0WPRCwAAgOhF0adX69SpkxnrrQtFbEbliSeeyPThpJXOs/jtUXWmde3atWYult6Dxx13nBkfe+yxofbb/N53332hHjNmTNLH1dlhnV/0KlSwEXjfb1b3Gu3fv3/CYxURKSj46ZScc845Zm7x4sUJjyFX+vRqPnelt2sWsZneadOmmbnHH388cwdWAj6L2qBBg1D7rPjGjRtDrbOlIrnVp1fzPZf99q97stVvrsjlPr2azzjfcccdZly3bt1QDxgwINOHU4TPb+rvTDRr1szMzZ49Oy3PGct7ZTL+/ebWW28N9ZYtW8zcxx9/bMa6l7P/3orms/3+XO7atSvUr776atLjrVevXqj936HO+/p8P316AQAAkLdY9AIAACB60cUb/Da/Rx99tBmfd955odZtM0REHn300VBnQ+uk3/3ud2ast+X12+4++eSTofa3WWO5ZeNb2Jx11lmh9m1YdBsUf+trwoQJZqzbtCRrZ+Zva/rn1BEG31LHt8d74403Qu1v4a9YsSLhMeRivMHTt01FRM4///xQ+1iIvg321FNPmbnRo0eb8datW9NyfPq2r45eiIj07t074XP6bVx1rObbb781c7kab/Dxn8svv9yMdXSlTp06mT6cUrPXXnuFWkdaRHI33uANGjTIjC+55JJQ69crEZGHHnoo48ej4wwiItu2bcv4c8byXllSf/zjH814yJAhZqzfH5O93o4fP96Mn3nmGTNevnx5iY7Pv6+uWrUq4b8l3gAAAIC8xaIXAAAA0WPRCwAAgOhFl+n1fMb31FNPDbXOEoqINGrUKNRVqlQxczpbKCLy3nvvhdpngz///POffcyfG+sWHO3btzdzuuWRiMhdd90V6nvuucfMbdiwQRKJNaek8z2/+MUvzNywYcNCvWPHDjPnc7ubNm0KdZs2bczc0qVLQ+2zfMkeZ/369QnnRGxmfNy4cWYuWXYthkyvp9vv+Vyh3gpVZ9pFRJo0aWLG+jr024+vW7cu1H67U7/dtc4c+7+dKVOmmPE777wT6g8//NDM+b8BLVczvZ7/3Y0cOTLUJ510kpl79tlnQ33BBReYOZ95Lmu+PWH9+vVDXa6c/awolkyvd8wxx4TavzfpzKZ+jRQpeu3lkljfK2Phrz2/LtPI9AIAACBvsegFAABA9Fj0AgAAIHrRZ3qT8flNnWHy/UJbtWplxnrbW5//1dvR+nym7z2oe8T63sCjRo0yY70loN8uMJl8zCnp7ah9P92uXbuasc6U+i1z9daGfutnn6PevHlzqPV5FSnat1Bvn5iKGDO9ms+I6i0o/dajPmOtz6vvJ6uz9D4r7/Okestgv32wP69r1qwJdSp9gmPJ9CbjX0NHjBgR6h49epg5vS23iP29PvLII2ZOb3+qt1//ucfRPZd/+OEHM+e3WJ03b17Cx/FZQi3WTK/me6S3a9cu1Pp1T6Ro79SqVauG2m8Vm23y8b0yVmR6AQAAkLdY9AIAACB6eR1vSMbfZtVxBhG7PWmtWrXMnL7lprevFLHb44rY26f+dnmydhyp4JZNcvrWnb6dLmLjDn4rYX+bfMmSJRk4Oiv2eEMq/LWlb6P6yJG/Xa351nM67uDjSdl8TWZbvCGZ6tWrm7F/fT300END7c+Pjhr4969PP/3UjHUcRccifk6yv5Fk8iHesCd0tCjb25nxXhkP4g0AAADIWyx6AQAAED0WvQAAAIgemd48QE4pHmR645Dvmd6YkOmNB++V8SDTCwAAgLzFohcAAADRY9ELAACA6LHoBQAAQPRY9AIAACB6LHoBAAAQvVJtWQYAAACUBT7pBQAAQPRY9AIAACB6LHoBAAAQPRa9AAAAiB6LXgAAAESPRS8AAACix6IXAAAA0WPRCwAAgOix6AUAAED0WPQCAAAgeix6AQAAED0WvQAAAIgei14AAABEj0UvAAAAoseiFwAAANFj0QsAAIDosegFAABA9Fj0AgAAIHoVSvPJCgoKCkvz+fB/CgsLC9L9mJzLspHuc8l5LBtck/HgXMaDcxmPROeST3oBAAAQPRa9AAAAiB6LXgAAAESPRS8AAACix6IXAAAA0SvV7g3pUq6cXavXrl071OvWrSvtwwGQRM2aNc14r732MuOvv/461Lt27SqVY0LxVatWLdRVq1Y1c6tWrSrtwwEgIhUq/LR869Spk5nzr7lffvllqPP9muWTXgAAAESPRS8AAACix6IXAAAA0cuJTG+LFi3MuEGDBmY8Z86cUjwalJZKlSqZ8fbt28voSLA7OvcpItK/f/9Qd+vWzczVqVPHjL/44otQT5gwwczpLBoyw2es9bkTEWnSpEmo9957bzO3Y8cOM9bnctKkSek6RJSxffbZx4x/+OGHUO/cubO0DycvtWrVyozbtWsX6l69epk5/T0nEZFvvvkm1H69NHbs2FCvX79+j48z2/FJLwAAAKLHohcAAADRy9p4g/4of+jQoWZu8eLFZrx58+bSOCSkgb892rNnTzPWt2waNWpk5ho3bmzGK1euDPXDDz9s5lasWLFHx4nk9C1vEZFzzjnHjI844ohQ9+nTx8zpFmUiIsuWLQv1oYceaubefffdUD/11FNmbtu2bSkcMTQdaRgyZIiZ22+//cxYtz/S16eISPv27c14zZo1oX7rrbfMnD5/a9euTfGIkQ716tUL9fHHH2/mRo8ebcb6XFapUsXM1a1bN9Rbtmwxc/pWOtKnVq1aZqzbBy5fvtzM1a9f34z1Ne3fR6tXrx7qadOmmTk/jgGf9AIAACB6LHoBAAAQPRa9AAAAiF7WZnq7du0a6osvvtjMPfroo2b85ptvlsYhoYR0ZujII480c507dzbjpk2bhtrnBXWOTMRmOn0+7d577w31gw8+mOIR4+dUrFgx1L4N2SmnnGLGus3ga6+9ZuZ+/PFHM9bbinfs2NHMlS9fPtQ+a+pfB2hvVnx6C1Pfbs7ntZO1LFuyZIkZ9+jRI9S+9Vnfvn1Dfffdd5s536oO6XHIIYeY8ciRI0Ptr0Of7b7//vtDPWbMmITP4V+nff7X501RMr49oP7ek24hJyLyn//8x4x1m0j/NzFixIhQ+1ZnPkf84YcfhjpXW4jySS8AAACix6IXAAAA0WPRCwAAgOhlbaZX9+v0Wb7bb789Lc+x7777mjH9fjPj+++/D7XPAfk8oR5PnDjRzPntUps1axZqn0O86qqrQt2hQwcz97e//c2MyZwVj86N+fPme+8uWrQo1I8//riZmz17thkXFBSE+uCDDzZzl156aah1plhE5JJLLjFjnR0m559c5cqVQ6378IoU7fG5YMGCUOvzKlK0V/KUKVNC7XP2J598cqh9v19/TT722GMJjx3F5zPyume2fl0WEXnggQfMWOe+9fcyREQ2bdoUap+l19/LELG92P0WuP4YkNgJJ5xgxvo7Lr6H+bx588x43bp1oZ4/f76ZO+2000J90kknmTn9HisiUqlSpVCPGzfOzPl+zdmKT3oBAAAQPRa9AAAAiF7Wxhvatm0b6nS2xtCt0I477jgz9+KLL4ba34ZByem2J35rYb9d8CeffBJqHzeZPHmyGetbbr4Ny9lnnx1qf54bNmxoxnfeeWeo9e1ZJOZblrVu3dqM//znP4f6pZdeKvbjbt261Yz17c+LLrrIzNWoUcOMd+3aFWp9G05EZNSoUcU+hnygI2O+xZS/BnSEQUcdfu7ffvfdd6EeP368mbvyyitDffjhh5u5W2+91Yz1drnpirPlo1tuucWMdSxh6NChZs6/vuoohH7fFLG3yDdu3Gjm9LbiIiI7d+4M9YEHHpjwOdlW3PKvb36s2wfq8ypif+fepEmTzFi3rlu1apWZO/bYY81Yxwh9CzUdR8zmqCif9AIAACB6LHoBAAAQPRa9AAAAiF7WZnq7d++ekcfVGVK/dd+5554bap8BZJvMkmvTpk2oq1ataub22WcfM9atVnSbFRGR1atXJxwvXbrUzOlcmc+C6q1SRUQuvPDCUPttGGl99fP0NsMiRTNkU6dOLdHj+m1tdTb3j3/8o5m77rrrzFjnjHW+V0Rkw4YNodZbaeYr3WLOZ3p9eyr9/QZ/XnWG1/O/Z/2cfjtjv2Xx1VdfHWqfV/SttfATn/vUf/ciIv/85z9D7TO8ns5+Hn300WZOtxZ87rnnkj7OypUrQ+23tdXb0r/yyitJHyff+JaqvXv3NmO9NfTatWvNnD/vyUybNi3U/jsVvp3hiSeeGGrfMlK/B2fzd6L4pBcAAADRY9ELAACA6LHoBQAAQPSyNtOre8f5/FcqfPZw8eLFoR45cqSZGzhwYKiHDBli5nye1G/lh8R0Js/3AfTbZOqthhcuXFjs5/BZJJ3F9XPnnHOOGev8uM9CrV+/PtT08P2JP4++37K+zjy97bCISPPmzRP+nM54621RRUReffVVM9bbaeqtT0VsJt/ntvU5zhc68+7PR+PGjc34vffeC7V+XRYp2p9Z/134LWZ1zu+ee+4xc36L6b59+4bab42qc8XTp08X/MTnQH2PY99nubj0Ft8i9m9EX78iyXOhPtNbrtxPn7sle6/OR77Xrf8uyv333x9qn+n113CXLl1C7V+7Z8yYEWqfxf3iiy/MWL+O+m2RBw8enPBxsgmf9AIAACB6LHoBAAAQvayNN4wbNy7Up556arF/zt8+8W0+kt0y+eCDD0L92WefmTm/5R6Kb8uWLaH2W0362EgyfitMfavu22+/TfhzemtjkaK3yXXcYXe3evF/Zs6caca+5ZSOqXi+hVynTp1C7W/pzZo1K9T+b6dVq1ZmrK/Z9u3bm7k6deqEurCwMOGx5Qu9tfuaNWvMXP369c1Yv6b686y3MxaxreL8teS3q9W+/PJLM9a3zPXfh4iNO8yePdvM+TaU+cbfuvYtAP0WwSWl40x6a1oRkQMOOMCM9fXmo4r6dcTHYfKdv150yz+RojEtrUGDBmasW87NnTvXzB122GGh/vzzz82cP1/6tUBHU0REzjvvvFDr120Ru7Yqa3zSCwAAgOix6AUAAED0WPQCAAAgejmR6T3ooIOK/XPXXnutGbdt29aMd7dl4v8vlW38kJzO9Pqcn9+GuF+/fqGuW7eumfM5TZ0v8vlOnd0uX768mfNjnW9s2bKlmdM5qnr16pk53yYmnyxatMiMfcsynTfz29oec8wxZqwzZX6bW701uM/8+fZZ+rz6NmS9evUKtd86159HvX1uPuQM/Vbb/veq89l+m1uf/x06dGio/bWt86X+OTzd5rBdu3Zm7uSTTw61327at0LLh2tUZyt9nlZn2UV2v/WwNmjQoFCPHz/ezOmt5Rs2bGjmfO5b/834999UWn/qLeyTfYcjVvfdd58Zjx07NuG/9W3j9Htws2bNzJzePthngf3vWbci8/9W/01ccMEFZk63IBQRmThxYqj1Wq808EkvAAAAoseiFwAAANHL2niDvp2iPzbfHX+7LVnrJJQOHRXxrUv0Di8iIgMGDAi1jzf43YZOP/30UK9bt87M6Z2ikrVK8vzt0kMOOSTUffr0MXOPPfaYGb/yyivFfp5c58+jbyfmW1BpPgqhb6/7llP6cXd3S1zz173ezcjfLvf/9qOPPgq134kq2S3FXOXbOL7//vtmrG9f+10S/S1O3eLI3/bWEaSdO3cW+/j8edcRpDPOOMPM+dv5Oqry/PPPm7lUjiGb6ciYf6/01+U333yT8HHOPvtsM9axBB8T1LEf3zrLvxbrsW9xp/+eli9fbuZ0nEHEXqdVqlQxcz6y5lsfxsC/5iZ7X/M7oulYi/8b0efEXz/J6FiEiI2N+Rhj586dzVhHn/zrsf/v1LtypgOf9AIAACB6LHoBAAAQPRa9AAAAiF5BaW7JWVBQUKIn8+2ofK5M+8Mf/mDGFStWNOMbbrihJIeQ0woLCwt2/69SU9JzuTuVK1cOtc9tdezY0YwvvvjiUOvMpojNGurH/DnJsn26nZnPFuptkEVsNvIf//iHmfM5yZJK97lM13n058q3BdN8Nlrz2S99zn3LuEqVKiV8HH+udAssn0Pt0KGDGeutWv3fxltvvWXGDz/8cKhTyRHm0jWZjG9Hd/zxx4e6adOmZk7n+Hw+328Nrlsu+Zy9/pvo3r27mfPnVp/Le++918w98MADZuzb3BVXNp1L/1qXrOWeby2m282J2DZX/jzr7dkbNWpk5nbs2GHG+mf9edatq9555x0z59uk6fZZPpfqx9OmTQu1zxgnk03nMp1q1qwZap/r1jl8/X7n50Ts+fN5cf167NvM+ry4zoH7bPLLL79sxvpvxOe+k0l0LvmkFwAAANFj0QsAAIDosegFAABA9HIi05uKFi1amLHP6/ktNzPN9zD0eRqfV8uEWHNKuhegz5XpDJPPhPu+rDq36TNNeutj3bNXRKRTp05mrDNNuterSNEs6KOPPiolka2Z3lzis6bDhw83Y71Nsu8Vqvu+ioh88cUXof73v/9t5pJlxfPhmvS5Xd13W2dC/c95/jVSP65/fdeZYhGRLl26JHzcl156yYx/+9vfhjpZJt2L5Vz618nmzZuHOlk+e9WqVWbOZ+31+fJ9uHUPVr91rt/GWr82z50718z5n9X5ZP9vfV9qLZZz6fn9CzT9Gue/m+G/E6Uzvf57E7Nnzw61v5595lpvPa+3mhcp+h6svx/zwgsvmLlkmXUyvQAAAMhbLHoBAAAQvSjiDfr2id8u0W9hVxrbE3bt2jXUffv2NXO+5cbXX38d6kmTJmXkeGK9ZVPafPuW888/34yPO+64UO+///5mzrffefbZZ0OdyvbFMcQbdGRExEaSpk+fbubWrl2b8ePxESR9y/zGG280c/6W66xZs0LtW2LNmDEj4XPm6jXpIz36FriIbS+UbMvbTPHt8HQLy0MPPdTM+Vujf/7zn0N99913F/s5c/VcpsKfd30re080adIk1MOGDTNzfpvbjz/+ONT+Fni5cvbzu169eoXav24nizjm6rn0UaKBAweasb4uXnzxRTPnX3NLg37NP+GEE8yc31pcn2v/nuu3W9aINwAAACBvsegFAABA9Fj0AgAAIHpRZHq7desW6l/96ldm7tVXXzXjsm5ZNmTIEDPWObOpU6eaufvvvz8tx5CrOSWfz/Nbjk6ZMiXUfmvqssgT6nN73nnnmbkDDjjAjB955JFQv/vuu2YuWbY7FzO9/fr1M+M//elPZqzb2YwaNcrM3XbbbaH2rZEypVatWqH2f3M33XSTGeutUfWWxCIir7/+eqh9C6xcvSZPPvlkM/Zbg+vMs28J5l/fSkPr1q1D7XO6gwcPNmPdfu6SSy4xcxMmTEj4HLl6LndHX5e+ZZn+boze8jedjjzySDPW2yL78+HfZ/XY51319rk+mxzLuezZs6cZX3755aH233PSbcA+//zzzB7Yz9DrNxGR008/3YyvuOKKUOv3AxH7XuK3RSbTCwAAgLzFohcAAADRq7D7f5L99O5bRxxxhJkrX768GZd2vGH9+vVm/Mwzz5ixbq1y++23m7l0xRtyib4VpdvZiBS91XzWWWeF2rcEu/XWW0NdWlGHMWPGhNrfavnd735nxrqtnW5bJ2JbXfnHyRV6R55NmzaZuZUrV5px586dQ+132DrooINCrX+/IplrP7hhw4ZQ+9ZIPnpy9NFHh9q3YtP/NpUdvrLZ5MmTzdi/3p555pmh9u3d9M5Y+lZ1Ji1YsCDU/rXf70Cmd5vyO0glizfkKr/bYLt27cxY3+r28QbdEmvmzJlmbseOHWk5Pn++qlWrlvDf+vdZHRusU6eOmfvss8/ScHTZbcmSJWas2/OdffbZZu61114rlWNKxLd2/PWvf23GOoKi3ytESvb+yCe9AAAAiB6LXgAAAESPRS8AAACiF0WmV+dX3nrrLTPns0gVK1YM9Q8//JDZAxOR+vXrm7HPJemtS33OzWeudHukWOmcpt7SVESkbt26Zqy3ePbZS70NcGnl8XT7P5/TfeONN8x4wIABodYZUhGbZZ4/f346D7HU6N+Fbwv0z3/+04zXrVsX6t69eyec8xnRdGV6/Ram+pqtXr26mevfv78Zr169OuHj+G1UY7Bs2TIz9nlO3QbMby+qfz+///3vzVy6sut++299LW3cuNHM+RZqemtUnwNt3LhxqFesWLGnh1lmdG754IMPNnN6S20Rew17d9xxR3oPrBiS5cAvvfRSM9bfDdDXqEjR7cJjpF83Rezfum+nqb9LpFtpihT9HkVJ20bqa0vEXl/+Ozd+6/mdO3eG2q8B9Pcovvzyy2IdC5/0AgAAIHosegEAABA9Fr0AAACIXhSZXr0F7b///W8zd8MNN5ix3ipU95TMlDVr1iQdayNHjhbGExIAACAASURBVDRj3VsvH/l+rp988okZ6y1rjznmGDPXsGHDUPtsZbqyllWqVDFjnWf0PUD9uFKlSgkfx2+bGRufU/7www9D7bcs1j0lfQZf/5yIzYAny//pbYZFim5HO3To0FD7LTJ9pkxvXavrfHHPPfeYsc6+XnbZZWbu8MMPD/WVV15p5nweT+dvk/V99Zlr/x0KPV+jRg0z579fMWfOnFD7XOHWrVsTHkMu0b9L/10Hn1fX29W+//77mT2wFF1//fVmrL/fIWK/D7N8+fJSOaZsdt9994Xa97tv3rx5qPV3TUSK9h7fe++9Q73XXnslfD5/PfvMvl4H6ccUKfrarf+t3/K6uDlejU96AQAAED0WvQAAAIheQbK2JGl/soKC0nuy/zrppJPM+LTTTgv13XffbebGjRtXKsdU2goLCwt2/69SUxbn0jvkkENCffHFF5u5QYMGhdq3WVq0aJEZ61Y9vn2Y5m/D+GtHb13aunVrM+fbz7333nuh9tti6tuO/tjTfS6z4Tzq35W/9XbccceFumPHjmbO317Tj1O5cuWEz+db5PhWaPr2mo5OiRS97a0jFQ899JCZ823rtFivyWRtyU455ZRQ+6iBbyuloyLJWjX6CJg/77plmW7XJVL03OrXBX9LXG/V+uOPP5q5WM6lj/Lotl/+dUi3xNLbS+8JvTW7SNH37v322y/UPmbkXxv0Nay3TBZJ3nYrlnOZjI936fetPn36JP1ZfZ3660Dzr5P+vVO3FvSvv35LaX2ufaRNb2Hst3pPdC75pBcAAADRY9ELAACA6LHoBQAAQPSiz/R6Z511Vqh9exvdfidd22Jmg3zIKfk2X9dcc02ofabLt2EpKPjp1+Pbh+ksW+3atc2c/xvRbY18Bm78+PFmPGnSpFDrbbRFimZ8tRgzvZrPm+nMpm55JVK0TZHOifrtaHWbunr16pk53wpNZ8p8G7KPPvrIjKdMmZLwcZLJh2vSGzZsWKjbtm1r5ho1amTG+hr1v1edzdVblIoUbQmps6c+r++zgzo7rK9PkfzMgervRfhWgrp9ZLt27cycboElYrOWPotbs2bNUPu2cP4abtCgQah9ntS3s7zwwgtDvWnTJimuWM9lMvp68lup+/Whfg/UeXkRm9vdtWuXmfPXsH7v9Jn9Zs2ambF+b/dbZfv3TnfsZHoBAACQn1j0AgAAIHosegEAABC9vMv0aro/nYhIixYtQv3555+buWSZrmyXjzklzedEdb9HEZHevXuHWmfMRETq1KkTat8H0GcEFyxYEGqdJRQRWbFiRcKf9X1jk4k905suvi+y3o7W5wF9Nnvz5s2ZO7D/yvdr0mcHk23b7ft46nO7fft2M+e3LNbzfntT30v0q6++CnUq74v5cC71tukidqtu/z0IT28p7fvp6q2g/bnz37mZPn16qPUW9CKpvYYmkw/nMhU6Ry1i37d69epl5qZOnRpqn9nX35sRsa/H5cuXN3P+OtXbYafy2kymFwAAAHmLRS8AAACil9fxhmT8R+5+C8umTZuGev78+aVyTCXFLZuS062UVq5caeb0LRoR214r2RaNe4J4Q/H469e3tiprXJMlp6MRvjWSj0L4NliZwLlMTreg8rGWxYsXh9q3M/MRsdLAuSw+fy71GsnHGfy24zomUdrvlXzSCwAAgOix6AUAAED0WPQCAAAgemR68wA5pXiQ6Y0D12Q8OJfx4FzGg0wvAAAA8haLXgAAAESPRS8AAACix6IXAAAA0WPRCwAAgOix6AUAAED0SrVlGQAAAFAW+KQXAAAA0WPRCwAAgOix6AUAAED0WPQCAAAgeix6AQAAED0WvQAAAIgei14AAABEj0UvAAAAoseiFwAAANFj0QsAAIDosegFAABA9Fj0AgAAIHosegEAABA9Fr0AAACIHoteAAAARI9FLwAAAKLHohcAAADRY9ELAACA6FUozScrKCgoLM3nw/8pLCwsSPdjVq9enXNZBjZt2pTWc8k1WTYycU1yLssG5zIenMt4JDqXfNILAACA6LHoBQAAQPRY9AIAACB6LHoBAAAQPRa9AAAAiF6pdm8Atm3bZsY//PBDRp5n3333zcjj4ufttddeZty0adNQ+3NRoYJ92fnmm29CvXz5cjO3ffv2dB0i0qBmzZpmXKVKFTNetWpVqHfu3Fkqx4T0qFSpkhlz7WWXggLbjKBGjRqh1q+hu1OunP2sc9euXXt2YDmGT3oBAAAQPRa9AAAAiB6LXgAAAEQv7zK9OhdTrVo1M7d58+bSPpy8oHO8mcrwevpc+iyUP+9Inc7sioh07drVjIcMGRLqxo0bm7ktW7aY8cqVK0P9ySefmLnPP/881PPmzSvZwSIlDRs2NON+/fqFulatWmbO53b19e3Plz63+ZYjzBa/+MUvQl23bl0z9/HHH5vxggULQu2/i7F169YMHB20Nm3amPH8+fPNOJUcr+avPf19jB9//LFEj5lL+KQXAAAA0WPRCwAAgOgVFBYWlt6TFRRk5Mn23nvvUA8aNMjMLVu2zIw3bdoUat9u57vvvgu1jzron8s1hYWFBbv/V6mpXr16sc9ltsdGcqm92aZNm9J6LlO5JnWk4aijjjJzw4YNM+Nu3bqF2t9OmzRpkhmvXbs21P71aMWKFaGeOHGimZs5c6YZ+9hENsvENVnS19fWrVubcf/+/c1Yn0vfsqx69epmrG+5zpkzx8zp2+U+xuJb1eWSbDqXzZs3N+Nrr73WjPWtbN3ySkSkQ4cOZjx58uRQ33777WZu6dKloc6l6253yvpcdurUKdTff/+9mVu4cGH6DqqYfCvKXIo/JDqXfNILAACA6LHoBQAAQPRY9AIAACB6OdmyrEmTJmb817/+NdQ+p6S3xRQRGTVqVKjHjBlj5po1axZqn/OsXLly0sdF5vXs2dOMp06dmpbH1ZljvxWnH+ezVq1ahXr//fc3cz7bqdsfffjhh2bO5+x1bsy3tNPPOXToUDPnWy7pnGguZ0RLW+3atc1Y/85F7HVXp04dM7do0SIz1u0Au3fvbub222+/UPvX6U8//dSMZ8yYsbvDxn/p66dPnz5mbp999jHj3r17h3rHjh1mbuPGjWbctm3bUH/wwQdm7uabbw71U089Zea+/vrrYhw1fo5u/eh/r2XBZ3grVqwY6tJqP5pufNILAACA6LHoBQAAQPRY9AIAACB6OZnp9T1CR4wYEWq/VZ/vB7lmzZpQ+0zghg0bQu0zvX4rVZ2D8/0o2WKz5PQWqDo3JiJyzjnnmLHu9/qHP/zBzL3//vslev7t27ebcfny5UNdoUJOXi5p06hRo1BXrVrVzL377rtm/Prrr4faX5O+57XOibVs2dLM6V6VOnMvItK+fXsz1sc0ZcoUMzd37lzBT/S57Nixo5nzeW39euaznf61T78u+tyufr098cQTE86J2B7qH330UZHjx0907tJ/38X3z9ZZ6XvvvdfM6X73Ijb/63vG/u1vfwt13759zZx/3dZ/I7w3Jue3ey6u+vXrm7Fe54iItGjRItSLFy8u0XOI2Byvv2bXrVtX4sctTXzSCwAAgOix6AUAAED0cvJ+7RVXXGHG+pbJxRdfbObGjx9vxvqWm79lo28R6NvaIkW30tW3Wv1toWnTpoW6NLd5joFul+Rv9Vx11VVmXKtWrVAPGTLEzOlz+eyzz5b4ePTfiN+22v+NxK5du3ahLiiwOzz6a0lfA7ujb8/6FnHz5s0Ltd8S029/quMN/parvg71Y+Yr/bvyv1f/WrdkyZJQT58+3cz56IFuaeZvf+rtjXX7MhGRgw8+2Iz167SOv4iUPLqUD/Q1KlI01nP33XeH2kdVvv32WzN+8sknQ33ppZeaOR31Gjx4sJnz21pfc801ofYxKFiHHXZYqF966aVi/9wf//hHM77kkkvMeE8iDYns3LnTjPX7sY6KZhs+6QUAAED0WPQCAAAgeix6AQAAEL2czPT69hx///vfQ/32228n/VmdPaxZs6aZ0+2q/DbDTZs2Tfg4vuWPziXSbic1+vflW1stXbrUjHXur1+/fmZOZ9v81py61VkqvvvuOzP2be1ipzOaviXO008/XeLH1flA39JIP6ffWti3EdQZ67Vr15o53X7Jb5OazfmzTNGvff617ptvvjFjvT3thAkTzJzO+4rY7KB/fdUtjc444wwz5zO++nz5/Lh+XfAZ43ykX4d8mzh/zejz419fk7nrrrvMWJ93fy6PPfZYM77nnntCfcMNN5i5F154odjHkA/833oyepvismjJ6F839Tbx/rtU/jsfZYlPegEAABA9Fr0AAACIHoteAAAARC8nM70TJ04041mzZpXocXx2zY81nzXUfS59plf3c/3ss8/MnO8tisRmz56ddF73E/V5X6Tf+vXrE86tXLnSjHVv1cqVK5s5v1Wq7nOtt8f1j+P7f/rekzrP6PsE6zxpvXr1zFw+Znp1/tn3Evevg/oc+Ayv7/GrtxH3+VKd3f7qq6/MnO7xKSLStWvXUDdv3tzM6X6/ZHrt66Dvsax/VyIiCxcuLPbj6ky2/5t45ZVXQt2gQQMz589Xly5dQn355ZebOf39imXLlhX72GKVyneATjvttFAff/zxmTiclLRp0ybUvofv2LFjS/twEuKTXgAAAESPRS8AAACil5PxBt+ew7cnygS/Faa/paPpj/b9LT7iDfHwtxL1LWO/ZXEM5s+fH2p9y/LnVKtWLdQnnHCCmfPt//SW3j76oFvddOjQwcz5lkv6Fpq+1SZib4P75/ettZLFnGKh4yjVq1c3c/o8i9iWf506dTJz/mf1NuI+RqLjDf5xfBvKFi1ahNq3O9Kvr3778XfeeUfymY/T+etUR4n8e5HfylZHUL788kszp2MtM2fONHO+TZpuR7fPPvuYuaOOOirU999/v+Q7HxNLRm8brbcvFhEZPXp0wp/za5kffvih2M+ZzKhRo0J95JFHpuUxM4FPegEAABA9Fr0AAACIHoteAAAARC8nM71bt2414xUrVhT7ZwcNGhRq375F55R8Frdz585mrNuy+AyczkLtrg2Lbn2mMzox0a2mfAbQZ/J0LsjnyLKdzhr6vK/emvrnxrlgzpw5ofbXh6fbm3Xv3t3M6VyhiM1z+i2C27ZtG2rfBkdvUSwi0rNnz1D7NmR6e88ePXqYOX+uvvjii1D7llixtDfT2UH9GiQiMnnyZDPWr4v+d+5zu/p61luBi9jXRZ35Fim6pbfe5tp/f0JvB37iiSeaOZ+l1+0s/et0jPz7jR/rTK3PvXfr1s2Mhw8fHmqf+9RbV/vMtd+uXf/efdtBvX38uHHjzJy/LlPJu+aqVNqv6nPif6/F/bl00uuwBQsWFPvn/NbLvoViuvFJLwAAAKLHohcAAADRy4l4Q+PGjc1Y32YVEZkxY0bCn9XtdkRsq5z27dubOb3rmt8hyNu+fXuo/a06ffvW37bT7ZlEbLs1v7uRfg6RoreRcoW+fXHggQeaOd3iSMTuMqOjDiJFfx/6VrO+3ZaN/LHrsf8byVb69nQqt6D87XK/u6GO9fhrct68eaHe3TWp+Z3d9DXpoxn+9UXvBubbm/nbj7pFVKZuG2aCPlbf8tFHOHQLKh8f8NEyHX/w50u3MfJt4op7rCIiO3bsCLVvgeX/W3QLPN/qcurUqWYcQ3RF75QmUvS/Sf++fMzIxxL8bWdNR4lSeS348ccfzfj0008P9UEHHWTmbr/9djN+7rnnQl2unP28LoZzJ5Ja+9UXXngh1D4aUtZSiSbu7u9H/x2mI/rAJ70AAACIHoteAAAARI9FLwAAAKJXkOn2EObJCgoy/mR6q0sRkcMPP9yMfRsdTedLfc7S/550jtfniz755JNQ+7yxbuUkIrJ48eJQ6yyhSNEcqG6flEp7s8LCwsThrBKqXr16sc+l3p63YcOGZs7nyvTvUm9FKiJy7LHHmvFxxx2X8Dn134HfrtZno/W59ZlFvbXt9ddfb+b0uUunZBnfTZs2pfVclvSa9Odt27ZtCf+tb4XkWxHp1oE+167blPn2WD5zmCz7pdui7W4bYt0iyx+Pzz1OmDAh1D7v669fLRPXZKZeX/Xv1V+//vVWtyyrU6eOmdNZXN8mTb9G+LH+Of+zvlWTb4moH+eNN94wc37L3o8++ijUGzdulOLKpXOZzNFHH23G+r3T5971tehfX33OWp8D/zqhWxT673f4jO/s2bNDfeONN5o5n2X27Q2LK5fOpW45l2vt+PRrrt/2PVl7T7/u8d8P0RKdSz7pBQAAQPRY9AIAACB6LHoBAAAQvegyvT/znGastyDVPRxFbB7MZ498Tklnev22yJ9++mmofebEZ0YrVqwYap+P85lA/bOTJk0ycz4Xo5V1prek9O9GpGgOsG/fvqE+4IADzNzQoUND7bdO9flBndv0WTCdP126dKmZ+/e//23Gjz32mKRDLmR6U+HPm+9rXdY9lvfff38z1r28PZ8H13mzRYsWmblp06YlfJxcyg4m4zPPent2/x0FnYf2/cr9dadfN/3rov578nP+cfW59O8FflvbZ555JtTvvPOOmYsln52KypUrh9rnqnUfbN8T279u69dQ33O5f//+oT7ssMPMnH+v1NepP5d///vfzfjxxx8PdSq9b2M9l6VBr5H8mihd/Bbyfpt4jUwvAAAA8haLXgAAAEQv+nhDMr5Fim6dlC5dunQxY397R2+f6LdH9S279C18v+3gxIkTEx5DLsUbfDurdPDn2d9+0/z50bfuDjnkEDPnt7jW22Q++eSTKR/nz/FRh1yJN+gIg48PdO7c2Yz1NeBvI3/44Yeh3pO/DX2r1F9n/nFr164d6pYtW5o53c5MxLYp8//2gw8+CPWWLVvMXK7eRvXthLp3755wfs2aNWYula1Ji8u3sdORJxGR3r17h9pvce1bqumWZa+//rqZ063pvFw9l55ugSViYyS+9WZpuPLKK81Y39rWW1qLFI33jRw5MtQPPPBAsZ8zl86lju35yN7q1asT/px//yvp9uk+2uS3mM6Ea665xozvuOOOhM9PvAEAAAB5i0UvAAAAoseiFwAAANGrsPt/Ehe9taHfflRvgZuubf38Vpd+G0/dTkVvyShSNM+psze+NY9ul1TWLaCyjc9qJ2sJ5ulzsmTJEjM3b948M9b537Zt2yb9t7HT25YOHjzYzPmcu95aUm/hLWJz1O+++66Z8+3DktHtsnxu2Le001vbrlu3zszpNk7eV199ZcY6y+xbDOYqn2n2Y52T7dChg5nTba/S9frq21G99dZbZqy3ffeZv379+pmx3qLe/1vdhtJvRZ2rfC72tNNOM2Odp/ctPDORz/b++te/mvGwYcNC7duZHXrooWZ80kknhXrmzJlmbvLkyek6xDKlry/ffi8Zn+HVf/f+uky2nXNpZHg9/3enW969//77xXoMPukFAABA9Fj0AgAAIHrRxxv87TfdnujEE080c/qWbLpuv3nJogd+9zbf8qdWrVqh9rsd6ZYyuRxv0BGBTZs2mbmyvq3oj2fs2LFmrI+vpG1gYqFvZfvfRbLdBP1OWLollo8WpBJvSHRsIkWvF72L11lnnWXmPv74YzPWO5D5Nle+RVYM/M6Uvv2bvmWuowUiNoKVqddXH11ZvHhxqL/44gszp2+NithWdf7WrX4fWbBgwZ4eZlbwbb78daFbfb388stmTscHSou+vnyrumOOOcaMBw4cGGq/w2Is8Qb9d7hixYoSP45+jdM72YmIzJ07t8SPm4yOp6Tyvj5q1CgzHj16dKiJNwAAAAD/xaIXAAAA0WPRCwAAgOhFl+n1W+P57fj23nvvUPvclm7Rkg185sxv+anprVxzmc5tDhgwwMz5diUl3Roz2Xa2ydqZZWKL5FjpFjpTpkwxczqbLmLbglWoYF+SdFbNZ2R9uyqdGU3WTsdvP+vb/+k84PDhw81cz549zfjOO+9M+Dyl0daptOn2ciJFt1cuKPhp50+f9/3d736X8HHfe+89My5pOySf+z7wwANDfcQRR5g5/5qpW8z5rVp1C8JYMr0+W++3WV++fHmo9bbiIrYN4bRp08zcxo0b03WIhm4l6P+2dEs5Efs64vO/sdiTHK+m10y7y/Dq67uwMPHuyi1atDBjna0XSd/3c/wW98XBJ70AAACIHoteAAAARI9FLwAAAKIXXaZ3d1kwnQMcOXKkmdP9RKtXr27mfI/WTPCZM71lsojdLnDixIlmzucbc5X+75g1a5aZ83lKraT5Xq8scrs+/9SkSZNQ+22sc4XuoevPo88O6rygz4npHLvPtPutSHVfWN+vVWvTpo0Z+22I9XWmz4VI0S2tM9XHMlvpnKdI0eyrzgf667VHjx4/W4sUPSf6dTzZa7rP4Pu/n/r164daf59DpGjWU28j7c+z/xuO0bPPPmvGOnt/1VVXmTmdybz11lvNnN9KXP+N+O/caN26dTNjve2wiEilSpVC7f8mfI973U86li3AM+XFF18s9r9NluPVfIY3XfzfREmuSz7pBQAAQPRY9AIAACB60cUbPH/7rVy5n9b5fntCfUs0Xdtk+vZMBx98sBn37ds31Pvtt5+Z69Klixnrdjz33ntvWo4vm+m2VyIiS5cuNeNjjz021L/+9a/NnN42029tWxYGDRpkxuecc06o/Xatzz33XKkcU2nxraH87U+9pbaP6egtQ32bG/87Pfroo0Ott5T1fMsyb/z48aF+9NFHzdxbb71lxjG2JUvFkiVLzFjfvvYRE73Nrd8e3reN09f+tm3bEj6/3rZcpOjrrY5N+DZXvvWYPre6/Z1IfrQr9OdyzJgxoe7YsaOZ0+fvqaeeMnPLli1LON66dWvC5/evg37L6wMOOCDUPpqyZs0aM77rrrtC7d83YPmWkpq/Lsv6dzlixAgzfuWVV1J+DD7pBQAAQPRY9AIAACB6LHoBAAAQvYLitqBIy5MVFJTekyWgM18dOnQwczqP5vO1DRs2NGPdksnP6eytzrGJ2K0U/c/6DJzPRt1xxx2hTqUNS2FhYcHu/1VqqlevXurnsnz58masW7odfvjhZk63f/NbaPpzonNBPrv3xhtvhNpnrH1G8Jhjjgl1r169kh776NGjQ/3444+buQkTJoTat+bZtGlTWs9lNlyTTZs2DbXe5lIkeYbMbzmr85v+cVq2bBlqve2xSNE2PDqnm6ntvTNxTWbDudQ5TN9O7Mgjjwy1vyarVq1qxjqz6R9H57V9RtS3rtIt1nxm1G/jqr/H4TOiyVowxXouNZ+Rv/LKK0Otr62f+7c6i+/fc/Xv1ee8dQtREdtSTmf9RURmzpxpxjoH7ttaJdsCNx/OZa7R7+2+rd3dd98dar9+SnQu+aQXAAAA0WPRCwAAgOix6AUAAED08i7T+/+1d6dBUpfHA8d7EFhOuZZjkftUbkFB8QBBvAARKCIihsSYlJSVWCmtWKmikjLJi1Qqh8YjWrFiSIxJaaIIAgFBbkGugFwiiNyE+z4EZP8vUv/H7l5m2JmdvZ75fl51VyPzY387s48zvd2p6H5b36frV5UePXo0xAMHDjS1Y8eOhbhBgwam5tcb63WXS5YsMbVVq1YV46qvLJae3lR8P5juqfX3R8+FFbH36JprrjE13Rvq+0T9DGjdB+h7zPysYH2v/Uxo38umxdjTq/nVwvprfurUqbK+nFKT672Dfu6r7wvVz0P9+xMi9rVYv9aKiFStakfPnzx5MsR+lfzGjRtNrnt8fd9/Krl+Lz29xlvEzqbXc/JFbO+tfz31M7D194F/jfS93X4Gc3FxLzOnzzb+uZaK7wH3ZyT9fP/kk09MLdXcZ3p6AQAAkLM49AIAACB6tDdkSH+M5lsY9Fv7fhRPaY1ASiUX2htS8R+b+Y/Q9Xg6v6JWf7TiP173H7vqj9T8CmX/Z/1opeKKvb2hJPRzMp2Pp8sDH6NmTreh+XXBfnVtqvFU2cK9LB01atQweap11NnCvcyc/rnqn3f+eanHEPqfjbolqSRobwAAAEDO4tALAACA6HHoBQAAQPTo6c0Bud7TGxN6euNA72A8uJfx4F7Gg55eAAAA5CwOvQAAAIgeh14AAABEj0MvAAAAosehFwAAANHj0AsAAIDolenIMgAAAKA88E4vAAAAosehFwAAANHj0AsAAIDocegFAABA9Dj0AgAAIHocegEAABA9Dr0AAACIHodeAAAARI9DLwAAAKLHoRcAAADR49ALAACA6HHoBQAAQPQ49AIAACB6HHoBAAAQPQ69AAAAiB6HXgAAAESPQy8AAACix6EXAAAA0atalg+WSCQKy/Lx8D+FhYWJbP+dBQUF3MtysG/fvqzeS56T5aM0npPcy/LBvYwH9zIeye4l7/QCAAAgehx6AQAAED0OvQAAAIgeh14AAABEj0MvAAAAolem0xvKQqtWrUzeuHFjk2/bti3ER48eLZNrwtfOnz9v8iNHjmTl761bt67Ja9eunZW/F5mpUaNGiJs2bWpqFy9eNPmePXvK5JpweYmE/SXnOnXqmLxTp06XjUVENm3aZPI1a9Zk+epQWmrVqmXyatWqmfz48eNleTlIkz7bNGzY0NS2bNli8kuXLpXJNVUGvNMLAACA6HHoBQAAQPQ49AIAACB6UfT05ufnh7hr166mNmzYMJNfddVVIV6wYIGpTZkyJcRnz57N5iXmNN3Hm60eXu/kyZNJc/39ISJStWoU3/blyveB9uvXz+R33313iNu2bWtq//3vf02+cePGEC9fvtzUPv300xJdJy5P37/OnTub2iOPPGLy++67L8T79+83tVdeecXk+t4ePnzY1C5cuJDZxSJjeXl5Jr/11ltD7Puzjx07ZnLdF6p/F0ak9F7H8bUGDRqYfMCAASa/6aabQuyfW6dPnzb52rVrQzxz5sxsXWKlxDu9AAAAiB6HXgAAAEQvis95vIcBXAAAIABJREFUDx06FOIVK1aYWrt27Uw+ePDgED/11FOmpj+ife2110xt/fr1Jb7OXFXeH4Xp7w8RO05LRKR+/fpleTlR8G1DY8aMMXnz5s1D3KZNG1Pz43X02Kv27dub2qJFi0K8atUqU2PkYOb01/kXv/iFqY0ePdrkkydPDrF/zfQtDChfjRo1Mvnjjz9u8rvuuivE1atXN7WPPvrI5HqkmR/9qVsftm7dmtnFogj9s+h73/ueqel7JyLSunXrEPv7s3fvXpNv3749xH379jW16dOnh3jlypXpXXAlxDu9AAAAiB6HXgAAAESPQy8AAACiF0VPr+b7N1966aWk+U9/+lNTGzt2bIg7duxoar7H99133y3RdaL8nDt3zuR6zJLvjdIj7nJdkyZNQtysWTNTq1mzpsn16mE/7mjp0qUm1/2l48aNMzW9VtyvM16yZInJdd8aUtOjHXv06GFqv/nNb0z+9NNPl8k1oeQeeOABk//oRz8y+alTp0I8e/ZsU/OjHP3IrGTq1atn8nXr1pncr55Hcvp5OWjQIFO7/fbbTf7ZZ5+FeMaMGabmf2+loKAgxL5nX/eBd+jQwdT+8Y9/FOeyKxXe6QUAAED0OPQCAAAgehx6AQAAEL0oenr1HNB0+vqeffZZk69evTrEP/vZz0xt0qRJJte9hn4V55dfflnsa0ByvpfQr8n0fdbZcPDgQZPrPtYqVXL7/xF1P6D+uojY54OI7Rf88MMPTc0/R3VPqZ9Fqfva/DpjP5NUPybri1PTM1r9Wmh6eCuXatWqhfjee+81tYsXL5p87ty5Idbzl0VEzp49a/I6deqEWK+8FRFp2bJliP1sfH09Irb3tLxntld0+fn5Ifa/XzJ//nyTT5kyJcRr1qwxNX/f9WzegQMHmppeM3711Vebmp9h/+abb4b4xIkT/vIrhdz+KQ4AAICcwKEXAAAA0auU7Q1+jNTIkSND/Ne//tXU/AizVKZNmxbijz/+2NT+8pe/mPyZZ54JsV7XKGJH/jCuJT16FaZvE/nVr35V1pcjBw4cCLEfmZVIJMr6csrVtddeG2L/MZgfS6bXWc6ZM8fU/H3dsGFDiH0Li/4IbciQIab22GOPmTwvLy/EvhVl48aNgq/dcMMNIb7xxhvL8UpQUhcuXAhxr169TM3/rPztb38bYv+RuKfX3Op2BhGRzp07h/iWW24xNT+yrLCwMMTr1683tdOnT6e8hlzTpUuXEOsxYyJF28RefvnlEOuv8eXotdHHjx83Nd1y4lsffHuD/l7T5yUR+7OyIuOdXgAAAESPQy8AAACix6EXAAAA0auUPb1+TaZeifr3v/89K4/h+1OGDRtm8kWLFoXYr3rctWtXiN94442sXE+u0Ksv/di4//znP2V9OcbRo0dN3rBhw3K6kvKh1zf7cWF79uwxue7NTWeE38yZM01+8uTJEPs+Xd/j+4Mf/OCyjy9i+9j8teaiunXrhtiPnELlkmpVtx9zdaU+Xm3Hjh0h9n26es2tX0Psx6bpnn7/WpDO9eSC5s2bh9j/vNm0aZPJr9THq+m+3QULFpiafo3VowxFRG677TaT6x5s//tSehWy7v2taHinFwAAANHj0AsAAIDocegFAABA9CplT+/w4cNNPm/evBD7lZp+JaKeSalnwl6JX+v3yCOPhHjt2rWm9uCDD4Z4+vTppub7dHJdt27dTL548eIQZ6s/uyS6du0aYj9TUve45gK9TrRqVfvSoVeWihSd26vpvjURO+far7bUz63u3bubWseOHU3eu3fvEI8ePdrUdN8qPb12TbNeL52usWPHmly/Ti5btszUateufdk/JyLy+eefZ3wNuU7/HoT/+afnZafrmmuuCbGfc62f77feequp+R7fu+++O+lj0NNr7d27N8QHDx40Nd/Tq/l5zNddd53J9blD92qL2O8f/7qtayIi/fv3D/HChQtNTa9N1v+OioZ3egEAABA9Dr0AAACIXqVsb/AfT/qPUzQ/OkOPOfIjj/Tb9bpl4nK2bt0a4rlz5yZ9jKFDh5oaI8wsv5ZStzdUBHpF7pXk5+eH2H/8HwP9kfT27dtNza/M1COx7rjjDlMbNGiQyfX4szNnzpiabinxI3p8m4Su65XJ/hr2799vavv27ZNcs3PnzhDrdbNX4u+lX8/+y1/+MsT+41k9/sjfS/8a6lsjkNzhw4dD7Fvt/MfTqYwfP97k+uNqP7JM19q0aWNqHTp0MLkeiaXXF4vYNomStNnEQrf5+PY5/ZoqYsfG+bPMuHHjTL558+YQ+/OSHj3m74//s7t37076Z3UrDe0NAAAAQDni0AsAAIDocegFAABA9Cpl4+Gvf/1rk7ds2bLY/+3SpUtDfPXVV5vapUuXQqxXG4sUHQWj+bFkeqzHxIkTTc2PWZo2bVqISzJeprL67ne/a/K+ffuGePLkyWV9OSXi1zJqugdOpOiImcrGrzft0aOHyXU/Z4sWLUzN56NGjQqxf57p56Tv90213tj3GOvH0CPSRERWrVplct0XGWu/rx5Bpb/GV6JH+InYlesiIj/5yU+S/rf66+x7ev04wEz5flL9uxex0r2wvqc3VZ+sX6P+7LPPmvyTTz4Jsf9ZqVcf169f39T8unDds+97jPv16xdi//sd/jXT12Okf4fEvy7532XS92DChAmmdvPNN5v8pptuCrF/7ulRgrpPWESkZs2aJtfr3f3rhn4O+8efPXu2yfX4y7LGO70AAACIHodeAAAARC/h3+ou1QdLJErlwfQoDz0epSTS+fhZj6oSsRvZ/DgXvRFOxH4k8OGHH5qa30jmP0osrsLCwkRG/2EKBQUFxb6XqVpDXnjhBZPrj7CfeeaZDK6sctAfF/qP21PZt29fVu9ltp6TXbp0MbneHuSfH0899ZTJ9fYgPz5Lf3SqP4a7Er+JUY/e0WN3RIqOyVu+fHmIdTuUSNGPj8+ePVvsa9JK4zmZzr3UH0Gn097g2wf8x6EV+SPovLw8k6dqj0lHed9LzW8/82PjVq9eHeI+ffqY2quvvmryVBvAfPtQcfnni/44X28JFBGpXr26yXW7xYEDB0wt1QbIdJT3vUwkvn5430qU6rn14x//2OQ9e/Y0uf67/M8bfX5KZ2yc36p4/PjxEPvzk792fS/92EN/bzOV7F7yTi8AAACix6EXAAAA0ePQCwAAgOhF0dNbmYwZM8bkI0eODHH37t1NzY+Feu6550Ks1xVeSXn39OreuaNHj5ra4MGDTa5XxF6pP7C4Kyx9f6dfTa35vtFsjVJKxa+XTNW7WlF7etPhe9V0f6BfLZxqnI6/r/q1zPebNW3aNOnjDxgwwOT6e/Bf//qXqb3//vsmX7FiRYjT6REt797BTPlxVOn0A8eqst7Lbt26mXzo0KEm173vfq267r33r6e+d1r3qfrVuvq/bd++van5389JNfJuyZIlJvfjvYqrst5Lv3bd/x6F/lmpx5f5mn6dFCn6Gqt7sv1rrB6h1qtXL1PTo85E7P1ZsGCBqfnXWD+6rbjo6QUAAEDO4tALAACA6HHoBQAAQPTo6a1A/Ly6u+66y+TvvPNOiCdNmmRqR44cSfr3lndPb6ZSzff1fC+oXoPo5xL6/rRrrrkmxF999ZWp6ZXXqXqBS8LPNPTrN7UYenrToXvKfE+v/7qdP38+xL7XVM/L9rOy/cpMvRpV96mJiLz55psm18/ZdHrPKmvvID29RVXWe+n5mfJ6Rrb/XQw999rPAvara3Xvp+/p1a/NvtdUz+8WEWnbtm2I9ZxXEZE5c+aYXK8P1z36VxLLvfRro0+cOFHqj6lfK2+77TZTGzdunMn79u0bYj8Lfe7cuSafMmVKiHfs2FHs66GnFwAAADmLQy8AAACiV/XKf6Ry82sz9Ygwv/Y309EY2fLNb37T5H/+859N/p3vfCfEelSSiMjrr79eatdVltJpadD27t1r8unTp4fYjwDTI1pERBo0aBBiP/qstFoaNN9uoce76I8YKyv/saluHxCxH53qNakiIocOHQpxSe6FXnHq1536Fab6e+COO+4wNf36ISKydevWEPv1xul8rFqRNWzYMMTDhw83tXvvvdfkb7/9doj9uDdUbL4NTI+9WrNmTak8ph5v5p8vvtXs8OHDIfYrlP2fXbRoUdK/N0a6XUCk6Fp43f7hX6eyZefOnZd9PJGiLTAjRowI8ejRo03Nt0zp0aV+FJ0fm1YcvNMLAACA6HHoBQAAQPQ49AIAACB60ff06p47EdujN2rUKFN7+umnQ+zXGpaHiRMnmnz8+PEhfvzxx01t1qxZIfb9rblIj8bxo638asVUfcTNmjUr1p8rCb++NtUa4spCjxfzfbF33nln0j/rx/YtXrw4xLqnL5v8c13fDz/655ZbbjH5Qw89FGLfjxxLL2G7du1C7F93/KjA73//+2VyTSg5v7r2qaeeMrn+OXLgwAFT8yPDMqWfa/PmzTO1zp07J/3v/HPLjx3UI8tWrlxZkkussPTvSjz66KOm5l9z9ddn6tSppqZ//yVb/P3Rr+MidiynX1t93333mVz3J3/xxRem5n8fozh4pxcAAADR49ALAACA6EXf3uD98Y9/DLHecCYics8994S4IrQ3nD171uSvvfZaiB944AFT0x9B0t5gP4bWH3WJiJw5cyajv1O3OojYDWAiqbfipTJhwgST6y2Js2fPzujvLG+6vcR/FOq39QwbNizE27dvN7W1a9eGuLTaGzz9cagfyeXH6eiP9/Vmv5jo72vffvLBBx+Y3I8mKm96ZKV+fRcRefHFF8v6cioUP+aqU6dOJtfP4datW5tattobUtm8eXPS6/Hb2/xo0l69eoVYt/6JFN0uV1npMW3+3+Q33+ntrn7jWaNGjUJcWq+x/meuvrd+ZNnnn39u8q5du4ZYj0QUsa2LfmxdMrzTCwAAgOhx6AUAAED0OPQCAAAgejnX0/vuu++GeMuWLab2rW99K8S+r2TKlCkm16ti01GzZk2T+77dVHSvoV+HWq9evYyuJ1bp9GTrr52/P1ppjSzz/WmfffZZqTxOefHrvTds2GBy3Qvr+331akvfm+fXCWfKr0nW4310v5uIXVktIrJjx44Qx9rTq8cC+XXnvnewrPmRaXfffbfJhw4dGuIbbrjB1PyIuY8++ijLV1ex6eeWiMi///1vk+ueX91XKSIybdq00ruwJPRzT/8Oi4jI9ddfb3L9GurXvMdi165dIX7vvfdMrUoV+36m/t2E9u3bm5oeRzd//vysXFv16tVN3r17d5P3798/xP411v8M1uvn/bmruH28Gu/0AgAAIHocegEAABA9Dr0AAACIXpzNLsX05JNPJs3HjRtnar7vUs+ZSzX3denSpSb3M2PT6emtW7duiH3/mV/PV1mVxdpf7/jx45eNy8qDDz5o8ueff77Mr6EsvfLKKyZv2LBhiP1qcD2P2vdv7tmzx+S6v1TPOvZatWplcr+OVa/FTNVfJmJnYvte5Rj52dQtWrQw+e7duzP6e/3K2UGDBoXYz8fWevbsaXK/jlbfk0mTJplarvXwegsXLjS571fXz0u/Klb/PKxfv76p+e+BVCvhNf9c69Onj8l1v3+/fv1Mzc+X1T87Ktrs6NLgVy23bNnS5HXq1AmxX/Or+6EHDx5salu3bjW5/l2nVOeeHj16JH0MEZEBAwaE2PeL+zOSfg773yFgTi8AAABwGRx6AQAAEL1Eqo8Bs/5giUTZPVgJ+fWjHTt2NPnJkydDnOpt9RkzZpj80KFDGV+T/qj36quvNrW33norxP4jgMLCwkTGD5pEQUFBqd/LL7/80uSxrI8UsR+LFxQUmJq+zx9//LGp7du3L6v3siI8J/UK0eHDh5uaXifapUsXU/MrgXWe6mNU/32k24ZE7EeBvt3Fj1+bM2dOiP1HjKnac0rjOVkR7mW2PPTQQyFO1d7grVmzxuTz5s3L2jUlE8u99F9nvbbZf1yuV1P7j7kvXrxocj0iq3bt2kkf37cd+RFzicTXX2b/sbtfX/6nP/0pxMuWLUv6mF4s99J/nfWq9/z8fFMbO3ZsiP1roR+F6uvJ+JYX/3qsW2D8uunly5eb/KWXXgqxb2lLJdm95J1eAAAARI9DLwAAAKLHoRcAAADRo6e3ErnjjjtC7Mcj6VWCXmXt6fX8uCTdV1YR6N7Uhx9+2NT8WDK9RvPpp582talTp4bY/5tj7OnVfM+YHmfj1wU3btzY5LrvsEaNGqame+D9mEDfA6/H8vhVx753UI8w0yuJrySW3kHEey/1802vJBYROX36dIh9L7AedSZiV9L6Fbi6790/n32vsB49tmXLFlPzvdy+T7S4Yr2XusfXf531GmA/Jq5bt24m1/fPv1brsXH+d478GDv9GuvHrfo+/EzHINLTCwAAgJzFoRcAAADR49ALAACA6NHTW4HVqlXL5LoXMZ37FktPbyqpZqI2bdrU5K1btzb5unXrQnz//febmv6a+9mH/fv3N7meM+nnu/qea923tH79elPz84m12Ht601GtWjWT634zPytU9/j6mp/bq+u6d/Fy/63PiyvW3sFclIv3Uq+N9s9Dr2rVqiH2M8n13FXfa7p//36Tb9u2LcSpVuCWRC7ey3ToPu9jx46Zmv79i127dpma/z2KslgNTU8vAAAAchaHXgAAAESP9oYckAvtDan4j6j1CB0RO2plyJAhpqY/hmnSpImp+Y/jdIuFH7Oi11aLFF2hW1y0N8SBj1Hjwb1MLS8vL2ktVStXeeBexoP2BgAAAOQsDr0AAACIHodeAAAARI+e3hyQ6z29MaGnNw70DsaDexkP7mU86OkFAABAzuLQCwAAgOhx6AUAAED0OPQCAAAgehx6AQAAED0OvQAAAIhemY4sAwAAAMoD7/QCAAAgehx6AQAAED0OvQAAAIgeh14AAABEj0MvAAAAosehFwAAANHj0AsAAIDocegFAABA9Dj0AgAAIHocegEAABA9Dr0AAACIHodeAAAARI9DLwAAAKLHoRcAAADR49ALAACA6HHoBQAAQPQ49AIAACB6HHoBAAAQvapl+WCJRKKwLB8P/1NYWJjI9t/JvSwfpXEvAQDIBbzTCwAAgOhx6AUAAED0OPQCAAAgehx6AQAAED0OvQAAAIgeh14AAABEj0MvAAAAosehFwAAANHj0AsAAIDocegFAABA9Dj0AgAAIHocegEAABC9quV9AdnQunXrECcSCVO7dOmSyXfu3Fkm14TLa9asmckLCwtN3rRp0xD37dvX1C5evGjy9evXh3jlypXZukQAABAh3ukFAABA9Dj0AgAAIHocegEAABC9CtvTW7169RDfeOONpta8eXOT5+fnJ/2zBQUFJl+3bl2IZ82aZWpz587N7GKRUs2aNUM8ePBgU+vdu7fJz507F+LatWubWvfu3U3esGHDEOv7KiIyadKkENPHDQAAeKcXAAAA0ePQCwAAgOhx6AUAAED0KmxP7/nz50Os+3tFRLp27WpyPdv1yJEjppaXl2dy3fPbq1cvU2vVqlWIX3/99TSvGMk0atQoxO3atTO148ePm1zfP9/T6/MzZ86E+Oabbza1t99+O8TPPPOMqc2fP78YVw0AAGLCO70AAACIHodeAAAARK/Ctjdo9evXN7lvYdAraLdv325qVavaf2L79u1DfM8995jaxIkTQ+zX5U6ePNnke/fuvcJV4//pkXK+VWXVqlUm/+KLL0Jct25dU3v11VdN3qJFixCPGjXK1IYPHx7iP/zhD6b2/PPPm/yVV15Jeu0AACAOvNMLAACA6HHoBQAAQPQ49AIAACB6Fbant0+fPiG+/vrrTW316tUmf//990Ps+0BPnjxp8g0bNoR4//79pnbfffeFuG/fvqbme4N1H+jBgweL/gMQNGnSJMR+hfTWrVtNvnbt2mL/vceOHQvx4cOHk9YefvhhU3viiSdMfurUqRC/8cYbxX58AABQefBOLwAAAKLHoRcAAADR49ALAACA6FXYnt5q1aqF2K+qTcX38HoXLlwI8eLFi01N59/4xjdMbcCAASYfOXJkiGfNmmVqO3bsKN7F5ohz586F+KqrrjK1EydOZOUx9u3bZ/Lf//73IfYzlUePHm3y8ePHh/izzz4zteXLl2fl+gAAQPninV4AAABEj0MvAAAAoldh2xvOnDkTYj1SSiS9dodMvfXWWyb3Y9KGDBkSYt2KgaJ0e4FfIe3XEqfSpk0bk/uV08n885//NPnp06dN3q9fvxD36NHD1GhvAAAgDrzTCwAAgOhx6AUAAED0OPQCAAAgehW2p/fSpUshbtu2ram1a9fO5Nu2bQvx+fPnTW3gwIEmX7FiRYh9P6nuN61Tp46pNWvWzOR6RFbLli1Nza/WzXX6a+XHuTVu3NjkN9xwQ4jbt29vatddd53J9eph3/et+3Zr1qxpan4VcsOGDUPsR6rpddT09wIAUHnxTi8AAACix6EXAAAA0UsUFhaW3YMlEsV+sHr16oV40KBBplalij2r64/Pb7vtNlPr37+/yfUoNL996+jRoyG+0mY3zbdU7Ny5M8R+rFZ5bGsrLCxMZPvvTOdeal27djW5bz24ePFiiH07g9+k1rt37xD7jWx6C1w6X3P9+CIie/bsCfHmzZtNbe7cuSY/ePBgsR8nU6VxLwEAyAW80wsAAIDocegFAABA9Dj0AgAAIHoVdmSZXjX8wQcfmJru1xQRqVu3bojz8vJMzY8au/POO0M8YMAAUysoKLjs44sU7fXUfcVVq9ovo+4h9euLZ8yYYfJFixaFWK/rjdWGDRuK/WcPHDhg8ho1aphcjyXr3r27qen73qtXr5SPo8edJRK2ZbZJkyYh9t9bU6dONfnf/va3EPs11gAAoHzxTi8AAACix6EXAAAA0ePQCwAAgOhV2Dm9mfIzfP0cWL0y2M+B1X2gejWtSNF+Ut1X7HuMO3XqFGK/BvnQoUMmX7BgQYh1T6hIev2vqVSkOb0l4e+lXhVdu3ZtU9PzgOvXr29qvgdb9/H63m39PdGzZ09T0/3hIiL6ufTCCy+Y2ksvvWTyEydOSCaY0wsAQGZ4pxcAAADR49ALAACA6EXX3pAt/iNw3zahXbp0yeR6zJUfpTVhwgST63aLpUuXmtrkyZNNnmm7QyztDeXNr1AePny4yfWa5Hbt2pnaz3/+c5M/99xzGV0D7Q0AAGSGd3oBAAAQPQ69AAAAiB6HXgAAAESvwq4hzpQfLab7a0VEdu7cWay/x4+uSodeJ+xX6e7atcvkI0aMCPGtt95qavfff7/JN2/enJXrqyxuuukmk+fn55t84cKFIc50BFg6fE/1F198YfJjx46FeOzYsab2xBNPmHzFihUhXrJkSbYuEQAAJME7vQAAAIgeh14AAABEL4qRZW3atAnxmDFjTE2PBBOxG9BWrlxpajt27Mj+xV1B8+bNQ+xHYg0ePNjk+tpXr15tavv370/6GJV1ZJkf7/bYY4+Z/NSpUyF+9NFHTW3fvn2ld2FJNG3aNMRPPvmkqfnvw8WLF4f41VdfLfZjMLIMAIDM8E4vAAAAosehFwAAANHj0AsAAIDoRTGy7ODBgyH+9NNPTU33zIqIjBo1KsR+7W950OPNjh49amq1atUyeZ06dUL85Zdflu6FVQBr1qwxuR8ZNmTIkBDrFcAiIi+++GLpXVgSuq/6d7/7nakNGzbM5PXr1w+xv89nzpwphasDACC38U4vAAAAosehFwAAANHj0AsAAIDoRdHTe/r06RD7Pt1WrVqZvFmzZiHW/b0iIi+//HKIL126lM1LLJZ27dqZPC8vz+Rt27YNcYsWLUzt+PHjIS7L2culae3atSZ/5513TK5XTH/72982tc8//zzEM2fOLIWrK6pBgwYhvuuuu0ytY8eOJte95/Xq1TO1KlW+/n9RPYsYAABkjnd6AQAAED0OvQAAAIheFO0N2qFDh0zuP9rOz88P8Q9/+ENTu/3220P83nvvmZpeGytS/JXF1apVM7lvt+jdu3eI+/TpY2rXXXedyZctWxZi/7G3HoHlR5/FYvbs2SZv1KhRiEeMGGFqr732WogXLlxoah9++KHJN23aFOITJ04kfXzdviBStB2ldevWIdbj5USKtqro+3Xttdea2rx585JeAwAAyAzv9AIAACB6HHoBAAAQPQ69AAAAiF6iLMdbJRKJcp+l1aFDhxD7HtrHH3/8sn9ORKR27dom37VrV4gvXLhganotsu/trFu3rslr1KgR4j179pia7uEVsX3Gen2xiMju3bslmcLCwkTSYoYqwr1MJL7+Zw0YMMDUBg4cGOJ+/fqZmh8fpvtr/XrnrVu3hrhp06am5u+Bvh7/PbFu3TqT617zOXPmSHGVxr0EACAX8E4vAAAAosehFwAAANHj0AsAAIDo5VxPr+b7bfWK4s6dO5tap06dTK5n6BYUFJianhXsZ7vq9bgiIlu2bAmx7+nVNRGR/fv3hzidWbyx9vSmUqtWrRD7e+dXOOt76efptmzZMsRHjhwxtYsXL5p827ZtIdazf0VENm7caPJU84BToacXAIDM8E4vAAAAosehFwAAANHL6faGdFSpYv//QLc0fPXVV6am2ySOHTtmav7P6vrJkydLfJ2Xk4vtDZnya6P1SDnfznDu3DmTl8VzifYGAAAywzu9AAAAiB6HXgAAAESPQy8AAACiR09vDqCnNx709AIAkBne6QUAAED0OPQCAAAgehx6AQAAED0OvQAAAIgeh14AAABEj0MvAAAAolemI8sAAACA8sA7vQAAAIgeh14AAABEj0MvAAAAosehFwAAANHj0AsAAIDocegFAABA9Dj0AgAAIHocegEAABA9Dr0AAACIHodeAAAARI9DLwAAAKLHoRcAAADR49ALAACA6HHoBQAAQPQ49AIAACB6HHoBAAAQPQ69AAAAiB6HXgAAAERPNJBRAAAAKUlEQVSPQy8AAACix6EXAAAA0ePQCwAAgOhx6AUAAED0OPQCAAAgev8HbexN5J5cB7wAAAAASUVORK5CYII=\n",
      "text/plain": [
       "<Figure size 864x864 with 32 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "# visualize output of each convolution filter for a selected item\n",
    "def visualize_convolution_output(index):\n",
    "    plt.figure(figsize=(12,12));\n",
    "\n",
    "    for n in range(nb_filters):\n",
    "        plt.subplot(6,6,n+1); \n",
    "        plt.imshow(l[index,:,:,n], cmap='gray')\n",
    "        plt.axis('off')\n",
    "\n",
    "# select one item (index) from the dataset\n",
    "item = 18\n",
    "visualize_convolution_output(item)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Exercises\n",
    "\n",
    "1. Play with model's parameters to enhance accuracy:\n",
    "  - number of convolution filters (<code>nb_filters</code>)\n",
    "  - convolution window (<code>kernel_size</code>)\n",
    "\n",
    "1. Try add stride, padding to Convolution and MaxPooling layers\n",
    "\n",
    "1. Add extra Conv2D, Dense, Dropout layers (uncomment them)\n",
    "\n",
    "1. Try to use only Dense layers and achieve comparable accuracy with this CNN model. How many parameters are you using with the respect the CNN one?\n",
    "\n",
    "1. Try to classify images from the\n",
    "   <a href='https://keras.io/datasets/#cifar10-small-image-classification'>CIFAR10</a> using Convolution layers."
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.6"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}