{ "cells": [ { "cell_type": "markdown", "metadata": {}, "source": [ "# Keras Activation functions" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Activation functions are used to modify the output response of the standard dot( input, weights ) operation performed by each node of a neural network layer. \n", "\n", "You can think the output of a node as:
\n", "output = activation( dot( input, weight ) + bias )\n", "\n", "Keras provide several predefined activation functions which can be used to enhance, dump, model the intensity of the basic response of the layer.\n" ] }, { "cell_type": "code", "execution_count": 1, "metadata": {}, "outputs": [], "source": [ "import tensorflow as tf\n", "from tensorflow.keras import backend, activations\n", "import numpy as np\n", "import matplotlib.pyplot as plt\n", "\n", "x = np.arange(-3, 3, 0.1)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "# Activation functions with open bounds\n", "\n", "These activation functions share the property of having no bounds over the range values of output. \n", "These functions are very versatile (most used activation is relu), yet a different response is provided on positive and negative input values." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "The simplest activation function is the linear response f(x) = x, which in practice does not alter the response of the default dot(input , weight) kernel operation of a single node of a layer.\n", "\n", "Another simple open bounds activation functions is the exponential function, which enhances response of high input values over lower ones.\n", "\n", "Other open bounds activation functions are elu, selu, relu, softplus which are later described." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## softplus\n", "\n", "\n", "SOFTPLUS implements the following activation function:
\n", "\n", "f(x) = log( 1 + exp(x) )" ] }, { "cell_type": "code", "execution_count": 2, "metadata": { "scrolled": false }, "outputs": [ { "data": { "image/png": "iVBORw0KGgoAAAANSUhEUgAAAXcAAAD8CAYAAACMwORRAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDMuMC4xLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvDW2N/gAAIABJREFUeJzt3Xl8lOW5//HPlZAQIISdsEoUEAVcIEFErSVVW2pbbY9aF6SiUrRHrXY5x6PtT1ttz9EudtG6UEEFl9Sq7VHUo9ZCFZUtyL7vhH3LMtmX6/dHpqecFMiQTPLMTL7v12tezGTumVwXSb55cs8992PujoiIJJakoAsQEZHoU7iLiCQghbuISAJSuIuIJCCFu4hIAlK4i4gkIIW7iEgCUriLiCQghbuISAJqF9Qn7tmzp2dlZTXpsaWlpXTq1Cm6BQVI/cSuROoFEqufROoFIu8nPz//gLv3amxcYOGelZXF4sWLm/TYuXPnMn78+OgWFCD1E7sSqRdIrH4SqReIvB8z2xbJ82laRkQkATUa7maWZmYLzWyZma0ysx8fZUx7M/uDmW00swVmltUSxYqISGQiOXKvBD7n7mcBZwMTzOzcBmNuBg67+xDgV8DD0S1TRERORKPh7vVC4Zsp4UvDfYIvB54LX38FuMjMLGpViojICYlozt3Mks1sKbAPeM/dFzQY0h/YAeDuNUAR0COahYqISOTsRE7WYWZdgT8Bd7j7yiM+vhKY4O4F4dubgLHufqDB46cCUwEyMzOz8/LymlR0KBQiPT29SY+NReondiVSL5BY/SRSLxB5P7m5ufnuntPoQHc/oQtwH/D9Bh97BxgXvt4OOED4F8exLtnZ2d5Uc+bMafJjY5H6iV2J1It7YvWTSL24R94PsNgjyOpIVsv0Ch+xY2YdgEuAtQ2GvQ7cEL5+JfDXcBEiInKEX/9lPat3Fbf454nkTUx9gefMLJn6OfqX3X22mT1A/W+Q14HpwCwz2wgcAq5psYpFROJU3sLt/PovG6iqqWN4v4wW/VyNhru7LwdGHeXj9x1xvQK4KrqliYgkjiXbD3Pff6/iM0N78r3PD2vxz6d3qIqItLB9xRXcOiufPl3SePTaUSQntfxKcYW7iEgLqqyp5dbn8ympqGHaN7Lp2jG1VT5vYBuHiYi0BT96fTVLthfyu+tGc1qflp1nP5KO3EVEWsiLC7bz0sLtfGv8YL50Zt9W/dwKdxGRFpC/7RD3v76Sz57ai++3wguoDSncRUSibE9RBbc+v4T+XTvw22ta5wXUhjTnLiISRRXVtdzyfD5llTW8MGUsXTqmBFKHwl1EJErcnR/+eSXLdhTy1KRsTs3sHFgtmpYREYmS5z7eyiv5Bdx50VC+MKJPoLUo3EVEouCTTQd58M01XDI8kzsvGhp0OQp3EZHmKjhcxm0vLuHknp145OtnkRTAC6gNKdxFRJqhrKqGqTPzqa6tY9qkbDqnBfMCakMKdxGRJnJ3/u2V5azdU8yj147ilF6xc/IQhbuISBM9PncTby7fzd0TTmP8sN5Bl/N/KNxFRJrg/TV7+cW767j87H5MvfCUoMv5Jwp3EZETtHFfCXfmLWVEvwwevuJMzIJ/AbUhhbuIyAkoKq/mmzPzSUtJ4qlJOaSlJAdd0lHpHaoiIhGqrXPueOlTdhwq48Vvnkv/rh2CLumYFO4iIhF66O01fLB+P//5tTM45+TuQZdzXJqWERGJwKv5Bfz+wy1MOncQ1409KehyGqVwFxFpxNIdhdzzpxWce0p37vvK8KDLiYjCXUTkOPYWVzB15mJ6d27P4xOzSUmOj9jUnLuIyDFUVNcydVY+ocoaXrv5PLp3ap2TW0eDwl1E5CjcnXtfW8GyHYU8eX12q57cOhri4+8LEZFWNu2Dzbz26U6+c/GpTBgZ7N7sTdFouJvZQDObY2arzWyVmd15lDHjzazIzJaGL/e1TLkiIi3vr2v38tD/rOVLZ/Tl2xcNCbqcJolkWqYG+J67LzGzzkC+mb3n7qsbjPvQ3b8c/RJFRFrPhr0lfPulpQzvm8EvrjorJrcWiESjR+7uvtvdl4SvlwBrgP4tXZiISGsrLKtiyszFpKUk8/tv5NAhNTa3FojECc25m1kWMApYcJS7x5nZMjN728xGRKE2EZFWU11bx20vLmF3YQVPTcqmXwxvLRAJc/fIBpqlA38DfururzW4LwOoc/eQmV0K/Mbd/+kkgmY2FZgKkJmZmZ2Xl9ekokOhEOnpsbMpfnOpn9iVSL1AYvUT7V5mra7k/e013Dwylc8MaP2zKUXaT25ubr675zQ60N0bvQApwDvAdyMcvxXoebwx2dnZ3lRz5sxp8mNjkfqJXYnUi3ti9RPNXmZ9stUH3T3bH3xjVdSe80RF2g+w2CPI4UhWyxgwHVjj7o8cY0yf8DjM7Bzqp3sONvqbRUQkYB9vOsCPXl/F+GG9uOfS04MuJ2oiWS1zPjAJWGFmS8Mfuxc4CcDdnwSuBL5lZjVAOXBN+DeMiEjM2nqglG89v4Ssnp347bWjSE6Kz5UxR9NouLv7POC4Hbv7Y8Bj0SpKRKSlFVdUM2XmYsxg+g05ZKS1/jx7S9I7VEWkzamtc+548VO2HijliYnZDOrRKeiSok57y4hIm/Ofb63hb+GTbowb3CPoclqEjtxFpE3JW7id6fO2MPm8rLg46UZTKdxFpM34eNMBfvjnlVx4ai9++KXEWRlzNAp3EWkTthyxMuax60bRLk5OutFUid2diAhQVFbNzc8tIilBV8YcjV5QFZGE9vc9Y3YcKuP5m8cm5MqYo1G4i0hCe+CN1czbeICfXXkmY09JzJUxR6NpGRFJWM9+tIVZ87cx9cJT+HrOwKDLaVUKdxFJSHPW7eOB2au5ZHgmd084LehyWp3CXUQSzro9Jdzx4qec1ieDX199dkLtGRMphbuIJJT9JZXc9OwiOrVPZvrkHDq1b5svLbbNrkUkIVVU1zJ11mIOlVbx8i3j6Nslvs+m1BwKdxFJCO7Ov72ynKU7CnliYjZnDOgSdEmB0rSMiCSEX723njeW7eLuCacxYWSfoMsJnMJdROLeq/kF/PavG7k6ZyC3XHhK0OXEBIW7iMS1+ZsP8h+vLef8IT34yddGEj7jZ5uncBeRuLV5f4hbZuUzqEcnHp+YTUqCbwZ2IvQ/ISJxqaTKuenZRbRLMp6ZPIYuHRJ/M7ATodUyIhJ3KmtqefTTCnaVwEvfPJeB3TsGXVLM0ZG7iMQVd+ffX1nO+sN1/PKqs8ge1C3okmKSwl1E4soj763nv5fu4sqhKXzlrH5BlxOzNC0jInHj5UU7ePSvG7lmzEC+0P1g0OXENB25i0hc+HDDfu790wo+M7QnD35VSx4bo3AXkZi3bk8J//r8Eob0TufxiaO15DECjf4PmdlAM5tjZqvNbJWZ3XmUMWZmvzWzjWa23MxGt0y5ItLW7Cuu4MZnFtIhNZkZk8fQuQ2c/zQaIplzrwG+5+5LzKwzkG9m77n76iPGfBEYGr6MBZ4I/ysi0mShyhpufHYRheXVvHzLOPp1bbu7PJ6oRo/c3X23uy8JXy8B1gD9Gwy7HJjp9eYDXc2sb9SrFZE2o7q2jtteWMLaPSX8buJoRvZv27s8nqgTmrgysyxgFLCgwV39gR1H3C7gn38BiIhExN35f39eyd/W7+cnXx1J7rDeQZcUd8zdIxtolg78Dfipu7/W4L7ZwEPuPi98+33gbndf3GDcVGAqQGZmZnZeXl6Tig6FQqSnpzfpsbFI/cSuROoF4qef1zdV8dqGar5ySgpXnJp61DHx0kukIu0nNzc3391zGh3o7o1egBTgHeC7x7j/KeDaI26vA/oe7zmzs7O9qebMmdPkx8Yi9RO7EqkX9/jo55XFO3zQ3bP9rrxPva6u7pjj4qGXExFpP8BijyC3I1ktY8B0YI27P3KMYa8D3wivmjkXKHL33Y3+ZhEROcJHGw9w96vLOW9wDx6+4kytZW+GSFbLnA9MAlaY2dLwx+4FTgJw9yeBt4BLgY1AGXBj9EsVkUS2elcxt8zKZ3CvdJ6clE1qO61lb45Gw93r59GP++sz/KfCbdEqSkTaloLDZUx+ZiHp7dvxzI1jyNBa9mbT3jIiEqjCsiomP7OI8upa/nir1rJHi/7uEZHAVFTXMuW5xWw/WMa0STmc1icj6JISho7cRSQQtXXOXXlLWbztMI9dN4pxg3sEXVJC0ZG7iLQ6d+fB2av5n1V7+OGXTufLZ2pf9mhTuItIq3t87iae/XgrN19wMlM+c0rQ5SQkhbuItKqXF+3g5++s4/Kz+/GDS08PupyEpXAXkVbzl9V7uSd8wo2fX3kWSUl6k1JLUbiLSKvI33aI215cwoh+GTxxvd6k1NL0vysiLW7D3hJuenYxfbukMWPyGNLba6FeS1O4i0iL2lVYzjdmLCS1XRKzbh5Lz/T2QZfUJijcRaTFHCqtYtL0BYQqanj2xjEM7N4x6JLaDP1tJCItIlRZw43PLKTgcDkzbzqHEf10JqXWpHAXkairrKnl1ln5rNxVzJPXZzP2FL37tLVpWkZEoqq2zvnuH5Yxb+MBHr7iTC4Znhl0SW2Swl1Eosbdue+/V/Lmit384NLTuTJ7QNAltVkKdxGJml++u54XFmzn1s8O5psXaluBICncRSQqnvrbJh6bs5FrzzmJuycMC7qcNk/hLiLN9tLC7fzX22v58pl9+clXR+rcpzFA4S4izfLGsl3c+6cV5A7rxSNfP5tk7RcTExTuItJkc9bu4zt/WMqYrO48PlH7xcQSfSVEpEkWbD7Irc/nc1rfzky/IYcOqclBlyRHULiLyAlbuqOQm59bzIBuHXjuxnPonJYSdEnSgMJdRE7Imt3F3DBjId07pfLClHPpoY3AYpLCXUQitnFfiOufXkDH1GRemDKWPl3Sgi5JjkHhLiIR2XGojOufXoCZ8cKUsdrhMcY1Gu5mNsPM9pnZymPcP97MisxsafhyX/TLFJEg7Smq4Lqn51NRU8vzU87hlF7pQZckjYhkV8hngceAmccZ86G7fzkqFYlITNlXXMF1v5/P4dJqXvzmWE7rkxF0SRKBRo/c3f0D4FAr1CIiMeZAqJKJTy9gT3EFz900hjMHdA26JIlQtObcx5nZMjN728xGROk5RSRAh0uruP7pBew4XMaMyWPIHtQ96JLkBJi7Nz7ILAuY7e4jj3JfBlDn7iEzuxT4jbsPPcbzTAWmAmRmZmbn5eU1qehQKER6euLM+amf2JVIvUDk/ZRWOz9bVMHOUB3fGZ3GiJ6x9waltvq1yc3NzXf3nEYHunujFyALWBnh2K1Az8bGZWdne1PNmTOnyY+NReondiVSL+6R9VNUXuWXPfqhD733LZ+zdm/LF9VEbfFr4+4OLPYIsrjZ0zJm1sfCW8CZ2TnUT/UcbO7zikjrK6moZvKMhazaVczjE0czfljvoEuSJmp0tYyZvQSMB3qaWQFwP5AC4O5PAlcC3zKzGqAcuCb820VE4khJRTU3zFjI8oIiHrtuFBfr9HhxrdFwd/drG7n/MeqXSopInGoY7BNG9g26JGkmvUNVpI0rqahm8jOLFOwJRuEu0ob9PdiX7Sjk0WsV7IlE4S7SRhU3CPYvnqFgTySRbD8gIgmmqKyabzyzkFU7ixTsCUrhLtLGHC6tYtKMBazfE+KJ67O5RKtiEpLCXaQNKa5yrv39fDYfKOWpb2STq3XsCUvhLtJG7Cup4KGF5RyqNGbcMIYLhvYMuiRpQQp3kTZgd1E5E59ewMFy59mbxjJucI+gS5IWpnAXSXDbD5Zx3dPzKSyr5ns5aQr2NkJLIUUS2MZ9JVz11MeEKmt48ZtjObVb7O3uKC1D4S6SoFbtKuLrT82ntg7+MHWcTrTRxijcRRJQ/rbDXDttPmntkvjjreMY1qdz0CVJK9Ocu0iCmbfhAFNnLaZ35/Y8P2UsA7p1DLokCYDCXSSBvL1iN3fmLeWUXp2YedM59M5IC7okCYjCXSRB5C3czr1/WsGok7ox44YxdOmYEnRJEiCFu0gCePJvm3jo7bV89tRePHH9aDqm6ke7rdN3gEgcc3ce+p+1PPW3zVx2Vj9+cdVZpLbTOglRuIvEreraOu55bQWv5Bcw6dxB/PiyESQlWdBlSYxQuIvEodLKGm57cQlz1+3nrouHcudFQwmfp14EULiLxJ0DoUpuenYRK3cW8dC/nME155wUdEkSgxTuInFk28FSbpixkD3FFUyblMPF2otdjkHhLhInlhcUctOzi6ipc16Yci7Zg7oFXZLEMIW7SBz4y+q93PHSp3TvlEreTecwpHd60CVJjFO4i8S45z7eyo/fWMXI/l14+oYcenfWu06lcQp3kRhVW+f89M01zPhoC5cMz+Q315ytNydJxPSdIhKDyqtquTPvU95dvZcbz8/ih18aTrLWsMsJaPStbGY2w8z2mdnKY9xvZvZbM9toZsvNbHT0yxRpO/YWV3D1tE94b81e7v/KcO7/yggFu5ywSN6n/Cww4Tj3fxEYGr5MBZ5oflkibdOKgiIue2wem/aF+P2kHG48/+SgS5I41Wi4u/sHwKHjDLkcmOn15gNdzaxvtAoUaSveWrGbq576mHZJSbzyrfO0hl2axdy98UFmWcBsdx95lPtmAw+5+7zw7feBu9198VHGTqX+6J7MzMzsvLy8JhUdCoVIT0+cpWDqJ3a1Ri/uzhubq3ltQzVDuiZxx6g0urRvmWkYfW1iV6T95Obm5rt7TqMD3b3RC5AFrDzGfbOBC464/T6Q09hzZmdne1PNmTOnyY+NReondrV0L2WVNX7Hi0t80N2z/a68T728qqZFP5++NrEr0n6AxR5BbkdjtcxOYOARtweEPyYix1FwuIxbZuWzencx//aFYfzr+MHa/EuiJhrh/jpwu5nlAWOBInffHYXnFUlYn2w6yG0vLqG6po7pN+TwudM0vy7R1Wi4m9lLwHigp5kVAPcDKQDu/iTwFnApsBEoA25sqWJF4p27M/OTbTwwezVZPToy7Rs5DO6VOPPGEjsaDXd3v7aR+x24LWoViSSoiupafvjnlbySX8DFp2fyq6vPonOaznMqLUPvUBVpBdsPlnHr8/Xz69++aCh3XTRUZ02SFqVwF2lhf1m9l+++vBQzY8Zkza9L61C4i7SQ2jrnkffW8bs5mxjZP4MnJmYzsHvHoMuSNkLhLtIC9pdUctcfPuWjjQe5ZsxAfnTZCNJSkoMuS9oQhbtIlM3bcIC7/rCUkopqfnbFmXx9zMDGHyQSZQp3kSipqa3jN+9v4LE5GxncK50XpoxlWJ/OQZclbZTCXSQKdheVc+dLS1m49RBXZQ/gx5eP0Ik1JFD67hNppndX7eHuV5dTWVPHr64+i6+NGhB0SSIKd5GmKquq4SdvruHFBdsZ3jeDR68bpXebSsxQuIs0wYqCIu7M+5QtB0u55bOn8L1LhpHaLpJz34i0DoW7yAmorXOe+mATj7y7np7p7XlhyljOG9wz6LJE/onCXSRCWw+U8v0/LmPxtsN86Yy+/PRrI+naMTXoskSOSuEu0oi6OmfW/G089PZa2iUbv7zqLP5ldH/tvS4xTeEuchz7y+qY+PQCPtl8kAtP7cXDV5xB3y4dgi5LpFEKd5GjqKtzXlq0nQc/Kic5uYr/+pczuGbMQB2tS9xQuIs0sGl/iHteW8HCLYc4vXsS06ZcqA2/JO4o3EXCqmvrmPbBZn7z/gbS2iXxsyvOpFdoo4Jd4pLCXQRYXlDI3a+uYM3uYi49ow8/+soIemekMXfupqBLE2kShbu0aUXl1fzinXU8v2AbvdLb89SkbL4wok/QZYk0m8Jd2iR3589Ld/LTN9dwqLSKG8Zl8d3Pn0qGzmkqCULhLm3Ohr0l/PDPK1mw5RBnDezKszeew8j+XYIuSySqFO7SZhSVVfPr99cz85NtpLdvx39+rX55o05ULYlI4S4Jr7bOeWnhdn757joKy6u5ZsxJfP/zp9IjvX3QpYm0GIW7JLSPNx3ggTdWs3ZPCWNP7s59XxnOiH6agpHEp3CXhLR+bwkPv72W99fuo3/XDjw+cTRfHNlH7zCVNiOicDezCcBvgGTgaXd/qMH9k4GfAzvDH3rM3Z+OYp0iEdlTVMGv3lvPH/N30Kl9O+6ecBo3np9FWkpy0KWJtKpGw93MkoHfAZcABcAiM3vd3Vc3GPoHd7+9BWoUaVRReTXTPtjE9HlbqKuDG88/mdtzh9Ctk7bklbYpkiP3c4CN7r4ZwMzygMuBhuEu0upClTU8+9EWpn2wmeKKGi4/ux/f//wwbRkgbZ65+/EHmF0JTHD3KeHbk4CxRx6lh6dl/gvYD6wHvuPuO47yXFOBqQCZmZnZeXl5TSo6FAqRnp4456pUPyeustb56/Ya3txcRagaRvVO5qtDUhiUEd3pF31tYlci9QKR95Obm5vv7jmNjYvWC6pvAC+5e6WZ3QI8B3yu4SB3nwZMA8jJyfHx48c36ZPNnTuXpj42FqmfyJVV1fDigu089dFm9pdUceGpvfjuJady9sCuLfL59LWJXYnUC0S/n0jCfScw8IjbA/jHC6cAuPvBI24+Dfys+aWJ/ENxRTUzP97K9HlbOFxWzbhTevD4xNGMyeoedGkiMSmScF8EDDWzk6kP9WuA644cYGZ93X13+OZlwJqoVilt1sFQJc98tJXnPtlKSUUNucN6cfvnhpA9SKEucjyNhru715jZ7cA71C+FnOHuq8zsAWCxu78OfNvMLgNqgEPA5BasWdqAzftDTJ+3hVfyC6iqreOLI/vwr+OHaA8YkQhFNOfu7m8BbzX42H1HXL8HuCe6pUlb4+7kbzvMtA82896avaQkJ3HF6P7cfMEpDOmdOC+cibQGvUNVAldZU8uby3fz3MdbWVZQRNeOKdyRO4RJ47Lo1Vn7v4g0hcJdArOrsJwXFmwjb+EODpZWMbhXJx68fARXZg+kQ6reUSrSHAp3aVW1dc6HG/aTt3AH763Zi7tz0emZTD4vi/MG99DeLyJRonCXVrG7qJyXFxXw8uId7Cwsp3unVKZ85mSuHztI7yYVaQEKd2kxFdW1vL9mH68uKWDuun3UOVwwpCf3Xno6lwzPJLVdUtAliiQshbtElbuz4XAt77y2gtnLd1FSUUOfjDS+NX4wV+ecxEk9dJQu0hoU7tJs7s66vSXMXrabN5bvYtvBCjqk7GTCyD5cMXoA4wb3IFmnshNpVQp3abLN+0O8sWw3s5fvYsO+EEkG5w3uySX9arjrylzS2+vbSyQo+umTiLk7q3YV8+6qPbyzai/r9pZgBmOyuvPg5SOYMLIvvTq3Z+7cuQp2kYDpJ1COq7q2jsVbD/OXNXt5Z9UeCg6Xk2SQk9Wd//fl4XzpjL706ZIWdJki0oDCXf7JwVAlc9ft569r9/HBhv2UVNSQmpzEBUN7csfnhnDx6Zn0SNc7R0VimcJdqKqpY8n2w8zbcIAPN+xn+c4i3KF35/ZcOrIvuaf15oKhPTXVIhJH9NPaBtXVOWv3lPDJ5oN8tPEA8zcfpKyqluQkY9TArnzn4lP53Gm9Gd43gyStchGJSwr3NqC2zlm7p5gFmw8xf/NBFmw5RFF5NQBZPTpyxegBfGZoT84d3IOMtJSAqxWRaFC4J6DSyhqW7ihk0dZD5G87zKfbCwlV1gAwqEdHJozow7mDuzP25B7069oh4GpFpCUo3ONcTW0dG/aFWLqjkGU7Clm6o5D1e0uoczCDYZmd+eqofuQM6s6Yk7vTX2Eu0iYo3ONIdW0dG/aGWLmriFU7i1i5q5jVu4opr64FoGvHFM4a0JXPD88kO6s7o07qqmkWkTZK4R6jDoQqWbu7hLV7ilkT/nfD3hBVtXUAdEpNZkS/Llw9ZiBnD+zK2QO7MqhHR22ZKyKAwj1Q7s6+kkpWH6xl28db2bCvhA17Q2zcF+JgadX/juvduT3D+nRm8vlZjOzfhZH9Msjq0UkrWUTkmBTuLczdKSyrZtuhMrYdLGXLgVI27y9l84EQW/aXUlpVGx65is5p7Tg1szOXDM9kSO90hvfNYFifznrDkIicMIV7FJRX1bKzsIwdh8spOFxOweEyCg6Vs+1QKdsOllFSUfO/Y82gf9cOnNyzEzk53Tm5ZydCuzZx1efPp1fn9ppWEZGoULg3oryqlj3FFewpqmBvcQV7iivYXVjOzsIKdheVs7uogkNHTKEApCYn0b9bBwZ278iogd0Y1KMjg3p0YlCPjpzUvSNpKf/3/KBz526ld4b2ZxGR6GmT4V5dW8fh0ioOhKo4EKrkYGklB0qq2FdSwf6SSvaVVLK/pJK9xRUUH3HU/Xed09rRr0sH+nVN46yBXenXJY2B3TsyoFsHBnTrSK/09poPF5FAxXW4uzvl1bUUlVdTVF5NYdnfL1UUlldzuKyKwtJqDpVVcai0isOlVRwqq6KwrPqoz9e+XRK9M9rTu3Mag3ulM25wDzIz0uiTkUafLmlkZrQnMyONzlpeKCIxLu7Cfc66fdzzYRk1896jqLya6lo/5tiUZKN7p1S6dUyle6dUhvfLoFvHVHqkp9IzvT0901Ppkd6enunt6ZGeSuf27TTnLSIJIaJwN7MJwG+AZOBpd3+owf3tgZlANnAQuNrdt0a31HpdOqQwsHMSQ07qQ5cOKf/n0rXj3y+pdOuYQoeUZIW1iLRJjYa7mSUDvwMuAQqARWb2uruvPmLYzcBhdx9iZtcADwNXt0TBo0/qxm1npzF+/Bkt8fQiIgkhKYIx5wAb3X2zu1cBecDlDcZcDjwXvv4KcJHpkFlEJDDmfuw5awAzuxKY4O5TwrcnAWPd/fYjxqwMjykI394UHnOgwXNNBaYCZGZmZufl5TWp6FAoRHp6epMeG4vUT+xKpF4gsfpJpF4g8n5yc3Pz3T2nsXGt+oKqu08DpgHk5OT4+PHjm/Q8c+fOpamPjUXqJ3YlUi+QWP0kUi8Q/X4imZbZCQw84vaA8MeOOsbM2gFdqH9hVUREAhBJuC8ChprZyWaWClwDvN5gzOvADeHrVwJ/9cbme0REpMU0Oi3j7jVmdjvwDvVLIWe4+yozewBY7O6vA9OBWWZF5ZPoAAADeklEQVS2EThE/S8AEREJSERz7u7+FvBWg4/dd8T1CuCq6JYmIiJNFcm0jIiIxJlGl0K22Cc22w9sa+LDewIHGh0VP9RP7EqkXiCx+kmkXiDyfga5e6/GBgUW7s1hZosjWecZL9RP7EqkXiCx+kmkXiD6/WhaRkQkASncRUQSULyG+7SgC4gy9RO7EqkXSKx+EqkXiHI/cTnnLiIixxevR+4iInIccRvuZvagmS03s6Vm9q6Z9Qu6puYws5+b2dpwT38ys65B19RUZnaVma0yszozi9vVDGY2wczWmdlGM/uPoOtpDjObYWb7wju4xjUzG2hmc8xsdfj77M6ga2oOM0szs4Vmtizcz4+j8rzxOi1jZhnuXhy+/m1guLvfGnBZTWZmn6d+T54aM3sYwN3vDrisJjGz04E64Cng++6+OOCSTlj4JDXrOeIkNcC1DU5SEzfM7EIgBMx095FB19McZtYX6OvuS8ysM5APfDWOvzYGdHL3kJmlAPOAO919fnOeN26P3P8e7GGdgPj8LRXm7u+6e0345nzqd9+MS+6+xt3XBV1HM0Vykpq44e4fUL/vU9xz993uviR8vQRYA/QPtqqm83qh8M2U8KXZeRa34Q5gZj81sx3AROC+xsbHkZuAt4Muoo3rD+w44nYBcRwgicrMsoBRwIJgK2keM0s2s6XAPuA9d292PzEd7mb2FzNbeZTL5QDu/gN3Hwi8ANx+/GcLXmP9hMf8AKihvqeYFUkvIi3JzNKBV4G7GvwlH3fcvdbdz6b+L/ZzzKzZU2eteiamE+XuF0c49AXqd628vwXLabbG+jGzycCXgYtifT/8E/jaxKtITlIjAQnPTb8KvODurwVdT7S4e6GZzQEmAM168Tumj9yPx8yGHnHzcmBtULVEg5lNAP4duMzdy4KuRyI6SY0EIPwC5HRgjbs/EnQ9zWVmvf6+Os7MOlD/In6z8yyeV8u8CgyjflXGNuBWd4/bI6vwiU7a84/TE86P19U/ZvY14FGgF1AILHX3LwRb1Ykzs0uBX/OPk9T8NOCSmszMXgLGU7/z4F7gfnefHmhRTWRmFwAfAiuo//kHuDd83om4Y2ZnAs9R/32WBLzs7g80+3njNdxFROTY4nZaRkREjk3hLiKSgBTuIiIJSOEuIpKAFO4iIglI4S4ikoAU7iIiCUjhLiKSgP4/EKfcflE5uikAAAAASUVORK5CYII=\n", "text/plain": [ "
" ] }, "metadata": { "needs_background": "light" }, "output_type": "display_data" } ], "source": [ "plt.plot(x, backend.get_value(activations.softplus(x)))\n", "plt.grid(True); plt.show()" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## elu\n", "\n", "\n", "ELU stands for exponential linear unit. The exponential linear activation:
\n", "\n", "x , if x > 0
\n", "alpha * (exp(x)-1) , if x < 0. # default: alpha=1.0
" ] }, { "cell_type": "code", "execution_count": 3, "metadata": { "scrolled": true }, "outputs": [ { "data": { "image/png": "iVBORw0KGgoAAAANSUhEUgAAAXYAAAD8CAYAAABjAo9vAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDMuMC4xLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvDW2N/gAAIABJREFUeJzt3Xl4U1X+x/H3aZumO11oC6WFguybS4usYosiKAoC7rsizPgbHXWGkU1lFGUZx20cF0RxQaUuCFTFBZQCLiDLsJe1tLSFAi20JV3SND2/P1IVFKUkKTdJv6/nuc9N0ntvvt+n8OFwcpIorTVCCCF8h5/RBQghhHAvCXYhhPAxEuxCCOFjJNiFEMLHSLALIYSPkWAXQggfI8EuhBA+RoJdCCF8jAS7EEL4mAAjnrR58+Y6OTnZqXMrKioIDQ11b0EG8qV+fKkXkH48mS/1Ag3vZ/369cVa69jTHWdIsCcnJ7Nu3Tqnzs3KyiItLc29BRnIl/rxpV5A+vFkvtQLNLwfpVReQ67n8lSMUipIKfWjUmqTUmqbUuoxV68phBDCee4YsVuBQVpri1LKBHyrlPpca73aDdcWQghxhlwOdu34eEhL/V1T/SYfGSmEEAZxy6oYpZS/UmojcBhYqrVe447rCiGEOHPKnZ/HrpSKBBYC92mtt/7qZ+OAcQDx8fEpGRkZTj2HxWIhLCzM1VI9hi/140u9gPTjyXypF2h4P+np6eu11qmnPVBr7dYNeBQY/0fHpKSkaGctX77c6XM9kS/140u9aC39eDJf6kXrhvcDrNMNyGF3rIqJrR+po5QKBgYDO1y9rhBCCOe4Y469JbBcKbUZWItjjv1TN1xXCCF8RlFxPg/MuZSi4vxGfy6Xg11rvVlrfb7WuqfWurvW+nF3FCaEEL7i241LuHPR5WSZivh89ZuN/nyGvPNUCCGaitcypzKn5CMC/WBK4j1ce+m9jf6cEuxCCNEIamttPDLvGj71y+Ecmz+Pp71Cz479zspzS7ALIYSbHTiSx4SPR7MxyEp/azNm3riIyPDmZ+355WN7hRDCjVZtyOSuxcPYaq7mpoAUXhqz4qyGOsiIXQgh3Oa1zEeZU7IAsx9MSfo/rrnkL4bUIcEuhBAuqq218fDbo/nMfx/tbf5MGzSH7u17G1aPBLsQQrig4HAukxZeUz+fHsmsmxbTLCza0Jpkjl0IIZy0akMmYzKvZKu5mptNF/LSmCzDQx1kxC6EEE6ZvWgyc48txqzg0Tb3MTL9z0aX9DMJdiGEOAM1NVYenjeKzwP208Hmz7RL5tLtnNN/4OLZJMEuhBANlF+Uw8TF17I5qIaLrNHMunkx4aGRRpf1GzLHLoQQDZC1biFjPh3OdrOVW019eGncCo8MdZARuxBCnNbLCyfxRmkmwQqmtvkrV6f/yeiS/pAEuxBC/I6aGitT5o3ii4D9dLQF8MSlc+nSLsXosk5Lgl0IIU4h/+BuJmbewOagGi6uiWHGzYs8durl12SOXQghfuWbHz9izGcjyTZbuS2wH/8dm+U1oQ4yYhdCiJO89PFDvFG2hFCl+We7vzF84BijSzpjEuxCCIFjPn3y21fzpamATjZ/pg95i45tzjO6LKdIsAshmry8A7uY9MkNbAmykVbTnJm3ZBIaEm50WU6TOXYhRJO2bM2H3L1kFDvMNdxhHsALY5d7daiDjNiFEE3YfxeM583yLwhXmmnt/8GwAXcYXZJbSLALIZqcamslU+aN5CvTATrbAnhyyNt0bNPT6LLcRoJdCNGk7CvcweTPbmSruZZBNXFMv2WR10+9/JrMsQshmoyvfpjP2M9Hs8tk486gi3l+7Nc+F+ogI3YhRBPxwocP8pZlKREKpnV4iCsG3GZ0SY1Ggl0I4dOqrZVMmnc1y0wH6WILYPrQd2jfurvRZTUqCXYhhM/Kyd/GpM9vYbu5lkE18cy4bTEhQaFGl9XoZI5dCOGTvvzhPcZ+dR17TDbGBKfx/NhlTSLUQUbsQggf9PyH9zPP8jURwBMdJ3B5/1uNLumskmAXQviMyuoKJs8bwdeBh5rMfPqpSLALIXzCnv1bmfzFLWSb7Vxqa8mTty1sMlMvvybBLoTwep9/N4+ndsyk3KQYG3Ipf732OaNLMpQEuxDCqz33wX3Mq1hOlIYnO09mSN+bjC7JcC4Hu1IqCXgbiAc08KrW+nlXryuEEH+ksrqCSfOG803gYbrZTMwYNp+2rTobXZZHcMeIvRb4u9Z6g1IqHFivlFqqtd7uhmsLIcRvHCnL5/a3H2CH2c5gWwLTb19IkDnE6LI8hsvBrrU+CBysv31cKZUNtAIk2IUQbrfk27d5uXgW5SbFuNDLuO+aZ4wuyeMorbX7LqZUMrAS6K61Lv/Vz8YB4wDi4+NTMjIynHoOi8VCWFiYa4V6EF/qx5d6AenHEy3f9RqfmjYSZddcE3ILnRP6Gl2SWzT0d5Oenr5ea5162gO11m7ZgDBgPTDqdMempKRoZy1fvtzpcz2RL/XjS71oLf14EktFub7v1XTd/c3u+obZ5+kFme8aXZJbNfR3A6zTDchjt3ykgFLKBCwA3tVaf+yOawohBMCuvI3c8c5FLA88whBbIm/c/h3R4QlGl+XR3LEqRgGvA9laa5nsEkK4zScr5/LM7qc5blL8Ofxy/jLqKaNL8gruWBXTH7gV2KKU2lj/2GSt9RI3XFsI0UQ9nXEP71avIkYrZnadyqW9rzW6JK/hjlUx3wLKDbUIIQQVlceZ8O5wVgQW08MayIyrMmiT0NHosryKvPNUCOExdu77H1OW3sFOcx1Da5N48s6FBAaajS7L60iwCyE8wuIVr/HsnmepMCn+L3wY94yaZXRJXkuCXQhhuH/NH8d86/fEasXU7o+R3mu00SV5NQl2IYRhjleUMvG9EawMPEpPq5mZwzNIatnB6LK8ngS7EMIQ2TnreXjZXewy13F5bWueuPNjmU93Ewl2IcRZt2j5bJ7N+Q9VJrg3Yjh/GjnD6JJ8igS7EOKsqbPbeer9P5FRs5o4rXi855NcnDLC6LJ8jgS7EOKsKLMcZdJ7V7PKfIxzrWZmjPiQpBbtjC7LJ0mwCyEa3ba963jk67vYbdYMs7fl8Ts/lPn0RiTBLoRoVAu+eYnn972I1QT3RV7NuBFPGl2Sz5NgF0I0ijq7nX9ljOV924/EacWT507noguGG11WkyDBLoRwuzLLUSa8N4LvzKWcaw1i5siPSIxLNrqsJkOCXQjhVlv3rOGRb8ayx6y50t6WaWMWEBBgMrqsJkWCXQjhNh99/SL/yXsJqwnujxrN3cMfN7qkJkmCXQjhsjq7nVkZY/jAto74OsX0lJkMOP9Ko8tqsiTYhRAuOXE+/fyaYGaO/IiE2DZGl9WkSbALIZy2efdqHs0ax16z5qq69jx+1wcyn+4BJNiFEE75YOl/eCF/NrYAeDD6Ou66aqrRJYl6EuxCiDNSZ7czY/5dfFi7npZ1ikd6PUW/cy83uixxAgl2IUSDlR4v5qH5I/jBXM4F1mBmjf6YFs2TjC5L/Iqf0QUIIbzDxp3fcntGOqsDyxhe14HXx3wvoe6hZMQuhDit95c+xwv5c6gNgAeb38CdVz5idEniD0iwCyF+V53dzpPv3c4C+0YS7IpHev+bvj2HGl2WOA0JdiHEKR0rO8JD749gtfk4KdYQZl2zkPiYVkaXJRpA5tiFEL+xYccqbv9gEGsCy7lad+a1Md9JqHsRGbELIU7y3pdP81LhXOz+8PfYm7h92BSjSxJnSIJdCAE45tOfePdWFtRtJtHux9Q+z3Jhj0uNLks4QYJdCMGxsiP8I2M4a4IspFpD+dd1C4mNSjC6LOEkmWMXoonbsH0Ft30wiB/NxxlJF16/+3sJdS8nI3YhmrD3vnyKFwvfRPvDP+Ju5dYrJhpdknADCXYhmqA6u53H37mZhXorSXY/pvZ7jl7dLjG6LOEmEuxCNDElpUU89P7V/BhUQS9rGLOu+1imXnyMW+bYlVJzlVKHlVJb3XE9IUTjWLvta27/cDDrzBZG043X7v5OQt0HuWvE/ibwX+BtN11PCOFma3MWsTBnKfjDQy1u5+ahDxldkmgkbgl2rfVKpVSyO64lhHCvOrudf75zI4v8ttPG5sfUfi+Q2i3N6LJEI1Jaa/dcyBHsn2qtu//Oz8cB4wDi4+NTMjIynHoei8VCWFiYk1V6Hl/qx5d6Ad/ox1J1jA9zp7MhpJrUimCuaTuF0OBmRpflMl/43Zyoof2kp6ev11qnnu64s/biqdb6VeBVgNTUVJ2WlubUdbKysnD2XE/kS/34Ui/g/f38uGUZT61+lIJgzTWqBxd1uptBl/jGyhdv/938mrv7kTcoCeGD3l4ynQfW3k+pfx0TWt7J1Nvm4+fvb3RZ4iyR5Y5C+JA6u52p825gMdkk2/34Z///ckHXi40uS5xl7lruOB/4AeiklCpQSo1xx3WFEA13qKSQMa/1Y5HawYXWcN667hsJ9SbKXatibnTHdYQQzlm95SseX/03Cs1wrerJw3fPk6mXJkymYoTwcm9+No3ZhzLw94dJCXdzw2UPGl2SMJgEuxBeqrbWxj/n3UCm2klbux+PDXyZ8zoNMLos4QEk2IXwQkXF+UxcMJr1QVX0sUbwr+sXE9Us1uiyhIeQ5Y5CeJnvN33OnYsuZ6O5kuv9zmP2mFUS6uIkMmIXwovM/eRxXi1+nwA/mNRqLNcPfsDokoQHkmAXwgvU1tqYOu86Mv320K7Wj2kXz6Znx35GlyU8lAS7EB6uqDifCQtGsSGomn7WZsy6cRGR4c2NLkt4MJljF8KDfbtxCXcuupxN5ipu8L+Al8eskFAXpyUjdiE81GuZU5lT8hGBfjAl8R6uvfReo0sSXkKCXQgPU1tr45F51/CpXw7tbf48lv4qPTv0Mbos4UUk2IXwIAeO5DHh49FsDLLS3xrJrJsW0yws2uiyhJeROXYhPMSqDZncuXgYW83V3BSQyktjsiTUhVNkxC6EB3h18SO8fvRjzH7wcOu/MHrQ/xldkvBiEuxCGKimxsqj71zLZ/776GDz5/FBc+jevrfRZQkvJ8EuhEEKDucyceE1bAqyMsAayUyZTxduInPsQhhgxfrFjMm8km3mam42XciLMp8u3EhG7EKcZbMXTeb1Y4sJVjC1zV+5Ov1PRpckfIwEuxBnSU2NlYfnjeLzgP10tAXwxKVz6dIuxeiyhA+SYBfiLMgvymHi4mvZHFTDwJpoZt68mPDQSKPLEj5K5tiFaGRZ6xYy5tPhbDdbudXUhxfHrpBQF41KRuxCNKKXF07ijdJMQhT8s+2DjLj4bqNLEk2ABLsQjaCmxsqUeaP4QubThQEk2IVws/yDu5mYeQObg2q4uCaGWbd8QmhIuNFliSZE5tiFcKNvfvyIMZ+NZLvZym2B/fjv2CwJdXHWyYhdCDd56eOHeKNsCaFKy3y6MJQEuxAuqrZW8vC80XxpKqCTzZ8nB79Jp7bnG12WaMIk2IVwQd6BXUz89Aa2mm2k1TRn5i2ZMvUiDCdz7EI4admaDxmzZCQ7A2u4wzyAF8Yul1AXHkFG7EI44b8LxvNm+ReEK5jWfjzDBtxhdElC/EyCXYgzUG2tZMq8kXxlOkBnWwBPDnmbjm16Gl2WECeRYBeigfYV7mDyZzey1VzLoJo4pt+ySKZehEeSOXYhGuCrH+Yz9vPR7DLZuDPoYp4f+7WEuvBYbhmxK6WGAs8D/sBrWuuZ7riuEJ7ghQ8f5C3LUiIUTOvwEFcMuM3okoT4Qy4Hu1LKH3gRGAwUAGuVUpla6+2uXlsII9lqrTz42mUsMx2kiy2A6UPfoX3r7kaXJcRpuWPEfiGwR2udA6CUygBGABLswmvl5G9jzt4JZAfZuaQmnum3LSYkKNTosoRoEHcEeysg/4T7BYB8zbrwPlpDwVoKV77Gg5WrKAj0Z0zwIB64/T9GVybEGVFaa9cuoNQ1wFCt9d31928Femut7/3VceOAcQDx8fEpGRkZTj2fxWIhLCzMpZo9iS/14629BFpLiD+URYuibwitLODfkTG8FRXKDfoK+idfbnR5buOtv59T8aVeoOH9pKenr9dap57uOHeM2AuBpBPuJ9Y/dhKt9avAqwCpqak6LS3NqSfLysrC2XM9kS/141W9VB6F7EzYugByvwVdx56g7jzl14/vI3/kiuQr6a8He08/DeBVv5/T8KVewP39uCPY1wIdlFJtcQT6DcBNbriuEO5VdQx2fgHbPoa930BdLcS05/D59/HA9o78WB5BYo9XifWPYUrfCWz4foPRFQvhFJeDXWtdq5S6F/gSx3LHuVrrbS5XJoQ7lO6HHUtg52eQ+x1oOzRLgr5/wd51FHN2h/HM0t1EhwZyw2UbWZyXxwuDXiAiMMLoyoVwmlvWsWutlwBL3HEtIVxir4WCtbBnGez+Eoq2OB6P7Qz974fOw6BVCnlHKxn/4SbW5hYytFsLxgwKYtzX7zCs3TDSktIMbUEIV8lHCgjvdywXclY4wjxnBVjLQPlD0oUweJojzGPOAUBrzfwf83nis+34+ymevf5chvWM45YltxBhjmBir4nG9iKEG0iwC+9TVgi5q2DfKti3Esr2Ox6PaAXdRkD7wdB2IARHnnRaYWkVUxZuIWvnEfq3j+Gpa84lITKYOZvnkH00m2fSniEyKPIUTyiEd5FgF56trg6OZMP+1ZC/xrEvzXP8LDgK2vSHfvdB24sc0y1K/eYStfY63vgul2eX7UJr+OdVXbmtbzJ+forskmxe3vQyg9sMZnCbwWe5OSEahwS78CzHi6BwAxzY4NgXroPqMsfPQmMhqTdcOM4xIo/vDn5//Dl2/9t/jMkLt5J9sJxLOsfx2IhuJEaFAFBpq+ShlQ8RZY7ikT6PNHZnQpw1EuzCGFpDeSEc3Ox4gbNoMxz4n+MxcMyRx3WBrldD6z6OQI9ud8oR+amUVtbw9Fe7eGdNHvHhQbxyywUM6dYCdcL5M36cQV55Hq8PeZ2ooKjG6FIIQ0iwi8ZXXQaHd8Dh7XCkfl+0FaqO1h+gHKHdui+0ugBapUCLnhAYcsZPZbPXMe+HPJ7/ejfHq23c0S+Zv1/WiTDzyX/Ul+QsYdGeRYzrOY5eLXq5oUkhPIcEu3CPujrM1cWwdzkU74aS3VC8y3G7/IQ3IptCIbYTdLkKWvRwBHh8NzC79vZwrTXLsg8zY0k2OcUVDGjfnCnDutCl5W/Xo+cfz2fa6mmcF3se95x7j0vPK4QnkmAXDWerhrICKM2FY3lwNAeO7nPsj+XSt7YKVtcfGxgOzTtA8gDHlEpsF8e+WdJp58XP1Kb8UmZ+voMfcko4JzaUN+7oRVqn2JOmXX5uoc7GxJUTUShmDZxFgJ/8FRC+R/5UCwetHZ+fUl7gWE5YXghl+Y7bZfmOd3AeP3jyOQFBENXWMY3S/hJ2FdfSsd8waN4RwuIbPB/urK2FZTy3bBfLsg8TFWLi8RHduPHC1pj8f/8fjhf/9yKbizfz74v/TUJYQqPWJ4RRJNh9nb0WKkvAcggsh+v39dvxg45VKD/t7TUnn+sfCBEJjlH2OZdAZGuIauPYR7aB8JYnjb4PZGXRse3ARm9pR1E5zy3dzRfbiogICmD8ZR25o3/b38yj/9rKgpXM3TqX0R1GMyR5SKPXKYRRJNi9ib3W8UJk1bFftsoSx4uQlSWOEXdlMVSUQMURx+2qY6e+VmA4hLeAiJaOFy3DWzq2iARolujYQpq7fdrEFf/bf4zZK3L4YlsR4eYA7r+kA3cNaEuzYNNpz91Xto8JKyfQObozEy6ccBaqFcI4Euxni90G1uOOrcYCVgvUHCf28BpYn1f/s3LHvrrshK20PszLHG+V/z3KH0KiITjasd47vhuENnfcDolxhHhYPITFQWicUytOjFBXp8nadZhXVuTw476jRAQFcG96e+6+qC2RIYENuoalxsL9y+/H5GfiufTnCA4IbuSqhTBW0w52rR3TD7XW+q36hH39ZquG2qqT97ZKsFU57tdU1t+v/OV2TYVjs1U6Qrym4rfTHPW6wclfIhgYDkHNftkiEiGum+N2SLTj3ZbBURAU6diHRDuC2xzhUaNrV1Xb7GRuOsBrq3LYdchCy2ZBPDysCzdc2Pq0Uy4nqtN1TPp2EvvL9zPnsjkyry6aBO8K9h2f0W7vh1D9pWMEXGdz7O02R3D+vD/xtvWX27X1938KcrvVtXoCgsAU7FjCFxjyy+3gKIhMgsAwCAx1bKZQMIc7lvUFhtXvw1m7eQe9LrrEEcyBYT4Vzs7IK6ngndV5fLCugLIqG51bhPPMdedy1bkJf/ii6O95ZdMrZOVnMfHCibJeXTQZ3hXsOStoVfgZHDKDfwD4mRwv8PkH1O8Dwd/0y21Ts18eCzD/8nhAEAQEgr/ZsQ8Idvw8IOiXn/30mCn4l8dNQY6A/ukxN4RwRU6VYz67Cau115G18wjvrMljxa4j+CnF0G4tuKVPG/q0iz7lssWG+Hr/17y86WWGnzOcmzrLd7+IpsO7gv2Kf7Eq5Aqf+kqspmz3oeN8uL6AjzcUUmyxEhdu5q+DOnBT79bERwS5du1ju5m8ajLdY7rzaN9Hnf7HQQhv5F3BLrxeicXKki0H+Wh9AZsKygjwU6R3juPalETSO8c5Nd3ya0UVRfx52Z8JNYXybPqzmP3NbqhcCO8hwS4aXVmljS+3FfHJ5gN8v7cEe52mc4twHh7WhavPb0XzMPcFb3lNOfcsu4cKWwVvDX2LFqEt3HZtIbyFBLtoFMUWK99kH+bLbUWs3H0Em13TOjqEPw1sx5U9E+jSMtzt0yM19hoeWP4AueW5vHzpy3SK7uTW6wvhLSTYhdsUVdQxe8Velm4/xPr9x9AaWkUGc0e/ZK7smUDPxGaNNtddp+uY8u0U1hatZcZFM+jTsk+jPI8Q3kCCXTitwlrLD3tLWLn7CCt2HSGvpArYQdeWEfx1UAcu6xZP15YRZ+WFy2fWPcMXuV/wwAUPcGW7Kxv9+YTwZBLsosFs9jq2FJbxw94SvttTzNrco9jsmmCTP33PieGiuFr+PLz/z99QdLa8sfUN3tr+Fjd2vpG7ut91Vp9bCE8kwS5+V01tHdsOlLFm31F+2FvCutyjVNTYAejcIpy7+rdlYMdYUpOjMAf4k5WVddZDfd72eTyz/hmGJg9lQq8JsqxRCCTYxQmOVtSwIe8Y6/KOsSHvGJsKSrHW1gHQPi6MURck0vecGHq3jSbGjStZnPVe9nv8a+2/GNxmMNMvmo6/n7/RJQnhESTYmyiLtZYtBWVsLihlc0EZmwpKKThWBYDJX9EtoRm39mlDSpsoUpKjiAt37Q1D7vbBzg+Y8eMM0pPSmTVwFia/03/CoxBNhQS7j9NaU1ReTfbBcrYfKGd7/T63pPLnYxKjgjk3MZJb+7Th/NZR9ExsRpDJc0e/C3YtYNrqaVyceDFPX/y0hLoQvyLB7iO01hw+bmXvYQu7Dh1n5yELuw8dZ+eh4xyvrv35uDYxIXRtGcGoCxLpkdiMnq2aecS0SkMt3L2Qx354jP6t+vNM2jOY/CXUhfg1CXYvU1ZpI+9oBbklleQWV5BzxEJOcQU5RyqwWH8J8GbBJjrFhzP83AQ6xofTNSGCzi3CCQ/y3iB8c+ubPL3+afol9OO5tOcI9G/Y57EL0dRIsHuYapudwtIqCo5VUXCssn5fxf6jleSVVFBaaTvp+FaRwbSLDWX0Ba04Jy6Mds3D6BgfRmy42WdWiGiteW7Dc8zdOpchyUOYPmC6hLoQf0CC/SyyWGspKqvmcHk1ReXVHCq3sm67lXfy1nGwrIqDZdUcrTj5CzlM/opWkcEkRoUwrEdLkmNCaR0TQpuYENpEhxIc6Llz4e5gr7MzbfU0FuxewHUdr2Ny78my+kWI05Bgd4HWmvKqWkoqrJRU1FBiqXHcttRQbLFy5Lhj++n2T2vATxQcAK1jKmkZGUTPxEgSmgXRKiqYpOgQEqOCiQsPwt/PN0beZ8pqtzJx5USW7V/GuJ7juPe8e33mfyFCNCYJdhzvqDxeXUt5lY3yahvlVbWUVdkoraqhrMpGWaWN0krH/WMVNo5V1tRvNux1+pTXbBZsIjbcTGyYmZ6JkTQPMxMfYaZFsyDiwoOIjzATHxHE2h++JS1t4Fnu2POVVpfytxV/Y23RWh7q9RC3dr3V6JKE8BouBbtS6lrgn0AX4EKt9Tp3FHU6NnsdlTV2qmrsVNnsVNbUUlljp8Lq2P9022KtpaJ+s1jtWKw2LNZaLNW1HLfWcrzacbvK9tuR9IkCA/yIDDYRGWIiKiSQ9nFhRIYEEh3quB8TFkhMqJno0ECah5mJCjVhDpDpAmfllOZw7zf3UlRRxIyLZshnvwhxhlwdsW8FRgGz3VDLaT32yTbe/r4C+xefN/icAD9FqDmAMHMAoWZ/wswBRIYEkhgdQkSQ4/GIIBMRwSYiggMIN/9yOyokkGbBJo9e0+1rviv8jvErxhPoH8jcIXM5L+48o0sSwuu4FOxa62zgrM179m4bw+EDhXRu35bgQH9CAgMICfQnONCf0MAAQsz1+0B/QgL9CTUHYA7wk3lZL6C15r0djo8IaB/ZnhcGvUBCWILRZQnhlbxqjn1o9xYEFe8gLa2D0aUIN7LarcxYM4MFuxeQnpTOzItmEmI6ux8mJoQvUVqf+sW/nw9Qahlwqu8Xm6K1Xlx/TBYw/o/m2JVS44BxAPHx8SkZGRlOFWyxWAgLC3PqXE/kS/0408sR2xHeKH6D/Jp8BkcM5srIK/FTrn/vqTv40u8GfKsfX+oFGt5Penr6eq116mkP1Fq7vAFZQGpDj09JSdHOWr58udPneiJf6udMe/kq9yvd590+ut97/fTy/Wd27tngS78brX2rH1/qReuG9wOs0w3IWK+aihG+wWa38fT6p3k3+116NO/BUxc/RauwVkaXJYTPcHW540jgBSAW+EwptVFrPcQtlQmflFuWy6RVk9haspVbutzC31L+Jh/kJYSbuboqZiGw0E21CB9Wp+uYv2M+z613fHjXs2nPcmmbS40uSwifJFMxotEdsBzg0e8eZU3RGga0GsBj/R49OJ2VAAAJ+klEQVQjLiTO6LKE8FkS7KLRaK1ZtGcRs9bOQmvN1L5TGd1htLyvQIhGJsEuGkVuWS5PrHmCNQfXkBKfwhP9nyAxPNHosoRoEiTYhVtZ7VbmbpnLnC1zMPubmdx7Mtd3ut5j1qYL0RRIsAu32VG1g39n/pu88jwub3s5/0j9B7EhsUaXJUSTI8EuXJZfns+zG55l6eGltA5vzezBs+mX0M/osoRosiTYhdPKrGXM3jyb+TvmY/IzcUWzK3j8qscx+3vPl2ML4Ysk2MUZs9ltZOzM4JVNr2CxWRjZfiR/Oe8vbPtxm4S6EB5Agl00mK3ORuaeTF7d/CoHKg7Qt2Vf/p76dzpFdzK6NCHECSTYxWnV1tXyyd5PmL15NoWWQrrHdOfRvo/SL6GfrEkXwgNJsIvfVWOv4bOcz3hty2vsP76frjFdmdx7Mhe1ukgCXQgPJsEufqO8ppwPd37Iu9nvcqTqCJ2jO/Of9P+QlpQmgS6EF5BgFz8rqijine3v8NHuj6iwVdCnZR+e6P8EfRP6SqAL4UUk2Js4rTVritaQsSODrPwsAIYkD+GObnfQJaaLscUJIZwiwd5EldeUk7knk/d3vk9ueS5R5ihu73Y713e6Xr5EWggvJ8HehNjr7KwpWkPm3ky+zvuaans158aey/QB07ks+TJZgy6Ej5BgbwJyy3LJ3JtJ5t5MDlUeIjwwnOHnDGd0x9F0jelqdHlCCDeTYPdRhZZCvsz9ki/2fUH20Wz8lB/9E/ozvtd40pPSZXQuhA+TYPchBccL+Gb/N3yZ+yWbizcD0KN5D8anjufytpfLtxYJ0URIsHuxOl3H9pLtLM9fzvL85ew+thuALtFdeOCCBxiSPES+3EKIJkiC3cuUWctYfXA13x/4nm8LvuVw1WH8lB8XxF3A+NTxDEoaRFJEktFlCiEMJMHu4Wx2G9tKtvHDgR/47sB3bCneQp2uI9wUTp+EPqQlpTGw1UAigyKNLlUI4SEk2D2Mrc7GtuJtrC1ay9qitWw8spGq2ioUiu7NuzO2x1j6t+pPj+Y9CPCTX58Q4rckGQxWYa9gRf4K/nf4f2w8spFtxduotlcD0CGqAyPbj6RXi16kxqfKqFwI0SAS7GdRVW0VO47uYFvxNraWbGVb8TZyy3OhAAJUAJ2jO3NNx2s4P+58UlukEh0UbXTJQggvJMHeSEqrS9l5bCc7ju5g17FdZB/NJqc0B7u2AxAXHEfX5l3p4deDUb1H0a15N4IDgg2uWgjhCyTYXVRpq2Rv6V72lO5x7Mv2sOfYHg5VHvr5mNjgWDpGdyQ9KZ3uMd3p1rzbz2vKs7KySG2RalT5QggfJMHeALV1tRRVFLG/fD/7yveRW5ZLbrljK6oo+vm4QL9A2kW2I7VFKp2jOtMxuiOdojoRExxjYPVCiKZGgh3HR9eW15RTaCnkgOUAhZZCCo4XkH88n/zj+RywHKBW1/58fJgpjOSIZFLjU0mOSKZ9VHvaR7YnMSwRfz9/AzsRQogmEOxaayw2C0cqj3C46jBFFUUUVRRxqPLQz7cPVhykwlZx0nnhpnASwxPpHN2ZwW0G0zqiNUnhSbRt1paYoBj54gkhhMfy2mCvrq3mWPUxjlYf5Wj1UYqriimpLqGkqoTiqmKKq4o5UnWEw5WHqaqt+s350UHRxIfEkxSeRO+WvUkITaBVWCsSwhJICEsgIjBCwlsI4ZW8Kthf2fQKGYUZVL5becqwBsc0SUxwDDFBMXSJ7sLAxIHEBccRGxJLXEgcLUJaEBcaJ59uKITwWS4Fu1LqKeAqoAbYC9yptS51R2GnEhcSR1tzW7q06UJ0UDTRQdFEmaOICoqieXBzYoJjZMmgEKLJc3XEvhSYpLWuVUrNAiYBE1wv69RGdRhFdGE0ab3SGusphBDC6/m5crLW+iutf14ushqQz4gVQgiDuRTsv3IX8LkbryeEEMIJSmv9xwcotQxocYofTdFaL64/ZgqQCozSv3NBpdQ4YBxAfHx8SkZGhlMFWywWwsLCnDrXE/lSP77UC0g/nsyXeoGG95Oenr5ea336t6prrV3agDuAH4CQhp6TkpKinbV8+XKnz/VEvtSPL/WitfTjyXypF60b3g+wTjcgY11dFTMUeAi4WGtd6cq1hBBCuIerc+z/BcKBpUqpjUqpV9xQkxBCCBe4NGLXWrd3VyFCCCHcw52rYoQQQniA066KaZQnVeoIkOfk6c2BYjeWYzRf6seXegHpx5P5Ui/Q8H7aaK1jT3eQIcHuCqXUOt2Q5T5ewpf68aVeQPrxZL7UC7i/H5mKEUIIHyPBLoQQPsYbg/1VowtwM1/qx5d6AenHk/lSL+Dmfrxujl0IIcQf88YRuxBCiD/glcGulJqmlNpc/27Xr5RSCUbX5Cyl1FNKqR31/SxUSkUaXZMrlFLXKqW2KaXqlFJeuWpBKTVUKbVTKbVHKTXR6HpcoZSaq5Q6rJTaanQt7qCUSlJKLVdKba//c3a/0TU5SykVpJT6USm1qb6Xx9x2bW+cilFKRWity+tv/xXoqrX+s8FlOUUpdRnwjf7ly0rQWjfal5U0NqVUF6AOmA2M11qvM7ikM6KU8gd2AYOBAmAtcKPWeruhhTlJKTUQsABva627G12Pq5RSLYGWWusNSqlwYD1wtTf+fpTjS5VDtdYWpZQJ+Ba4X2u92tVre+WI/adQrxcKeN+/TvW0j31ZidY6W2u90+g6XHAhsEdrnaO1rgEygBEG1+Q0rfVK4KjRdbiL1vqg1npD/e3jQDbQytiqnFP/gY2W+rum+s0tWeaVwQ6glHpSKZUP3Aw8anQ9biJfVmK8VkD+CfcL8NLg8HVKqWTgfGCNsZU4Tynlr5TaCBwGlmqt3dKLxwa7UmqZUmrrKbYRAFrrKVrrJOBd4F5jq/1jp+ul/pgpQC2OfjxaQ/oRojEppcKABcADv/ofvFfRWtu11ufh+J/6hUopt0yXufpl1o1Ga31pAw99F1gCTG3Eclxyul6UUncAVwKXaC940eMMfjfeqBBIOuF+Yv1jwkPUz0cvAN7VWn9sdD3uoLUuVUotB4YCLr/Q7bEj9j+ilOpwwt0RwA6janHVCV9WMly+rMQjrAU6KKXaKqUCgRuATINrEvXqX3B8HcjWWj9jdD2uUErF/rQKTikVjOMFe7dkmbeuilkAdMKx+iIP+LPW2itHVUqpPYAZKKl/aLW3rvABUEqNBF4AYoFSYKPWeoixVZ0ZpdQVwHOAPzBXa/2kwSU5TSk1H0jD8emBh4CpWuvXDS3KBUqpAcAqYAuOv/8Ak7XWS4yryjlKqZ7AWzj+nPkBH2itH3fLtb0x2IUQQvw+r5yKEUII8fsk2IUQwsdIsAshhI+RYBdCCB8jwS6EED5Ggl0IIXyMBLsQQvgYCXYhhPAx/w9kqMgG2Ua5YwAAAABJRU5ErkJggg==\n", "text/plain": [ "
" ] }, "metadata": { "needs_background": "light" }, "output_type": "display_data" } ], "source": [ "plt.plot(x, backend.get_value(activations.elu(x, alpha=1.0))) # default\n", "plt.plot(x, backend.get_value(activations.elu(x, alpha=0.5))) \n", "plt.plot(x, backend.get_value(activations.elu(x, alpha=2.0)))\n", "plt.grid(True); plt.show()" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## selu\n", "\n", "SELU stands for scaled ELU:\n", "\n", "scale * elu(x, alpha)\n", "\n", "where alpha and scale are chosen so that the mean and variance of the inputs are preserved between two consecutive layers (as long as the weights are initialized correctly)." ] }, { "cell_type": "code", "execution_count": 4, "metadata": {}, "outputs": [ { "data": { "image/png": "iVBORw0KGgoAAAANSUhEUgAAAXYAAAD8CAYAAABjAo9vAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDMuMC4xLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvDW2N/gAAH6NJREFUeJzt3Xd8VfX9x/HXN5sMVhJCIEAAWZGdMB2FOopV6yhuVIYErdu2aqvV2lZ/tVNbrRMMIBBRsVKKCw0oKpKEEUbYGAgjAQKZZH9/fxCttQghueTce+77+XjcxyM3nHvu+2OStyfnfnOusdYiIiLuEeB0ABER8SwVu4iIy6jYRURcRsUuIuIyKnYREZdRsYuIuIyKXUTEZVTsIiIuo2IXEXGZICeeNCYmxiYmJjbpseXl5URERHg2kIPcNI+bZgHN483cNAs0fp7s7OyD1trYk23nSLEnJiaSlZXVpMcuXbqUMWPGeDaQg9w0j5tmAc3jzdw0CzR+HmNMXmP2p1MxIiIuo2IXEXEZFbuIiMuo2EVEXEbFLiLiMip2ERGXUbGLiLhMs4vdGBNmjFlpjFlrjNlgjHnME8FERNykrKqWXy/cQEllzWl/Lk8csVcB37fWDgIGA+OMMSM9sF8REVfYe+Qo45/7jNkr8sjcWXTan6/Zf3lqj70bdlnD3eCGm94hW0QEyMk/wpSZWVRW1/HKxGGc2/ukVwRoNnOsl5u5E2MCgWzgDOBZa+0Dx9kmFUgFiIuLS05PT2/Sc5WVlREZGdmMtN7FTfO4aRbQPN7MV2bJLqjlhbVVRIUY7k0OIyHq+CdJGjvP2LFjs621KSfd0FrrsRvQFsgA+p9ou+TkZNtUGRkZTX6sN3LTPG6axVrN4828fZb6+nr7wrJtNvHBRfayZ5bbwpLKE27f2HmALNuILvboRcCstUeMMRnAOGC9J/ctIuILaurqeeTtDcxbuYuLB8Tz56sHERYc2KIZml3sxphYoKah1FsBFwBPNjuZiIiPKT5aw+1zVrF820F+MqYnP7uwDwEBpsVzeOKIPR6Y2XCePQCYb61d5IH9ioj4jN1FFUxOy2TnwXL+MH4gV6d0cSyLJ1bF5ABDPJBFRMQnZecdJnVWFjV19cyaMpzRPWMczePIG22IiLjFopy93Dd/LfFtwpgxcRg9Y51fraNiFxFpAmstz2Zs40/vbyGlWztevCmF9hEhTscCVOwiIqesuraeXyxYx5ur8rl8cCeeHD+Q0KCWXflyIip2EZFTcKSimmmzs/liZxH3nN+Lu8/rhTEtv/LlRFTsIiKNtPNgOZPTMtlz+ChPXTOYy4d0djrScanYRUQaYeXOIlJnZ2GAOVNHMCyxvdORvpOKXUTkJBasyueBN3Po0i6cGROHkRgT4XSkE1Kxi4h8B2stf/1gC3/7aBsje7Tn+QnJtA33jpUvJ6JiFxE5jsqaOu5/I4eFa/dyVXICj18xgJAg33jTORW7iMi3HCqrInV2Ntl5h7l/XB9u+15Pr1v5ciIqdhGRb9hWWMbktEwKSip59vqhXDww3ulIp0zFLiLS4LNtB7n11WxCggJITx3JkK7tnI7UJCp2ERFgfuZufvnWOnrERjD95mF0aR/udKQmU7GLiF+rr7f84b3NPL9sO+f0iuHZG4bSOizY6VjNomIXEb91tLqO++av4Z31+7lhRFce+9GZBAX6xsqXE1Gxi4hfKiytZOqsbHLyj/Dwxf2YcnZ3n1r5ciIqdhHxO5v3lzI5LZOi8mpemJDMhWd2dDqSR6nYRcSvLNtygNvnrCIiNJDXbx1F/85tnI7kcSp2EfEbs1fk8euFG+gdF8WMiSnEt2nldKTTQsUuIq5XV295YnEu05fv5Ly+HfjbdUOICHVv/bl3MhERoLyqlrvT17Akt4BJZyXy8MVJBAa440XS76JiFxHX2l9cyZSZmeTuK+E3l53JTaMSnY7UIlTsIuJKG/YWMyUti9LKGqZPHMbYPh2cjtRiVOwi4jpLNhZwV/pq2rYK5o3bRtMvvrXTkVqUil1EXMNay4xPv+R3/97IgM5tePmmFDq0DnM6VotTsYuIK9TW1fPYvzYye0UePzgzjqeuGUKrkECnYzlCxS4iPq+0soY75q5m2ZYDTDu3Bw+M60uAy1e+nIiKXUR8Wv7hCqakZbHtQBn/d+UArhve1elIjlOxi4jPWrv7CFNmZlFVU8fMScM5u1eM05G8gopdRHzSO+v2ce/8NcREhjJv6gh6xUU5HclrqNhFxKdYa1m8o5r5765iSNe2vHRTCjGRoU7H8ioqdhHxGTV19Tz81nrmb6nhkoHx/OmqQYQF++fKlxNpdrEbY7oAs4A4wAIvWmufbu5+RUS+qbiihtvmZPPZ9kNc2jOYp68d4tcrX07EE0fstcBPrbWrjDFRQLYx5gNr7UYP7FtEhF2HKpiUtpJdRRX86apBxJRuU6mfQLPf3M9au89au6rh41IgF+jc3P2KiABk5xVxxT8+5WBZNbOnjGB8coLTkbyeR8+xG2MSgSHAF57cr4j4p7fX7OHnb+TQqU0YMyYOo0dspNORfIKx1npmR8ZEAsuAx621C47z76lAKkBcXFxyenp6k56nrKyMyEj3fHHdNI+bZgHN4yRrLQu31/DWthp6twvgriFhRIb859SLL83SGI2dZ+zYsdnW2pSTbmitbfYNCAbeA+5rzPbJycm2qTIyMpr8WG/kpnncNIu1mscplTW19t701bbbA4vsvemrbWVN7f9s4yuzNFZj5wGybCM61hOrYgwwHci11v6lufsTEf91uLyaabOzWfllEfdd0Js7v38GxypGToUnzrGfBdwIrDPGrGn43C+ttYs9sG8R8RM7DpQxOS2TvcWVPH3tYC4brDUYTdXsYrfWLgf0v1QRabIVOw4xbXY2gQGGeVNHkNytvdORfJr+8lREHPVGdj6/WJBD1/bhvDJxOF2jw52O5PNU7CLiiPp6y18+2MIzGdsY3TOa525Ipk14sNOxXEHFLiItrrKmjp+9vpZFOfu4JqULv7uiP8GBzf57SWmgYheRFnWwrIqps7JYvesID17Ul2nn9tDKFw9TsYtIi9laUMqktEwOlFbx3A1DuWhAvNORXEnFLiItYvnWg9w2J5vQoEBemzaKwV3aOh3JtVTsInLazVu5i4f/uZ4zYiOZPjGFhHZa+XI6qdhF5LSpr7c8+e4mXvh4B9/rHcsz1w8hKkwrX043FbuInBZHq+u457XVvLehgBtHduPRS5MI0sqXFqFiFxGPKyyp5JZZWazbU8wjlyQx6axErXxpQSp2EfGo3H0lTEnL5MjRGl66MYXzk+KcjuR3VOwi4jEZmwq5Y+4qosKCmT9tFP07t3E6kl9SsYuIR8z6/Et+vXADSZ1aM/3mYcS1DnM6kt9SsYtIs9TVW367aCNpn33JBUlxPH3tYMJDVC1O0n99EWmysqpa7pq3mo82FTL1nO48eFE/AgP0IqnTVOwi0iT7io8yOS2LLQWl/O7y/kwY2c3pSNJAxS4ip2z9nmImp2VytLqOVyYO49zesU5Hkm9QsYvIKXl/w37uTl9D+4gQZt82gj4do5yOJN+iYheRRrHWMn35Th5fnMughLa8dFMKsVGhTseS41Cxi8hJ1dTV8+jCDcz9YhcXD4jnz1cPIiw40OlY8h1U7CJyQiWVNdw+ZxWfbD3IT8b05GcX9iFAK1+8mopdRL7T7qIKpszMZMeBcv4wfiBXp3RxOpI0gopdRI5r9a7DTJ2VRXVtPbOmDGd0zxinI0kjqdhF5H8sXrePe19bQ1zrMNJTh3FGh0inI8kpULGLyNestTy3bDt/eHczKd3a8eJNKbSPCHE6lpwiFbuIAFBdW89Db63j9ex8LhvciSd/PFArX3yUil1EOFJRza2vZrNiRxF3n9eLe87vpTfG8GEqdhE/9+XBcianZZJ/+ChPXTOYy4d0djqSNJOKXcSPZX5ZROqsLABevWUEw7u3dziReIKKXcRP/XP1Hu5/I4eEdq2YMXEYiTERTkcSD1Gxi/gZay1PLdnK0x9uZWSP9jw/IZm24Vr54iYqdhE/UllTxwNv5vD2mr2MT07giSsGEBIU4HQs8TAVu4ifKKm2THj5C7LyDvPzH/ThJ2N6auWLS3mk2I0xM4BLgEJrbX9P7FNEPGf7gTJ++/lRimsqeeb6IVwysJPTkeQ08tTvYGnAOA/tS0Q86LPtB7nyH59RWWdJTx2pUvcDHjlit9Z+bIxJ9MS+RMRz5mft5pcL1tE9JoLUvkEM7drO6UjSAvSqiYgL1ddbnnx3E/e/kcOontG8cdtoYsP14+4vjLXWMzs6dsS+6LvOsRtjUoFUgLi4uOT09PQmPU9ZWRmRke650pyb5nHTLOC781TXWV7MqSKroI4xCUFMSAohKMD47DzH46ZZoPHzjB07Nttam3LSDa21HrkBicD6xmybnJxsmyojI6PJj/VGbprHTbNY65vzFJZU2h89s9wmPrjIvrhsu62vr//633xxnu/iplmsbfw8QJZtRMdquaOIS2zeX8rktEyKyqt5fkIyPzizo9ORxCEeOelmjJkHfA70McbkG2OmeGK/ItI4y7YcYPxzn1FTV8/8aaNU6n7OU6tirvPEfkTk1L26Io9HF26gV4dIZkwcRqe2rZyOJA7TqRgRH1VXb3licS7Tl+9kbJ9Y/n79UCJD9SMtKnYRn1RRXcvd6Wv4YGMBN4/qxq8uSSIoUMsZ5RgVu4iPKSipZHJaJrn7Snj00iQmndXd6UjiZVTsIj5kw95ipqRlUVpZw8s3p/D9vnFORxIvpGIX8REfbSrgjrmradMqmNdvHU1Sp9ZORxIvpWIX8QFpn+7kN4s2ktSpNdNvHkZc6zCnI4kXU7GLeLHaunp+u2gjMz/P44KkOJ6+djDhIfqxlRPTd4iIlyqrquXOuavI2HyAqed058GL+hEYoDfGkJNTsYt4ob1HjjI5LZOthWU8fkV/bhjRzelI4kNU7CJeJif/CFNmZlFZXccrE4dxbu9YpyOJj1Gxi3iR9zbs5+701URHhDLnJyPoHRfldCTxQSp2ES9greWlT3bwf+9sYlBCW166KYXYqFCnY4mPUrGLOKymrp5H3l7PvJW7uXhAPH++ehBhwYFOxxIfpmIXcVDx0Rpun7OK5dsOcvvYnvz0gj4EaOWLNJOKXcQhu4sqmJSWSd6hcv44fiBXpXRxOpK4hIpdxAHZeYdJnZVFbb1l5uThjO4Z43QkcREVu0gL+9favfz09bXEtwljxsRh9Ix1z5syi3dQsYu0EGstz2Zs40/vb2FYYjteuDGF9hEhTscSF1Kxi7SA6tp6frFgHW+uyueKIZ35/Y8HEBqklS9yeqjYRU6zIxXVTJudzRc7i7jn/F7cfV4vjNHKFzl9VOwip9HOg+VMTstkz+GjPH3tYC4b3NnpSOIHVOwip8nKnUWkzs4iwBjmTh1BSmJ7pyOJn1Cxi5wGC1bl88CbOXRpH84rE4fRLTrC6UjiR1TsIh5kreWvS7bytw+3MqpHNM9PSKZNeLDTscTPqNhFPKSypo7738hh4dq9XJ2SwO8uH0BIUIDTscQPqdhFPOBQWRWps7PJzjvMA+P6cuv3emjlizhGxS7STNsKS5mUlklhSRX/uGEoPxwQ73Qk8XMqdpFm+GzbQaa9mk1oUCCvTRvF4C5tnY4komIXaarXMnfx0Fvr6REbwYyJw0hoF+50JBFAxS5yyurrLX94bzPPL9vOOb1iePaGobQO08oX8R4qdpFTcLS6jvvmr+Gd9fuZMLIrv770TIICtfJFvIuKXaSRCksrmTozi5w9xfzqkiQmn5WolS/ilVTsIo2waX8JU9KyKCqv5sUbU7ggKc7pSCLfySO/QxpjxhljNhtjthljHvTEPkW8xdLNhYx/7nNq6+t5/dZRKnXxes0udmNMIPAscBGQBFxnjElq7n5FvMHsz79kclomXduH88/bz6J/5zZORxI5KU+cihkObLPW7gAwxqQDlwEbPbBvEUfU1Vvm5lbxft4Gzu/XgaevHUJEqM5cim/wxKmYzsDub9zPb/iciE+y1nJX+mrez6tl0lmJvHBjikpdfIqx1jZvB8aMB8ZZa29puH8jMMJae8e3tksFUgHi4uKS09PTm/R8ZWVlREa6581/3TSPW2Z5/8sa5m6q5tJulh/38/15vuKWrw+4axZo/Dxjx47NttamnGw7TxyG7AG6fON+QsPn/ou19kXgRYCUlBQ7ZsyYJj3Z0qVLaepjvZGb5nHDLOvyi3n9g085v18cV3Yt9fl5vskNX5+vuGkW8Pw8njgVkwn0MsZ0N8aEANcCCz2wX5EWVVpZwx3zVhETGcofxw/UGnXxWc0+YrfW1hpj7gDeAwKBGdbaDc1OJtKCrLU8/M/17C6qID11FO0iQpyOJNJkHnlFyFq7GFjsiX2JOOH17HzeXrOX+y7ozfDuem9S8W26yIX4vW2FpTz69gZG9Yjm9rFnOB1HpNlU7OLXKqpruWPuasJDAnnq2sEEBui8uvg+Lc4Vv2Wt5eev57C5oJS0ScOJax3mdCQRj9ARu/itfyzdzr/X7ePBcX35Xu9Yp+OIeIyKXfzSko0F/On9zVw2uBOp5/ZwOo6IR6nYxe9sKyzlntfW0L9TG578sdari/uo2MWvFFfUMHVWNmHBgbx4UzJhwYFORxLxOBW7+I3aunruTF9N/uEKnp8wlPg2rZyOJHJaaFWM+AVrLb96ez0fbznA768cQEqi/ghJ3EtH7OIXnlqylXkrd3PH2DO4dnhXp+OInFYqdnG9V1fk8fSHW7k6JYGfXtjb6Tgip52KXVzt3fX7eeTt9Xy/bweeuGKAVsCIX1Cxi2tlflnEXemrGZjQlmeuH0JQoL7dxT/oO11cKXdfCVPSMklo24oZE4cRHqJ1AuI/VOziOrn7Srj+pRWEhwQxc/Jw2uva6uJnVOziKpv3l3LDy18QEhRAeupIurQPdzqSSItTsYtrbCko5fqXVhAcaEhPHUViTITTkUQcoWIXV9jaUOqBAYZ5U0fSXaUufkzFLj5va0Ep1730BQHGMC91JD1iI52OJOIoFbv4tOy8w1z1wucYA3OnjqSnSl1ExS6+K2NzITe8vIK2rYJZcNtozuigUhcBXQRMfNSCVfnc/0YOfeOjSJs0nJjIUKcjiXgNFbv4nJc+3sHji3MZ3TOaF25MJios2OlIIl5FxS4+o67e8sTiXKYv38nFA+L5yzWDCA3SG2WIfJuKXXxCSWUNd85dzbItB5g4OpFfXZJEYIAu6CVyPCp28Xo7D5Zzy8xM8g5V8MQVA7h+hK6nLnIiKnbxasu3HuT2uasIMPDqLSMY2SPa6UgiXk/FLl7JWssrn37J44tzOSM2kpdvTtF1X0QaScUuXqe4oob731zLexsKuCApjr9eM5jIUH2rijSWflrEq6zedZg7561mf3ElD1/cjylnd9e7HomcIhW7eAVrLdOX7+T372wirnUYr986iiFd2zkdS8QnqdjFcYWllTz45jo+2lTIhUlx/HH8INqE64+ORJpKxS6Osdbyr5x9PPL2eiqq63j00iQmjk7UqReRZmpWsRtjrgJ+DfQDhltrszwRStzvUFkVv3p7PYvX7WdQl7b8+apBuoiXiIc094h9PXAl8IIHsoifeHf9Ph56az2llbXcP64Pqef0IChQFxoV8ZRmFbu1NhfQr87SKLuLKnjsXxtZkltA/86tmXvVYPp0jHI6lojr6By7nHbVtfW8vHwHf/twKwbDLy7qy+SzuxOso3SR08JYa0+8gTFLgI7H+aeHrLVvN2yzFPjZic6xG2NSgVSAuLi45PT09CYFLisrIzLSPedi3TTP8WbJPVTHrI1V7Cu3JMcFcn3fEKJb+Uahu+lrA+6ax02zQOPnGTt2bLa1NuWkG1prm30DlgIpjd0+OTnZNlVGRkaTH+uN3DTPN2fZcaDMTpuVZbs9sMie/eSH9sPc/c4FayI3fW2sddc8bprF2sbPA2TZRnSsTsWIRxWVV/O3D7fy6oo8QoICuPf83kz7Xg/CgnXddJGW0tzljlcAfwdigX8bY9ZYa3/gkWTiUypr6vj3jmruzMigvLqWa4d35Z7ze9EhKszpaCJ+p7mrYt4C3vJQFvFBlTV1zP1iF88t286B0hrO69uBBy/qS684rXYRcYpOxUiT/HehVzGqRzS39DNMu3KY09FE/J6KXU5JSWUN6St38dInO78u9L9fN4SRPaJZunSp0/FEBBW7NNKeI0d5ZflO0jN3U1ZVy+ie/yl0EfEuKnY5oZz8I0xfvpNFOfsAuGRgPFPP6UH/zm0cTiYi30XFLv/jaHUdC9fu4dUVu1i3p5jI0CAmjU5k0tnd6dy2ldPxROQkVOzyta0Fpcz5YhdvrsqntLKWXh0ieexHZ3LF0M60DtP10UV8hYrdzx0ur2bh2r0sWJXP2vxiggMNF/WPZ8LIbgxLbKcLvIn4IBW7H6qsqWPZlgMsWJXPR5sKqamz9ItvzcMX9+PyIZ2JiQx1OqKINIOK3U9U1dbxyZaD/HvdPj7YWEBZVS0xkaHcPCqRK4cmkNSptdMRRcRDVOwuVl5VyydbD/D+hgI+2FhAaVUtbVoF88MBHfnhgHjOPiNGb3Ah4kIqdpfZV3yUJbmFLNlYwOfbD1FdV0/rsCDG9e/IxQPjGd0zhpAglbmIm6nYfVxlTR0rdxbx8ZYDfLL1IJsLSgHoFh3OTaO6cV6/OFIS2+lNLUT8iIrdx9TU1bNuTzErdhzi8+2H+GJnEdW19YQEBTA8sT1XDu3Mef060DM2UitaRPyUit3LVdbUkZNfTOaXRazYcYjsvMNUVNcB0DsukhtHduOcXjGM6B5NqxBd81xEVOxexVrL3uJKVu86THbeYVblHWbD3hJq64+9fWGfuCiuSk5gRI9ohndvr2WJInJcKnaHWGspKKlidWEtq97fTM6eYtblF3OovBqAsOAABia0Zeq5PRjatR3J3drRPiLE4dQi4gtU7C2gsqaO7QfK2LSvlI37SshtuB2uqAEgwGyjd1wU3+/bgYEJbRiY0JakTq31gqeINImK3YPKqmrZeaCcHQfL2FJQypaCMrYVlpF3qJyGsymEBgXQp2MUFyZ1pF98FDWFO5hw8RidHxcRj1Gxn6Kyqlp2HapgV1E5eYcq+PJQBTsPlrHjQDmFpVVfbxcYYEiMDqdvxyguHdSJXh0i6RcfRWJ0xH/9UdDSpXkqdRHxKBX7t5RU1rC/uJI9R46Sf/go+Ycr2HP42Me7iyq+Pgf+lbbhwfSIieDc3rF0j4mgZ2wE3WMi6R4ToT8EEhFH+E2xV9fWc6i8isKSKgpKKikoraKwpJKCkkr2FR+77S+upKyq9r8eFxIYQKe2YSS0C+eCpDi6RofTrX0EXduH0zU6nDatdDlbEfEuPlvsNXX1HKmo4UhFNUXlx26Hyv/z8cGyqoZbNQdKqyg+WvM/+wgwEBMZSnybMHrGRnD2GTF0ahtGxzat6NxQ5rGRoQQE6A99RMR3+FSxP71kK69+WkFlxnuUfuvI+puiwoKIjgghJjKUXh0iGdUjmtioUGIiQ4lrHUqHqDDiWocSHRlKoEpbRFzGp4q9Y5tQzmgbQJ/uCbQLD6FdRDBtw0NoFx5MdEQo0ZEhtAsP0bltEfFrPlXs1wzrSlz5DsaMOdPpKCIiXkuHtiIiLqNiFxFxGRW7iIjLqNhFRFxGxS4i4jIqdhERl1Gxi4i4jIpdRMRljLW25Z/UmANAXhMfHgMc9GAcp7lpHjfNAprHm7lpFmj8PN2stbEn28iRYm8OY0yWtTbF6Rye4qZ53DQLaB5v5qZZwPPz6FSMiIjLqNhFRFzGF4v9RacDeJib5nHTLKB5vJmbZgEPz+Nz59hFROTEfPGIXURETsAni90Y81tjTI4xZo0x5n1jTCenMzWVMeaPxphNDfO8ZYxp63Sm5jDGXGWM2WCMqTfG+OSqBWPMOGPMZmPMNmPMg07naQ5jzAxjTKExZr3TWTzBGNPFGJNhjNnY8H12t9OZmsoYE2aMWWmMWdswy2Me27cvnooxxrS21pY0fHwXkGStvdXhWE1ijLkQ+MhaW2uMeRLAWvuAw7GazBjTD6gHXgB+Zq3NcjjSKTHGBAJbgAuAfCATuM5au9HRYE1kjDkXKANmWWv7O52nuYwx8UC8tXaVMSYKyAYu98WvjzHGABHW2jJjTDCwHLjbWruiufv2ySP2r0q9QQTge/93amCtfd9a+9UbuK4AEpzM01zW2lxr7WanczTDcGCbtXaHtbYaSAcuczhTk1lrPwaKnM7hKdbafdbaVQ0flwK5QGdnUzWNPaas4W5ww80jXeaTxQ5gjHncGLMbuAF4xOk8HjIZeMfpEH6uM7D7G/fz8dHicDtjTCIwBPjC2SRNZ4wJNMasAQqBD6y1HpnFa4vdGLPEGLP+OLfLAKy1D1lruwBzgDucTXtiJ5ulYZuHgFqOzePVGjOPyOlkjIkE3gTu+dZv8D7FWltnrR3Msd/UhxtjPHK6zGvfzNpae34jN50DLAYePY1xmuVksxhjJgKXAOdZH3jR4xS+Nr5oD9DlG/cTGj4nXqLhfPSbwBxr7QKn83iCtfaIMSYDGAc0+4Vurz1iPxFjTK9v3L0M2ORUluYyxowD7gd+ZK2tcDqPkAn0MsZ0N8aEANcCCx3OJA0aXnCcDuRaa//idJ7mMMbEfrUKzhjTimMv2Huky3x1VcybQB+Orb7IA2611vrkUZUxZhsQChxq+NQKX13hA2CMuQL4OxALHAHWWGt/4GyqU2OM+SHwFBAIzLDWPu5wpCYzxswDxnDs6oEFwKPW2umOhmoGY8zZwCfAOo79/AP80lq72LlUTWOMGQjM5Nj3WQAw31r7G4/s2xeLXUREvptPnooREZHvpmIXEXEZFbuIiMuo2EVEXEbFLiLiMip2ERGXUbGLiLiMil1ExGX+H119a7h1l2o0AAAAAElFTkSuQmCC\n", "text/plain": [ "
" ] }, "metadata": { "needs_background": "light" }, "output_type": "display_data" } ], "source": [ "plt.plot(x, backend.get_value(activations.selu(x)))\n", "plt.grid(True); plt.show()" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## relu\n", "\n", "RELU stands for rectified ELU. In its default form it is very similar to a linear responce function f(x) = max(0,x), which cutoff negative values and enhance stong input over lighter ones.\n", "\n", "RELU is probably one of the most used activation function in input or hidden layers of a neural network.\n", "\n", "RELU can be also customized with some parameters:\n", "* max_value which sets a plateau over output maximum responce value\n", "* alpha which model responce over negative input values\n", "\n", "RELU form:\n", "f(x) = max_value , for x >= max_value,
\n", "f(x) = x , for threshold <= x < max_value,
\n", "f(x) = alpha * (x - threshold) otherwise." ] }, { "cell_type": "code", "execution_count": 5, "metadata": {}, "outputs": [ { "data": { "image/png": "iVBORw0KGgoAAAANSUhEUgAAAXcAAAD8CAYAAACMwORRAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDMuMC4xLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvDW2N/gAAHxRJREFUeJzt3Xl4lPXVxvHvIYSwhNVARAFBQRYVJEGWavsa96217orV1lpRAXetuLe1tta2ahWt7VttXQKBAlqKe2uopa9aISxhk0VFQER2CBDIct4/Mm3TiJlhMpNn5sn9ua65rll+M7kPA+f5cWYyY+6OiIiES7OgA4iISOKpuYuIhJCau4hICKm5i4iEkJq7iEgIqbmLiISQmruISAipuYuIhJCau4hICDWPtsDMWgJvA1mR9VPc/b46a7KA54B8YBNwkbt/XN/j5uTkeM+ePeMKvXPnTtq0aRPXfVOR6kldYaoFwlVPmGqB2OuZM2fORnfvHHWhu9d7AgzIjpzPBN4DhtdZMxp4KnL+YmBStMfNz8/3eBUXF8d931SkelJXmGpxD1c9YarFPfZ6gNkepb+6e/SxTOTxyiIXMyOnuh9IczbwbOT8FOBEM7OoRxYREUkK8xg+OMzMMoA5QG/gCXe/vc7tC4HT3H1N5PJKYJi7b6yzbhQwCiA3Nze/qKgortBlZWVkZ2fHdd9UpHpSV5hqgXDVE6ZaIPZ6CgoK5rj7kKgLY9ne+3/GLx2AYuDIOtcvBLrVurwSyKnvsTSW+Q/Vk7rCVIt7uOoJUy3uAYxl6hwItkaa+2l1bloLdAcws+ZAe2peWBURkQBEbe5m1tnMOkTOtwJOBpbWWTYd+Hbk/PnAW5EjjIiIBCDqWyGBrsCzkbl7M2Cyu88wsx9R89+D6cDTwPNmtgLYTM07ZkREJCBRm7u7LwAG7+P6e2udLwcuSGw0ERGJl35DVUSkEf3qL8tZuHZb0n9OLGMZERFJgBfnruGRvyxjb1UVRx7cPqk/Szt3EZFGsHz9Du6ctpChPTtx00mHJ/3nqbmLiCTZrr2VjC4soXWLDB4fOZjmGclvvRrLiIgkkbtz90sLWbGhjOe/O4zcdi0b5edq5y4ikkR/nL2GaSVruf6EPhzXJ6fRfq6au4hIkixZt517/rSQY3sfwPUn9mnUn63mLiKSBDvKKxhdWEL7Vpk8etFgMpo17gflauYuIpJg7s4d00pZtWknE68aTue2WY2eQTt3EZEEe+G9T5ixYB23nNKXYYceEEgGNXcRkQQqXbON+/+8mOP7duba/zkssBxq7iIiCbJtdwWjJ8whJ7sFj1x4NM0aec5em2buIiIJ4O58f8p81m0tZ9LVI+jYpkWgebRzFxFJgGf+8TGvL1rPuNP7kX9Ix6DjqLmLiDRUySdb+OkrSzh5QC5XHtcr6DiAmruISINs2bmXsYUlHNi+Jb84fxBmwc3Za9PMXUQkTtXVzs2T57GxbC9Trh1B+9aZQUf6N+3cRUTi9Ju3P6T4gw3cfVZ/BnbrEHSc/6LmLiISh/c+3MQv3viAMwd25bLhhwQd5wvU3EVE9tPGsj1cN3EuPTq15sFzj0qZOXttau4iIvuhqtq5sWge23ZX8MTIPNq2TJ05e216QVVEZD+Mf2sFs1Zs5MFzj2LAQe2CjvOltHMXEYnRP1Zs5NG/LuPcwQdz0THdg45TLzV3EZEYfL69nBuK5tK7czY/PufIlJyz16axjIhIFJVV1Vw3cS4791Qx8ao8WrdI/dYZdeduZt3NrNjMFpvZIjO7YR9rjjezbWY2L3K6NzlxRUQa3yN/WcZ7H23mgXOOpE9u26DjxCSWw08lcIu7l5hZW2COmb3p7ovrrPu7u5+V+IgiIsEp/uBzniheycXHdOfcvG5Bx4lZ1J27u69z95LI+R3AEuDgZAcTEQnap1t3c/OkefQ7sC0/+MYRQcfZL/v1gqqZ9QQGA+/t4+YRZjbfzF41s/T6UxARqaOiqpqxE0rYW1nNk5fm0TIzI+hI+8XcPbaFZtnA34AH3H1andvaAdXuXmZmZwC/cvc++3iMUcAogNzc3PyioqK4QpeVlZGdnR3XfVOR6kldYaoFwlVPsmspWrqH1z6u5NpBWQzrmvwXUGOtp6CgYI67D4m60N2jnoBM4HXg5hjXfwzk1LcmPz/f41VcXBz3fVOR6kldYarFPVz1JLOW1xeu80Nun+H3vFSatJ9RV6z1ALM9hj4cy7tlDHgaWOLuD3/JmgMj6zCzodSMezZFPbKIiKSY1Zt3cesf53PUwe2568z+QceJWyz/1zgWuAwoNbN5kevuBHoAuPtTwPnAtWZWCewGLo4cYURE0saeyirGTCjBgSdG5pHVPL3m7LVFbe7uPguo91ex3H08MD5RoUREgvCTl5ewYM02fnNZPj0OaB10nAbRxw+IiAAzFnzKs++s4nvH9eLUIw4MOk6DqbmLSJP30cadjJtayuAeHbj99H5Bx0kINXcRadLKK6oYXVhC8wzjiZF5ZGaEoy2m/qffiIgk0Q//vIgl67bz++8cw0EdWgUdJ2HCcYgSEYnDtJI1TPznakYffxgF/boEHSeh1NxFpElavn4Hd724kKG9OnHzyYcHHSfh1NxFpMnZtbeS0YUltMnK4PFLBtM8JHP22jRzF5Emxd25+6WFrNhQxgtXDiO3XcugIyVF+A5XIiL1mDx7NdNK1nL9CX04tndO0HGSRs1dRJqMJeu2c++fFnFc7xyuP/ELH1wbKmruItIk7CivYHRhCe1bZfLoxUeT0Sy1v+C6oTRzF5HQc3fumFbKqk07mXjVcHKys4KOlHTauYtI6L3w7ipmLFjHraf2ZdihBwQdp1GouYtIqJWu2cb9M5ZQ0Lcz13ztsKDjNBo1dxEJrW27Kxg9YQ452S14+MKjaRbyOXttmrmLSCi5O9+fMp91W8uZdPUIOrZpEXSkRqWdu4iE0jP/+JjXF61n3On9yD+kY9BxGp2au4iETsknW/jpK0s4ZUAuVx7XK+g4gVBzF5FQ2bJzL2MLS+jaoSU/P38QZk1nzl6bZu4iEhrV1c7Nk+exsWwvU64dQfvWmUFHCox27iISGk+9vZLiDzZwz1n9GditQ9BxAqXmLiKh8N6Hm/jlG8s4a2BXvjX8kKDjBE7NXUTS3sayPVw3cS49OrXmp+ce1WTn7LWpuYtIWquqdm4smse23RU8MTKPti2b7py9Nr2gKiJp7fG3ljNrxUZ+dt5RDDioXdBxUkbUnbuZdTezYjNbbGaLzOyGfawxM3vMzFaY2QIzy0tOXBGR/5i1fCO/+utyzh18MBcO6R50nJQSy1imErjF3QcAw4ExZjagzprTgT6R0yjg1wlNKSJSx5byam6cNJfenbP58TlHas5eR9Tm7u7r3L0kcn4HsAQ4uM6ys4HnvMa7QAcz65rwtCIiQGVVNU/N38POPVU8eWkerVtowlzXfr2gamY9gcHAe3VuOhhYXevyGr54ABARSYiH31zGB1uqeeCcI+mT2zboOCnJ3D22hWbZwN+AB9x9Wp3bZgAPuvusyOW/Are7++w660ZRM7YhNzc3v6ioKK7QZWVlZGdnx3XfVKR6UleYaoFw1DN/QyWPzNnDiFzn6sHpXUttsT43BQUFc9x9SNSF7h71BGQCrwM3f8ntvwEuqXX5A6BrfY+Zn5/v8SouLo77vqlI9aSuMNXinv71rN2yywf98HU/9ZG/+et/eSvoOAkV63MDzPYY+nYs75Yx4Glgibs//CXLpgOXR941MxzY5u7roh5ZRERitLeymjETSqiscp68NI8WGXoBtT6xvApxLHAZUGpm8yLX3Qn0AHD3p4BXgDOAFcAu4IrERxWRpuyh15Yy95OtjB85mEM7Z/NJ0IFSXNTm7jVz9HoPkZH/KoxJVCgRkdreWPQZv5v1EZePOISzBh4UdJy0oI8fEJGUtnrzLm7543yOOrg9d53ZP+g4aUPNXURS1p7KKkYXlgDw5KV5ZDXPCDhR+tA7/0UkZf3k5SWUrt3Gby/Lp3un1kHHSSvauYtISpqx4FOefWcV3zuuF6cccWDQcdKOmruIpJwPN5QxbmopeT06cPvp/YKOk5bU3EUkpZRX1MzZMzOM8SPzyMxQm4qHZu4iklJ+MH0RSz/bwe+vOIaDOrQKOk7a0iFRRFLGtJI1FL2/mjEFh1HQt0vQcdKamruIpITl63dw14sLGdqrEzeddHjQcdKemruIBG7X3kpGF5bQJiuDxy8ZTHPN2RtMM3cRCZS7c/eLC1mxoYwXrhxGbruWQUcKBR0eRSRQk2evZtrctdxwYh+O7Z0TdJzQUHMXkcAsWbede/+0iON653DdCX2CjhMqau4iEogd5RWMLiyhfatMHr34aDKa6fPZE0kzdxFpdO7OHdNKWbVpJxOvGk5OdlbQkUJHO3cRaXQvvLuKGQvWccspfRl26AFBxwklNXcRaVSla7Zx/4wlFPTtzLX/c1jQcUJLzV1EGs223RWMnjCHnOwWPHzh0TTTnD1pNHMXkUbh7tz2x/ms21rO5GtG0LFNi6AjhZp27iLSKJ6e9RFvLF7PuNP7kdejY9BxQk/NXUSSbs6qLTz46lJOGZDLlcf1CjpOk6DmLiJJtWXnXq6bUELXDi35+QWDMNOcvTFo5i4iSVNd7dw8eR4by/Yy9dqv0L5VZtCRmgzt3EUkaZ56eyXFH2zgnrP6c1S39kHHaVLU3EUkKd79cBO/eP0DzhrYlW8NPyToOE1O1OZuZs+Y2edmtvBLbj/ezLaZ2bzI6d7ExxSRdLJhxx6unziXnge04cHzBmrOHoBYZu5/AMYDz9Wz5u/uflZCEolIWquqdm6cNJdtuyt49rtDyc7SS3tBiLpzd/e3gc2NkEVEQuDxt5bzjxWb+NHZR9C/a7ug4zRZiZq5jzCz+Wb2qpkdkaDHFJE0M2v5Rn711+Wcm3cwFw7pHnScJs3cPfois57ADHc/ch+3tQOq3b3MzM4AfuXu+/zUfTMbBYwCyM3NzS8qKoordFlZGdnZ2XHdNxWpntQVplogufVsKa/m3v/bTdsWxn3DW5HVPLlz9qb63BQUFMxx9yFRF7p71BPQE1gY49qPgZxo6/Lz8z1excXFcd83Fame1BWmWtyTV09FZZVf8NT/eb+7X/Xl67cn5WfU1VSfG2C2x9CLGzyWMbMDLfJSuJkNpWbUs6mhjysi6ePhN5fxz48285Nzj6R3l7ZBxxFieLeMmU0EjgdyzGwNcB+QCeDuTwHnA9eaWSWwG7g4cnQRkSageOnnPDlzJZcM7c45g7sFHUciojZ3d78kyu3jqXmrpIg0MZ9u3c1Nk+fRv2s77vu63kuRSvQbqiISl72V1YyZUEJllfPkpXm0zMwIOpLUot8uEJG4PPTaUuZ+spXxIwfTK6dN0HGkDu3cRWS/vbHoM3436yO+PeIQzhp4UNBxZB/U3EVkv6zevItb/zifgd3ac+eZ/YOOI19CzV1EYransorRhSU48MTIPLKaa86eqjRzF5GYPfDyEkrXbuO3l+XTvVProONIPbRzF5GY/Hn+pzz3ziqu+movTjniwKDjSBRq7iIS1YcbyrhjWil5PTrw/dP6BR1HYqDmLiL1Kq+ombNnZhjjR+aRmaG2kQ40cxeRev1g+iKWfraD319xDAd1aBV0HImRDsEi8qWmzllD0furGVNwGAV9uwQdR/aDmruI7NPy9Tu4+6WFDOvViZtOOjzoOLKf1NxF5At27a3k2sIS2mRl8Nglg2muOXva0cxdRP6Lu3P3iwtZuaGMF64cRm67lkFHkjjocCwi/2XS+6uZNnctN5zYh2N75wQdR+Kk5i4i/7b40+3cO30Rx/XO4boT9vlVyJIm1NxFBIAd5RWMmVBCh1aZPHrx0WQ0S+4XXEtyaeYuIrg746aW8snmXUz43jBysrOCjiQNpJ27iPD8u6t4uXQdt5xyOMMOPSDoOJIAau4iTdyCNVv58YwlFPTtzDVfOyzoOJIgau4iTdi2XRWMLiwhJ7sFD194NM00Zw8NzdxFmih357Yp8/lsWzmTrxlBxzYtgo4kCaSdu0gT9fSsj3hj8XrGnd6PvB4dg44jCabmLtIEzVm1hQdfXcqpR+Ry5XG9go4jSaDmLtLEbN65l7ETSujaoSUPnT8IM83ZwyhqczezZ8zsczNb+CW3m5k9ZmYrzGyBmeUlPqaIJEK1OzdPnsemsr08OTKf9q0yg44kSRLLzv0PwGn13H460CdyGgX8uuGxRCQZXvmwgpkfbOCerw/gqG7tg44jSRS1ubv728DmepacDTznNd4FOphZ10QFFJHEeO/DTUxdXsFZA7vyrWE9go4jSZaImfvBwOpal9dErhORFLFhxx6umziX3NbGg+cN1Jy9CWjU97mb2ShqRjfk5uYyc+bMuB6nrKws7vumItWTusJQS7U7v5hdzpad1dw6yJn9zqygIyVEGJ6b2hJdTyKa+1qge63L3SLXfYG7/xb4LcCQIUP8+OOPj+sHzpw5k3jvm4pUT+oKQy2PvLmMxZuW89B5A+myc2Xa1/MvYXhuakt0PYkYy0wHLo+8a2Y4sM3d1yXgcUWkgWYt38hjby3nvLxuXDCkW9BxpBFF3bmb2UTgeCDHzNYA9wGZAO7+FPAKcAawAtgFXJGssCISu/Xby7mhaC69O2dz/zeP0Jy9iYna3N39kii3OzAmYYlEpMEqq6q5bsJcdu2tYtLVebRuoY+Ramr0jIuE0MNvLuOfH2/mkYsG0btL26DjSAD08QMiIVO89HOenLmSS4Z255zBmrM3VWruIiGydutubpo8j/5d23Hf148IOo4ESM1dJCT2VlYzdkIJlVXOk5fm0TIzI+hIEiDN3EVC4qHXljL3k608MTKPXjltgo4jAdPOXSQEXlv4Gb+b9RHfHnEIZw7URzuJmrtI2vtk0y5umzKfgd3ac+eZ/YOOIylCzV0kje2prGLMhBIMeGJkHlnNNWeXGpq5i6SxB15eQunabfzv5UPo3ql10HEkhWjnLpKm/jz/U557ZxVXfbUXJw/IDTqOpBg1d5E09OGGMsZNXUBejw58/7R+QceRFKTmLpJmyiuqGF1YQovmzRg/Mo/MDP0zli/SzF0kzdz3p0Us/WwHf7jiGA7q0CroOJKidMgXSSNT56xh0uzVjC3ozfF9uwQdR1KYmrtImli2fgd3v7SQYb06ceNJfYKOIylOzV0kDezcU8nowhLaZGXw+CWDaa45u0ShmbtIinN37nlpISs3lFF45TC6tGsZdCRJAzr8i6S4Se+vZtrctdx44uF8pXdO0HEkTai5i6SwxZ9u597pi/hqnxzGntA76DiSRtTcRVLUjvIKxkwooWPrTB656GgymukLriV2mrmLpCB3Z9zUUj7ZvIuJVw0nJzsr6EiSZrRzF0lBz7+7ipdL13HrKX0Z2qtT0HEkDam5i6SYBWu28uMZSzihXxeu/tqhQceRNKXmLpJCtu2qYHRhCZ3bZvHLCwbRTHN2iZNm7iIpwt25dcp81m8vZ/LVI+jYpkXQkSSNxbRzN7PTzOwDM1thZuP2cft3zGyDmc2LnL6X+Kgi4fb0rI94c/F6xp3en8E9OgYdR9Jc1J27mWUATwAnA2uA981sursvrrN0kruPTUJGkdCbs2oLD766lFOPyOW7x/YMOo6EQCw796HACnf/0N33AkXA2cmNJdJ0bN65l7ETSjioQyseOn8QZpqzS8PF0twPBlbXurwmcl1d55nZAjObYmbdE5JOJOSqq52bJ89jU9lenrw0j/atMoOOJCFh7l7/ArPzgdPc/XuRy5cBw2qPYMzsAKDM3feY2dXARe5+wj4eaxQwCiA3Nze/qKgortBlZWVkZ2fHdd9UpHpSV7JrmbFyL1OWV3D5gBac0CP5jV3PTeqKtZ6CgoI57j4k6kJ3r/cEjABer3X5DuCOetZnANuiPW5+fr7Hq7i4OO77piLVk7qSWcs7Kzd6r3EzfOyEEq+urk7az6lNz03qirUeYLZH6a/uHtNY5n2gj5n1MrMWwMXA9NoLzKxrrYvfAJbE8LgiTdaGHXu4buJceh7Qhp+ee5Tm7JJwUd8t4+6VZjYWeJ2aXfkz7r7IzH5EzRFkOnC9mX0DqAQ2A99JYmaRtFZV7dw4aS7bd1fw3HeHkp2lXzeRxIvpb5W7vwK8Uue6e2udv4OacY2IRPHYX5fzjxWbeOi8gfTv2i7oOBJS+vgBkUb09+UbeOyt5ZyX140LhnQLOo6EmJq7SCNZv72cG4vm0adLNvd/8wjN2SWp1NxFGkFlVTXXTZjL7ooqnrw0j9YtNGeX5NLfMJFG8Ms3l/HPjzfzyEWD6N2lbdBxpAnQzl0kyd5aup5fz1zJJUN7cM5gzdmlcai5iyTR2q27uXnyfAZ0bcd9Xx8QdBxpQtTcRZJkb2U1YyeUUFnlPHlpHi0zM4KOJE2IZu4iSfKz15Yy95OtPDEyj545bYKOI02Mdu4iSfDaws94etZHfOcrPTlzYNfodxBJMDV3kQT7ZNMubpsyn0Hd2nPHGf2CjiNNlJq7SAKVV1QxesIcDBg/Mo+s5pqzSzA0cxdJoAdeXsLCtdv538uH0L1T66DjSBOmnbtIgkyf/ynPv7uKUV87lJMH5AYdR5o4NXeRBFi5oYw7pi4g/5CO3HZq36DjiKi5izRUeUUVYwpLaNG8GeNHDiYzQ/+sJHiauYs00H1/WsTSz3bwhyuOoWv7VkHHEQG0cxdpkClz1jBp9mrGFvTm+L5dgo4j8m9q7iJxWrZ+B3e/VMrwQztx40l9go4j8l/U3EXisHNPJaMLS8jOyuSxiwfTXHN2STGauYvsJ3fnrhdLWbmhjMIrh9GlXcugI4l8gbYbIvup6P3VvDTvU2466XC+0jsn6Dgi+6TmLrIfFn26jfumL+KrfXIYW9A76DgiX0rNXSRGO8orGFNYQsfWmTx60dE0a6YvuJbUpZm7SAzcnXFTS1m9ZTdFo4ZzQHZW0JFE6qWdu0gMnntnFS+XruO2U/tyTM9OQccRiSqm5m5mp5nZB2a2wszG7eP2LDObFLn9PTPrmeigIkGZv3orP355MSf268Korx4adByRmERt7maWATwBnA4MAC4xs7rf9HslsMXdewOPAD9LdFCRIOyscEYXltClbUt+eeEgzdklbcSycx8KrHD3D919L1AEnF1nzdnAs5HzU4ATzUz/CiStuTu/K93D5zvKGT9yMB1atwg6kkjMYnlB9WBgda3La4BhX7bG3SvNbBtwALAxESFr+9uyDdw5axdtSv6W6IcOzM5dqicVVVRV8/GmKu49awCDe3QMOo7IfmnUd8uY2ShgFEBubi4zZ87c78dYsaWK3KxqMmx3gtMFJ1v1pKbmcOQhTq+Kj5k5c1XQaRKirKwsrn93qShMtUAS6nH3ek/ACOD1WpfvAO6os+Z1YETkfHNqduxW3+Pm5+d7vIqLi+O+bypSPakrTLW4h6ueMNXiHns9wGyP0rfdPaaZ+/tAHzPrZWYtgIuB6XXWTAe+HTl/PvBWJISIiAQg6ljGa2boY6nZnWcAz7j7IjP7ETVHkOnA08DzZrYC2EzNAUBERAIS08zd3V8BXqlz3b21zpcDFyQ2moiIxEu/oSoiEkJq7iIiIaTmLiISQmruIiIhpOYuIhJCFtTb0c1sAxDvr/3lkISPNgiQ6kldYaoFwlVPmGqB2Os5xN07R1sUWHNvCDOb7e5Dgs6RKKondYWpFghXPWGqBRJfj8YyIiIhpOYuIhJC6drcfxt0gARTPakrTLVAuOoJUy2Q4HrScuYuIiL1S9edu4iI1CNtm7uZ3W9mC8xsnpm9YWYHBZ2pIczs52a2NFLTi2bWIehM8TKzC8xskZlVm1navpsh2hfDpxMze8bMPjezhUFnaSgz625mxWa2OPL37IagMzWEmbU0s3+a2fxIPT9MyOOm61jGzNq5+/bI+euBAe5+TcCx4mZmp1DzOfiVZvYzAHe/PeBYcTGz/kA18BvgVnefHXCk/Rb5YvhlwMnUfLXk+8Al7r440GBxMrOvAWXAc+5+ZNB5GsLMugJd3b3EzNoCc4BvpvFzY0Abdy8zs0xgFnCDu7/bkMdN2537vxp7RBsgPY9SEe7+hrtXRi6+C3QLMk9DuPsSd/8g6BwNFMsXw6cNd3+bmu9aSHvuvs7dSyLndwBLqPke57QU+YKlssjFzMipwf0sbZs7gJk9YGargUuBe6OtTyPfBV4NOkQTt68vhk/bBhJWZtYTGAy8F2yShjGzDDObB3wOvOnuDa4npZu7mf3FzBbu43Q2gLvf5e7dgUJgbLBpo4tWT2TNXUAlNTWlrFhqEUkmM8sGpgI31vmffNpx9yp3P5qa/7EPNbMGj85i+iamoLj7STEuLaTmm6LuS2KcBotWj5l9BzgLODHVv4N2P56bdLUW6F7rcrfIdZICIrPpqUChu08LOk+iuPtWMysGTgMa9OJ3Su/c62NmfWpdPBtYGlSWRDCz04DvA99w911B55GYvhheAhB5AfJpYIm7Pxx0noYys87/enecmbWi5kX8BvezdH63zFSgLzXvylgFXOPuabuziny5eBawKXLVu+n67h8zOwd4HOgMbAXmufupwabaf2Z2BvAo//li+AcCjhQ3M5sIHE/NJw+uB+5z96cDDRUnMzsO+DtQSs2/f4A7I9/1nHbMbCDwLDV/z5oBk939Rw1+3HRt7iIi8uXSdiwjIiJfTs1dRCSE1NxFREJIzV1EJITU3EVEQkjNXUQkhNTcRURCSM1dRCSE/h/cjmwJFR7zUgAAAABJRU5ErkJggg==\n", "text/plain": [ "
" ] }, "metadata": { "needs_background": "light" }, "output_type": "display_data" } ], "source": [ "plt.plot(x, backend.get_value(activations.relu(x, alpha=0.0, max_value=None) )) # default\n", "\n", "plt.grid(True)\n", "plt.show()" ] }, { "cell_type": "code", "execution_count": 6, "metadata": {}, "outputs": [ { "data": { "image/png": "iVBORw0KGgoAAAANSUhEUgAAAX8AAAD8CAYAAACfF6SlAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDMuMC4xLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvDW2N/gAAIABJREFUeJzt3Xt4VPW97/H3NwkJIQnXZCL3ayRQBRQELahJ0KpUyz6Knlpbd4utveyeuveWeuuW7tZTq6329Knd7dZWd0uPu1SUU5XqtiKkolUR3ASBhDtylQRCCOGSkOR7/sjoQxEkZCasmTWf1/PMw6yZNbM+Xyb55pffWlnL3B0REUktaUEHEBGRM0/NX0QkBan5i4ikIDV/EZEUpOYvIpKC1PxFRFKQmr+ISApS8xcRSUFq/iIiKSgj6AAnk5+f70OGDOnw6w8ePEhOTk78AgUoTLVAuOoJUy2gehJZe2tZvnz5HncvONV6Cdv8hwwZwrJlyzr8+vLyckpKSuIXKEBhqgXCVU+YagHVk8jaW4uZvdee99O0j4hIClLzFxFJQWr+IiIpSM1fRCQFqfmLiKSgmJu/mQ00s8VmtsbMVpvZbSdYx8zsZ2a2wcxWmtn5sW5XREQ6Lh6HejYDt7v7O2aWByw3s5fdfc0x61wFFEVvk4BfRv8VEZEAxNz83X0XsCt6/4CZVQL9gWOb/3RgjrddM/JNM+tpZn2jrxURiavmllZ+v3Qrb69v4p2mtUHHOW1n9cjmc5MGdeo2LJ7X8DWzIcCrwDnuXn/M4wuAB9z9tejyK8Cd7r7suNffCtwKUFhYOH7u3LkdztLQ0EBubm6HX59IwlQLhKueMNUC4ann+Y1NPLP+KIYDFnSc0zasRxr3XpT9N4+197MpLS1d7u4TTrVe3P7C18xygWeAfzy28Z8Od38MeAxgwoQJHstf5qXiX/YlizDVE6ZaIBz1rNlZz3Mvv8bVY/oyo1990tfzgXh/NnE52sfMutDW+J909/knWGUHMPCY5QHRx0RE4qapuZV/fmoFPbIzuW/6OUHHSWjxONrHgMeBSnf/yUlWew64OXrUz4XAfs33i0i8PbJoPVXvH+CBa8+lV05m0HESWjymfSYDXwDeNbMV0cfuAQYBuPu/Ay8A04ANwCHgS3HYrojIh1Zsq+MX5RuZMX4Al40uDDpOwovH0T6vcYo9KtGjfP4h1m2JiJzIkaMt3P7UCgrzsph9zeig4ySFhD2ls4hIez300lo21hzkd7dMpHvXLkHHSQo6vYOIJLWlm2t5/PXNfP7CQVxcdMprmEiUmr+IJK2Djc3MmlfBwF7duPuqUUHHSSqa9hGRpPXDFyvZtu8Qf7j1InKy1M5Oh0b+IpKUXl1Xw/99cyu3TB7KxKG9g46TdNT8RSTp7D98lDufWcnwghxmXTEy6DhJSb8niUjSuW/BGqoPNPLM1z9J1y7pQcdJShr5i0hSeXnNbp5evp2vXzqccQN7Bh0naan5i0jSqD3YxN3z36X4rDy+NbUo6DhJTdM+IpI07n12FfsPNzFn5kQyMzR2jYX+90QkKTxfsZM/rdzFP152NqP7dQ86TtJT8xeRhFd94Aj3PruKsQN78tVLhgUdJxTU/EUkobk798xfxeGmFh6+fiwZ6Wpb8aD/RRFJaM+8s4OFlbv59hUjGRFJ/ktMJgo1fxFJWDvrDvO951YzcWhvZk4eGnScUFHzF5GE5O7c+cxKWtx5aMZY0tKS70LsiUzNX0QS0pNvbWXJ+j3cM20Ug/p0CzpO6Kj5i0jC2br3EPe/UMnFRfncNGlQ0HFCSc1fRBJKa6sza14F6WnGg9eNwUzTPZ1BzV9EEsoTr29m6ZZavnvNJ+jXMzvoOKGl5i8iCWNDdQM/emktl40q5Lrz+wcdJ9TU/EUkITS3tHL7vApyMtO5/9pzNN3TyXRiNxFJCI++uomKbXX8/HPnEcnrGnSc0NPIX0QCt2ZnPT9duI6rx/Tl6jH9go6TEtT8RSRQTc1t0z09sjO5b/o5QcdJGZr2EZFA/eyV9VTuqudXN0+gV05m0HFShkb+IhKYFdvq+OVfNjJj/AAuH10YdJyUouYvIoE4crSF259aQWFeFrOvGR10nJSjaR8RCcRDL61lY81BfnfLRLp37RJ0nJQTl5G/mT1hZtVmtuokz5eY2X4zWxG9zY7HdkUkOS3dXMvjr2/m8xcO4uKigqDjpKR4jfx/A/wcmPMx6yxx96vjtD0RSVIHG5uZNa+Cgb26cfdVo4KOk7LiMvJ391eB2ni8l4iE2w9frGTbvkM8dP1YcrI08xwUc/f4vJHZEGCBu3/kQF0zKwGeAbYDO4FZ7r76BOvdCtwKUFhYOH7u3LkdztPQ0EBubjgu+RamWiBc9YSpFuj8elbtaeahZY1cMTiDG0dlddp2PhCmz6e9tZSWli539wmnXNHd43IDhgCrTvJcdyA3en8asP5U7zd+/HiPxeLFi2N6fSIJUy3u4aonTLW4d249dYea/ML7F3rZQ4v9cFNzp23nWGH6fNpbC7DM29Gzz8ihnu5e7+4N0fsvAF3MLP9MbFtEEsN9C9awu/4ID98wjq5d0oOOk/LOSPM3s7Mseoo+M5sY3e7eM7FtEQney2t28/Ty7XyjZATjBvYMOo4Qp6N9zOz3QAmQb2bbge8CXQDc/d+BGcDXzawZOAx8NvrriYiE3L6DTdw9/12Kz8rjW1OLgo4jUXFp/u5+4yme/zlth4KKSIq599lV7D/cxJyZE8nM0EkFEoU+CRHpNM9X7GTByl3cNrWI0f26Bx1HjqHmLyKdovrAEWY/u4qxA3rwtUuHBx1HjqPmLyJx5+7cM38VB5taePiGsWSkq9UkGn0iIhJ3z7yzg4WVu7njipGMiOQFHUdOQM1fROJqZ91hvvfcaiYO6c2XJg8NOo6chJq/iMSNu3PnMytpcefH148hPc2CjiQnoeYvInHz5FtbWbJ+D/dMG8XgPjlBx5GPoeYvInGxde8h7n+hkouL8rlp0qCg48gpqPmLSMxaW51Z8ypIN+PB68YQPZuLJDA1fxGJ2ROvb2bpllpmXzOafj2zg44j7aDmLyIx2VDdwI9eWstlowqZMX5A0HGkndT8RaTDmltauX1eBd0y07n/2nM03ZNEdA01EemwR1/dRMW2On7+ufOI5HUNOo6cBo38RaRD1uys56cL13H1mL5cPaZf0HHkNKn5i8hpa2pum+7pkZ3JfdM/ctluSQKa9hGR0/bIovVU7qrnVzdPoFdOZtBxpAM08heR01KxrY5flG9kxvgBXD66MOg40kFq/iLSbkeOtnD7vAoieVnce/XooONIDDTtIyLt9vCf17KhuoE5MyfSI7tL0HEkBhr5i0i7LN1cy69f28znLxzEJWcXBB1HYqTmLyKndLCxmVnzKhjYqxt3XzUq6DgSB5r2EZFTeuDFKrbtO8Tcr1xITpbaRhho5C8iH2vJ+hp+9+Z73DJ5KJOG9Qk6jsSJmr+InFT9kaPc8fRKhhfkMOuKkUHHkTjS728iclLff34Nu+uPMP8bk+naJT3oOBJHGvmLyAktXLObp5dv5+slwxk3sGfQcSTO1PxF5CP2HWzirvnvUnxWHrdNPTvoONIJNO0jIh9x77Or2H+4iTkzJ5KZoTFiGOlTFZG/sWDlThas3MVtU4sY3a970HGkk8Sl+ZvZE2ZWbWarTvK8mdnPzGyDma00s/PjsV0Ria+6xlbu/eMqxg7owdcuHR50HOlE8Rr5/wa48mOevwooit5uBX4Zp+2KSJy4O79d3cShphYevmEcGemaGAizuHy67v4qUPsxq0wH5nibN4GeZtY3HtsWkfh45p0d/Hd1C9++YiQjIrlBx5FOZu4enzcyGwIscPePXNbHzBYAD7j7a9HlV4A73X3ZcevdSttvBhQWFo6fO3duh/M0NDSQmxuOL+Aw1QLhqicstew93Mq/vH6Yft2c71yUQ1pILsQels8H2l9LaWnpcnefcKr1EupoH3d/DHgMYMKECV5SUtLh9yovLyeW1yeSMNUC4aonDLW4Ozc/sRRLa+Kr4zIpKy0NOlLchOHz+UC8azlTk3o7gIHHLA+IPiYiAXvyra0sWb+Hu6eNItJN8/yp4kx90s8BN0eP+rkQ2O/uu87QtkXkJLbuPcT9L1RycVE+n580KOg4cgbFZdrHzH4PlAD5ZrYd+C7QBcDd/x14AZgGbAAOAV+Kx3ZFpONaW51Z8ypIN+PB68ZgIZnnl/aJS/N39xtP8bwD/xCPbYlIfDzx+maWbqnloevH0q9ndtBx5AzTBJ9ICtpQ3cCPX1rLZaMiXHd+/6DjSADU/EVSTHNLK7fPqyA7M537rz1X0z0pKqEO9RSRzvfoq5uo2FbHIzeeRySva9BxJCAa+YukkMpd9fx04To+PaYv14ztF3QcCZCav0iKaGpu5Z+fqqBHdib3Tf/IH+JLitG0j0iKeGTReip31fOrmyfQOycz6DgSMI38RVJAxbY6flG+kevOH8DlowuDjiMJQM1fJOSOHG3h9nkVRPKymH3N6KDjSILQtI9IyD3857VsqG5gzsyJ9MjuEnQcSRAa+YuE2NLNtfz6tc18btIgLjm7IOg4kkDU/EVC6mBjM7PmVTCgVzbfmTYq6DiSYDTtIxJSD7xYxbZ9h5j7lQvJydK3uvwtjfxFQmjJ+hp+9+Z7zJw8lEnD+gQdRxKQmr9IyNQfOcodT69kWEEO375iZNBxJEHpd0GRkLnv+TXsrj/C/G9MpmuX9KDjSILSyF8kRBau2c285dv5RskIxg3sGXQcSWBq/iIhse9gE3fNf5fis/L41tSioONIgtO0j0hI3PvsKvYfbmLOzIlkZmhcJx9PXyEiIbBg5U4WrNzFbVOLGN2ve9BxJAmo+YskuZoDjdz7x1WMHdCDr106POg4kiTU/EWSmLtz9/x3OdjUwsM3jCUjXd/S0j76ShFJYvPf2cHCyt3cccVIRkTygo4jSUTNXyRJ7dp/mH99fjUXDOnFlyYPDTqOJBk1f5Ek5O7c8fRKmluch64fS3qaBR1Jkoyav0gS+s+lW1myfg/3TCtmcJ+coONIElLzF0kyW/ce4gd/qmTKiHxumjQ46DiSpNT8RZJIa6sz6+kK0s14cMYY0jTdIx2k5i+SRP7jr1tYurmW2deMpn/P7KDjSBJT8xdJEhuqG/jRf1UxtTjCjPEDgo4jSS4uzd/MrjSztWa2wczuOsHzXzSzGjNbEb19OR7bFUkVzS2t3D6vguzMdH547bmYabpHYhPzid3MLB34N+ByYDvwtpk95+5rjlv1D+7+zVi3J5KKHn11ExXb6njkxvOIdO8adBwJgXiM/CcCG9x9k7s3AXOB6XF4XxEBKnfV89OF6/j0mL5cM7Zf0HEkJMzdY3sDsxnAle7+5ejyF4BJx47yzeyLwA+BGmAd8E/uvu0E73UrcCtAYWHh+Llz53Y4V0NDA7m5uR1+fSIJUy0Qrno6u5bmVud7bxxhf2MrP5jSjbzMzp3uCdNnA+Gqp721lJaWLnf3Cadc0d1jugEzgF8fs/wF4OfHrdMHyIre/yqw6FTvO378eI/F4sWLY3p9IglTLe7hqqeza3nopSoffOcC//Pq9zt1Ox8I02fjHq562lsLsMzb0bvjMe2zAxh4zPKA6GPH/oDZ6+6N0cVfA+PjsF2RUKvYVscvyjdy3fkDuHx0YdBxJGTi0fzfBorMbKiZZQKfBZ47dgUz63vM4meAyjhsVyS0jhxt4fZ5FUTysph9zeig40gIxXy0j7s3m9k3gZeAdOAJd19tZt+n7deP54BvmdlngGagFvhirNsVCbOfvLyODdUNzJk5kR7ZXYKOIyEUl2v4uvsLwAvHPTb7mPt3A3fHY1siYff2llp+tWQTN00axCVnFwQdR0JKf+ErkkAONTUza14FA3plc8+0UUHHkRCLy8hfROLjgRer2Fp7iLlfuZCcLH17SufRyF8kQby2fg9z3niPmZOHMmlYn6DjSMip+YskgPojR7nj6QqGFeTw7StGBh1HUoB+rxRJAP97wRrerz/C/G9MpmuX9KDjSArQyF8kYK9U7uapZdv5eslwxg3sGXQcSRFq/iIB2newibvmv0vxWXl8a2pR0HEkhWjaRyRAs59bzb6DTfzmSxeQlaHpHjlzNPIXCcifVu7i+Yqd3Da1iE/06xF0HEkxav4iAag50Mi//PFdxgzowddLhgcdR1KQmr/IGebu3D3/XQ42tfDw9WPJSNe3oZx5+qoTOcPmv7ODhZW7+fanRlJUmBd0HElRav4iZ9Cu/Yf51+dXc8GQXsycMjToOJLC1PxFzhB3546nV9Lc4jx0/VjS0zr3kowiH0fNX+QM+c+lW1myfg/3TCtmcJ+coONIilPzFzkDtu49xA/+VMmUEfncNGlw0HFE1PxFOltrqzPr6QrSzXhwxhjSNN0jCUDNX6ST/cdft7B0cy2zrxlN/57ZQccRAdT8RTrVhuoGfvRfVVw2KsKM8QOCjiPyITV/kU7S3NLK7fMqyM5M5/5rz8VM0z2SOHRiN5FO8uirm6jYVsfPbjyPSF7XoOOI/A2N/EU6QeWuen66cB2fHtOXz4ztF3QckY9Q8xeJs6bmVv75qQp6ZHfhvunnBB1H5IQ07SMSZ48sWk/lrnp+dfMEeudkBh1H5IQ08heJo4ptdfyifCPXnT+Ay0cXBh1H5KTU/EXi5MjRFm6fV0FBbhazrxkddByRj6VpH5E4+cnL69hQ3cCcmRPpkd0l6DgiH0sjf5E4eHtLLb9asombJg3ikrMLgo4jckpq/iIxOtTUzKx5FQzolc3d00YFHUekXeLS/M3sSjNba2YbzOyuEzyfZWZ/iD7/lpkNicd2RRLBAy9WsbX2ED+eMZbcLM2kSnKIufmbWTrwb8BVwGjgRjM7fm/XLcA+dx8B/B/gwVi3K5IIVu9pYc4b7zFz8lAuHNYn6Dgi7RaPkf9EYIO7b3L3JmAuMP24daYDv43efxqYajrRiSSx1lZnxbY6Hl/VyLCCHL59xcigI4mcFnP32N7AbAZwpbt/Obr8BWCSu3/zmHVWRdfZHl3eGF1nz3HvdStwK0BhYeH4uXPndjhXQ0MDubm5HX59IglTLZC89RxudlbtaaGipoWVNc3UN0GXNOeuidkM75kedLy4SNbP5mTCVE97ayktLV3u7hNOtV5CTVC6+2PAYwATJkzwkpKSDr9XeXk5sbw+kYSpFkiuejbVNLCoqppFVdW8vaWWoy1Oj+wulIzqR1lxhIyadVz9qdKgY8ZNMn027RGmeuJdSzya/w5g4DHLA6KPnWid7WaWAfQA9sZh2yJx1dTcytLNtdGGv5stew8BMLIwj1umDKOsOML5g3qSkd42Y1pevj7IuCIdFo/m/zZQZGZDaWvynwU+d9w6zwF/D7wBzAAWeazzTSJxUn3gCOVra1hUWc1rG/bQ0NhMZkYanxzeh1umDKW0OMKAXt2CjikSVzE3f3dvNrNvAi8B6cAT7r7azL4PLHP354DHgd+Z2QaglrYfECKBaG11Vu+s55Wq3SyuqqZi+34Azurelc+M60fZyAifHNGHbpkJNSsqEldx+ep29xeAF457bPYx948A18djWyId0dDYzGvr97C4qppFa6upOdCIGYwb2JPbLz+bslERRvftrqttScrQ0EZC6729B3mlsprFa6t5a1MtTS2t5GVlcMnIAspGRigZWUCf3KygY4oEQs1fQuNoSytvb6llcVU1r1RVs6nmIADDC3L44uQhlI6MMGFIL7qk66wmImr+ktT2NDRSvraGxVXVvLquhgONzWSmpzFpWG9uvnAwZcWFDOqjnbUix1Pzl6Ti3raz9oNj7yu21+EOkbwsPj2mL6XFEaaMyCdH59gR+Vj6DpGEd6ipmdc37GVR1W4WVVWzu74RgLEDe/JPl51NWXGET/TTzlqR06HmLwlpW+0hFq+t5pXKat7YtJem5lZyszK4uCifsuIIJSMjFORpZ61IR6n5S0Jobmll+Xv7WLS2mkWV1ayvbgBgWH4OX7hwMGXFES4Y0pvMDO2sFYkHNX8JTO3BJv6yrppFVTX8ZW019Uea6ZJuTBzam89OHETpyAKGFYTjpFwiiUbNX84Yd6fq/QMs2NjEI5V/5b+37qPVIT83k0994iymFkeYUpRPXldd/1aks6n5S6c63NTCXzfuYVFVNYurqtm5/wgA5/Zv5X+VFVFWHOHc/j1IS9POWpEzSc1f4m5H3eG2QzErd/PXjXtpbG6lW2Y6Fxflc9tlRWTt3cDfXTkl6JgiKU3NX2LW0ur899Z9Hx57X/X+AQAG9e7GjRMHMXVUhIlDe5OV0XbBk/LyTUHGFRHU/KWD6g418Zd1bX9ZW76uhrpDR8lIMyYM6cV3po2itDjC8IIcHXsvkqDU/KVd3J311Q1tJ0qrqmb51n20tDq9czIpK45QVhzh4qICemRrZ61IMlDzl5M6crSFNzbtbTtRWmU1O+oOAzC6b3e+UTKc0uIIYwf0JF07a0WSjpq//I1d+w9/eGTOaxv2cORoK9ld0pk8Ip9/KB1BaXEBfXtkBx1TRGKk5p/iWlqdFdvqPjwNcuWuegAG9MrmhgkDKS2OcNGwPnTtkh5wUhGJJzX/FLT/8FGWrG+7Zm35uhpqDzaRnmaMH9SLu64qpqw4QlEkVztrRUJMzT8FuDsbaw5GR/e7eXtL287ant26UHJ2AWWjCrm0qIAe3bSzViRVqPmHVGNzC29tqm2bv19bzXt7DwFQfFYeX71kGGXFEc4b1Es7a0VSlJp/iOyuP9J2gfLoztpDTS1kZaQxeUQ+X7l4GKXFEfr31M5aEVHzT2qtrc7KHfujf1m7m1U72nbW9uvRlWvP709ZcYSLhuWTnamdtSLyt9T8k8yBI0d5bf0eXqmqpnxtNXsamkgzOG9QL759xUimjoowsjBPO2tF5GOp+SeB9w+28uslm1i8tpqlm2s52uJ075rBpSMjTC2OcOnZBfTKyQw6pogkETX/BNTU3MrbW2rbTqWwtprNew4DlZxdmMvMKUMpGxlh/OBeZKTrqlYi0jFq/gmi5kAji9e2/WXtkvV7aGhsJjMjjYuG9WFywVG+es1kBvbuFnRMEQkJNf+AtLY6q3fW80rVbhZXVVOxfT8AZ3XvyjVj+1FWHGHyiD50y8ygvLxcjV9E4krN/wxqaGzmtfV72g7HXFtNzYFGzGDcwJ7M+tTZlBZHGN23u3bWikinU/PvZO/tPfjhRU7e2lRLU0sreVkZXHJ2AWXFES4dWUB+blbQMUUkxcTU/M2sN/AHYAiwBbjB3fedYL0W4N3o4lZ3/0ws201kR1taWbZlH4uqdrOoqpqNNQcBGF6Qw99/cjBlxYVMGNKLLtpZKyIBinXkfxfwirs/YGZ3RZfvPMF6h919XIzbSlh7GxopX1vDorXVvLquhgNHmslMT2PSsN58/sLBlBVHGNwnJ+iYIiIfirX5TwdKovd/C5Rz4uYfKu7Oml31LKpsm7tfsa0OdyjIy2LaOX0pGxVhyoh8crI0qyYiicncveMvNqtz957R+wbs+2D5uPWagRVAM/CAu//xJO93K3ArQGFh4fi5c+d2OFtDQwO5ubkdfv3xGpudNbUtVFS3UFHTwr7Gtv+3oT3SGFeQztiCdAZ1TyOtE3bWxruWoIWpnjDVAqonkbW3ltLS0uXuPuFU651yaGpmC4GzTvDUd45dcHc3s5P9JBns7jvMbBiwyMzedfeNx6/k7o8BjwFMmDDBS0pKThXvpMrLy4nl9QDbag+xeG3bJQzf2LSXpuZWcrMyuLiokNLiCCUjC4jkdY1pG+0Rj1oSSZjqCVMtoHoSWbxrOWXzd/fLTvacme02s77uvsvM+gLVJ3mPHdF/N5lZOXAe8JHmH7Tmllbe2Vr34bH363Y3ADA0P4cvROfuLxjSm8wM7awVkeQW66T0c8DfAw9E/332+BXMrBdwyN0bzSwfmAz8KMbtxs2+g038ZV0Ni6qq+cu6GvYfPkqXdGPi0N78zwsGUVYcYWi+dtaKSLjE2vwfAJ4ys1uA94AbAMxsAvA1d/8yMAp41MxagTTa5vzXxLjdDnN3qt4/8OFFyt/Zuo9Wh/zcTC4fXcjU4ghTivLJ66qrWolIeMXU/N19LzD1BI8vA74cvf9X4NxYthOrw00tvLFpT9uJ0qqq2bn/CADn9O/ON8uKKCuOMKZ/D9J0VSsRSRGhPRZx7+FWfvfmeyyuqub1DXtobG6lW2Y6U0bkc9tlRZSMjFDYvfN31oqIJKLQNf8ddYe55TdvU/X+YWAVg3p348aJg5g6KsLEob3JytBVrUREQtf8C/Oy6N8zm7E9GvnKpy9ieEGOTpQmInKc0B2zmJGexuNfvICrhnZhRCRXjV9E5ARC1/xFROTU1PxFRFKQmr+ISApS8xcRSUFq/iIiKUjNX0QkBan5i4ikIDV/EZEUFNOVvDqTmdXQdqbQjsoH9sQpTtDCVAuEq54w1QKqJ5G1t5bB7l5wqpUStvnHysyWtedSZskgTLVAuOoJUy2gehJZvGvRtI+ISApS8xcRSUFhbv6PBR0gjsJUC4SrnjDVAqonkcW1ltDO+YuIyMmFeeQvIiInEdrmb2b3mdlKM1thZn82s35BZ4qFmf3YzKqiNf0/M+sZdKZYmNn1ZrbazFrNLCmPxjCzK81srZltMLO7gs4TCzN7wsyqzWxV0FliZWYDzWyxma2Jfo3dFnSmWJhZVzNbamYV0Xq+F5f3Deu0j5l1d/f66P1vAaPd/WsBx+owM/sUsMjdm83sQQB3vzPgWB1mZqOAVuBRYJa7Lws40mkxs3RgHXA5sB14G7jR3dcEGqyDzOwSoAGY4+7nBJ0nFmbWF+jr7u+YWR6wHPi7JP5sDMhx9wYz6wK8Btzm7m/G8r6hHfl/0PijcoCk/inn7n929+bo4pvAgCDzxMrdK919bdA5YjAR2ODum9y9CZgLTA84U4e5+6tAbdA54sHdd7n7O9H7B4BKoH+wqTrO2zREF7tEbzH3s9A2fwAz+4GZbQNuAmYHnSeOZgIvBh0ixfUHth2zvJ0kbjBhZWZDgPOAt4JNEhszSzezFUBmeGg5AAABjUlEQVQ18LK7x1xPUjd/M1toZqtOcJsO4O7fcfeBwJPAN4NNe2qnqie6zneAZtpqSmjtqUeks5hZLvAM8I/HzQQkHXdvcfdxtP3GP9HMYp6ay4g9VnDc/bJ2rvok8ALw3U6ME7NT1WNmXwSuBqZ6EuysOY3PJxntAAYeszwg+pgkgOjc+DPAk+4+P+g88eLudWa2GLgSiGnnfFKP/D+OmRUdszgdqAoqSzyY2ZXAHcBn3P1Q0HmEt4EiMxtqZpnAZ4HnAs4kfLiD9HGg0t1/EnSeWJlZwQdH95lZNm0HGcTcz8J8tM8zwEjajih5D/iauyftyMzMNgBZwN7oQ28m+dFL/wN4BCgA6oAV7n5FsKlOj5lNA34KpANPuPsPAo7UYWb2e6CEtjNH7ga+6+6PBxqqg8xsCrAEeJe273+Ae9z9heBSdZyZjQF+S9vXWRrwlLt/P+b3DWvzFxGRkwvttI+IiJycmr+ISApS8xcRSUFq/iIiKUjNX0QkBan5i4ikIDV/EZEUpOYvIpKC/j+WSW835tJ8qgAAAABJRU5ErkJggg==\n", "text/plain": [ "
" ] }, "metadata": { "needs_background": "light" }, "output_type": "display_data" } ], "source": [ "plt.plot(x, backend.get_value(activations.relu(x, alpha=0.2, max_value=2.0) )) # custom relu with plateau and tail\n", "\n", "plt.grid(True)\n", "plt.show()" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "# Activation functions with plateau\n", "\n", "The following activation functions share the property of having a max plateu value for positive/negative values. \n", "These plateau functions are best used for \"normalized\" input and/or to put some \"normalization\" over output values." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## sigmoid" ] }, { "cell_type": "code", "execution_count": 7, "metadata": {}, "outputs": [ { "data": { "image/png": "iVBORw0KGgoAAAANSUhEUgAAAXcAAAD8CAYAAACMwORRAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDMuMC4xLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvDW2N/gAAIABJREFUeJzt3Xl4VPW9x/H3NyEsEvYl7HtAEVAkLGrVoKBovS7VtoqgKEvVorba9tq69Nbu7bX39t7SKiqKylKtG1ZcW5BWBQJEZIcQISQsgbCGmHW+94/E3pQiGcIkZ2byeT1PnmeWk8nnx8CHk9+c8zvm7oiISHxJCDqAiIhEnspdRCQOqdxFROKQyl1EJA6p3EVE4pDKXUQkDtVY7mY2y8zyzWztFzxvZvY/ZpZlZp+Y2TmRjykiIicjnD33Z4BxJ3j+ciC16msa8IdTjyUiIqeixnJ39yXA/hNscjXwrFdaCrQ2s86RCigiIievUQReoyuwo9r93KrHdh27oZlNo3LvnmbNmg3r3r17rX5gKBQiISF+Pi6Ip/HE01hA44lm8TQWCH88mzdv3ufuHWraLhLlHjZ3nwnMBEhLS/MVK1bU6nUWL15Menp6BJMFK57GE09jAY0nmsXTWCD88ZjZ9nBeLxL/7eUB1XfBu1U9JiIiAYlEuS8Abq46amYUcMjd/2VKRkRE6k+N0zJmNg9IB9qbWS7wQyAJwN0fAxYCVwBZQBFwa12FFRGR8NRY7u5+Yw3PO/DNiCUSEZFTFj8fNYuIyD+o3EVE4pDKXUQkDqncRUTikMpdRKQeHS0pp6i0vM5/Tr2eoSoi0pCEQs7WvYVk5hwkc8dBMnMOsHnPEX5x3RC+lla75VfCpXIXEYmQ4rIKPsk9RMa2/WRs28/K7Qc4Uly5l96yaSPO7tGGy87sxJBureo8i8pdRKSWSsoryMw5yIdZ+/gou4DVuYcoLQ8BkNoxmSuHdGFYzzYM7dGa3u2ak5Bg9ZZN5S4iEqZQyFm38zB/y9rLh1kFZGzbT0l5iASDwV1bccu5PRneqy1pvdrStnnjQLOq3EVETuDA0VKWbNnL+5v3smTzXvYVlgIwIKUFN47owfn92jOyT1taNk0KOOk/U7mLiBwjK7+Qd9fv4b0Ne8jMOUDIofVpSVyY2oH0AR34Ump7OrZoGnTME1K5i0iDFwo5mTsO8M66Pby7fg/Z+44CMKhrS+66OJX0AR0Y0q01ifU4Z36qVO4i0iBVFvpB3vhkF2+u3cWuQ8UkJRqj+rRj0vm9GHNGCl1aNws6Zq2p3EWkwXCv/ED01cw8Fq7Zxc5DxTROTODC/h343rgBXHJGStTNndeWyl1E4t6uQ5/xauZOXsnMZfOeQpISjQtTO/CdywYwZmD8FHp1KncRiUvFZRW8vW43L6zYwYdbC3CHc3q05sfXDOLKwZ1pE/ChinVN5S4icWXzniPMW57DK5l5HCwqo3vbZtx9cSrXDu1Kr/bNg45Xb1TuIhLzSsorWLhmF899tJ1VOQdJSjQuPbMTNw7vwXl929XrmaHRQuUuIjFrz+Fi5izdztzlOewrLKVP++Y8cMUZfOWcrrRLbhJ0vECp3EUk5mTmHOD3Hxez6p2/UuHOxQM6cvN5vbigX/sGuZd+PCp3EYkJoZCzaFM+jy/JZvmn+2nWCCad15uJ5/akZ7uGM5ceLpW7iES10vIQr32cx8wl2WzJL6RLq6Y8dOVAuhRv4/IxA4OOF7VU7iISlYrLKnhxxQ5+v3gruw4Vc3qnFvzX18/iyiFdSEpMYPHi7UFHjGoqdxGJKsVlFcxfnsNj72ez+3Axw3q24edfGcxF/Ttgpvn0cKncRSQqlJRXMG9ZDr9fvJX8IyWM6NWWR792Fuf1badSrwWVu4gEqiLkvJqZx2/e3Uzewc8Y2bst/33D2ZzbR6V+KlTuIhIId+e9Dfn8+u2NbN5TyKCuLfnFdYP5Ur/2KvUIULmLSL3LzDnAT97YwMrtB+jdvjkzxp/D5YM66Rj1CFK5i0i92XXoM3711iZeycyjfXITfnrtIL6W1p2kxISgo8UdlbuI1LnPSit4fMlWHnt/KyGHb47uyx3p/UhuogqqK/qTFZE64+4sXLObn7yxnl2HivnykM7cP+50urc9LehocU/lLiJ14tN9R3n4tbX8bcs+BnZuyf/cOJThvdoGHavBULmLSEQVl1Xw+8VbeWzxVpo0SuA//m0gE0b1pJHm1euVyl1EIuZvW/bywCtrydlfxNVnd+GBK86gY8umQcdqkFTuInLKDhWV8ZM31vPiylz6tG/O3CkjOa9f+6BjNWhhlbuZjQN+CyQCT7r7L455vgcwG2hdtc397r4wwllFJAq9tXYXD722jv1HS7kzvS93X5JK06TEoGM1eDWWu5klAjOAsUAukGFmC9x9fbXNHgRecPc/mNlAYCHQqw7yikiU2HukhIdfW8uba3czsHNLnp40nEFdWwUdS6qEs+c+Ashy92wAM5sPXA1UL3cHWlbdbgXsjGRIEYkub63dzQ9eWUNhSTnfGzeAqRf00YlIUcbc/cQbmF0PjHP3KVX3JwIj3X16tW06A+8AbYDmwBh3X3mc15oGTANISUkZNn/+/FqFLiwsJDk5uVbfG43iaTzxNBbQeI5VVObM2VDKBzvL6dUygalDmtA1OZhSb6jvzejRo1e6e1qNG7r7Cb+A66mcZ//8/kTgd8dscy9wX9Xtc6ncq0840esOGzbMa2vRokW1/t5oFE/jiaexuGs81X2wZa+f+7P3vM/33/BH39nkpeUVkQtWCw31vQFWeA297e5hTcvkAd2r3e9W9Vh1k4FxVf9ZfGRmTYH2QH4Yry8iUaykvIJfv7WJJ//+KX3aN+elO87j7O6tg44lNQin3DOAVDPrTWWp3wCMP2abHOAS4BkzOwNoCuyNZFARqX/Zewu5e34ma/MOM3FUT35wxRk0a6wjYWJBjeXu7uVmNh14m8rDHGe5+zoze4TKXw8WAPcBT5jZt6n8cHVS1a8PIhKjXlqZy0OvraVxowRmThzGpWd2CjqSnISwjnP3ymPWFx7z2MPVbq8Hzo9sNBEJQmFJOQ+9upZXMvMY0bstv73hbDq3ahZ0LDlJOkNVRP5hw67D3DlnFdsLjvKtMancdXEqibqARkxSuYsIUDkN88Cra2jRNIm5U0cxqk+7oCPJKVC5izRwxWUV/Oj19cxbnsPI3m353/FD6dhCi33FOpW7SAO2Y38Rd8xZydq8w9x+UV++c2l/Lc0bJ1TuIg3U37fs45tzVxFy54mb0xg7MCXoSBJBKneRBsbdeervn/KzhRvo1zGZmRPT6NW+edCxJMJU7iINSGmFc98Lq3k5M4/Lzkzh0a+drYtUxym9qyINxK5Dn/HzZcV8ejiPe8f2Z/rofiToMMe4pXIXaQAycw4w9dmVFH4W0tmmDYQ+FheJc6+v3snXZy6lWeMEHhrVTMXeQKjcReKUu/M/f9nCXfMyOatbK16983y6ttA/+YZC0zIicaikvIL7X1rDK5l5XDu0K7+4bjBNGmk1x4ZE5S4SZ/YfLeUbz60gY9sB7hvbn+kX98NMH5w2NCp3kTiSU1DEpKeXk3vwM343fihXDukSdCQJiMpdJE58knuQ257JoDzkzJ0ykrRebYOOJAFSuYvEgUUb87lzziraJTdm9m0j6Nshfi4cLbWjcheJcfOX5/DAq2s5o3MLZk0arhUdBVC5i8SsykMds/iv9zZzUf8O/P6mc2iupQSkiv4miMSgUMj50evrmP3Rdq47pxu/uG4wSVqqV6pRuYvEmNLyEPe9uJrXV+9k2oV9+P7lp+tQR/kXKneRGFJUWs7tz69iyea93H/56dx+Ud+gI0mUUrmLxIiDRaXc+kwGq3cc5FfXDeFrw7sHHUmimMpdJAbkHy5mwlPL2FZQxB8mDOMyLf4lNVC5i0S53ANFTHhyGflHSnh60nDO79c+6EgSA1TuIlHs031HuemJpRwpKee5ySMZ1rNN0JEkRqjcRaLUxt2HmfDkckLuzJs6ikFdWwUdSWKIyl0kCq3ecZCbZy2naVIC86eMol/HFkFHkhijcheJMiu37+eWWRm0Pi2JuVNG0aPdaUFHkhikcheJIsuyC7jtmQw6tmzK3Kkj6dyqWdCRJEbpfGWRKPFh1j4mPZ1Bp1ZNmT9tlIpdTonKXSQKLNm8l1ufyaB722bMn3YuKS21sqOcGk3LiARs0cZ8vvH8Svp2SOb5ySNol9wk6EgSB1TuIgH668Y93P7cKvp3Sub5ySNpfVrjoCNJnFC5iwRk0cZ8bn9uFQM6teD5ySNpdVpS0JEkjoQ1525m48xsk5llmdn9X7DN18xsvZmtM7O5kY0pEl8WbcznG8+tVLFLnalxz93MEoEZwFggF8gwswXuvr7aNqnA94Hz3f2AmXWsq8Aise7zYv98KkbFLnUhnD33EUCWu2e7eykwH7j6mG2mAjPc/QCAu+dHNqZIfFi06f+Lfc7kUSp2qTPm7ifewOx6YJy7T6m6PxEY6e7Tq23zKrAZOB9IBP7D3d86zmtNA6YBpKSkDJs/f36tQhcWFpKcHD9Xd4+n8cTTWCCy41mzt5zfriqha4sEvpvWlOTG9X/1pHh6f+JpLBD+eEaPHr3S3dNq2i5SH6g2AlKBdKAbsMTMBrv7weobuftMYCZAWlqap6en1+qHLV68mNp+bzSKp/HE01ggcuP5IGsfv3svg/6dWjJ3anBHxcTT+xNPY4HIjyecaZk8oPolX7pVPVZdLrDA3cvc/VMq9+JTIxNRJLYtzS5g8uwMerVrzvNTdLij1I9wyj0DSDWz3mbWGLgBWHDMNq9SudeOmbUH+gPZEcwpEpNWbNvPbc9k0K3NacyZOpK2zVXsUj9qLHd3LwemA28DG4AX3H2dmT1iZldVbfY2UGBm64FFwHfdvaCuQovEglU5ByrXimnZlLlTRtJeZ55KPQprzt3dFwILj3ns4Wq3Hbi36kukwVuTe4hbZi2nXXJj5k4dRUetFSP1TAuHiUTYxt2HmThrGS2bJjF36ig6tVKxS/1TuYtEUFZ+IROeXEbTRonMmzqKrq21bK8EQ+UuEiHbC45y05NLAWPO1JG6gpIESuUuEgF5Bz9j/BPLKC0PMWfKSPp2iJ+TayQ2aVVIkVOUf7iYm55YyuHiMuZNHcWATrqYtQRPe+4ip6CgsISbnlzG3iMlzL5tBIO6tgo6kgigPXeRWjv0WRk3z1pOzv4iZt82gnN6tAk6ksg/aM9dpBaOlpRz69PL2bznCI9PHMaoPu2CjiTyT7TnLnKSissqmDJ7BatzDzFj/DmkD9DlCyT6aM9d5CSUloe44/mVLP20gEe/ehbjBnUKOpLIcancRcJUXhHinvmZLNq0l59dO5hrhnYNOpLIF1K5i4QhFHK+99InvLl2Nw9++QxuHNEj6EgiJ6RyF6mBu/PDBet4eVUe947tz5QL+gQdSaRGKneRE3B3fvHWRp5bup1vXNiHuy7uF3QkkbDoaBmRE3g9u4yXt2QzYVQP7r/8dMzq/7qnIrWhPXeRLzDr75/y8pYyvjK0K49cNUjFLjFF5S5yHH/MyOGRP69nWEoiv7p+CAkJKnaJLZqWETnGgtU7uf/lNVzUvwMTeh2lUaL2gST26G+tSDXvrd/DvX/8mOG92vLYhGEkaY9dYpTKXaTKB1n7uHPuKs7s0pKnbkmjWePEoCOJ1JrKXQRYuX0/U2avoE/75sy+bQQtmiYFHUnklKjcpcFbm3eISbMy6NyqKc9NHknr0xoHHUnklKncpUHbvOcIE59aRstmSTw/ZSQdWjQJOpJIRKjcpcHatu8oNz25jMaNEpg7dSRdWjcLOpJIxKjcpUHKPVDETU8uoyLkzJkykp7tmgcdSSSiVO7S4OQfLmbCk8s4UlzGc5NH0K+jLmgt8UflLg3K/qOlTHiq8oLWz9w2gjO76ILWEp90hqo0GIeKypjw5DK2FxTxzK26oLXEN+25S4NwpLiMm59eTlZ+ITNvTuPcvrqgtcQ3lbvEvaLSciY/s4J1eYeYcdM5XNS/Q9CRROqcyl3iWnFZBdOeXcmK7fv57xvOZuzAlKAjidQLzblL3CotD3HnnFV8sHUf/3n9WVw5pEvQkUTqjfbcJS6VVYSYPncVf92Yz0+vGcx1w7oFHUmkXqncJe6UV4S4Z34m76zfw4+uOpPxI3sEHUmk3qncJa5UhJx7X1jNwjW7efDLZ3DLeb2CjiQSiLDK3czGmdkmM8sys/tPsN11ZuZmlha5iCLhqQg5331xNQtW7+Tfx53OlAv6BB1JJDA1lruZJQIzgMuBgcCNZjbwONu1AO4BlkU6pEhNQiHn+y9/wsuZedw3tj93pPcNOpJIoMLZcx8BZLl7truXAvOBq4+z3Y+BXwLFEcwnUqNQyHng1bW8sCKXuy/ux12XpAYdSSRw5u4n3sDsemCcu0+puj8RGOnu06ttcw7wgLtfZ2aLge+4+4rjvNY0YBpASkrKsPnz59cqdGFhIcnJybX63mgUT+Op77GE3Hl2XSmLc8u5sk8S16UmYRa5657G03sD8TWeeBoLhD+e0aNHr3T3Gqe+T/k4dzNLAH4DTKppW3efCcwESEtL8/T09Fr9zMWLF1Pb741G8TSe+hxLKOQ8+NpaFufmcGd6X7572YCIFjvE13sD8TWeeBoLRH484UzL5AHdq93vVvXY51oAg4DFZrYNGAUs0IeqUpc+L/a5y+qu2EViWTjlngGkmllvM2sM3AAs+PxJdz/k7u3dvZe79wKWAlcdb1pGJBJCIeehqmK/Q8Uuclw1lru7lwPTgbeBDcAL7r7OzB4xs6vqOqBIdZ/vsc+pKvbvqdhFjiusOXd3XwgsPOaxh79g2/RTjyXyrypCzv0vfcKLK3M1FSNSAy0cJjGhvCLEd//0Ca9k5nHPJal8a0yqil3kBFTuEvXKKkLc+8JqXl+9k+9c2p/pF+s4dpGaqNwlqpWWVy4C9uba3Xz/8tP5xkU681QkHCp3iVrFZRVMn7uK9zbk8/CVA7ntS72DjiQSM1TuEpWOlpQz7bkVfJBVwI+vGcTEUT2DjiQSU1TuEnUOfVbGrU8v5+MdB3n0q2fpQhsitaByl6hSUFjCxKeWsyX/CDPGn8PlgzsHHUkkJqncJWrsPlTMhKeWsWN/EU/cnEb6gI5BRxKJWSp3iQrb9h1l4qxl7C8sZfZtIxjVp13QkURimspdArc27xCTnl5ORciZM3UUZ3dvHXQkkZincpdALc0uYOrsFbRo2ohnp42kX8f4WZ9bJEgqdwnMO+t2M31eJj3ansazt42gS+tmQUcSiRsqdwnECyt2cP9LnzC4W2uemTScNs0bBx1JJK6o3KVeuTszFmXxn+9s5oLU9jw2YRjNm+ivoUik6V+V1JvyihAPvbaOectzuHZoV3553RAaNwrnejEicrJU7lIvikrLuWtuJn/ZmK+12EXqgcpd6ty+whImz17BmtyDWidGpJ6o3KVObd1byG3PZLDncDGPT0xj7MCUoCOJNAgqd6kzH27dx+3PrSQpMYG5U0dxTo82QUcSaTBU7lIn5i/P4cFX19KnQ3OeumU43dueFnQkkQZF5S4RFXLnZws3MHNJNhf278Dvxg+lZdOkoGOJNDgqd4mYoyXl/G9mCZn52dx8bk8evnIgjRJ1qKNIEFTuEhE5BUVMe24Fm/Ir+I9/G8ik83VJPJEgqdzllP19yz6mz1uFO9yX1lTFLhIF9Duz1Jq78+Tfsrl51jI6tmjCgunnM6h9YtCxRATtuUstFZdV8INX1vDyqjwuHZjCb75+NslNGvFp0MFEBFC5Sy3kFBRxx5yVrNt5mG+NSeXui1NJSNBSAiLRROUuJ+W99Xu494WPAXjy5jTG6IxTkaikcpewlFeE+M27m/n94q2c2aUlf7hpGD3a6cQkkWilcpca7T1Swt3zMvkou4AbR3Tnh/92Jk2T9MGpSDRTucsJ/W3LXr79x9UcKS7j19cP4atp3YOOJCJhULnLcZVVhHj0nc089v5WUjsm8/yUEZzeqWXQsUQkTCp3+Rc79hdx17xMPt5xkBtH9ODhKwfSrLGmYURiicpd/smC1Tt54OU1YDBj/Dl8eUjnoCOJSC2EdYaqmY0zs01mlmVm9x/n+XvNbL2ZfWJmfzEzXWonxhwsKuWueZncPS+T1JRkFt59gYpdJIbVuOduZonADGAskAtkmNkCd19fbbNMIM3di8zsDuBXwNfrIrBE3uJN+XzvT5+w/2gp37m0P7df1FerOYrEuHCmZUYAWe6eDWBm84GrgX+Uu7svqrb9UmBCJENK3SgqLeenb2xgzrIc+qckM2vScAZ1bRV0LBGJAHP3E29gdj0wzt2nVN2fCIx09+lfsP3vgN3u/pPjPDcNmAaQkpIybP78+bUKXVhYSHJycq2+NxoFMZ4NBRU8va6EvUXOZb0a8ZXUxjROPPUlBPTeRLd4Gk88jQXCH8/o0aNXuntaTdtF9ANVM5sApAEXHe95d58JzARIS0vz9PT0Wv2cxYsXU9vvjUb1OZ7DxWX8fOFG5mXk0LPdacybMIRRfdpF7PX13kS3eBpPPI0FIj+ecMo9D6h+5kq3qsf+iZmNAR4ALnL3ksjEk0h6b/0eHnh1DXuPlDDtwj58e0x/HeIoEqfCKfcMINXMelNZ6jcA46tvYGZDgcepnL7Jj3hKOSX5R4r58Z838PrqnZzeqQUzJ6ZxVvfWQccSkTpUY7m7e7mZTQfeBhKBWe6+zsweAVa4+wLg10Ay8KKZAeS4+1V1mFvCUBFynl+6nf98exMl5SG+PaY/d6T3pXEjHQkjEu/CmnN394XAwmMee7ja7TERziWnKDPnAA++upZ1Ow9zQWp7fnTVmfTpED8fPonIiekM1Tiz/2gpv357E/MzcujYogkzxp/DFYM7UfUblYg0ECr3OFFaHuLZj7bx279soai0gsnn9+ZbY/uT3ERvsUhDpH/5Mc7deW9DPj99Yz3bCoq4qH8HHvzyGaSmtAg6mogESOUew9bmHeLnb27gg6wC+nZoztO3Dmf0gI5BxxKRKKByj0Hb9h3l0Xc38/rqnbQ+LYkfXXUm40f2IEnrwYhIFZV7DMk/XMxv/7KFP2bsICkxgemj+zH1wj60apYUdDQRiTIq9xiwr7CEJ5ZkM/ujbZRXOONH9mD6xf3o2KJp0NFEJEqp3KNY/pFiZr6fzfPLtlNaHuKqs7rw7bH96dmuedDRRCTKqdyj0J7DxTz+fjZzlm2nrCLENWd35ZsX96OvTkISkTCp3KNIVv4RZi7J5pXMPEIO1w7tyjdH96N3e+2pi8jJUblHgS0HKnh+9gre27CHJo0SuGF4D6Zc0FvTLyJSayr3gJSWh3hz7S6e+XAbmTnFtDmtgnsuSeXmc3vSLrlJ0PFEJMap3OtZ/uFi5izLYe7yHPYeKaF3++ZMOKMxD9x4sdZWF5GIUbnXA3dnafZ+5i3P4c21uyircNIHdGDSeb24MLUDS5a8r2IXkYhSudehvUdK+NPKXP6YkcO2giJaNG3EhFE9ufncXvqQVETqlMo9wkrLQ7y/eS8vrczlvQ17KA85I3q15a6LU7licGftoYtIvVC5R4C78/GOg7ySmcfrq3dyoKiMds0bc+v5vfj68B7066jj00Wkfqnca8nd2bynkDfW7OLPq3eSve8ojRslMHZgCl8Z2pUL+3fQQl4iEhiV+0nasucIf/5kF2+s2UVWfiFmMLJ3W75xUR8uH9yZlk21iJeIBE/lXoOKkPPxjgO8s34P767fQ/beo5jBiF5tueXqM7lsUCct4CUiUUflfhyFJeV8kLWPv27I5y8b97CvsJRGCcaoPu245dxeXD6oEx1bqtBFJHqp3KmcP9+4+wjvb97L4k35rNh2gPKQ06JJI9JP78jYgSmkD+igKRcRiRkNttxzDxTx4dYCPszax4dbC8g/UgLA6Z1aMPmC3qT378iwnm1o3EgfiopI7GkQ5e7u5B74jIxt+1n+6X4+yi5ge0ERAO2TG3Nu3/Z8qV87LurfkU6tNN0iIrEvLsu9vCLExt1HyMw5QMa2A2Rs28+uQ8UAtGjaiJG9K+fOz+/Xnv4pyZhZwIlFRCIr5svd3dl5qJg1uQfJzDlI5o6DrMk9xGdlFQCktGzC8F5tGdG7LcN7tWVASgsSElTmIhLfYq7c8w8Xk7G7nOVvbWRN3iHW7TzM/qOlADROTGBgl5bcMKI7Q3u0YWj31nRr00x75iLS4MRcub+4MpcZH5fQKCGb1JQWjDmjI4O7tmJQ11YM7NKSJo20douISMyV+zVDu9Ls0HbGfzmdpkkqchGR44m54/y6tm5Gn9aJKnYRkROIuXIXEZGaqdxFROKQyl1EJA6p3EVE4pDKXUQkDoVV7mY2zsw2mVmWmd1/nOebmNkfq55fZma9Ih1URETCV2O5m1kiMAO4HBgI3GhmA4/ZbDJwwN37Af8F/DLSQUVEJHzh7LmPALLcPdvdS4H5wNXHbHM1MLvq9p+AS0zn/IuIBCacM1S7Ajuq3c8FRn7RNu5ebmaHgHbAvuobmdk0YFrV3UIz21Sb0ED7Y187xsXTeOJpLKDxRLN4GguEP56e4bxYvS4/4O4zgZmn+jpmtsLd0yIQKSrE03jiaSyg8USzeBoLRH484UzL5AHdq93vVvXYcbcxs0ZAK6AgEgFFROTkhVPuGUCqmfU2s8bADcCCY7ZZANxSdft64K/u7pGLKSIiJ6PGaZmqOfTpwNtAIjDL3deZ2SPACndfADwFPGdmWcB+Kv8DqEunPLUTZeJpPPE0FtB4olk8jQUiPB7TDraISPzRGaoiInFI5S4iEodittzN7Mdm9omZfWxm75hZl6AznQoz+7WZbawa0ytm1jroTLVlZl81s3VmFjKzmD1UraZlN2KJmc0ys3wzWxt0llNlZt3NbJGZra/6e3ZP0Jlqy8yamtlyM1tdNZYfRey1Y3XO3cxauvvhqtt3AwPd/faAY9WamV1K5VFG5Wb2SwB3//eAY9WKmZ0BhIDHge+4+4qAI520qmU3NgNjqTxxLwO40d3XBxqslszsQqAQeNbdBwWd51SYWWegs7sOXPK5AAACFklEQVSvMrMWwErgmlh8b6rO5G/u7oVmlgT8HbjH3Zee6mvH7J7758VepTkQm/9LVXH3d9y9vOruUirPJ4hJ7r7B3Wt79nG0CGfZjZjh7kuoPJIt5rn7LndfVXX7CLCByrPkY45XKqy6m1T1FZEui9lyBzCzn5rZDuAm4OGg80TQbcCbQYdo4I637EZMFkg8q1qBdiiwLNgktWdmiWb2MZAPvOvuERlLVJe7mb1nZmuP83U1gLs/4O7dgTnA9GDT1qym8VRt8wBQTuWYolY4YxGpS2aWDLwEfOuY3+RjirtXuPvZVP62PsLMIjJtVq9ry5wsdx8T5qZzgIXAD+swzimraTxmNgm4Ergk2s/wPYn3JlaFs+yGBKRqfvolYI67vxx0nkhw94NmtggYB5zyB99Rved+ImaWWu3u1cDGoLJEgpmNA74HXOXuRUHnkbCW3ZAAVH0I+RSwwd1/E3SeU2FmHT4/Ms7MmlH5AX5EuiyWj5Z5CRhA5VEZ24Hb3T1m96yqlm5owv8vuLY0Vo/+MbNrgf8FOgAHgY/d/bJgU508M7sC+G/+f9mNnwYcqdbMbB6QTuWysnuAH7r7U4GGqiUz+xLwN2ANlf/+AX7g7guDS1U7ZjaEymthJFK5s/2Cuz8SkdeO1XIXEZEvFrPTMiIi8sVU7iIicUjlLiISh1TuIiJxSOUuIhKHVO4iInFI5S4iEof+D8EmLYGi5zrSAAAAAElFTkSuQmCC\n", "text/plain": [ "
" ] }, "metadata": { "needs_background": "light" }, "output_type": "display_data" } ], "source": [ "plt.plot(x, backend.get_value(activations.sigmoid(x)))\n", "plt.grid(True); plt.ylim(0, 1); plt.show()" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## softsign\n", "\n", "The softplus activation is:
\n", "f(x) = x / (abs(x) + 1)" ] }, { "cell_type": "code", "execution_count": 8, "metadata": {}, "outputs": [ { "data": { "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYYAAAD8CAYAAABzTgP2AAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDMuMC4xLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvDW2N/gAAIABJREFUeJzt3Xl81PWdx/HXh0A4EkKAQAhJuBG5CkgEr1VQVOy2YFu1aOvRqmy71dp2e9i1a1trW3vsuu3W7paqLdYDj1qlFhcRSdFVENAoN4RwhTMHhByQ87N/zECTNCGQmWQyk/fz8ZhH5vf7fX8znw8Jeed3jrk7IiIiJ3WJdAEiItKxKBhERKQBBYOIiDSgYBARkQYUDCIi0oCCQUREGghLMJjZ42Z22Mw2NLPczOyXZpZrZh+a2Xn1lt1qZtuDj1vDUY+IiLReuLYYfg/MPs3ya4DRwcd84L8BzKwf8F1gOjAN+K6Z9Q1TTSIi0gphCQZ3XwkUn2bIXOAJD1gFJJtZGnA1sMzdi939CLCM0weMiIi0sa7t9D7pwN560/nBec3N/ztmNp/A1gY9e/acmpmZ2apC6urq6NIldg6txFI/sdQLxFY/sdQLdN5+tm3bVujuA1oa117BEDJ3XwAsAMjKyvK1a9e26nWys7OZMWNGGCuLrFjqJ5Z6gdjqJ5Z6gc7bj5ntPpPXa6/I3AfU/xM/IzivufkiIhIh7RUMi4FbgmcnXQCUuPsBYClwlZn1DR50vio4T0REIiQsu5LM7BlgBpBiZvkEzjTqBuDu/wMsAT4K5AIVwOeCy4rN7AfAmuBLPeDupzuILSIibSwsweDuN7aw3IEvNbPsceDxcNQhIiKhi53D8iIiEhYKBhERaUDBICIiDSgYRESkAQWDiIg0oGAQEZEGFAwiItKAgkFERBpQMIiISAMKBhERaUDBICIiDSgYRESkAQWDiIg0oGAQEZEGFAwiItKAgkFERBpQMIiISAMKBhERaSAswWBms81sq5nlmtm9TSx/2Mxygo9tZna03rLaessWh6MeERFpvZA/89nM4oBHgCuBfGCNmS12900nx7j7V+uNvxuYUu8ljrv75FDrEBGR8AjHFsM0INfd89y9ClgEzD3N+BuBZ8LwviIi0gZC3mIA0oG99abzgelNDTSzocBw4I16s3uY2VqgBnjI3V8KQ00iIlGvrs45VHqCXYUV7CoqZ1dROXfNHEXvHt3a9H3DEQxnYx7wgrvX1ps31N33mdkI4A0zW+/uOxqvaGbzgfkAqampZGdnt6qAsrKyVq/bEcVSP7HUC8RWP7HUC3S8fkqrnIPldRwsr+NAuXOoIvD8cIVTXfe3cV0NMmoOkNm74c6ecPcTjmDYB2TWm84IzmvKPOBL9We4+77g1zwzyyZw/OHvgsHdFwALALKysnzGjBmtKjY7O5vWrtsRxVI/sdQLxFY/sdQLRKafujon/8hxcgtK2XG4nNzDZewoCDyOVFSfGtctzhjSrxdjMxP5aEovhvZPYFj/BIal9CKtT0/iutjfvXa4+wlHMKwBRpvZcAKBMA+4qfEgMzsX6Au8U29eX6DC3SvNLAW4GPhpGGoSEYkId2d/yQm2HjzG1oNlbD9cyvZDZeQeLuN49d92lqQkxjNiQCLXTExjREoCIwckMmJAAunJPekaF9krCUIOBnevMbO7gKVAHPC4u280sweAte5+8hTUecAid/d6q48FfmNmdQQOhD9U/2wmEZGOrKKqhi0HS9m0/xhbDh5j68FSthwspfREzakxg5J6MDo1kRunDeGc1ERGDQw8knvFR7Dy0wvLMQZ3XwIsaTTv/kbT32tivbeBieGoQUSkLR0pr2L9vhI27C9h0/5jbDpwjJ2F5Zz8U7d3966cm9abuZMHc+6gJM4d1JvRqb3p07NtDxS3hfY++Cwi0uEdO1HNh3tL+CD/KBv2lbB+Xwn5R46fWp7Rtyfj0pKYM2kw49KSGDc4ifTknpj9/f7/aKRgEJFOraa2ji0HS3l/zxHe33uUD/YeZUdB+anlQ/v3YlJmMp+9YCgT0/swYXAf+vSKvq2As6FgEJFOpaSimg8Kali7dCvrdh8hZ+/RUweFUxLjmZyZzCempDMpM5mPpCfHfAg0RcEgIjHtcOkJ1uw8wrs7i1i9s5ith0pxh7guOxiXlsSnz8/kvKF9OW9IckztDgqFgkFEYsqR8ireySvi7R2FvL2jiLzgbqGe3eKYOrQvH52YRreje7j145fRK16/ApuifxURiWonqmtZs6uYN7cX8tb2QjYfPIY7JMTHMW14P27IymT68H5MSO9Dt+D1AdnZ+xQKp6F/GRGJKu5OXmE5f91awMrtBazKK+JEdR3xcV04b2gyX5t1DheN6s9HMpJPBYGcHQWDiHR41bV1rNlZzOubD7N8yyF2F1UAMCIlgXnnD+GycwYwfUQ/bQWEif4VRaRDKqusYcWWwyzdeJC/biug9EQN8V27cNHI/txxyXBmjBlIZr9ekS4zJikYRKTDKKmoZtnmQ/zvhgOs3F5IVU0dKYnd+eiENK4YO5BLRqdoq6Ad6F9YRCKqrLKGZZsOsjhnP29uL6SmzhncpwefnT6UayYO4rwhfZu8o6i0HQWDiLS7qpo6Vm4r4OUP9rNs00FOVNeRntyT2y8ZzjUT05iU0UfXE0SQgkFE2s3mA8d4bu1eXs7ZT3F5Ff0S4rl+aiZzJg9m6pC+dNGWQYegYBCRNlVSUc3LH+zj+bX5rN9XQnxcF64cn8p152VwyegUnVLaASkYRKRNrM8v4Yl3drH4g/1U1tQxLi2J7318HHMnp9M3oeN+FoEoGEQkjE5U1/KXDw/wxKrdfLD3KL3i4/jU1AxumjaECel9Il2enCEFg4iErKiskife2c0fVu2muLyKkQMS+N7Hx/HJqRkk9eh8dyeNdgoGEWm1XYXl/PbNPF5Yl09lTR2zxg7kcxcP56KR/XVWURRTMIjIWdu4v4RfvZHL/248SLcuXfjElHTuvHQ4owb2jnRpEgZhCQYzmw38AogDHnX3hxotvw34GbAvOOtX7v5ocNmtwHeC8x9094XhqElEwm/j/hJ+8fp2Xtt0iN49uvLFy0Zy20XDGJjUI9KlSRiFHAxmFgc8AlwJ5ANrzGyxu29qNPRZd7+r0br9gO8CWYAD64LrHgm1LhEJnz3HavmnP6xl6cZAIHxl1mg+d/HwqPyge2lZOLYYpgG57p4HYGaLgLlA42BoytXAMncvDq67DJgNPBOGukQkRHuLK/jZ0q0s/uAEvbvXcM8Vo/n8JQqEWBeOYEgH9tabzgemNzHuU2Z2KbAN+Kq7721m3fSm3sTM5gPzAVJTU8nOzm5VsWVlZa1etyOKpX5iqReI7n7Kq51X8qpZtqsaM7gqw5k7Jp6Ebvt5f/X+SJcXsmj+3jQl3P2018HnPwPPuHulmf0TsBC4/GxewN0XAAsAsrKyfMaMGa0qJDs7m9au2xHFUj+x1AtEZz/VtXU8tWo3v1i5naPHq/nklAy+fvU5bH1/ddT1cjrR+L05nXD3E45g2Adk1pvO4G8HmQFw96J6k48CP6237oxG62aHoSYROUvv7izmvj+tZ/vhMi4a2Z9//ejYUxelbY1wbdK+whEMa4DRZjacwC/6ecBN9QeYWZq7HwhOzgE2B58vBX5kZn2D01cB3w5DTSJyhorLq/jxks08vy6f9OSe/PaWLGaNHajrEDqxkIPB3WvM7C4Cv+TjgMfdfaOZPQCsdffFwJfNbA5QAxQDtwXXLTazHxAIF4AHTh6IFpG2VVfnPL9uLz9+dQtlJ2r44oyR3H35KH0QjoTnGIO7LwGWNJp3f73n36aZLQF3fxx4PBx1iMiZ2Vtcwdef/4DVO4uZNqwfD35iAuek6uI0CdCfBiKdiLvz/Np8HnglcDb5Tz41kRuyMrXbSBpQMIh0EgWllXz7xfW8vvkQF4zox8+vn0RG316RLks6IAWDSCfw2saD3Pviesoqa/jOP47l8xcP16elSbMUDCIxrKa2jp8u3cqClXlMSE/i4RsmM1rHEqQFCgaRGHW49AR3Pf0+7+4s5uYLhvKdj42le9e4SJclUUDBIBKD3t1ZzJeefo/SE9U8/OlJfGJKRqRLkiiiYBCJIe7OY2/t5MevbiGzb0/+cPs0zh2UFOmyJMooGERiRE1tHd9dvJGnVu/hqnGp/PyGSfpYTWkVBYNIDKioquHup99n+ZbDfOGykXzz6jE660haTcEgEuUKSiu5feEaNuwr4Qdzx3PzhcMiXZJEOQWDSBTbUVDGbb97l4LSSn5zcxZXjkuNdEkSAxQMIlFqw74SPvvYauLMWDT/QiZnJke6JIkRCgaRKLRhXwmfeXQ1id278vSd0xnaPyHSJUkMUTCIRJn6obBo/gVk9tP9jiS8ukS6ABE5cxv3B3YfKRSkLSkYRKLExv2BLYWE+K48c6dCQdqOgkEkCmw7VMpnHl1Nr25xPHPnBQzpr1CQtqNgEOngDpee4HO/W0N8XBcWzb9QoSBtLizBYGazzWyrmeWa2b1NLP+amW0ysw/NbLmZDa23rNbMcoKPxeGoRyRWVFTVcMfCtRypqOLx285XKEi7CPmsJDOLAx4BrgTygTVmttjdN9Ub9j6Q5e4VZvZF4KfAp4PLjrv75FDrEIk1tXXOPYty2LCvhN/eksWE9D6RLkk6iXBsMUwDct09z92rgEXA3PoD3H2Fu1cEJ1cBugewSAse/Msmlm06xHc/Pp4rxuqKZmk/5u6hvYDZdcBsd78jOH0zMN3d72pm/K+Ag+7+YHC6BsgBaoCH3P2lZtabD8wHSE1Nnbpo0aJW1VtWVkZiYmKr1u2IYqmfWOoFQutn2e5qntpcxVVDu3LT2O5hruzs6XvTsZ1pPzNnzlzn7lktDnT3kB7AdcCj9aZvBn7VzNjPEthi6F5vXnrw6whgFzCypfecOnWqt9aKFStavW5HFEv9xFIv7q3v540th3z4va/4HQvXeE1tXXiLaiV9bzq2M+0HWOtn8Hs9HLuS9gGZ9aYzgvMaMLNZwH3AHHevrBdM+4Jf84BsYEoYahKJSgdKjvPVZ3MYMyiJX8ybTJxunS0REI5gWAOMNrPhZhYPzAManF1kZlOA3xAIhcP15vc1s+7B5ynAxUD9g9YinUZNbR33PJNDVU0dj9w0hV7xumONREbIP3nuXmNmdwFLgTjgcXffaGYPENhsWQz8DEgEnjczgD3uPgcYC/zGzOoIhNRD3vBsJpFO4xfLt/PurmL+89OTGTEgdvZ/S/QJy58k7r4EWNJo3v31ns9qZr23gYnhqEEkmr21vZBfrcjl+qkZXDslPdLlSCenK59FIqygtJKvPJvDyAGJfH/u+EiXI6LbbotEUl2d87Xncig9Uc1Td0zXcQXpEPRTKBJBC97M483thfz4kxMZM6h3pMsRAbQrSSRidhWW8x/LtnH1+FTmnZ/Z8goi7UTBIBIB7s53XtpA97guPDB3AsGz9UQ6BAWDSAS8nLOft3IL+ebsMaQm9Yh0OSINKBhE2tnRiip+8MomJmcmc9P0oS2vINLOdPBZpJ39eMkWjh6v5slPTtQtL6RD0haDSDtanVfEs2v3csc/DGdsWlKkyxFpkoJBpJ1U1tTyr39aT0bfntxzxehIlyPSLO1KEmknC/6ax46Ccn73ufN1IZt0aNpiEGkHBaWV/Dp7Bx+dOIiZYwZGuhyR01IwiLSDR1bkUlVbxzeuPjfSpYi0SMEg0sb2HT3O06v3cP3UDIanJES6HJEWKRhE2tgvX98OwJd1wFmihIJBpA3lFZTxwnv5fOaCIQxO7hnpckTOiIJBpA09/Pp2unftwj/PGBXpUkTOmIJBpI3sOVbLnz/Yz+cvHs6A3t0jXY7IGQtLMJjZbDPbama5ZnZvE8u7m9mzweWrzWxYvWXfDs7famZXh6MekY7gxe3VJPXoyp2Xjoh0KSJnJeRgMLM44BHgGmAccKOZjWs07HbgiLuPAh4GfhJcdxwwDxgPzAZ+HXw9kaj23p4j5BTU8k+XjaRPz26RLkfkrIRji2EakOvuee5eBSwC5jYaMxdYGHz+AnCFBW5APxdY5O6V7r4TyA2+nkhU+/nSrSTFw+cuHhbpUkTOWjiuy08H9tabzgemNzfG3WvMrAToH5y/qtG66U29iZnNB+YDpKamkp2d3apiy8rKWr1uRxRL/cRKL5uKanl7xwmuG+G8+/ZbkS4nLGLle3OS+jm9qLlhi7svABYAZGVl+YwZM1r1OtnZ2bR23Y4olvqJhV7cnYd//TaD+8DVo7pEfT8nxcL3pj71c3rh2JW0D6j/gbUZwXlNjjGzrkAfoOgM1xWJGq9vPswHe49yz6zRdNNnLUiUCkcwrAFGm9lwM4sncDB5caMxi4Fbg8+vA95wdw/Onxc8a2k4MBp4Nww1ibS7ujrn50u3MjwlgU+dlxHpckRaLeRdScFjBncBS4E44HF332hmDwBr3X0x8BjwBzPLBYoJhAfBcc8Bm4Aa4EvuXhtqTSKR8OcP97P1UCn/deMUusbpEiGJXmE5xuDuS4AljebdX+/5CeD6Ztb9IfDDcNQhEinVtXU8vGwbY9OS+MeJaZEuRyQk+rNGJAxeWJfPrqIKvn7VOXTRsQWJcgoGkRCdqK7ll8u3M2VIMpefqw/hkeinYBAJ0VOr93Cg5ATfuHoMges2RaKbgkEkBKUnqvn1ilwuGZXCRSNTIl2OSFgoGERC8LOlWzlSUcW3ZusjOyV2KBhEWun9PUf4w6rd3HLhMCZm9Il0OSJho2AQaYXq2jq+/eJ6Unv34F+uOifS5YiEVdTcK0mkI3n8rZ1sOVjKb26eSu8euq22xBZtMYicpb3FFTz8+jauHJfK1eMHRbockbBTMIicBXfn/pc3EGfG9+eMj3Q5Im1CwSByFv6y/gArthbwL1eNYXByz0iXI9ImFAwiZ6ikoprv/3kTE9P7cOtFwyJdjkib0cFnkTPg7tz74occKa/id7edT5zuhyQxTFsMImfgydV7eHXDQb45ewwT0nXNgsQ2BYNICzbuL+EHr2xixpgB3HHJiEiXI9LmFAwip1FeWcPdT79P317d+PfrJ+mW2tIp6BiDyGn820sb2FVUztN3XkD/xO6RLkekXWiLQaQZL6zL58X39/HlK0ZzwYj+kS5HpN2EFAxm1s/MlpnZ9uDXvk2MmWxm75jZRjP70Mw+XW/Z781sp5nlBB+TQ6lHJFy2Hyrl317awAUj+nH35aMjXY5Iuwp1i+FeYLm7jwaWB6cbqwBucffxwGzgP80sud7yb7j75OAjJ8R6REJWWFbJ5xeuIaF7HL+YN0WnpkqnE2owzAUWBp8vBK5tPMDdt7n79uDz/cBhYECI7yvSJk5U13LHwrUUlFby2K3nk5rUI9IlibQ7c/fWr2x21N2Tg88NOHJyupnx0wgEyHh3rzOz3wMXApUEtzjcvbKZdecD8wFSU1OnLlq0qFU1l5WVkZiY2Kp1O6JY6ifSvdS58+ucStYdquWuKd2ZmhrauRmR7iecYqkX6Lz9zJw5c527Z7U40N1P+wBeBzY08ZgLHG009shpXicN2Apc0GieAd0JBMb9LdXj7kydOtVba8WKFa1etyOKpX4i3csP/7LJh37rFf/tyh1heb1I9xNOsdSLe+ftB1jrZ/A7tsU/idx9VnPLzOyQmaW5+wEzSyOwm6ipcUnAX4D73H1Vvdc+EHxaaWa/A77eUj0ibeHJVbtZsDKPWy4cyu2XDI90OSIRFeoxhsXArcHntwIvNx5gZvHAn4An3P2FRsvSgl+NwPGJDSHWI3LW3thyiPtf3sDl5w7k/o+NI/DjKNJ5hRoMDwFXmtl2YFZwGjPLMrNHg2NuAC4FbmvitNSnzGw9sB5IAR4MsR6Rs/LXbQV84cn3GD+4D/914xS6xunSHpGQjq65exFwRRPz1wJ3BJ8/CTzZzPqXh/L+IqH467YC7nxiLaMHJvKH26eR0F03AhABXfksndTJUBg1IJGn7phOcq/4SJck0mEoGKTTWalQEDktBYN0KidDYWQwFPomKBREGlMwSKfx4nv53L5wDSMUCiKnpaNtEvPcnUdW5PLz17Zx4Yj+/M/NU+nTs1ukyxLpsBQMEtNqauv4t5c38My7e7l28mB+et0k4rtqQ1nkdBQMErPKK2u46+n3WLG1gC/NHMnXrxqji9dEzoCCQWLS3uIKvvDkOjYfOMaPPjGRm6YPiXRJIlFDwSAxZ8WWw3zl2Rzq3Hns1vOZee7ASJckElUUDBIzauucX7y+jV++kcvYtCT+57PnMbR/QqTLEok6CgaJCcXlVdyz6H3e3F7IdVMzePDaCfToFhfpskSikoJBot47O4r4l+dyKCyr4sefnMi88zN1kFkkBAoGiVonqmv52dKtPPbWToanJPDCFy/kIxnNfoCgiJwhBYNEpfX5JXz1uRxyD5dxy4VDufeac+kVrx9nkXDQ/ySJKtW1dfx39g5+uXw7KYndeeLz07j0nAGRLkskpigYJGq8u7OY77y0nm2Hypg7eTAPzJlAn166tYVIuCkYpMMrLq/ix0s28/y6fNKTe/LoLVnMGpca6bJEYpaCQTqsujrnhXX5/OjVzZSdqOELl43ky1eM0rEEkTYW0v8wM+sHPAsMA3YBN7j7kSbG1RL4XGeAPe4+Jzh/OLAI6A+sA25296pQapLYsCqviB8t2cyH+SWcP6wvD147kTGDeke6LJFOIdTbTN4LLHf30cDy4HRTjrv75OBjTr35PwEedvdRwBHg9hDrkSi3o6CMO59Yy7wFqygoreTfr5/Es/MvVCiItKNQt8nnAjOCzxcC2cC3zmRFC1yBdDlwU731vwf8d4g1SRQqLKvkD5sqyX5tJT27xfGNq8dw+yXDdfWySASYu7d+ZbOj7p4cfG7AkZPTjcbVADlADfCQu79kZinAquDWAmaWCbzq7hOaea/5wHyA1NTUqYsWLWpVzWVlZSQmJrZq3Y4o2vs5Vum8uqua5Xuqqal1ZgzpxrUj40nqHv1XLkf796a+WOoFOm8/M2fOXOfuWS2Na3GLwcxeBwY1sei++hPu7mbWXMoMdfd9ZjYCeMPM1gMlLb13o9dfACwAyMrK8hkzZpzN6qdkZ2fT2nU7omjtp6iskgUr83jind1U1tQyd3I60xOLmfePl0e6tLCJ1u9NU2KpF1A/LWkxGNx9VnPLzOyQmaW5+wEzSwMON/Ma+4Jf88wsG5gC/BFINrOu7l4DZAD7WtGDRJEDJcd5/K2dPLlqDydqapk7aTB3XzGakQMSyc7OjnR5IkLoxxgWA7cCDwW/vtx4gJn1BSrcvTK4++hi4KfBLYwVwHUEzkxqcn2JDZsPHOO3K/NY/MF+HPjYR9K4+/LRjBoYO5vzIrEi1GB4CHjOzG4HdgM3AJhZFvAFd78DGAv8xszqCJwF9ZC7bwqu/y1gkZk9CLwPPBZiPdKBuDtv5Rby2zd3snJbAb3i47j5wqF8/uLhZPbrFenyRKQZIQWDuxcBVzQxfy1wR/D528DEZtbPA6aFUoN0PCXHq/njunyeXLWbvMJyUhK7842rx/CZ6UNI7hUf6fJEpAW6hFTCZuP+Ep5ctZuX3t/P8epapgxJ5uFPT+KaCWk67VQkiigYJCRHK6p4OWc/z6/by4Z9x+jRrQtzJ6Vz84VDmZDeJ9LliUgrKBjkrNXU1vF/O4p4fu1eXtt4iKraOsYPTuL7c8Zz7eR03fFUJMopGOSMuDs5e4/ycs5+XvnwAIVllST36sZN04dwfVYG4wdr60AkVigYpFnuzpaDpSxZf4CXc/azp7iC+K5duOLcgcyZNJjLxw6ke1cdOxCJNQoGacDd+TC/hFc3HOR/NxxgV1EFXQwuHpXC3ZeP4uoJg0jqoV1FIrFMwSBU1tTy7s5ilm8+zLJNh9h39DhduxgXjuzP/EtHcuW4VAb07h7pMkWknSgYOqnCskqytxawfPMhVm4roLyqlh7dunDJqBS+euU5zBo7UNcciHRSCoZOorq2jvd2H+Gv2wpYub2ADfuOAZCa1J05k9OZNXYgF49K0fUGIqJgiFXuzrZDZby9o5D/yy1iVV4RZZU1xHUxpg7py9evOofLzhnIhPQkAndMFxEJUDDECHcnr7Cc1XnFvJNXxDs7CiksC3xK6tD+vZgzeTCXjh7ARaP66+CxiJyWgiFK1dY5Ww4eY9nuap57ah3v7iw+FQQDenfnklEpXDQqhYtG9iejr25YJyJnTsEQJY6dqOb9PUdZt/sI7+0+Qs7eo5RV1gCQnlzCP4wewLTh/Th/WD9GDkjQ7iERaTUFQwdUXVvHlgOl5OQf5YO9R8nZe5QdBWW4QxeDcwcl8Ykp6Zw3NJnag9u47prY+dQzEYk8BUOEVdfWsf1QGRv2lbA++Nh84BiVNXUA9E+IZ3JmMnMmDWbq0L5Mykwmsfvfvm3Z2bmRKl1EYpSCoR2VV9aw5eAxNu0/xqYDga9bDpaeCoHE7l0ZPziJmy8YyuQhyUzKSCajb0/tFhKRdqVgaAO1dc7uonK2Hixly8HS4Ndj7C6uwD0wpk/PbqdCYGJGHyam92FY/wS6dFEIiEhkKRhCUFvn5B+pYPuhMrYdLg18PVRK7uGyU1sBZjCsfwJj05K4dko64wf3YfzgJNL69NCWgIh0SCEFg5n1A54FhgG7gBvc/UijMTOBh+vNOheY5+4vmdnvgcuAkuCy29w9J5Sa2kJFVQ15BeXkFZaz43AZuQVl7Dhcxs7C8lMBAJDWpwejU3tzwYj+jEntzblpvRk9sDc943U1sYhEj1C3GO4Flrv7Q2Z2b3D6W/UHuPsKYDKcCpJc4LV6Q77h7i+EWEfITlTXsre4gl1FFewuKmdnYTl5BYGvB4+dODWui0Fmv16MHJDIpecMYOSABEYN7M3o1ERdOCYiMSHUYJgLzAg+Xwhk0ygYGrkOeNXdK0J831YpqahmZ0ktZR/uZ3dRBXuKKthTHHjsLzl+av8/QHKvboxISeDiUSmMGJDAiJQEhg9IYFj/BN1PSERiWqjBkOruB4LPDwKpLYyfB/xHo3k/NLP7geXAve5eGWJNzfrnp9fxf7kn4J3hmKM5AAAGB0lEQVT3AUhJ7M6Qfj2ZNrwfw/onMCylF8P6JzC0fy/dWVREOi3z+n8mNzXA7HVgUBOL7gMWuntyvbFH3L1vM6+TBnwIDHb36nrzDgLxwAJgh7s/0Mz684H5AKmpqVMXLVrUQmt/b0NhLSVlxxnSvxcDeho9ukb/wd+ysjISExMjXUZYxFIvEFv9xFIv0Hn7mTlz5jp3z2pxoLu3+gFsBdKCz9OAracZew+w4DTLZwCvnMn7Tp061VtrxYoVrV63I4qlfmKpF/fY6ieWenHvvP0Aa/0Mfsd2aV0+nbIYuDX4/Fbg5dOMvRF4pv6M4BYDFjhv81pgQ4j1iIhIiEINhoeAK81sOzArOI2ZZZnZoycHmdkwIBP4a6P1nzKz9cB6IAV4MMR6REQkRCEdfHb3IuCKJuavBe6oN70LSG9inO7+JiLSwYS6xSAiIjFGwSAiIg0oGEREpAEFg4iINKBgEBGRBhQMIiLSgIJBREQaUDCIiEgDCgYREWlAwSAiIg0oGEREpAEFg4iINKBgEBGRBhQMIiLSgIJBREQaUDCIiEgDCgYREWlAwSAiIg0oGEREpIGQgsHMrjezjWZWZ2ZZpxk328y2mlmumd1bb/5wM1sdnP+smcWHUo+IiIQu1C2GDcAngZXNDTCzOOAR4BpgHHCjmY0LLv4J8LC7jwKOALeHWI+IiIQopGBw983uvrWFYdOAXHfPc/cqYBEw18wMuBx4IThuIXBtKPWIiEjourbDe6QDe+tN5wPTgf7AUXevqTc/vbkXMbP5wPzgZJmZtRRIzUkBClu5bkcUS/3EUi8QW/3EUi/QefsZeiYv1mIwmNnrwKAmFt3n7i+fyZuEg7svABaE+jpmttbdmz0eEm1iqZ9Y6gViq59Y6gXUT0taDAZ3nxXie+wDMutNZwTnFQHJZtY1uNVwcr6IiERQe5yuugYYHTwDKR6YByx2dwdWANcFx90KtNsWiIiINC3U01U/YWb5wIXAX8xsaXD+YDNbAhDcGrgLWApsBp5z943Bl/gW8DUzyyVwzOGxUOo5QyHvjupgYqmfWOoFYqufWOoF1M9pWeAPdxERkQBd+SwiIg0oGEREpIFOGQxm9gMz+9DMcszsNTMbHOmaWsvMfmZmW4L9/MnMkiNdUyjO9DYrHVlzt4CJRmb2uJkdNrMNka4lHMws08xWmNmm4M/ZPZGuqbXMrIeZvWtmHwR7+X7YXrszHmMwsyR3PxZ8/mVgnLt/IcJltYqZXQW84e41ZvYTAHf/VoTLajUzGwvUAb8Bvu7uayNc0lkJ3gJmG3AlgYs21wA3uvumiBbWSmZ2KVAGPOHuEyJdT6jMLA1Ic/f3zKw3sA64Nhq/P8G7RyS4e5mZdQPeAu5x91Whvnan3GI4GQpBCUDUpqO7v1bv6vFVBK4HiVpneJuVjqzJW8BEuKZWc/eVQHGk6wgXdz/g7u8Fn5cSOFOy2TsudGQeUBac7BZ8hOV3WacMBgAz+6GZ7QU+A9wf6XrC5PPAq5EuopNr6hYwUfmLJ9aZ2TBgCrA6spW0npnFmVkOcBhY5u5h6SVmg8HMXjezDU085gK4+33ungk8ReA6iw6rpV6CY+4Dagj006GdST8ibcnMEoE/Al9ptAchqrh7rbtPJrCnYJqZhWV3X3vcRC8izuJWHk8BS4DvtmE5IWmpFzO7DfgYcIVHwUGjMNxmpSNr7hYw0kEE98f/EXjK3V+MdD3h4O5HzWwFMJvAxyGEJGa3GE7HzEbXm5wLbIlULaEys9nAN4E57l4R6Xqk6VvARLgmCQoesH0M2Ozu/xHpekJhZgNOnoVoZj0JnPAQlt9lnfWspD8CYwic/bIb+IK7R+VfdcHbiXQncFNCgFXReoYVBG6zAvwXMAA4CuS4+9WRrersmNlHgf8E4oDH3f2HES6p1czsGWAGgds6HwK+6+7tceuaNmFmlwBvAusJ/P8H+Fd3XxK5qlrHzD5C4HNs4gj8kf+cuz8QltfujMEgIiLN65S7kkREpHkKBhERaUDBICIiDSgYRESkAQWDiIg0oGAQEZEGFAwiItLA/wO4A4imNMvaVwAAAABJRU5ErkJggg==\n", "text/plain": [ "
" ] }, "metadata": { "needs_background": "light" }, "output_type": "display_data" } ], "source": [ "plt.plot(x, backend.get_value(activations.softsign(x)))\n", "plt.grid(True); plt.ylim(-1, 1); plt.show()" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## tanh" ] }, { "cell_type": "code", "execution_count": 9, "metadata": { "scrolled": true }, "outputs": [ { "data": { "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYYAAAD8CAYAAABzTgP2AAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDMuMC4xLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvDW2N/gAAIABJREFUeJzt3Xl81PW1//HXyQpJ2JeABAQEWVRciLhgLVhE9FrpglZrW/Tq5drWbmqr1lvttbWut/b+WqulaqW3VtT2qrSigEq0LiigKDsEkCUS9hBC9sz5/TFfehNMWDKTzPZ+Ph7fx3yXz3fmnExmznzXj7k7IiIiB6TFOgAREYkvKgwiItKECoOIiDShwiAiIk2oMIiISBMqDCIi0oQKg4iINKHCICIiTagwiIhIExmxDqA1evbs6QMHDmzVuvv37yc3Nze6AcVQMuWTTLlAcuWTTLlA6uazePHine7e63DtErIwDBw4kEWLFrVq3aKiIsaNGxfdgGIomfJJplwgufJJplwgdfMxs41H8nzalSQiIk2oMIiISBMqDCIi0oQKg4iINBGVwmBmj5vZdjNb1sJyM7P/Z2bFZvaRmZ3WaNlUM1sbDFOjEY+IiLRetLYYngAmHWL5hcDQYJgGPAxgZt2BO4AzgDHAHWbWLUoxiYhIK0SlMLj7G8DuQzSZDPzRwxYAXc2sL3ABMM/dd7v7HmAehy4wIiLSxtrrOoZ+wOZG01uCeS3NFxFJOqGQU1MfoqquITzUNlBd10BNfQM1dSFq6kPBdIja+hA1DSHq6kPUNnq8euwguudmtWmcCXOBm5lNI7wbivz8fIqKilr1PBUVFa1eNx4lUz7JlAskVz7JlAu0Lh93p7oB9tU6++vCQ0Ud/xyvqoeqeqeqzqlqgOr68LyaBqc6eKxpiCxuA/rWlnBMXtOdPdF+f9qrMJQA/RtNFwTzSoBxB80vau4J3H06MB2gsLDQW3vVYqpe8ZgIkikXSK58kikXaJpPKOTsrKjhk73VbCuvZnt5NdvKaygtr2b7vhp2769hV0Utu/bXUlsfavE5szLS6Nwhg04dMsnLzqBHpwzyOmSQk5VOTlYGuVnp4fHsDDpmptMxM53szDQ6ZqbTITOd7Iw0sg88ZqSRFQzZ6elkZaSRmW5kpDe/9z/a7097FYZZwPVmNpPwgea97r7VzOYAv2h0wHkicGs7xSQiKaK6roFNuytZv6OC9Tv38+6yGh5b9y5b9lRRUlb1qS/89DSjd6dsenXKpldeNsP7dKZHbhY98rLolhMeuuRk0rVjJl1yMunSMZPsjPQYZRd9USkMZvYU4V/+Pc1sC+EzjTIB3P0RYDZwEVAMVAJXB8t2m9nPgIXBU93p7oc6iC0i0qK6hhAbdu5n5dZyVm7dx6rScoq3V1BSVoX7/7XrnAWD8usZeUxnJp6QT0HXjhzTtSP5nTuQ37kD3XOzSE+z2CUSY1EpDO5+xWGWO/DtFpY9DjwejThEJHW4Oxt27ueDTWV8sHkPSzaXsaa0gtqG8K//zHTjuF55nDagG18+rYDBvXIZ3DOPQb1yWfTOm4wbNzbGGcSvhDn4LCKpLRRyVpaW81bxTt5Zt4sPNpdRVlkHQF52Bif378LVYwcyom9nhvftxHG98shsYZ+8HJoKg4jErW3l1cxftZ03i3fy9rpd7N5fC8DgXrlcMLIPpw7oyqkDujGkd15K7/qJNhUGEYkrH+/cz5zlpby8vJQPNpUBkN85m3HH92LskJ6MHdKTPl06xDjK5KbCICIxV7q3mr++v4W/ffgJq0r3AXBiv87ceP7xTDyhD8fn52GmLYL2osIgIjFRU9/AKyu28+zizbyxZgchh9MHduMnF49k4sh8+nfPiXWIKUuFQUTa1SdlVfzhrQ08u3gLZZV19O3SgW+PH8KU0QUc2yN5+mFOZCoMItIuln+yl9+/sZ6/f7QVByad0IfLTu/POUN66sBxnFFhEJE29VbxTh4uWsebxTvJzUpn6tkDuXrsQAq6aVdRvFJhEJE2sXTLXu59eRVvFu+kd6dsbp40nK+OGUCXnMxYhyaHocIgIlG1aVclD8xdzawPP6FbTiY/uXgkXztzQFLdSyjZqTCISFSUV9fx4Lw1/GnBRtLTjOvHD2HaZwfTuYO2EBKNCoOIRGzu8lJuf2E52/dV85XTB/D9CUPJ76yL0BKVCoOItNr28mp++rflzF5ayvA+nfjd10dzcv+usQ5LIqTCICJHzd15euFm7pq9kpr6ED+8YBjTzh2sm9YlCRUGETkqeyvr+OFfPmTuim2cObg7v/jiSQzulRfrsCSKVBhE5Igt2VzG9X9+n9K91fzHv4zgmnMG6R5GSShaPbhNAv4bSAcedfd7Dlr+IDA+mMwBert712BZA7A0WLbJ3S+JRkwiEj3uzuNvbuDul1bSu1MHnr3uLE4d0O3wK0pCirgwmFk68BBwPrAFWGhms9x9xYE27v6DRu2/A5za6Cmq3P2USOMQkbZRXl3Hrz+o4f3tK5gwIp8HLh1F15ysWIclbSgaWwxjgGJ3Xw9gZjOBycCKFtpfQbhPaBGJcyVlVVz9h/dYt6NBu45SiHnjHrJb8wRmU4BJ7n5tMP114Ax3v76ZtscCC4ACd28I5tUDS4B64B53f76F15kGTAPIz88fPXPmzFbFW1FRQV5e8hwoS6Z8kikXSPx8NpY38ODiGmoanGuHO6MLEjeXgyX6e3OwI81n/Pjxi9298HDt2vvg8+XAXw4UhcCx7l5iZoOB18xsqbuvO3hFd58OTAcoLCz0cePGtSqAoqIiWrtuPEqmfJIpF0jsfOav3s59r71Pl47ZPH31GLauWpywuTQnkd+b5kQ7n2icdFwC9G80XRDMa87lwFONZ7h7SfC4Hiii6fEHEWlnT723iWtnLOLYHrk89+2xDOvTKdYhSTuLRmFYCAw1s0FmlkX4y3/WwY3MbDjQDXin0bxuZpYdjPcExtLysQkRaWO/fnUtt/7vUs4Z0pNnrjtLt7VIURHvSnL3ejO7HphD+HTVx919uZndCSxy9wNF4nJgpjc9qDEC+J2ZhQgXqXsan80kIu3n16+u5b/mreFLp/bjvimjyNBVzCkrKscY3H02MPugebcfNP3TZtZ7GzgpGjGISOv95rX/Kwr3X3qyelRLcfpJIJLiHppfzANz1/BFFQUJqDCIpLDfFhVz/5zVTD7lGB5QUZCACoNIinr0H+u57+XVXHLyMfyXioI0osIgkoL+9uEn/PzFlVx0Uh9+ednJOtAsTei/QSTFLPx4Nzc++yGnD+zGLy87RUVBPkX/ESIpZP2OCv7tj4so6NqR6V8vpENmeqxDkjikwiCSInZV1HD1EwtJN+MPV59Ot1zdIVWap456RFJAdV0D1/5xEaV7q3lq2pkc2yM31iFJHFNhEEly7s6Nz3zIks1lPHzlaZymDnbkMLQrSSTJPfqPDby4dCu3TBrOpBP7xjocSQAqDCJJ7L0Nu7nn5VVceGIfpp07ONbhSIJQYRBJUtv3VXP9n99nQPcc7psySj2vyRFTYRBJQvUNIb7z5w8or67j4a+dRqcOmbEOSRKIDj6LJKEH5q7h3Q27+eVlJzO8T+dYhyMJRlsMIklm7vJSHnl9HVeeMYAvnVYQ63AkAakwiCSRT8qquPHZDxlV0IXbPz8y1uFIgopKYTCzSWa22syKzeyWZpZfZWY7zGxJMFzbaNlUM1sbDFOjEY9IKgqFnJue/ZBQyPnNFaeRnaHbXUjrRHyMwczSgYeA84EtwEIzm9VMF51Pu/v1B63bHbgDKAQcWBysuyfSuERSzR/e/pi31+3i3i+fxIAeObEORxJYNLYYxgDF7r7e3WuBmcDkI1z3AmCeu+8OisE8YFIUYhJJKWu37ePel1cxYURvLivsH+twJMFF46ykfsDmRtNbgDOaafdlMzsXWAP8wN03t7Buv+ZexMymAdMA8vPzKSoqalWwFRUVrV43HiVTPsmUC7RfPvUh52cLqslOC/H5PhW8/vrrUX8NvTfxLdr5tNfpqn8DnnL3GjP7d2AGcN7RPIG7TwemAxQWFvq4ceNaFUhRURGtXTceJVM+yZQLtF8+D8xZzcbyYn739dFccEKfNnkNvTfxLdr5RGNXUgnQeNu1IJj3T+6+y91rgslHgdFHuq6ItGzxxj38tqiYS0cXtFlRkNQTjcKwEBhqZoPMLAu4HJjVuIGZNb5z1yXAymB8DjDRzLqZWTdgYjBPRA6jsraeG55ZwjFdO+rUVImqiHcluXu9mV1P+As9HXjc3Zeb2Z3AInefBXzXzC4B6oHdwFXBurvN7GeEiwvAne6+O9KYRFLBr15Zy8ZdlcycdqZueSFRFZVjDO4+G5h90LzbG43fCtzawrqPA49HIw6RVLGsZC+P/mM9V4wZwJmDe8Q6HEkyuvJZJMHUN4S49X+X0iMvm1suHB7rcCQJ6SZ6Ignmibc/ZmnJXh766ml06ahdSBJ92mIQSSBb9lTyy3lrOG94by46SWchSdtQYRBJEO7O7S8sB+DOySeo4x1pMyoMIgnixaVbeW3Vdm6cOIyCbroXkrQdFQaRBLC3so6fzlrBSf26cNXZA2MdjiQ5HXwWSQAPvrKG3ftreOLq00lP0y4kaVvaYhCJc6tKy/mfBRu58oxjObFfl1iHIylAhUEkjrk7/zlrBZ06ZHDD+cfHOhxJESoMInHspWWlvLN+FzdOHEa33KxYhyMpQoVBJE5V1TZw14srGd6nE18dMyDW4UgK0cFnkTj1yOvrKCmrYua0M3XAWdqVthhE4tCWPZU88vo6Lh7VVzfJk3anwiASh34xeyVm8OOLRsQ6FElBKgwicebtdTuZvbSUb48bwjFdO8Y6HElBKgwicSQUcn7+95X069qRfzt3cKzDkRQVlcJgZpPMbLWZFZvZLc0sv8HMVpjZR2b2qpkd22hZg5ktCYZZB68rkkqe+6CEFVvL+dGkYXTITI91OJKiIj4ryczSgYeA84EtwEIzm+XuKxo1+wAodPdKM/smcB/wlWBZlbufEmkcIomuuq6BB+auZlRBFz4/6phYhyMpLBpbDGOAYndf7+61wExgcuMG7j7f3SuDyQVAQRReVySpPPbmBrburebHF40gTaenSgyZu0f2BGZTgEnufm0w/XXgDHe/voX2vwFK3f3nwXQ9sASoB+5x9+dbWG8aMA0gPz9/9MyZM1sVb0VFBXl5ea1aNx4lUz7JlAscXT7ltc6PXq9kRI90vndahzaO7Oil8nuTCI40n/Hjxy9298LDNnT3iAZgCvBoo+mvA79poe3XCG8xZDea1y94HAx8DBx3uNccPXq0t9b8+fNbvW48SqZ8kikX96PL5yfPL/XBt77oa7fta7uAIpDK700iONJ8gEV+BN/r0diVVAL0bzRdEMxrwswmALcBl7h7TaPCVBI8rgeKgFOjEJNIwli3o4I/v7uJK8b0Z0jv5PkVK4krGoVhITDUzAaZWRZwOdDk7CIzOxX4HeGisL3R/G5mlh2M9wTGAo0PWoskvXtfWkWHzHS+P0F3T5X4EPFZSe5eb2bXA3OAdOBxd19uZncS3myZBdwP5AHPBv3UbnL3S4ARwO/MLES4SN3jTc9mEklq723YzdwV27hp4vH0zMuOdTgiQJRuoufus4HZB827vdH4hBbWexs4KRoxiCQad+ful1bSp3MHrjlHF7NJ/NCVzyIxMm/FNj7YVMb3JwylY5YuZpP4ocIgEgMNIef+OasZ3DOXKaN1WY/EFxUGkRh47oMS1m6v4KYLhpGRro+hxBf9R4q0s5r6Bh6ct4ZRBV248MQ+sQ5H5FNUGETa2ZMLNlFSVsXNk4YTnKUnEldUGETaUUVNPb+ZX8w5Q3oydkjPWIcj0iwVBpF29Og/1rN7fy0/vGBYrEMRaZEKg0g72VVRw+/fWM9FJ/Xh5P5dYx2OSItUGETayUPz11FdH+LGidpakPimwiDSDkrKqvjTgo1MOa2A43rpRnkS31QYRNrBr19dC8B3JwyNcSQih6fCINLGNuzcz7OLt/DVMwbQr2vHWIcjclgqDCJt7FevrCErPY1vjx8S61BEjogKg0gbWlVazqwPP+HqsQPp1Um31ZbEoMIg0ob+a+4a8rIz+Pdzj4t1KCJHLCqFwcwmmdlqMys2s1uaWZ5tZk8Hy981s4GNlt0azF9tZhdEIx6ReLC+rIF5K7Yx7TOD6ZKTGetwRI5YxIXBzNKBh4ALgZHAFWY28qBm1wB73H0I8CBwb7DuSMJdgZ4ATAJ+GzyfSML769pauudmcfU5g2IdishRicYWwxig2N3Xu3stMBOYfFCbycCMYPwvwOcsfPewycBMd69x9w1AcfB8IgntnXW7WL4rxLfGHUdedlQ6ShRpN9EoDP2AzY2mtwTzmm3j7vXAXqDHEa4rklDcnQfmrqZbtvG1M4+NdTgiRy1hfsqY2TRgGkB+fj5FRUWtep6KiopWrxuPkimfZMnlwx31LN5Yw+XHOQve+kesw4mKZHlvDlA+hxaNwlAC9G80XRDMa67NFjPLALoAu45wXQDcfTowHaCwsNDHjRvXqmCLiopo7brxKJnySYZcQiHnvl+/yYDu6Uw4joTP54BkeG8aUz6HFo1dSQuBoWY2yMyyCB9MnnVQm1nA1GB8CvCau3sw//LgrKVBwFDgvSjEJBITLy0rZcXWcn5w/lAy0tQJjySmiAtDcMzgemAOsBJ4xt2Xm9mdZnZJ0OwxoIeZFQM3ALcE6y4HngFWAC8D33b3hkhjEomF+oYQv5y3mqG987jkZB0qk8QVlWMM7j4bmH3QvNsbjVcDl7aw7l3AXdGIQySWnvughHU79vPI10aTrq0FSWC68lkkCmrqG/jVK2sZVdCFC07Ij3U4IhFRYRCJgqcXbqakrIobJw4jfImOSOJSYRCJUFVtA79+rZgxA7tz7tCesQ5HJGIqDCIR+uM7H7NjXw03XaCtBUkOKgwiESivruPh19fx2eN7MWZQ91iHIxIVKgwiEXj0jfWUVdZx08RhsQ5FJGpUGERaace+Gh59cwP/MqovJxV0iXU4IlGjwiDSSg/NL6amPsSN5x8f61BEokqFQaQVNu+u5Ml3N3JZYX8G98qLdTgiUaXCINIKD85bQ5oZ3/vc0FiHIhJ1KgwiR2lVaTnPLSnhqrED6dOlQ6zDEYk6FQaRo/TAnNXkZWfwzc8eF+tQRNqECoPIUVj08W5eWbmd6z57HF1zsmIdjkibUGEQOULuzr0vr6JXp2yuHjsw1uGItBkVBpEj9Nqq7Sz8eA/f/dxQcrISpldckaOmwiByBOobQtz90ioG9czl8tP7H34FkQQWUWEws+5mNs/M1gaP3Zppc4qZvWNmy83sIzP7SqNlT5jZBjNbEgynRBKPSFt5ZtEWirdXcPOk4WSm6/eUJLdI/8NvAV5196HAq8H0wSqBb7j7CcAk4Fdm1rXR8h+6+ynBsCTCeESirqKmnl/OW8PpA7upEx5JCZEWhsnAjGB8BvCFgxu4+xp3XxuMfwJsB3pF+Loi7Wb6G+vZWVHDjy8aodtqS0qItDDku/vWYLwUOOTPKTMbA2QB6xrNvivYxfSgmWVHGI9IVG0rr+b3b6zn4lF9OXXAp/aUiiQlc/dDNzB7BejTzKLbgBnu3rVR2z3u3uynx8z6AkXAVHdf0GheKeFiMR1Y5+53trD+NGAaQH5+/uiZM2ceOrMWVFRUkJeXPPe2SaZ84jGXx5fV8FZJPXd/piO9c47ud1Q85tNayZQLpG4+48ePX+zuhYdt6O6tHoDVQN9gvC+wuoV2nYH3gSmHeK5xwN+P5HVHjx7trTV//vxWrxuPkimfeMtl5da9PvCWv/udf1veqvXjLZ9IJFMu7qmbD7DIj+A7NtJdSbOAqcH4VOCFgxuYWRbwHPBHd//LQcv6Bo9G+PjEsgjjEYmau2evolN2Bt85b0isQxFpV5EWhnuA881sLTAhmMbMCs3s0aDNZcC5wFXNnJb6pJktBZYCPYGfRxiPSFS8sWYHr6/ZwXfOG6pbX0jKiejyTXffBXyumfmLgGuD8T8Bf2ph/fMieX2RtlDXEOLOv69gQPccvnH2sbEOR6Td6UodkYP88Z2NFG+v4CcXjyQ7Iz3W4Yi0OxUGkUZ2VtTwq3lrOPf4XkwY0TvW4YjEhAqDSCP3v7yaqroGbr94pC5mk5SlwiAS+GhLGc8s3szVYwcypHfynOMucrRUGESAUMj56azl9MjN5rvqx1lSnAqDCPD8khLe31TGzZOG0alDZqzDEYkpFQZJeRU19dz90ipO7t+VL59WEOtwRGJO3VBJyvvvV9awY18Nv/9GIWlpOuAsoi0GSWnLSvby+Fsfc/np/Tmlf9fDryCSAlQYJGU1hJwfP7eUbjlZ3HrhiFiHIxI3VBgkZc14+2M+2rKXOz4/ki45OuAscoAKg6SkkrIqHpi7mvHDenHxqL6xDkckrqgwSMpxd25/fhnucOfkE3WFs8hBVBgk5by0rJRXV23nxonH0797TqzDEYk7KgySUvZW1XHHrOWc2K8zV509MNbhiMQlXccgKeWel1axq6KGP1x1Ohnp+l0k0pyIPhlm1t3M5pnZ2uCxWwvtGhr13jar0fxBZvaumRWb2dNBN6AibWL+6u089d4mrv3MYE7s1yXW4YjErUh/Mt0CvOruQ4FXg+nmVLn7KcFwSaP59wIPuvsQYA9wTYTxiDRr9/5afvSXjxiW34kbzj8+1uGIxLVIC8NkYEYwPgP4wpGuaOFTQc4D/tKa9UWOlLvzH88vpayylge/cgodMtUrm8ihRFoY8t19azBeCuS30K6DmS0yswVmduDLvwdQ5u71wfQWoF+E8Yh8yvNLSpi9tJQbzh/GyGM6xzockbhn7n7oBmavAH2aWXQbMMPduzZqu8fdP3Wcwcz6uXuJmQ0GXgM+B+wFFgS7kTCz/sBL7n5iC3FMA6YB5Ofnj545c+aR5PcpFRUV5OUlTycsyZRPW+SyqyrEf7xVRf9OadwypgNp7XjNgt6b+JWq+YwfP36xuxcetqG7t3oAVgN9g/G+wOojWOcJYApgwE4gI5h/FjDnSF539OjR3lrz589v9brxKJnyiXYuDQ0hv/x37/jIn7zkm3btj+pzHwm9N/ErVfMBFvkRfMdGuitpFjA1GJ8KvHBwAzPrZmbZwXhPYCywIghyflAkWlxfpLUef2sD76zfxe2fH6kL2USOQqSF4R7gfDNbC0wIpjGzQjN7NGgzAlhkZh8SLgT3uPuKYNnNwA1mVkz4mMNjEcYjAoT7b77v5dVMGJHPZYX9Yx2OSEKJ6AI3d99F+HjBwfMXAdcG428DJ7Ww/npgTCQxiBxsz/5avvmn9+nVKZv7p4zSvZBEjpKufJakEgo5P3hmCTv21fDsdWfRLVfXTIocLd0TQJLKb+YXU7R6B7d/fiQnq0c2kVZRYZCk8caaHTz4yhq+eGo/rjxjQKzDEUlYKgySFD4pq+J7Mz9gaO887vqi+lgQiYQKgyS86roGvvXk+9Q1OA9/bTQ5WTp0JhIJfYIkoYVCzk3PfsiSzWU8fOVpHNcrea5mFYkVbTFIQrtvzmr+/tFWbrlwOBeepL6bRaJBhUES1pPvbuSR19dx5RkD+PdzB8c6HJGkocIgCWn+qu385PlljB/Wi/+85AQdbBaJIhUGSTjLSvby7T+/z4i+nfnNV09TF50iUaZPlCSUTbsq+dcnFtK1YyaPX3U6udk6f0Ik2lQYJGFs3l3JFb9fQE19iD9cPYb8zh1iHZJIUlJhkISweXcll09fQEVNPU9eewbD+nSKdUgiSUuFQeLelj3hLYV91XX86ZozOLFfl1iHJJLUVBgkrpWUVXHF7xdQXlXHn649g5MKVBRE2pqO3EncOrClUFYZ3lIYVaC7pYq0h4i2GMysu5nNM7O1wWO3ZtqMN7MljYZqM/tCsOwJM9vQaNkpkcQjyWPplr188bdvU1ZZx/9cc4ZuoS3SjiLdlXQL8Kq7DwVeDaabcPf57n6Ku58CnAdUAnMbNfnhgeXuviTCeCQJzF+1na9Mf4es9DT++s2zOUVFQaRdRVoYJgMzgvEZwBcO034K8JK7V0b4upKknnx3I9fMWMjgXrk8962zOT5fZx+JtLdIC0O+u28NxkuB/MO0vxx46qB5d5nZR2b2oJllRxiPJKhQyHlmdS23PbeMccN68/S0s+it6xREYsLc/dANzF4B+jSz6DZghrt3bdR2j7t/6jhDsKwv8BFwjLvXNZpXCmQB04F17n5nC+tPA6YB5Ofnj545c+ZhUmteRUUFeXnJc2vmZMhnf53z6NIaPtjewHn9M7hyRBbpaYl/76NkeG8OSKZcIHXzGT9+/GJ3LzxsQ3dv9QCsBvoG432B1Ydo+z1g+iGWjwP+fiSvO3r0aG+t+fPnt3rdeJTo+XywaY+ffferPuTHL/qtf5jroVAo1iFFTaK/N40lUy7uqZsPsMiP4Ds20l1Js4CpwfhU4IVDtL2Cg3YjBVsMWPjWmF8AlkUYjyQId+exNzdw6SNvYwbPXnc2Ewdm6i6pInEg0usY7gGeMbNrgI3AZQBmVghc5+7XBtMDgf7A6wet/6SZ9QIMWAJcF2E8kgD2VtZx018+ZN6KbUwcmc/9U06mS04mRetiHZmIQISFwd13AZ9rZv4i4NpG0x8D/Zppd14kry+J5+Vlpdz+wjL2VNZy+8UjuXrsQG0liMQZXfks7WJbeTV3vLCcl5eXMqJvZx6berpubyESp1QYpE2FQs7MhZu5+6WV1NaHuHnScK79zCAy1bmOSNxSYZA2s3jjbn4xexWLN+7hrME9uPtLJzGwZ26swxKRw1BhkKgr3l7BfS+vYu6KbfTqlM19U0Zx6egCHUsQSRAqDBI128qr+dUra3h64WZysjK48fzjueYzg8jJ0r+ZSCLRJ1Yitm5HBY/+YwN/fX8L7s7Uswdy/fgh9MjTHU5EEpEKg7SKu7No4x6mv7GeV1ZuIzM9jS+fVsA3P3scA3rkxDo8EYmACoMclf019by4dCt/fncTSzaX0TUnk++MH8I3zh5IT20hiCQFFQY5rANbB88s3MyLS7dSWdvA4J653Dklp+JDAAAHy0lEQVT5BKaMLtAxBJEko0+0NMvdWf5JOS8vK+XFpVvZsHM/uVnpfH7UMVx2egGnDeims4xEkpQKg/xTfUOIxRv3MGf5NuYsL6WkrIo0gzMG9eBb447jopP6kputfxmRZKdPeQpzd9btqODNtTt5s3gX767fxb6aerIy0vjMkJ58b8JQJozIp3tuVqxDFZF2pMKQQmrrQ6zYWs4Hm/bw/qYy3tuwi23lNQD0796Ri0/uyzlDevHZYb3I05aBSMrSpz9JVdc1sHZbBStLy1m5tZyPtuxlacleautDAPTt0oHCgd05Z0hPxh7XU6eYisg/qTAkuMo658PNZazfWcGGHftZt3M/q0v3sWHnfhpC4W5bO2amM/KYzkw961hOHdCNUwd0pW+XjjGOXETilQpDHAuFnD2VtWwrr6G0vIqSPVVs2VPFlrLgcXclu/bXwqtvAZCeZhR068jx+Z246MQ+jOjbmeF9O3Ns9xzSkqAPZRFpHxEVBjO7FPgpMAIYE3TQ01y7ScB/A+nAo+5+TzB/EDAT6AEsBr7u7rWRxBTPGkJOeVUde6vqKDvwWFnLzopadlXUsHt/ML6/hu3lNWzfV01dgzd5jqyMNAq6dqRft45MPCGfhr2lTBgzisG98hjQPYesDN3OWkQiE+kWwzLgS8DvWmpgZunAQ8D5wBZgoZnNcvcVwL3Ag+4+08weAa4BHo4wplYLhZzahhB1DSFq60PU/HNooKYuPF5V10B1MFTVNlBV10BlbQP7a+qprG2gsrae/bUNVFTXs6+6joqaevZV14ena+pbfO2MNKN7bhbdc7PomZfNmYPzyO+cTX7nDuR3zqZ35w4UdO1Iz7zsJr/+i4p2M+6EPu3x5xGRFBFp154rgcNd6DQGKHb39UHbmcBkM1sJnAd8NWg3g/DWR5sVhtueW8qryyrJfO816hucuoYQdf98DH3q1/nRSE8zcrLSyc3KICcrnbwOGXTqkEGvTtl06pBJXnYGnTtm0rVjJl1zwkOXjll0zcmkZ242nTtm6IIxEYkL7XGMoR+wudH0FuAMwruPyty9vtH8T/ULfYCZTQOmAeTn51NUVHTUgdTsruWYjiGys2rJMCM9DdIt/Gs9My2DdIPMtPCXfGZaeDwzvdF4mpGVDlnpkB3Mz043sjMgww4ukPXBcJB6oDw87CM8bP50qyNWUVHRqr9FPEqmXCC58kmmXED5HM5hC4OZvQI0t6/iNnd/IWqRHIa7TwemAxQWFvq4ceOO+jnGjYOioiJas268SqZ8kikXSK58kikXUD6Hc9jC4O4TInyNEqB/o+mCYN4uoKuZZQRbDQfmi4hIDLXHKSwLgaFmNsjMsoDLgVnu7sB8YErQbirQblsgIiLSvIgKg5l90cy2AGcBL5rZnGD+MWY2GyDYGrgemAOsBJ5x9+XBU9wM3GBmxYSPOTwWSTwiIhK5SM9Keg54rpn5nwAXNZqeDcxupt16wmctiYhInNDVUCIi0oQKg4iINKHCICIiTagwiIhIExY+azSxmNkOYGMrV+8J7IxiOLGWTPkkUy6QXPkkUy6Quvkc6+69DtcoIQtDJMxskbsXxjqOaEmmfJIpF0iufJIpF1A+h6NdSSIi0oQKg4iINJGKhWF6rAOIsmTKJ5lygeTKJ5lyAeVzSCl3jEFERA4tFbcYRETkEFKyMJjZz8zsIzNbYmZzzeyYWMfUWmZ2v5mtCvJ5zsy6xjqmSJjZpWa23MxCZpaQZ42Y2SQzW21mxWZ2S6zjiYSZPW5m281sWaxjiQYz629m881sRfB/9r1Yx9RaZtbBzN4zsw+DXP4zas+diruSzKyzu5cH498FRrr7dTEOq1XMbCLwmrvXm9m9AO5+c4zDajUzGwGECPcjfpO7L4pxSEcl6ON8DY36OAeuCPo4Tzhmdi5QAfzR3U+MdTyRMrO+QF93f9/MOgGLgS8k4vtj4S4jc929wswygTeB77n7gkifOyW3GA4UhUAukLDV0d3nNuoedQHhDo8SlruvdPfVsY4jAv/s49zda4GZwOQYx9Rq7v4GsDvWcUSLu2919/eD8X2EuwJosUvheOZhFcFkZjBE5bssJQsDgJndZWabgSuB22MdT5T8K/BSrINIcc31cZ6QXzzJzswGAqcC78Y2ktYzs3QzWwJsB+a5e1RySdrCYGavmNmyZobJAO5+m7v3B54k3JFQ3DpcLkGb24B6wvnEtSPJR6QtmVke8Ffg+wftQUgo7t7g7qcQ3lMwxsyisrsvoo564tlR9FX9JOFOhO5ow3AicrhczOwq4GLgc54AB42i0I94PGupj3OJE8H++L8CT7r7/8Y6nmhw9zIzmw9MAiI+USBptxgOxcyGNpqcDKyKVSyRMrNJwI+AS9y9MtbxSPN9nMc4JgkEB2wfA1a6+y9jHU8kzKzXgbMQzawj4RMeovJdlqpnJf0VGEb47JeNwHXunpC/6oL+srOBXcGsBYl6hhWE+xEHfg30AsqAJe5+QWyjOjpmdhHwKyAdeNzd74pxSK1mZk8B4wjfvXMbcIe7J2zf7GZ2DvAPYCnhzz/Aj4PuhxOKmY0CZhD+P0sDnnH3O6Py3KlYGEREpGUpuStJRERapsIgIiJNqDCIiEgTKgwiItKECoOIiDShwiAiIk2oMIiISBMqDCIi0sT/B4oh55M5vWcKAAAAAElFTkSuQmCC\n", "text/plain": [ "
" ] }, "metadata": { "needs_background": "light" }, "output_type": "display_data" } ], "source": [ "plt.plot(x, backend.get_value(activations.tanh(x)))\n", "plt.grid(True); plt.show()" ] } ], "metadata": { "kernelspec": { "display_name": "Python 3", "language": "python", "name": "python3" }, "language_info": { "codemirror_mode": { "name": "ipython", "version": 3 }, "file_extension": ".py", "mimetype": "text/x-python", "name": "python", "nbconvert_exporter": "python", "pygments_lexer": "ipython3", "version": "3.6.6" } }, "nbformat": 4, "nbformat_minor": 2 }