{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Your owe a bike store. During week $t$, the (random) demand is $D_t$ units. \n",
    "On Monday morning you may choose to command $a_t$ additional units: they are delivered immediately before the shop opens. For each week:\n",
    "\n",
    "  * Maintenance cost: $h(s)$ for $s$ units in stock left from the previous week (no maintenance is needed for newly commanded items)\n",
    "  * Command cost: $C(a)$ for $a$ units\n",
    "  * Sales profit: $f(q)$ for $q$ units sold\n",
    "  * Constraint: \n",
    "    - your warehouse has a maximal capacity of $M$ unit (any additionnal bike gets stolen)\n",
    "    - you cannot sell bikes that you don't have in stock\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "* State: number of bikes in stock left from the last week => state space $\\mathcal{S} = \\{0,\\dots, M\\}$\n",
    "* Action: number of bikes commanded at the beginning of the week => action space $\\mathcal{A} = \\{0, \\dots ,M\\}$ \n",
    "* Reward = balance of the week: if you command $A_t$ bikes,\n",
    "$$X_t = - C(A_t) - h(S_t) + f\\big(\\min(D_t, S_t+A_t, M) \\big)$$\n",
    "* Transitions: you end the week with $S_{t+1} = \\max\\big(0, \\min(M, S_t + A_t)  - D_t \\big)$ bikes"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Here we assume that $h(s) = h\\cdot s$, $f(q) = p\\cdot q$, $C(a) = c_0 \\mathbb{1}\\{a>0\\} + c\\cdot a$"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "rdemand (generic function with 1 method)"
      ]
     },
     "execution_count": 1,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Initialization\n",
    "\n",
    "## Loading useful packages\n",
    "using LinearAlgebra\n",
    "using Statistics\n",
    "# using StatsBase\n",
    "using Plots\n",
    "\n",
    "## Constants of the problem\n",
    "M = 15 # stock capacity\n",
    "h = 0.3 # maintenance cost per unit and per week-end\n",
    "p = 1 # selling price per unit\n",
    "c0 = 0.8 # shpping cost per command\n",
    "c = 0.5 # buying price per unit\n",
    "\n",
    "s0 = M # initial stock\n",
    "gamma = 0.99 # discount factor: I do not care what happens after more than 100 weeks (but in fact the optimal policy has a limit when gamma tends to 1)\n",
    "\n",
    "## The demand \n",
    "\n",
    "avgdem = 9 # average demand\n",
    "# demande distribution function (everything >=M is grouped together since at most M bikes can be sold anyway)\n",
    "# pdem = [exp(-avgdem) * avgdem^(big(k)) / factorial(big(k)) for k in 0:M] # Poisson demand (small tail)\n",
    "pdem = [1/(1+avgdem)*(1-1/(1+avgdem))^k for k in 0:M]# Exponential demand (large tail)\n",
    "pdem[M+1] = 1 - sum(pdem[1:M])\n",
    "cpdem = cumsum(pdem)\n",
    "function rdemand()\n",
    "    u = rand()# random demand generator\n",
    "    return( sum([u>cpdem[k] for k in 1:(M+1)]) )\n",
    "end"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "simu (generic function with 1 method)"
      ]
     },
     "execution_count": 2,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Transition and Reward functions\n",
    "\n",
    "function reward(s, a, d)\n",
    "   return( -h*(s) -c0*(a>0) - c*a + p * min(d, s+a, M) )\n",
    "end\n",
    "\n",
    "function nextState(s, a, d)\n",
    "    return( max(0, min(M, s+a)-d) )\n",
    "end\n",
    "\n",
    "function simu(T, pi)\n",
    "    X = zeros(T+1)\n",
    "    S = s0\n",
    "    for t in 0:T\n",
    "        A = pi(S)\n",
    "        D = rdemand()\n",
    "        X[1+t] = reward(S, A, D)\n",
    "        S = nextState(S, A, D)\n",
    "    end\n",
    "    return(X)\n",
    "end"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/svg+xml": [
       "<?xml version=\"1.0\" encoding=\"utf-8\"?>\n",
       "<svg xmlns=\"http://www.w3.org/2000/svg\" xmlns:xlink=\"http://www.w3.org/1999/xlink\" width=\"800\" height=\"800\" viewBox=\"0 0 3200 3200\">\n",
       "<defs>\n",
       "  <clipPath id=\"clip7200\">\n",
       "    <rect x=\"0\" y=\"0\" width=\"3200\" height=\"3200\"/>\n",
       "  </clipPath>\n",
       "</defs>\n",
       "<polygon clip-path=\"url(#clip7200)\" points=\"\n",
       "0,3200 3200,3200 3200,0 0,0 \n",
       "  \" fill=\"#ffffff\" fill-rule=\"evenodd\" fill-opacity=\"1\"/>\n",
       "<defs>\n",
       "  <clipPath id=\"clip7201\">\n",
       "    <rect x=\"640\" y=\"320\" width=\"2241\" height=\"2241\"/>\n",
       "  </clipPath>\n",
       "</defs>\n",
       "<polygon clip-path=\"url(#clip7200)\" points=\"\n",
       "229.135,1425.62 3152.76,1425.62 3152.76,47.2441 229.135,47.2441 \n",
       "  \" fill=\"#ffffff\" fill-rule=\"evenodd\" fill-opacity=\"1\"/>\n",
       "<defs>\n",
       "  <clipPath id=\"clip7202\">\n",
       "    <rect x=\"229\" y=\"47\" width=\"2925\" height=\"1379\"/>\n",
       "  </clipPath>\n",
       "</defs>\n",
       "<polyline clip-path=\"url(#clip7202)\" style=\"stroke:#000000; stroke-width:2; stroke-opacity:0.1; fill:none\" points=\"\n",
       "  302.655,1425.62 302.655,47.2441 \n",
       "  \"/>\n",
       "<polyline clip-path=\"url(#clip7202)\" style=\"stroke:#000000; stroke-width:2; stroke-opacity:0.1; fill:none\" points=\"\n",
       "  1225.11,1425.62 1225.11,47.2441 \n",
       "  \"/>\n",
       "<polyline clip-path=\"url(#clip7202)\" style=\"stroke:#000000; stroke-width:2; stroke-opacity:0.1; fill:none\" points=\"\n",
       "  2147.56,1425.62 2147.56,47.2441 \n",
       "  \"/>\n",
       "<polyline clip-path=\"url(#clip7202)\" style=\"stroke:#000000; stroke-width:2; stroke-opacity:0.1; fill:none\" points=\"\n",
       "  3070.01,1425.62 3070.01,47.2441 \n",
       "  \"/>\n",
       "<polyline clip-path=\"url(#clip7202)\" style=\"stroke:#000000; stroke-width:2; stroke-opacity:0.1; fill:none\" points=\"\n",
       "  229.135,1170.14 3152.76,1170.14 \n",
       "  \"/>\n",
       "<polyline clip-path=\"url(#clip7202)\" style=\"stroke:#000000; stroke-width:2; stroke-opacity:0.1; fill:none\" points=\"\n",
       "  229.135,911.21 3152.76,911.21 \n",
       "  \"/>\n",
       "<polyline clip-path=\"url(#clip7202)\" style=\"stroke:#000000; stroke-width:2; stroke-opacity:0.1; fill:none\" points=\"\n",
       "  229.135,652.279 3152.76,652.279 \n",
       "  \"/>\n",
       "<polyline clip-path=\"url(#clip7202)\" style=\"stroke:#000000; stroke-width:2; stroke-opacity:0.1; fill:none\" points=\"\n",
       "  229.135,393.347 3152.76,393.347 \n",
       "  \"/>\n",
       "<polyline clip-path=\"url(#clip7202)\" style=\"stroke:#000000; stroke-width:2; stroke-opacity:0.1; fill:none\" points=\"\n",
       "  229.135,134.416 3152.76,134.416 \n",
       "  \"/>\n",
       "<polyline clip-path=\"url(#clip7200)\" style=\"stroke:#000000; stroke-width:4; stroke-opacity:1; fill:none\" points=\"\n",
       "  229.135,1425.62 3152.76,1425.62 \n",
       "  \"/>\n",
       "<polyline clip-path=\"url(#clip7200)\" style=\"stroke:#000000; stroke-width:4; stroke-opacity:1; fill:none\" points=\"\n",
       "  229.135,1425.62 229.135,47.2441 \n",
       "  \"/>\n",
       "<polyline clip-path=\"url(#clip7200)\" style=\"stroke:#000000; stroke-width:4; stroke-opacity:1; fill:none\" points=\"\n",
       "  302.655,1425.62 302.655,1404.94 \n",
       "  \"/>\n",
       "<polyline clip-path=\"url(#clip7200)\" style=\"stroke:#000000; stroke-width:4; stroke-opacity:1; fill:none\" points=\"\n",
       "  1225.11,1425.62 1225.11,1404.94 \n",
       "  \"/>\n",
       "<polyline clip-path=\"url(#clip7200)\" style=\"stroke:#000000; stroke-width:4; stroke-opacity:1; fill:none\" points=\"\n",
       "  2147.56,1425.62 2147.56,1404.94 \n",
       "  \"/>\n",
       "<polyline clip-path=\"url(#clip7200)\" style=\"stroke:#000000; stroke-width:4; stroke-opacity:1; fill:none\" points=\"\n",
       "  3070.01,1425.62 3070.01,1404.94 \n",
       "  \"/>\n",
       "<polyline clip-path=\"url(#clip7200)\" style=\"stroke:#000000; stroke-width:4; stroke-opacity:1; fill:none\" points=\"\n",
       "  229.135,1170.14 272.99,1170.14 \n",
       "  \"/>\n",
       "<polyline clip-path=\"url(#clip7200)\" style=\"stroke:#000000; stroke-width:4; stroke-opacity:1; fill:none\" points=\"\n",
       "  229.135,911.21 272.99,911.21 \n",
       "  \"/>\n",
       "<polyline clip-path=\"url(#clip7200)\" style=\"stroke:#000000; stroke-width:4; stroke-opacity:1; fill:none\" points=\"\n",
       "  229.135,652.279 272.99,652.279 \n",
       "  \"/>\n",
       "<polyline clip-path=\"url(#clip7200)\" style=\"stroke:#000000; stroke-width:4; stroke-opacity:1; fill:none\" points=\"\n",
       "  229.135,393.347 272.99,393.347 \n",
       "  \"/>\n",
       "<polyline clip-path=\"url(#clip7200)\" style=\"stroke:#000000; stroke-width:4; stroke-opacity:1; fill:none\" points=\"\n",
       "  229.135,134.416 272.99,134.416 \n",
       "  \"/>\n",
       "<g clip-path=\"url(#clip7200)\">\n",
       "<text style=\"fill:#000000; fill-opacity:1; font-family:Arial,Helvetica Neue,Helvetica,sans-serif; font-size:48px; text-anchor:middle;\" transform=\"rotate(0, 302.655, 1483.62)\" x=\"302.655\" y=\"1483.62\">0</text>\n",
       "</g>\n",
       "<g clip-path=\"url(#clip7200)\">\n",
       "<text style=\"fill:#000000; fill-opacity:1; font-family:Arial,Helvetica Neue,Helvetica,sans-serif; font-size:48px; text-anchor:middle;\" transform=\"rotate(0, 1225.11, 1483.62)\" x=\"1225.11\" y=\"1483.62\">100</text>\n",
       "</g>\n",
       "<g clip-path=\"url(#clip7200)\">\n",
       "<text style=\"fill:#000000; fill-opacity:1; font-family:Arial,Helvetica Neue,Helvetica,sans-serif; font-size:48px; text-anchor:middle;\" transform=\"rotate(0, 2147.56, 1483.62)\" x=\"2147.56\" y=\"1483.62\">200</text>\n",
       "</g>\n",
       "<g clip-path=\"url(#clip7200)\">\n",
       "<text style=\"fill:#000000; fill-opacity:1; font-family:Arial,Helvetica Neue,Helvetica,sans-serif; font-size:48px; text-anchor:middle;\" transform=\"rotate(0, 3070.01, 1483.62)\" x=\"3070.01\" y=\"1483.62\">300</text>\n",
       "</g>\n",
       "<g clip-path=\"url(#clip7200)\">\n",
       "<text style=\"fill:#000000; fill-opacity:1; font-family:Arial,Helvetica Neue,Helvetica,sans-serif; font-size:48px; text-anchor:end;\" transform=\"rotate(0, 197.135, 1187.64)\" x=\"197.135\" y=\"1187.64\">0</text>\n",
       "</g>\n",
       "<g clip-path=\"url(#clip7200)\">\n",
       "<text style=\"fill:#000000; fill-opacity:1; font-family:Arial,Helvetica Neue,Helvetica,sans-serif; font-size:48px; text-anchor:end;\" transform=\"rotate(0, 197.135, 928.71)\" x=\"197.135\" y=\"928.71\">50</text>\n",
       "</g>\n",
       "<g clip-path=\"url(#clip7200)\">\n",
       "<text style=\"fill:#000000; fill-opacity:1; font-family:Arial,Helvetica Neue,Helvetica,sans-serif; font-size:48px; text-anchor:end;\" transform=\"rotate(0, 197.135, 669.779)\" x=\"197.135\" y=\"669.779\">100</text>\n",
       "</g>\n",
       "<g clip-path=\"url(#clip7200)\">\n",
       "<text style=\"fill:#000000; fill-opacity:1; font-family:Arial,Helvetica Neue,Helvetica,sans-serif; font-size:48px; text-anchor:end;\" transform=\"rotate(0, 197.135, 410.847)\" x=\"197.135\" y=\"410.847\">150</text>\n",
       "</g>\n",
       "<g clip-path=\"url(#clip7200)\">\n",
       "<text style=\"fill:#000000; fill-opacity:1; font-family:Arial,Helvetica Neue,Helvetica,sans-serif; font-size:48px; text-anchor:end;\" transform=\"rotate(0, 197.135, 151.916)\" x=\"197.135\" y=\"151.916\">200</text>\n",
       "</g>\n",
       "<g clip-path=\"url(#clip7200)\">\n",
       "<text style=\"fill:#000000; fill-opacity:1; font-family:Arial,Helvetica Neue,Helvetica,sans-serif; font-size:66px; text-anchor:middle;\" transform=\"rotate(0, 1690.95, 1567.48)\" x=\"1690.95\" y=\"1567.48\">week</text>\n",
       "</g>\n",
       "<g clip-path=\"url(#clip7200)\">\n",
       "<text style=\"fill:#000000; fill-opacity:1; font-family:Arial,Helvetica Neue,Helvetica,sans-serif; font-size:66px; text-anchor:middle;\" transform=\"rotate(-90, 73.2861, 736.431)\" x=\"73.2861\" y=\"736.431\">cumulated profit</text>\n",
       "</g>\n",
       "<polyline clip-path=\"url(#clip7202)\" style=\"stroke:#009af9; stroke-width:12; stroke-opacity:1; fill:none\" points=\"\n",
       "  311.879,1156.16 321.104,1131.82 330.328,1121.98 339.553,1146.84 348.777,1185.68 358.002,1186.71 367.226,1147.87 376.451,1146.84 385.675,1145.8 394.9,1160.3 \n",
       "  404.124,1130.27 413.349,1116.28 422.574,1117.84 431.798,1116.8 441.023,1139.07 450.247,1119.91 459.472,1116.28 468.696,1086.77 477.921,1083.14 487.145,1082.1 \n",
       "  496.37,1071.23 505.594,1071.23 514.819,1054.66 524.043,1056.21 533.268,1073.3 542.492,1121.46 551.717,1112.66 560.941,1097.64 570.166,1091.43 579.39,1092.98 \n",
       "  588.615,1120.43 597.839,1138.55 607.064,1173.77 616.289,1148.91 625.513,1181.53 634.738,1205.36 643.962,1161.86 653.187,1145.28 662.411,1164.96 671.636,1195.52 \n",
       "  680.86,1178.43 690.085,1198.11 699.309,1178.43 708.534,1183.09 717.758,1173.77 726.983,1152.02 736.207,1133.37 745.432,1127.16 754.656,1126.12 763.881,1150.98 \n",
       "  773.105,1198.11 782.33,1224 791.554,1198.62 800.779,1192.93 810.004,1182.05 819.228,1172.21 828.453,1202.25 837.677,1234.87 846.902,1204.84 856.126,1204.84 \n",
       "  865.351,1193.45 874.575,1205.36 883.8,1191.37 893.024,1168.07 902.249,1155.64 911.473,1157.19 920.698,1158.75 929.922,1147.87 939.147,1141.66 948.371,1135.44 \n",
       "  957.596,1170.66 966.82,1200.18 976.045,1208.98 985.269,1240.57 994.494,1271.64 1003.72,1301.16 1012.94,1301.16 1022.17,1308.41 1031.39,1270.09 1040.62,1292.36 \n",
       "  1049.84,1255.59 1059.07,1241.61 1068.29,1230.21 1077.51,1229.18 1086.74,1197.07 1095.96,1188.27 1105.19,1161.86 1114.41,1171.18 1123.64,1196.55 1132.86,1157.19 \n",
       "  1142.09,1157.19 1151.31,1157.19 1160.54,1163.93 1169.76,1178.43 1178.98,1218.82 1188.21,1204.32 1197.43,1243.68 1206.66,1276.3 1215.88,1311.52 1225.11,1304.27 \n",
       "  1234.33,1285.11 1243.56,1276.3 1252.78,1275.27 1262.01,1300.12 1271.23,1319.29 1280.45,1289.77 1289.68,1304.27 1298.9,1350.88 1308.13,1365.38 1317.35,1386.61 \n",
       "  1326.58,1371.07 1335.8,1352.95 1345.03,1362.27 1354.25,1338.45 1363.47,1327.05 1372.7,1297.54 1381.92,1293.91 1391.15,1262.32 1400.37,1297.54 1409.6,1340 \n",
       "  1418.82,1291.84 1428.05,1306.34 1437.27,1335.86 1446.5,1343.63 1455.72,1326.02 1464.94,1350.88 1474.17,1379.36 1483.39,1361.23 1492.62,1361.23 1501.84,1329.13 \n",
       "  1511.07,1312.55 1520.29,1312.55 1529.52,1329.64 1538.74,1356.05 1547.97,1308.93 1557.19,1323.43 1566.41,1299.09 1575.64,1277.34 1584.86,1265.95 1594.09,1265.95 \n",
       "  1603.31,1287.7 1612.54,1316.7 1621.76,1308.41 1630.99,1301.16 1640.21,1273.71 1649.44,1273.71 1658.66,1301.16 1667.88,1332.23 1677.11,1313.59 1686.33,1297.02 \n",
       "  1695.56,1315.66 1704.78,1311.52 1714.01,1349.84 1723.23,1321.36 1732.46,1322.91 1741.68,1321.88 1750.9,1352.95 1760.13,1330.68 1769.35,1355.54 1778.58,1327.57 \n",
       "  1787.8,1336.89 1797.03,1349.32 1806.25,1332.23 1815.48,1331.71 1824.7,1301.16 1833.93,1318.25 1843.15,1285.62 1852.37,1292.36 1861.6,1260.25 1870.82,1233.32 \n",
       "  1880.05,1233.32 1889.27,1227.11 1898.5,1259.73 1907.72,1214.16 1916.95,1205.36 1926.17,1186.2 1935.4,1190.34 1944.62,1174.8 1953.84,1166 1963.07,1133.89 \n",
       "  1972.29,1122.5 1981.52,1153.05 1990.74,1129.23 1999.97,1104.89 2009.19,1127.16 2018.42,1174.28 2027.64,1143.21 2036.87,1175.84 2046.09,1153.05 2055.31,1136.48 \n",
       "  2064.54,1135.44 2073.76,1134.41 2082.99,1130.78 2092.21,1132.34 2101.44,1119.91 2110.66,1128.19 2119.89,1150.98 2129.11,1171.69 2138.33,1135.44 2147.56,1155.12 \n",
       "  2156.78,1197.59 2166.01,1183.61 2175.23,1177.39 2184.46,1176.36 2193.68,1167.55 2202.91,1145.8 2212.13,1150.46 2221.36,1138.55 2230.58,1109.03 2239.8,1126.12 \n",
       "  2249.03,1140.62 2258.25,1190.86 2267.48,1143.21 2276.7,1142.18 2285.93,1130.78 2295.15,1121.46 2304.38,1130.78 2313.6,1155.12 2322.83,1186.71 2332.05,1206.91 \n",
       "  2341.27,1163.93 2350.5,1188.78 2359.72,1156.16 2368.95,1149.94 2378.17,1187.75 2387.4,1144.77 2396.62,1159.27 2405.85,1167.55 2415.07,1188.78 2424.3,1173.77 \n",
       "  2433.52,1135.96 2442.74,1124.57 2451.97,1096.6 2461.19,1089.35 2470.42,1093.5 2479.64,1097.64 2488.87,1072.27 2498.09,1067.6 2507.32,1104.37 2516.54,1085.21 \n",
       "  2525.77,1054.14 2534.99,1053.1 2544.21,1052.07 2553.44,1053.62 2562.66,1034.46 2571.89,1028.25 2581.11,1001.32 2590.34,1012.19 2599.56,1038.09 2608.79,999.765 \n",
       "  2618.01,975.425 2627.23,971.8 2636.46,946.943 2645.68,948.496 2654.91,937.103 2664.13,925.71 2673.36,945.389 2682.58,975.425 2691.81,946.425 2701.03,932.442 \n",
       "  2710.26,900.853 2719.48,900.853 2728.7,928.299 2737.93,956.782 2747.15,996.657 2756.38,1043.27 2765.6,1056.21 2774.83,1070.19 2784.05,1115.77 2793.28,1140.62 \n",
       "  2802.5,1145.8 2811.73,1108.52 2820.95,1143.73 2830.17,1113.69 2839.4,1117.84 2848.62,1147.36 2857.85,1119.39 2867.07,1104.89 2876.3,1119.39 2885.52,1100.75 \n",
       "  2894.75,1085.73 2903.97,1111.11 2913.2,1158.23 2922.42,1197.07 2931.64,1199.66 2940.87,1204.84 2950.09,1178.43 2959.32,1164.44 2968.54,1176.36 2977.77,1146.84 \n",
       "  2986.99,1112.14 2996.22,1121.46 3005.44,1173.77 3014.66,1203.8 3023.89,1202.77 3033.11,1167.03 3042.34,1168.59 3051.56,1170.14 3060.79,1161.34 3070.01,1156.16 \n",
       "  \n",
       "  \"/>\n",
       "<polyline clip-path=\"url(#clip7202)\" style=\"stroke:#e26f46; stroke-width:12; stroke-opacity:1; fill:none\" points=\"\n",
       "  311.879,1213.12 321.104,1178.43 330.328,1167.03 339.553,1155.64 348.777,1149.43 358.002,1134.41 367.226,1148.91 376.451,1171.18 385.675,1128.71 394.9,1117.32 \n",
       "  404.124,1116.28 413.349,1097.64 422.574,1086.25 431.798,1095.57 441.023,1069.68 450.247,1058.28 459.472,1046.89 468.696,1035.5 477.921,1044.82 487.145,1039.64 \n",
       "  496.37,1034.46 505.594,1039.64 514.819,1068.64 524.043,1106.96 533.268,1077.44 542.492,1067.6 551.717,1052.59 560.941,1041.19 570.166,1029.8 579.39,1018.41 \n",
       "  588.615,1007.01 597.839,1021.51 607.064,991.997 616.289,1001.32 625.513,1011.68 634.738,974.907 643.962,963.514 653.187,952.121 662.411,971.8 671.636,938.657 \n",
       "  680.86,958.336 690.085,925.192 699.309,944.871 708.534,953.157 717.758,912.764 726.983,901.371 736.207,905.514 745.432,924.674 754.656,884.281 763.881,903.96 \n",
       "  773.105,870.817 782.33,859.424 791.554,848.031 800.779,867.71 810.004,896.71 819.228,857.352 828.453,845.959 837.677,834.566 846.902,823.173 856.126,811.78 \n",
       "  865.351,800.387 874.575,788.994 883.8,803.495 893.024,810.227 902.249,840.781 911.473,870.299 920.698,910.174 929.922,875.477 939.147,874.442 948.371,855.799 \n",
       "  957.596,844.406 966.82,838.191 976.045,828.352 985.269,839.227 994.494,840.781 1003.72,823.173 1012.94,800.905 1022.17,820.584 1031.39,802.977 1040.62,796.245 \n",
       "  1049.84,773.976 1059.07,762.583 1068.29,761.548 1077.51,768.798 1086.74,749.637 1095.96,772.423 1105.19,788.994 1114.41,799.87 1123.64,824.727 1132.86,833.531 \n",
       "  1142.09,836.12 1151.31,802.459 1160.54,783.816 1169.76,772.423 1178.98,761.03 1188.21,749.637 1197.43,748.601 1206.66,729.958 1215.88,718.565 1225.11,712.351 \n",
       "  1234.33,697.333 1243.56,701.476 1252.78,705.101 1262.01,732.547 1271.23,769.316 1280.45,796.762 1289.68,835.084 1298.9,841.816 1308.13,868.745 1317.35,903.442 \n",
       "  1326.58,867.192 1335.8,855.799 1345.03,844.406 1354.25,833.013 1363.47,821.62 1372.7,810.227 1381.92,824.727 1391.15,826.281 1400.37,855.281 1409.6,867.71 \n",
       "  1418.82,892.567 1428.05,922.085 1437.27,956.782 1446.5,930.889 1455.72,948.496 1464.94,911.728 1474.17,900.335 1483.39,920.014 1492.62,886.87 1501.84,875.477 \n",
       "  1511.07,879.62 1520.29,893.603 1529.52,882.728 1538.74,904.996 1547.97,888.424 1557.19,869.263 1566.41,881.692 1575.64,900.335 1584.86,888.942 1594.09,897.228 \n",
       "  1603.31,918.978 1612.54,879.62 1621.76,888.942 1630.99,914.835 1640.21,934.514 1649.44,895.156 1658.66,883.763 1667.88,872.37 1677.11,876.513 1686.33,885.317 \n",
       "  1695.56,878.067 1704.78,889.978 1714.01,916.907 1723.23,879.103 1732.46,867.71 1741.68,887.388 1750.9,906.031 1760.13,941.246 1769.35,984.229 1778.58,1022.03 \n",
       "  1787.8,1042.75 1797.03,1001.84 1806.25,995.622 1815.48,990.961 1824.7,972.318 1833.93,991.997 1843.15,958.853 1852.37,947.46 1861.6,956.782 1870.82,967.139 \n",
       "  1880.05,930.371 1889.27,918.978 1898.5,907.585 1907.72,916.907 1916.95,932.442 1926.17,897.228 1935.4,913.281 1944.62,895.674 1953.84,888.942 1963.07,877.031 \n",
       "  1972.29,894.638 1981.52,857.87 1990.74,846.477 1999.97,835.084 2009.19,823.691 2018.42,827.834 2027.64,815.923 2036.87,797.28 2046.09,816.959 2055.31,783.816 \n",
       "  2064.54,772.423 2073.76,786.923 2082.99,788.477 2092.21,812.298 2101.44,771.387 2110.66,759.994 2119.89,753.78 2129.11,738.762 2138.33,758.441 2147.56,725.297 \n",
       "  2156.78,713.904 2166.01,702.511 2175.23,711.833 2184.46,685.94 2193.68,674.547 2202.91,663.154 2212.13,667.297 2221.36,681.279 2230.58,644.511 2239.8,633.118 \n",
       "  2249.03,637.261 2258.25,651.243 2267.48,650.725 2276.7,650.207 2285.93,613.439 2295.15,602.046 2304.38,590.653 2313.6,579.26 2322.83,588.582 2332.05,562.688 \n",
       "  2341.27,572.01 2350.5,582.367 2359.72,602.564 2368.95,613.439 2378.17,570.974 2387.4,585.474 2396.62,587.028 2405.85,553.885 2415.07,542.492 2424.3,546.635 \n",
       "  2433.52,524.367 2442.74,512.974 2451.97,501.581 2461.19,490.188 2470.42,478.795 2479.64,482.938 2488.87,460.669 2498.09,449.276 2507.32,458.598 2516.54,484.491 \n",
       "  2525.77,442.026 2534.99,440.991 2544.21,422.348 2553.44,410.955 2562.66,425.455 2571.89,395.937 2581.11,410.437 2590.34,422.348 2599.56,444.098 2608.79,472.062 \n",
       "  2618.01,434.258 2627.23,422.865 2636.46,411.472 2645.68,410.437 2654.91,391.794 2664.13,380.401 2673.36,389.722 2682.58,363.829 2691.81,352.436 2701.03,341.043 \n",
       "  2710.26,334.829 2719.48,319.811 2728.7,323.954 2737.93,337.936 2747.15,301.168 2756.38,289.775 2765.6,278.382 2774.83,272.167 2784.05,267.507 2793.28,269.578 \n",
       "  2802.5,285.114 2811.73,244.721 2820.95,233.328 2830.17,227.113 2839.4,212.095 2848.62,200.702 2857.85,189.309 2867.07,208.988 2876.3,175.845 2885.52,164.452 \n",
       "  2894.75,178.952 2903.97,149.434 2913.2,138.041 2922.42,152.541 2931.64,148.916 2940.87,119.398 2950.09,108.005 2959.32,122.505 2968.54,108.523 2977.77,86.2547 \n",
       "  2986.99,100.755 2996.22,97.1298 3005.44,124.577 3014.66,125.094 3023.89,157.202 3033.11,188.274 3042.34,150.47 3051.56,159.791 3060.79,185.684 3070.01,143.22 \n",
       "  \n",
       "  \"/>\n",
       "<polygon clip-path=\"url(#clip7200)\" points=\"\n",
       "325.135,360.204 963.492,360.204 963.492,178.764 325.135,178.764 \n",
       "  \" fill=\"#ffffff\" fill-rule=\"evenodd\" fill-opacity=\"1\"/>\n",
       "<polyline clip-path=\"url(#clip7200)\" style=\"stroke:#000000; stroke-width:4; stroke-opacity:1; fill:none\" points=\"\n",
       "  325.135,360.204 963.492,360.204 963.492,178.764 325.135,178.764 325.135,360.204 \n",
       "  \"/>\n",
       "<polyline clip-path=\"url(#clip7200)\" style=\"stroke:#009af9; stroke-width:12; stroke-opacity:1; fill:none\" points=\"\n",
       "  357.135,239.244 549.135,239.244 \n",
       "  \"/>\n",
       "<g clip-path=\"url(#clip7200)\">\n",
       "<text style=\"fill:#000000; fill-opacity:1; font-family:Arial,Helvetica Neue,Helvetica,sans-serif; font-size:48px; text-anchor:start;\" transform=\"rotate(0, 581.135, 256.744)\" x=\"581.135\" y=\"256.744\">random policy</text>\n",
       "</g>\n",
       "<polyline clip-path=\"url(#clip7200)\" style=\"stroke:#e26f46; stroke-width:12; stroke-opacity:1; fill:none\" points=\"\n",
       "  357.135,299.724 549.135,299.724 \n",
       "  \"/>\n",
       "<g clip-path=\"url(#clip7200)\">\n",
       "<text style=\"fill:#000000; fill-opacity:1; font-family:Arial,Helvetica Neue,Helvetica,sans-serif; font-size:48px; text-anchor:start;\" transform=\"rotate(0, 581.135, 317.224)\" x=\"581.135\" y=\"317.224\">constant policy</text>\n",
       "</g>\n",
       "<polygon clip-path=\"url(#clip7200)\" points=\"\n",
       "229.135,3025.62 3152.76,3025.62 3152.76,1647.24 229.135,1647.24 \n",
       "  \" fill=\"#ffffff\" fill-rule=\"evenodd\" fill-opacity=\"1\"/>\n",
       "<defs>\n",
       "  <clipPath id=\"clip7203\">\n",
       "    <rect x=\"229\" y=\"1647\" width=\"2925\" height=\"1379\"/>\n",
       "  </clipPath>\n",
       "</defs>\n",
       "<polyline clip-path=\"url(#clip7203)\" style=\"stroke:#000000; stroke-width:2; stroke-opacity:0.1; fill:none\" points=\"\n",
       "  302.655,3025.62 302.655,1647.24 \n",
       "  \"/>\n",
       "<polyline clip-path=\"url(#clip7203)\" style=\"stroke:#000000; stroke-width:2; stroke-opacity:0.1; fill:none\" points=\"\n",
       "  1225.11,3025.62 1225.11,1647.24 \n",
       "  \"/>\n",
       "<polyline clip-path=\"url(#clip7203)\" style=\"stroke:#000000; stroke-width:2; stroke-opacity:0.1; fill:none\" points=\"\n",
       "  2147.56,3025.62 2147.56,1647.24 \n",
       "  \"/>\n",
       "<polyline clip-path=\"url(#clip7203)\" style=\"stroke:#000000; stroke-width:2; stroke-opacity:0.1; fill:none\" points=\"\n",
       "  3070.01,3025.62 3070.01,1647.24 \n",
       "  \"/>\n",
       "<polyline clip-path=\"url(#clip7203)\" style=\"stroke:#000000; stroke-width:2; stroke-opacity:0.1; fill:none\" points=\"\n",
       "  229.135,2748.08 3152.76,2748.08 \n",
       "  \"/>\n",
       "<polyline clip-path=\"url(#clip7203)\" style=\"stroke:#000000; stroke-width:2; stroke-opacity:0.1; fill:none\" points=\"\n",
       "  229.135,2400.72 3152.76,2400.72 \n",
       "  \"/>\n",
       "<polyline clip-path=\"url(#clip7203)\" style=\"stroke:#000000; stroke-width:2; stroke-opacity:0.1; fill:none\" points=\"\n",
       "  229.135,2053.36 3152.76,2053.36 \n",
       "  \"/>\n",
       "<polyline clip-path=\"url(#clip7203)\" style=\"stroke:#000000; stroke-width:2; stroke-opacity:0.1; fill:none\" points=\"\n",
       "  229.135,1705.99 3152.76,1705.99 \n",
       "  \"/>\n",
       "<polyline clip-path=\"url(#clip7200)\" style=\"stroke:#000000; stroke-width:4; stroke-opacity:1; fill:none\" points=\"\n",
       "  229.135,3025.62 3152.76,3025.62 \n",
       "  \"/>\n",
       "<polyline clip-path=\"url(#clip7200)\" style=\"stroke:#000000; stroke-width:4; stroke-opacity:1; fill:none\" points=\"\n",
       "  229.135,3025.62 229.135,1647.24 \n",
       "  \"/>\n",
       "<polyline clip-path=\"url(#clip7200)\" style=\"stroke:#000000; stroke-width:4; stroke-opacity:1; fill:none\" points=\"\n",
       "  302.655,3025.62 302.655,3004.94 \n",
       "  \"/>\n",
       "<polyline clip-path=\"url(#clip7200)\" style=\"stroke:#000000; stroke-width:4; stroke-opacity:1; fill:none\" points=\"\n",
       "  1225.11,3025.62 1225.11,3004.94 \n",
       "  \"/>\n",
       "<polyline clip-path=\"url(#clip7200)\" style=\"stroke:#000000; stroke-width:4; stroke-opacity:1; fill:none\" points=\"\n",
       "  2147.56,3025.62 2147.56,3004.94 \n",
       "  \"/>\n",
       "<polyline clip-path=\"url(#clip7200)\" style=\"stroke:#000000; stroke-width:4; stroke-opacity:1; fill:none\" points=\"\n",
       "  3070.01,3025.62 3070.01,3004.94 \n",
       "  \"/>\n",
       "<polyline clip-path=\"url(#clip7200)\" style=\"stroke:#000000; stroke-width:4; stroke-opacity:1; fill:none\" points=\"\n",
       "  229.135,2748.08 272.99,2748.08 \n",
       "  \"/>\n",
       "<polyline clip-path=\"url(#clip7200)\" style=\"stroke:#000000; stroke-width:4; stroke-opacity:1; fill:none\" points=\"\n",
       "  229.135,2400.72 272.99,2400.72 \n",
       "  \"/>\n",
       "<polyline clip-path=\"url(#clip7200)\" style=\"stroke:#000000; stroke-width:4; stroke-opacity:1; fill:none\" points=\"\n",
       "  229.135,2053.36 272.99,2053.36 \n",
       "  \"/>\n",
       "<polyline clip-path=\"url(#clip7200)\" style=\"stroke:#000000; stroke-width:4; stroke-opacity:1; fill:none\" points=\"\n",
       "  229.135,1705.99 272.99,1705.99 \n",
       "  \"/>\n",
       "<g clip-path=\"url(#clip7200)\">\n",
       "<text style=\"fill:#000000; fill-opacity:1; font-family:Arial,Helvetica Neue,Helvetica,sans-serif; font-size:48px; text-anchor:middle;\" transform=\"rotate(0, 302.655, 3083.62)\" x=\"302.655\" y=\"3083.62\">0</text>\n",
       "</g>\n",
       "<g clip-path=\"url(#clip7200)\">\n",
       "<text style=\"fill:#000000; fill-opacity:1; font-family:Arial,Helvetica Neue,Helvetica,sans-serif; font-size:48px; text-anchor:middle;\" transform=\"rotate(0, 1225.11, 3083.62)\" x=\"1225.11\" y=\"3083.62\">100</text>\n",
       "</g>\n",
       "<g clip-path=\"url(#clip7200)\">\n",
       "<text style=\"fill:#000000; fill-opacity:1; font-family:Arial,Helvetica Neue,Helvetica,sans-serif; font-size:48px; text-anchor:middle;\" transform=\"rotate(0, 2147.56, 3083.62)\" x=\"2147.56\" y=\"3083.62\">200</text>\n",
       "</g>\n",
       "<g clip-path=\"url(#clip7200)\">\n",
       "<text style=\"fill:#000000; fill-opacity:1; font-family:Arial,Helvetica Neue,Helvetica,sans-serif; font-size:48px; text-anchor:middle;\" transform=\"rotate(0, 3070.01, 3083.62)\" x=\"3070.01\" y=\"3083.62\">300</text>\n",
       "</g>\n",
       "<g clip-path=\"url(#clip7200)\">\n",
       "<text style=\"fill:#000000; fill-opacity:1; font-family:Arial,Helvetica Neue,Helvetica,sans-serif; font-size:48px; text-anchor:end;\" transform=\"rotate(0, 197.135, 2765.58)\" x=\"197.135\" y=\"2765.58\">0</text>\n",
       "</g>\n",
       "<g clip-path=\"url(#clip7200)\">\n",
       "<text style=\"fill:#000000; fill-opacity:1; font-family:Arial,Helvetica Neue,Helvetica,sans-serif; font-size:48px; text-anchor:end;\" transform=\"rotate(0, 197.135, 2418.22)\" x=\"197.135\" y=\"2418.22\">20</text>\n",
       "</g>\n",
       "<g clip-path=\"url(#clip7200)\">\n",
       "<text style=\"fill:#000000; fill-opacity:1; font-family:Arial,Helvetica Neue,Helvetica,sans-serif; font-size:48px; text-anchor:end;\" transform=\"rotate(0, 197.135, 2070.86)\" x=\"197.135\" y=\"2070.86\">40</text>\n",
       "</g>\n",
       "<g clip-path=\"url(#clip7200)\">\n",
       "<text style=\"fill:#000000; fill-opacity:1; font-family:Arial,Helvetica Neue,Helvetica,sans-serif; font-size:48px; text-anchor:end;\" transform=\"rotate(0, 197.135, 1723.49)\" x=\"197.135\" y=\"1723.49\">60</text>\n",
       "</g>\n",
       "<g clip-path=\"url(#clip7200)\">\n",
       "<text style=\"fill:#000000; fill-opacity:1; font-family:Arial,Helvetica Neue,Helvetica,sans-serif; font-size:66px; text-anchor:middle;\" transform=\"rotate(0, 1690.95, 3167.48)\" x=\"1690.95\" y=\"3167.48\">week</text>\n",
       "</g>\n",
       "<g clip-path=\"url(#clip7200)\">\n",
       "<text style=\"fill:#000000; fill-opacity:1; font-family:Arial,Helvetica Neue,Helvetica,sans-serif; font-size:66px; text-anchor:middle;\" transform=\"rotate(-90, 100.048, 2336.43)\" x=\"100.048\" y=\"2336.43\">cumulated profit with discount</text>\n",
       "</g>\n",
       "<polyline clip-path=\"url(#clip7203)\" style=\"stroke:#009af9; stroke-width:12; stroke-opacity:1; fill:none\" points=\"\n",
       "  311.879,2701.19 321.104,2620.37 330.328,2588.03 339.553,2668.92 348.777,2794.05 358.002,2797.35 367.226,2674.71 376.451,2671.48 385.675,2668.27 394.9,2712.7 \n",
       "  404.124,2621.59 413.349,2579.61 422.574,2584.23 431.798,2581.18 441.023,2646.06 450.247,2590.79 459.472,2580.44 468.696,2496.99 477.921,2486.84 487.145,2483.97 \n",
       "  496.37,2454.14 505.594,2454.14 514.819,2409.59 524.043,2413.72 533.268,2458.75 542.492,2584.39 551.717,2561.65 560.941,2523.26 570.166,2507.53 579.39,2511.42 \n",
       "  588.615,2579.51 597.839,2624.03 607.064,2709.65 616.289,2649.81 625.513,2727.56 634.738,2783.76 643.962,2682.16 653.187,2643.84 662.411,2688.89 671.636,2758.13 \n",
       "  680.86,2719.79 690.085,2763.5 699.309,2720.23 708.534,2730.38 717.758,2710.29 726.983,2663.88 736.207,2624.5 745.432,2611.5 754.656,2609.36 763.881,2660.31 \n",
       "  773.105,2755.93 782.33,2807.94 791.554,2757.48 800.779,2746.26 810.004,2725.07 819.228,2706.08 828.453,2763.46 837.677,2825.16 846.902,2768.92 856.126,2768.92 \n",
       "  865.351,2748.02 874.575,2769.65 883.8,2744.51 893.024,2703.01 902.249,2681.1 911.473,2683.82 920.698,2686.5 929.922,2667.9 939.147,2657.38 948.371,2646.96 \n",
       "  957.596,2705.4 966.82,2753.9 976.045,2768.22 985.269,2819.09 994.494,2868.62 1003.72,2915.21 1012.94,2915.21 1022.17,2926.42 1031.39,2867.74 1040.62,2901.5 \n",
       "  1049.84,2846.31 1059.07,2825.54 1068.29,2808.78 1077.51,2807.27 1086.74,2760.98 1095.96,2748.41 1105.19,2711.09 1114.41,2724.13 1123.64,2759.27 1132.86,2705.31 \n",
       "  1142.09,2705.31 1151.31,2705.31 1160.54,2714.27 1169.76,2733.37 1178.98,2786.03 1188.21,2767.32 1197.43,2817.61 1206.66,2858.89 1215.88,2903 1225.11,2894.01 \n",
       "  1234.33,2870.49 1243.56,2859.79 1252.78,2858.54 1262.01,2888.15 1271.23,2910.75 1280.45,2876.28 1289.68,2893.04 1298.9,2946.37 1308.13,2962.8 1317.35,2986.61 \n",
       "  1326.58,2969.36 1335.8,2949.44 1345.03,2959.58 1354.25,2933.92 1363.47,2921.77 1372.7,2890.6 1381.92,2886.81 1391.15,2854.13 1400.37,2890.2 1409.6,2933.27 \n",
       "  1418.82,2884.91 1428.05,2899.33 1437.27,2928.37 1446.5,2935.94 1455.72,2918.96 1464.94,2942.69 1474.17,2969.62 1483.39,2952.66 1492.62,2952.66 1501.84,2923.21 \n",
       "  1511.07,2908.16 1520.29,2908.16 1529.52,2923.37 1538.74,2946.64 1547.97,2905.53 1557.19,2918.05 1566.41,2897.25 1575.64,2878.84 1584.86,2869.29 1594.09,2869.29 \n",
       "  1603.31,2887.15 1612.54,2910.73 1621.76,2904.06 1630.99,2898.28 1640.21,2876.63 1649.44,2876.63 1658.66,2897.85 1667.88,2921.64 1677.11,2907.51 1686.33,2895.08 \n",
       "  1695.56,2908.92 1704.78,2905.88 1714.01,2933.77 1723.23,2913.25 1732.46,2914.36 1741.68,2913.62 1750.9,2935.35 1760.13,2919.94 1769.35,2936.97 1778.58,2918 \n",
       "  1787.8,2924.26 1797.03,2932.52 1806.25,2921.27 1815.48,2920.94 1824.7,2901.22 1833.93,2912.14 1843.15,2891.51 1852.37,2895.72 1861.6,2875.82 1870.82,2859.3 \n",
       "  1880.05,2859.3 1889.27,2855.56 1898.5,2874.98 1907.72,2848.12 1916.95,2842.99 1926.17,2831.92 1935.4,2834.29 1944.62,2825.49 1953.84,2820.55 1963.07,2802.74 \n",
       "  1972.29,2796.48 1981.52,2813.1 1990.74,2800.27 1999.97,2787.29 2009.19,2799.05 2018.42,2823.67 2027.64,2807.6 2036.87,2824.3 2046.09,2812.75 2055.31,2804.43 \n",
       "  2064.54,2803.92 2073.76,2803.41 2082.99,2801.65 2092.21,2802.39 2101.44,2796.46 2110.66,2800.38 2119.89,2811.04 2129.11,2820.63 2138.33,2804.01 2147.56,2812.94 \n",
       "  2156.78,2832.02 2166.01,2825.8 2175.23,2823.07 2184.46,2822.61 2193.68,2818.81 2202.91,2809.52 2212.13,2811.49 2221.36,2806.5 2230.58,2794.26 2239.8,2801.28 \n",
       "  2249.03,2807.17 2258.25,2827.38 2267.48,2808.4 2276.7,2808 2285.93,2803.55 2295.15,2799.95 2304.38,2803.51 2313.6,2812.73 2322.83,2824.58 2332.05,2832.08 \n",
       "  2341.27,2816.28 2350.5,2825.32 2359.72,2813.57 2368.95,2811.35 2378.17,2824.7 2387.4,2809.68 2396.62,2814.7 2405.85,2817.53 2415.07,2824.73 2424.3,2819.69 \n",
       "  2433.52,2807.13 2442.74,2803.38 2451.97,2794.27 2461.19,2791.93 2470.42,2793.25 2479.64,2794.56 2488.87,2786.62 2498.09,2785.18 2507.32,2796.45 2516.54,2790.64 \n",
       "  2525.77,2781.3 2534.99,2780.99 2544.21,2780.68 2553.44,2781.14 2562.66,2775.6 2571.89,2773.83 2581.11,2766.21 2590.34,2769.25 2599.56,2776.43 2608.79,2765.91 \n",
       "  2618.01,2759.29 2627.23,2758.32 2636.46,2751.7 2645.68,2752.11 2654.91,2749.13 2664.13,2746.19 2673.36,2751.22 2682.58,2758.83 2691.81,2751.56 2701.03,2748.09 \n",
       "  2710.26,2740.32 2719.48,2740.32 2728.7,2746.93 2737.93,2753.73 2747.15,2763.14 2756.38,2774.04 2765.6,2777.04 2774.83,2780.24 2784.05,2790.58 2793.28,2796.16 \n",
       "  2802.5,2797.32 2811.73,2789.11 2820.95,2796.78 2830.17,2790.3 2839.4,2791.19 2848.62,2797.43 2857.85,2791.58 2867.07,2788.57 2876.3,2791.54 2885.52,2787.76 \n",
       "  2894.75,2784.74 2903.97,2789.79 2913.2,2799.08 2922.42,2806.66 2931.64,2807.16 2940.87,2808.15 2950.09,2803.15 2959.32,2800.53 2968.54,2802.74 2977.77,2797.31 \n",
       "  2986.99,2791 2996.22,2792.68 3005.44,2802 3014.66,2807.3 3023.89,2807.12 3033.11,2800.94 3042.34,2801.21 3051.56,2801.47 3060.79,2800 3070.01,2799.14 \n",
       "  \n",
       "  \"/>\n",
       "<polyline clip-path=\"url(#clip7203)\" style=\"stroke:#e26f46; stroke-width:12; stroke-opacity:1; fill:none\" points=\"\n",
       "  311.879,2892.24 321.104,2777.03 330.328,2739.58 339.553,2702.51 348.777,2682.49 358.002,2634.59 367.226,2680.37 376.451,2749.98 385.675,2618.57 394.9,2583.66 \n",
       "  404.124,2580.52 413.349,2524.54 422.574,2490.67 431.798,2518.1 441.023,2442.66 450.247,2409.8 459.472,2377.27 468.696,2345.06 477.921,2371.15 487.145,2356.8 \n",
       "  496.37,2342.59 505.594,2356.65 514.819,2434.62 524.043,2536.62 533.268,2458.84 542.492,2433.17 551.717,2394.39 560.941,2365.26 570.166,2336.42 579.39,2307.87 \n",
       "  588.615,2279.61 597.839,2315.22 607.064,2243.45 616.289,2265.89 625.513,2290.57 634.738,2203.82 643.962,2177.21 653.187,2150.87 662.411,2195.92 671.636,2120.81 \n",
       "  680.86,2164.96 690.085,2091.34 699.309,2134.61 708.534,2152.65 717.758,2065.6 726.983,2041.29 736.207,2050.04 745.432,2090.11 754.656,2006.48 763.881,2046.82 \n",
       "  773.105,1979.56 782.33,1956.68 791.554,1934.02 800.779,1972.77 810.004,2029.29 819.228,1953.35 828.453,1931.58 837.677,1910.03 846.902,1888.7 856.126,1867.58 \n",
       "  865.351,1846.68 874.575,1825.98 883.8,1852.06 893.024,1864.05 902.249,1917.91 911.473,1969.42 920.698,2038.31 929.922,1978.96 939.147,1977.21 948.371,1945.96 \n",
       "  957.596,1927.05 966.82,1916.84 976.045,1900.84 985.269,1918.35 994.494,1920.82 1003.72,1893.04 1012.94,1858.24 1022.17,1888.68 1031.39,1861.72 1040.62,1851.51 \n",
       "  1049.84,1818.09 1059.07,1801.16 1068.29,1799.64 1077.51,1810.2 1086.74,1782.57 1095.96,1815.09 1105.19,1838.51 1114.41,1853.72 1123.64,1888.15 1132.86,1900.22 \n",
       "  1142.09,1903.74 1151.31,1858.5 1160.54,1833.7 1169.76,1818.69 1178.98,1803.84 1188.21,1789.13 1197.43,1787.81 1206.66,1764.22 1215.88,1749.95 1225.11,1742.25 \n",
       "  1234.33,1723.81 1243.56,1728.84 1252.78,1733.21 1262.01,1765.9 1271.23,1809.26 1280.45,1841.3 1289.68,1885.59 1298.9,1893.29 1308.13,1923.8 1317.35,1962.71 \n",
       "  1326.58,1922.46 1335.8,1909.94 1345.03,1897.54 1354.25,1885.27 1363.47,1873.12 1372.7,1861.09 1381.92,1876.25 1391.15,1877.85 1400.37,1907.56 1409.6,1920.17 \n",
       "  1418.82,1945.13 1428.05,1974.47 1437.27,2008.61 1446.5,1983.39 1455.72,2000.37 1464.94,1965.26 1474.17,1954.49 1483.39,1972.91 1492.62,1942.2 1501.84,1931.75 \n",
       "  1511.07,1935.51 1520.29,1948.08 1529.52,1938.4 1538.74,1958.02 1547.97,1943.57 1557.19,1927.02 1566.41,1937.65 1575.64,1953.43 1584.86,1943.88 1594.09,1950.75 \n",
       "  1603.31,1968.62 1612.54,1936.62 1621.76,1944.12 1630.99,1964.75 1640.21,1980.28 1649.44,1949.54 1658.66,1940.73 1667.88,1932.01 1677.11,1935.15 1686.33,1941.75 \n",
       "  1695.56,1936.37 1704.78,1945.13 1714.01,1964.73 1723.23,1937.49 1732.46,1929.36 1741.68,1943.26 1750.9,1956.29 1760.13,1980.67 1769.35,2010.13 1778.58,2035.78 \n",
       "  1787.8,2049.69 1797.03,2022.49 1806.25,2018.4 1815.48,2015.36 1824.7,2003.33 1833.93,2015.9 1843.15,1994.94 1852.37,1987.81 1861.6,1993.58 1870.82,1999.94 \n",
       "  1880.05,1977.6 1889.27,1970.75 1898.5,1963.97 1907.72,1969.46 1916.95,1978.53 1926.17,1958.19 1935.4,1967.37 1944.62,1957.4 1953.84,1953.62 1963.07,1947.02 \n",
       "  1972.29,1956.69 1981.52,1936.69 1990.74,1930.56 1999.97,1924.48 2009.19,1918.47 2018.42,1920.63 2027.64,1914.47 2036.87,1904.93 2046.09,1914.9 2055.31,1898.27 \n",
       "  2064.54,1892.61 2073.76,1899.74 2082.99,1900.5 2092.21,1911.98 2101.44,1892.46 2110.66,1887.07 2119.89,1884.17 2129.11,1877.21 2138.33,1886.23 2147.56,1871.19 \n",
       "  2156.78,1866.07 2166.01,1861 2175.23,1865.11 2184.46,1853.82 2193.68,1848.9 2202.91,1844.03 2212.13,1845.79 2221.36,1851.64 2230.58,1836.4 2239.8,1831.72 \n",
       "  2249.03,1833.4 2258.25,1839.03 2267.48,1838.82 2276.7,1838.62 2285.93,1824.27 2295.15,1819.86 2304.38,1815.5 2313.6,1811.19 2322.83,1814.68 2332.05,1805.07 \n",
       "  2341.27,1808.5 2350.5,1812.27 2359.72,1819.54 2368.95,1823.42 2378.17,1808.43 2387.4,1813.5 2396.62,1814.03 2405.85,1802.68 2415.07,1798.82 2424.3,1800.21 \n",
       "  2433.52,1792.81 2442.74,1789.06 2451.97,1785.34 2461.19,1781.67 2470.42,1778.03 2479.64,1779.34 2488.87,1772.37 2498.09,1768.84 2507.32,1771.7 2516.54,1779.57 \n",
       "  2525.77,1766.8 2534.99,1766.49 2544.21,1761 2553.44,1757.68 2562.66,1761.86 2571.89,1753.43 2581.11,1757.53 2590.34,1760.87 2599.56,1766.9 2608.79,1774.58 \n",
       "  2618.01,1764.3 2627.23,1761.24 2636.46,1758.2 2645.68,1757.93 2654.91,1753.06 2664.13,1750.11 2673.36,1752.5 2682.58,1745.94 2691.81,1743.08 2701.03,1740.25 \n",
       "  2710.26,1738.72 2719.48,1735.07 2728.7,1736.06 2737.93,1739.4 2747.15,1730.72 2756.38,1728.05 2765.6,1725.42 2774.83,1723.99 2784.05,1722.93 2793.28,1723.4 \n",
       "  2802.5,1726.85 2811.73,1717.96 2820.95,1715.48 2830.17,1714.14 2839.4,1710.93 2848.62,1708.52 2857.85,1706.14 2867.07,1710.22 2876.3,1703.42 2885.52,1701.1 \n",
       "  2894.75,1704.02 2903.97,1698.14 2913.2,1695.89 2922.42,1698.72 2931.64,1698.02 2940.87,1692.38 2950.09,1690.22 2959.32,1692.94 2968.54,1690.35 2977.77,1686.25 \n",
       "  2986.99,1688.89 2996.22,1688.24 3005.44,1693.13 3014.66,1693.22 3023.89,1698.83 3033.11,1704.21 3042.34,1697.73 3051.56,1699.31 3060.79,1703.66 3070.01,1696.6 \n",
       "  \n",
       "  \"/>\n",
       "<polygon clip-path=\"url(#clip7200)\" points=\"\n",
       "325.135,1960.2 963.492,1960.2 963.492,1778.76 325.135,1778.76 \n",
       "  \" fill=\"#ffffff\" fill-rule=\"evenodd\" fill-opacity=\"1\"/>\n",
       "<polyline clip-path=\"url(#clip7200)\" style=\"stroke:#000000; stroke-width:4; stroke-opacity:1; fill:none\" points=\"\n",
       "  325.135,1960.2 963.492,1960.2 963.492,1778.76 325.135,1778.76 325.135,1960.2 \n",
       "  \"/>\n",
       "<polyline clip-path=\"url(#clip7200)\" style=\"stroke:#009af9; stroke-width:12; stroke-opacity:1; fill:none\" points=\"\n",
       "  357.135,1839.24 549.135,1839.24 \n",
       "  \"/>\n",
       "<g clip-path=\"url(#clip7200)\">\n",
       "<text style=\"fill:#000000; fill-opacity:1; font-family:Arial,Helvetica Neue,Helvetica,sans-serif; font-size:48px; text-anchor:start;\" transform=\"rotate(0, 581.135, 1856.74)\" x=\"581.135\" y=\"1856.74\">random policy</text>\n",
       "</g>\n",
       "<polyline clip-path=\"url(#clip7200)\" style=\"stroke:#e26f46; stroke-width:12; stroke-opacity:1; fill:none\" points=\"\n",
       "  357.135,1899.72 549.135,1899.72 \n",
       "  \"/>\n",
       "<g clip-path=\"url(#clip7200)\">\n",
       "<text style=\"fill:#000000; fill-opacity:1; font-family:Arial,Helvetica Neue,Helvetica,sans-serif; font-size:48px; text-anchor:start;\" transform=\"rotate(0, 581.135, 1917.22)\" x=\"581.135\" y=\"1917.22\">constant policy</text>\n",
       "</g>\n",
       "</svg>\n"
      ]
     },
     "execution_count": 3,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Simulation: comparing two (basic) policies\n",
    "\n",
    "function randomPolicy(s)\n",
    "    return( rand(0:M) )\n",
    "end\n",
    "\n",
    "function constantPolicy(s)\n",
    "    return(6)\n",
    "end\n",
    "\n",
    "T = trunc(Int, 3/(1-gamma))\n",
    "X1 = simu(T, randomPolicy)\n",
    "X2 = simu(T, constantPolicy)\n",
    "\n",
    "\n",
    "# without discount:\n",
    "p1 = plot([cumsum(X1), cumsum(X2)], xlabel = \"week\", ylabel = \"cumulated profit\", label=[\"random policy\" \"constant policy\"], legend=:topleft, lw=3)\n",
    "\n",
    "# with discount:\n",
    "p2 = plot([cumsum(X1 .* [gamma^t for t in 0:T]), cumsum(X2.* [gamma^t for t in 0:T])], xlabel = \"week\", ylabel = \"cumulated profit with discount\", label=[\"random policy\" \"constant policy\"], legend=:topleft, lw=3)\n",
    "\n",
    "plot(p1, p2, layout = (2,1), size = (800, 800))\n",
    "#png(\"shop_run_profit.png\")\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Transition and Reward kernels\n",
    "K = Dict() # K[s,a][ns] = probability to skip from state s to state ns for the action a\n",
    "avgR = Dict() # avgR[s,a] = average reward received in state s when playing action a\n",
    "\n",
    "## initialization\n",
    "for a in 0:M\n",
    "    for s in 0:M\n",
    "        K[s,a] = zeros(1+M)\n",
    "        avgR[s,a] = 0\n",
    "    end\n",
    "end\n",
    "\n",
    "## computation: iterate over all possible states, actions, and possible demand values\n",
    "for a in 0:M\n",
    "    for s in 0:M\n",
    "        for d in 0:M\n",
    "            ns = nextState(s, a, d); \n",
    "            K[s,a][1+ns] += pdem[1+d]\n",
    "            avgR[s,a] += pdem[1+d] * reward(s, a, d)\n",
    "        end\n",
    "    end\n",
    "end"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "16-element Array{Float64,1}:\n",
       " 47.53820008510554 \n",
       " 47.68531763137929 \n",
       " 47.764349168493375\n",
       " 47.77853042281694 \n",
       " 47.73159355170817 \n",
       " 47.629650367710276\n",
       " 47.478201502112164\n",
       " 47.282197523073876\n",
       " 47.0460939419394  \n",
       " 46.77390071891838 \n",
       " 46.47390071891839 \n",
       " 46.17390071891838 \n",
       " 45.87390071891838 \n",
       " 45.573900718918374\n",
       " 45.27390071891837 \n",
       " 44.97390071891837 "
      ]
     },
     "execution_count": 5,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Markov transition matrix of a *deterministic* policy pi\n",
    "\n",
    "function policyTransitionMatrix(pi) # warning: requires pi to be *deterministic*\n",
    "    Kpi = zeros(1+M, 1+M)\n",
    "    for s in 0:M\n",
    "        Kpi[1+s,:] = K[s, pi(s)]\n",
    "    end\n",
    "    return(Kpi)\n",
    "end\n",
    "\n",
    "function policyAvgReward(pi)\n",
    "    avgRpi = zeros(1+M)\n",
    "    for s in 0:M\n",
    "        avgRpi[1+s] = avgR[s, pi(s)]\n",
    "    end\n",
    "    return(avgRpi)\n",
    "end\n",
    "  \n",
    "# example: for the constant policy\n",
    "# Kpi = policyTransitionMatrix(constantPolicy)\n",
    "# avgRpi = policyAvgReward(constantPolicy)\n",
    "\n",
    "# policy evaluation\n",
    "\n",
    "function evaluatePolicy(pi)\n",
    "    Kpi = policyTransitionMatrix(pi)\n",
    "    avgRpi = policyAvgReward(pi)\n",
    "    return( (Matrix{Float64}(I, M+1, M+1) - gamma * Kpi) \\ avgRpi )\n",
    "end\n",
    "\n",
    "evaluatePolicy(constantPolicy)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Planning"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[118.22 118.42 118.62 118.82 119.122 119.523 119.89 120.223 120.522 120.788 121.02 121.219 121.384 121.516 121.614 121.679]"
     ]
    },
    {
     "data": {
      "text/plain": [
       "1×16 Adjoint{Int64,Array{Int64,1}}:\n",
       " 10  9  8  7  0  0  0  0  0  0  0  0  0  0  0  0"
      ]
     },
     "execution_count": 23,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Bellmann Operator\n",
    "function BellmannOperator(V) # TODO maybe: merge BellmannOperator and greedyPolicy by having BellmannOperator return the associated policy\n",
    "    nV = copy(V)    \n",
    "    for s in 0:M\n",
    "        nV[1+s] = maximum([avgR[s, a] + gamma * dot(K[s,a], V) for a in 0:M]) \n",
    "    end    \n",
    "    return(nV)\n",
    "end\n",
    "\n",
    "# Value Iteration\n",
    "function valueIteration(epsilon=1e-6) # epsilon = guaranteed precision\n",
    "    V = zeros(1+M)  # in theory, can be anything\n",
    "    oV = [v + 2*epsilon for v in V]\n",
    "    while norm(V-oV, Inf) > epsilon*(1-gamma)/gamma\n",
    "        oV = copy(V)\n",
    "        V = BellmannOperator(V)\n",
    "    end\n",
    "    return(V)\n",
    "end\n",
    "\n",
    "function greedyPolicy(V)\n",
    "    return([ findmax([avgR[s, a] + gamma * dot(K[s,a], V) for a in 0:M])[2]-1 for s in 0:M])\n",
    "end\n",
    "\n",
    "optimalValue = valueIteration()\n",
    "print(optimalValue')\n",
    "piOpt = greedyPolicy(optimalValue)'"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "1×16 Adjoint{Int64,Array{Int64,1}}:\n",
       " 10  9  8  7  0  0  0  0  0  0  0  0  0  0  0  0"
      ]
     },
     "execution_count": 9,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Policy Iteration\n",
    "function policyIteration()\n",
    "    pi = rand(0:M, 1+M)\n",
    "    opi = -ones(1+M)\n",
    "    while any(pi!=opi)\n",
    "        opi = copy(pi)\n",
    "        V = evaluatePolicy(x->pi[1+x])\n",
    "        pi = greedyPolicy(V)\n",
    "    end\n",
    "    return(pi)\n",
    "end\n",
    "\n",
    "policyIteration()'"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Kpi = zeros(1+M, 1+M)\n",
    "Kpi[1+0, :] = K[(0, 0)]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "┌ Warning: Problem status UnknownError; solution may be inaccurate.\n",
      "└ @ Convex /home/agarivie/.julia/packages/Convex/nR5LG/src/solution.jl:51\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "1×16 Adjoint{Int64,Array{Int64,1}}:\n",
       " 10  9  8  7  0  0  0  0  0  0  0  0  0  0  0  0"
      ]
     },
     "execution_count": 10,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Linear Programming\n",
    "# WARNING: packages JuMP and Convex are incompatible! only one of the two should be uncommented\n",
    "\n",
    "### Using package JuMP\n",
    "# using JuMP\n",
    "# using Clp\n",
    "# model = Model(solver = ClpSolver())\n",
    "# @variable(model, V[s=0:M])\n",
    "# @objective(model, Min, sum(V))\n",
    "# #@constraint(model, con00, V[0] >= avgR[0, 0] + gamma * sum(K[0,0][1+k] * V[k] for k in 0:M) )\n",
    "# @constraint(model, con[s=0:M, a=0:M], V[s] >= avgR[s, a] + gamma * sum(K[s, a][1+k] * V[k] for k in 0:M) )\n",
    "# status = solve(model)\n",
    "# piOpt = greedyPolicy(getvalue(V)[:])'\n",
    "\n",
    "## Using Package Convex\n",
    "using Convex\n",
    "using SCS\n",
    "solver = SCSSolver(verbose=0)\n",
    "#set_default_solver(solver);\n",
    "\n",
    "V = Variable(M+1)\n",
    "convexPb = minimize(sum(V))\n",
    "convexPb.constraints +=  [ V[1+s] >= avgR[s, a] + gamma * sum(K[s, a][1+k] * V[1+k] for k in 0:M) for s in 0:M, a in 0:M]\n",
    "solve!(convexPb, solver)\n",
    "piOpt = greedyPolicy(V.value)'"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 36,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/svg+xml": [
       "<?xml version=\"1.0\" encoding=\"utf-8\"?>\n",
       "<svg xmlns=\"http://www.w3.org/2000/svg\" xmlns:xlink=\"http://www.w3.org/1999/xlink\" width=\"600\" height=\"400\" viewBox=\"0 0 2400 1600\">\n",
       "<defs>\n",
       "  <clipPath id=\"clip3000\">\n",
       "    <rect x=\"0\" y=\"0\" width=\"2400\" height=\"1600\"/>\n",
       "  </clipPath>\n",
       "</defs>\n",
       "<polygon clip-path=\"url(#clip3000)\" points=\"\n",
       "0,1600 2400,1600 2400,0 0,0 \n",
       "  \" fill=\"#ffffff\" fill-rule=\"evenodd\" fill-opacity=\"1\"/>\n",
       "<defs>\n",
       "  <clipPath id=\"clip3001\">\n",
       "    <rect x=\"480\" y=\"0\" width=\"1681\" height=\"1600\"/>\n",
       "  </clipPath>\n",
       "</defs>\n",
       "<polygon clip-path=\"url(#clip3000)\" points=\"\n",
       "229.135,1425.62 2352.76,1425.62 2352.76,47.2441 229.135,47.2441 \n",
       "  \" fill=\"#ffffff\" fill-rule=\"evenodd\" fill-opacity=\"1\"/>\n",
       "<defs>\n",
       "  <clipPath id=\"clip3002\">\n",
       "    <rect x=\"229\" y=\"47\" width=\"2125\" height=\"1379\"/>\n",
       "  </clipPath>\n",
       "</defs>\n",
       "<polyline clip-path=\"url(#clip3002)\" style=\"stroke:#000000; stroke-width:2; stroke-opacity:0.1; fill:none\" points=\"\n",
       "  289.238,1425.62 289.238,47.2441 \n",
       "  \"/>\n",
       "<polyline clip-path=\"url(#clip3002)\" style=\"stroke:#000000; stroke-width:2; stroke-opacity:0.1; fill:none\" points=\"\n",
       "  957.043,1425.62 957.043,47.2441 \n",
       "  \"/>\n",
       "<polyline clip-path=\"url(#clip3002)\" style=\"stroke:#000000; stroke-width:2; stroke-opacity:0.1; fill:none\" points=\"\n",
       "  1624.85,1425.62 1624.85,47.2441 \n",
       "  \"/>\n",
       "<polyline clip-path=\"url(#clip3002)\" style=\"stroke:#000000; stroke-width:2; stroke-opacity:0.1; fill:none\" points=\"\n",
       "  2292.65,1425.62 2292.65,47.2441 \n",
       "  \"/>\n",
       "<polyline clip-path=\"url(#clip3002)\" style=\"stroke:#000000; stroke-width:2; stroke-opacity:0.1; fill:none\" points=\"\n",
       "  229.135,1178.02 2352.76,1178.02 \n",
       "  \"/>\n",
       "<polyline clip-path=\"url(#clip3002)\" style=\"stroke:#000000; stroke-width:2; stroke-opacity:0.1; fill:none\" points=\"\n",
       "  229.135,811.566 2352.76,811.566 \n",
       "  \"/>\n",
       "<polyline clip-path=\"url(#clip3002)\" style=\"stroke:#000000; stroke-width:2; stroke-opacity:0.1; fill:none\" points=\"\n",
       "  229.135,445.115 2352.76,445.115 \n",
       "  \"/>\n",
       "<polyline clip-path=\"url(#clip3002)\" style=\"stroke:#000000; stroke-width:2; stroke-opacity:0.1; fill:none\" points=\"\n",
       "  229.135,78.6633 2352.76,78.6633 \n",
       "  \"/>\n",
       "<polyline clip-path=\"url(#clip3000)\" style=\"stroke:#000000; stroke-width:4; stroke-opacity:1; fill:none\" points=\"\n",
       "  229.135,1425.62 2352.76,1425.62 \n",
       "  \"/>\n",
       "<polyline clip-path=\"url(#clip3000)\" style=\"stroke:#000000; stroke-width:4; stroke-opacity:1; fill:none\" points=\"\n",
       "  229.135,1425.62 229.135,47.2441 \n",
       "  \"/>\n",
       "<polyline clip-path=\"url(#clip3000)\" style=\"stroke:#000000; stroke-width:4; stroke-opacity:1; fill:none\" points=\"\n",
       "  289.238,1425.62 289.238,1404.94 \n",
       "  \"/>\n",
       "<polyline clip-path=\"url(#clip3000)\" style=\"stroke:#000000; stroke-width:4; stroke-opacity:1; fill:none\" points=\"\n",
       "  957.043,1425.62 957.043,1404.94 \n",
       "  \"/>\n",
       "<polyline clip-path=\"url(#clip3000)\" style=\"stroke:#000000; stroke-width:4; stroke-opacity:1; fill:none\" points=\"\n",
       "  1624.85,1425.62 1624.85,1404.94 \n",
       "  \"/>\n",
       "<polyline clip-path=\"url(#clip3000)\" style=\"stroke:#000000; stroke-width:4; stroke-opacity:1; fill:none\" points=\"\n",
       "  2292.65,1425.62 2292.65,1404.94 \n",
       "  \"/>\n",
       "<polyline clip-path=\"url(#clip3000)\" style=\"stroke:#000000; stroke-width:4; stroke-opacity:1; fill:none\" points=\"\n",
       "  229.135,1178.02 260.99,1178.02 \n",
       "  \"/>\n",
       "<polyline clip-path=\"url(#clip3000)\" style=\"stroke:#000000; stroke-width:4; stroke-opacity:1; fill:none\" points=\"\n",
       "  229.135,811.566 260.99,811.566 \n",
       "  \"/>\n",
       "<polyline clip-path=\"url(#clip3000)\" style=\"stroke:#000000; stroke-width:4; stroke-opacity:1; fill:none\" points=\"\n",
       "  229.135,445.115 260.99,445.115 \n",
       "  \"/>\n",
       "<polyline clip-path=\"url(#clip3000)\" style=\"stroke:#000000; stroke-width:4; stroke-opacity:1; fill:none\" points=\"\n",
       "  229.135,78.6633 260.99,78.6633 \n",
       "  \"/>\n",
       "<g clip-path=\"url(#clip3000)\">\n",
       "<text style=\"fill:#000000; fill-opacity:1; font-family:Arial,Helvetica Neue,Helvetica,sans-serif; font-size:48px; text-anchor:middle;\" transform=\"rotate(0, 289.238, 1479.62)\" x=\"289.238\" y=\"1479.62\">0</text>\n",
       "</g>\n",
       "<g clip-path=\"url(#clip3000)\">\n",
       "<text style=\"fill:#000000; fill-opacity:1; font-family:Arial,Helvetica Neue,Helvetica,sans-serif; font-size:48px; text-anchor:middle;\" transform=\"rotate(0, 957.043, 1479.62)\" x=\"957.043\" y=\"1479.62\">5</text>\n",
       "</g>\n",
       "<g clip-path=\"url(#clip3000)\">\n",
       "<text style=\"fill:#000000; fill-opacity:1; font-family:Arial,Helvetica Neue,Helvetica,sans-serif; font-size:48px; text-anchor:middle;\" transform=\"rotate(0, 1624.85, 1479.62)\" x=\"1624.85\" y=\"1479.62\">10</text>\n",
       "</g>\n",
       "<g clip-path=\"url(#clip3000)\">\n",
       "<text style=\"fill:#000000; fill-opacity:1; font-family:Arial,Helvetica Neue,Helvetica,sans-serif; font-size:48px; text-anchor:middle;\" transform=\"rotate(0, 2292.65, 1479.62)\" x=\"2292.65\" y=\"1479.62\">15</text>\n",
       "</g>\n",
       "<g clip-path=\"url(#clip3000)\">\n",
       "<text style=\"fill:#000000; fill-opacity:1; font-family:Arial,Helvetica Neue,Helvetica,sans-serif; font-size:48px; text-anchor:end;\" transform=\"rotate(0, 205.135, 1195.52)\" x=\"205.135\" y=\"1195.52\">114</text>\n",
       "</g>\n",
       "<g clip-path=\"url(#clip3000)\">\n",
       "<text style=\"fill:#000000; fill-opacity:1; font-family:Arial,Helvetica Neue,Helvetica,sans-serif; font-size:48px; text-anchor:end;\" transform=\"rotate(0, 205.135, 829.066)\" x=\"205.135\" y=\"829.066\">116</text>\n",
       "</g>\n",
       "<g clip-path=\"url(#clip3000)\">\n",
       "<text style=\"fill:#000000; fill-opacity:1; font-family:Arial,Helvetica Neue,Helvetica,sans-serif; font-size:48px; text-anchor:end;\" transform=\"rotate(0, 205.135, 462.615)\" x=\"205.135\" y=\"462.615\">118</text>\n",
       "</g>\n",
       "<g clip-path=\"url(#clip3000)\">\n",
       "<text style=\"fill:#000000; fill-opacity:1; font-family:Arial,Helvetica Neue,Helvetica,sans-serif; font-size:48px; text-anchor:end;\" transform=\"rotate(0, 205.135, 96.1633)\" x=\"205.135\" y=\"96.1633\">120</text>\n",
       "</g>\n",
       "<g clip-path=\"url(#clip3000)\">\n",
       "<text style=\"fill:#000000; fill-opacity:1; font-family:Arial,Helvetica Neue,Helvetica,sans-serif; font-size:66px; text-anchor:middle;\" transform=\"rotate(0, 1290.95, 1559.48)\" x=\"1290.95\" y=\"1559.48\">action</text>\n",
       "</g>\n",
       "<g clip-path=\"url(#clip3000)\">\n",
       "<text style=\"fill:#000000; fill-opacity:1; font-family:Arial,Helvetica Neue,Helvetica,sans-serif; font-size:66px; text-anchor:middle;\" transform=\"rotate(-90, 89.2861, 736.431)\" x=\"89.2861\" y=\"736.431\">Q(s,a)</text>\n",
       "</g>\n",
       "<polyline clip-path=\"url(#clip3002)\" style=\"stroke:#009af9; stroke-width:12; stroke-opacity:1; fill:none\" points=\"\n",
       "  289.238,772.645 422.799,855.175 556.36,803.892 689.921,760.121 823.482,722.866 957.043,691.749 1090.6,666.891 1224.17,648.303 1357.73,635.539 1491.29,629.844 \n",
       "  1624.85,630.082 1758.41,635.932 1891.97,647.845 2025.53,666.107 2159.09,690.351 2292.65,781.963 \n",
       "  \"/>\n",
       "<polyline clip-path=\"url(#clip3002)\" style=\"stroke:#e26f46; stroke-width:12; stroke-opacity:1; fill:none\" points=\"\n",
       "  289.238,503.605 422.799,612.93 556.36,581.813 689.921,556.955 823.482,538.368 957.043,525.604 1090.6,519.909 1224.17,520.146 1357.73,525.996 1491.29,537.909 \n",
       "  1624.85,556.172 1758.41,580.415 1891.97,672.028 2025.53,763.641 2159.09,855.254 2292.65,946.866 \n",
       "  \"/>\n",
       "<polyline clip-path=\"url(#clip3002)\" style=\"stroke:#3da44d; stroke-width:12; stroke-opacity:1; fill:none\" points=\"\n",
       "  289.238,86.2547 422.799,244.748 556.36,263.011 689.921,287.254 823.482,378.867 957.043,470.48 1090.6,562.093 1224.17,653.705 1357.73,745.318 1491.29,836.931 \n",
       "  1624.85,928.544 1758.41,1020.16 1891.97,1111.77 2025.53,1203.38 2159.09,1295 2292.65,1386.61 \n",
       "  \"/>\n",
       "<polygon clip-path=\"url(#clip3000)\" points=\"\n",
       "1935.05,372.684 2280.76,372.684 2280.76,130.764 1935.05,130.764 \n",
       "  \" fill=\"#ffffff\" fill-rule=\"evenodd\" fill-opacity=\"1\"/>\n",
       "<polyline clip-path=\"url(#clip3000)\" style=\"stroke:#000000; stroke-width:4; stroke-opacity:1; fill:none\" points=\"\n",
       "  1935.05,372.684 2280.76,372.684 2280.76,130.764 1935.05,130.764 1935.05,372.684 \n",
       "  \"/>\n",
       "<polyline clip-path=\"url(#clip3000)\" style=\"stroke:#009af9; stroke-width:12; stroke-opacity:1; fill:none\" points=\"\n",
       "  1959.05,191.244 2103.05,191.244 \n",
       "  \"/>\n",
       "<g clip-path=\"url(#clip3000)\">\n",
       "<text style=\"fill:#000000; fill-opacity:1; font-family:Arial,Helvetica Neue,Helvetica,sans-serif; font-size:48px; text-anchor:start;\" transform=\"rotate(0, 2127.05, 208.744)\" x=\"2127.05\" y=\"208.744\">s=2</text>\n",
       "</g>\n",
       "<polyline clip-path=\"url(#clip3000)\" style=\"stroke:#e26f46; stroke-width:12; stroke-opacity:1; fill:none\" points=\"\n",
       "  1959.05,251.724 2103.05,251.724 \n",
       "  \"/>\n",
       "<g clip-path=\"url(#clip3000)\">\n",
       "<text style=\"fill:#000000; fill-opacity:1; font-family:Arial,Helvetica Neue,Helvetica,sans-serif; font-size:48px; text-anchor:start;\" transform=\"rotate(0, 2127.05, 269.224)\" x=\"2127.05\" y=\"269.224\">s=5</text>\n",
       "</g>\n",
       "<polyline clip-path=\"url(#clip3000)\" style=\"stroke:#3da44d; stroke-width:12; stroke-opacity:1; fill:none\" points=\"\n",
       "  1959.05,312.204 2103.05,312.204 \n",
       "  \"/>\n",
       "<g clip-path=\"url(#clip3000)\">\n",
       "<text style=\"fill:#000000; fill-opacity:1; font-family:Arial,Helvetica Neue,Helvetica,sans-serif; font-size:48px; text-anchor:start;\" transform=\"rotate(0, 2127.05, 329.704)\" x=\"2127.05\" y=\"329.704\">s=13</text>\n",
       "</g>\n",
       "</svg>\n"
      ]
     },
     "execution_count": 36,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# computing the optimal Q-table\n",
    "optimalQ = zeros(M+1, M+1)\n",
    "for s in 0:M\n",
    "    for a in 0:M\n",
    "        optimalQ[1+s, 1+a]= avgR[s, a] + gamma * sum([K[s, a][1+ns] * V.value[1+ns] for ns in 0:M ])\n",
    "    end\n",
    "end\n",
    "\n",
    "lS = [2, Int64(floor(M/3)), M-2]\n",
    "plot(0:M, optimalQ[lS, :]',  xlabel = \"action\", ylabel = \"Q(s,a)\", label=[ string(\"s=\", x) for x in lS], lw=3)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Stochastic Approximation for Policy Evaluation: TD(0) and TD($\\lambda$)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "16×2 Adjoint{Float64,Array{Float64,2}}:\n",
       " 47.3413  47.5382\n",
       " 47.511   47.6853\n",
       " 47.6031  47.7643\n",
       " 47.4968  47.7785\n",
       " 47.489   47.7316\n",
       " 47.4259  47.6297\n",
       " 47.0499  47.4782\n",
       " 47.0806  47.2822\n",
       " 46.5944  47.0461\n",
       " 46.4258  46.7739\n",
       " 46.1666  46.4739\n",
       " 45.9161  46.1739\n",
       " 45.5645  45.8739\n",
       " 45.4735  45.5739\n",
       " 44.9925  45.2739\n",
       " 45.1997  44.9739"
      ]
     },
     "execution_count": 13,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "function TD0(pi, T)\n",
    "    V = zeros(1+M) # V[S] = estimated value of each state under policy pi\n",
    "    N = zeros(1+M) # N[S] =number of visits to state s in the loop\n",
    "    S = s0\n",
    "    for t in 1:T\n",
    "        N[1+S] += 1\n",
    "        A = pi(S)\n",
    "        D = rdemand()\n",
    "        X = reward(S, A, D)\n",
    "        nS = nextState(S, A, D) \n",
    "        alpha = 1/(1+N[1+S])^0.5 \n",
    "        V[1+S] = (1-alpha)*V[1+S] + alpha * (X + gamma*V[1+nS])\n",
    "        S = nS\n",
    "    end\n",
    "    return(V)\n",
    "end\n",
    "\n",
    "T = 10^7\n",
    "[TD0(constantPolicy, T)'; evaluatePolicy(constantPolicy)']'\n",
    "# requires a huge number of iterations to converge... at least 10^7\n",
    "# because the initialization is too far from the solution: it takes forever to raise from 0 to 47"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Model-free Learning: Q-learning"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 245,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "10%\t20%\t30%\t40%\t50%\t60%\t70%\t80%\t90%\t100%\t\n",
      "[117.996, 118.13, 118.552, 118.673, 118.926, 119.383, 119.784, 120.051, 120.336, 120.495, 120.788, 120.985, 121.182, 121.325, 121.242, 121.404]\n",
      "[11 11 8 5 0 0 0 0 0 0 0 0 0 0 0 0]\n",
      "[117.488 117.654 117.888 117.989 118.386 118.787 119.155 119.489 119.789 120.055 120.288 120.488 120.654 120.786 120.885 120.951]\n",
      "[118.22 118.42 118.62 118.82 119.122 119.523 119.89 120.223 120.522 120.788 121.02 121.219 121.384 121.516 121.614 121.679]\n"
     ]
    }
   ],
   "source": [
    "function Qlearning(T)\n",
    "    # Q = zeros(1+M, 1+M) # Q-table\n",
    "    Q = 118*ones(1+M, 1+M) # Q-table cheating with initialization\n",
    "    N = zeros(1+M) # number of visits to state s # rather take state-value?\n",
    "    epsilon = 0.3\n",
    "    S = s0\n",
    "    for t in 1:T\n",
    "        if mod(t, floor(T/10))==0 print(Int64(100*t/T), \"%\\t\"); if t==T println() end; flush(stdout) end       \n",
    "        N[1+S] += 1\n",
    "        if rand()<epsilon# /t^(1/4) # epsilon-greedy choice of action\n",
    "            A = rand(0:M)\n",
    "        else\n",
    "            A = findmax(Q[1+S, :])[2]-1 # greedy action            \n",
    "        end\n",
    "        D = rdemand()\n",
    "        X = reward(S, A, D)\n",
    "        nS = nextState(S, A, D)\n",
    "        delta = X + gamma * maximum(Q[1+nS, :]) - Q[1+S, 1+A]\n",
    "        alpha = 1/(1+N[1+S])^0.5 # alpha = 1/t^0.3 # warning: alpha should not be taken for an integer...\n",
    "        Q[1+S, 1+A] += alpha * delta\n",
    "        S = nS        \n",
    "    end;  \n",
    "    pi = [findmax(Q[1+s,:])[2]-1  for s in 0:M] \n",
    "    println( [Q[1+s, 1+pi[1+s]] for s in 0:M] )\n",
    "    return(pi)\n",
    "end\n",
    "\n",
    "T = 10^7\n",
    "pi = Qlearning(T)\n",
    "println(pi')\n",
    "println(evaluatePolicy(s->pi[1+s])')\n",
    "println(optimalValue')\n",
    "\n",
    "\n",
    "# for T >= 10^7, we obtain correct policies\n",
    "# for T >= 10^8, we almost obtain the right one"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 333,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "fittedQ (generic function with 1 method)"
      ]
     },
     "execution_count": 333,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# minimalistic Neural Network\n",
    "widths = [3 20 10 5 1]\n",
    "depth = length(widths)\n",
    "activation = [zeros(widths[k]) for k in 1:depth]\n",
    "delta = copy(activation)\n",
    "coefficients = [rand(widths[k+1], widths[k]).-1/2 for k in 1:(depth-1)]\n",
    "\n",
    "# sigma(x) = [1/(1+exp(-u)) for u in x]; dsigma(x) = [exp(-u)/(1+exp(-u))^2 for u in x]\n",
    "sigma(x) = [max(0, u) for u in x]; dsigma(x) = [u>0 ? 1.0 : 0.0 for u in x]\n",
    "function forward(x)\n",
    "    activation[1] = x\n",
    "    for k in 1:(depth-2)\n",
    "       activation[k+1] = sigma(coefficients[k] * activation[k])\n",
    "    end\n",
    "    activation[depth] = coefficients[depth-1] * activation[depth - 1]\n",
    "end\n",
    "function update(d, eta, lambda)\n",
    "    delta[depth] = d\n",
    "    for k in (depth-1):-1:1\n",
    "#        println(delta[k+1] * activation[k]')\n",
    "        coefficients[k] -= eta * (delta[k+1] * activation[k]' + lambda * coefficients[k]) #for i in 1:N[k+1] for j in 1:N[k] A[k][i,j] -= eta*delta[k+1][i] * X[k][j] end end\n",
    "        if k>1 delta[k] = dsigma(activation[k]) .* (coefficients[k]'* delta[k+1]) end # useless for k=1\n",
    "    end\n",
    "end\n",
    "\n",
    "function fittedQ(s, a)\n",
    "    forward([s, a, 1.0])\n",
    "    return 117+activation[depth][1]\n",
    "end"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 334,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "100%\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "1×16 Adjoint{Int64,Array{Int64,1}}:\n",
       " 10  8  7  7  6  0  0  0  0  0  0  0  0  0  0  0"
      ]
     },
     "execution_count": 334,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "function QlearningWFA(T) # Q-learning with function approximation\n",
    "    eta = 1e-6\n",
    "    epsilon = 1 # even with epsilon=1 the algorithm diverges\n",
    "    lambda = 1e-2 # regularization for the parameters of the NN\n",
    "    S = s0\n",
    "    for t in 1:T\n",
    "        if mod(t, floor(T/20))==0 print(\"\\b\\b \\b\\b\", Int64(100*t/T), \"%\"); if t==T println() end; flush(stdout); end\n",
    "        if rand()<epsilon # epsilon-greedy choice of action\n",
    "            A = rand(0:M)\n",
    "        else\n",
    "            A = findmax([fittedQ(S, a) for a in 0:M])[2]-1 # greedy action\n",
    "        end\n",
    "        D = rdemand()\n",
    "        X = reward(S, A, D)\n",
    "        nS = nextState(S, A, D)\n",
    "        delta = X + gamma * maximum([fittedQ(nS, a) for a in 0:M]) - fittedQ(S, A)\n",
    "        # delta = optimalQ[1+S, 1+A] - fittedQ(S, A) # cheating a lot\n",
    "        # delta = avgR[S, A] + gamma * maximum(optimalQ[1+nS, :]) - fittedQ(S, A) # cheating a bit less\n",
    "        #delta = X + gamma * maximum(optimalQ[1+nS, :]) - fittedQ(S, A) # cheating even less\n",
    "        update([-delta], eta, lambda) \n",
    "        S = nS        \n",
    "    end\n",
    "    pi = [findmax([fittedQ(s, a) for a in 0:M])[2]-1  for s in 0:M] \n",
    "    return pi\n",
    "end\n",
    "\n",
    "T = 2*10^7\n",
    "QlearningWFA(T)'\n",
    "# it diverges at every run!!! very unstable method\n",
    "# -> add batch updates?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 335,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Tuple{Float64,Float64}[(115.588, 113.671), (115.212, 113.289), (115.562, 113.341), (115.842, 113.572), (116.081, 113.813), (116.284, 114.054), (116.454, 114.274), (116.59, 114.36), (116.691, 114.434), (116.761, 114.508), (116.792, 114.535), (116.79, 114.407), (116.759, 114.279), (116.694, 114.151), (116.594, 114.015), (116.462, 113.877)]\n"
     ]
    },
    {
     "data": {
      "image/svg+xml": [
       "<?xml version=\"1.0\" encoding=\"utf-8\"?>\n",
       "<svg xmlns=\"http://www.w3.org/2000/svg\" xmlns:xlink=\"http://www.w3.org/1999/xlink\" width=\"600\" height=\"400\" viewBox=\"0 0 2400 1600\">\n",
       "<defs>\n",
       "  <clipPath id=\"clip4600\">\n",
       "    <rect x=\"0\" y=\"0\" width=\"2400\" height=\"1600\"/>\n",
       "  </clipPath>\n",
       "</defs>\n",
       "<polygon clip-path=\"url(#clip4600)\" points=\"\n",
       "0,1600 2400,1600 2400,0 0,0 \n",
       "  \" fill=\"#ffffff\" fill-rule=\"evenodd\" fill-opacity=\"1\"/>\n",
       "<defs>\n",
       "  <clipPath id=\"clip4601\">\n",
       "    <rect x=\"480\" y=\"0\" width=\"1681\" height=\"1600\"/>\n",
       "  </clipPath>\n",
       "</defs>\n",
       "<polygon clip-path=\"url(#clip4600)\" points=\"\n",
       "269.279,1425.62 2352.76,1425.62 2352.76,47.2441 269.279,47.2441 \n",
       "  \" fill=\"#ffffff\" fill-rule=\"evenodd\" fill-opacity=\"1\"/>\n",
       "<defs>\n",
       "  <clipPath id=\"clip4602\">\n",
       "    <rect x=\"269\" y=\"47\" width=\"2084\" height=\"1379\"/>\n",
       "  </clipPath>\n",
       "</defs>\n",
       "<polyline clip-path=\"url(#clip4602)\" style=\"stroke:#000000; stroke-width:2; stroke-opacity:0.1; fill:none\" points=\"\n",
       "  328.245,1425.62 328.245,47.2441 \n",
       "  \"/>\n",
       "<polyline clip-path=\"url(#clip4602)\" style=\"stroke:#000000; stroke-width:2; stroke-opacity:0.1; fill:none\" points=\"\n",
       "  983.427,1425.62 983.427,47.2441 \n",
       "  \"/>\n",
       "<polyline clip-path=\"url(#clip4602)\" style=\"stroke:#000000; stroke-width:2; stroke-opacity:0.1; fill:none\" points=\"\n",
       "  1638.61,1425.62 1638.61,47.2441 \n",
       "  \"/>\n",
       "<polyline clip-path=\"url(#clip4602)\" style=\"stroke:#000000; stroke-width:2; stroke-opacity:0.1; fill:none\" points=\"\n",
       "  2293.79,1425.62 2293.79,47.2441 \n",
       "  \"/>\n",
       "<polyline clip-path=\"url(#clip4602)\" style=\"stroke:#000000; stroke-width:2; stroke-opacity:0.1; fill:none\" points=\"\n",
       "  269.279,1308.2 2352.76,1308.2 \n",
       "  \"/>\n",
       "<polyline clip-path=\"url(#clip4602)\" style=\"stroke:#000000; stroke-width:2; stroke-opacity:0.1; fill:none\" points=\"\n",
       "  269.279,1122.6 2352.76,1122.6 \n",
       "  \"/>\n",
       "<polyline clip-path=\"url(#clip4602)\" style=\"stroke:#000000; stroke-width:2; stroke-opacity:0.1; fill:none\" points=\"\n",
       "  269.279,936.993 2352.76,936.993 \n",
       "  \"/>\n",
       "<polyline clip-path=\"url(#clip4602)\" style=\"stroke:#000000; stroke-width:2; stroke-opacity:0.1; fill:none\" points=\"\n",
       "  269.279,751.388 2352.76,751.388 \n",
       "  \"/>\n",
       "<polyline clip-path=\"url(#clip4602)\" style=\"stroke:#000000; stroke-width:2; stroke-opacity:0.1; fill:none\" points=\"\n",
       "  269.279,565.782 2352.76,565.782 \n",
       "  \"/>\n",
       "<polyline clip-path=\"url(#clip4602)\" style=\"stroke:#000000; stroke-width:2; stroke-opacity:0.1; fill:none\" points=\"\n",
       "  269.279,380.176 2352.76,380.176 \n",
       "  \"/>\n",
       "<polyline clip-path=\"url(#clip4602)\" style=\"stroke:#000000; stroke-width:2; stroke-opacity:0.1; fill:none\" points=\"\n",
       "  269.279,194.571 2352.76,194.571 \n",
       "  \"/>\n",
       "<polyline clip-path=\"url(#clip4600)\" style=\"stroke:#000000; stroke-width:4; stroke-opacity:1; fill:none\" points=\"\n",
       "  269.279,1425.62 2352.76,1425.62 \n",
       "  \"/>\n",
       "<polyline clip-path=\"url(#clip4600)\" style=\"stroke:#000000; stroke-width:4; stroke-opacity:1; fill:none\" points=\"\n",
       "  269.279,1425.62 269.279,47.2441 \n",
       "  \"/>\n",
       "<polyline clip-path=\"url(#clip4600)\" style=\"stroke:#000000; stroke-width:4; stroke-opacity:1; fill:none\" points=\"\n",
       "  328.245,1425.62 328.245,1404.94 \n",
       "  \"/>\n",
       "<polyline clip-path=\"url(#clip4600)\" style=\"stroke:#000000; stroke-width:4; stroke-opacity:1; fill:none\" points=\"\n",
       "  983.427,1425.62 983.427,1404.94 \n",
       "  \"/>\n",
       "<polyline clip-path=\"url(#clip4600)\" style=\"stroke:#000000; stroke-width:4; stroke-opacity:1; fill:none\" points=\"\n",
       "  1638.61,1425.62 1638.61,1404.94 \n",
       "  \"/>\n",
       "<polyline clip-path=\"url(#clip4600)\" style=\"stroke:#000000; stroke-width:4; stroke-opacity:1; fill:none\" points=\"\n",
       "  2293.79,1425.62 2293.79,1404.94 \n",
       "  \"/>\n",
       "<polyline clip-path=\"url(#clip4600)\" style=\"stroke:#000000; stroke-width:4; stroke-opacity:1; fill:none\" points=\"\n",
       "  269.279,1308.2 300.531,1308.2 \n",
       "  \"/>\n",
       "<polyline clip-path=\"url(#clip4600)\" style=\"stroke:#000000; stroke-width:4; stroke-opacity:1; fill:none\" points=\"\n",
       "  269.279,1122.6 300.531,1122.6 \n",
       "  \"/>\n",
       "<polyline clip-path=\"url(#clip4600)\" style=\"stroke:#000000; stroke-width:4; stroke-opacity:1; fill:none\" points=\"\n",
       "  269.279,936.993 300.531,936.993 \n",
       "  \"/>\n",
       "<polyline clip-path=\"url(#clip4600)\" style=\"stroke:#000000; stroke-width:4; stroke-opacity:1; fill:none\" points=\"\n",
       "  269.279,751.388 300.531,751.388 \n",
       "  \"/>\n",
       "<polyline clip-path=\"url(#clip4600)\" style=\"stroke:#000000; stroke-width:4; stroke-opacity:1; fill:none\" points=\"\n",
       "  269.279,565.782 300.531,565.782 \n",
       "  \"/>\n",
       "<polyline clip-path=\"url(#clip4600)\" style=\"stroke:#000000; stroke-width:4; stroke-opacity:1; fill:none\" points=\"\n",
       "  269.279,380.176 300.531,380.176 \n",
       "  \"/>\n",
       "<polyline clip-path=\"url(#clip4600)\" style=\"stroke:#000000; stroke-width:4; stroke-opacity:1; fill:none\" points=\"\n",
       "  269.279,194.571 300.531,194.571 \n",
       "  \"/>\n",
       "<g clip-path=\"url(#clip4600)\">\n",
       "<text style=\"fill:#000000; fill-opacity:1; font-family:Arial,Helvetica Neue,Helvetica,sans-serif; font-size:48px; text-anchor:middle;\" transform=\"rotate(0, 328.245, 1479.62)\" x=\"328.245\" y=\"1479.62\">0</text>\n",
       "</g>\n",
       "<g clip-path=\"url(#clip4600)\">\n",
       "<text style=\"fill:#000000; fill-opacity:1; font-family:Arial,Helvetica Neue,Helvetica,sans-serif; font-size:48px; text-anchor:middle;\" transform=\"rotate(0, 983.427, 1479.62)\" x=\"983.427\" y=\"1479.62\">5</text>\n",
       "</g>\n",
       "<g clip-path=\"url(#clip4600)\">\n",
       "<text style=\"fill:#000000; fill-opacity:1; font-family:Arial,Helvetica Neue,Helvetica,sans-serif; font-size:48px; text-anchor:middle;\" transform=\"rotate(0, 1638.61, 1479.62)\" x=\"1638.61\" y=\"1479.62\">10</text>\n",
       "</g>\n",
       "<g clip-path=\"url(#clip4600)\">\n",
       "<text style=\"fill:#000000; fill-opacity:1; font-family:Arial,Helvetica Neue,Helvetica,sans-serif; font-size:48px; text-anchor:middle;\" transform=\"rotate(0, 2293.79, 1479.62)\" x=\"2293.79\" y=\"1479.62\">15</text>\n",
       "</g>\n",
       "<g clip-path=\"url(#clip4600)\">\n",
       "<text style=\"fill:#000000; fill-opacity:1; font-family:Arial,Helvetica Neue,Helvetica,sans-serif; font-size:48px; text-anchor:end;\" transform=\"rotate(0, 245.279, 1325.7)\" x=\"245.279\" y=\"1325.7\">113.5</text>\n",
       "</g>\n",
       "<g clip-path=\"url(#clip4600)\">\n",
       "<text style=\"fill:#000000; fill-opacity:1; font-family:Arial,Helvetica Neue,Helvetica,sans-serif; font-size:48px; text-anchor:end;\" transform=\"rotate(0, 245.279, 1140.1)\" x=\"245.279\" y=\"1140.1\">114.0</text>\n",
       "</g>\n",
       "<g clip-path=\"url(#clip4600)\">\n",
       "<text style=\"fill:#000000; fill-opacity:1; font-family:Arial,Helvetica Neue,Helvetica,sans-serif; font-size:48px; text-anchor:end;\" transform=\"rotate(0, 245.279, 954.493)\" x=\"245.279\" y=\"954.493\">114.5</text>\n",
       "</g>\n",
       "<g clip-path=\"url(#clip4600)\">\n",
       "<text style=\"fill:#000000; fill-opacity:1; font-family:Arial,Helvetica Neue,Helvetica,sans-serif; font-size:48px; text-anchor:end;\" transform=\"rotate(0, 245.279, 768.888)\" x=\"245.279\" y=\"768.888\">115.0</text>\n",
       "</g>\n",
       "<g clip-path=\"url(#clip4600)\">\n",
       "<text style=\"fill:#000000; fill-opacity:1; font-family:Arial,Helvetica Neue,Helvetica,sans-serif; font-size:48px; text-anchor:end;\" transform=\"rotate(0, 245.279, 583.282)\" x=\"245.279\" y=\"583.282\">115.5</text>\n",
       "</g>\n",
       "<g clip-path=\"url(#clip4600)\">\n",
       "<text style=\"fill:#000000; fill-opacity:1; font-family:Arial,Helvetica Neue,Helvetica,sans-serif; font-size:48px; text-anchor:end;\" transform=\"rotate(0, 245.279, 397.676)\" x=\"245.279\" y=\"397.676\">116.0</text>\n",
       "</g>\n",
       "<g clip-path=\"url(#clip4600)\">\n",
       "<text style=\"fill:#000000; fill-opacity:1; font-family:Arial,Helvetica Neue,Helvetica,sans-serif; font-size:48px; text-anchor:end;\" transform=\"rotate(0, 245.279, 212.071)\" x=\"245.279\" y=\"212.071\">116.5</text>\n",
       "</g>\n",
       "<g clip-path=\"url(#clip4600)\">\n",
       "<text style=\"fill:#000000; fill-opacity:1; font-family:Arial,Helvetica Neue,Helvetica,sans-serif; font-size:66px; text-anchor:middle;\" transform=\"rotate(0, 1311.02, 1559.48)\" x=\"1311.02\" y=\"1559.48\">action</text>\n",
       "</g>\n",
       "<g clip-path=\"url(#clip4600)\">\n",
       "<text style=\"fill:#000000; fill-opacity:1; font-family:Arial,Helvetica Neue,Helvetica,sans-serif; font-size:66px; text-anchor:middle;\" transform=\"rotate(-90, 89.2861, 736.431)\" x=\"89.2861\" y=\"736.431\"> estimated Q(s,a)</text>\n",
       "</g>\n",
       "<polyline clip-path=\"url(#clip4602)\" style=\"stroke:#009af9; stroke-width:12; stroke-opacity:1; fill:none\" points=\"\n",
       "  328.245,533.017 459.281,672.535 590.318,542.771 721.354,438.872 852.39,350.192 983.427,274.714 1114.46,211.671 1245.5,161.309 1376.54,123.651 1507.57,97.792 \n",
       "  1638.61,86.2547 1769.64,86.7354 1900.68,98.5873 2031.72,122.723 2162.75,159.723 2293.79,208.839 \n",
       "  \"/>\n",
       "<polyline clip-path=\"url(#clip4602)\" style=\"stroke:#e26f46; stroke-width:12; stroke-opacity:1; fill:none\" points=\"\n",
       "  328.245,1244.67 459.281,1386.61 590.318,1367.13 721.354,1281.47 852.39,1192.07 983.427,1102.68 1114.46,1020.72 1245.5,988.986 1376.54,961.589 1507.57,934.193 \n",
       "  1638.61,923.94 1769.64,971.431 1900.68,1018.92 2031.72,1066.41 2162.75,1117.06 2293.79,1168.31 \n",
       "  \"/>\n",
       "<polygon clip-path=\"url(#clip4600)\" points=\"\n",
       "1989.93,312.204 2280.76,312.204 2280.76,130.764 1989.93,130.764 \n",
       "  \" fill=\"#ffffff\" fill-rule=\"evenodd\" fill-opacity=\"1\"/>\n",
       "<polyline clip-path=\"url(#clip4600)\" style=\"stroke:#000000; stroke-width:4; stroke-opacity:1; fill:none\" points=\"\n",
       "  1989.93,312.204 2280.76,312.204 2280.76,130.764 1989.93,130.764 1989.93,312.204 \n",
       "  \"/>\n",
       "<polyline clip-path=\"url(#clip4600)\" style=\"stroke:#009af9; stroke-width:12; stroke-opacity:1; fill:none\" points=\"\n",
       "  2013.93,191.244 2157.93,191.244 \n",
       "  \"/>\n",
       "<g clip-path=\"url(#clip4600)\">\n",
       "<text style=\"fill:#000000; fill-opacity:1; font-family:Arial,Helvetica Neue,Helvetica,sans-serif; font-size:48px; text-anchor:start;\" transform=\"rotate(0, 2181.93, 208.744)\" x=\"2181.93\" y=\"208.744\">y1</text>\n",
       "</g>\n",
       "<polyline clip-path=\"url(#clip4600)\" style=\"stroke:#e26f46; stroke-width:12; stroke-opacity:1; fill:none\" points=\"\n",
       "  2013.93,251.724 2157.93,251.724 \n",
       "  \"/>\n",
       "<g clip-path=\"url(#clip4600)\">\n",
       "<text style=\"fill:#000000; fill-opacity:1; font-family:Arial,Helvetica Neue,Helvetica,sans-serif; font-size:48px; text-anchor:start;\" transform=\"rotate(0, 2181.93, 269.224)\" x=\"2181.93\" y=\"269.224\">y2</text>\n",
       "</g>\n",
       "</svg>\n"
      ]
     },
     "execution_count": 335,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "s = 0\n",
    "println([(optimalQ[1+s, 1+a], fittedQ(s, a)) for a in 0:M])\n",
    "\n",
    "myPlot = plot(0:M, optimalQ[1+s, :], xlabel = \"action\", ylabel = \"optQ(s,a)\", lw=3)\n",
    "plot!(myPlot, 0:M, [fittedQ(s, a) for a in 0:M],  xlabel = \"action\", ylabel = \" estimated Q(s,a)\", lw=3)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Additional garbage:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "┌ Warning: Problem status UnknownError; solution may be inaccurate.\n",
      "└ @ Convex /home/agarivie/.julia/packages/Convex/nR5LG/src/solution.jl:51\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "1×16 Adjoint{Int64,Array{Int64,1}}:\n",
       " 10  9  8  7  0  0  0  0  0  0  0  0  0  0  0  0"
      ]
     },
     "execution_count": 11,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "## Using Package Convex\n",
    "using Convex\n",
    "using SCS\n",
    "solver = SCSSolver(verbose=0)\n",
    "#set_default_solver(solver);\n",
    "\n",
    "V = Variable(M+1)\n",
    "convexPb = minimize(sum(V))\n",
    "convexPb.constraints +=  [ V[1+s] >= avgR[s, a] + gamma * sum(K[s, a][1+k] * V[1+k] for k in 0:M) for s in 0:M, a in 0:M]\n",
    "solve!(convexPb, solver)\n",
    "piOpt = greedyPolicy(V.value)'"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 32,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "┌ Warning: Problem status UnknownError; solution may be inaccurate.\n",
      "└ @ Convex /home/agarivie/.julia/packages/Convex/nR5LG/src/solution.jl:51\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Array{Float64,1}[[118.214, 118.418, 118.612, 118.829, 119.121, 119.521, 119.887, 120.223, 120.522, 120.788, 121.02, 121.212, 121.381, 121.515, 121.611, 120.807, 121.612] [118.22, 118.42, 118.62, 118.82, 119.122, 119.523, 119.89, 120.223, 120.522, 120.788, 121.02, 121.219, 121.384, 121.516, 121.614, 121.679]]"
     ]
    }
   ],
   "source": [
    "V = Variable(M+1)\n",
    "convexPb = minimize(V[M])\n",
    "convexPb.constraints +=  [ V[1+s] >= avgR[s, a] + gamma * sum(K[s, a][1+k] * V[1+k] for k in 0:M) for s in 0:M, a in 0:M]\n",
    "solve!(convexPb, solver)\n",
    "piOpt = greedyPolicy(V.value)'\n",
    "print([convexPb.solution.primal',valueIteration()']')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 259,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Dict{Any,Any} with 256 entries:\n",
       "  (0, 0)   => 0.0\n",
       "  (6, 9)   => 0.0469798\n",
       "  (13, 9)  => -2.05302\n",
       "  (9, 1)   => 1.86189\n",
       "  (8, 9)   => -0.55302\n",
       "  (14, 0)  => 2.74109\n",
       "  (0, 7)   => 0.395328\n",
       "  (1, 12)  => -0.387679\n",
       "  (4, 5)   => 1.01322\n",
       "  (4, 0)   => 1.8951\n",
       "  (11, 8)  => -0.95302\n",
       "  (15, 4)  => -0.15302\n",
       "  (4, 9)   => 0.212321\n",
       "  (8, 4)   => 1.25813\n",
       "  (9, 0)   => 2.81322\n",
       "  (15, 0)  => 2.64698\n",
       "  (1, 2)   => 0.339\n",
       "  (14, 2)  => 1.14698\n",
       "  (6, 11)  => -0.95302\n",
       "  (9, 6)   => 0.64698\n",
       "  (8, 2)   => 1.66189\n",
       "  (11, 4)  => 1.04698\n",
       "  (8, 12)  => -2.05302\n",
       "  (2, 5)   => 0.795328\n",
       "  (8, 5)   => 1.01232\n",
       "  ⋮        => ⋮"
      ]
     },
     "execution_count": 259,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "avgR"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 65,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/svg+xml": [
       "<?xml version=\"1.0\" encoding=\"utf-8\"?>\n",
       "<svg xmlns=\"http://www.w3.org/2000/svg\" xmlns:xlink=\"http://www.w3.org/1999/xlink\" width=\"600\" height=\"400\" viewBox=\"0 0 2400 1600\">\n",
       "<defs>\n",
       "  <clipPath id=\"clip6200\">\n",
       "    <rect x=\"0\" y=\"0\" width=\"2400\" height=\"1600\"/>\n",
       "  </clipPath>\n",
       "</defs>\n",
       "<polygon clip-path=\"url(#clip6200)\" points=\"\n",
       "0,1600 2400,1600 2400,0 0,0 \n",
       "  \" fill=\"#ffffff\" fill-rule=\"evenodd\" fill-opacity=\"1\"/>\n",
       "<defs>\n",
       "  <clipPath id=\"clip6201\">\n",
       "    <rect x=\"480\" y=\"0\" width=\"1681\" height=\"1600\"/>\n",
       "  </clipPath>\n",
       "</defs>\n",
       "<polygon clip-path=\"url(#clip6200)\" points=\"\n",
       "234.185,1487.47 2352.76,1487.47 2352.76,47.2441 234.185,47.2441 \n",
       "  \" fill=\"#ffffff\" fill-rule=\"evenodd\" fill-opacity=\"1\"/>\n",
       "<defs>\n",
       "  <clipPath id=\"clip6202\">\n",
       "    <rect x=\"234\" y=\"47\" width=\"2120\" height=\"1441\"/>\n",
       "  </clipPath>\n",
       "</defs>\n",
       "<polyline clip-path=\"url(#clip6202)\" style=\"stroke:#000000; stroke-width:2; stroke-opacity:0.1; fill:none\" points=\"\n",
       "  294.145,1487.47 294.145,47.2441 \n",
       "  \"/>\n",
       "<polyline clip-path=\"url(#clip6202)\" style=\"stroke:#000000; stroke-width:2; stroke-opacity:0.1; fill:none\" points=\"\n",
       "  960.362,1487.47 960.362,47.2441 \n",
       "  \"/>\n",
       "<polyline clip-path=\"url(#clip6202)\" style=\"stroke:#000000; stroke-width:2; stroke-opacity:0.1; fill:none\" points=\"\n",
       "  1626.58,1487.47 1626.58,47.2441 \n",
       "  \"/>\n",
       "<polyline clip-path=\"url(#clip6202)\" style=\"stroke:#000000; stroke-width:2; stroke-opacity:0.1; fill:none\" points=\"\n",
       "  2292.8,1487.47 2292.8,47.2441 \n",
       "  \"/>\n",
       "<polyline clip-path=\"url(#clip6202)\" style=\"stroke:#000000; stroke-width:2; stroke-opacity:0.1; fill:none\" points=\"\n",
       "  234.185,1306.82 2352.76,1306.82 \n",
       "  \"/>\n",
       "<polyline clip-path=\"url(#clip6202)\" style=\"stroke:#000000; stroke-width:2; stroke-opacity:0.1; fill:none\" points=\"\n",
       "  234.185,1121.23 2352.76,1121.23 \n",
       "  \"/>\n",
       "<polyline clip-path=\"url(#clip6202)\" style=\"stroke:#000000; stroke-width:2; stroke-opacity:0.1; fill:none\" points=\"\n",
       "  234.185,935.638 2352.76,935.638 \n",
       "  \"/>\n",
       "<polyline clip-path=\"url(#clip6202)\" style=\"stroke:#000000; stroke-width:2; stroke-opacity:0.1; fill:none\" points=\"\n",
       "  234.185,750.045 2352.76,750.045 \n",
       "  \"/>\n",
       "<polyline clip-path=\"url(#clip6202)\" style=\"stroke:#000000; stroke-width:2; stroke-opacity:0.1; fill:none\" points=\"\n",
       "  234.185,564.452 2352.76,564.452 \n",
       "  \"/>\n",
       "<polyline clip-path=\"url(#clip6202)\" style=\"stroke:#000000; stroke-width:2; stroke-opacity:0.1; fill:none\" points=\"\n",
       "  234.185,378.859 2352.76,378.859 \n",
       "  \"/>\n",
       "<polyline clip-path=\"url(#clip6202)\" style=\"stroke:#000000; stroke-width:2; stroke-opacity:0.1; fill:none\" points=\"\n",
       "  234.185,193.266 2352.76,193.266 \n",
       "  \"/>\n",
       "<polyline clip-path=\"url(#clip6200)\" style=\"stroke:#000000; stroke-width:4; stroke-opacity:1; fill:none\" points=\"\n",
       "  234.185,1487.47 2352.76,1487.47 \n",
       "  \"/>\n",
       "<polyline clip-path=\"url(#clip6200)\" style=\"stroke:#000000; stroke-width:4; stroke-opacity:1; fill:none\" points=\"\n",
       "  234.185,1487.47 234.185,47.2441 \n",
       "  \"/>\n",
       "<polyline clip-path=\"url(#clip6200)\" style=\"stroke:#000000; stroke-width:4; stroke-opacity:1; fill:none\" points=\"\n",
       "  294.145,1487.47 294.145,1465.87 \n",
       "  \"/>\n",
       "<polyline clip-path=\"url(#clip6200)\" style=\"stroke:#000000; stroke-width:4; stroke-opacity:1; fill:none\" points=\"\n",
       "  960.362,1487.47 960.362,1465.87 \n",
       "  \"/>\n",
       "<polyline clip-path=\"url(#clip6200)\" style=\"stroke:#000000; stroke-width:4; stroke-opacity:1; fill:none\" points=\"\n",
       "  1626.58,1487.47 1626.58,1465.87 \n",
       "  \"/>\n",
       "<polyline clip-path=\"url(#clip6200)\" style=\"stroke:#000000; stroke-width:4; stroke-opacity:1; fill:none\" points=\"\n",
       "  2292.8,1487.47 2292.8,1465.87 \n",
       "  \"/>\n",
       "<polyline clip-path=\"url(#clip6200)\" style=\"stroke:#000000; stroke-width:4; stroke-opacity:1; fill:none\" points=\"\n",
       "  234.185,1306.82 265.964,1306.82 \n",
       "  \"/>\n",
       "<polyline clip-path=\"url(#clip6200)\" style=\"stroke:#000000; stroke-width:4; stroke-opacity:1; fill:none\" points=\"\n",
       "  234.185,1121.23 265.964,1121.23 \n",
       "  \"/>\n",
       "<polyline clip-path=\"url(#clip6200)\" style=\"stroke:#000000; stroke-width:4; stroke-opacity:1; fill:none\" points=\"\n",
       "  234.185,935.638 265.964,935.638 \n",
       "  \"/>\n",
       "<polyline clip-path=\"url(#clip6200)\" style=\"stroke:#000000; stroke-width:4; stroke-opacity:1; fill:none\" points=\"\n",
       "  234.185,750.045 265.964,750.045 \n",
       "  \"/>\n",
       "<polyline clip-path=\"url(#clip6200)\" style=\"stroke:#000000; stroke-width:4; stroke-opacity:1; fill:none\" points=\"\n",
       "  234.185,564.452 265.964,564.452 \n",
       "  \"/>\n",
       "<polyline clip-path=\"url(#clip6200)\" style=\"stroke:#000000; stroke-width:4; stroke-opacity:1; fill:none\" points=\"\n",
       "  234.185,378.859 265.964,378.859 \n",
       "  \"/>\n",
       "<polyline clip-path=\"url(#clip6200)\" style=\"stroke:#000000; stroke-width:4; stroke-opacity:1; fill:none\" points=\"\n",
       "  234.185,193.266 265.964,193.266 \n",
       "  \"/>\n",
       "<g clip-path=\"url(#clip6200)\">\n",
       "<text style=\"fill:#000000; fill-opacity:1; font-family:Arial,Helvetica Neue,Helvetica,sans-serif; font-size:48px; text-anchor:middle;\" transform=\"rotate(0, 294.145, 1541.47)\" x=\"294.145\" y=\"1541.47\">0</text>\n",
       "</g>\n",
       "<g clip-path=\"url(#clip6200)\">\n",
       "<text style=\"fill:#000000; fill-opacity:1; font-family:Arial,Helvetica Neue,Helvetica,sans-serif; font-size:48px; text-anchor:middle;\" transform=\"rotate(0, 960.362, 1541.47)\" x=\"960.362\" y=\"1541.47\">5</text>\n",
       "</g>\n",
       "<g clip-path=\"url(#clip6200)\">\n",
       "<text style=\"fill:#000000; fill-opacity:1; font-family:Arial,Helvetica Neue,Helvetica,sans-serif; font-size:48px; text-anchor:middle;\" transform=\"rotate(0, 1626.58, 1541.47)\" x=\"1626.58\" y=\"1541.47\">10</text>\n",
       "</g>\n",
       "<g clip-path=\"url(#clip6200)\">\n",
       "<text style=\"fill:#000000; fill-opacity:1; font-family:Arial,Helvetica Neue,Helvetica,sans-serif; font-size:48px; text-anchor:middle;\" transform=\"rotate(0, 2292.8, 1541.47)\" x=\"2292.8\" y=\"1541.47\">15</text>\n",
       "</g>\n",
       "<g clip-path=\"url(#clip6200)\">\n",
       "<text style=\"fill:#000000; fill-opacity:1; font-family:Arial,Helvetica Neue,Helvetica,sans-serif; font-size:48px; text-anchor:end;\" transform=\"rotate(0, 210.185, 1324.32)\" x=\"210.185\" y=\"1324.32\">115.75</text>\n",
       "</g>\n",
       "<g clip-path=\"url(#clip6200)\">\n",
       "<text style=\"fill:#000000; fill-opacity:1; font-family:Arial,Helvetica Neue,Helvetica,sans-serif; font-size:48px; text-anchor:end;\" transform=\"rotate(0, 210.185, 1138.73)\" x=\"210.185\" y=\"1138.73\">116.00</text>\n",
       "</g>\n",
       "<g clip-path=\"url(#clip6200)\">\n",
       "<text style=\"fill:#000000; fill-opacity:1; font-family:Arial,Helvetica Neue,Helvetica,sans-serif; font-size:48px; text-anchor:end;\" transform=\"rotate(0, 210.185, 953.138)\" x=\"210.185\" y=\"953.138\">116.25</text>\n",
       "</g>\n",
       "<g clip-path=\"url(#clip6200)\">\n",
       "<text style=\"fill:#000000; fill-opacity:1; font-family:Arial,Helvetica Neue,Helvetica,sans-serif; font-size:48px; text-anchor:end;\" transform=\"rotate(0, 210.185, 767.545)\" x=\"210.185\" y=\"767.545\">116.50</text>\n",
       "</g>\n",
       "<g clip-path=\"url(#clip6200)\">\n",
       "<text style=\"fill:#000000; fill-opacity:1; font-family:Arial,Helvetica Neue,Helvetica,sans-serif; font-size:48px; text-anchor:end;\" transform=\"rotate(0, 210.185, 581.952)\" x=\"210.185\" y=\"581.952\">116.75</text>\n",
       "</g>\n",
       "<g clip-path=\"url(#clip6200)\">\n",
       "<text style=\"fill:#000000; fill-opacity:1; font-family:Arial,Helvetica Neue,Helvetica,sans-serif; font-size:48px; text-anchor:end;\" transform=\"rotate(0, 210.185, 396.359)\" x=\"210.185\" y=\"396.359\">117.00</text>\n",
       "</g>\n",
       "<g clip-path=\"url(#clip6200)\">\n",
       "<text style=\"fill:#000000; fill-opacity:1; font-family:Arial,Helvetica Neue,Helvetica,sans-serif; font-size:48px; text-anchor:end;\" transform=\"rotate(0, 210.185, 210.766)\" x=\"210.185\" y=\"210.766\">117.25</text>\n",
       "</g>\n",
       "<polyline clip-path=\"url(#clip6202)\" style=\"stroke:#009af9; stroke-width:4; stroke-opacity:1; fill:none\" points=\"\n",
       "  294.145,199.293 427.388,615.843 560.631,464.897 693.875,338.821 827.118,238.104 960.362,162.793 1093.61,111.078 1226.85,88.0053 1360.09,88.9668 1493.34,112.669 \n",
       "  1626.58,160.936 1759.82,234.931 1893.07,333.156 2026.31,704.342 2159.55,1075.53 2292.8,1446.71 \n",
       "  \"/>\n",
       "<polygon clip-path=\"url(#clip6200)\" points=\"\n",
       "1989.93,251.724 2280.76,251.724 2280.76,130.764 1989.93,130.764 \n",
       "  \" fill=\"#ffffff\" fill-rule=\"evenodd\" fill-opacity=\"1\"/>\n",
       "<polyline clip-path=\"url(#clip6200)\" style=\"stroke:#000000; stroke-width:4; stroke-opacity:1; fill:none\" points=\"\n",
       "  1989.93,251.724 2280.76,251.724 2280.76,130.764 1989.93,130.764 1989.93,251.724 \n",
       "  \"/>\n",
       "<polyline clip-path=\"url(#clip6200)\" style=\"stroke:#009af9; stroke-width:4; stroke-opacity:1; fill:none\" points=\"\n",
       "  2013.93,191.244 2157.93,191.244 \n",
       "  \"/>\n",
       "<g clip-path=\"url(#clip6200)\">\n",
       "<text style=\"fill:#000000; fill-opacity:1; font-family:Arial,Helvetica Neue,Helvetica,sans-serif; font-size:48px; text-anchor:start;\" transform=\"rotate(0, 2181.93, 208.744)\" x=\"2181.93\" y=\"208.744\">y1</text>\n",
       "</g>\n",
       "</svg>\n"
      ]
     },
     "execution_count": 65,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "plot(0:M, optimalQ[1+s, :])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Julia 1.1.0",
   "language": "julia",
   "name": "julia-1.1"
  },
  "language_info": {
   "file_extension": ".jl",
   "mimetype": "application/julia",
   "name": "julia",
   "version": "1.1.0"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}
