{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "6xoz29ZbhlWk"
   },
   "source": [
    "# ENSEMBLE methods for MNIST dataset\n",
    "\n",
    "Titouan Vayer, titouan.vayer@inria.fr\n",
    "\n",
    "In this session, we try to use the possibilities of sklearn in terms of ENSEMBLE methods (random forests, boosting) for character recognition.\n",
    "</p>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 35
    },
    "colab_type": "code",
    "id": "pygIPffThlWm",
    "outputId": "a375cb2f-1adf-46d9-8373-f97bac3dcd8c"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Populating the interactive namespace from numpy and matplotlib\n"
     ]
    }
   ],
   "source": [
    "# loading a lot of facilities for numerical computations and graphs\n",
    "%pylab inline \n",
    "pylab.rcParams['figure.figsize'] = (20, 6) # to have larger plot\n",
    "\n",
    "# set working directory\n",
    "workingdir = './'"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "moFV5JWfhlWq"
   },
   "source": [
    "## 1. Loading the data set\n",
    "\n",
    "### 1.1 Description of the data set\n",
    "The MNIST data contains $n\\_tot=1797$ images of size $dimX\\times dimY=8\\times8$, and their *labels* (a digit between $0$ and $9$). \n",
    "The images are stored as lines of the matrix *X*. \n",
    "Each image can be reshaped as a matrix $im \\in \\mathcal{M}_{dimX, dimY}(\\mathbb{R})$, where $im[i,j]$ is the intensity level of pixel (i,j): 0 means 'white', 255 means 'black'.\n",
    "\n",
    "The labels, stored in *y*, are numbers (here, we cast them to integers) between $0$ and $9$. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 89
    },
    "colab_type": "code",
    "id": "weoZ7FrOhlWr",
    "outputId": "02082a1d-4f9b-4911-ce1d-fa19b8536caa"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Loaded 1797 images of size 8x8\n",
      "Number of classes: 10\n",
      "Classes:\n",
      "[0, 1, 2, 3, 4, 5, 6, 7, 8, 9]\n"
     ]
    }
   ],
   "source": [
    "from sklearn.datasets  import load_digits\n",
    "X, y = load_digits(return_X_y=True)\n",
    "\n",
    "\n",
    "# defining general variables for use throuhout the notebook  \n",
    "n_tot, p = shape(X)\n",
    "dimX = int(sqrt(p))\n",
    "dimY = dimX \n",
    "nc = len(unique(y)) # number of classes\n",
    "\n",
    "print(\"Loaded %d images of size %dx%d\"%(n_tot, dimX, dimY))\n",
    "print(\"Number of classes: %d\"%(nc))\n",
    "print(\"Classes:\")\n",
    "print(sorted(unique(y)))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Classification Trees\n",
    "See https://scikit-learn.org/stable/modules/generated/sklearn.tree.DecisionTreeClassifier.html#sklearn.tree.DecisionTreeClassifier"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Proportion of misclassified images in testing set: 0.176\n",
      "elapsed: 0.00064373 seconds\n"
     ]
    }
   ],
   "source": [
    "import time\n",
    "\n",
    "n_train = 600\n",
    "I = random.choice(range(n_tot), n_train, replace=False)\n",
    "trainingX = X[I, :] \n",
    "trainingTarget = y[I]\n",
    "\n",
    "# testing set: we may not use them, except at the end to assert the quality of the classifier\n",
    "mask = np.ones(n_tot, dtype=bool) #np.ones_like(a,dtype=bool)\n",
    "mask[I] = False\n",
    "testingX = X[mask, :]\n",
    "testingTarget = y[mask]\n",
    "\n",
    "from sklearn.tree import DecisionTreeClassifier\n",
    "treeClassifier = DecisionTreeClassifier()\n",
    "treeClassifier.fit(trainingX, trainingTarget)\n",
    "\n",
    "J = random.choice(range(len(testingTarget)), 500, replace=False)\n",
    "\n",
    "start_time = time.time()\n",
    "pred = treeClassifier.predict(testingX[J])\n",
    "print(\"Proportion of misclassified images in testing set: %g\" %(mean(pred != testingTarget[J])))\n",
    "print(\"elapsed: %g seconds\"%(time.time()-start_time))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "169"
      ]
     },
     "execution_count": 5,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from sklearn import tree\n",
    "treeClassifier.tree_.node_count"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Random Forests\n",
    "See https://scikit-learn.org/stable/modules/generated/sklearn.ensemble.RandomForestClassifier.html"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 53
    },
    "colab_type": "code",
    "id": "nhZAupUJhlWx",
    "outputId": "0593d4e3-4b82-4312-f8a7-eb38a7631c7d"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Proportion of misclassified images in testing set: 0.032\n",
      "elapsed: 0.0139301 seconds\n"
     ]
    }
   ],
   "source": [
    "import time\n",
    "\n",
    "n_train = 600\n",
    "I = random.choice(range(n_tot), n_train, replace=False)\n",
    "trainingX = X[I, :] \n",
    "trainingTarget = y[I]\n",
    "\n",
    "# testing set: we may not use them, except at the end to assert the quality of the classifier\n",
    "mask = np.ones(n_tot, dtype=bool) #np.ones_like(a,dtype=bool)\n",
    "mask[I] = False\n",
    "testingX = X[mask, :]\n",
    "testingTarget = y[mask]\n",
    "\n",
    "from sklearn.ensemble import RandomForestClassifier\n",
    "rfClassifier = RandomForestClassifier()\n",
    "rfClassifier.fit(trainingX, trainingTarget)\n",
    "\n",
    "J = random.choice(range(len(testingTarget)), 500, replace=False)\n",
    "\n",
    "start_time = time.time()\n",
    "pred = rfClassifier.predict(testingX[J])\n",
    "print(\"Proportion of misclassified images in testing set: %g\" %(mean(pred != testingTarget[J])))\n",
    "print(\"elapsed: %g seconds\"%(time.time()-start_time))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Boosting \n",
    "See https://scikit-learn.org/stable/modules/generated/sklearn.ensemble.AdaBoostClassifier.html for adaboost, and \n",
    "https://scikit-learn.org/stable/modules/generated/sklearn.ensemble.GradientBoostingClassifier.html for gradient boosting"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Proportion of misclassified images in testing set: 0.606\n",
      "elapsed: 0.008919 seconds\n"
     ]
    }
   ],
   "source": [
    "# adaboost\n",
    "\n",
    "import time\n",
    "\n",
    "n_train = 600\n",
    "I = random.choice(range(n_tot), n_train, replace=False)\n",
    "trainingX = X[I, :] \n",
    "trainingTarget = y[I]\n",
    "\n",
    "# testing set: we may not use them, except at the end to assert the quality of the classifier\n",
    "mask = np.ones(n_tot, dtype=bool) #np.ones_like(a,dtype=bool)\n",
    "mask[I] = False\n",
    "testingX = X[mask, :]\n",
    "testingTarget = y[mask]\n",
    "\n",
    "from sklearn.ensemble import AdaBoostClassifier\n",
    "adaboostClassifier = AdaBoostClassifier()\n",
    "adaboostClassifier.fit(trainingX, trainingTarget)\n",
    "\n",
    "J = random.choice(range(len(testingTarget)), 500, replace=False)\n",
    "\n",
    "start_time = time.time()\n",
    "pred = adaboostClassifier.predict(testingX[J])\n",
    "print(\"Proportion of misclassified images in testing set: %g\" %(mean(pred != testingTarget[J])))\n",
    "print(\"elapsed: %g seconds\"%(time.time()-start_time))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Proportion of misclassified images in testing set: 0.064\n",
      "elapsed: 0.00798798 seconds\n"
     ]
    }
   ],
   "source": [
    "# gradient boosting\n",
    "\n",
    "import time\n",
    "\n",
    "n_train = 600\n",
    "I = random.choice(range(n_tot), n_train, replace=False)\n",
    "trainingX = X[I, :] \n",
    "trainingTarget = y[I]\n",
    "\n",
    "# testing set: we may not use them, except at the end to assert the quality of the classifier\n",
    "mask = np.ones(n_tot, dtype=bool) #np.ones_like(a,dtype=bool)\n",
    "mask[I] = False\n",
    "testingX = X[mask, :]\n",
    "testingTarget = y[mask]\n",
    "\n",
    "from sklearn.ensemble import GradientBoostingClassifier\n",
    "gboostClassifier = GradientBoostingClassifier()\n",
    "gboostClassifier.fit(trainingX, trainingTarget)\n",
    "\n",
    "J = random.choice(range(len(testingTarget)), 500, replace=False)\n",
    "\n",
    "start_time = time.time()\n",
    "pred = gboostClassifier.predict(testingX[J])\n",
    "print(\"Proportion of misclassified images in testing set: %g\" %(mean(pred != testingTarget[J])))\n",
    "print(\"elapsed: %g seconds\"%(time.time()-start_time))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "CLbpuDoWhlW5"
   },
   "source": [
    "## Questions"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Q0. What is the problems with the previous analysis ? How can we properly compare these different models ? "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.model_selection import cross_val_score"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "jR6_Y-UlhlW5"
   },
   "source": [
    "### Q1. Play with the parameters of the CART classifier (max_depth, max_leaf_nodes, criterion, etc.) and measure the influence on:\n",
    "  * the performance\n",
    "  * the execution time"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "NK2nezchhlW6"
   },
   "source": [
    "### Q2. Play with the parameters of the Random Forest classifier (max_depth, max_features, etc.) and measure the influence on:\n",
    "  * the performance\n",
    "  * the execution time"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Q3. Use Grid-Search for proper model selection and performance evaluation"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.model_selection import train_test_split\n",
    "\n",
    "X_train, X_test, y_train, y_test = train_test_split(\n",
    "    X, y, random_state=2408, test_size=0.33\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "The hyper-parameters for RandomForestClassifier are:\n",
      "bootstrap\n",
      "ccp_alpha\n",
      "class_weight\n",
      "criterion\n",
      "max_depth\n",
      "max_features\n",
      "max_leaf_nodes\n",
      "max_samples\n",
      "min_impurity_decrease\n",
      "min_samples_leaf\n",
      "min_samples_split\n",
      "min_weight_fraction_leaf\n",
      "n_estimators\n",
      "n_jobs\n",
      "oob_score\n",
      "random_state\n",
      "verbose\n",
      "warm_start\n"
     ]
    }
   ],
   "source": [
    "print(\"The hyper-parameters for RandomForestClassifier are:\")\n",
    "for param_name in RandomForestClassifier().get_params().keys():\n",
    "    print(param_name)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Fitting 5 folds for each of 8 candidates, totalling 40 fits\n"
     ]
    }
   ],
   "source": [
    "from sklearn.model_selection import GridSearchCV\n",
    "import time\n",
    "import numpy as np\n",
    "\n",
    "model = RandomForestClassifier()\n",
    "\n",
    "\n",
    "param_grid = {'n_estimators': (100, 300, 500, 1000), 'max_depth': (None, 5)}\n",
    "model_grid_search = GridSearchCV(model, param_grid=param_grid, n_jobs=4, cv=5, verbose=2)\n",
    "start = time.time()\n",
    "model_grid_search.fit(X_train, y_train)\n",
    "elapsed_time = time.time() - start\n",
    "\n",
    " "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "The accuracy score using a GridSearchCV is 0.97 in 10.757 seconds\n"
     ]
    }
   ],
   "source": [
    "print(\n",
    "    f\"The accuracy score using a {model_grid_search.__class__.__name__} is \"\n",
    "    f\"{model_grid_search.score(X_test, y_test):.2f} in \"\n",
    "    f\"{elapsed_time:.3f} seconds\"\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'max_depth': None, 'n_estimators': 500}"
      ]
     },
     "execution_count": 22,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "model_grid_search.best_params_"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "CfYHph4ThlW7"
   },
   "source": [
    "### Q4. Play with the number of iterations for the Adaboost classifier and measure the influence on:\n",
    "  * the performance\n",
    "  * the execution time"
   ]
  }
 ],
 "metadata": {
  "colab": {
   "collapsed_sections": [],
   "name": "correction_mnist_dimred.ipynb",
   "provenance": []
  },
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.8.5"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 1
}
