{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Séance 2: Data and methodology of Machine Learning \n",
    "\n",
    "Mathurin Massias, mathurin.massias@inria.fr\n",
    "\n",
    "Material taken from the Euroscipy 2019 scikit-learn tutorial by Loic Esteve and Olivier Grisel."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Today we will build our first Machine Learning pipeline, from dataloading to prediction through feature extraction and hyperparameter tuning.\n",
    "\n",
    "In this notebook, we will look at necessary steps that happen before any machine learning takes place. \n",
    "* load the data\n",
    "* look at the variables in the dataset, in particular make the difference\n",
    "  between numerical and categorical variables, which need different\n",
    "  preprocessing in most machine learning workflows\n",
    "* visualize the distribution of the variables to gain some insights into the dataset."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import pandas as pd\n",
    "import seaborn as sns"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Loading the adult census dataset"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We will use data from the \"Current Population adult_census\" from 1994 that we\n",
    "downloaded from [OpenML](http://openml.org/)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "adult_census = pd.read_csv(\"https://www.openml.org/data/get_csv/1595261/adult-census.csv\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "What does csv mean? Do you notice anything special about our use of the `pandas.read_csv` function?"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can look at the OpenML webpage to know more about this dataset: https://www.openml.org/d/1590"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Look at the variables in the dataset\n",
    "The data are stored in a pandas dataframe."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "adult_census.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The column named **class** is our target variable (i.e., the variable which\n",
    "we want to predict). The two possible classes are `<= 50K` (low-revenue) and\n",
    "`> 50K` (high-revenue)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "What type of data do we have? And which problem are we considering? Can you envision practical usecases of this dataset?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "adult_census.shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "target_column = 'class'\n",
    "adult_census[target_column].value_counts()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Note: classes are slighly imbalanced. Class imbalance happens often in\n",
    "practice and may need special techniques for machine learning. For example in\n",
    "a medical setting, if we are trying to predict whether patients will develop\n",
    "a rare disease, there will be a lot more sane patients than ill patients in\n",
    "the dataset."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The dataset contains both numerical and categorical data. Numerical values\n",
    "can take continuous values, for example `age`. Categorical values can have a\n",
    "finite number of values, for exemple `native-country`.\n",
    "\n",
    "In practice, `age` takes a finite number of values in the dataset. Yet, how is\n",
    "it different from a categorical value?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "numerical_columns = ['age', 'education-num', 'capital-gain', 'capital-loss',\n",
    "                     'hours-per-week']\n",
    "categorical_columns = ['workclass', 'education', 'marital-status', 'occupation',\n",
    "                       'relationship', 'race', 'sex', 'native-country']\n",
    "all_columns = numerical_columns + categorical_columns + [target_column]\n",
    "\n",
    "adult_census = adult_census[all_columns]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "adult_census"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Note that for simplicity, we have ignored the \"fnlwgt\" (final weight) column\n",
    "that was crafted by the creators of the dataset when sampling the dataset to\n",
    "be representative of the full census database."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "adult_census.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Inspect the data\n",
    "Before building a machine learning model, it is a good idea to look at the\n",
    "data:\n",
    "* maybe the task you are trying to achieve can be solved without machine\n",
    "  learning\n",
    "* you need to check that the data you need for your task is indeed present in\n",
    "the dataset\n",
    "* inspecting the data is a good way to find peculiarities. These can can\n",
    "  arise in the data collection (for example, malfunctioning sensor or missing\n",
    "  values), or the way the data is processed afterwards (for example capped\n",
    "  values)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's look at the distribution of individual variables, to get some insights\n",
    "about the data. We can start by plotting histograms."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "lines_to_next_cell": 2
   },
   "outputs": [],
   "source": [
    "adult_census.hist(figsize=(20, 10));  # what did pandas do?"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can already make a few comments about some of the variables:\n",
    "* age: there are not that many points for 'age > 70'. The dataset description\n",
    "does indicate that retired people have been filtered out (`hours-per-week > 0`).\n",
    "* education-num: peak at 10 and 13, hard to tell what it corresponds to\n",
    "without looking much further. We'll do that later in this notebook.\n",
    "* hours per week at 40, this was very likely the standard of working hours at\n",
    "the time of the data collection\n",
    "* most values of capital-gain and capital-loss are close to zero"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "lines_to_next_cell": 2
   },
   "source": [
    "For categorical variables, we can look at the distribution of values:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "adult_census['sex'].value_counts()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "adult_census['native-country'].value_counts()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "What's the issue with the approach we've adopted so far?"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "`pandas_profiling` is a nice tool for inspecting the data (both numerical and\n",
    "categorical variables)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "!pip install -U pandas_profiling"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import ydata_profiling\n",
    "adult_census.profile_report()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "lines_to_next_cell": 0
   },
   "source": [
    "As noted above, `education-num` distribution has two clear peaks around 10\n",
    "and 13. It would be reasonable to expect that 'education-num' is the number of\n",
    "years of education. Let's look at the relationship between education and\n",
    "education-num."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "pd.crosstab(index=adult_census['education'], columns=adult_census['education-num'])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This shows that education and education-num are redundant. For\n",
    "example, `education-num=2` is equivalent to `education='1st-4th'`. In\n",
    "practice that means we can remove `education-num` without losing information.\n",
    "Note that having redundant (or highly correlated) columns can be a problem\n",
    "for machine learning algorithms."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Another way to inspect the data is to do a pairplot and show how variable\n",
    "differ according to the class. In the diagonal you can see the distribution\n",
    "of individual variables. The plots on the off-diagonal can reveal interesting\n",
    "interactions between variables."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In the snippet below, what is the purpose of `n_samples_to_plot`? \n",
    "What's the advantage? What's the risk? How can we circumvent it?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "n_samples_to_plot = 5000\n",
    "columns = ['age', 'education-num', 'hours-per-week']\n",
    "sns.pairplot(data=adult_census[0:n_samples_to_plot] , vars=columns,\n",
    "             hue=target_column, plot_kws={'alpha': 0.2}, height=4,\n",
    "             diag_kind='hist');"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "adult_census.columns"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "sns.pairplot(data=adult_census[:n_samples_to_plot], x_vars='age', y_vars='hours-per-week',\n",
    "             hue=target_column, markers=['o', 'v'], height=6)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "lines_to_next_cell": 2
   },
   "source": [
    "\n",
    "By looking at the data you could infer some hand-written rules to predict the\n",
    "class:\n",
    "* if you are young (less than 25 year-old roughly), you are in the `<= 50K` class.\n",
    "* if you are old (more than 70 year-old roughly), you are in the `<= 50K` class.\n",
    "* if you work part-time (less than 40 hours roughly) you are in the `<= 50K` class.\n",
    "\n",
    "These hand-written rules could work reasonably well without the need for any\n",
    "machine learning. Note however that it is not very easy to create rules for\n",
    "the region `40 < hours-per-week < 60` and `30 < age < 70`. We can hope that\n",
    "machine learning can help in this region. Also note that visualization can\n",
    "help creating hand-written rules but is limited to 2 dimensions (maybe 3\n",
    "dimensions), whereas machine learning models can build models in\n",
    "high-dimensional spaces.\n",
    "\n",
    "Another thing worth mentioning in this plot: if you are young (less than 25\n",
    "year-old roughly) you tend to work less and if you are old (more than 70\n",
    "year-old roughly). This is a non-linear relationship between age and hours\n",
    "per week. **Some machine learning models can only capture linear interaction so\n",
    "this may be a factor when deciding which model to chose.**\n",
    "\n",
    "In a machine-learning setting, we will use an algorithm to automatically\n",
    "decide what should be the \"rules\" in order to predict on new data. We can\n",
    "check which set of simple rule a decision tree would grasp using the same\n",
    "data."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "def plot_tree_decision_function(tree, X, y, ax):\n",
    "    \"\"\"Plot the different decision rules found by a `DecisionTreeClassifier`.\n",
    "\n",
    "    Parameters\n",
    "    ----------\n",
    "    tree : DecisionTreeClassifier instance\n",
    "        The decision tree to inspect.\n",
    "    X : dataframe of shape (n_samples, n_features)\n",
    "        The data used to train the `tree` estimator.\n",
    "    y : ndarray of shape (n_samples,)\n",
    "        The target used to train the `tree` estimator.\n",
    "    ax : matplotlib axis\n",
    "        The matplotlib axis where to plot the different decision rules.\n",
    "    \"\"\"\n",
    "    import numpy as np\n",
    "    from scipy import ndimage\n",
    "\n",
    "    h = 0.02\n",
    "    x_min, x_max = 0, 100\n",
    "    y_min, y_max = 0, 100\n",
    "    xx, yy = np.meshgrid(\n",
    "        np.arange(x_min, x_max, h), np.arange(y_min, y_max, h)\n",
    "    )\n",
    "\n",
    "    Z = tree.predict_proba(np.c_[xx.ravel(), yy.ravel()])[:, 1]\n",
    "    Z = Z.reshape(xx.shape)\n",
    "    faces = tree.tree_.apply(\n",
    "        np.c_[xx.ravel(), yy.ravel()].astype(np.float32)\n",
    "    )\n",
    "    faces = faces.reshape(xx.shape)\n",
    "    border = ndimage.laplace(faces) != 0\n",
    "    ax.scatter(\n",
    "        X.iloc[:, 0], X.iloc[:, 1], c=np.array(['tab:blue', 'tab:red'])[y],\n",
    "        s=60, alpha=0.7\n",
    "    )\n",
    "    ax.contourf(xx, yy, Z, alpha=.4, cmap='RdBu_r')\n",
    "    ax.scatter(xx[border], yy[border], marker='.', s=1)\n",
    "    ax.set_xlabel(X.columns[0])\n",
    "    ax.set_ylabel(X.columns[1])\n",
    "    ax.set_xlim([x_min, x_max])\n",
    "    ax.set_ylim([y_min, y_max])\n",
    "    sns.despine(offset=10)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import matplotlib as mpl\n",
    "import matplotlib.pyplot as plt\n",
    "mpl.rcParams.update(mpl.rcParamsDefault)  # reset the plotting style"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.preprocessing import LabelEncoder\n",
    "\n",
    "# select a subset of data\n",
    "data_subset = adult_census[:n_samples_to_plot]\n",
    "X = data_subset[[\"age\", \"hours-per-week\"]]\n",
    "y = LabelEncoder().fit_transform(data_subset[target_column].to_numpy())\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "y "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "data_subset[target_columncolumn]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We will create a decision tree which we will keep really simple on purpose.\n",
    "We will only allow a maximum of 2 rules to interpret the results."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.tree import DecisionTreeClassifier\n",
    "from sklearn.tree import plot_tree\n",
    "\n",
    "max_leaf_nodes = 3\n",
    "tree = DecisionTreeClassifier(max_leaf_nodes=max_leaf_nodes, random_state=0)\n",
    "tree.fit(X, y)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can now first check the set of rules learnt by the tree and check\n",
    "visually what theses rules look like."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# plot the tree structure\n",
    "fig, ax = plt.subplots()\n",
    "plot_tree(tree, ax=ax)\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# plot the decision function learned by the tree\n",
    "fig, ax = plt.subplots()\n",
    "plot_tree_decision_function(tree, X, y, ax=ax)\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Allowing only 3 leaves in the tree, we get similar rules than the one\n",
    "designed by hand:\n",
    "* the persons younger than 28.5 year-old will be considered in the class\n",
    "  earning `<= 50K`.\n",
    "* the persons older than 28.5 and working more than 40.5 hours-per-week\n",
    "  will be considered in the class earning `> 50K`, while the one working\n",
    "  below 40.5 hours-per-week, will be considered in the class\n",
    "  earning `<= 50K`."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "lines_to_next_cell": 2
   },
   "source": [
    "\n",
    "So far we have:\n",
    "* loaded the data from a CSV file using `pandas`\n",
    "* looked at the kind of variables in the dataset, and make the difference\n",
    "  between categorical and numerical variables.\n",
    "* inspected the data with `pandas`, `seaborn` and `pandas_profiling`. Data inspection\n",
    "  can allow you to decide whether using machine learning is appropriate for\n",
    "  your data and to notice potential peculiarities in your data.\n",
    "  \n",
    "**Let's move to more advanced model fitting**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "target_name = \"class\"\n",
    "target = adult_census[target_name]\n",
    "data = adult_census.drop(columns=[target_name])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "print(\n",
    "    f\"The dataset contains {data.shape[0]} samples and {data.shape[1]} \"\n",
    "    \"features\"\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's check again the type of data we have: "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "data.dtypes"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import numpy as np\n",
    "data_numeric = data.select_dtypes(include=np.number)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "data_numeric"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We're going to train various models. How can we choose the best? \n",
    "- choice of metric (here OK: classification)\n",
    "- applied to what?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.model_selection import train_test_split\n",
    "\n",
    "data_train, data_test, target_train, target_test = train_test_split(\n",
    "    data_numeric, target, random_state=2408\n",
    ")\n",
    "\n",
    "print(\n",
    "    f\"The training dataset contains {data_train.shape[0]} samples and \"\n",
    "    f\"{data_train.shape[1]} features\"\n",
    ")\n",
    "print(\n",
    "    f\"The testing dataset contains {data_test.shape[0]} samples and \"\n",
    "    f\"{data_test.shape[1]} features\"\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We will build a linear classification model called \"Logistic Regression\". The\n",
    "`fit` method is called to train the model from the input and target data. Only\n",
    "the training data should be given for this purpose.\n",
    "\n",
    "In addition, when checking the time required to train the model and internally\n",
    "check the number of iterations done by the solver to find a solution."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.linear_model import LogisticRegression\n",
    "import time\n",
    "\n",
    "model = LogisticRegression(solver='lbfgs', max_iter=1000)\n",
    "start = time.time()\n",
    "model.fit(data_train, target_train)\n",
    "elapsed_time = time.time() - start\n",
    "\n",
    "print(\n",
    "    f\"The model {model.__class__.__name__} was trained in \"\n",
    "    f\"{elapsed_time:.3f} seconds for {model.n_iter_} iterations\"\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "target_predicted = model.predict(data_test)\n",
    "predictions = data_test.copy()\n",
    "predictions['predicted-class'] = target_predicted\n",
    "predictions['expected-class'] = target_test\n",
    "predictions['correct'] = target_predicted == target_test\n",
    "predictions.head(n=10)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "print(\n",
    "    f\"The test accuracy using a {model.__class__.__name__} is \"\n",
    "    f\"{model.score(data_test, target_test):.5f}\"\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Exercise 1\n",
    "\n",
    "- What would be the score of a model that always predicts `' >50K'`?\n",
    "- What would be the score of a model that always predicts `' <= 50K'`?\n",
    "- Is 81% or 82% accuracy a good score for this problem?\n",
    "\n",
    "Hint: You can compute the cross-validated of a [DummyClassifier](https://scikit-learn.org/stable/modules/model_evaluation.html#dummy-estimators) the performance of such baselines."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "data_train.describe()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.preprocessing import StandardScaler\n",
    "\n",
    "scaler = StandardScaler()\n",
    "data_train_scaled = scaler.fit_transform(data_train)\n",
    "data_train_scaled"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's have a look at the transformed data:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "data_train_scaled = pd.DataFrame(data_train_scaled, columns=data_train.columns)\n",
    "data_train_scaled.describe()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.pipeline import make_pipeline\n",
    "\n",
    "model = make_pipeline(StandardScaler(), LogisticRegression(solver='lbfgs'))\n",
    "start = time.time()\n",
    "model.fit(data_train, target_train)\n",
    "elapsed_time = time.time() - start"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "print(\n",
    "    f\"The accuracy using a {model.__class__.__name__} is \"\n",
    "    f\"{model.score(data_test, target_test):.3f} with a fitting time of \"\n",
    "    f\"{elapsed_time:.3f} seconds in {model[-1].n_iter_} iterations\"\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's now perform 5 fold cross validation:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "for k in range(5):\n",
    "    train, test = ..., ..."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Don't do as above, coding the CV yourself! \n",
    "\n",
    "Always look for functions already coded (in reference packages): \n",
    "- they'll be more correct than our version\n",
    "- they'll be faster than our version\n",
    "- they'll have nicer functionalities that we didn't know we wanted\n",
    "- we'll leverage the work of many smart people that asked themselves the same question as we are asking now"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.model_selection import cross_val_score\n",
    "\n",
    "scores = cross_val_score(model, data_numeric, target, cv=5)\n",
    "print(f\"The different scores obtained are: \\n{scores}\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "print(f\"The mean cross-validation accuracy is: \"\n",
    "      f\"{scores.mean():.3f} +/- {scores.std():.3f}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Working with categorical variables\n",
    "\n",
    "As we have seen in the previous section, a numerical variable is a continuous\n",
    "quantity represented by a real or integer number. Those variables can be\n",
    "naturally handled by machine learning algorithms that typically composed of\n",
    "a sequence of arithmetic instructions such as additions and multiplications.\n",
    "\n",
    "By opposition, categorical variables have discrete values typically represented\n",
    "by string labels taken in a finite list of possible choices. For instance, the\n",
    "variable `native-country` in our dataset is a categorical variable because it\n",
    "encodes the data using a finite list of possible countries (along with the `?`\n",
    "marker when this information is missing):"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "data[\"native-country\"].value_counts()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In the remainder of this section, we will present different strategies to\n",
    "encode categorical data into numerical data which can be used by a\n",
    "machine-learning algorithm."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "data_categorical = adult_census.select_dtypes(include=[object]).drop(columns=[\"class\"])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "data_categorical.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "print(f\"The datasets is composed of {data_categorical.shape[1]} features\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Encoding ordinal categories\n",
    "\n",
    "The most intuitive strategy is to encode each category with a number.\n",
    "The `OrdinalEncoder` will transform the data in such manner."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.preprocessing import OrdinalEncoder\n",
    "\n",
    "encoder = OrdinalEncoder()\n",
    "data_encoded = encoder.fit_transform(data_categorical)\n",
    "\n",
    "print(f\"The dataset encoded contains {data_encoded.shape[1]} features\")\n",
    "data_encoded[:5]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can see that all categories have been encoded for each feature\n",
    "independently. We can also notice that the number of features before and after\n",
    "the encoding is the same.\n",
    "\n",
    "However, one has to be careful when using this encoding strategy. Using this\n",
    "integer representation can lead the downstream models to make the assumption\n",
    "that the categories are ordered: 0 is smaller than 1 which is smaller than 2,\n",
    "etc.\n",
    "\n",
    "By default, `OrdinalEncoder` uses a lexicographical strategy to map string\n",
    "category labels to integers. This strategy is completely arbitrary and often be\n",
    "meaningless. For instance suppose the dataset has a categorical variable named\n",
    "\"size\" with categories such as \"S\", \"M\", \"L\", \"XL\". We would like the integer\n",
    "representation to respect the meaning of the sizes by mapping them to increasing\n",
    "integers such as 0, 1, 2, 3. However lexicographical strategy used by default\n",
    "would map the labels \"S\", \"M\", \"L\", \"XL\" to 2, 1, 0, 3.\n",
    "\n",
    "The `OrdinalEncoder` class accepts a \"categories\" constructor argument to pass\n",
    "an the correct ordering explicitly.\n",
    "\n",
    "If a categorical variable does not carry any meaningful order information then\n",
    "this encoding might be misleading to downstream statistical models and you might\n",
    "consider using one-hot encoding instead (see below).\n",
    "\n",
    "Note however that the impact a violation of this ordering assumption is really\n",
    "dependent on the downstream models (for instance linear models are much more\n",
    "sensitive than models built from a ensemble of decision trees).\n",
    "\n",
    "### Encoding nominal categories (without assuming any order)\n",
    "\n",
    "`OneHotEncoder` is an alternative encoder that can prevent the dowstream\n",
    "models to make a false assumption about the ordering of categories. For a\n",
    "given feature, it will create as many new columns as there are possible\n",
    "categories. For a given sample, the value of the column corresponding to the\n",
    "category will be set to `1` while all the columns of the other categories will\n",
    "be set to `0`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.preprocessing import OneHotEncoder\n",
    "\n",
    "encoder = OneHotEncoder(sparse=False)\n",
    "data_encoded = encoder.fit_transform(data_categorical)\n",
    "print(f\"The dataset encoded contains {data_encoded.shape[1]} features\")\n",
    "data_encoded"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "columns_encoded = encoder.get_feature_names_out(data_categorical.columns)\n",
    "pd.DataFrame(data_encoded, columns=columns_encoded).head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "model = make_pipeline(\n",
    "    OneHotEncoder(handle_unknown='ignore'),\n",
    "    LogisticRegression(solver='lbfgs', max_iter=1000)\n",
    ")\n",
    "scores = cross_val_score(model, data_categorical, target, cv=10)\n",
    "print(f\"The different scores obtained are: \\n{scores}\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "print(f\"The accuracy is: {scores.mean():.3f} +/- {scores.std():.5f}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Exercise 2:\n",
    "\n",
    "- Try to fit a logistic regression model on categorical data transformed by\n",
    "  the OrdinalEncoder instead. What do you observe?"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Using all the data at hand and specifying which columns to transform how"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "binary_encoding_columns = ['sex']\n",
    "one_hot_encoding_columns = ['workclass', 'education', 'marital-status',\n",
    "                            'occupation', 'relationship',\n",
    "                            'race', 'native-country']\n",
    "scaling_columns = ['age', 'education-num', 'hours-per-week',\n",
    "                   'capital-gain', 'capital-loss']\n",
    "\n",
    "from sklearn.compose import ColumnTransformer\n",
    "\n",
    "preprocessor = ColumnTransformer([\n",
    "    ('binary-encoder', OrdinalEncoder(), binary_encoding_columns),\n",
    "    ('one-hot-encoder', OneHotEncoder(handle_unknown='ignore'),\n",
    "     one_hot_encoding_columns),\n",
    "    ('standard-scaler', StandardScaler(), scaling_columns)\n",
    "])\n",
    "model = make_pipeline(\n",
    "    preprocessor,\n",
    "    LogisticRegression(solver='lbfgs', max_iter=1000)\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Pipelines, as all sklearn models, can be displayed!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "model"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "scores = cross_val_score(model, data, target, cv=5)\n",
    "print(f\"The different scores obtained are: \\n{scores}\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "print(f\"The accuracy is: {scores.mean():.3f} +- {scores.std():.3f}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The compound model has a higher predictive accuracy than the\n",
    "two models that used numerical and categorical variables in\n",
    "isolation."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## The issue of finding the best model parameters\n",
    "\n",
    "In the previous example, we created a `LogisticRegression` classifier using\n",
    "the default parameters by omitting setting explicitly these parameters.\n",
    "\n",
    "For this classifier, the parameter `C` governes the penalty; in other\n",
    "words, how much our model should \"trust\" (or fit) the training data.\n",
    "\n",
    "Therefore, the default value of `C` is never certified to give the best\n",
    "performing model.\n",
    "\n",
    "We can make a quick experiment by changing the value of `C` and see the\n",
    "impact of this parameter on the model performance."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "data_train, data_test, target_train, target_test = train_test_split(\n",
    "    data, target, random_state=0)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "C = 1\n",
    "model = make_pipeline(\n",
    "    preprocessor, LogisticRegression(C=C, max_iter=1000, solver='lbfgs')\n",
    ")\n",
    "model.fit(data_train, target_train)\n",
    "print(\n",
    "    f\"The accuracy score using a {model.__class__.__name__} is \"\n",
    "    f\"{model.score(data_test, target_test):.2f} with C={C}\"\n",
    ")\n",
    "\n",
    "C = 1e-5\n",
    "model = make_pipeline(\n",
    "    preprocessor, LogisticRegression(C=C, max_iter=1000, solver='lbfgs')\n",
    ")\n",
    "model.fit(data_train, target_train)\n",
    "print(\n",
    "    f\"The accuracy score using a {model.__class__.__name__} is \"\n",
    "    f\"{model.score(data_test, target_test):.2f} with C={C}\"\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "\n",
    "## Finding the best model hyper-parameters via exhaustive parameters search\n",
    "\n",
    "We see that the parameter `C` as a significative impact on the model\n",
    "performance. This parameter should be tuned to get the best cross-validation\n",
    "score, so as to avoid over-fitting problems.\n",
    "\n",
    "In short, we will set the parameter, train our model on some data, and\n",
    "evaluate the model performance on some left out data. Ideally, we will select\n",
    "the parameter leading to the optimal performance on the testing set.\n",
    "Scikit-learn provides a `GridSearchCV` estimator which will handle the\n",
    "cross-validation and hyper-parameter search for us."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.model_selection import GridSearchCV\n",
    "\n",
    "model = make_pipeline(\n",
    "    preprocessor, LogisticRegression(max_iter=1000, solver='lbfgs')\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We will see that we need to provide the name of the parameter to be set.\n",
    "Thus, we can use the method `get_params()` to have the list of the parameters\n",
    "of the model which can set during the grid-search."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "print(\"The hyper-parameters are for a logistic regression model are:\")\n",
    "for param_name in LogisticRegression().get_params().keys():\n",
    "    print(param_name)\n",
    "\n",
    "print(\"The hyper-parameters are for the full-pipeline are:\")\n",
    "for param_name in model.get_params().keys():\n",
    "    print(param_name)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The parameter `'logisticregression__C'` is the parameter for which we would\n",
    "like different values. Let see how to use the `GridSearchCV` estimator for\n",
    "doing such search."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import time\n",
    "import numpy as np\n",
    "\n",
    "param_grid = {'logisticregression__C': (0.1, 1.0, 10.0)}\n",
    "model_grid_search = GridSearchCV(model, param_grid=param_grid, n_jobs=4, cv=5)\n",
    "start = time.time()\n",
    "model_grid_search.fit(data_train, target_train)\n",
    "elapsed_time = time.time() - start\n",
    "print(\n",
    "    f\"The accuracy score using a {model_grid_search.__class__.__name__} is \"\n",
    "    f\"{model_grid_search.score(data_test, target_test):.2f} in \"\n",
    "    f\"{elapsed_time:.3f} seconds\"\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The `GridSearchCV` estimator takes a `param_grid` parameter which defines\n",
    "all possible parameters combination. Once the grid-search fitted, it can be\n",
    "used as any other predictor by calling `predict` and `predict_proba`.\n",
    "Internally, it will use the model with the best parameters found during\n",
    "`fit`. You can know about these parameters by looking at the `best_params_`\n",
    "attribute."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "print(f\"The best set of parameters is: {model_grid_search.best_params_}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "With the `GridSearchCV` estimator, the parameters need to be specified\n",
    "explicitely. Instead, one could randomly generate (following a specific\n",
    "distribution) the parameter candidates. The `RandomSearchCV` allows for such\n",
    "stochastic search. It is used similarly to the `GridSearchCV` but the\n",
    "sampling distributions need to be specified instead of the parameter values."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from scipy.stats import uniform\n",
    "from sklearn.model_selection import RandomizedSearchCV\n",
    "\n",
    "param_distributions = {'logisticregression__C': uniform(loc=50, scale=100)}\n",
    "model_grid_search = RandomizedSearchCV(\n",
    "    model, param_distributions=param_distributions, n_iter=3, n_jobs=4, cv=5\n",
    ")\n",
    "model_grid_search.fit(data_train, target_train)\n",
    "print(\n",
    "    f\"The accuracy score using a {model_grid_search.__class__.__name__} is \"\n",
    "    f\"{model_grid_search.score(data_test, target_test):.2f}\"\n",
    ")\n",
    "print(f\"The best set of parameters is: {model_grid_search.best_params_}\")\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Notes on search efficiency\n",
    "\n",
    "Be aware that sometimes, scikit-learn provides some `EstimatorCV` classes\n",
    "which will perform internally the cross-validation in such way that it will\n",
    "more computationally efficient. We can give the example of the\n",
    "`LogisticRegressionCV` which can be used to find the best `C` in a more\n",
    "efficient way than what we previously did with the `GridSearchCV`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.linear_model import LogisticRegressionCV\n",
    "\n",
    "# define the different Cs to try out\n",
    "param_grid = {\"C\": (0.1, 1.0, 10.0)}\n",
    "\n",
    "model = make_pipeline(\n",
    "    preprocessor,\n",
    "    LogisticRegressionCV(Cs=param_grid['C'], max_iter=1000, solver='lbfgs',\n",
    "                         n_jobs=4, cv=5)\n",
    ")\n",
    "start = time.time()\n",
    "model.fit(data_train, target_train)\n",
    "elapsed_time = time.time() - start\n",
    "print(f\"Time elapsed to train LogisticRegressionCV: \"\n",
    "      f\"{elapsed_time:.3f} seconds\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The `fit` time for the `CV` version of `LogisticRegression` give a speed-up\n",
    "x2. This speed-up is provided by re-using the values of coefficients to\n",
    "warm-start the estimator for the different `C` values.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Exercises:\n",
    "\n",
    "- Build a machine learning pipeline:\n",
    "      * preprocess the categorical columns using an `OrdinalEncoder` and let\n",
    "        the numerical columns as they are.\n",
    "      * use an `HistGradientBoostingClassifier` as a predictive model.\n",
    "- Make an hyper-parameters search using `RandomizedSearchCV` and tuning the\n",
    "  parameters:\n",
    "      * `learning_rate` with values ranging from 0.001 to 0.5. You can use\n",
    "        an exponential distribution to sample the possible values.\n",
    "      * `l2_regularization` with values ranging from 0 to 0.5. You can use\n",
    "        a uniform distribution.\n",
    "      * `max_lead_nodes` with values ranging from 5 to 30. The values should\n",
    "        be integer following a uniform distribution.\n",
    "      * `min_samples_leaf` with values ranging from 5 to 30. The values\n",
    "        should be integer following a uniform distribution.\n",
    "\n",
    "In case you have issues of with unknown categories, try to precompute the\n",
    "list of possible categories ahead of time and pass it explicitly to the\n",
    "constructor of the encoder:\n",
    "\n",
    "```python\n",
    "categories = [data[column].unique()\n",
    "              for column in data[categorical_columns]]\n",
    "OrdinalEncoder(categories=categories)\n",
    "```\n",
    "\n",
    "## Combining evaluation and hyper-parameters search\n",
    "\n",
    "Cross-validation was used for searching the best model parameters. We\n",
    "previously evaluate model performance through cross-validation as well. If we\n",
    "would like to combine both aspects, we need to perform a \"nested\"\n",
    "cross-validation. The \"outer\" cross-validation is applied to assess the\n",
    "model while the \"inner\" cross-validation set the hyper-parameters of the\n",
    "model on the data set provided by the \"outer\" cross-validation. In practice,\n",
    "it is equivalent of including, `GridSearchCV`, `RandomSearchCV`, or any\n",
    "`EstimatorCV` in a `cross_val_score` or `cross_validate` function call."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.model_selection import cross_val_score\n",
    "\n",
    "model = make_pipeline(\n",
    "    preprocessor, LogisticRegressionCV(max_iter=1000, solver='lbfgs', cv=5)\n",
    ")\n",
    "score = cross_val_score(model, data, target, n_jobs=4, cv=5)\n",
    "print(f\"The accuracy score is: {score.mean():.2f} +- {score.std():.3f}\")\n",
    "print(f\"The different scores obtained are: \\n{score}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Be aware that such training might involve a variation of the hyper-parameters\n",
    "of the model. When analyzing such model, you should not only look at the\n",
    "overall model performance but look at the hyper-parameters variations as\n",
    "well."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "jupytext": {
   "formats": "notebooks//ipynb,python_scripts//py:percent"
  },
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.10.6"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}
