{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Session 4: Model selection \n",
    "\n",
    "Titouan Vayer, titouan.vayer@inria.fr\n",
    "\n",
    "Material taken from the Euroscipy 2019 scikit-learn tutorial by Loic Esteve and Olivier Grisel."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Inline plots\n",
    "\n",
    "import pandas as pd"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Loading the adult census dataset"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We will use data from the \"Current Population adult_census\" from 1994 that we\n",
    "downloaded from [OpenML](http://openml.org/)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "adult_census = pd.read_csv(\"https://www.openml.org/data/get_csv/1595261/adult-census.csv\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "adult_census.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The column named **class** is our target variable (i.e., the variable which\n",
    "we want to predict). The two possible classes are `<= 50K` (low-revenue) and\n",
    "`> 50K` (high-revenue)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "target_column = 'class'\n",
    "adult_census[target_column].value_counts()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Note: classes are slighly imbalanced. Class imbalance happens often in\n",
    "practice and may need special techniques for machine learning. For example in\n",
    "a medical setting, if we are trying to predict whether patients will develop\n",
    "a rare disease, there will be a lot more sane patients than ill patients in\n",
    "the dataset."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "numerical_columns = ['age', 'education-num', 'capital-gain', 'capital-loss',\n",
    "                     'hours-per-week']\n",
    "categorical_columns = ['workclass', 'education', 'marital-status', 'occupation',\n",
    "                       'relationship', 'race', 'sex', 'native-country']\n",
    "all_columns = numerical_columns + categorical_columns + [target_column]\n",
    "\n",
    "adult_census = adult_census[all_columns]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "target_name = \"class\"\n",
    "target = adult_census[target_name]\n",
    "data = adult_census.drop(columns=[target_name])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "print(\n",
    "    f\"The dataset contains {data.shape[0]} samples and {data.shape[1]} \"\n",
    "    \"features\"\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "data.dtypes"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import numpy as np\n",
    "data_numeric = data.select_dtypes(include=np.number)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "data_numeric"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We're going to train various models. \n",
    "- choice of metric?\n",
    "- applied to what?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.model_selection import train_test_split\n",
    "\n",
    "data_train, data_test, target_train, target_test = train_test_split(\n",
    "    data_numeric, target, random_state=2408\n",
    ")\n",
    "\n",
    "print(\n",
    "    f\"The training dataset contains {data_train.shape[0]} samples and \"\n",
    "    f\"{data_train.shape[1]} features\"\n",
    ")\n",
    "print(\n",
    "    f\"The testing dataset contains {data_test.shape[0]} samples and \"\n",
    "    f\"{data_test.shape[1]} features\"\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We will build a linear classification model called \"Logistic Regression\". The\n",
    "`fit` method is called to train the model from the input and target data. Only\n",
    "the training data should be given for this purpose.\n",
    "\n",
    "In addition, when checking the time required to train the model and internally\n",
    "check the number of iterations done by the solver to find a solution."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.linear_model import LogisticRegression\n",
    "import time\n",
    "\n",
    "model = LogisticRegression(solver='lbfgs', max_iter=1000)\n",
    "start = time.time()\n",
    "model.fit(data_train, target_train)\n",
    "elapsed_time = time.time() - start\n",
    "\n",
    "print(\n",
    "    f\"The model {model.__class__.__name__} was trained in \"\n",
    "    f\"{elapsed_time:.3f} seconds for {model.n_iter_} iterations\"\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "target_predicted = model.predict(data_test)\n",
    "predictions = data_test.copy()\n",
    "predictions['predicted-class'] = target_predicted\n",
    "predictions['expected-class'] = target_test\n",
    "predictions['correct'] = target_predicted == target_test\n",
    "predictions.head(n=10)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "print(\n",
    "    f\"The test accuracy using a {model.__class__.__name__} is \"\n",
    "    f\"{model.score(data_test, target_test):.5f}\"\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Exercise 1\n",
    "\n",
    "- What would be the score of a model that always predicts `' >50K'`?\n",
    "- What would be the score of a model that always predicts `' <= 50K'`?\n",
    "- Is 81% or 82% accuracy a good score for this problem?\n",
    "\n",
    "Hint: You can compute the cross-validated of a [DummyClassifier](https://scikit-learn.org/stable/modules/model_evaluation.html#dummy-estimators) the performance of such baselines."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Look also at the cross-validation score"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "And for the dummy model"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Working with categorical variables\n",
    "\n",
    "As we have seen in the previous section, a numerical variable is a continuous\n",
    "quantity represented by a real or integer number. Those variables can be\n",
    "naturally handled by machine learning algorithms that typically composed of\n",
    "a sequence of arithmetic instructions such as additions and multiplications.\n",
    "\n",
    "By opposition, categorical variables have discrete values typically represented\n",
    "by string labels taken in a finite list of possible choices. For instance, the\n",
    "variable `native-country` in our dataset is a categorical variable because it\n",
    "encodes the data using a finite list of possible countries (along with the `?`\n",
    "marker when this information is missing):"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "data[\"native-country\"].value_counts()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "data_categorical = adult_census.select_dtypes(include=[object]).drop(columns=[\"class\"])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "data_categorical.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "print(f\"The datasets is composed of {data_categorical.shape[1]} features\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Encoding ordinal categories\n",
    "\n",
    "The most intuitive strategy is to encode each category with a number.\n",
    "The `OrdinalEncoder` will transform the data in such manner"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.preprocessing import OrdinalEncoder\n",
    "\n",
    "encoder = OrdinalEncoder()\n",
    "data_encoded = encoder.fit_transform(data_categorical)\n",
    "\n",
    "print(f\"The dataset encoded contains {data_encoded.shape[1]} features\")\n",
    "data_encoded[:5]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can see that all categories have been encoded for each feature\n",
    "independently. We can also notice that the number of features before and after\n",
    "the encoding is the same.\n",
    "\n",
    "However, one has to be careful when using this encoding strategy. Using this\n",
    "integer representation can lead the downstream models to make the assumption\n",
    "that the categories are ordered: 0 is smaller than 1 which is smaller than 2,\n",
    "etc.\n",
    "\n",
    "By default, `OrdinalEncoder` uses a lexicographical strategy to map string\n",
    "category labels to integers. This strategy is completely arbitrary and often be\n",
    "meaningless. For instance suppose the dataset has a categorical variable named\n",
    "\"size\" with categories such as \"S\", \"M\", \"L\", \"XL\". We would like the integer\n",
    "representation to respect the meaning of the sizes by mapping them to increasing\n",
    "integers such as 0, 1, 2, 3. However lexicographical strategy used by default\n",
    "would map the labels \"S\", \"M\", \"L\", \"XL\" to 2, 1, 0, 3.\n",
    "\n",
    "The `OrdinalEncoder` class accepts a \"categories\" constructor argument to pass\n",
    "an the correct ordering explicitly.\n",
    "\n",
    "If a categorical variable does not carry any meaningful order information then\n",
    "this encoding might be misleading to downstream statistical models and you might\n",
    "consider using one-hot encoding instead (see below).\n",
    "\n",
    "Note however that the impact a violation of this ordering assumption is really\n",
    "dependent on the downstream models (for instance linear models are much more\n",
    "sensitive than models built from a ensemble of decision trees).\n",
    "\n",
    "### Encoding nominal categories (without assuming any order)\n",
    "\n",
    "`OneHotEncoder` is an alternative encoder that can prevent the dowstream\n",
    "models to make a false assumption about the ordering of categories. For a\n",
    "given feature, it will create as many new columns as there are possible\n",
    "categories. For a given sample, the value of the column corresponding to the\n",
    "category will be set to `1` while all the columns of the other categories will\n",
    "be set to `0`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.preprocessing import OneHotEncoder\n",
    "\n",
    "encoder = OneHotEncoder(sparse=False)\n",
    "data_encoded = encoder.fit_transform(data_categorical)\n",
    "print(f\"The dataset encoded contains {data_encoded.shape[1]} features\")\n",
    "data_encoded"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "columns_encoded = encoder.get_feature_names_out(data_categorical.columns)\n",
    "pd.DataFrame(data_encoded, columns=columns_encoded).head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We will use the useful pipeline objects from sklearn:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.pipeline import make_pipeline\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "model = make_pipeline(\n",
    "    OneHotEncoder(handle_unknown='ignore'),\n",
    "    LogisticRegression(solver='lbfgs', max_iter=1000)\n",
    ")\n",
    "scores = cross_val_score(model, data_categorical, target, cv=10)\n",
    "print(f\"The different scores obtained are: \\n{scores}\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "print(f\"The accuracy is: {scores.mean():.3f} +/- {scores.std():.5f}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Using all the data at hand and specifying which columns to transform how"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "binary_encoding_columns = ['sex']\n",
    "one_hot_encoding_columns = ['workclass', 'education', 'marital-status',\n",
    "                            'occupation', 'relationship',\n",
    "                            'race', 'native-country']\n",
    "scaling_columns = ['age', 'education-num', 'hours-per-week',\n",
    "                   'capital-gain', 'capital-loss']\n",
    "\n",
    "from sklearn.compose import ColumnTransformer\n",
    "from sklearn.preprocessing import StandardScaler\n",
    "\n",
    "preprocessor = ColumnTransformer([\n",
    "    ('binary-encoder', OrdinalEncoder(), binary_encoding_columns),\n",
    "    ('one-hot-encoder', OneHotEncoder(handle_unknown='ignore'),\n",
    "     one_hot_encoding_columns),\n",
    "    ('standard-scaler', StandardScaler(), scaling_columns)\n",
    "])\n",
    "model = make_pipeline(\n",
    "    preprocessor,\n",
    "    LogisticRegression(solver='lbfgs', max_iter=1000)\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Pipelines, as all sklearn models, can be displayed!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "model"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "scores = cross_val_score(model, data, target, cv=5)\n",
    "print(f\"The different scores obtained are: \\n{scores}\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "print(f\"The accuracy is: {scores.mean():.3f} +- {scores.std():.3f}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The compound model has a higher predictive accuracy than the\n",
    "two models that used numerical and categorical variables in\n",
    "isolation."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## The issue of finding the best model parameters\n",
    "\n",
    "In the previous example, we created a `LogisticRegression` classifier using\n",
    "the default parameters by omitting setting explicitly these parameters.\n",
    "\n",
    "For this classifier, the parameter `C` governes the penalty; in other\n",
    "words, how much our model should \"trust\" (or fit) the training data.\n",
    "\n",
    "Therefore, the default value of `C` is never certified to give the best\n",
    "performing model.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "data_train, data_test, target_train, target_test = train_test_split(\n",
    "    data, target, random_state=0)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Make a quick experiment by changing the value of `C` and see the impact of this parameter on the model performance."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "\n",
    "## Finding the best model hyper-parameters via exhaustive parameters search\n",
    "\n",
    "We see that the parameter `C` as a significative impact on the model\n",
    "performance. This parameter should be tuned to get the best cross-validation\n",
    "score, so as to avoid over-fitting problems.\n",
    "\n",
    "In short, we will set the parameter, train our model on some data, and\n",
    "evaluate the model performance on some left out data. Ideally, we will select\n",
    "the parameter leading to the optimal performance on the testing set.\n",
    "Scikit-learn provides a `GridSearchCV` estimator which will handle the\n",
    "cross-validation and hyper-parameter search for us."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.model_selection import GridSearchCV\n",
    "\n",
    "model = make_pipeline(\n",
    "    preprocessor, LogisticRegression(max_iter=1000, solver='lbfgs')\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We will see that we need to provide the name of the parameter to be set.\n",
    "Thus, we can use the method `get_params()` to have the list of the parameters\n",
    "of the model which can set during the grid-search."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "print(\"The hyper-parameters are for a logistic regression model are:\")\n",
    "for param_name in LogisticRegression().get_params().keys():\n",
    "    print(param_name)\n",
    "\n",
    "print(\"The hyper-parameters are for the full-pipeline are:\")\n",
    "for param_name in model.get_params().keys():\n",
    "    print(param_name)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The parameter `'logisticregression__C'` is the parameter for which we would\n",
    "like different values. Let see how to use the `GridSearchCV` estimator for\n",
    "doing such search."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import time\n",
    "import numpy as np\n",
    "\n",
    "param_grid = {'logisticregression__C': (0.1, 1.0, 10.0)}\n",
    "model_grid_search = GridSearchCV(model, param_grid=param_grid, n_jobs=4, cv=5)\n",
    "start = time.time()\n",
    "model_grid_search.fit(data_train, target_train)\n",
    "elapsed_time = time.time() - start\n",
    "print(\n",
    "    f\"The accuracy score using a {model_grid_search.__class__.__name__} is \"\n",
    "    f\"{model_grid_search.score(data_test, target_test):.2f} in \"\n",
    "    f\"{elapsed_time:.3f} seconds\"\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The `GridSearchCV` estimator takes a `param_grid` parameter which defines\n",
    "all possible parameters combination. Once the grid-search fitted, it can be\n",
    "used as any other predictor by calling `predict` and `predict_proba`.\n",
    "Internally, it will use the model with the best parameters found during\n",
    "`fit`. You can know about these parameters by looking at the `best_params_`\n",
    "attribute."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "print(f\"The best set of parameters is: {model_grid_search.best_params_}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "With the `GridSearchCV` estimator, the parameters need to be specified\n",
    "explicitely. Instead, one could randomly generate (following a specific\n",
    "distribution) the parameter candidates. The `RandomSearchCV` allows for such\n",
    "stochastic search. It is used similarly to the `GridSearchCV` but the\n",
    "sampling distributions need to be specified instead of the parameter values."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from scipy.stats import uniform\n",
    "from sklearn.model_selection import RandomizedSearchCV\n",
    "\n",
    "param_distributions = {'logisticregression__C': uniform(loc=50, scale=100)}\n",
    "model_grid_search = RandomizedSearchCV(\n",
    "    model, param_distributions=param_distributions, n_iter=3, n_jobs=4, cv=5\n",
    ")\n",
    "model_grid_search.fit(data_train, target_train)\n",
    "print(\n",
    "    f\"The accuracy score using a {model_grid_search.__class__.__name__} is \"\n",
    "    f\"{model_grid_search.score(data_test, target_test):.2f}\"\n",
    ")\n",
    "print(f\"The best set of parameters is: {model_grid_search.best_params_}\")\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "EnvPose3D",
   "language": "python",
   "name": "envpose3d"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.9"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}
