{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Session 02: Unsupervised ML and dimension reduction\n",
    "\n",
    "Titouan Vayer, titouan.vayer@inria.fr\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import sklearn\n",
    "import numpy as np\n",
    "import pandas as pd\n",
    "import matplotlib.pyplot as plt "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Data Loading: IRIS\n",
    "We begin by loading the IRIS dataset, just like we did in session 1. It is worth noting that the IRIS dataset contains measurements of petal and sepal dimensions (length, width) from various samples of three iris varieties (Setosa, Versicolour, Virginica).\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.datasets import load_iris\n",
    "iris = load_iris()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "X = iris.data  \n",
    "y = iris.target "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Question 1: How many samples $n$  ? What is the dimension $d$ ?"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Format Conversion\n",
    "Later on, it will be convenient to utilize the functionalities of pandas that were used in Session 2 to visualize the dataset contents. The following conversion function (taken from https://stackoverflow.com/questions/38105539/how-to-convert-a-scikit-learn-dataset-to-a-pandas-dataset) can be useful to us again."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "def sklearn_to_df(sklearn_dataset):\n",
    "    df = pd.DataFrame(sklearn_dataset.data, columns=sklearn_dataset.feature_names)\n",
    "    df['target'] = pd.Series(sklearn_dataset.target)\n",
    "    return df"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We apply it to our dataset"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "df_iris = sklearn_to_df(iris)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "df_iris.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Question 2: Data Inspection\n",
    "Recalling yesterday's session, what do you suggest for a first overview of the data structure?"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Question 3: Which variable contains the class (iris type)? How many classes are there, and how many examples are there per class?"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Part 1 - Dimensionality Reduction\n",
    "\n",
    "We would like to visualize our data. The following function displays a dataset in 2D, where the color represents the class (Setosa, Versicolour, Virginica)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "def plot2d(X,y,label=['first','second']): # affichage d'un tableau nx2 de données X avec des couleurs déterminées par y de taille nx1\n",
    "    x_min, x_max = X[:, 0].min() - 0.5, X[:, 0].max() + 0.5\n",
    "    y_min, y_max = X[:, 1].min() - 0.5, X[:, 1].max() + 0.5\n",
    "    plt.figure(figsize=(4, 3))\n",
    "    plt.clf()\n",
    "    # Plot the training points\n",
    "    plt.scatter(X[:, 0], X[:, 1], c=y, cmap=plt.cm.Set1, edgecolor=\"k\")\n",
    "    plt.xlabel(label[0])\n",
    "    plt.ylabel(label[1])\n",
    "    plt.xlim(x_min, x_max)\n",
    "    plt.ylim(y_min, y_max)\n",
    "    plt.xticks(())\n",
    "    plt.yticks(())"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "labels = df_iris.columns; labels"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Question 4: which problem do we have for visualizing the data ? What do you suggest ? \n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Feature Selection\n",
    "What we have just done is dimensionality reduction through feature selection!\n",
    "To go further (not covered in this training), there are automated methods for feature selection\n",
    "See https://scikit-learn.org/stable/modules/feature_selection.html for more details"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.feature_selection import SelectKBest\n",
    "from sklearn.feature_selection import chi2\n",
    "X_new = SelectKBest(chi2, k=2).fit_transform(X, y)\n",
    "X_new.shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "plot2d(X_new,y)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Principal component analysis\n",
    "### First example "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.decomposition import PCA\n",
    "\n",
    "X_reduced = PCA(n_components=3).fit_transform(iris.data)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# To getter a better understanding of interaction of the dimensions\n",
    "# plot the first three PCA dimensions\n",
    "fig = plt.figure(1, figsize=(8, 6))\n",
    "ax = fig.add_subplot(111, projection=\"3d\", elev=-150, azim=110)\n",
    "\n",
    "ax.scatter(\n",
    "    X_reduced[:, 0],\n",
    "    X_reduced[:, 1],\n",
    "    X_reduced[:, 2],\n",
    "    c=y,\n",
    "    cmap=plt.cm.Set1,\n",
    "    edgecolor=\"k\",\n",
    "    s=40,\n",
    ")\n",
    "\n",
    "ax.set_title(\"First three PCA directions\")\n",
    "ax.set_xlabel(\"1st eigenvector\")\n",
    "ax.xaxis.set_ticklabels([])\n",
    "ax.set_ylabel(\"2nd eigenvector\")\n",
    "ax.yaxis.set_ticklabels([])\n",
    "ax.set_zlabel(\"3rd eigenvector\")\n",
    "ax.zaxis.set_ticklabels([])\n",
    "\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Question 5: Can you display the first 2 principal components? And compare them with the 2D plot obtained by selecting the first two features of X?"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Question 6: any comments ?"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Homework PCA in *high dim*\n",
    "https://scikit-learn.org/stable/auto_examples/decomposition/plot_faces_decomposition.html"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Non-linear dimension reduction with tSNE on IRIS\n",
    "https://github.com/olekscode/Examples-PCA-tSNE/blob/master/Python/Visualizing%20Iris%20Dataset%20using%20PCA%20and%20t-SNE.ipynb"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Kmeans"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We will now do a clustering algorithm: the K-means. We first load some packages"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.cluster import KMeans\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The data considered here are cities along with some informations. We first load the data using pandas."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "df = pd.read_csv('/Users/titouanvayer/Documents/Kmeanscountry/data/worldcities.csv', sep=',')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "df.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Q7: Keep only three columns: city of a given country (e.g. France) + latitude and longitude"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Q8: Convert the data frame to a numpy array and plot the corresponding points"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Q9: Do a simple k-means with k = 3, 4 different initialization and plot the corresponding centroids and the partition of the points\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Q9: Is it really a good partition according to you ? what are the limitations ? "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can also plot the Voronoi diagram (see course) for many value of $K$ as described below:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from scipy.spatial import Voronoi, voronoi_plot_2d\n",
    "\n",
    "size = 10\n",
    "fs = 18\n",
    "fig, ax = plt.subplots(2,2, figsize = (10,10))\n",
    "cmap = plt.cm.get_cmap('tab20')\n",
    "\n",
    "ax[0,0].scatter(X[:,0], X[:,1], color= cmap(0), s=size, alpha=0.8)\n",
    "ax[0,0].set_title('Data', fontsize = fs)\n",
    "ax[0,0].set_xlabel('Latitude', fontsize = fs-2)\n",
    "ax[0,0].set_ylabel('Longitude', fontsize = fs-2)\n",
    "\n",
    "m=0\n",
    "for K, axx in zip([5,10,15],[ax[0,1], ax[1,0], ax[1,1]]):\n",
    "\n",
    "    kmeans = KMeans(init=\"k-means++\", n_clusters=K, n_init=4, random_state=0)\n",
    "    kmeans.fit(X)\n",
    "    classes = kmeans.predict(X)\n",
    "    h = 0.02\n",
    "    x_min, x_max = X[:, 0].min() - 1, X[:, 0].max() + 1\n",
    "    y_min, y_max = X[:, 1].min() - 1, X[:, 1].max() + 1\n",
    "    xx, yy = np.meshgrid(np.arange(x_min, x_max, h), np.arange(y_min, y_max, h))\n",
    "    Z = kmeans.predict(np.c_[xx.ravel(), yy.ravel()])\n",
    "    centroids = kmeans.cluster_centers_\n",
    "    vor = Voronoi(centroids)\n",
    "    Z = Z.reshape(xx.shape)\n",
    "    # define color map \n",
    "    color_map = 255 * np.array(cmap.colors)[1:K+1]\n",
    "    data_3d = color_map[Z]\n",
    "    \n",
    "    fig = voronoi_plot_2d(vor, ax=axx, show_points=False,show_vertices=False, line_colors='k', \n",
    "                          line_alpha =0.4)\n",
    "    axx.scatter(X[:,0], X[:,1], color= [cmap(classes[i]+1) for i in range(X.shape[0])], s=size, alpha=0.8)\n",
    "    axx.scatter(\n",
    "        centroids[:, 0],\n",
    "        centroids[:, 1],\n",
    "        marker=\"x\",\n",
    "        s=50,\n",
    "        linewidths=3,\n",
    "        #color=[cmap(i+1) for i in range(centroids.shape[0])],\n",
    "        color=\"k\",\n",
    "        zorder=10,\n",
    "    )\n",
    "\n",
    "    axx.imshow(\n",
    "        data_3d,\n",
    "        interpolation=\"nearest\",\n",
    "        extent=(xx.min(), xx.max(), yy.min(), yy.max()),\n",
    "        aspect=\"auto\",\n",
    "        origin=\"lower\",\n",
    "        alpha = 0.1\n",
    "    )\n",
    "\n",
    "    axx.set_title('K-means with K = {}'.format(K), fontsize = fs)\n",
    "    axx.set_xlim([x_min,x_max])\n",
    "    axx.set_ylim([y_min,y_max])\n",
    "    axx.set_xticks([])\n",
    "    axx.set_yticks([])\n",
    "    \n",
    "    m+=1\n",
    "\n",
    "plt.tight_layout()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "EnvPose3D",
   "language": "python",
   "name": "envpose3d"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.9"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}
