{
 "cells": [
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "CPython 3.5.5\n",
      "IPython 6.2.1\n",
      "\n",
      "numpy 1.14.1\n",
      "scipy 1.0.0\n",
      "sklearn 0.19.1\n",
      "pandas 0.22.0\n",
      "matplotlib 2.2.2\n"
     ]
    }
   ],
   "source": [
    "%load_ext watermark\n",
    "%watermark -v -p numpy,scipy,sklearn,pandas,matplotlib"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**6장 – 결정 트리**"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "_이 노트북은 6장에 있는 모든 샘플 코드와 연습문제 해답을 가지고 있습니다._"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 설정"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "파이썬 2와 3을 모두 지원합니다. 공통 모듈을 임포트하고 맷플롯립 그림이 노트북 안에 포함되도록 설정하고 생성한 그림을 저장하기 위한 함수를 준비합니다:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "# 파이썬 2와 파이썬 3 지원\n",
    "from __future__ import division, print_function, unicode_literals\n",
    "\n",
    "# 공통\n",
    "import numpy as np\n",
    "import os\n",
    "\n",
    "# 일관된 출력을 위해 유사난수 초기화\n",
    "np.random.seed(42)\n",
    "\n",
    "# 맷플롯립 설정\n",
    "%matplotlib inline\n",
    "import matplotlib\n",
    "import matplotlib.pyplot as plt\n",
    "plt.rcParams['axes.labelsize'] = 14\n",
    "plt.rcParams['xtick.labelsize'] = 12\n",
    "plt.rcParams['ytick.labelsize'] = 12\n",
    "\n",
    "# 한글출력\n",
    "matplotlib.rc('font', family='NanumBarunGothic')\n",
    "matplotlib.rcParams['axes.unicode_minus'] = False\n",
    "\n",
    "# 그림을 저장할 폴드\n",
    "PROJECT_ROOT_DIR = \".\"\n",
    "CHAPTER_ID = \"decision_trees\"\n",
    "\n",
    "def image_path(fig_id):\n",
    "    return os.path.join(PROJECT_ROOT_DIR, \"images\", CHAPTER_ID, fig_id)\n",
    "\n",
    "def save_fig(fig_id, tight_layout=True):\n",
    "    if tight_layout:\n",
    "        plt.tight_layout()\n",
    "    plt.savefig(image_path(fig_id) + \".png\", format='png', dpi=300)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 훈련과 시각화"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "DecisionTreeClassifier(class_weight=None, criterion='gini', max_depth=2,\n",
       "            max_features=None, max_leaf_nodes=None,\n",
       "            min_impurity_decrease=0.0, min_impurity_split=None,\n",
       "            min_samples_leaf=1, min_samples_split=2,\n",
       "            min_weight_fraction_leaf=0.0, presort=False, random_state=42,\n",
       "            splitter='best')"
      ]
     },
     "execution_count": 3,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from sklearn.datasets import load_iris\n",
    "from sklearn.tree import DecisionTreeClassifier\n",
    "\n",
    "iris = load_iris()\n",
    "X = iris.data[:, 2:] # petal length and width\n",
    "y = iris.target\n",
    "\n",
    "tree_clf = DecisionTreeClassifier(max_depth=2, random_state=42)\n",
    "tree_clf.fit(X, y)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.tree import export_graphviz\n",
    "\n",
    "export_graphviz(\n",
    "        tree_clf,\n",
    "        out_file=image_path(\"iris_tree.dot\"),\n",
    "        feature_names=[\"꽃잎 길이 (cm)\", \"꽃잎 너비 (cm)\"],\n",
    "        class_names=iris.target_names,\n",
    "        rounded=True,\n",
    "        filled=True\n",
    "    )"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/svg+xml": [
       "<?xml version=\"1.0\" encoding=\"UTF-8\" standalone=\"no\"?>\n",
       "<!DOCTYPE svg PUBLIC \"-//W3C//DTD SVG 1.1//EN\"\n",
       " \"http://www.w3.org/Graphics/SVG/1.1/DTD/svg11.dtd\">\n",
       "<!-- Generated by graphviz version 2.38.0 (20140413.2041)\n",
       " -->\n",
       "<!-- Title: Tree Pages: 1 -->\n",
       "<svg width=\"346pt\" height=\"314pt\"\n",
       " viewBox=\"0.00 0.00 346.00 314.00\" xmlns=\"http://www.w3.org/2000/svg\" xmlns:xlink=\"http://www.w3.org/1999/xlink\">\n",
       "<g id=\"graph0\" class=\"graph\" transform=\"scale(1 1) rotate(0) translate(4 310)\">\n",
       "<title>Tree</title>\n",
       "<polygon fill=\"white\" stroke=\"none\" points=\"-4,4 -4,-310 342,-310 342,4 -4,4\"/>\n",
       "<!-- 0 -->\n",
       "<g id=\"node1\" class=\"node\"><title>0</title>\n",
       "<path fill=\"none\" stroke=\"black\" d=\"M199,-306C199,-306 70,-306 70,-306 64,-306 58,-300 58,-294 58,-294 58,-235 58,-235 58,-229 64,-223 70,-223 70,-223 199,-223 199,-223 205,-223 211,-229 211,-235 211,-235 211,-294 211,-294 211,-300 205,-306 199,-306\"/>\n",
       "<text text-anchor=\"middle\" x=\"134.5\" y=\"-290.8\" font-family=\"Helvetica,sans-Serif\" font-size=\"14.00\">꽃잎 길이 (cm) &lt;= 2.45</text>\n",
       "<text text-anchor=\"middle\" x=\"134.5\" y=\"-275.8\" font-family=\"Helvetica,sans-Serif\" font-size=\"14.00\">gini = 0.667</text>\n",
       "<text text-anchor=\"middle\" x=\"134.5\" y=\"-260.8\" font-family=\"Helvetica,sans-Serif\" font-size=\"14.00\">samples = 150</text>\n",
       "<text text-anchor=\"middle\" x=\"134.5\" y=\"-245.8\" font-family=\"Helvetica,sans-Serif\" font-size=\"14.00\">value = [50, 50, 50]</text>\n",
       "<text text-anchor=\"middle\" x=\"134.5\" y=\"-230.8\" font-family=\"Helvetica,sans-Serif\" font-size=\"14.00\">class = setosa</text>\n",
       "</g>\n",
       "<!-- 1 -->\n",
       "<g id=\"node2\" class=\"node\"><title>1</title>\n",
       "<path fill=\"#e58139\" stroke=\"black\" d=\"M105,-179.5C105,-179.5 12,-179.5 12,-179.5 6,-179.5 -7.10543e-15,-173.5 -7.10543e-15,-167.5 -7.10543e-15,-167.5 -7.10543e-15,-123.5 -7.10543e-15,-123.5 -7.10543e-15,-117.5 6,-111.5 12,-111.5 12,-111.5 105,-111.5 105,-111.5 111,-111.5 117,-117.5 117,-123.5 117,-123.5 117,-167.5 117,-167.5 117,-173.5 111,-179.5 105,-179.5\"/>\n",
       "<text text-anchor=\"middle\" x=\"58.5\" y=\"-164.3\" font-family=\"Helvetica,sans-Serif\" font-size=\"14.00\">gini = 0.0</text>\n",
       "<text text-anchor=\"middle\" x=\"58.5\" y=\"-149.3\" font-family=\"Helvetica,sans-Serif\" font-size=\"14.00\">samples = 50</text>\n",
       "<text text-anchor=\"middle\" x=\"58.5\" y=\"-134.3\" font-family=\"Helvetica,sans-Serif\" font-size=\"14.00\">value = [50, 0, 0]</text>\n",
       "<text text-anchor=\"middle\" x=\"58.5\" y=\"-119.3\" font-family=\"Helvetica,sans-Serif\" font-size=\"14.00\">class = setosa</text>\n",
       "</g>\n",
       "<!-- 0&#45;&gt;1 -->\n",
       "<g id=\"edge1\" class=\"edge\"><title>0&#45;&gt;1</title>\n",
       "<path fill=\"none\" stroke=\"black\" d=\"M108.132,-222.907C100.821,-211.652 92.8743,-199.418 85.5263,-188.106\"/>\n",
       "<polygon fill=\"black\" stroke=\"black\" points=\"88.4269,-186.146 80.0443,-179.667 82.5566,-189.96 88.4269,-186.146\"/>\n",
       "<text text-anchor=\"middle\" x=\"74.8494\" y=\"-200.421\" font-family=\"Helvetica,sans-Serif\" font-size=\"14.00\">True</text>\n",
       "</g>\n",
       "<!-- 2 -->\n",
       "<g id=\"node3\" class=\"node\"><title>2</title>\n",
       "<path fill=\"none\" stroke=\"black\" d=\"M276,-187C276,-187 147,-187 147,-187 141,-187 135,-181 135,-175 135,-175 135,-116 135,-116 135,-110 141,-104 147,-104 147,-104 276,-104 276,-104 282,-104 288,-110 288,-116 288,-116 288,-175 288,-175 288,-181 282,-187 276,-187\"/>\n",
       "<text text-anchor=\"middle\" x=\"211.5\" y=\"-171.8\" font-family=\"Helvetica,sans-Serif\" font-size=\"14.00\">꽃잎 너비 (cm) &lt;= 1.75</text>\n",
       "<text text-anchor=\"middle\" x=\"211.5\" y=\"-156.8\" font-family=\"Helvetica,sans-Serif\" font-size=\"14.00\">gini = 0.5</text>\n",
       "<text text-anchor=\"middle\" x=\"211.5\" y=\"-141.8\" font-family=\"Helvetica,sans-Serif\" font-size=\"14.00\">samples = 100</text>\n",
       "<text text-anchor=\"middle\" x=\"211.5\" y=\"-126.8\" font-family=\"Helvetica,sans-Serif\" font-size=\"14.00\">value = [0, 50, 50]</text>\n",
       "<text text-anchor=\"middle\" x=\"211.5\" y=\"-111.8\" font-family=\"Helvetica,sans-Serif\" font-size=\"14.00\">class = versicolor</text>\n",
       "</g>\n",
       "<!-- 0&#45;&gt;2 -->\n",
       "<g id=\"edge2\" class=\"edge\"><title>0&#45;&gt;2</title>\n",
       "<path fill=\"none\" stroke=\"black\" d=\"M161.215,-222.907C167.008,-214.105 173.195,-204.703 179.178,-195.612\"/>\n",
       "<polygon fill=\"black\" stroke=\"black\" points=\"182.259,-197.298 184.832,-187.021 176.411,-193.45 182.259,-197.298\"/>\n",
       "<text text-anchor=\"middle\" x=\"189.881\" y=\"-207.806\" font-family=\"Helvetica,sans-Serif\" font-size=\"14.00\">False</text>\n",
       "</g>\n",
       "<!-- 3 -->\n",
       "<g id=\"node4\" class=\"node\"><title>3</title>\n",
       "<path fill=\"#39e581\" fill-opacity=\"0.898039\" stroke=\"black\" d=\"M191,-68C191,-68 94,-68 94,-68 88,-68 82,-62 82,-56 82,-56 82,-12 82,-12 82,-6 88,-0 94,-0 94,-0 191,-0 191,-0 197,-0 203,-6 203,-12 203,-12 203,-56 203,-56 203,-62 197,-68 191,-68\"/>\n",
       "<text text-anchor=\"middle\" x=\"142.5\" y=\"-52.8\" font-family=\"Helvetica,sans-Serif\" font-size=\"14.00\">gini = 0.168</text>\n",
       "<text text-anchor=\"middle\" x=\"142.5\" y=\"-37.8\" font-family=\"Helvetica,sans-Serif\" font-size=\"14.00\">samples = 54</text>\n",
       "<text text-anchor=\"middle\" x=\"142.5\" y=\"-22.8\" font-family=\"Helvetica,sans-Serif\" font-size=\"14.00\">value = [0, 49, 5]</text>\n",
       "<text text-anchor=\"middle\" x=\"142.5\" y=\"-7.8\" font-family=\"Helvetica,sans-Serif\" font-size=\"14.00\">class = versicolor</text>\n",
       "</g>\n",
       "<!-- 2&#45;&gt;3 -->\n",
       "<g id=\"edge3\" class=\"edge\"><title>2&#45;&gt;3</title>\n",
       "<path fill=\"none\" stroke=\"black\" d=\"M185.807,-103.726C180.289,-94.9703 174.45,-85.7032 168.906,-76.9051\"/>\n",
       "<polygon fill=\"black\" stroke=\"black\" points=\"171.776,-74.8941 163.483,-68.2996 165.853,-78.6259 171.776,-74.8941\"/>\n",
       "</g>\n",
       "<!-- 4 -->\n",
       "<g id=\"node5\" class=\"node\"><title>4</title>\n",
       "<path fill=\"#8139e5\" fill-opacity=\"0.976471\" stroke=\"black\" d=\"M326,-68C326,-68 233,-68 233,-68 227,-68 221,-62 221,-56 221,-56 221,-12 221,-12 221,-6 227,-0 233,-0 233,-0 326,-0 326,-0 332,-0 338,-6 338,-12 338,-12 338,-56 338,-56 338,-62 332,-68 326,-68\"/>\n",
       "<text text-anchor=\"middle\" x=\"279.5\" y=\"-52.8\" font-family=\"Helvetica,sans-Serif\" font-size=\"14.00\">gini = 0.043</text>\n",
       "<text text-anchor=\"middle\" x=\"279.5\" y=\"-37.8\" font-family=\"Helvetica,sans-Serif\" font-size=\"14.00\">samples = 46</text>\n",
       "<text text-anchor=\"middle\" x=\"279.5\" y=\"-22.8\" font-family=\"Helvetica,sans-Serif\" font-size=\"14.00\">value = [0, 1, 45]</text>\n",
       "<text text-anchor=\"middle\" x=\"279.5\" y=\"-7.8\" font-family=\"Helvetica,sans-Serif\" font-size=\"14.00\">class = virginica</text>\n",
       "</g>\n",
       "<!-- 2&#45;&gt;4 -->\n",
       "<g id=\"edge4\" class=\"edge\"><title>2&#45;&gt;4</title>\n",
       "<path fill=\"none\" stroke=\"black\" d=\"M236.821,-103.726C242.258,-94.9703 248.013,-85.7032 253.477,-76.9051\"/>\n",
       "<polygon fill=\"black\" stroke=\"black\" points=\"256.519,-78.6413 258.821,-68.2996 250.572,-74.9484 256.519,-78.6413\"/>\n",
       "</g>\n",
       "</g>\n",
       "</svg>\n"
      ],
      "text/plain": [
       "<graphviz.files.Source at 0x7fcb1ea5c5c0>"
      ]
     },
     "execution_count": 5,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "import graphviz\n",
    "with open(\"images/decision_trees/iris_tree.dot\") as f:\n",
    "    dot_graph = f.read()\n",
    "dot = graphviz.Source(dot_graph)\n",
    "dot.format = 'png'\n",
    "dot.render(filename='iris_tree', directory='images/decision_trees', cleanup=True)\n",
    "dot"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "[encoded data, 18496 characters]\n",
      "text/plain": [
       "<Figure size 576x288 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "from matplotlib.colors import ListedColormap\n",
    "\n",
    "def plot_decision_boundary(clf, X, y, axes=[0, 7.5, 0, 3], iris=True, legend=False, plot_training=True):\n",
    "    x1s = np.linspace(axes[0], axes[1], 100)\n",
    "    x2s = np.linspace(axes[2], axes[3], 100)\n",
    "    x1, x2 = np.meshgrid(x1s, x2s)\n",
    "    X_new = np.c_[x1.ravel(), x2.ravel()]\n",
    "    y_pred = clf.predict(X_new).reshape(x1.shape)\n",
    "    custom_cmap = ListedColormap(['#fafab0','#9898ff','#a0faa0'])\n",
    "    plt.contourf(x1, x2, y_pred, alpha=0.3, cmap=custom_cmap)\n",
    "    if not iris:\n",
    "        custom_cmap2 = ListedColormap(['#7d7d58','#4c4c7f','#507d50'])\n",
    "        plt.contour(x1, x2, y_pred, cmap=custom_cmap2, alpha=0.8)\n",
    "    if plot_training:\n",
    "        plt.plot(X[:, 0][y==0], X[:, 1][y==0], \"yo\", label=\"Iris-Setosa\")\n",
    "        plt.plot(X[:, 0][y==1], X[:, 1][y==1], \"bs\", label=\"Iris-Versicolor\")\n",
    "        plt.plot(X[:, 0][y==2], X[:, 1][y==2], \"g^\", label=\"Iris-Virginica\")\n",
    "        plt.axis(axes)\n",
    "    if iris:\n",
    "        plt.xlabel(\"꽃잎 길이\", fontsize=14)\n",
    "        plt.ylabel(\"꽃잎 너비\", fontsize=14)\n",
    "    else:\n",
    "        plt.xlabel(r\"$x_1$\", fontsize=18)\n",
    "        plt.ylabel(r\"$x_2$\", fontsize=18, rotation=0)\n",
    "    if legend:\n",
    "        plt.legend(loc=\"lower right\", fontsize=14)\n",
    "\n",
    "plt.figure(figsize=(8, 4))\n",
    "plot_decision_boundary(tree_clf, X, y)\n",
    "plt.plot([2.45, 2.45], [0, 3], \"k-\", linewidth=2)\n",
    "plt.plot([2.45, 7.5], [1.75, 1.75], \"k--\", linewidth=2)\n",
    "plt.plot([4.95, 4.95], [0, 1.75], \"k:\", linewidth=2)\n",
    "plt.plot([4.85, 4.85], [1.75, 3], \"k:\", linewidth=2)\n",
    "plt.text(1.40, 1.0, \"깊이=0\", fontsize=15)\n",
    "plt.text(3.2, 1.80, \"깊이=1\", fontsize=13)\n",
    "plt.text(4.05, 0.5, \"(깊이=2)\", fontsize=11)\n",
    "\n",
    "save_fig(\"decision_tree_decision_boundaries_plot\")\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 클래스와 클래스 확률을 예측하기"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([[0.        , 0.90740741, 0.09259259]])"
      ]
     },
     "execution_count": 7,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "tree_clf.predict_proba([[5, 1.5]])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([1])"
      ]
     },
     "execution_count": 8,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "tree_clf.predict([[5, 1.5]])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 훈련 세트에 민감함"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([[4.8, 1.8]])"
      ]
     },
     "execution_count": 9,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "X[(X[:, 1]==X[:, 1][y==1].max()) & (y==1)] # 가장 너비가 큰 Iris-Versicolor"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "DecisionTreeClassifier(class_weight=None, criterion='gini', max_depth=2,\n",
       "            max_features=None, max_leaf_nodes=None,\n",
       "            min_impurity_decrease=0.0, min_impurity_split=None,\n",
       "            min_samples_leaf=1, min_samples_split=2,\n",
       "            min_weight_fraction_leaf=0.0, presort=False, random_state=40,\n",
       "            splitter='best')"
      ]
     },
     "execution_count": 10,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "not_widest_versicolor = (X[:, 1]!=1.8) | (y==2)\n",
    "X_tweaked = X[not_widest_versicolor]\n",
    "y_tweaked = y[not_widest_versicolor]\n",
    "\n",
    "tree_clf_tweaked = DecisionTreeClassifier(max_depth=2, random_state=40)\n",
    "tree_clf_tweaked.fit(X_tweaked, y_tweaked)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "[encoded data, 14848 characters]\n",
      "text/plain": [
       "<Figure size 576x288 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "plt.figure(figsize=(8, 4))\n",
    "plot_decision_boundary(tree_clf_tweaked, X_tweaked, y_tweaked, legend=False)\n",
    "plt.plot([0, 7.5], [0.8, 0.8], \"k-\", linewidth=2)\n",
    "plt.plot([0, 7.5], [1.75, 1.75], \"k--\", linewidth=2)\n",
    "plt.text(1.0, 0.9, \"깊이=0\", fontsize=15)\n",
    "plt.text(1.0, 1.80, \"깊이=1\", fontsize=13)\n",
    "\n",
    "save_fig(\"decision_tree_instability_plot\")\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "[encoded data, 21600 characters]\n",
      "text/plain": [
       "<Figure size 792x288 with 2 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "from sklearn.datasets import make_moons\n",
    "Xm, ym = make_moons(n_samples=100, noise=0.25, random_state=53)\n",
    "\n",
    "deep_tree_clf1 = DecisionTreeClassifier(random_state=42)\n",
    "deep_tree_clf2 = DecisionTreeClassifier(min_samples_leaf=4, random_state=42)\n",
    "deep_tree_clf1.fit(Xm, ym)\n",
    "deep_tree_clf2.fit(Xm, ym)\n",
    "\n",
    "plt.figure(figsize=(11, 4))\n",
    "plt.subplot(121)\n",
    "plot_decision_boundary(deep_tree_clf1, Xm, ym, axes=[-1.5, 2.5, -1, 1.5], iris=False)\n",
    "plt.title(\"규제 없음\", fontsize=16)\n",
    "plt.subplot(122)\n",
    "plot_decision_boundary(deep_tree_clf2, Xm, ym, axes=[-1.5, 2.5, -1, 1.5], iris=False)\n",
    "plt.title(\"min_samples_leaf = {}\".format(deep_tree_clf2.min_samples_leaf), fontsize=14)\n",
    "\n",
    "save_fig(\"min_samples_leaf_plot\")\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "[encoded data, 10776 characters]\n",
      "text/plain": [
       "<Figure size 576x216 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "angle = np.pi / 180 * 20\n",
    "rotation_matrix = np.array([[np.cos(angle), -np.sin(angle)], [np.sin(angle), np.cos(angle)]])\n",
    "Xr = X.dot(rotation_matrix)\n",
    "\n",
    "tree_clf_r = DecisionTreeClassifier(random_state=42)\n",
    "tree_clf_r.fit(Xr, y)\n",
    "\n",
    "plt.figure(figsize=(8, 3))\n",
    "plot_decision_boundary(tree_clf_r, Xr, y, axes=[0.5, 7.5, -1.0, 1], iris=False)\n",
    "\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "[encoded data, 19656 characters]\n",
      "text/plain": [
       "<Figure size 792x288 with 2 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "np.random.seed(6)\n",
    "Xs = np.random.rand(100, 2) - 0.5\n",
    "ys = (Xs[:, 0] > 0).astype(np.float32) * 2\n",
    "\n",
    "angle = np.pi / 4\n",
    "rotation_matrix = np.array([[np.cos(angle), -np.sin(angle)], [np.sin(angle), np.cos(angle)]])\n",
    "Xsr = Xs.dot(rotation_matrix)\n",
    "\n",
    "tree_clf_s = DecisionTreeClassifier(random_state=42)\n",
    "tree_clf_s.fit(Xs, ys)\n",
    "tree_clf_sr = DecisionTreeClassifier(random_state=42)\n",
    "tree_clf_sr.fit(Xsr, ys)\n",
    "\n",
    "plt.figure(figsize=(11, 4))\n",
    "plt.subplot(121)\n",
    "plot_decision_boundary(tree_clf_s, Xs, ys, axes=[-0.7, 0.7, -0.7, 0.7], iris=False)\n",
    "plt.subplot(122)\n",
    "plot_decision_boundary(tree_clf_sr, Xsr, ys, axes=[-0.7, 0.7, -0.7, 0.7], iris=False)\n",
    "\n",
    "save_fig(\"sensitivity_to_rotation_plot\")\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 회귀 트리"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [],
   "source": [
    "# 2차식으로 만든 데이터셋 + 잡음\n",
    "np.random.seed(42)\n",
    "m = 200\n",
    "X = np.random.rand(m, 1)\n",
    "y = 4 * (X - 0.5) ** 2\n",
    "y = y + np.random.randn(m, 1) / 10"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "DecisionTreeRegressor(criterion='mse', max_depth=2, max_features=None,\n",
       "           max_leaf_nodes=None, min_impurity_decrease=0.0,\n",
       "           min_impurity_split=None, min_samples_leaf=1,\n",
       "           min_samples_split=2, min_weight_fraction_leaf=0.0,\n",
       "           presort=False, random_state=42, splitter='best')"
      ]
     },
     "execution_count": 16,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from sklearn.tree import DecisionTreeRegressor\n",
    "\n",
    "tree_reg = DecisionTreeRegressor(max_depth=2, random_state=42)\n",
    "tree_reg.fit(X, y)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "[encoded data, 36612 characters]\n",
      "text/plain": [
       "<Figure size 792x288 with 2 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "from sklearn.tree import DecisionTreeRegressor\n",
    "\n",
    "tree_reg1 = DecisionTreeRegressor(random_state=42, max_depth=2)\n",
    "tree_reg2 = DecisionTreeRegressor(random_state=42, max_depth=3)\n",
    "tree_reg1.fit(X, y)\n",
    "tree_reg2.fit(X, y)\n",
    "\n",
    "def plot_regression_predictions(tree_reg, X, y, axes=[0, 1, -0.2, 1], ylabel=\"$y$\"):\n",
    "    x1 = np.linspace(axes[0], axes[1], 500).reshape(-1, 1)\n",
    "    y_pred = tree_reg.predict(x1)\n",
    "    plt.axis(axes)\n",
    "    plt.xlabel(\"$x_1$\", fontsize=18)\n",
    "    if ylabel:\n",
    "        plt.ylabel(ylabel, fontsize=18, rotation=0)\n",
    "    plt.plot(X, y, \"b.\")\n",
    "    plt.plot(x1, y_pred, \"r.-\", linewidth=2, label=r\"$\\hat{y}$\")\n",
    "\n",
    "plt.figure(figsize=(11, 4))\n",
    "plt.subplot(121)\n",
    "plot_regression_predictions(tree_reg1, X, y)\n",
    "for split, style in ((0.1973, \"k-\"), (0.0917, \"k--\"), (0.7718, \"k--\")):\n",
    "    plt.plot([split, split], [-0.2, 1], style, linewidth=2)\n",
    "plt.text(0.21, 0.65, \"깊이=0\", fontsize=15)\n",
    "plt.text(0.01, 0.2, \"깊이=1\", fontsize=13)\n",
    "plt.text(0.65, 0.8, \"깊이=1\", fontsize=13)\n",
    "plt.legend(loc=\"upper center\", fontsize=18)\n",
    "plt.title(\"max_depth=2\", fontsize=14)\n",
    "\n",
    "plt.subplot(122)\n",
    "plot_regression_predictions(tree_reg2, X, y, ylabel=None)\n",
    "for split, style in ((0.1973, \"k-\"), (0.0917, \"k--\"), (0.7718, \"k--\")):\n",
    "    plt.plot([split, split], [-0.2, 1], style, linewidth=2)\n",
    "for split in (0.0458, 0.1298, 0.2873, 0.9040):\n",
    "    plt.plot([split, split], [-0.2, 1], \"k:\", linewidth=1)\n",
    "plt.text(0.3, 0.5, \"깊이=2\", fontsize=13)\n",
    "plt.title(\"max_depth=3\", fontsize=14)\n",
    "\n",
    "save_fig(\"tree_regression_plot\")\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [],
   "source": [
    "export_graphviz(\n",
    "        tree_reg1,\n",
    "        out_file=image_path(\"regression_tree.dot\"),\n",
    "        feature_names=[\"x1\"],\n",
    "        rounded=True,\n",
    "        filled=True\n",
    "    )"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/svg+xml": [
       "<?xml version=\"1.0\" encoding=\"UTF-8\" standalone=\"no\"?>\n",
       "<!DOCTYPE svg PUBLIC \"-//W3C//DTD SVG 1.1//EN\"\n",
       " \"http://www.w3.org/Graphics/SVG/1.1/DTD/svg11.dtd\">\n",
       "<!-- Generated by graphviz version 2.38.0 (20140413.2041)\n",
       " -->\n",
       "<!-- Title: Tree Pages: 1 -->\n",
       "<svg width=\"462pt\" height=\"269pt\"\n",
       " viewBox=\"0.00 0.00 462.00 269.00\" xmlns=\"http://www.w3.org/2000/svg\" xmlns:xlink=\"http://www.w3.org/1999/xlink\">\n",
       "<g id=\"graph0\" class=\"graph\" transform=\"scale(1 1) rotate(0) translate(4 265)\">\n",
       "<title>Tree</title>\n",
       "<polygon fill=\"white\" stroke=\"none\" points=\"-4,4 -4,-265 458,-265 458,4 -4,4\"/>\n",
       "<!-- 0 -->\n",
       "<g id=\"node1\" class=\"node\"><title>0</title>\n",
       "<path fill=\"#e58139\" fill-opacity=\"0.325490\" stroke=\"black\" d=\"M266,-261C266,-261 184,-261 184,-261 178,-261 172,-255 172,-249 172,-249 172,-205 172,-205 172,-199 178,-193 184,-193 184,-193 266,-193 266,-193 272,-193 278,-199 278,-205 278,-205 278,-249 278,-249 278,-255 272,-261 266,-261\"/>\n",
       "<text text-anchor=\"middle\" x=\"225\" y=\"-245.8\" font-family=\"Helvetica,sans-Serif\" font-size=\"14.00\">x1 &lt;= 0.197</text>\n",
       "<text text-anchor=\"middle\" x=\"225\" y=\"-230.8\" font-family=\"Helvetica,sans-Serif\" font-size=\"14.00\">mse = 0.098</text>\n",
       "<text text-anchor=\"middle\" x=\"225\" y=\"-215.8\" font-family=\"Helvetica,sans-Serif\" font-size=\"14.00\">samples = 200</text>\n",
       "<text text-anchor=\"middle\" x=\"225\" y=\"-200.8\" font-family=\"Helvetica,sans-Serif\" font-size=\"14.00\">value = 0.354</text>\n",
       "</g>\n",
       "<!-- 1 -->\n",
       "<g id=\"node2\" class=\"node\"><title>1</title>\n",
       "<path fill=\"#e58139\" fill-opacity=\"0.780392\" stroke=\"black\" d=\"M202,-157C202,-157 128,-157 128,-157 122,-157 116,-151 116,-145 116,-145 116,-101 116,-101 116,-95 122,-89 128,-89 128,-89 202,-89 202,-89 208,-89 214,-95 214,-101 214,-101 214,-145 214,-145 214,-151 208,-157 202,-157\"/>\n",
       "<text text-anchor=\"middle\" x=\"165\" y=\"-141.8\" font-family=\"Helvetica,sans-Serif\" font-size=\"14.00\">x1 &lt;= 0.092</text>\n",
       "<text text-anchor=\"middle\" x=\"165\" y=\"-126.8\" font-family=\"Helvetica,sans-Serif\" font-size=\"14.00\">mse = 0.038</text>\n",
       "<text text-anchor=\"middle\" x=\"165\" y=\"-111.8\" font-family=\"Helvetica,sans-Serif\" font-size=\"14.00\">samples = 44</text>\n",
       "<text text-anchor=\"middle\" x=\"165\" y=\"-96.8\" font-family=\"Helvetica,sans-Serif\" font-size=\"14.00\">value = 0.689</text>\n",
       "</g>\n",
       "<!-- 0&#45;&gt;1 -->\n",
       "<g id=\"edge1\" class=\"edge\"><title>0&#45;&gt;1</title>\n",
       "<path fill=\"none\" stroke=\"black\" d=\"M205.52,-192.884C200.49,-184.332 195.008,-175.013 189.748,-166.072\"/>\n",
       "<polygon fill=\"black\" stroke=\"black\" points=\"192.675,-164.144 184.588,-157.299 186.641,-167.693 192.675,-164.144\"/>\n",
       "<text text-anchor=\"middle\" x=\"178.314\" y=\"-177.799\" font-family=\"Helvetica,sans-Serif\" font-size=\"14.00\">True</text>\n",
       "</g>\n",
       "<!-- 4 -->\n",
       "<g id=\"node5\" class=\"node\"><title>4</title>\n",
       "<path fill=\"#e58139\" fill-opacity=\"0.200000\" stroke=\"black\" d=\"M326,-157C326,-157 244,-157 244,-157 238,-157 232,-151 232,-145 232,-145 232,-101 232,-101 232,-95 238,-89 244,-89 244,-89 326,-89 326,-89 332,-89 338,-95 338,-101 338,-101 338,-145 338,-145 338,-151 332,-157 326,-157\"/>\n",
       "<text text-anchor=\"middle\" x=\"285\" y=\"-141.8\" font-family=\"Helvetica,sans-Serif\" font-size=\"14.00\">x1 &lt;= 0.772</text>\n",
       "<text text-anchor=\"middle\" x=\"285\" y=\"-126.8\" font-family=\"Helvetica,sans-Serif\" font-size=\"14.00\">mse = 0.074</text>\n",
       "<text text-anchor=\"middle\" x=\"285\" y=\"-111.8\" font-family=\"Helvetica,sans-Serif\" font-size=\"14.00\">samples = 156</text>\n",
       "<text text-anchor=\"middle\" x=\"285\" y=\"-96.8\" font-family=\"Helvetica,sans-Serif\" font-size=\"14.00\">value = 0.259</text>\n",
       "</g>\n",
       "<!-- 0&#45;&gt;4 -->\n",
       "<g id=\"edge4\" class=\"edge\"><title>0&#45;&gt;4</title>\n",
       "<path fill=\"none\" stroke=\"black\" d=\"M244.48,-192.884C249.51,-184.332 254.992,-175.013 260.252,-166.072\"/>\n",
       "<polygon fill=\"black\" stroke=\"black\" points=\"263.359,-167.693 265.412,-157.299 257.325,-164.144 263.359,-167.693\"/>\n",
       "<text text-anchor=\"middle\" x=\"271.686\" y=\"-177.799\" font-family=\"Helvetica,sans-Serif\" font-size=\"14.00\">False</text>\n",
       "</g>\n",
       "<!-- 2 -->\n",
       "<g id=\"node3\" class=\"node\"><title>2</title>\n",
       "<path fill=\"#e58139\" stroke=\"black\" d=\"M86,-53C86,-53 12,-53 12,-53 6,-53 7.10543e-15,-47 7.10543e-15,-41 7.10543e-15,-41 7.10543e-15,-12 7.10543e-15,-12 7.10543e-15,-6 6,-0 12,-0 12,-0 86,-0 86,-0 92,-0 98,-6 98,-12 98,-12 98,-41 98,-41 98,-47 92,-53 86,-53\"/>\n",
       "<text text-anchor=\"middle\" x=\"49\" y=\"-37.8\" font-family=\"Helvetica,sans-Serif\" font-size=\"14.00\">mse = 0.018</text>\n",
       "<text text-anchor=\"middle\" x=\"49\" y=\"-22.8\" font-family=\"Helvetica,sans-Serif\" font-size=\"14.00\">samples = 20</text>\n",
       "<text text-anchor=\"middle\" x=\"49\" y=\"-7.8\" font-family=\"Helvetica,sans-Serif\" font-size=\"14.00\">value = 0.854</text>\n",
       "</g>\n",
       "<!-- 1&#45;&gt;2 -->\n",
       "<g id=\"edge2\" class=\"edge\"><title>1&#45;&gt;2</title>\n",
       "<path fill=\"none\" stroke=\"black\" d=\"M124.429,-88.9485C112.75,-79.4346 100.033,-69.074 88.4247,-59.6175\"/>\n",
       "<polygon fill=\"black\" stroke=\"black\" points=\"90.5557,-56.8392 80.5921,-53.2367 86.1345,-62.2663 90.5557,-56.8392\"/>\n",
       "</g>\n",
       "<!-- 3 -->\n",
       "<g id=\"node4\" class=\"node\"><title>3</title>\n",
       "<path fill=\"#e58139\" fill-opacity=\"0.596078\" stroke=\"black\" d=\"M202,-53C202,-53 128,-53 128,-53 122,-53 116,-47 116,-41 116,-41 116,-12 116,-12 116,-6 122,-0 128,-0 128,-0 202,-0 202,-0 208,-0 214,-6 214,-12 214,-12 214,-41 214,-41 214,-47 208,-53 202,-53\"/>\n",
       "<text text-anchor=\"middle\" x=\"165\" y=\"-37.8\" font-family=\"Helvetica,sans-Serif\" font-size=\"14.00\">mse = 0.013</text>\n",
       "<text text-anchor=\"middle\" x=\"165\" y=\"-22.8\" font-family=\"Helvetica,sans-Serif\" font-size=\"14.00\">samples = 24</text>\n",
       "<text text-anchor=\"middle\" x=\"165\" y=\"-7.8\" font-family=\"Helvetica,sans-Serif\" font-size=\"14.00\">value = 0.552</text>\n",
       "</g>\n",
       "<!-- 1&#45;&gt;3 -->\n",
       "<g id=\"edge3\" class=\"edge\"><title>1&#45;&gt;3</title>\n",
       "<path fill=\"none\" stroke=\"black\" d=\"M165,-88.9485C165,-80.7153 165,-71.848 165,-63.4814\"/>\n",
       "<polygon fill=\"black\" stroke=\"black\" points=\"168.5,-63.2367 165,-53.2367 161.5,-63.2367 168.5,-63.2367\"/>\n",
       "</g>\n",
       "<!-- 5 -->\n",
       "<g id=\"node6\" class=\"node\"><title>5</title>\n",
       "<path fill=\"none\" stroke=\"black\" d=\"M326,-53C326,-53 244,-53 244,-53 238,-53 232,-47 232,-41 232,-41 232,-12 232,-12 232,-6 238,-0 244,-0 244,-0 326,-0 326,-0 332,-0 338,-6 338,-12 338,-12 338,-41 338,-41 338,-47 332,-53 326,-53\"/>\n",
       "<text text-anchor=\"middle\" x=\"285\" y=\"-37.8\" font-family=\"Helvetica,sans-Serif\" font-size=\"14.00\">mse = 0.015</text>\n",
       "<text text-anchor=\"middle\" x=\"285\" y=\"-22.8\" font-family=\"Helvetica,sans-Serif\" font-size=\"14.00\">samples = 110</text>\n",
       "<text text-anchor=\"middle\" x=\"285\" y=\"-7.8\" font-family=\"Helvetica,sans-Serif\" font-size=\"14.00\">value = 0.111</text>\n",
       "</g>\n",
       "<!-- 4&#45;&gt;5 -->\n",
       "<g id=\"edge5\" class=\"edge\"><title>4&#45;&gt;5</title>\n",
       "<path fill=\"none\" stroke=\"black\" d=\"M285,-88.9485C285,-80.7153 285,-71.848 285,-63.4814\"/>\n",
       "<polygon fill=\"black\" stroke=\"black\" points=\"288.5,-63.2367 285,-53.2367 281.5,-63.2367 288.5,-63.2367\"/>\n",
       "</g>\n",
       "<!-- 6 -->\n",
       "<g id=\"node7\" class=\"node\"><title>6</title>\n",
       "<path fill=\"#e58139\" fill-opacity=\"0.678431\" stroke=\"black\" d=\"M442,-53C442,-53 368,-53 368,-53 362,-53 356,-47 356,-41 356,-41 356,-12 356,-12 356,-6 362,-0 368,-0 368,-0 442,-0 442,-0 448,-0 454,-6 454,-12 454,-12 454,-41 454,-41 454,-47 448,-53 442,-53\"/>\n",
       "<text text-anchor=\"middle\" x=\"405\" y=\"-37.8\" font-family=\"Helvetica,sans-Serif\" font-size=\"14.00\">mse = 0.036</text>\n",
       "<text text-anchor=\"middle\" x=\"405\" y=\"-22.8\" font-family=\"Helvetica,sans-Serif\" font-size=\"14.00\">samples = 46</text>\n",
       "<text text-anchor=\"middle\" x=\"405\" y=\"-7.8\" font-family=\"Helvetica,sans-Serif\" font-size=\"14.00\">value = 0.615</text>\n",
       "</g>\n",
       "<!-- 4&#45;&gt;6 -->\n",
       "<g id=\"edge6\" class=\"edge\"><title>4&#45;&gt;6</title>\n",
       "<path fill=\"none\" stroke=\"black\" d=\"M326.97,-88.9485C339.051,-79.4346 352.208,-69.074 364.216,-59.6175\"/>\n",
       "<polygon fill=\"black\" stroke=\"black\" points=\"366.628,-62.1734 372.318,-53.2367 362.297,-56.6739 366.628,-62.1734\"/>\n",
       "</g>\n",
       "</g>\n",
       "</svg>\n"
      ],
      "text/plain": [
       "<graphviz.files.Source at 0x7fcb15eb8f98>"
      ]
     },
     "execution_count": 19,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "import graphviz\n",
    "with open(\"images/decision_trees/regression_tree.dot\") as f:\n",
    "    dot_graph = f.read()\n",
    "dot = graphviz.Source(dot_graph)\n",
    "dot.format = 'png'\n",
    "dot.render(filename='regression_tree', directory='images/decision_trees', cleanup=True)\n",
    "dot"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "[encoded data, 46004 characters]\n",
      "text/plain": [
       "<Figure size 792x288 with 2 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "tree_reg1 = DecisionTreeRegressor(random_state=42)\n",
    "tree_reg2 = DecisionTreeRegressor(random_state=42, min_samples_leaf=10)\n",
    "tree_reg1.fit(X, y)\n",
    "tree_reg2.fit(X, y)\n",
    "\n",
    "x1 = np.linspace(0, 1, 500).reshape(-1, 1)\n",
    "y_pred1 = tree_reg1.predict(x1)\n",
    "y_pred2 = tree_reg2.predict(x1)\n",
    "\n",
    "plt.figure(figsize=(11, 4))\n",
    "\n",
    "plt.subplot(121)\n",
    "plt.plot(X, y, \"b.\")\n",
    "plt.plot(x1, y_pred1, \"r.-\", linewidth=2, label=r\"$\\hat{y}$\")\n",
    "plt.axis([0, 1, -0.2, 1.1])\n",
    "plt.xlabel(\"$x_1$\", fontsize=18)\n",
    "plt.ylabel(\"$y$\", fontsize=18, rotation=0)\n",
    "plt.legend(loc=\"upper center\", fontsize=18)\n",
    "plt.title(\"규제 없음\", fontsize=14)\n",
    "\n",
    "plt.subplot(122)\n",
    "plt.plot(X, y, \"b.\")\n",
    "plt.plot(x1, y_pred2, \"r.-\", linewidth=2, label=r\"$\\hat{y}$\")\n",
    "plt.axis([0, 1, -0.2, 1.1])\n",
    "plt.xlabel(\"$x_1$\", fontsize=18)\n",
    "plt.title(\"min_samples_leaf={}\".format(tree_reg2.min_samples_leaf), fontsize=14)\n",
    "\n",
    "save_fig(\"tree_regression_regularization_plot\")\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true
   },
   "source": [
    "# 연습문제 해답"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 1. to 6."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "부록 A 참조."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true
   },
   "source": [
    "## 7."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "_문제: moons 데이터셋에 결정 트리를 훈련시키고 세밀하게 튜닝해보세요._"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "a. `make_moons(n_samples=1000, noise=0.4)`를 사용해 데이터셋을 생성합니다."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "`random_state=42`를 지정하여 결과를 일정하게 만듭니다:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.datasets import make_moons\n",
    "\n",
    "X, y = make_moons(n_samples=10000, noise=0.4, random_state=42)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "b. 이를 `train_test_split()`을 사용해 훈련 세트와 테스트 세트로 나눕니다"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.model_selection import train_test_split\n",
    "\n",
    "X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, random_state=42)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "c. `DecisionTreeClassifier`의 최적의 매개변수를 찾기 위해 교차 검증과 함께 그리드 탐색을 수행합니다(`GridSearchCV`를 사용하면 됩니다). 힌트: 여러 가지 `max_leaf_nodes` 값을 시도해보세요."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Fitting 3 folds for each of 294 candidates, totalling 882 fits\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "[Parallel(n_jobs=-1)]: Done 882 out of 882 | elapsed:    1.7s finished\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "GridSearchCV(cv=None, error_score='raise',\n",
       "       estimator=DecisionTreeClassifier(class_weight=None, criterion='gini', max_depth=None,\n",
       "            max_features=None, max_leaf_nodes=None,\n",
       "            min_impurity_decrease=0.0, min_impurity_split=None,\n",
       "            min_samples_leaf=1, min_samples_split=2,\n",
       "            min_weight_fraction_leaf=0.0, presort=False, random_state=42,\n",
       "            splitter='best'),\n",
       "       fit_params=None, iid=True, n_jobs=-1,\n",
       "       param_grid={'min_samples_split': [2, 3, 4], 'max_leaf_nodes': [2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31, 32, 33, 34, 35, 36, 37, 38, 39, 40, 41, 42, 43, 44, 45, 46, 47, 48, 49, 50, 51, 52, 53, 54, 55, 56, 57, 58, 59, 60, 61, 62, 63, 64, 65, 66, 67, 68, 69, 70, 71, 72, 73, 74, 75, 76, 77, 78, 79, 80, 81, 82, 83, 84, 85, 86, 87, 88, 89, 90, 91, 92, 93, 94, 95, 96, 97, 98, 99]},\n",
       "       pre_dispatch='2*n_jobs', refit=True, return_train_score='warn',\n",
       "       scoring=None, verbose=1)"
      ]
     },
     "execution_count": 23,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from sklearn.model_selection import GridSearchCV\n",
    "\n",
    "params = {'max_leaf_nodes': list(range(2, 100)), 'min_samples_split': [2, 3, 4]}\n",
    "grid_search_cv = GridSearchCV(DecisionTreeClassifier(random_state=42), params, n_jobs=-1, verbose=1)\n",
    "\n",
    "grid_search_cv.fit(X_train, y_train)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "DecisionTreeClassifier(class_weight=None, criterion='gini', max_depth=None,\n",
       "            max_features=None, max_leaf_nodes=17,\n",
       "            min_impurity_decrease=0.0, min_impurity_split=None,\n",
       "            min_samples_leaf=1, min_samples_split=2,\n",
       "            min_weight_fraction_leaf=0.0, presort=False, random_state=42,\n",
       "            splitter='best')"
      ]
     },
     "execution_count": 24,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "grid_search_cv.best_estimator_"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "d. 찾은 매개변수를 사용해 전체 훈련 세트에 대해 모델을 훈련시키고 테스트 세트에서 성능을 측정합니다. 대략 85~87%의 정확도가 나올 것입니다."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "기본적으로 `GridSearchCV`는 전체 훈련 세트로 찾은 최적의 모델을 다시 훈련시킵니다(`refit=False`로 지정해서 바꿀 수 있습니다). 그래서 별도로 작업할 필요가 없습니다. 모델의 정확도를 바로 평가할 수 있습니다:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.8695"
      ]
     },
     "execution_count": 25,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from sklearn.metrics import accuracy_score\n",
    "\n",
    "y_pred = grid_search_cv.predict(X_test)\n",
    "accuracy_score(y_test, y_pred)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 8."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "_Exercise: 랜덤 포레스트를 만들어보세요._"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "a. 이전 연습문제에 이어서, 훈련 세트의 서브셋을 1,000개 생성합니다. 각각은 무작위로 선택된 100개의 샘플을 담고 있습니다. 힌트: 사이킷런의 `ShuffleSplit`을 사용할 수 있습니다."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.model_selection import ShuffleSplit\n",
    "\n",
    "n_trees = 1000\n",
    "n_instances = 100\n",
    "\n",
    "mini_sets = []\n",
    "\n",
    "rs = ShuffleSplit(n_splits=n_trees, test_size=len(X_train) - n_instances, random_state=42)\n",
    "for mini_train_index, mini_test_index in rs.split(X_train):\n",
    "    X_mini_train = X_train[mini_train_index]\n",
    "    y_mini_train = y_train[mini_train_index]\n",
    "    mini_sets.append((X_mini_train, y_mini_train))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "b. 앞에서 찾은 최적의 매개변수를 사용해 각 서브셋에 결정 트리를 훈련시킵니다. 테스트 세트로 이 1,000개의 결정 트리를 평가합니다. 더 작은 데이터셋에서 훈련되었기 때문에 이 결정 트리는 앞서 만든 결정 트리보다 성능이 떨어져 약8 0%의 정확도를 냅니다."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.8054494999999999"
      ]
     },
     "execution_count": 27,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from sklearn.base import clone\n",
    "\n",
    "forest = [clone(grid_search_cv.best_estimator_) for _ in range(n_trees)]\n",
    "\n",
    "accuracy_scores = []\n",
    "\n",
    "for tree, (X_mini_train, y_mini_train) in zip(forest, mini_sets):\n",
    "    tree.fit(X_mini_train, y_mini_train)\n",
    "    \n",
    "    y_pred = tree.predict(X_test)\n",
    "    accuracy_scores.append(accuracy_score(y_test, y_pred))\n",
    "\n",
    "np.mean(accuracy_scores)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "c. 이제 마술을 부릴 차례입니다. 각 테스트 세트 샘플에 대해 1,000개의 결정 트리 예측을 만들고 다수로 나온 예측만 취합니다(사이파이의 `mode()` 함수를 사용할 수 있습니다). 그러면 테스트 세트에 대한 _다수결 예측_이 만들어집니다."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "metadata": {},
   "outputs": [],
   "source": [
    "Y_pred = np.empty([n_trees, len(X_test)], dtype=np.uint8)\n",
    "\n",
    "for tree_index, tree in enumerate(forest):\n",
    "    Y_pred[tree_index] = tree.predict(X_test)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "metadata": {},
   "outputs": [],
   "source": [
    "from scipy.stats import mode\n",
    "\n",
    "y_pred_majority_votes, n_votes = mode(Y_pred, axis=0)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "d. 테스트 세트에서 이 예측을 평가합니다. 앞서 만든 모델보다 조금 높은(약 0.5~1.5% 정도) 정확도를 얻게 될 것입니다. 축하합니다. 랜덤 포레스트 분류기를 훈련시켰습니다!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.872"
      ]
     },
     "execution_count": 30,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "accuracy_score(y_test, y_pred_majority_votes.reshape([-1]))"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.5.5"
  },
  "nav_menu": {
   "height": "309px",
   "width": "468px"
  },
  "toc": {
   "navigate_menu": true,
   "number_sections": true,
   "sideBar": true,
   "threshold": 6,
   "toc_cell": false,
   "toc_section_display": "block",
   "toc_window_display": false
  }
 },
 "nbformat": 4,
 "nbformat_minor": 1
}
