{
 "cells": [
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "CPython 3.5.5\n",
      "IPython 6.2.1\n",
      "\n",
      "numpy 1.14.1\n",
      "scipy 1.0.0\n",
      "sklearn 0.19.1\n",
      "pandas 0.22.0\n",
      "matplotlib 2.2.2\n"
     ]
    }
   ],
   "source": [
    "%load_ext watermark\n",
    "%watermark -v -p numpy,scipy,sklearn,pandas,matplotlib"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**5 장 – 서포트 벡터 머신**\n",
    "\n",
    "_이 노트북은 2장에 있는 모든 샘플 코드와 연습문제 해답을 가지고 있습니다._"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 설정"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "파이썬 2와 3을 모두 지원합니다. 공통 모듈을 임포트하고 맷플롯립 그림이 노트북 안에 포함되도록 설정하고 생성한 그림을 저장하기 위한 함수를 준비합니다:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "# 파이썬 2와 파이썬 3 지원\n",
    "from __future__ import division, print_function, unicode_literals\n",
    "\n",
    "# 공통\n",
    "import numpy as np\n",
    "import os\n",
    "\n",
    "# 일관된 출력을 위해 유사난수 초기화\n",
    "np.random.seed(42)\n",
    "\n",
    "# 맷플롯립 설정\n",
    "%matplotlib inline\n",
    "import matplotlib\n",
    "import matplotlib.pyplot as plt\n",
    "plt.rcParams['axes.labelsize'] = 14\n",
    "plt.rcParams['xtick.labelsize'] = 12\n",
    "plt.rcParams['ytick.labelsize'] = 12\n",
    "\n",
    "# 한글출력\n",
    "plt.rcParams['font.family'] = 'NanumBarunGothic'\n",
    "plt.rcParams['axes.unicode_minus'] = False\n",
    "\n",
    "# 그림을 저장할 폴드\n",
    "PROJECT_ROOT_DIR = \".\"\n",
    "CHAPTER_ID = \"svm\"\n",
    "\n",
    "def save_fig(fig_id, tight_layout=True):\n",
    "    path = os.path.join(PROJECT_ROOT_DIR, \"images\", CHAPTER_ID, fig_id + \".png\")\n",
    "    if tight_layout:\n",
    "        plt.tight_layout()\n",
    "    plt.savefig(path, format='png', dpi=300)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 라지 마진 분류"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "다음 몇 개의 코드 셀은 5장 앞부분의 그래프를 만듭니다. 실제 코드 예제는 그 이후에 나옵니다:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "SVC(C=inf, cache_size=200, class_weight=None, coef0=0.0,\n",
       "  decision_function_shape='ovr', degree=3, gamma='auto', kernel='linear',\n",
       "  max_iter=-1, probability=False, random_state=None, shrinking=True,\n",
       "  tol=0.001, verbose=False)"
      ]
     },
     "execution_count": 3,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from sklearn.svm import SVC\n",
    "from sklearn import datasets\n",
    "\n",
    "iris = datasets.load_iris()\n",
    "X = iris[\"data\"][:, (2, 3)]  # 꽃잎 길이, 꽃잎 너비\n",
    "y = iris[\"target\"]\n",
    "\n",
    "setosa_or_versicolor = (y == 0) | (y == 1)\n",
    "X = X[setosa_or_versicolor]\n",
    "y = y[setosa_or_versicolor]\n",
    "\n",
    "# SVM 분류 모델\n",
    "svm_clf = SVC(kernel=\"linear\", C=float(\"inf\"))\n",
    "svm_clf.fit(X, y)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "[encoded data, 30092 characters]\n",
      "text/plain": [
       "<Figure size 864x194.4 with 2 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "# 나쁜 모델\n",
    "x0 = np.linspace(0, 5.5, 200)\n",
    "pred_1 = 5*x0 - 20\n",
    "pred_2 = x0 - 1.8\n",
    "pred_3 = 0.1 * x0 + 0.5\n",
    "\n",
    "def plot_svc_decision_boundary(svm_clf, xmin, xmax):\n",
    "    w = svm_clf.coef_[0]\n",
    "    b = svm_clf.intercept_[0]\n",
    "\n",
    "    # 결정 경계에서 w0*x0 + w1*x1 + b = 0 이므로\n",
    "    # => x1 = -w0/w1 * x0 - b/w1\n",
    "    x0 = np.linspace(xmin, xmax, 200)\n",
    "    decision_boundary = -w[0]/w[1] * x0 - b/w[1]\n",
    "\n",
    "    margin = 1/w[1]\n",
    "    gutter_up = decision_boundary + margin\n",
    "    gutter_down = decision_boundary - margin\n",
    "\n",
    "    svs = svm_clf.support_vectors_\n",
    "    plt.scatter(svs[:, 0], svs[:, 1], s=180, facecolors='#FFAAAA')\n",
    "    plt.plot(x0, decision_boundary, \"k-\", linewidth=2)\n",
    "    plt.plot(x0, gutter_up, \"k--\", linewidth=2)\n",
    "    plt.plot(x0, gutter_down, \"k--\", linewidth=2)\n",
    "\n",
    "plt.figure(figsize=(12,2.7))\n",
    "\n",
    "plt.subplot(121)\n",
    "plt.plot(x0, pred_1, \"g--\", linewidth=2)\n",
    "plt.plot(x0, pred_2, \"m-\", linewidth=2)\n",
    "plt.plot(x0, pred_3, \"r-\", linewidth=2)\n",
    "plt.plot(X[:, 0][y==1], X[:, 1][y==1], \"bs\", label=\"Iris-Versicolor\")\n",
    "plt.plot(X[:, 0][y==0], X[:, 1][y==0], \"yo\", label=\"Iris-Setosa\")\n",
    "plt.xlabel(\"꽃잎 길이\", fontsize=14)\n",
    "plt.ylabel(\"꽃잎 너비\", fontsize=14)\n",
    "plt.legend(loc=\"upper left\", fontsize=14)\n",
    "plt.axis([0, 5.5, 0, 2])\n",
    "\n",
    "plt.subplot(122)\n",
    "plot_svc_decision_boundary(svm_clf, 0, 5.5)\n",
    "plt.plot(X[:, 0][y==1], X[:, 1][y==1], \"bs\")\n",
    "plt.plot(X[:, 0][y==0], X[:, 1][y==0], \"yo\")\n",
    "plt.xlabel(\"꽃잎 길이\", fontsize=14)\n",
    "plt.axis([0, 5.5, 0, 2])\n",
    "\n",
    "save_fig(\"large_margin_classification_plot\")\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 특성의 스케일에 민감함"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "[encoded data, 30536 characters]\n",
      "text/plain": [
       "<Figure size 864x230.4 with 2 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "Xs = np.array([[1, 50], [5, 20], [3, 80], [5, 60]]).astype(np.float64)\n",
    "ys = np.array([0, 0, 1, 1])\n",
    "svm_clf = SVC(kernel=\"linear\", C=100)\n",
    "svm_clf.fit(Xs, ys)\n",
    "\n",
    "plt.figure(figsize=(12,3.2))\n",
    "plt.subplot(121)\n",
    "plt.plot(Xs[:, 0][ys==1], Xs[:, 1][ys==1], \"bo\")\n",
    "plt.plot(Xs[:, 0][ys==0], Xs[:, 1][ys==0], \"ms\")\n",
    "plot_svc_decision_boundary(svm_clf, 0, 6)\n",
    "plt.xlabel(\"$x_0$\", fontsize=20)\n",
    "plt.ylabel(\"$x_1$  \", fontsize=20, rotation=0)\n",
    "plt.title(\"스케일 조정 전\", fontsize=16)\n",
    "plt.axis([0, 6, 0, 90])\n",
    "\n",
    "from sklearn.preprocessing import StandardScaler\n",
    "scaler = StandardScaler()\n",
    "X_scaled = scaler.fit_transform(Xs)\n",
    "svm_clf.fit(X_scaled, ys)\n",
    "\n",
    "plt.subplot(122)\n",
    "plt.plot(X_scaled[:, 0][ys==1], X_scaled[:, 1][ys==1], \"bo\")\n",
    "plt.plot(X_scaled[:, 0][ys==0], X_scaled[:, 1][ys==0], \"ms\")\n",
    "plot_svc_decision_boundary(svm_clf, -2, 2)\n",
    "plt.xlabel(\"$x_0$\", fontsize=20)\n",
    "plt.title(\"스케일 조정 후\", fontsize=16)\n",
    "plt.axis([-2, 2, -2, 2])\n",
    "\n",
    "save_fig(\"sensitivity_to_feature_scales_plot\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 이상치에 민감함"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "[encoded data, 34952 characters]\n",
      "text/plain": [
       "<Figure size 864x194.4 with 2 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "X_outliers = np.array([[3.4, 1.3], [3.2, 0.8]])\n",
    "y_outliers = np.array([0, 0])\n",
    "Xo1 = np.concatenate([X, X_outliers[:1]], axis=0)\n",
    "yo1 = np.concatenate([y, y_outliers[:1]], axis=0)\n",
    "Xo2 = np.concatenate([X, X_outliers[1:]], axis=0)\n",
    "yo2 = np.concatenate([y, y_outliers[1:]], axis=0)\n",
    "\n",
    "svm_clf2 = SVC(kernel=\"linear\", C=10**9)\n",
    "svm_clf2.fit(Xo2, yo2)\n",
    "\n",
    "plt.figure(figsize=(12,2.7))\n",
    "\n",
    "plt.subplot(121)\n",
    "plt.plot(Xo1[:, 0][yo1==1], Xo1[:, 1][yo1==1], \"bs\")\n",
    "plt.plot(Xo1[:, 0][yo1==0], Xo1[:, 1][yo1==0], \"yo\")\n",
    "plt.text(0.3, 1.0, \"불가능!\", fontsize=24, color=\"red\")\n",
    "plt.xlabel(\"꽃잎 길이\", fontsize=14)\n",
    "plt.ylabel(\"꽃잎 너비\", fontsize=14)\n",
    "plt.annotate(\"이상치\",\n",
    "             xy=(X_outliers[0][0], X_outliers[0][1]),\n",
    "             xytext=(2.5, 1.7),\n",
    "             ha=\"center\",\n",
    "             arrowprops=dict(facecolor='black', shrink=0.1),\n",
    "             fontsize=16,\n",
    "            )\n",
    "plt.axis([0, 5.5, 0, 2])\n",
    "\n",
    "plt.subplot(122)\n",
    "plt.plot(Xo2[:, 0][yo2==1], Xo2[:, 1][yo2==1], \"bs\")\n",
    "plt.plot(Xo2[:, 0][yo2==0], Xo2[:, 1][yo2==0], \"yo\")\n",
    "plot_svc_decision_boundary(svm_clf2, 0, 5.5)\n",
    "plt.xlabel(\"꽃잎 길이\", fontsize=14)\n",
    "plt.annotate(\"이상치\",\n",
    "             xy=(X_outliers[1][0], X_outliers[1][1]),\n",
    "             xytext=(3.2, 0.08),\n",
    "             ha=\"center\",\n",
    "             arrowprops=dict(facecolor='black', shrink=0.1),\n",
    "             fontsize=16,\n",
    "            )\n",
    "plt.axis([0, 5.5, 0, 2])\n",
    "\n",
    "save_fig(\"sensitivity_to_outliers_plot\")\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 라지 마진 *vs* 마진 오류"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "다음이 5장의 첫 번째 코드 예제입니다:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Pipeline(memory=None,\n",
       "     steps=[('scaler', StandardScaler(copy=True, with_mean=True, with_std=True)), ('linear_svc', LinearSVC(C=1, class_weight=None, dual=True, fit_intercept=True,\n",
       "     intercept_scaling=1, loss='hinge', max_iter=1000, multi_class='ovr',\n",
       "     penalty='l2', random_state=42, tol=0.0001, verbose=0))])"
      ]
     },
     "execution_count": 7,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "import numpy as np\n",
    "from sklearn import datasets\n",
    "from sklearn.pipeline import Pipeline\n",
    "from sklearn.preprocessing import StandardScaler\n",
    "from sklearn.svm import LinearSVC\n",
    "\n",
    "iris = datasets.load_iris()\n",
    "X = iris[\"data\"][:, (2, 3)]  # petal length, petal width\n",
    "y = (iris[\"target\"] == 2).astype(np.float64)  # Iris-Virginica\n",
    "\n",
    "svm_clf = Pipeline([\n",
    "        (\"scaler\", StandardScaler()),\n",
    "        (\"linear_svc\", LinearSVC(C=1, loss=\"hinge\", random_state=42)),\n",
    "    ])\n",
    "\n",
    "svm_clf.fit(X, y)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([1.])"
      ]
     },
     "execution_count": 8,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "svm_clf.predict([[5.5, 1.7]])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "여러가지 규제 설정을 비교하는 그래프를 만들겠습니다:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Pipeline(memory=None,\n",
       "     steps=[('scaler', StandardScaler(copy=True, with_mean=True, with_std=True)), ('linear_svc', LinearSVC(C=100, class_weight=None, dual=True, fit_intercept=True,\n",
       "     intercept_scaling=1, loss='hinge', max_iter=1000, multi_class='ovr',\n",
       "     penalty='l2', random_state=42, tol=0.0001, verbose=0))])"
      ]
     },
     "execution_count": 9,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "scaler = StandardScaler()\n",
    "svm_clf1 = LinearSVC(C=1, loss=\"hinge\", random_state=42)\n",
    "svm_clf2 = LinearSVC(C=100, loss=\"hinge\", random_state=42)\n",
    "\n",
    "scaled_svm_clf1 = Pipeline([\n",
    "        (\"scaler\", scaler),\n",
    "        (\"linear_svc\", svm_clf1),\n",
    "    ])\n",
    "scaled_svm_clf2 = Pipeline([\n",
    "        (\"scaler\", scaler),\n",
    "        (\"linear_svc\", svm_clf2),\n",
    "    ])\n",
    "\n",
    "scaled_svm_clf1.fit(X, y)\n",
    "scaled_svm_clf2.fit(X, y)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [],
   "source": [
    "# 스케일되지 않은 파라미터로 변경\n",
    "b1 = svm_clf1.decision_function([-scaler.mean_ / scaler.scale_])\n",
    "b2 = svm_clf2.decision_function([-scaler.mean_ / scaler.scale_])\n",
    "w1 = svm_clf1.coef_[0] / scaler.scale_\n",
    "w2 = svm_clf2.coef_[0] / scaler.scale_\n",
    "svm_clf1.intercept_ = np.array([b1])\n",
    "svm_clf2.intercept_ = np.array([b2])\n",
    "svm_clf1.coef_ = np.array([w1])\n",
    "svm_clf2.coef_ = np.array([w2])\n",
    "\n",
    "# 서포트 벡터 찾기 (libsvm과 달리 liblinear 라이브러리에서 제공하지 않기 때문에 \n",
    "# LinearSVC에는 서포트 벡터가 저장되어 있지 않습니다.)\n",
    "t = y * 2 - 1\n",
    "support_vectors_idx1 = (t * (X.dot(w1) + b1) < 1).ravel()\n",
    "support_vectors_idx2 = (t * (X.dot(w2) + b2) < 1).ravel()\n",
    "svm_clf1.support_vectors_ = X[support_vectors_idx1]\n",
    "svm_clf2.support_vectors_ = X[support_vectors_idx2]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "[encoded data, 44140 characters]\n",
      "text/plain": [
       "<Figure size 864x230.4 with 2 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "plt.figure(figsize=(12,3.2))\n",
    "plt.subplot(121)\n",
    "plt.plot(X[:, 0][y==1], X[:, 1][y==1], \"g^\", label=\"Iris-Virginica\")\n",
    "plt.plot(X[:, 0][y==0], X[:, 1][y==0], \"bs\", label=\"Iris-Versicolor\")\n",
    "plot_svc_decision_boundary(svm_clf1, 4, 6)\n",
    "plt.xlabel(\"꽃잎 길이\", fontsize=14)\n",
    "plt.ylabel(\"꽃잎 너비\", fontsize=14)\n",
    "plt.legend(loc=\"upper left\", fontsize=14)\n",
    "plt.title(\"$C = {}$\".format(svm_clf1.C), fontsize=16)\n",
    "plt.axis([4, 6, 0.8, 2.8])\n",
    "\n",
    "plt.subplot(122)\n",
    "plt.plot(X[:, 0][y==1], X[:, 1][y==1], \"g^\")\n",
    "plt.plot(X[:, 0][y==0], X[:, 1][y==0], \"bs\")\n",
    "plot_svc_decision_boundary(svm_clf2, 4, 6)\n",
    "plt.xlabel(\"꽃잎 길이\", fontsize=14)\n",
    "plt.title(\"$C = {}$\".format(svm_clf2.C), fontsize=16)\n",
    "plt.axis([4, 6, 0.8, 2.8])\n",
    "\n",
    "save_fig(\"regularization_plot\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true
   },
   "source": [
    "# 비선형 분류"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "[encoded data, 7728 characters]\n",
      "text/plain": [
       "<Figure size 792x288 with 2 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "X1D = np.linspace(-4, 4, 9).reshape(-1, 1)\n",
    "X2D = np.c_[X1D, X1D**2]\n",
    "y = np.array([0, 0, 1, 1, 1, 1, 1, 0, 0])\n",
    "\n",
    "plt.figure(figsize=(11, 4))\n",
    "\n",
    "plt.subplot(121)\n",
    "plt.grid(True, which='both')\n",
    "plt.axhline(y=0, color='k')\n",
    "plt.plot(X1D[:, 0][y==0], np.zeros(4), \"bs\")\n",
    "plt.plot(X1D[:, 0][y==1], np.zeros(5), \"g^\")\n",
    "plt.gca().get_yaxis().set_ticks([])\n",
    "plt.xlabel(r\"$x_1$\", fontsize=20)\n",
    "plt.axis([-4.5, 4.5, -0.2, 0.2])\n",
    "\n",
    "plt.subplot(122)\n",
    "plt.grid(True, which='both')\n",
    "plt.axhline(y=0, color='k')\n",
    "plt.axvline(x=0, color='k')\n",
    "plt.plot(X2D[:, 0][y==0], X2D[:, 1][y==0], \"bs\")\n",
    "plt.plot(X2D[:, 0][y==1], X2D[:, 1][y==1], \"g^\")\n",
    "plt.xlabel(r\"$x_1$\", fontsize=20)\n",
    "plt.ylabel(r\"$x_2$\", fontsize=20, rotation=0)\n",
    "plt.gca().get_yaxis().set_ticks([0, 4, 8, 12, 16])\n",
    "plt.plot([-4.5, 4.5], [6.5, 6.5], \"r--\", linewidth=3)\n",
    "plt.axis([-4.5, 4.5, -1, 17])\n",
    "\n",
    "plt.subplots_adjust(right=1)\n",
    "\n",
    "save_fig(\"higher_dimensions_plot\", tight_layout=False)\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "[encoded data, 10860 characters]\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "from sklearn.datasets import make_moons\n",
    "X, y = make_moons(n_samples=100, noise=0.15, random_state=42)\n",
    "\n",
    "def plot_dataset(X, y, axes):\n",
    "    plt.plot(X[:, 0][y==0], X[:, 1][y==0], \"bs\")\n",
    "    plt.plot(X[:, 0][y==1], X[:, 1][y==1], \"g^\")\n",
    "    plt.axis(axes)\n",
    "    plt.grid(True, which='both')\n",
    "    plt.xlabel(r\"$x_1$\", fontsize=20)\n",
    "    plt.ylabel(r\"$x_2$\", fontsize=20, rotation=0)\n",
    "\n",
    "plot_dataset(X, y, [-1.5, 2.5, -1, 1.5])\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Pipeline(memory=None,\n",
       "     steps=[('poly_features', PolynomialFeatures(degree=3, include_bias=True, interaction_only=False)), ('scaler', StandardScaler(copy=True, with_mean=True, with_std=True)), ('svm_clf', LinearSVC(C=10, class_weight=None, dual=True, fit_intercept=True,\n",
       "     intercept_scaling=1, loss='hinge', max_iter=1000, multi_class='ovr',\n",
       "     penalty='l2', random_state=42, tol=0.0001, verbose=0))])"
      ]
     },
     "execution_count": 14,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from sklearn.datasets import make_moons\n",
    "from sklearn.pipeline import Pipeline\n",
    "from sklearn.preprocessing import PolynomialFeatures\n",
    "\n",
    "polynomial_svm_clf = Pipeline([\n",
    "        (\"poly_features\", PolynomialFeatures(degree=3)),\n",
    "        (\"scaler\", StandardScaler()),\n",
    "        (\"svm_clf\", LinearSVC(C=10, loss=\"hinge\", random_state=42))\n",
    "    ])\n",
    "\n",
    "polynomial_svm_clf.fit(X, y)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "[encoded data, 18444 characters]\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "def plot_predictions(clf, axes):\n",
    "    x0s = np.linspace(axes[0], axes[1], 100)\n",
    "    x1s = np.linspace(axes[2], axes[3], 100)\n",
    "    x0, x1 = np.meshgrid(x0s, x1s)\n",
    "    X = np.c_[x0.ravel(), x1.ravel()]\n",
    "    y_pred = clf.predict(X).reshape(x0.shape)\n",
    "    y_decision = clf.decision_function(X).reshape(x0.shape)\n",
    "    plt.contourf(x0, x1, y_pred, cmap=plt.cm.brg, alpha=0.2)\n",
    "    plt.contourf(x0, x1, y_decision, cmap=plt.cm.brg, alpha=0.1)\n",
    "\n",
    "plot_predictions(polynomial_svm_clf, [-1.5, 2.5, -1, 1.5])\n",
    "plot_dataset(X, y, [-1.5, 2.5, -1, 1.5])\n",
    "\n",
    "save_fig(\"moons_polynomial_svc_plot\")\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Pipeline(memory=None,\n",
       "     steps=[('scaler', StandardScaler(copy=True, with_mean=True, with_std=True)), ('svm_clf', SVC(C=5, cache_size=200, class_weight=None, coef0=1,\n",
       "  decision_function_shape='ovr', degree=3, gamma='auto', kernel='poly',\n",
       "  max_iter=-1, probability=False, random_state=None, shrinking=True,\n",
       "  tol=0.001, verbose=False))])"
      ]
     },
     "execution_count": 16,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from sklearn.svm import SVC\n",
    "\n",
    "poly_kernel_svm_clf = Pipeline([\n",
    "        (\"scaler\", StandardScaler()),\n",
    "        (\"svm_clf\", SVC(kernel=\"poly\", degree=3, coef0=1, C=5))\n",
    "    ])\n",
    "poly_kernel_svm_clf.fit(X, y)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Pipeline(memory=None,\n",
       "     steps=[('scaler', StandardScaler(copy=True, with_mean=True, with_std=True)), ('svm_clf', SVC(C=5, cache_size=200, class_weight=None, coef0=100,\n",
       "  decision_function_shape='ovr', degree=10, gamma='auto', kernel='poly',\n",
       "  max_iter=-1, probability=False, random_state=None, shrinking=True,\n",
       "  tol=0.001, verbose=False))])"
      ]
     },
     "execution_count": 17,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "poly100_kernel_svm_clf = Pipeline([\n",
    "        (\"scaler\", StandardScaler()),\n",
    "        (\"svm_clf\", SVC(kernel=\"poly\", degree=10, coef0=100, C=5))\n",
    "    ])\n",
    "poly100_kernel_svm_clf.fit(X, y)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "[encoded data, 32700 characters]\n",
      "text/plain": [
       "<Figure size 792x288 with 2 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "plt.figure(figsize=(11, 4))\n",
    "\n",
    "plt.subplot(121)\n",
    "plot_predictions(poly_kernel_svm_clf, [-1.5, 2.5, -1, 1.5])\n",
    "plot_dataset(X, y, [-1.5, 2.5, -1, 1.5])\n",
    "plt.title(r\"$d=3, r=1, C=5$\", fontsize=18)\n",
    "\n",
    "plt.subplot(122)\n",
    "plot_predictions(poly100_kernel_svm_clf, [-1.5, 2.5, -1, 1.5])\n",
    "plot_dataset(X, y, [-1.5, 2.5, -1, 1.5])\n",
    "plt.title(r\"$d=10, r=100, C=5$\", fontsize=18)\n",
    "\n",
    "save_fig(\"moons_kernelized_polynomial_svc_plot\")\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "data": {
      "image/png": "[encoded data, 38144 characters]\n",
      "text/plain": [
       "<Figure size 792x288 with 2 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "def gaussian_rbf(x, landmark, gamma):\n",
    "    return np.exp(-gamma * np.linalg.norm(x - landmark, axis=1)**2)\n",
    "\n",
    "gamma = 0.3\n",
    "\n",
    "x1s = np.linspace(-4.5, 4.5, 200).reshape(-1, 1)\n",
    "x2s = gaussian_rbf(x1s, -2, gamma)\n",
    "x3s = gaussian_rbf(x1s, 1, gamma)\n",
    "\n",
    "XK = np.c_[gaussian_rbf(X1D, -2, gamma), gaussian_rbf(X1D, 1, gamma)]\n",
    "yk = np.array([0, 0, 1, 1, 1, 1, 1, 0, 0])\n",
    "\n",
    "plt.figure(figsize=(11, 4))\n",
    "\n",
    "plt.subplot(121)\n",
    "plt.grid(True, which='both')\n",
    "plt.axhline(y=0, color='k')\n",
    "plt.scatter(x=[-2, 1], y=[0, 0], s=150, alpha=0.5, c=\"red\")\n",
    "plt.plot(X1D[:, 0][yk==0], np.zeros(4), \"bs\")\n",
    "plt.plot(X1D[:, 0][yk==1], np.zeros(5), \"g^\")\n",
    "plt.plot(x1s, x2s, \"g--\")\n",
    "plt.plot(x1s, x3s, \"b:\")\n",
    "plt.gca().get_yaxis().set_ticks([0, 0.25, 0.5, 0.75, 1])\n",
    "plt.xlabel(r\"$x_1$\", fontsize=20)\n",
    "plt.ylabel(r\"유사도\", fontsize=14)\n",
    "plt.annotate(r'$\\mathbf{x}$',\n",
    "             xy=(X1D[3, 0], 0),\n",
    "             xytext=(-0.5, 0.20),\n",
    "             ha=\"center\",\n",
    "             arrowprops=dict(facecolor='black', shrink=0.1),\n",
    "             fontsize=18,\n",
    "            )\n",
    "plt.text(-2, 0.9, \"$x_2$\", ha=\"center\", fontsize=20)\n",
    "plt.text(1, 0.9, \"$x_3$\", ha=\"center\", fontsize=20)\n",
    "plt.axis([-4.5, 4.5, -0.1, 1.1])\n",
    "\n",
    "plt.subplot(122)\n",
    "plt.grid(True, which='both')\n",
    "plt.axhline(y=0, color='k')\n",
    "plt.axvline(x=0, color='k')\n",
    "plt.plot(XK[:, 0][yk==0], XK[:, 1][yk==0], \"bs\")\n",
    "plt.plot(XK[:, 0][yk==1], XK[:, 1][yk==1], \"g^\")\n",
    "plt.xlabel(r\"$x_2$\", fontsize=20)\n",
    "plt.ylabel(r\"$x_3$  \", fontsize=20, rotation=0)\n",
    "plt.annotate(r'$\\phi\\left(\\mathbf{x}\\right)$',\n",
    "             xy=(XK[3, 0], XK[3, 1]),\n",
    "             xytext=(0.65, 0.50),\n",
    "             ha=\"center\",\n",
    "             arrowprops=dict(facecolor='black', shrink=0.1),\n",
    "             fontsize=18,\n",
    "            )\n",
    "plt.plot([-0.1, 1.1], [0.57, -0.1], \"r--\", linewidth=3)\n",
    "plt.axis([-0.1, 1.1, -0.1, 1.1])\n",
    "    \n",
    "plt.subplots_adjust(right=1)\n",
    "\n",
    "save_fig(\"kernel_method_plot\")\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Phi(-1.0, -2) = [0.74081822]\n",
      "Phi(-1.0, 1) = [0.30119421]\n"
     ]
    }
   ],
   "source": [
    "x1_example = X1D[3, 0]\n",
    "for landmark in (-2, 1):\n",
    "    k = gaussian_rbf(np.array([[x1_example]]), np.array([[landmark]]), gamma)\n",
    "    print(\"Phi({}, {}) = {}\".format(x1_example, landmark, k))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Pipeline(memory=None,\n",
       "     steps=[('scaler', StandardScaler(copy=True, with_mean=True, with_std=True)), ('svm_clf', SVC(C=0.001, cache_size=200, class_weight=None, coef0=0.0,\n",
       "  decision_function_shape='ovr', degree=3, gamma=5, kernel='rbf',\n",
       "  max_iter=-1, probability=False, random_state=None, shrinking=True,\n",
       "  tol=0.001, verbose=False))])"
      ]
     },
     "execution_count": 21,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "rbf_kernel_svm_clf = Pipeline([\n",
    "        (\"scaler\", StandardScaler()),\n",
    "        (\"svm_clf\", SVC(kernel=\"rbf\", gamma=5, C=0.001))\n",
    "    ])\n",
    "rbf_kernel_svm_clf.fit(X, y)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "[encoded data, 77368 characters]\n",
      "text/plain": [
       "<Figure size 792x504 with 4 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "from sklearn.svm import SVC\n",
    "\n",
    "gamma1, gamma2 = 0.1, 5\n",
    "C1, C2 = 0.001, 1000\n",
    "hyperparams = (gamma1, C1), (gamma1, C2), (gamma2, C1), (gamma2, C2)\n",
    "\n",
    "svm_clfs = []\n",
    "for gamma, C in hyperparams:\n",
    "    rbf_kernel_svm_clf = Pipeline([\n",
    "            (\"scaler\", StandardScaler()),\n",
    "            (\"svm_clf\", SVC(kernel=\"rbf\", gamma=gamma, C=C))\n",
    "        ])\n",
    "    rbf_kernel_svm_clf.fit(X, y)\n",
    "    svm_clfs.append(rbf_kernel_svm_clf)\n",
    "\n",
    "plt.figure(figsize=(11, 7))\n",
    "\n",
    "for i, svm_clf in enumerate(svm_clfs):\n",
    "    plt.subplot(221 + i)\n",
    "    plot_predictions(svm_clf, [-1.5, 2.5, -1, 1.5])\n",
    "    plot_dataset(X, y, [-1.5, 2.5, -1, 1.5])\n",
    "    gamma, C = hyperparams[i]\n",
    "    plt.title(r\"$\\gamma = {}, C = {}$\".format(gamma, C), fontsize=16)\n",
    "\n",
    "save_fig(\"moons_rbf_svc_plot\")\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 회귀"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {},
   "outputs": [],
   "source": [
    "np.random.seed(42)\n",
    "m = 50\n",
    "X = 2 * np.random.rand(m, 1)\n",
    "y = (4 + 3 * X + np.random.randn(m, 1)).ravel()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "LinearSVR(C=1.0, dual=True, epsilon=1.5, fit_intercept=True,\n",
       "     intercept_scaling=1.0, loss='epsilon_insensitive', max_iter=1000,\n",
       "     random_state=42, tol=0.0001, verbose=0)"
      ]
     },
     "execution_count": 24,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from sklearn.svm import LinearSVR\n",
    "\n",
    "svm_reg = LinearSVR(epsilon=1.5, random_state=42)\n",
    "svm_reg.fit(X, y)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {},
   "outputs": [],
   "source": [
    "svm_reg1 = LinearSVR(epsilon=1.5, random_state=42)\n",
    "svm_reg2 = LinearSVR(epsilon=0.5, random_state=42)\n",
    "svm_reg1.fit(X, y)\n",
    "svm_reg2.fit(X, y)\n",
    "\n",
    "def find_support_vectors(svm_reg, X, y):\n",
    "    y_pred = svm_reg.predict(X)\n",
    "    off_margin = (np.abs(y - y_pred) >= svm_reg.epsilon)\n",
    "    return np.argwhere(off_margin)\n",
    "\n",
    "svm_reg1.support_ = find_support_vectors(svm_reg1, X, y)\n",
    "svm_reg2.support_ = find_support_vectors(svm_reg2, X, y)\n",
    "\n",
    "eps_x1 = 1\n",
    "eps_y_pred = svm_reg1.predict([[eps_x1]])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "[encoded data, 47808 characters]\n",
      "text/plain": [
       "<Figure size 648x288 with 2 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "def plot_svm_regression(svm_reg, X, y, axes):\n",
    "    x1s = np.linspace(axes[0], axes[1], 100).reshape(100, 1)\n",
    "    y_pred = svm_reg.predict(x1s)\n",
    "    plt.plot(x1s, y_pred, \"k-\", linewidth=2, label=r\"$\\hat{y}$\")\n",
    "    plt.plot(x1s, y_pred + svm_reg.epsilon, \"k--\")\n",
    "    plt.plot(x1s, y_pred - svm_reg.epsilon, \"k--\")\n",
    "    plt.scatter(X[svm_reg.support_], y[svm_reg.support_], s=180, facecolors='#FFAAAA')\n",
    "    plt.plot(X, y, \"bo\")\n",
    "    plt.xlabel(r\"$x_1$\", fontsize=18)\n",
    "    plt.legend(loc=\"upper left\", fontsize=18)\n",
    "    plt.axis(axes)\n",
    "\n",
    "plt.figure(figsize=(9, 4))\n",
    "plt.subplot(121)\n",
    "plot_svm_regression(svm_reg1, X, y, [0, 2, 3, 11])\n",
    "plt.title(r\"$\\epsilon = {}$\".format(svm_reg1.epsilon), fontsize=18)\n",
    "plt.ylabel(r\"$y$\", fontsize=18, rotation=0)\n",
    "#plt.plot([eps_x1, eps_x1], [eps_y_pred, eps_y_pred - svm_reg1.epsilon], \"k-\", linewidth=2)\n",
    "plt.annotate(\n",
    "        '', xy=(eps_x1, eps_y_pred), xycoords='data',\n",
    "        xytext=(eps_x1, eps_y_pred - svm_reg1.epsilon),\n",
    "        textcoords='data', arrowprops={'arrowstyle': '<->', 'linewidth': 1.5}\n",
    "    )\n",
    "plt.text(0.91, 5.6, r\"$\\epsilon$\", fontsize=20)\n",
    "plt.subplot(122)\n",
    "plot_svm_regression(svm_reg2, X, y, [0, 2, 3, 11])\n",
    "plt.title(r\"$\\epsilon = {}$\".format(svm_reg2.epsilon), fontsize=18)\n",
    "save_fig(\"svm_regression_plot\")\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "metadata": {},
   "outputs": [],
   "source": [
    "np.random.seed(42)\n",
    "m = 100\n",
    "X = 2 * np.random.rand(m, 1) - 1\n",
    "y = (0.2 + 0.1 * X + 0.5 * X**2 + np.random.randn(m, 1)/10).ravel()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "SVR(C=100, cache_size=200, coef0=0.0, degree=2, epsilon=0.1, gamma='auto',\n",
       "  kernel='poly', max_iter=-1, shrinking=True, tol=0.001, verbose=False)"
      ]
     },
     "execution_count": 28,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from sklearn.svm import SVR\n",
    "\n",
    "svm_poly_reg = SVR(kernel=\"poly\", degree=2, C=100, epsilon=0.1)\n",
    "svm_poly_reg.fit(X, y)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "SVR(C=0.01, cache_size=200, coef0=0.0, degree=2, epsilon=0.1, gamma='auto',\n",
       "  kernel='poly', max_iter=-1, shrinking=True, tol=0.001, verbose=False)"
      ]
     },
     "execution_count": 29,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from sklearn.svm import SVR\n",
    "\n",
    "svm_poly_reg1 = SVR(kernel=\"poly\", degree=2, C=100, epsilon=0.1)\n",
    "svm_poly_reg2 = SVR(kernel=\"poly\", degree=2, C=0.01, epsilon=0.1)\n",
    "svm_poly_reg1.fit(X, y)\n",
    "svm_poly_reg2.fit(X, y)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "[encoded data, 51792 characters]\n",
      "text/plain": [
       "<Figure size 648x288 with 2 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "plt.figure(figsize=(9, 4))\n",
    "plt.subplot(121)\n",
    "plot_svm_regression(svm_poly_reg1, X, y, [-1, 1, 0, 1])\n",
    "plt.title(r\"$degree={}, C={}, \\epsilon = {}$\".format(svm_poly_reg1.degree, svm_poly_reg1.C, svm_poly_reg1.epsilon), fontsize=18)\n",
    "plt.ylabel(r\"$y$\", fontsize=18, rotation=0)\n",
    "plt.subplot(122)\n",
    "plot_svm_regression(svm_poly_reg2, X, y, [-1, 1, 0, 1])\n",
    "plt.title(r\"$degree={}, C={}, \\epsilon = {}$\".format(svm_poly_reg2.degree, svm_poly_reg2.C, svm_poly_reg2.epsilon), fontsize=18)\n",
    "save_fig(\"svm_with_polynomial_kernel_plot\")\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# SVM 이론"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "metadata": {},
   "outputs": [],
   "source": [
    "iris = datasets.load_iris()\n",
    "X = iris[\"data\"][:, (2, 3)]  # petal length, petal width\n",
    "y = (iris[\"target\"] == 2).astype(np.float64)  # Iris-Virginica"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 32,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "[encoded data, 203052 characters]\n",
      "text/plain": [
       "<Figure size 792x432 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "from mpl_toolkits.mplot3d import Axes3D\n",
    "\n",
    "def plot_3D_decision_function(ax, w, b, x1_lim=[4, 6], x2_lim=[0.8, 2.8]):\n",
    "    x1_in_bounds = (X[:, 0] > x1_lim[0]) & (X[:, 0] < x1_lim[1])\n",
    "    X_crop = X[x1_in_bounds]\n",
    "    y_crop = y[x1_in_bounds]\n",
    "    x1s = np.linspace(x1_lim[0], x1_lim[1], 20)\n",
    "    x2s = np.linspace(x2_lim[0], x2_lim[1], 20)\n",
    "    x1, x2 = np.meshgrid(x1s, x2s)\n",
    "    xs = np.c_[x1.ravel(), x2.ravel()]\n",
    "    df = (xs.dot(w) + b).reshape(x1.shape)\n",
    "    m = 1 / np.linalg.norm(w)\n",
    "    boundary_x2s = -x1s*(w[0]/w[1])-b/w[1]\n",
    "    margin_x2s_1 = -x1s*(w[0]/w[1])-(b-1)/w[1]\n",
    "    margin_x2s_2 = -x1s*(w[0]/w[1])-(b+1)/w[1]\n",
    "    ax.plot_surface(x1s, x2, np.zeros_like(x1),\n",
    "                    color=\"b\", alpha=0.2, cstride=100, rstride=100)\n",
    "    ax.plot(x1s, boundary_x2s, 0, \"k-\", linewidth=2, label=r\"$h=0$\")\n",
    "    ax.plot(x1s, margin_x2s_1, 0, \"k--\", linewidth=2, label=r\"$h=\\pm 1$\")\n",
    "    ax.plot(x1s, margin_x2s_2, 0, \"k--\", linewidth=2)\n",
    "    ax.plot(X_crop[:, 0][y_crop==1], X_crop[:, 1][y_crop==1], 0, \"g^\")\n",
    "    ax.plot_wireframe(x1, x2, df, alpha=0.3, color=\"k\")\n",
    "    ax.plot(X_crop[:, 0][y_crop==0], X_crop[:, 1][y_crop==0], 0, \"bs\")\n",
    "    ax.axis(x1_lim + x2_lim)\n",
    "    ax.text(4.5, 2.5, 3.8, \"결정 함수 $h$\", fontsize=15)\n",
    "    ax.set_xlabel(r\"꽃잎 길이\", fontsize=15, labelpad=15)\n",
    "    ax.set_ylabel(r\"꽃잎 너비\", fontsize=15, rotation=25, labelpad=15)\n",
    "    ax.set_zlabel(r\"$h = \\mathbf{w}^t \\cdot \\mathbf{x} + b$\", fontsize=18, labelpad=10)\n",
    "    ax.legend(loc=\"upper left\", fontsize=16)\n",
    "\n",
    "fig = plt.figure(figsize=(11, 6))\n",
    "ax1 = fig.add_subplot(111, projection='3d')\n",
    "plot_3D_decision_function(ax1, w=svm_clf2.coef_[0], b=svm_clf2.intercept_[0])\n",
    "\n",
    "save_fig(\"iris_3D_plot\")\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 작은 가중치 벡터가 라지 마진을 만듭니다"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 33,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "[encoded data, 19544 characters]\n",
      "text/plain": [
       "<Figure size 864x230.4 with 2 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "def plot_2D_decision_function(w, b, ylabel=True, x1_lim=[-3, 3]):\n",
    "    x1 = np.linspace(x1_lim[0], x1_lim[1], 200)\n",
    "    y = w * x1 + b\n",
    "    m = 1 / w\n",
    "\n",
    "    plt.plot(x1, y)\n",
    "    plt.plot(x1_lim, [1, 1], \"k:\")\n",
    "    plt.plot(x1_lim, [-1, -1], \"k:\")\n",
    "    plt.axhline(y=0, color='k')\n",
    "    plt.axvline(x=0, color='k')\n",
    "    plt.plot([m, m], [0, 1], \"k--\")\n",
    "    plt.plot([-m, -m], [0, -1], \"k--\")\n",
    "    plt.plot([-m, m], [0, 0], \"k-o\", linewidth=3)\n",
    "    plt.axis(x1_lim + [-2, 2])\n",
    "    plt.xlabel(r\"$x_1$\", fontsize=16)\n",
    "    if ylabel:\n",
    "        plt.ylabel(r\"$w_1 x_1$  \", rotation=0, fontsize=16)\n",
    "    plt.title(r\"$w_1 = {}$\".format(w), fontsize=16)\n",
    "\n",
    "plt.figure(figsize=(12, 3.2))\n",
    "plt.subplot(121)\n",
    "plot_2D_decision_function(1, 0)\n",
    "plt.subplot(122)\n",
    "plot_2D_decision_function(0.5, 0, ylabel=False)\n",
    "save_fig(\"small_w_large_margin_plot\")\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 34,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([1.])"
      ]
     },
     "execution_count": 34,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from sklearn.svm import SVC\n",
    "from sklearn import datasets\n",
    "\n",
    "iris = datasets.load_iris()\n",
    "X = iris[\"data\"][:, (2, 3)] # 꽃잎 길이, 꽃잎 너비\n",
    "y = (iris[\"target\"] == 2).astype(np.float64) # Iris-Virginica\n",
    "\n",
    "svm_clf = SVC(kernel=\"linear\", C=1)\n",
    "svm_clf.fit(X, y)\n",
    "svm_clf.predict([[5.3, 1.3]])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 힌지 손실"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 35,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "[encoded data, 9492 characters]\n",
      "text/plain": [
       "<Figure size 360x201.6 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "t = np.linspace(-2, 4, 200)\n",
    "h = np.where(1 - t < 0, 0, 1 - t)  # max(0, 1-t)\n",
    "\n",
    "plt.figure(figsize=(5,2.8))\n",
    "plt.plot(t, h, \"b-\", linewidth=2, label=\"$max(0, 1 - t)$\")\n",
    "plt.grid(True, which='both')\n",
    "plt.axhline(y=0, color='k')\n",
    "plt.axvline(x=0, color='k')\n",
    "plt.yticks(np.arange(-1, 2.5, 1))\n",
    "plt.xlabel(\"$t$\", fontsize=16)\n",
    "plt.axis([-2, 4, -1, 2.5])\n",
    "plt.legend(loc=\"upper right\", fontsize=16)\n",
    "save_fig(\"hinge_plot\")\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 추가 내용"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 훈련 시간"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 36,
   "metadata": {
    "scrolled": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[<matplotlib.lines.Line2D at 0x7f4d717f2cc0>]"
      ]
     },
     "execution_count": 36,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "image/png": "[encoded data, 26288 characters]\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "X, y = make_moons(n_samples=1000, noise=0.4, random_state=42)\n",
    "plt.plot(X[:, 0][y==0], X[:, 1][y==0], \"bs\")\n",
    "plt.plot(X[:, 0][y==1], X[:, 1][y==1], \"g^\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 37,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[LibSVM]0 0.1 0.18813824653625488\n",
      "[LibSVM]1 0.01 0.1933610439300537\n",
      "[LibSVM]2 0.001 0.22491121292114258\n",
      "[LibSVM]3 0.0001 0.40105342864990234\n",
      "[LibSVM]4 1e-05 0.6449289321899414\n",
      "[LibSVM]5 1.0000000000000002e-06 0.6017529964447021\n",
      "[LibSVM]6 1.0000000000000002e-07 4.703836917877197\n",
      "[LibSVM]7 1.0000000000000002e-08 0.6525232791900635\n",
      "[LibSVM]8 1.0000000000000003e-09 0.642498254776001\n",
      "[LibSVM]9 1.0000000000000003e-10 0.6429502964019775\n"
     ]
    },
    {
     "data": {
      "image/png": "[encoded data, 10064 characters]\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "import time\n",
    "\n",
    "tol = 0.1\n",
    "tols = []\n",
    "times = []\n",
    "for i in range(10):\n",
    "    svm_clf = SVC(kernel=\"poly\", gamma=3, C=10, tol=tol, verbose=1)\n",
    "    t1 = time.time()\n",
    "    svm_clf.fit(X, y)\n",
    "    t2 = time.time()\n",
    "    times.append(t2-t1)\n",
    "    tols.append(tol)\n",
    "    print(i, tol, t2-t1)\n",
    "    tol /= 10\n",
    "plt.rcParams['font.family'] = 'stixgeneral'\n",
    "plt.semilogx(tols, times)\n",
    "plt.rcParams['font.family'] = 'NanumBarunGothic'"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 배치 경사 하강법을 사용한 선형 SVM 분류기 구현"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 38,
   "metadata": {},
   "outputs": [],
   "source": [
    "# 훈련 세트\n",
    "X = iris[\"data\"][:, (2, 3)] # 꽃잎 길이, 꽃잎 너비\n",
    "y = (iris[\"target\"] == 2).astype(np.float64).reshape(-1, 1) # Iris-Virginica"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 39,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([[1.],\n",
       "       [0.]])"
      ]
     },
     "execution_count": 39,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from sklearn.base import BaseEstimator\n",
    "\n",
    "class MyLinearSVC(BaseEstimator):\n",
    "    def __init__(self, C=1, eta0=1, eta_d=10000, n_epochs=1000, random_state=None):\n",
    "        self.C = C\n",
    "        self.eta0 = eta0\n",
    "        self.n_epochs = n_epochs\n",
    "        self.random_state = random_state\n",
    "        self.eta_d = eta_d\n",
    "\n",
    "    def eta(self, epoch):\n",
    "        return self.eta0 / (epoch + self.eta_d)\n",
    "        \n",
    "    def fit(self, X, y):\n",
    "        # Random initialization\n",
    "        if self.random_state:\n",
    "            np.random.seed(self.random_state)\n",
    "        w = np.random.randn(X.shape[1], 1) # n feature weights\n",
    "        b = 0\n",
    "\n",
    "        m = len(X)\n",
    "        t = y * 2 - 1  # -1 if t==0, +1 if t==1\n",
    "        X_t = X * t\n",
    "        self.Js=[]\n",
    "\n",
    "        # Training\n",
    "        for epoch in range(self.n_epochs):\n",
    "            support_vectors_idx = (X_t.dot(w) + t * b < 1).ravel()\n",
    "            X_t_sv = X_t[support_vectors_idx]\n",
    "            t_sv = t[support_vectors_idx]\n",
    "\n",
    "            J = 1/2 * np.sum(w * w) + self.C * (np.sum(1 - X_t_sv.dot(w)) - b * np.sum(t_sv))\n",
    "            self.Js.append(J)\n",
    "\n",
    "            w_gradient_vector = w - self.C * np.sum(X_t_sv, axis=0).reshape(-1, 1)\n",
    "            b_derivative = -C * np.sum(t_sv)\n",
    "                \n",
    "            w = w - self.eta(epoch) * w_gradient_vector\n",
    "            b = b - self.eta(epoch) * b_derivative\n",
    "            \n",
    "\n",
    "        self.intercept_ = np.array([b])\n",
    "        self.coef_ = np.array([w])\n",
    "        support_vectors_idx = (X_t.dot(w) + t * b < 1).ravel()\n",
    "        self.support_vectors_ = X[support_vectors_idx]\n",
    "        return self\n",
    "\n",
    "    def decision_function(self, X):\n",
    "        return X.dot(self.coef_[0]) + self.intercept_[0]\n",
    "\n",
    "    def predict(self, X):\n",
    "        return (self.decision_function(X) >= 0).astype(np.float64)\n",
    "\n",
    "C=2\n",
    "svm_clf = MyLinearSVC(C=C, eta0 = 10, eta_d = 1000, n_epochs=60000, random_state=2)\n",
    "svm_clf.fit(X, y)\n",
    "svm_clf.predict(np.array([[5, 2], [4, 1]]))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 40,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[0, 60000, 0, 100]"
      ]
     },
     "execution_count": 40,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "image/png": "[encoded data, 7588 characters]\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "plt.plot(range(svm_clf.n_epochs), svm_clf.Js)\n",
    "plt.axis([0, svm_clf.n_epochs, 0, 100])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 41,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[-15.56780998] [[[2.28129013]\n",
      "  [2.71597487]]]\n"
     ]
    }
   ],
   "source": [
    "print(svm_clf.intercept_, svm_clf.coef_)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 42,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[-15.51721253] [[2.27128546 2.71287145]]\n"
     ]
    }
   ],
   "source": [
    "svm_clf2 = SVC(kernel=\"linear\", C=C)\n",
    "svm_clf2.fit(X, y.ravel())\n",
    "print(svm_clf2.intercept_, svm_clf2.coef_)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 43,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[4, 6, 0.8, 2.8]"
      ]
     },
     "execution_count": 43,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "image/png": "[encoded data, 44836 characters]\n",
      "text/plain": [
       "<Figure size 864x230.4 with 2 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "yr = y.ravel()\n",
    "plt.figure(figsize=(12,3.2))\n",
    "plt.subplot(121)\n",
    "plt.plot(X[:, 0][yr==1], X[:, 1][yr==1], \"g^\", label=\"Iris-Virginica\")\n",
    "plt.plot(X[:, 0][yr==0], X[:, 1][yr==0], \"bs\", label=\"Not Iris-Virginica\")\n",
    "plot_svc_decision_boundary(svm_clf, 4, 6)\n",
    "plt.xlabel(\"꽃잎 길이\", fontsize=14)\n",
    "plt.ylabel(\"꽃잎 너비\", fontsize=14)\n",
    "plt.title(\"MyLinearSVC\", fontsize=14)\n",
    "plt.axis([4, 6, 0.8, 2.8])\n",
    "\n",
    "plt.subplot(122)\n",
    "plt.plot(X[:, 0][yr==1], X[:, 1][yr==1], \"g^\")\n",
    "plt.plot(X[:, 0][yr==0], X[:, 1][yr==0], \"bs\")\n",
    "plot_svc_decision_boundary(svm_clf2, 4, 6)\n",
    "plt.xlabel(\"꽃잎 길이\", fontsize=14)\n",
    "plt.title(\"SVC\", fontsize=14)\n",
    "plt.axis([4, 6, 0.8, 2.8])\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 44,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[-14.062485     2.24179316   1.79750198]\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "[4, 6, 0.8, 2.8]"
      ]
     },
     "execution_count": 44,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "image/png": "[encoded data, 27508 characters]\n",
      "text/plain": [
       "<Figure size 396x230.4 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "from sklearn.linear_model import SGDClassifier\n",
    "\n",
    "sgd_clf = SGDClassifier(loss=\"hinge\", alpha = 0.017, max_iter = 50, random_state=42)\n",
    "sgd_clf.fit(X, y.ravel())\n",
    "\n",
    "m = len(X)\n",
    "t = y * 2 - 1  # -1 if t==0, +1 if t==1\n",
    "X_b = np.c_[np.ones((m, 1)), X]  # 편향 x0=1을 추가\n",
    "X_b_t = X_b * t\n",
    "sgd_theta = np.r_[sgd_clf.intercept_[0], sgd_clf.coef_[0]]\n",
    "print(sgd_theta)\n",
    "support_vectors_idx = (X_b_t.dot(sgd_theta) < 1).ravel()\n",
    "sgd_clf.support_vectors_ = X[support_vectors_idx]\n",
    "sgd_clf.C = C\n",
    "\n",
    "plt.figure(figsize=(5.5,3.2))\n",
    "plt.plot(X[:, 0][yr==1], X[:, 1][yr==1], \"g^\")\n",
    "plt.plot(X[:, 0][yr==0], X[:, 1][yr==0], \"bs\")\n",
    "plot_svc_decision_boundary(sgd_clf, 4, 6)\n",
    "plt.xlabel(\"꽃잎 길이\", fontsize=14)\n",
    "plt.ylabel(\"꽃잎 너비\", fontsize=14)\n",
    "plt.title(\"SGDClassifier\", fontsize=14)\n",
    "plt.axis([4, 6, 0.8, 2.8])\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 연습문제 해답"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 1. to 7."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true
   },
   "source": [
    "부록 A 참조."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 8."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "_문제: 선형적으로 분리되는 데이터셋에 `LinearSVC`를 훈련시켜보세요. 그런 다음 같은 데이터셋에 `SVC`와`SGDClassifier`를 적용해보세요. 거의 비슷한 모델이 만들어지는지 확인해보세요._"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Iris 데이터셋을 사용하겠습니다. Iris Setosa와 Iris Versicolor 클래스는 선형적으로 구분이 가능합니다."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 45,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn import datasets\n",
    "\n",
    "iris = datasets.load_iris()\n",
    "X = iris[\"data\"][:, (2, 3)]  # petal length, petal width\n",
    "y = iris[\"target\"]\n",
    "\n",
    "setosa_or_versicolor = (y == 0) | (y == 1)\n",
    "X = X[setosa_or_versicolor]\n",
    "y = y[setosa_or_versicolor]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 46,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "LinearSVC:                    [0.28480668] [[1.05542422 1.09851637]]\n",
      "SVC:                          [0.31933577] [[1.1223101  1.02531081]]\n",
      "SGDClassifier(alpha=0.00200): [0.32] [[1.12293103 1.02620763]]\n"
     ]
    }
   ],
   "source": [
    "from sklearn.svm import SVC, LinearSVC\n",
    "from sklearn.linear_model import SGDClassifier\n",
    "from sklearn.preprocessing import StandardScaler\n",
    "\n",
    "C = 5\n",
    "alpha = 1 / (C * len(X))\n",
    "\n",
    "lin_clf = LinearSVC(loss=\"hinge\", C=C, random_state=42)\n",
    "svm_clf = SVC(kernel=\"linear\", C=C)\n",
    "sgd_clf = SGDClassifier(loss=\"hinge\", learning_rate=\"constant\", eta0=0.001, alpha=alpha,\n",
    "                        max_iter=100000, random_state=42)\n",
    "\n",
    "scaler = StandardScaler()\n",
    "X_scaled = scaler.fit_transform(X)\n",
    "\n",
    "lin_clf.fit(X_scaled, y)\n",
    "svm_clf.fit(X_scaled, y)\n",
    "sgd_clf.fit(X_scaled, y)\n",
    "\n",
    "print(\"LinearSVC:                   \", lin_clf.intercept_, lin_clf.coef_)\n",
    "print(\"SVC:                         \", svm_clf.intercept_, svm_clf.coef_)\n",
    "print(\"SGDClassifier(alpha={:.5f}):\".format(sgd_clf.alpha), sgd_clf.intercept_, sgd_clf.coef_)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "이 세개 모델의 결정 경계를 그려 보겠습니다:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 47,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "[encoded data, 32844 characters]\n",
      "text/plain": [
       "<Figure size 792x288 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "# 각 결정 경계의 기울기와 편향을 계산합니다\n",
    "w1 = -lin_clf.coef_[0, 0]/lin_clf.coef_[0, 1]\n",
    "b1 = -lin_clf.intercept_[0]/lin_clf.coef_[0, 1]\n",
    "w2 = -svm_clf.coef_[0, 0]/svm_clf.coef_[0, 1]\n",
    "b2 = -svm_clf.intercept_[0]/svm_clf.coef_[0, 1]\n",
    "w3 = -sgd_clf.coef_[0, 0]/sgd_clf.coef_[0, 1]\n",
    "b3 = -sgd_clf.intercept_[0]/sgd_clf.coef_[0, 1]\n",
    "\n",
    "# 결정 경계를 원본 스케일로 변환합니다\n",
    "line1 = scaler.inverse_transform([[-10, -10 * w1 + b1], [10, 10 * w1 + b1]])\n",
    "line2 = scaler.inverse_transform([[-10, -10 * w2 + b2], [10, 10 * w2 + b2]])\n",
    "line3 = scaler.inverse_transform([[-10, -10 * w3 + b3], [10, 10 * w3 + b3]])\n",
    "\n",
    "# 세 개의 결정 경계를 모두 그립니다\n",
    "plt.figure(figsize=(11, 4))\n",
    "plt.plot(line1[:, 0], line1[:, 1], \"k:\", label=\"LinearSVC\")\n",
    "plt.plot(line2[:, 0], line2[:, 1], \"b--\", linewidth=2, label=\"SVC\")\n",
    "plt.plot(line3[:, 0], line3[:, 1], \"r-\", label=\"SGDClassifier\")\n",
    "plt.plot(X[:, 0][y==1], X[:, 1][y==1], \"bs\") # label=\"Iris-Versicolor\"\n",
    "plt.plot(X[:, 0][y==0], X[:, 1][y==0], \"yo\") # label=\"Iris-Setosa\"\n",
    "plt.xlabel(\"꽃잎 길이\", fontsize=14)\n",
    "plt.ylabel(\"꽃잎 너비\", fontsize=14)\n",
    "plt.legend(loc=\"upper center\", fontsize=14)\n",
    "plt.axis([0, 5.5, 0, 2])\n",
    "\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "아주 비슷하네요!"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 9."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "_문제: MNIST 데이터셋에 SVM 분류기를 훈련시켜보세요. SVM 분류기는 이진 분류기라서 OvA 전략을 사용해 10개의 숫자를 분류해야 합니다. 처리 속도를 높이기 위해 작은 검증 세트로 하이퍼파라미터를 조정하는 것이 좋습니다. 어느 정도까지 정확도를 높일 수 있나요?_"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "먼저 데이터셋을 로드하고 훈련 세트와 테스트 세트로 나눕니다. `train_test_split()` 함수를 사용할 수 있지만 보통 처음 60,000개의 샘플을 훈련 세트로 사용하고 나머지는 10,000개를 테스트 세트로 사용합니다(이렇게 하면 다른 사람들의 모델과 성능을 비교하기 좋습니다):"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 48,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.datasets import fetch_mldata\n",
    "\n",
    "mnist = fetch_mldata(\"MNIST original\")\n",
    "X = mnist[\"data\"]\n",
    "y = mnist[\"target\"]\n",
    "\n",
    "X_train = X[:60000]\n",
    "y_train = y[:60000]\n",
    "X_test = X[60000:]\n",
    "y_test = y[60000:]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "많은 훈련 알고리즘은 훈련 샘플의 순서에 민감하므로 먼저 이를 섞는 것이 좋은 습관입니다:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 49,
   "metadata": {},
   "outputs": [],
   "source": [
    "np.random.seed(42)\n",
    "rnd_idx = np.random.permutation(60000)\n",
    "X_train = X_train[rnd_idx]\n",
    "y_train = y_train[rnd_idx]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "선형 SVM 분류기부터 시작해보죠. 이 모델은 자동으로 OvA(또는 OvR) 전략을 사용하므로 특별히 처리해 줄 것이 없습니다. 간단하네요!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 50,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "LinearSVC(C=1.0, class_weight=None, dual=True, fit_intercept=True,\n",
       "     intercept_scaling=1, loss='squared_hinge', max_iter=1000,\n",
       "     multi_class='ovr', penalty='l2', random_state=42, tol=0.0001,\n",
       "     verbose=0)"
      ]
     },
     "execution_count": 50,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "lin_clf = LinearSVC(random_state=42)\n",
    "lin_clf.fit(X_train, y_train)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "훈련 세트에 대한 예측을 만들어 정확도를 측정해 보겠습니다(최종 모델을 선택해 훈련시킨 것이 아니기 때문에 아직 테스트 세트를 사용해서는 안됩니다):"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 51,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.8263333333333334"
      ]
     },
     "execution_count": 51,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from sklearn.metrics import accuracy_score\n",
    "\n",
    "y_pred = lin_clf.predict(X_train)\n",
    "accuracy_score(y_train, y_pred)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "와우, MNIST에서 82% 정확도면 정말 나쁜 성능입니다. 선형 모델이 MNIST 문제에 너무 단순하기 때문이지만 먼저 데이터의 스케일을 조정할 필요가 있습니다:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 52,
   "metadata": {},
   "outputs": [],
   "source": [
    "scaler = StandardScaler()\n",
    "X_train_scaled = scaler.fit_transform(X_train.astype(np.float32))\n",
    "X_test_scaled = scaler.transform(X_test.astype(np.float32))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 53,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "LinearSVC(C=1.0, class_weight=None, dual=True, fit_intercept=True,\n",
       "     intercept_scaling=1, loss='squared_hinge', max_iter=1000,\n",
       "     multi_class='ovr', penalty='l2', random_state=42, tol=0.0001,\n",
       "     verbose=0)"
      ]
     },
     "execution_count": 53,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "lin_clf = LinearSVC(random_state=42)\n",
    "lin_clf.fit(X_train_scaled, y_train)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 54,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.9224"
      ]
     },
     "execution_count": 54,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "y_pred = lin_clf.predict(X_train_scaled)\n",
    "accuracy_score(y_train, y_pred)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "훨씬 나아졌지만(에러율을 절반으로 줄였습니다) 여전히 MNIST에서 좋은 성능은 아닙니다. SVM을 사용한다면 커널 함수를 사용해야 합니다. RBF 커널(기본값)로 `SVC`를 적용해 보겠습니다.\n",
    "\n",
    "**경고**: 사이킷런 0.19버전 이하를 사용하면 기본적으로 OvO 전략을 사용할 것이므로 `decision_function_shape=\"ovr\"`로 지정해 OvR 전략으로 바꾸어 주어야 합니다(OvR은 0.19 버전부터 기본이 되었습니다)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 55,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "SVC(C=1.0, cache_size=200, class_weight=None, coef0=0.0,\n",
       "  decision_function_shape='ovr', degree=3, gamma='auto', kernel='rbf',\n",
       "  max_iter=-1, probability=False, random_state=None, shrinking=True,\n",
       "  tol=0.001, verbose=False)"
      ]
     },
     "execution_count": 55,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "svm_clf = SVC(decision_function_shape=\"ovr\")\n",
    "svm_clf.fit(X_train_scaled[:10000], y_train[:10000])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 56,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.94615"
      ]
     },
     "execution_count": 56,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "y_pred = svm_clf.predict(X_train_scaled)\n",
    "accuracy_score(y_train, y_pred)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "아주 좋네요 6배나 적은 데이터에서 모델을 훈련시켰지만 더 좋은 성능을 얻었습니다. 교차 검증을 사용한 랜덤 서치로 하이퍼파라미터 튜닝을 해보겠습니다. 진행을 빠르게 하기 위해 작은 데이터셋으로 작업하겠습니다:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 57,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Fitting 3 folds for each of 10 candidates, totalling 30 fits\n",
      "[CV] gamma=0.001766074650481071, C=8.852316058423087 .................\n",
      "[CV] gamma=0.001766074650481071, C=8.852316058423087 .................\n",
      "[CV] gamma=0.006364737055453384, C=1.8271960104746645 ................\n",
      "[CV] gamma=0.001766074650481071, C=8.852316058423087 .................\n",
      "[CV] .. gamma=0.001766074650481071, C=8.852316058423087, total=   0.8s\n",
      "[CV] gamma=0.006364737055453384, C=1.8271960104746645 ................\n",
      "[CV] .. gamma=0.001766074650481071, C=8.852316058423087, total=   0.8s\n",
      "[CV] gamma=0.006364737055453384, C=1.8271960104746645 ................\n",
      "[CV] .. gamma=0.001766074650481071, C=8.852316058423087, total=   0.8s\n",
      "[CV] gamma=0.051349833451870636, C=9.875199193765326 .................\n",
      "[CV] . gamma=0.006364737055453384, C=1.8271960104746645, total=   0.9s\n",
      "[CV] gamma=0.051349833451870636, C=9.875199193765326 .................\n",
      "[CV] . gamma=0.006364737055453384, C=1.8271960104746645, total=   0.9s\n",
      "[CV] gamma=0.051349833451870636, C=9.875199193765326 .................\n",
      "[CV] . gamma=0.006364737055453384, C=1.8271960104746645, total=   0.9s\n",
      "[CV] gamma=0.05991666578466177, C=6.59992909281409 ...................\n",
      "[CV] .. gamma=0.051349833451870636, C=9.875199193765326, total=   1.0s\n",
      "[CV] gamma=0.05991666578466177, C=6.59992909281409 ...................\n",
      "[CV] .. gamma=0.051349833451870636, C=9.875199193765326, total=   1.0s\n",
      "[CV] gamma=0.05991666578466177, C=6.59992909281409 ...................\n",
      "[CV] .. gamma=0.051349833451870636, C=9.875199193765326, total=   1.0s\n",
      "[CV] gamma=0.003596490522533181, C=9.053435975487119 .................\n",
      "[CV] .... gamma=0.05991666578466177, C=6.59992909281409, total=   1.0s\n",
      "[CV] gamma=0.003596490522533181, C=9.053435975487119 .................\n",
      "[CV] .... gamma=0.05991666578466177, C=6.59992909281409, total=   1.0s\n",
      "[CV] gamma=0.003596490522533181, C=9.053435975487119 .................\n",
      "[CV] .... gamma=0.05991666578466177, C=6.59992909281409, total=   1.0s\n",
      "[CV] gamma=0.004002330992905356, C=2.701062804458301 .................\n",
      "[CV] .. gamma=0.003596490522533181, C=9.053435975487119, total=   0.9s\n",
      "[CV] gamma=0.004002330992905356, C=2.701062804458301 .................\n",
      "[CV] .. gamma=0.003596490522533181, C=9.053435975487119, total=   0.9s\n",
      "[CV] gamma=0.004002330992905356, C=2.701062804458301 .................\n",
      "[CV] .. gamma=0.003596490522533181, C=9.053435975487119, total=   0.9s\n",
      "[CV] gamma=0.017596957507461645, C=3.2711787843881437 ................\n",
      "[CV] .. gamma=0.004002330992905356, C=2.701062804458301, total=   0.9s\n",
      "[CV] gamma=0.017596957507461645, C=3.2711787843881437 ................\n",
      "[CV] .. gamma=0.004002330992905356, C=2.701062804458301, total=   0.9s\n",
      "[CV] gamma=0.017596957507461645, C=3.2711787843881437 ................\n",
      "[CV] .. gamma=0.004002330992905356, C=2.701062804458301, total=   0.9s\n",
      "[CV] gamma=0.01573529056426603, C=6.848991127746501 ..................\n",
      "[CV] . gamma=0.017596957507461645, C=3.2711787843881437, total=   1.0s\n",
      "[CV] gamma=0.01573529056426603, C=6.848991127746501 ..................\n",
      "[CV] . gamma=0.017596957507461645, C=3.2711787843881437, total=   1.0s\n",
      "[CV] gamma=0.01573529056426603, C=6.848991127746501 ..................\n",
      "[CV] . gamma=0.017596957507461645, C=3.2711787843881437, total=   1.0s\n",
      "[CV] gamma=0.03834647526105027, C=2.893035364914488 ..................\n",
      "[CV] ... gamma=0.01573529056426603, C=6.848991127746501, total=   1.0s\n",
      "[CV] gamma=0.03834647526105027, C=2.893035364914488 ..................\n",
      "[CV] ... gamma=0.01573529056426603, C=6.848991127746501, total=   1.0s\n",
      "[CV] gamma=0.03834647526105027, C=2.893035364914488 ..................\n",
      "[CV] ... gamma=0.01573529056426603, C=6.848991127746501, total=   1.0s\n",
      "[CV] gamma=0.008808538172595842, C=5.336260835426313 .................\n",
      "[CV] ... gamma=0.03834647526105027, C=2.893035364914488, total=   1.0s\n",
      "[CV] gamma=0.008808538172595842, C=5.336260835426313 .................\n",
      "[CV] ... gamma=0.03834647526105027, C=2.893035364914488, total=   1.0s\n",
      "[CV] gamma=0.008808538172595842, C=5.336260835426313 .................\n",
      "[CV] ... gamma=0.03834647526105027, C=2.893035364914488, total=   1.0s\n",
      "[CV] .. gamma=0.008808538172595842, C=5.336260835426313, total=   0.9s\n",
      "[CV] .. gamma=0.008808538172595842, C=5.336260835426313, total=   0.9s\n",
      "[CV] .. gamma=0.008808538172595842, C=5.336260835426313, total=   0.9s\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "[Parallel(n_jobs=-1)]: Done  30 out of  30 | elapsed:   10.6s finished\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "RandomizedSearchCV(cv=None, error_score='raise',\n",
       "          estimator=SVC(C=1.0, cache_size=200, class_weight=None, coef0=0.0,\n",
       "  decision_function_shape='ovr', degree=3, gamma='auto', kernel='rbf',\n",
       "  max_iter=-1, probability=False, random_state=None, shrinking=True,\n",
       "  tol=0.001, verbose=False),\n",
       "          fit_params=None, iid=True, n_iter=10, n_jobs=-1,\n",
       "          param_distributions={'gamma': <scipy.stats._distn_infrastructure.rv_frozen object at 0x7f4d6bfdcf60>, 'C': <scipy.stats._distn_infrastructure.rv_frozen object at 0x7f4d6bfdc518>},\n",
       "          pre_dispatch='2*n_jobs', random_state=None, refit=True,\n",
       "          return_train_score='warn', scoring=None, verbose=2)"
      ]
     },
     "execution_count": 57,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from sklearn.model_selection import RandomizedSearchCV\n",
    "from scipy.stats import reciprocal, uniform\n",
    "\n",
    "param_distributions = {\"gamma\": reciprocal(0.001, 0.1), \"C\": uniform(1, 10)}\n",
    "rnd_search_cv = RandomizedSearchCV(svm_clf, param_distributions, n_iter=10, verbose=2, n_jobs=-1)\n",
    "rnd_search_cv.fit(X_train_scaled[:1000], y_train[:1000])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 58,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "SVC(C=8.852316058423087, cache_size=200, class_weight=None, coef0=0.0,\n",
       "  decision_function_shape='ovr', degree=3, gamma=0.001766074650481071,\n",
       "  kernel='rbf', max_iter=-1, probability=False, random_state=None,\n",
       "  shrinking=True, tol=0.001, verbose=False)"
      ]
     },
     "execution_count": 58,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "rnd_search_cv.best_estimator_"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 59,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.856"
      ]
     },
     "execution_count": 59,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "rnd_search_cv.best_score_"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "이 점수는 낮지만 1,000개의 샘플만 사용한 것을 기억해야 합니다. 전체 데이터셋으로 최선의 모델을 재훈련시켜 보겠습니다(몇 시간이 걸릴지 모르니 퇴근하기 전에 돌려 보세요):"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 60,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "SVC(C=8.852316058423087, cache_size=200, class_weight=None, coef0=0.0,\n",
       "  decision_function_shape='ovr', degree=3, gamma=0.001766074650481071,\n",
       "  kernel='rbf', max_iter=-1, probability=False, random_state=None,\n",
       "  shrinking=True, tol=0.001, verbose=False)"
      ]
     },
     "execution_count": 60,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "rnd_search_cv.best_estimator_.fit(X_train_scaled, y_train)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 61,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.99965"
      ]
     },
     "execution_count": 61,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "y_pred = rnd_search_cv.best_estimator_.predict(X_train_scaled)\n",
    "accuracy_score(y_train, y_pred)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "아주 훌륭하네요! 이 모델을 선택하겠습니다. 이제 테스트 세트로 모델을 테스트합니다:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 62,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.9709"
      ]
     },
     "execution_count": 62,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "y_pred = rnd_search_cv.best_estimator_.predict(X_test_scaled)\n",
    "accuracy_score(y_test, y_pred)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "아주 나쁘지 않지만 확실히 모델이 다소 과대적합되었습니다. 하이퍼파라미터를 조금 더 수정할 수 있지만(가령, `C`와/나 `gamma`를 감소시킵니다) 그렇게 하면 테스트 세트에 과대적합될 위험이 있습니다. 다른 사람들은 하이퍼파라미터 `C=5`와 `gamma=0.005`에서 더 나은 성능(98% 이상의 정확도)을 얻었습니다. 훈련 세트를 더 많이 사용해서 더 오래 랜덤 서치를 수행하면 이런 값을 얻을 수 있을지 모릅니다."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 10."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "_문제: 캘리포니아 주택 가격 데이터셋에 SVM 회귀를 훈련시켜보세요._"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "사이킷런의 `fetch_california_housing()` 함수를 사용해 데이터셋을 로드합니다:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 63,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.datasets import fetch_california_housing\n",
    "\n",
    "housing = fetch_california_housing()\n",
    "X = housing[\"data\"]\n",
    "y = housing[\"target\"]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "훈련 세트와 테스트 세트로 나눕니다:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 64,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.model_selection import train_test_split\n",
    "\n",
    "X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, random_state=42)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "데이터의 스케일을 조정하는 것을 잊지 마세요:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 65,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.preprocessing import StandardScaler\n",
    "\n",
    "scaler = StandardScaler()\n",
    "X_train_scaled = scaler.fit_transform(X_train)\n",
    "X_test_scaled = scaler.transform(X_test)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "먼저 간단한 `LinearSVR`을 훈련시켜 보죠:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 66,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "LinearSVR(C=1.0, dual=True, epsilon=0.0, fit_intercept=True,\n",
       "     intercept_scaling=1.0, loss='epsilon_insensitive', max_iter=1000,\n",
       "     random_state=42, tol=0.0001, verbose=0)"
      ]
     },
     "execution_count": 66,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from sklearn.svm import LinearSVR\n",
    "\n",
    "lin_svr = LinearSVR(random_state=42)\n",
    "lin_svr.fit(X_train_scaled, y_train)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "훈련 세트에 대한 성능을 확인해 보겠습니다:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 67,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.9499688222172291"
      ]
     },
     "execution_count": 67,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from sklearn.metrics import mean_squared_error\n",
    "\n",
    "y_pred = lin_svr.predict(X_train_scaled)\n",
    "mse = mean_squared_error(y_train, y_pred)\n",
    "mse"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "RMSE를 확인해 보겠습니다:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 68,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.9746634404845752"
      ]
     },
     "execution_count": 68,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "np.sqrt(mse)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "훈련 세트에서 타깃은 만달러 단위입니다. RMSE는 기대할 수 있는 에러의 정도를 대략 가늠하게 도와줍니다(에러가 클수록 큰 폭으로 증가합니다). 이 모델의 에러가 대략 $10,000 정도로 예상할 수 있습니다. 썩 훌륭하지 않네요. RBF 커널이 더 나을지 확인해 보겠습니다. 하이퍼파라미터 `C`와 `gamma`의 적절한 값을 찾기 위해 교차 검증을 사용한 랜덤 서치를 적용하겠습니다:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 69,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Fitting 3 folds for each of 10 candidates, totalling 30 fits\n",
      "[CV] gamma=0.07969454818643928, C=4.745401188473625 ..................\n",
      "[CV] gamma=0.07969454818643928, C=4.745401188473625 ..................\n",
      "[CV] gamma=0.015751320499779724, C=8.31993941811405 ..................\n",
      "[CV] gamma=0.07969454818643928, C=4.745401188473625 ..................\n",
      "[CV] ... gamma=0.015751320499779724, C=8.31993941811405, total=   5.9s\n",
      "[CV] gamma=0.015751320499779724, C=8.31993941811405 ..................\n",
      "[CV] ... gamma=0.07969454818643928, C=4.745401188473625, total=   6.2s\n",
      "[CV] gamma=0.015751320499779724, C=8.31993941811405 ..................\n",
      "[CV] ... gamma=0.07969454818643928, C=4.745401188473625, total=   6.2s\n",
      "[CV] gamma=0.002051110418843397, C=2.560186404424365 .................\n",
      "[CV] ... gamma=0.07969454818643928, C=4.745401188473625, total=   6.2s\n",
      "[CV] gamma=0.002051110418843397, C=2.560186404424365 .................\n",
      "[CV] .. gamma=0.002051110418843397, C=2.560186404424365, total=   5.4s\n",
      "[CV] gamma=0.002051110418843397, C=2.560186404424365 .................\n",
      "[CV] ... gamma=0.015751320499779724, C=8.31993941811405, total=   5.7s\n",
      "[CV] gamma=0.05399484409787431, C=1.5808361216819946 .................\n",
      "[CV] .. gamma=0.002051110418843397, C=2.560186404424365, total=   5.5s\n",
      "[CV] gamma=0.05399484409787431, C=1.5808361216819946 .................\n",
      "[CV] ... gamma=0.015751320499779724, C=8.31993941811405, total=   5.8s\n",
      "[CV] gamma=0.05399484409787431, C=1.5808361216819946 .................\n",
      "[CV] .. gamma=0.05399484409787431, C=1.5808361216819946, total=   5.4s\n",
      "[CV] gamma=0.026070247583707663, C=7.011150117432088 .................\n",
      "[CV] .. gamma=0.002051110418843397, C=2.560186404424365, total=   5.5s\n",
      "[CV] gamma=0.026070247583707663, C=7.011150117432088 .................\n",
      "[CV] .. gamma=0.05399484409787431, C=1.5808361216819946, total=   5.4s\n",
      "[CV] gamma=0.026070247583707663, C=7.011150117432088 .................\n",
      "[CV] .. gamma=0.05399484409787431, C=1.5808361216819946, total=   5.4s\n",
      "[CV] gamma=0.0870602087830485, C=1.2058449429580245 ..................\n",
      "[CV] ... gamma=0.0870602087830485, C=1.2058449429580245, total=   5.3s\n",
      "[CV] gamma=0.0870602087830485, C=1.2058449429580245 ..................\n",
      "[CV] .. gamma=0.026070247583707663, C=7.011150117432088, total=   5.8s\n",
      "[CV] gamma=0.0870602087830485, C=1.2058449429580245 ..................\n",
      "[CV] .. gamma=0.026070247583707663, C=7.011150117432088, total=   5.8s\n",
      "[CV] gamma=0.0026587543983272693, C=9.324426408004218 ................\n",
      "[CV] .. gamma=0.026070247583707663, C=7.011150117432088, total=   6.0s\n",
      "[CV] gamma=0.0026587543983272693, C=9.324426408004218 ................\n",
      "[CV] ... gamma=0.0870602087830485, C=1.2058449429580245, total=   5.4s\n",
      "[CV] gamma=0.0026587543983272693, C=9.324426408004218 ................\n",
      "[CV] ... gamma=0.0870602087830485, C=1.2058449429580245, total=   5.3s\n",
      "[CV] gamma=0.0023270677083837795, C=2.818249672071006 ................\n",
      "[CV] . gamma=0.0026587543983272693, C=9.324426408004218, total=   5.6s\n",
      "[CV] gamma=0.0023270677083837795, C=2.818249672071006 ................\n",
      "[CV] . gamma=0.0026587543983272693, C=9.324426408004218, total=   5.5s\n",
      "[CV] gamma=0.0023270677083837795, C=2.818249672071006 ................\n",
      "[CV] . gamma=0.0026587543983272693, C=9.324426408004218, total=   5.5s\n",
      "[CV] gamma=0.011207606211860567, C=4.042422429595377 .................\n",
      "[CV] . gamma=0.0023270677083837795, C=2.818249672071006, total=   5.4s\n",
      "[CV] gamma=0.011207606211860567, C=4.042422429595377 .................\n",
      "[CV] . gamma=0.0023270677083837795, C=2.818249672071006, total=   5.5s\n",
      "[CV] gamma=0.011207606211860567, C=4.042422429595377 .................\n",
      "[CV] . gamma=0.0023270677083837795, C=2.818249672071006, total=   5.4s\n",
      "[CV] gamma=0.003823475224675185, C=5.319450186421157 .................\n",
      "[CV] .. gamma=0.011207606211860567, C=4.042422429595377, total=   5.4s\n",
      "[CV] gamma=0.003823475224675185, C=5.319450186421157 .................\n",
      "[CV] .. gamma=0.011207606211860567, C=4.042422429595377, total=   5.5s\n",
      "[CV] gamma=0.003823475224675185, C=5.319450186421157 .................\n",
      "[CV] .. gamma=0.011207606211860567, C=4.042422429595377, total=   5.5s\n",
      "[CV] .. gamma=0.003823475224675185, C=5.319450186421157, total=   5.4s\n",
      "[CV] .. gamma=0.003823475224675185, C=5.319450186421157, total=   5.3s\n",
      "[CV] .. gamma=0.003823475224675185, C=5.319450186421157, total=   5.3s\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "[Parallel(n_jobs=-1)]: Done  30 out of  30 | elapsed:  1.1min finished\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "RandomizedSearchCV(cv=None, error_score='raise',\n",
       "          estimator=SVR(C=1.0, cache_size=200, coef0=0.0, degree=3, epsilon=0.1, gamma='auto',\n",
       "  kernel='rbf', max_iter=-1, shrinking=True, tol=0.001, verbose=False),\n",
       "          fit_params=None, iid=True, n_iter=10, n_jobs=-1,\n",
       "          param_distributions={'gamma': <scipy.stats._distn_infrastructure.rv_frozen object at 0x7f4d70d2cac8>, 'C': <scipy.stats._distn_infrastructure.rv_frozen object at 0x7f4d70d2cb70>},\n",
       "          pre_dispatch='2*n_jobs', random_state=42, refit=True,\n",
       "          return_train_score='warn', scoring=None, verbose=2)"
      ]
     },
     "execution_count": 69,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from sklearn.svm import SVR\n",
    "from sklearn.model_selection import RandomizedSearchCV\n",
    "from scipy.stats import reciprocal, uniform\n",
    "\n",
    "param_distributions = {\"gamma\": reciprocal(0.001, 0.1), \"C\": uniform(1, 10)}\n",
    "rnd_search_cv = RandomizedSearchCV(SVR(), param_distributions, n_iter=10, verbose=2, random_state=42, n_jobs=-1)\n",
    "rnd_search_cv.fit(X_train_scaled, y_train)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 70,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "SVR(C=4.745401188473625, cache_size=200, coef0=0.0, degree=3, epsilon=0.1,\n",
       "  gamma=0.07969454818643928, kernel='rbf', max_iter=-1, shrinking=True,\n",
       "  tol=0.001, verbose=False)"
      ]
     },
     "execution_count": 70,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "rnd_search_cv.best_estimator_"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "이제 훈련 세트에서 RMSE를 측정해 보겠습니다:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 71,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.5727524770785356"
      ]
     },
     "execution_count": 71,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "y_pred = rnd_search_cv.best_estimator_.predict(X_train_scaled)\n",
    "mse = mean_squared_error(y_train, y_pred)\n",
    "np.sqrt(mse)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "선형 모델보다 훨씬 나아졌네요. 이 모델을 선택하고 테스트 세트에서 평가해 보겠습니다:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 72,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.592916838552874"
      ]
     },
     "execution_count": 72,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "y_pred = rnd_search_cv.best_estimator_.predict(X_test_scaled)\n",
    "mse = mean_squared_error(y_test, y_pred)\n",
    "np.sqrt(mse)"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.5.5"
  },
  "nav_menu": {},
  "toc": {
   "navigate_menu": true,
   "number_sections": true,
   "sideBar": true,
   "threshold": 6,
   "toc_cell": false,
   "toc_section_display": "block",
   "toc_window_display": false
  }
 },
 "nbformat": 4,
 "nbformat_minor": 1
}
