{ "cells": [ { "cell_type": "markdown", "id": "3f5d1031-4a3c-4bd7-9089-4766917156f2", "metadata": {}, "source": [ "# Time to Merge Prediction Inference Service\n", "\n", "In the previous notebook, we explored some basic machine learning models for predicting time to merge of a PR. We then deployed the model with the highest f1-score as a service using Seldon. The purpose of this notebook is to check whether this service is running as intended, and more specifically to ensure that the model performance is what we expect it to be. So here, we will use the test set from the aforementioned notebook as the query payload for the service, and then verify that the return values are the same as those obtained during training/testing locally." ] }, { "cell_type": "code", "execution_count": 1, "id": "091c5463", "metadata": {}, "outputs": [ { "data": { "text/plain": [ "True" ] }, "execution_count": 1, "metadata": {}, "output_type": "execute_result" } ], "source": [ "import sys\n", "import json\n", "import os\n", "import requests\n", "from dotenv import load_dotenv, find_dotenv\n", "import numpy as np\n", "\n", "from sklearn.metrics import classification_report\n", "\n", "metric_template_path = \"../data-sources/TestGrid/metrics\"\n", "if metric_template_path not in sys.path:\n", " sys.path.insert(1, metric_template_path)\n", "\n", "from ipynb.fs.defs.metric_template import ( # noqa: E402\n", " CephCommunication,\n", ")\n", "\n", "load_dotenv(find_dotenv())" ] }, { "cell_type": "code", "execution_count": 2, "id": "e4672acb", "metadata": {}, "outputs": [], "source": [ "## CEPH Bucket variables\n", "## Create a .env file on your local with the correct configs,\n", "s3_endpoint_url = os.getenv(\"S3_ENDPOINT\")\n", "s3_access_key = os.getenv(\"S3_ACCESS_KEY\")\n", "s3_secret_key = os.getenv(\"S3_SECRET_KEY\")\n", "s3_bucket = os.getenv(\"S3_BUCKET\")\n", "s3_path = \"github\"\n", "REMOTE = os.getenv(\"REMOTE\")\n", "INPUT_DATA_PATH = \"../../../data/processed/github\"" ] }, { "cell_type": "code", "execution_count": 31, "id": "29a28262", "metadata": {}, "outputs": [], "source": [ "if REMOTE:\n", " cc = CephCommunication(s3_endpoint_url, s3_access_key, s3_secret_key, s3_bucket)\n", " X_test = cc.read_from_ceph(s3_path, \"X_test.parquet\")\n", " y_test = cc.read_from_ceph(s3_path, \"y_test.parquet\")\n", "\n", "else:\n", " print(\n", " \"The X_test.parquet and y_test.parquet files are not included in the ocp-ci-analysis github repo.\"\n", " )\n", " print(\n", " \"Please set REMOTE=1 in the .env file and read this data from the S3 bucket instead.\"\n", " )" ] }, { "cell_type": "code", "execution_count": 32, "id": "c392d8d1", "metadata": {}, "outputs": [ { "data": { "text/html": [ "
\n", "\n", "\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
sizeis_revieweris_approvercreated_at_daycreated_at_monthcreated_at_weekdaycreated_at_hourchange_in_.githubchange_in_docschange_in_pkg...title_wordcount_fixtitle_wordcount_haproxytitle_wordcount_octitle_wordcount_publishingtitle_wordcount_reverttitle_wordcount_routertitle_wordcount_shtitle_wordcount_stagingtitle_wordcount_supporttitle_wordcount_travis
35993TrueTrue67021001...0000000000
145624TrueTrue96422001...0000000000
101020FalseFalse29743001...0000000000
5783FalseFalse1612113001...0000000000
94011TrueTrue17645001...0000000000
..................................................................
104381FalseTrue16812000...0000000000
60273FalseFalse2311016000...0000000000
115583FalseFalse251018001...0000000000
184424FalseFalse52010000...1000000000
234850FalseFalse257313001...0000000000
\n", "

2706 rows × 96 columns

\n", "
" ], "text/plain": [ " size is_reviewer is_approver created_at_day created_at_month \\\n", "3599 3 True True 6 7 \n", "14562 4 True True 9 6 \n", "10102 0 False False 29 7 \n", "578 3 False False 16 12 \n", "9401 1 True True 17 6 \n", "... ... ... ... ... ... \n", "10438 1 False True 16 8 \n", "6027 3 False False 23 11 \n", "11558 3 False False 25 10 \n", "18442 4 False False 5 2 \n", "23485 0 False False 25 7 \n", "\n", " created_at_weekday created_at_hour change_in_.github change_in_docs \\\n", "3599 0 21 0 0 \n", "14562 4 22 0 0 \n", "10102 4 3 0 0 \n", "578 1 13 0 0 \n", "9401 4 5 0 0 \n", "... ... ... ... ... \n", "10438 1 2 0 0 \n", "6027 0 16 0 0 \n", "11558 1 8 0 0 \n", "18442 0 10 0 0 \n", "23485 3 13 0 0 \n", "\n", " change_in_pkg ... title_wordcount_fix title_wordcount_haproxy \\\n", "3599 1 ... 0 0 \n", "14562 1 ... 0 0 \n", "10102 1 ... 0 0 \n", "578 1 ... 0 0 \n", "9401 1 ... 0 0 \n", "... ... ... ... ... \n", "10438 0 ... 0 0 \n", "6027 0 ... 0 0 \n", "11558 1 ... 0 0 \n", "18442 0 ... 1 0 \n", "23485 1 ... 0 0 \n", "\n", " title_wordcount_oc title_wordcount_publishing title_wordcount_revert \\\n", "3599 0 0 0 \n", "14562 0 0 0 \n", "10102 0 0 0 \n", "578 0 0 0 \n", "9401 0 0 0 \n", "... ... ... ... \n", "10438 0 0 0 \n", "6027 0 0 0 \n", "11558 0 0 0 \n", "18442 0 0 0 \n", "23485 0 0 0 \n", "\n", " title_wordcount_router title_wordcount_sh title_wordcount_staging \\\n", "3599 0 0 0 \n", "14562 0 0 0 \n", "10102 0 0 0 \n", "578 0 0 0 \n", "9401 0 0 0 \n", "... ... ... ... \n", "10438 0 0 0 \n", "6027 0 0 0 \n", "11558 0 0 0 \n", "18442 0 0 0 \n", "23485 0 0 0 \n", "\n", " title_wordcount_support title_wordcount_travis \n", "3599 0 0 \n", "14562 0 0 \n", "10102 0 0 \n", "578 0 0 \n", "9401 0 0 \n", "... ... ... \n", "10438 0 0 \n", "6027 0 0 \n", "11558 0 0 \n", "18442 0 0 \n", "23485 0 0 \n", "\n", "[2706 rows x 96 columns]" ] }, "execution_count": 32, "metadata": {}, "output_type": "execute_result" } ], "source": [ "X_test" ] }, { "cell_type": "code", "execution_count": 33, "id": "220eb258", "metadata": {}, "outputs": [ { "data": { "text/html": [ "
\n", "\n", "\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
ttm_class
35999
145625
101022
5784
94016
......
104386
60278
115585
184424
234856
\n", "

2706 rows × 1 columns

\n", "
" ], "text/plain": [ " ttm_class\n", "3599 9\n", "14562 5\n", "10102 2\n", "578 4\n", "9401 6\n", "... ...\n", "10438 6\n", "6027 8\n", "11558 5\n", "18442 4\n", "23485 6\n", "\n", "[2706 rows x 1 columns]" ] }, "execution_count": 33, "metadata": {}, "output_type": "execute_result" } ], "source": [ "y_test" ] }, { "cell_type": "code", "execution_count": 34, "id": "7ab0777c", "metadata": {}, "outputs": [], "source": [ "# endpoint from the seldon deployment\n", "base_url = \"http://ttm-pipeline-opf-seldon.apps.zero.massopen.cloud/predict\"" ] }, { "cell_type": "code", "execution_count": 35, "id": "bcd142cc", "metadata": {}, "outputs": [], "source": [ "# convert the dataframe into a numpy array and then to a list (required by seldon)\n", "data = {\"data\": {\"ndarray\": X_test.to_numpy().tolist()}}\n", "\n", "# create the query payload\n", "json_data = json.dumps(data)\n", "headers = {\"content-Type\": \"application/json\"}" ] }, { "cell_type": "code", "execution_count": 36, "id": "2f051ce0", "metadata": {}, "outputs": [ { "data": { "text/plain": [ "" ] }, "execution_count": 36, "metadata": {}, "output_type": "execute_result" } ], "source": [ "# query our inference service\n", "response = requests.post(base_url, data=json_data, headers=headers)\n", "response" ] }, { "cell_type": "code", "execution_count": 37, "id": "4781b783", "metadata": {}, "outputs": [ { "data": { "text/plain": [ "['t:0', 't:1', 't:2', 't:3', 't:4', 't:5', 't:6', 't:7', 't:8', 't:9']" ] }, "execution_count": 37, "metadata": {}, "output_type": "execute_result" } ], "source": [ "# what are the names of the prediction classes\n", "json_response = response.json()\n", "json_response[\"data\"][\"names\"]" ] }, { "cell_type": "code", "execution_count": 38, "id": "4b52a698", "metadata": {}, "outputs": [ { "data": { "text/plain": [ "[0.02, 0.03, 0.085, 0.165, 0.09, 0.155, 0.09, 0.135, 0.07, 0.16]" ] }, "execution_count": 38, "metadata": {}, "output_type": "execute_result" } ], "source": [ "# probabality estimates for each of the class for a sample PR\n", "json_response[\"data\"][\"ndarray\"][0]" ] }, { "cell_type": "code", "execution_count": 39, "id": "088ef754", "metadata": {}, "outputs": [], "source": [ "# get predicted classes from probabilities for each PR\n", "preds = np.argmax(json_response[\"data\"][\"ndarray\"], axis=1)" ] }, { "cell_type": "code", "execution_count": 40, "id": "287da01d", "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ " precision recall f1-score support\n", "\n", " 0 0.31 0.42 0.36 249\n", " 1 0.14 0.10 0.12 217\n", " 2 0.23 0.27 0.25 364\n", " 3 0.15 0.17 0.16 240\n", " 4 0.13 0.10 0.11 275\n", " 5 0.14 0.10 0.12 236\n", " 6 0.23 0.23 0.23 333\n", " 7 0.16 0.14 0.15 270\n", " 8 0.18 0.17 0.17 260\n", " 9 0.23 0.28 0.25 262\n", "\n", " accuracy 0.20 2706\n", " macro avg 0.19 0.20 0.19 2706\n", "weighted avg 0.19 0.20 0.20 2706\n", "\n" ] } ], "source": [ "# evaluate results\n", "print(classification_report(y_test, preds))" ] }, { "cell_type": "markdown", "id": "0b3d3e60-469e-4dd9-aed9-853c05b76ba8", "metadata": {}, "source": [ "# Conclusion\n", "\n", "The evaluation scores in the above classification report match the ones we saw in the training notebook. Great, looks like our inference service and model are working as expected, and are ready to predict some times to merge of PRs! " ] } ], "metadata": { "kernelspec": { "display_name": "Python 3", "language": "python", "name": "python3" }, "language_info": { "codemirror_mode": { "name": "ipython", "version": 3 }, "file_extension": ".py", "mimetype": "text/x-python", "name": "python", "nbconvert_exporter": "python", "pygments_lexer": "ipython3", "version": "3.8.3" } }, "nbformat": 4, "nbformat_minor": 5 }