{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Getting started: Introducing workload data"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "As a data scientist, in order to approach the analysis, we first need to have a knowledge about the data-set we will be working on. The objective of this notebook is to take a quick look at the data to understand what kind of information we have captured.\n",
    "\n",
    "The goal is to collect and analyse the workload data from the clusters. Insight operator is collecting the workload data from the 4.8+ clusters. The data can be found in the [Insight Operator Archive](https://github.com/openshift/insights-operator/blob/master/docs/insights-archive-sample/config/workload_info.json), which has two kinds of information. The image_layers and containers information."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Data Collection"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Here we import the two kinds of dataset from the DH-PLAYPEN bucket."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-06-14T15:20:03.036744Z",
     "start_time": "2021-06-14T15:20:03.032104Z"
    }
   },
   "outputs": [],
   "source": [
    "import io\n",
    "import boto3\n",
    "import pandas as pd\n",
    "import warnings\n",
    "import os\n",
    "\n",
    "from dotenv import load_dotenv, find_dotenv\n",
    "\n",
    "load_dotenv(find_dotenv())\n",
    "warnings.filterwarnings(\"ignore\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-06-14T15:20:04.307754Z",
     "start_time": "2021-06-14T15:20:03.974648Z"
    }
   },
   "outputs": [],
   "source": [
    "# CEPH Bucket variables\n",
    "s3_endpoint_url = os.getenv(\"S3_ENDPOINT\")\n",
    "s3_access_key = os.getenv(\"S3_ACCESS_KEY\")\n",
    "s3_secret_key = os.getenv(\"S3_SECRET_KEY\")\n",
    "s3_bucket = os.getenv(\"S3_BUCKET\")\n",
    "\n",
    "# s3 resource to communicate with storage\n",
    "s3 = boto3.resource(\n",
    "    \"s3\",\n",
    "    endpoint_url=s3_endpoint_url,\n",
    "    aws_access_key_id=s3_access_key,\n",
    "    aws_secret_access_key=s3_secret_key,\n",
    ")\n",
    "\n",
    "# access the parquet file as an s3 object\n",
    "\n",
    "obj1 = s3.Object(\n",
    "    \"DH-PLAYPEN\", \"ccx/fingerprinting/image_layers/date=2021-05-12/2021-05-12.parquet\"\n",
    ")\n",
    "obj2 = s3.Object(\n",
    "    \"DH-PLAYPEN\", \"ccx/fingerprinting/containers/date=2021-05-12/2021-05-12.parquet\"\n",
    ")\n",
    "# download the file into the buffer\n",
    "buffer1 = io.BytesIO()\n",
    "obj1.download_fileobj(buffer1)\n",
    "buffer2 = io.BytesIO()\n",
    "obj2.download_fileobj(buffer2)\n",
    "\n",
    "# read the buffer and create the dataframe\n",
    "image_layers_df = pd.read_parquet(buffer1)\n",
    "containers_df = pd.read_parquet(buffer2)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-11-12T18:05:52.606425Z",
     "start_time": "2021-11-12T18:05:52.604600Z"
    }
   },
   "source": [
    "### Image Layers Dataset"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-06-11T15:05:35.540212Z",
     "start_time": "2021-06-11T15:05:35.535159Z"
    }
   },
   "source": [
    "An overview of Image layers dataset can be seen in the dataframe below."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-06-14T15:20:05.394898Z",
     "start_time": "2021-06-14T15:20:05.384871Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>cluster_id</th>\n",
       "      <th>image_id</th>\n",
       "      <th>layer_image_id</th>\n",
       "      <th>layer_image_level</th>\n",
       "      <th>first_command</th>\n",
       "      <th>first_arg</th>\n",
       "      <th>archive_path</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>00003d61-9db1-4757-9cd1-84df271daeb9</td>\n",
       "      <td>sha256:337c22cabe530213b14965f9ea69a92dbeb5104...</td>\n",
       "      <td>sha256:9ebb302e1fb002fb643091710dac46f8258781d...</td>\n",
       "      <td>0</td>\n",
       "      <td>icTsn2s_EIax</td>\n",
       "      <td>2v1NneeWoS_9</td>\n",
       "      <td>archives/compressed/00/00003d61-9db1-4757-9cd1...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>00003d61-9db1-4757-9cd1-84df271daeb9</td>\n",
       "      <td>sha256:337c22cabe530213b14965f9ea69a92dbeb5104...</td>\n",
       "      <td>sha256:a74396a32e85c2feeedf76052ed3297859810c8...</td>\n",
       "      <td>1</td>\n",
       "      <td>icTsn2s_EIax</td>\n",
       "      <td>2v1NneeWoS_9</td>\n",
       "      <td>archives/compressed/00/00003d61-9db1-4757-9cd1...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>00003d61-9db1-4757-9cd1-84df271daeb9</td>\n",
       "      <td>sha256:337c22cabe530213b14965f9ea69a92dbeb5104...</td>\n",
       "      <td>sha256:7db62383a7470afbacfc0fab55d5a182e3c5fa2...</td>\n",
       "      <td>2</td>\n",
       "      <td>icTsn2s_EIax</td>\n",
       "      <td>2v1NneeWoS_9</td>\n",
       "      <td>archives/compressed/00/00003d61-9db1-4757-9cd1...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>00003d61-9db1-4757-9cd1-84df271daeb9</td>\n",
       "      <td>sha256:337c22cabe530213b14965f9ea69a92dbeb5104...</td>\n",
       "      <td>sha256:f24250419f728db23957454976d6d38b679a349...</td>\n",
       "      <td>3</td>\n",
       "      <td>icTsn2s_EIax</td>\n",
       "      <td>2v1NneeWoS_9</td>\n",
       "      <td>archives/compressed/00/00003d61-9db1-4757-9cd1...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>00003d61-9db1-4757-9cd1-84df271daeb9</td>\n",
       "      <td>sha256:337c22cabe530213b14965f9ea69a92dbeb5104...</td>\n",
       "      <td>sha256:267f7bb0f5dcf1b83f8ce89831d05f3a44a3abe...</td>\n",
       "      <td>4</td>\n",
       "      <td>icTsn2s_EIax</td>\n",
       "      <td>2v1NneeWoS_9</td>\n",
       "      <td>archives/compressed/00/00003d61-9db1-4757-9cd1...</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                             cluster_id  \\\n",
       "0  00003d61-9db1-4757-9cd1-84df271daeb9   \n",
       "1  00003d61-9db1-4757-9cd1-84df271daeb9   \n",
       "2  00003d61-9db1-4757-9cd1-84df271daeb9   \n",
       "3  00003d61-9db1-4757-9cd1-84df271daeb9   \n",
       "4  00003d61-9db1-4757-9cd1-84df271daeb9   \n",
       "\n",
       "                                            image_id  \\\n",
       "0  sha256:337c22cabe530213b14965f9ea69a92dbeb5104...   \n",
       "1  sha256:337c22cabe530213b14965f9ea69a92dbeb5104...   \n",
       "2  sha256:337c22cabe530213b14965f9ea69a92dbeb5104...   \n",
       "3  sha256:337c22cabe530213b14965f9ea69a92dbeb5104...   \n",
       "4  sha256:337c22cabe530213b14965f9ea69a92dbeb5104...   \n",
       "\n",
       "                                      layer_image_id  layer_image_level  \\\n",
       "0  sha256:9ebb302e1fb002fb643091710dac46f8258781d...                  0   \n",
       "1  sha256:a74396a32e85c2feeedf76052ed3297859810c8...                  1   \n",
       "2  sha256:7db62383a7470afbacfc0fab55d5a182e3c5fa2...                  2   \n",
       "3  sha256:f24250419f728db23957454976d6d38b679a349...                  3   \n",
       "4  sha256:267f7bb0f5dcf1b83f8ce89831d05f3a44a3abe...                  4   \n",
       "\n",
       "  first_command     first_arg  \\\n",
       "0  icTsn2s_EIax  2v1NneeWoS_9   \n",
       "1  icTsn2s_EIax  2v1NneeWoS_9   \n",
       "2  icTsn2s_EIax  2v1NneeWoS_9   \n",
       "3  icTsn2s_EIax  2v1NneeWoS_9   \n",
       "4  icTsn2s_EIax  2v1NneeWoS_9   \n",
       "\n",
       "                                        archive_path  \n",
       "0  archives/compressed/00/00003d61-9db1-4757-9cd1...  \n",
       "1  archives/compressed/00/00003d61-9db1-4757-9cd1...  \n",
       "2  archives/compressed/00/00003d61-9db1-4757-9cd1...  \n",
       "3  archives/compressed/00/00003d61-9db1-4757-9cd1...  \n",
       "4  archives/compressed/00/00003d61-9db1-4757-9cd1...  "
      ]
     },
     "execution_count": 15,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "image_layers_df.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-06-14T02:41:17.874826Z",
     "start_time": "2021-06-14T02:41:17.870828Z"
    }
   },
   "source": [
    "#### Inspect the Image Layers Data"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-06-14T02:48:21.493458Z",
     "start_time": "2021-06-14T02:48:21.489088Z"
    }
   },
   "source": [
    "We inspect the image layers data to see the kind of information we have access to."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-06-14T15:20:06.848689Z",
     "start_time": "2021-06-14T15:20:06.842834Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "cluster_id                        00003d61-9db1-4757-9cd1-84df271daeb9\n",
       "image_id             sha256:337c22cabe530213b14965f9ea69a92dbeb5104...\n",
       "layer_image_id       sha256:a74396a32e85c2feeedf76052ed3297859810c8...\n",
       "layer_image_level                                                    1\n",
       "first_command                                             icTsn2s_EIax\n",
       "first_arg                                                 2v1NneeWoS_9\n",
       "archive_path         archives/compressed/00/00003d61-9db1-4757-9cd1...\n",
       "Name: 1, dtype: object"
      ]
     },
     "execution_count": 16,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "image_layers_df.iloc[1]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-06-14T02:48:52.581783Z",
     "start_time": "2021-06-14T02:48:52.578304Z"
    }
   },
   "source": [
    "**Available fields:**\n",
    "\n",
    "- **cluster_id**: id of the cluster\n",
    "\n",
    "- **image_id**: provide the 'sha' of the image that the container is running.\n",
    "\n",
    "- **layer_image_id**: provide the 'sha' of the image_layers that is linked to image id.\n",
    "\n",
    "- **layer_image_level**: order of the image layer.\n",
    "\n",
    "- **first_command**: first command in that image.\n",
    "\n",
    "- **first_arg**: first argument in that image. We do not have information about the kind of first command and first argument provided, but we can compare if the two image runs the same command/argument. \n",
    "\n",
    "- **archive_path**: path to the archive from which the images are extracted."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-06-14T02:38:34.113302Z",
     "start_time": "2021-06-14T02:38:34.110134Z"
    }
   },
   "source": [
    "### Containers Dataset"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-06-14T15:20:08.314974Z",
     "start_time": "2021-06-14T15:20:08.303317Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>cluster_id</th>\n",
       "      <th>namespace</th>\n",
       "      <th>shape</th>\n",
       "      <th>shape_instances</th>\n",
       "      <th>image_id</th>\n",
       "      <th>first_command</th>\n",
       "      <th>first_arg</th>\n",
       "      <th>init_container</th>\n",
       "      <th>archive_path</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>00003d61-9db1-4757-9cd1-84df271daeb9</td>\n",
       "      <td>0LiT6ZNtbpYL</td>\n",
       "      <td>sha256:3ecf29979b2722bf4a82a5e7a954e8685820720...</td>\n",
       "      <td>1</td>\n",
       "      <td>sha256:f46f210d6023bec16e68340b484a8881ce46d5e...</td>\n",
       "      <td>None</td>\n",
       "      <td>47DEQpj8HBSa</td>\n",
       "      <td>False</td>\n",
       "      <td>archives/compressed/00/00003d61-9db1-4757-9cd1...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>00003d61-9db1-4757-9cd1-84df271daeb9</td>\n",
       "      <td>0LiT6ZNtbpYL</td>\n",
       "      <td>sha256:3ecf29979b2722bf4a82a5e7a954e8685820720...</td>\n",
       "      <td>1</td>\n",
       "      <td>sha256:edb9aaacf421c6dc45b20324e8699cec02f26bf...</td>\n",
       "      <td>n9CdwzVF-cwZ</td>\n",
       "      <td>RNOaw_AuQeIY</td>\n",
       "      <td>False</td>\n",
       "      <td>archives/compressed/00/00003d61-9db1-4757-9cd1...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>00003d61-9db1-4757-9cd1-84df271daeb9</td>\n",
       "      <td>0LiT6ZNtbpYL</td>\n",
       "      <td>sha256:542d007d13008cc1be2dbf03601b954c4452947...</td>\n",
       "      <td>1</td>\n",
       "      <td>sha256:a693c315b775c693dc49c19b7f217762676bc28...</td>\n",
       "      <td>b51B0EZ1bw3c</td>\n",
       "      <td>ua-xlwwsvdYd</td>\n",
       "      <td>False</td>\n",
       "      <td>archives/compressed/00/00003d61-9db1-4757-9cd1...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>00003d61-9db1-4757-9cd1-84df271daeb9</td>\n",
       "      <td>0LiT6ZNtbpYL</td>\n",
       "      <td>sha256:542d007d13008cc1be2dbf03601b954c4452947...</td>\n",
       "      <td>1</td>\n",
       "      <td>sha256:a693c315b775c693dc49c19b7f217762676bc28...</td>\n",
       "      <td>Cl6kTzfbYztA</td>\n",
       "      <td>None</td>\n",
       "      <td>True</td>\n",
       "      <td>archives/compressed/00/00003d61-9db1-4757-9cd1...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>00003d61-9db1-4757-9cd1-84df271daeb9</td>\n",
       "      <td>0LiT6ZNtbpYL</td>\n",
       "      <td>sha256:542d007d13008cc1be2dbf03601b954c4452947...</td>\n",
       "      <td>1</td>\n",
       "      <td>sha256:d9c64d038f16e04c52142bc9e7dfa0645ce7e34...</td>\n",
       "      <td>Cl6kTzfbYztA</td>\n",
       "      <td>None</td>\n",
       "      <td>True</td>\n",
       "      <td>archives/compressed/00/00003d61-9db1-4757-9cd1...</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                             cluster_id     namespace  \\\n",
       "0  00003d61-9db1-4757-9cd1-84df271daeb9  0LiT6ZNtbpYL   \n",
       "1  00003d61-9db1-4757-9cd1-84df271daeb9  0LiT6ZNtbpYL   \n",
       "2  00003d61-9db1-4757-9cd1-84df271daeb9  0LiT6ZNtbpYL   \n",
       "3  00003d61-9db1-4757-9cd1-84df271daeb9  0LiT6ZNtbpYL   \n",
       "4  00003d61-9db1-4757-9cd1-84df271daeb9  0LiT6ZNtbpYL   \n",
       "\n",
       "                                               shape  shape_instances  \\\n",
       "0  sha256:3ecf29979b2722bf4a82a5e7a954e8685820720...                1   \n",
       "1  sha256:3ecf29979b2722bf4a82a5e7a954e8685820720...                1   \n",
       "2  sha256:542d007d13008cc1be2dbf03601b954c4452947...                1   \n",
       "3  sha256:542d007d13008cc1be2dbf03601b954c4452947...                1   \n",
       "4  sha256:542d007d13008cc1be2dbf03601b954c4452947...                1   \n",
       "\n",
       "                                            image_id first_command  \\\n",
       "0  sha256:f46f210d6023bec16e68340b484a8881ce46d5e...          None   \n",
       "1  sha256:edb9aaacf421c6dc45b20324e8699cec02f26bf...  n9CdwzVF-cwZ   \n",
       "2  sha256:a693c315b775c693dc49c19b7f217762676bc28...  b51B0EZ1bw3c   \n",
       "3  sha256:a693c315b775c693dc49c19b7f217762676bc28...  Cl6kTzfbYztA   \n",
       "4  sha256:d9c64d038f16e04c52142bc9e7dfa0645ce7e34...  Cl6kTzfbYztA   \n",
       "\n",
       "      first_arg  init_container  \\\n",
       "0  47DEQpj8HBSa           False   \n",
       "1  RNOaw_AuQeIY           False   \n",
       "2  ua-xlwwsvdYd           False   \n",
       "3          None            True   \n",
       "4          None            True   \n",
       "\n",
       "                                        archive_path  \n",
       "0  archives/compressed/00/00003d61-9db1-4757-9cd1...  \n",
       "1  archives/compressed/00/00003d61-9db1-4757-9cd1...  \n",
       "2  archives/compressed/00/00003d61-9db1-4757-9cd1...  \n",
       "3  archives/compressed/00/00003d61-9db1-4757-9cd1...  \n",
       "4  archives/compressed/00/00003d61-9db1-4757-9cd1...  "
      ]
     },
     "execution_count": 17,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "containers_df.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-06-14T03:06:59.675878Z",
     "start_time": "2021-06-14T03:06:59.671527Z"
    }
   },
   "source": [
    "#### Inspecting the Container Dataset"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-06-14T15:20:09.264037Z",
     "start_time": "2021-06-14T15:20:09.257370Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "cluster_id                      00003d61-9db1-4757-9cd1-84df271daeb9\n",
       "namespace                                               0LiT6ZNtbpYL\n",
       "shape              sha256:3ecf29979b2722bf4a82a5e7a954e8685820720...\n",
       "shape_instances                                                    1\n",
       "image_id           sha256:edb9aaacf421c6dc45b20324e8699cec02f26bf...\n",
       "first_command                                           n9CdwzVF-cwZ\n",
       "first_arg                                               RNOaw_AuQeIY\n",
       "init_container                                                 False\n",
       "archive_path       archives/compressed/00/00003d61-9db1-4757-9cd1...\n",
       "Name: 1, dtype: object"
      ]
     },
     "execution_count": 18,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "containers_df.iloc[1]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Available fields:**\n",
    "\n",
    "- **cluster_id**: id of the cluster\n",
    "\n",
    "- **namespace**: namespace in the cluster\n",
    "\n",
    "- **shape**: These are POD's templete. They are set of containers in the POD. If two POD used the same set of containers, same command, they fall into same shape.\n",
    "\n",
    "- **shape_instances**: number of PODs of that shape.\n",
    "\n",
    "- **containers (image_id/first_command/first_argument/init_container)**: provide the information about the containers in the shape. Their image_id, first_command, first argument and also the number of containers in that shape (init_container).\n",
    "\n",
    "- **archive_path**: archive path to the cluster id."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-06-14T15:20:11.291157Z",
     "start_time": "2021-06-14T15:20:10.204205Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>shape</th>\n",
       "      <th>cluster_id</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>6315</th>\n",
       "      <td>sha256:ff62cfd4da3beb77d886f8935a1b7a6aaf54bdb...</td>\n",
       "      <td>2721</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2969</th>\n",
       "      <td>sha256:78fc0fdc25942f43a44b10330813a19f04ea88e...</td>\n",
       "      <td>2551</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2040</th>\n",
       "      <td>sha256:524148cb8d81907984141cb8d210decf75657d7...</td>\n",
       "      <td>1792</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2370</th>\n",
       "      <td>sha256:5fe22686d7266cfc828498c6674f3268fa6bb78...</td>\n",
       "      <td>1790</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3728</th>\n",
       "      <td>sha256:97af185840a1f8c688608aa199bc6a8fb45f9ae...</td>\n",
       "      <td>1361</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>...</th>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2908</th>\n",
       "      <td>sha256:76cde139b6a84f92e5f5d273aaec928589957f4...</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2909</th>\n",
       "      <td>sha256:76d83926eb2df6f554f519bfcc9f74904a16b75...</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2911</th>\n",
       "      <td>sha256:76e3aa55c87e23ff1c7beef873bcf399b89ca30...</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2912</th>\n",
       "      <td>sha256:77063f77b9a5d1513981bbe202ebceeecc5f80f...</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6328</th>\n",
       "      <td>sha256:ffe906ed042207a1a05260ecf1c46f93218b830...</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>6329 rows × 2 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "                                                  shape  cluster_id\n",
       "6315  sha256:ff62cfd4da3beb77d886f8935a1b7a6aaf54bdb...        2721\n",
       "2969  sha256:78fc0fdc25942f43a44b10330813a19f04ea88e...        2551\n",
       "2040  sha256:524148cb8d81907984141cb8d210decf75657d7...        1792\n",
       "2370  sha256:5fe22686d7266cfc828498c6674f3268fa6bb78...        1790\n",
       "3728  sha256:97af185840a1f8c688608aa199bc6a8fb45f9ae...        1361\n",
       "...                                                 ...         ...\n",
       "2908  sha256:76cde139b6a84f92e5f5d273aaec928589957f4...           1\n",
       "2909  sha256:76d83926eb2df6f554f519bfcc9f74904a16b75...           1\n",
       "2911  sha256:76e3aa55c87e23ff1c7beef873bcf399b89ca30...           1\n",
       "2912  sha256:77063f77b9a5d1513981bbe202ebceeecc5f80f...           1\n",
       "6328  sha256:ffe906ed042207a1a05260ecf1c46f93218b830...           1\n",
       "\n",
       "[6329 rows x 2 columns]"
      ]
     },
     "execution_count": 19,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "containers_df.groupby([\"shape\"]).agg(\n",
    "    {\"cluster_id\": pd.Series.nunique}\n",
    ").reset_index().sort_values(by=[\"cluster_id\"], ascending=False)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-06-11T15:59:28.735394Z",
     "start_time": "2021-06-11T15:59:28.731156Z"
    }
   },
   "source": [
    "To have some idea about the shape, we use the groupby method in order to examine if different clusters have same shape or not. Turns out, form the information above, almost all clusters have the same shape configuration. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-06-14T03:35:31.159352Z",
     "start_time": "2021-06-14T03:35:31.155679Z"
    }
   },
   "source": [
    "## Conclusion"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Our next goal is to do the exploratory data analysis of the dataset to get some insight about the relationships between the features. This is then followed by the use of ML for identifying and analysing the types (clusters) of workloads that customer runs. "
   ]
  }
 ],
 "metadata": {
  "finalized": {
   "timestamp": 1623682743734,
   "trusted": false
  },
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.8.3"
  },
  "requirements": {
   "aliases": {},
   "dev-packages": {},
   "packages": {
    "pandas": "==1.2.4"
   },
   "requires": {
    "python_version": "3.8"
   },
   "sources": [
    {
     "name": "pypi",
     "url": "https://pypi.org/simple",
     "verify_ssl": true
    }
   ]
  },
  "toc": {
   "base_numbering": 1,
   "nav_menu": {},
   "number_sections": true,
   "sideBar": true,
   "skip_h1_title": false,
   "title_cell": "Table of Contents",
   "title_sidebar": "Contents",
   "toc_cell": false,
   "toc_position": {},
   "toc_section_display": true,
   "toc_window_display": false
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}