{ "cells": [ { "cell_type": "markdown", "metadata": {}, "source": [ "# Compare groups" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "install retentioneering if running from google.colab or for the first time:" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "!pip3 install retentioneering" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Statistical comparison" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Very often we need to compare two groups of users using some metrics. It can be during A/B test results analysis, or comparing two user segments from different channels, or comparing cohorts of users and etc." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "In this tutorial we will use a simple dataset of user activity logs in app or web-site during hypothetical A/B test. It has raw behavior event-level logs as well as additional information, specifying the particular user in the test or control and some transaction information.\n", "\n", "We start from importing retentioneering and sample datasets:" ] }, { "cell_type": "code", "execution_count": 2, "metadata": {}, "outputs": [ { "data": { "text/html": [ "
\n", "\n", "\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
user_ideventtimestampuser_backettransaction_valuetransaction_ID
0219483890catalog2019-11-01 17:59:13.273932testNaNNone
1219483890product2019-11-01 17:59:28.459271testNaNNone
2219483890cart2019-11-01 17:59:29.502214testNaNNone
3219483890catalog2019-11-01 17:59:32.557029testNaNNone
4964964743catalog2019-11-01 21:38:19.283663testNaNNone
\n", "
" ], "text/plain": [ " user_id event timestamp user_backet \\\n", "0 219483890 catalog 2019-11-01 17:59:13.273932 test \n", "1 219483890 product 2019-11-01 17:59:28.459271 test \n", "2 219483890 cart 2019-11-01 17:59:29.502214 test \n", "3 219483890 catalog 2019-11-01 17:59:32.557029 test \n", "4 964964743 catalog 2019-11-01 21:38:19.283663 test \n", "\n", " transaction_value transaction_ID \n", "0 NaN None \n", "1 NaN None \n", "2 NaN None \n", "3 NaN None \n", "4 NaN None " ] }, "execution_count": 2, "metadata": {}, "output_type": "execute_result" } ], "source": [ "import retentioneering\n", "\n", "# load sample data\n", "data = retentioneering.datasets.load_simple_ab_test()\n", "\n", "data.head()" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "We can see regular columns with information about user actions (‘user_id’, ‘event’, ‘timestamp’) as well as column regarding A/B test: ‘user_backet’, and columns with transactions information (events ‘payment_done’): ‘transaction_value’ and ‘transaction_ID’.\n", "\n", "Next, as usually we need to update retentioneering.config to specify column names for events, user_ids and time:" ] }, { "cell_type": "code", "execution_count": 3, "metadata": {}, "outputs": [], "source": [ "retentioneering.config.update({\n", " 'user_col': 'user_id',\n", " 'event_col':'event',\n", " 'event_time_col':'timestamp',\n", "})" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Let’s explore column ‘user_backet’:" ] }, { "cell_type": "code", "execution_count": 4, "metadata": {}, "outputs": [ { "data": { "text/plain": [ "not_in_test 2624\n", "control 573\n", "test 554\n", "Name: user_backet, dtype: int64" ] }, "execution_count": 4, "metadata": {}, "output_type": "execute_result" } ], "source": [ "(data\n", " .drop_duplicates(subset=['user_id'])['user_backet']\n", " .value_counts())" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "We can see that our dataset has 554 and 573 unique users in test and control groups, correspondingly. Let’s put those user_id’s in separate variables ‘test’ and ‘control’:" ] }, { "cell_type": "code", "execution_count": 5, "metadata": {}, "outputs": [], "source": [ "test = data[data['user_backet']=='test']['user_id'].unique()\n", "control = data[data['user_backet']=='control']['user_id'].unique()" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Now everything is ready to start comparing these two groups using rete.compare() function.\n", "\n", "Let’s say we would like to compare conversion rate in the test vs control groups.\n", "\n", "For this we would need to specify a function that given one user trajectory (in form of dataframe) will return a numerical value, 1 (converted) or 0 (not converted) in our case. Importantly, function must take as an argument a dataframe of one user trajectory, perform any type of calculation and return a single numerical value.\n", "\n", "In our case user is considered converted is they have ‘payment_done’ event, so the function definition is very straightforward:" ] }, { "cell_type": "code", "execution_count": 8, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "test (mean ± SD): 0.227 ± 0.419, n = 554\n", "control (mean ± SD): 0.148 ± 0.355, n = 573\n", "'test' is greater than 'control' with P-value: 0.00034\n", "power of the test: 96.15%\n" ] }, { "data": { "image/png": "iVBORw0KGgoAAAANSUhEUgAAAbgAAAFOCAYAAAD0AQi3AAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjMuMSwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy/d3fzzAAAACXBIWXMAAAsTAAALEwEAmpwYAAAn20lEQVR4nO3df1iUdb7/8RcMjiEKBplL5MaqgakgSpo/yg6oae3YVq4ef+Tq1nXMEsU1jTXMVcof/fDXsTrlqe3sVWpqrqXtRoR5Qo9WUmuLCaiZSGb+WGhlEAKG+f7R17liVZhBmBk+Ph/XxXUx9/2Z+36/R8YX93DfnzvA6XQ6BQCAYQJ9XQAAAM2BgAMAGImAAwAYiYADABjJqIBzOp1yOBzivBkAgFEBV1tbq3379qm2ttbt5zgcDn322WdyOBzNWJn3mNaPRE8tgWn9SGb2dKUxKuAAADiPgAMAGImAAwAYyasBN3XqVMXGxrq+brnlFklSVlaWhgwZol69emnKlCk6ffq06znr1q3Trbfeqt69e2v27NkqLy/3ZskAgBbKqwF37NgxZWdnq7CwUIWFhfrkk09UXFystLQ0paena8eOHQoPD9f8+fMlSbm5uVq1apVWr16trKwslZaWasWKFd4sGQDQQnkt4JxOp06dOqWoqKg6y7dt26bBgwcrOTlZ4eHhSk1NVU5Ojs6ePastW7Zo1KhR6t27tzp06KCHH35Y7777rrdKBgC0YEHe2tF3332ngIAAjR8/XoWFhercubMef/xxFRQUqGfPnq5xkZGRCg4O1vHjx5Wfn68HH3zQtS4mJkalpaU6e/asQkNDL7kvT07rPT/WlFOBTetHoqeWwLR+pMb3ZLFYmqMcNILXAq6kpERdunTRY489ppiYGL311luaOnWqunXrpnbt2tUZGxISovLyctnt9jrrQkJCJEnl5eX1BlxeXp7H9TXmOf7MtH4kemoJTOtH8rynxMTEZqoEnvJawPXo0UNvvvmm6/HkyZO1adMmff7550pOTq4ztqKiQqGhoQoNDVVlZWWd5ZLqDTdJiouLc/u3KIfDoby8PI+e489M60eip5bAtH4kM3u60ngt4D766CNVVVVp2LBhrmU1NTWKi4tTfn6+a9mpU6dUXV2t6Ohode3aVfn5+brjjjskSYcOHdINN9zgOpK7FIvF4vEPZGOe489M60eip5bAtH4kM3u6UnjtJJOqqiotWLBA+/btk91u1//8z/+oqqpKCxcuVHZ2tnJzc1VaWqrFixfLZrPJarXqnnvu0caNG3Xo0CGdPHlSy5Yt03333eetkgEALZjXjuCGDRum4uJipaamym63q0ePHnrllVfUpUsXZWRkaM6cOSotLdXtt9+utLQ0SVL//v31H//xH5o8ebIqKip099131znpBACAS/FawEnSAw88oAceeOCC5TabTTab7aLPmTx5siZPntzMlQEATMNUXQAAI3n1CM4E73z8pH6oaXi6sNZBIfpV/ye8UBEA4GIIOA/9UFOuH6qZDxMA/B0fUQIAjETAAQCMRMABAIxEwAEAjETAAQCMRMABAIxEwAEAjETAAQCMRMABAIxEwAEAjETAAQCMRMABAIxEwAEAjETAAQCMRMABAIxEwAEAjETAAQCMRMABAIxEwAEAjETAAQCMRMABAIxEwAEAjETAAQCMRMABAIxEwAEAjETAAQCMRMABAIxEwAEAjETAAQCMRMABAIxEwAEAjETAAQCMRMABAIwU5OsC/MW81w+qujag3jFtr7IooXMb/VBd7qWqAACNRcD9f+U/OFTtqD/gAAAtBx9RAgCMRMABAIxEwAEAjETAAQCMRMABAIxEwAEAjETAAQCMRMABAIxEwAEAjETAAQCMRMABAIzkk4A7deqUbrnlFu3evVuStHfvXtlsNsXHx2vcuHE6cuSIa2xWVpaGDBmiXr16acqUKTp9+rQvSgYAtDA+Cbh58+aprKxMkmS325WSkqJJkyZp586d6tevn2bOnClJKi4uVlpamtLT07Vjxw6Fh4dr/vz5vigZANDCeD3gNm3apDZt2uhnP/uZJGn79u3q1KmTRo8erbCwME2bNk1FRUU6dOiQtm3bpsGDBys5OVnh4eFKTU1VTk6Ozp496+2yAQAtjFdvl/PNN99ozZo12rBhg379619LkgoKCtS9e3fXGKvVqujoaBUVFamgoEA9e/Z0rYuMjFRwcLCOHz+u0NDQS+7H4XC4XdP5sa0szgbHtgp0KkBBClSrBscGKsijOprK+X36Yt/NhZ78n2n9SI3vyWKxNEc5aASvBZzT6dTcuXM1e/ZshYeHu5aXlZWpffv2dcaGhITIbrerrKxM7dq1u2BdeXn9NxzNy8vzuL7xvavcHDlcbuSbJGnfvn0e19FUGvMa+Dt68n+m9SN53lNiYmIzVQJPeS3g3njjDUVERGj48OF1loeGhqqysrLOsoqKCoWFhV1yXX1Hb5IUFxfn9m9RDodDeXl5Wvc3a4M3PA1pbVH/bltkrzjT4HZbt2qjewcsdKuGpnS+H09eA39HT/7PtH4kM3u60ngt4Pbs2aPt27frvffecy377W9/q3bt2ik2Nta1rKqqSkVFRerevbv279+v/Px817pTp06purpa0dHR9e7LYrF4/ANZ7QhoMOCqawPkVI1qVd3g9mpV49M3RWNeA39HT/7PtH4kM3u6UnjtJJMXX3xRhYWFrq+oqCi99tpr2rFjhwoLC5WZmamysjItW7ZMCQkJ6tixo0aOHKns7Gzl5uaqtLRUixcvls1mk9Vq9VbZAIAWyucXerdr106rV6/WypUrNWjQIB08eFBLliyRJEVHRysjI0Nz5sxRUlKSnE6n0tLSfFwxAKAl8OpZlD/14Ycfur4fMGCAMjMzLzrOZrPJZrN5qywAgCF8fgQHAEBzIOAAAEYi4AAARiLgAABGIuAAAEYi4AAARiLgAABGIuAAAEYi4AAARiLgAABGIuAAAEYi4AAARiLgAABGIuAAAEYi4AAARiLgAABGIuAAAEYi4AAARiLgAABGIuAAAEYi4AAARiLgAABGIuAAAEYi4AAARiLgAABGIuAAAEYi4AAARiLgAABGIuAAAEYi4AAARiLgAABGIuAAAEYi4AAARiLgAABGIuAAAEYi4AAARiLgAABGIuAAAEYi4AAARiLgAABGIuAAAEYi4AAARiLgAABGIuAAAEYi4AAARiLgAABGIuAAAEYi4AAARiLgAABG8mrAbdmyRcnJyerVq5dGjRql3NxcSdLevXtls9kUHx+vcePG6ciRI67nZGVlaciQIerVq5emTJmi06dPe7NkAEAL5bWAO3LkiDIyMvTEE09o9+7duvPOOzV9+nTZ7XalpKRo0qRJ2rlzp/r166eZM2dKkoqLi5WWlqb09HTt2LFD4eHhmj9/vrdKBgC0YF4LuD179ujmm29WUlKSQkJCNHbsWJWUlCg7O1udOnXS6NGjFRYWpmnTpqmoqEiHDh3Stm3bNHjwYCUnJys8PFypqanKycnR2bNnvVU2AKCFCvLWjsaOHasxY8bI6XSqvLxc69ev189//nMdOHBA3bt3d42zWq2Kjo5WUVGRCgoK1LNnT9e6yMhIBQcH6/jx4woNDb3kvhwOh9t1nR/byuJscGyrQKcCFKRAtWpwbKCCPKqjqZzfpy/23Vzoyf+Z1o/U+J4sFktzlING8FrAWSwWWSwW7d69W7/97W8lSSkpKTp58qTat29fZ2xISIjsdrvKysrUrl27C9aVl5fXu6+8vDyP6xvfu8rNkcPlRr5Jkvbt2+dxHU2lMa+Bv6Mn/2daP5LnPSUmJjZTJfCU1wLuvIEDByovL09ffPGFUlJS1L59e9122211xlRUVCgsLEyhoaGqrKy8YF19R2+SFBcX5/ZvUQ6HQ3l5eVr3N6uqHQH1jg1pbVH/bltkrzjT4HZbt2qjewcsdKuGpnS+H09eA39HT/7PtH4kM3u60ngt4NasWaPQ0FCNHTtWVqtVffv2Ve/evXXLLbcoOzvbNa6qqkpFRUXq3r279u/fr/z8fNe6U6dOqbq6WtHR0fXu6/zRoieqHQENBlx1bYCcqlGtqhvcXq1qfPqmaMxr4O/oyf+Z1o9kZk9XCq+dZBIREaE1a9bowIEDqqys1M6dO/X5559rwIABKiwsVGZmpsrKyrRs2TIlJCSoY8eOGjlypLKzs5Wbm6vS0lItXrxYNptNVqvVW2UDAFoorx3B3XvvvSouLtYjjzyi77//XtHR0VqyZIm6deum1atXa+HChXrssceUmJiopUuXSpKio6OVkZGhOXPmqLS0VLfffrvS0tK8VTIAoAXzWsAFBgZq5syZrmvcfmrAgAHKzMy86PNsNptsNlszVwcAMA1TdQEAjETAAQCMRMABAIxEwAEAjOR2wN10000qLi6+YPnhw4fVp0+fJi0KAIDL1eBZlPfcc48CAgLkdDr18MMPq1WruvNUlZSUXDCdFgAAvtZgwA0dOlSSlJ+fr/79+18wb6TValVSUlKzFAcAQGM1GHApKSmu73/zm980OA8kAAD+wO0LvVNSUnTkyBHt3r37ggmQpR8/ygQAtCy1tbUKDDTzfEO3A+7ll1/WypUrZbFYFBwcXGddQEAAAQcAfuqf//yn0tLS9P3336tLly46duyYjh49qj59+qh379667bbb9OSTT6qmpkatW7fWggULFBAQoFmzZmnjxo2SpDFjxmj58uXasmWLDh48qO+//17ff/+9ZsyYoWHDhmnZsmX69NNP5XQ69atf/UoTJkzwcdceBNxrr72mqVOnKiUlhZm1AaAFWbNmjW677TZNmDBBn3zyiZ5//nn94x//0GOPPaaoqCiNHz9ejz/+uHr27KkPP/xQixcvVnp6+iW3FxQUpNdff12nT5/Wvffeq9tvv12bN2/WX//6V7Vt21avvfaaF7u7NLePS6urqzVixAjCDQBamIMHD7puxBoXFydJuvrqqxUVFSVJOnbsmHr27ClJ6tOnj44dO3bBNmpra13f33zzzZKkDh066Oqrr5bdbtczzzyjJUuWaNasWX7zkafbVdx11116//33m7MWAEAziIyMVEFBgSRp9+7dF6y/5ppr9NVXX0mS9u7dq+7du8tqtaqiokKSdO7cuTrXQX/55ZeSfrxHZ0VFhZxOp3bt2qWnn35aq1at0saNG3X69OnmbqtBbn9EGRYWpldffVUff/yxunfvfsGR3Ny5c5u8OADA5Zs6darS0tL05ptvqnPnzhccYWVkZCg9PV2tWrVScHCwnnzySV177bXq0aOHZs+eLafTqfDwcNf4kydP6v7779fZs2c1f/58RUREqKamRmPGjFFNTY1uv/12dejQwdttXsDtgPviiy8UHx8vSSosLKyzLiCg/jthAwB8p7i4WHPmzFF8fLw+/fRTvf322/rTn/7kWh8fH68333zzguedvzfnvxo6dKjGjRtXZ9m8efOatugm4HbAvf76681ZBwCgmVxzzTWuI7SrrrpKTz31lK9L8gq3A2779u31rh8yZMhlFwMAaHpdunS56BFaY0yfPr1JtuMNbgfctGnTLrrcYrGoffv2BBwAwK+4HXDnz8A5r6qqSl999ZVWrVrlmq8SAAB/0eiLFaxWq2666SY9++yzevnll5uyJgAALttlX41XXV3tF9c7AADwU25/RLlkyZILllVWVurjjz92XdUOAGi8BesOyV7paLbtt73KogXjb2y27fsbtwPuwIEDFyw7fy+4KVOmNGlRAHAlslc6VN6MAeep2NhYZWVl6YYbbvD4ud98842GDBmiL7/8UkFBbkdNk+I6OACAkTz6G9yJEyf07LPP6pFHHtGjjz6q//7v/9bZs2ebqzYAgI8MHjxYknTHHXfo73//u1544QUlJSWpb9++WrRokaqqqiRJhw8f1tixY5WQkKARI0YoKytLNTU1rkvHevTooZKSEp/04HbA7d+/X3fddZdyc3PVoUMHBQUF6c0339SQIUOUl5fXnDUCALwsJydHkpSVlaW//e1vysnJ0dq1a/X2228rLy9Pa9askSQtXLhQSUlJ+uSTTzRv3jzNnj1bFRUVrslBvvzyyzrzWHqT2wH3zDPPaOLEidqwYYMWLlyop59+Wtu3b9fw4cMvOV8ZAKDl27Bhg2bOnKnrrrtOUVFRmj59ut577z1JUmBgoPbv369vvvlGgwYN0o4dO9SmTRsfV/wjj47gbDbbBcvHjh3runUCAMA8x48f1+TJkxUbG6vY2Fg98MADOnnypCTpueee089+9jNNnTpVgwYNch3Z+QO3TzJp27atjh49qpiYmDrLT548qdatWzd5YQAA/3DNNddoxYoVrjvK2O12nThxQg6HQ7t371ZaWprS09P19ddfa9KkSerbt6+6devm46o9CLhRo0ZpwYIFOnfunPr27augoCDt27dPS5cu1Z133tmcNQLAFaHtVZaGB3lx+xaLRSUlJRo5cqT+67/+S4sWLVJNTY3S09N1ww03aN68eVq5cqVKS0s1ZswYVVVVqaqqSldffbXr0oCSkhJde+21zdFOg9wOuOnTp8tisSgjI8N1l1eLxaJRo0bp97//fbMVCABXCn+7CHvYsGGaOHGi3nrrLZWXl+uuu+6Sw+HQ0KFDNWvWLEk/np/x1FNPacWKFWrfvr0efPBBJSYmqra2Vr1791ZycrJ27dql9u3be73+BgOutrZW7777rqKjo5WSkqIpU6boyJEjWr58ucaOHavk5GRv1AkA8LJVq1a5vk9PT1d6evoFY/r27at33nnnguWBgYFNdouexqr3JJPy8nJNnDhRjz/+uGu+SavVqp///OcqLS3VtGnT9MQTT3ilUAAAPFFvwK1atUpnzpzRe++9V+d+b23atNGmTZu0du1affjhh3VufQ4AgD+oN+Cys7M1c+ZMderU6aLr+/Tpo9mzZ+utt95qluIAAGisegPu1KlTuvHG+v/oGR8fr6KioiYtCgCAy1VvwLVv316nTp2qdwNnzpxRcHBwkxYFAMDlqjfgBg8erJdfflm1tbWXHPOnP/1JAwcObPLCAAC4HPUGXGpqqr766ivdf//9+uijj1RaWqra2lqdPn1aH330kSZOnKjc3FzNnDnTS+UCAOCeeq+D69ixo9avX68FCxbooYceUkBAgGud0+lU3759tW7dukbdDA8AgObU4IXenTp10quvvqoTJ07owIEDOnfunNq1a6ebbrpJHTt29EaNAHBFeOfjJ/VDTXmzbb91UIh+1d931y5PnDhRd999t0aPHu3R8xp7d3C3R0ZGRioyMtKjogAA7vuhplw/VDdfwPmrmpoaj4LLXR7d0RsAcGXZs2eP7r77biUkJGjChAk6evSoTpw4oYceekiJiYkaPny4Nm/e7BofGxur9evXKzk5Wb169dLcuXPldDo1e/Zsffrpp5o3b55effVV/fnPf9bEiRM1e/Zs/fKXv5Qk5ebm6r777lPv3r01ZswY7du377JqJ+AAABd14sQJTZ8+XXPnztWePXsUHx+vP/zhD3r44YcVExOjHTt2aMmSJVq2bJl27drlet4HH3ygTZs2acOGDfrrX/+qzz77TM8995z69eunp556Sg8++KAkae/everXr5+2bdumkydP6qGHHtKUKVO0c+dOjR07Vg899JDOnDnT6PoJOADARb377rsaOHCgBgwYoODgYD3yyCPq27evTpw4odTUVIWGhqpPnz4aO3as/vKXv7ieN336dEVERKhbt27q0qWLjh8/ftHtd+7cWWPGjJHVatXWrVvVr18/jRgxQm3bttV9992nG2+8sU5weoqAAwBc1PHjx3X99de7Hrdr106dO3fW9ddfX+dvZhEREXWOtMLDw13ft27dWg6H46LbDwsLq7Ov6OjoOuvDw8NdE/03BgEHALiofw2YkydPau3atTp58qScTqdr+ddff10nCBsjIiJC3333XZ1lR48evaztEnAA4CdaB4Wodatm/AoK8aieESNG6MMPP9TevXtVVlamZ555Rh07dpTVatVLL70ku92u3Nxcbd26Vffdd1+D2zt/h/CLufPOO/W///u/+uijj2S327Vx40adOXNG//Zv/+ZRzT/V9OdlAgAaxZfXqF1MTEyMMjIy9Pjjj+sf//iHBg4cqCeffFJnzpzRH/7wB7300kuKiIjQnDlzFBcX1+D2kpKS9Nxzz6lNmzYKCakbtl27dtXTTz+tpUuX6ptvvtGNN96o559//rLmOvZqwOXk5OiZZ57RsWPHdN1112natGkaOXKk9u7dq4ULF+rYsWPq0aOHFi1apM6dO0uSsrKy9PTTT+vMmTO65ZZbtGjRInXo0MGbZQPAFeuXv/yl6zT+866++mqtW7fuouMLCwvrPF6/fr3r+0mTJmnSpEmux/961HfHHXfojjvuuGCb119//QXbdYfXPqIsLS1VamqqJk6cqN27d+vRRx/V3LlzVVBQoJSUFE2aNEk7d+5Uv379XHNbFhcXKy0tTenp6dqxY4fCw8M1f/58b5UMAGjBvBZwubm5ioqK0r//+7+rbdu2GjZsmGJiYpSdna1OnTpp9OjRCgsL07Rp01RUVKRDhw5p27ZtGjx4sJKTkxUeHq7U1FTl5OTo7Nmz3iobANBCee0jysTERK1cudL1uKSkRMXFxXr77bfr3G7HarUqOjpaRUVFKigoUM+ePV3rIiMjFRwcrOPHjys0NPSS+7rUKan1jW1lcTYwUmoV6FSAghSoVg2ODVSQR3U0lfP79MW+mws9+T/T+pEa35PFYmmOctAIXgu48PBw17URubm5mjdvnuLj4xUZGXlBWIWEhMhut6usrEzt2rW7YF15ef1zteXl5Xlc3/jeVW6OHC438k2SLnuamcvRmNfA39GT/zOtH8nznhITE5upEnjKqyeZ2O12ZWRkKDs7W9OmTdOkSZO0fPlyVVZW1hlXUVGhsLAwhYaGXnRdfUdvkhQXF+f2b1EOh0N5eXla9zerqh0B9Y4NaW1R/25bZK9oeOqY1q3a6N4BC92qoSmd78eT18Df0ZP/M60fycyerjReC7jKykqNHz9eHTt2VGZmpq699lpJP54a+tOJOquqqlRUVKTu3btr//79ys/Pd607deqUqqurL7ja/V9ZLBaPfyCrHQENBlx1bYCcqlGtqhvcXq1qfPqmaMxr4O/oyf+Z1o9kZk9XCq+dZLJt2zZVVVXphRdecIWbJA0bNkyFhYXKzMxUWVmZli1bpoSEBHXs2FEjR45Udna2cnNzVVpaqsWLF8tms8lqtXqrbABAC+W1gMvPz9fXX3+tuLg4xcbGur4++OADrV69WitXrtSgQYN08OBBLVmyRJIUHR2tjIwMzZkzR0lJSXI6nUpLS/NWyQCAFsxrH1HOnz+/3mvYMjMzL7rcZrPJZrM1V1kAAEMxFyUAwEgEHADASAQcAMBIBBwAwEgEHADASAQcAMBIBBwAwEgEHADASAQcAMBIBBwAwEgEHADASAQcAMBIBBwAwEgEHADASAQcAMBIBBwAwEgEHADASAQcAMBIBBwAwEgEHADASAQcAMBIBBwAwEgEHADASAQcAMBIBBwAwEgEHADASAQcAMBIBBwAwEgEHADASAQcAMBIBBwAwEgEHADASAQcAMBIBBwAwEgEHADASAQcAMBIBBwAwEgEHADASEG+LgAALteCdYdkr3Q0OK7tVRYtGH+jFyqCPyDgALR49kqHyt0IOFxZ+IgSAGAkAg4AYCQCDgBgJAIOAGAkAg4AYCQCDgBgJAIOAGAkAg4AYCQCDgBgJK8HXFpamjZt2uR6vHfvXtlsNsXHx2vcuHE6cuSIa11WVpaGDBmiXr16acqUKTp9+rS3ywUAtFBeC7hdu3ZpyZIl2rp1q2uZ3W5XSkqKJk2apJ07d6pfv36aOXOmJKm4uFhpaWlKT0/Xjh07FB4ervnz53urXABAC+e1gPv8889lt9sVERHhWrZ9+3Z16tRJo0ePVlhYmKZNm6aioiIdOnRI27Zt0+DBg5WcnKzw8HClpqYqJydHZ8+e9VbJAIAWzGuTLc+YMUOSNHHiRNeygoICde/e3fXYarUqOjpaRUVFKigoUM+ePV3rIiMjFRwcrOPHjys0NLTefTkc7k+6en5sK4uzwbGtAp0KUJAC1arBsYEK8qiOpnJ+n77Yd3OhJ//n635aBTrdfg+7W2Nje7JYLB6NR/Px6d0EysrK1L59+zrLQkJCZLfbVVZWpnbt2l2wrry8vMHt5uXleVzL+N5Vbo4cLjfyTZK0b98+j+toKo15DfwdPfk/X/Xz6zj3x3r6vvS0p8TERI/Go/n4NOBCQ0NVWVlZZ1lFRYXCwsIuua6hozdJiouLc/u3KIfDoby8PK37m1XVjoB6x4a0tqh/ty2yV5xpcLutW7XRvQMWulVDUzrfjyevgb+jJ//n637mvX5Q5T80fKQV0tqipybGuLVNX/eEy+fTgOvatas2b97selxVVaWioiJ1795d+/fvV35+vmvdqVOnVF1drejo6Aa3a7FYPP6BrHYENBhw1bUBcqpGtapucHu1qvHpm6Ixr4G/oyf/56t+qmsbfv+eH+dpfab9G11JfHod3LBhw1RYWKjMzEyVlZVp2bJlSkhIUMeOHTVy5EhlZ2crNzdXpaWlWrx4sWw2m6xWqy9LBgC0ED4NuHbt2mn16tVauXKlBg0apIMHD2rJkiWSpOjoaGVkZGjOnDlKSkqS0+lUWlqaL8sFALQgXv+I8vXXX6/zeMCAAcrMzLzoWJvNJpvN5o2yAACGYaouAICRCDgAgJEIOACAkXx6mQAAeNt7uc+prLLha1mvCgpTVKs7vFARmgsBB+CK8kPNOf1Q3fCMSIEKcnvWIvgnPqIEABiJgAMAGImAAwAYiYADABiJgAMAGImAAwAYiYADABiJgAMAGImAAwAYiYADABiJgAMAGImAAwAYiYADABiJgAMAGImAAwAYiYADABiJgAMAGImAAwAYiYADABiJgAMAGImAAwAYiYADABiJgAMAGImAAwAYiYADABiJgAMAGImAAwAYiYADABiJgAMAGImAAwAYiYADABiJgAMAGImAAwAYiYADABiJgAMAGImAAwAYiYADABiJgAMAGImAAwAYiYADABiJgAMAGImAAwAYiYADABiJgAMAGMnvA27v3r2y2WyKj4/XuHHjdOTIEV+XBABoAfw64Ox2u1JSUjRp0iTt3LlT/fr108yZM31dFgCgBfDrgNu+fbs6deqk0aNHKywsTNOmTVNRUZEOHTrk69IAAH4uyNcF1KegoEDdu3d3PbZarYqOjlZRUZFuvPHGC8Y7nU5JUlVVlSwWi1v7cDgckqSIkADV1AbUOzbkqgAFB7WXo1Vtg9u1BoWoqqrKrRqa0vl+PHkN/B09+T9f93NN2wAFB9X//pU8ew+3DgqV5HlPFotFgYGBCghouB40L78OuLKyMrVv377OspCQENnt9ouOr6398Yf2wIEDHu/LdlOlmyMH6Zpg90bm5eV5XEdTacxr4O/oyf/5qp9hXTwZ7f57WGpcTwkJCUb84tLS+XXAhYaGqrKybvBUVFQoLCzsouODgoIUFxfHb08AfCow0K//+nPF8OuA69q1qzZv3ux6XFVVpaKiojofW/5UYGCgrFart8oDAPgxv/41Y9iwYSosLFRmZqbKysq0bNkyJSQkqGPHjr4uDQDg5wKc58/M8FN79uzRwoUL9e233yoxMVFLly4l4AAADfL7gAMAoDH8+iNKAAAa64oIOHem+7Lb7ZoxY4Z69+6tpKQkbdq0yQeVus+dnkpKSpSSkqLevXtr4MCBeuKJJ3Tu3DkfVOseT6dlmzVrlmbPnu2l6jznbj8vvPCCBg4cqL59+yo1NVVlZWVertR97vR05MgR/eY3v3H93C1atEg1NTU+qNZ9aWlpl3zP19TUaP78+UpMTNSgQYP04osverk6NJbxAefudF9LlixRVVWVPvjgAz377LNaunSp9u/f7/2C3eBuT08++aSqq6uVmZmpDRs26IsvvtDq1au9X7AbPJ2W7b333lNmZqb3CvSQu/1s3rxZWVlZWrt2rd5//32VlJTolVde8X7BbnC3pyeeeEJxcXHas2ePNmzYoJycHP35z3/2fsFu2LVrl5YsWaKtW7decsyaNWv05Zdfatu2bXrttde0fv16ffDBB16sEo1lfMC5M91XVVWV3n33Xc2ePVvXXHONbr75Zg0fPlx/+ctffFj5pbk7hVlOTo5SUlLUsWNH1/hdu3b5qOr6eTIt25kzZ7RixQqNGjXKB5W6x91+1q1bp8cee0y/+MUvFB4eruXLl+vuu+/2UdX1c7cni8WigIAA/fTP+/86YYO/+Pzzz2W32xUREXHJMVu2bNGMGTN03XXXKSYmRmPHjtW2bdu8WCUay/iAq2+6r/OOHj2q2tpade3a1bUsJiamzhh/4k5PkvTHP/5RsbGxrsdffPGFIiMjvVanJ9ztSZLmzZunRx55RNdee603S/SIO/1UVVUpPz9feXl5uvXWW9W3b1+tWLGixf8bPf7441q7dq0SEhI0dOhQRUVFadiwYd4u1y0zZszQokWL9Itf/OKi68vLy3Xs2LE6fcfGxvrt/w2oy/iAKysrU2hoaJ1l/zrd16XGlJeXe6VGT7nTkyT16tVLVqtVJSUl+v3vf6+cnBzNmjXLm6W6zd2ezl/4f88993irtEZxp59//vOfcjgc+uyzz7RlyxZt3bpVhw4d0ooVK7xdrlvc6cnhcOh3v/udJkyYoM8++0ybN2/WkSNH9MYbb3i73CZxvref9u3P/zegLuMDzp3pvsLCwlRRUVHvGH/iyRRmmzdv1ogRI2S32/XOO++oW7du3irTI+709N133+mFF17QwoULvV2exzz5N0pNTVWHDh0UGRmpBx98UB9//LG3yvSIOz0VFhbq22+/1e9+9zu1bdtWPXv21IQJE7Rz505vl9skzgfbT/9/OHfu3AVBD/9kfMB17dpV+fn5rscXm+4rKipKDodDxcXFrmWHDx/WTTfd5NVa3eVOT5L00ksvadmyZVq+fLmef/55v/3oS3Kvp/379+v48eMaPHiwYmNj9fzzz2vbtm26+eabfVFyvdzpJyIiQm3atHHNxC/9OGF4cLAHMwF7kTs9tW7d+oK/vwUFBalNmzZerbWpBAcHKyoqqk7fhw8fvuR0gfAvxgecO9N9BQcHa8SIEVq+fLnKysr0f//3f3r//fdls9l8WPmludPTuXPn9OKLL2rlypW69dZbfVite9zpaejQoSosLHR9paSkaOTIkcrNzfVh5RfnTj+BgYGy2WxatWqVTp06pW+//VavvPJKi/65i46OVlRUlFauXCm73a7Dhw9r/fr1uvPOO31Y+eW555579OKLL6qkpER5eXl64403dO+99/q6LLjDeQXYvXu3c/jw4c64uDjn5MmTnd99953T6XQ6Y2JinB9//LHT6XQ6S0tLnQ8//LAzLi7OOWTIEOf777/vy5Ib1FBPf//7350xMTEXfN1///0+rvzS3Pl3+qn//M//dD766KPeLtNt7vRz7tw5Z0ZGhrN///7O2267zblixQpnTU2NL8uulzs9FRUVOadMmeLs06ePMykpyfnHP/7RlyW75f7773du3LjR9TgpKcm5efNmp9PpdP7www/OuXPnOhMSEpyDBg1yrlu3zldlwkNM1QUAMJLxH1ECAK5MBBwAwEgEHADASAQcAMBIBBwAwEgEHADASAQcAMBIBBwAwEj/D2/piK9mbfDmAAAAAElFTkSuQmCC\n", "text/plain": [ "
" ] }, "metadata": {}, "output_type": "display_data" } ], "source": [ "conversion = lambda x: int(['payment_done'] in x['event'].unique())\n", "\n", "data.rete.compare(groups=(test, control),\n", " function=conversion,\n", " test='mannwhitneyu',\n", " group_names=('test','control'))" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Parameters for function rete.compare():\n", "\n", "* groups: tuple (g1, g2), where g1 and g2 are collections of user_id`s (list, tuple or set) of two groups for comparison.\n", "\n", "* function(x): function that takes a single user dataset as an argument and returns a single numerical value (see below for more examples).\n", "\n", "* test: {‘mannwhitneyu’, 'ttest', ‘ks_2samp’}, selected statistical test to test the null hypothesis that 2 independent samples are drawn from the same distribution. One-sided tests are used, meaning that distributions are compared for ‘less’ or ‘greater’. Rule of thumb is: for discrete variables (like conversions, number of purchases) use Mann-Whitney test (‘mannwhitneyu’) or t-test ('ttest'). For continuous variables (like average_check) use Kolmogorov-Smirnov test (‘ks_2samp’).\n", "\n", "* group_names - optional parameter to set group names for the output.\n", "\n", "* alpha - Selected level of significance, to calculate power of the test: probability to correctly reject a H0 when H1 is true. Default value is 0.05.\n", "\n", "We can see that in the example above the test group has statistically significantly higher conversion rate than the control group (given P-value threshold for significance 0.05), therefore change must be implemented. Histogram just illustrates how the selected metric is distributed between groups (in the example above metrics can only be 0 or 1). \n", "\n", "To illustrate better how to define custom metrics and pass it as an argument to rete.compare() function let’s compare a couple more metrics. Suppose we would like to compare the average check between test and control groups. Again, it’s very easy:\n", "\n" ] }, { "cell_type": "code", "execution_count": 9, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "test (mean ± SD): 736.026 ± 149.001, n = 126\n", "control (mean ± SD): 732.980 ± 139.960, n = 85\n", "'control' is greater than 'test' with P-value: 0.55199\n", "power of the test: 3.65%\n" ] }, { "data": { "image/png": "iVBORw0KGgoAAAANSUhEUgAAAbgAAAFOCAYAAAD0AQi3AAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjMuMSwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy/d3fzzAAAACXBIWXMAAAsTAAALEwEAmpwYAAAwOklEQVR4nO3de1gTd74/8HfIBSkKFlk9XfRoFUGlIIjXctYutghUsFZ/3mg5unqOYhW8lK0VraLWqr24+mg96lmt1WotVm29rFqVLdWqxUupaFW0ysVLvYFKEA2ZzO8PlxxCAiQBkmF4v57H5yEz38nn+zHDvJOQzChEURRBREQkMy7OngAREVF9YMAREZEsMeCIiEiWGHBERCRLsg04URQhCAL4GRoiosZJtgFnMBiQlZUFg8Fg1XhBEHDq1CkIglDPM3MMufUDyK8n9iN9cuypMZFtwBERUePGgCMiIlliwBERkSwx4IiISJYYcEREJEsMOCIikiUGHBERyRIDjoiIZIkBR0REssSAIyIiWWLAERGRLDHgiIhIlhhwREQkSww4IiKSJZWzJyA1C7f+hvuPrLvETkVNmyiRGtexHmZERET2YMBVUvJYQMlj2wOOiIikhW9REhGRLDHgiIhIlhhwREQkSww4IiKSJQYcERHJEgOOiIhkiQFHRESyxIAjIiJZYsAREZEsMeCIiEiWGHBERCRLDDgiIpIlBhwREckSA46IiGSJAUdERLLEgCMiIlliwBERkSw5POCmT5+OrVu3AgCWL18Of39/s38zZsww2+7cuXNm47744gtHT5+IiBoIlaMKHTlyBIcPH8bOnTvRvXt3AEBiYiISExONY+7du4fhw4dj7NixZtvn5eVh0KBBWLx4saOmTEREDZjDXsGdPn0aWq0WLVq0qHLMe++9h7i4OPj6+pqtKygoQLt27epxhkREJCcOewWXlJQEAIiPj7e4PiMjAxcuXMDSpUstrs/Ly8Ply5exadMmCIKA2NhYJCcnQ6PRVFtXEASr5lc+TuUiQq0UrdqmIrWLaHUtRyifi5TmVFty64n9SJ+tPSmVyvqcDtnIYQFXk1WrViEpKanKwFIoFOjduzdGjx6NoqIiTJ06FStWrMC0adOqvd/s7Gyb5jEooNSm8RVlZWXZvW19sbX/hkBuPbEf6bO2p9DQ0HqeCdlCEgF35swZ5OfnY8CAAVWOWbBggfFnLy8vTJgwAStXrqwx4AIDA616ViUIArKzs/HNOTc8KDVYP/l/cXdV4v14P5u3qy/l/Vjbf0Mgt57Yj/TJsafGRBIBl5aWhujoaKjVaovrdTodVq5cibFjx6JZs2YAAL1eD3d39xrvW6lU2rRj6g0KlAkKq8eXKzMoJPkLYGv/DYHcemI/0ifHnhoDSXwP7ujRowgLC6tyvUajQXp6OpYtW4b79+/jypUrWLNmDWJjYx04SyIiakicHnDXrl3D9evXERQUZLauX79+2L59O4Cn35m7cuUKwsPDMW7cOERHR2PkyJGOni4RETUQDn+LcuPGjSa3W7dujYsXL1ocm56ebvy5bdu2WLduXb3OjYiI5MPpr+CIiIjqgyQ+ZEKOl7r5ErSPbfu+UtMmSqTGdaynGRER1S0GXCOlfSygxMaAIyJqSPgWJRERyRIDjoiIZIkBR0REssSAIyIiWWLAERGRLDHgiIhIlhhwREQkSww4IiKSJQYcERHJEgOOiIhkiQFHRESyxIAjIiJZYsAREZEsMeCIiEiWGHBERCRLDDgiIpIlBhwREckSA46IiGSJAUdERLLEgCMiIlliwBERkSwx4IiISJYYcEREJEsMOCIikiUGHBERyZLDA2769OnYunWr8XZCQgL8/f2N/3r16mVxO61Wi6SkJISEhCA8PNzkPoiIiCpTOarQkSNHcPjwYezcuRPdu3c3Ls/Pz8fBgwfRpk2bardfuHAhdDodDhw4gNzcXIwfPx6dO3fGCy+8UN9TJyKiBshhAXf69GlotVq0aNHCuEwURdy+fRs+Pj7VbqvT6bB7925s27YN3t7e8Pb2RmRkJPbs2cOAIyIiixwWcElJSQCA+Ph447Lff/8dCoUCcXFxuHjxItq3b4+UlBSEhoaabJubmwuDwQBfX1/jMj8/P2RmZtZYVxAEq+ZXPk7lIkKtFK3apiK1i2h1LUcon0tVc1Lb0aeze6ypp4aG/UifrT0plcr6nA7ZyGEBZ0lhYSE6dOiAd955B35+fvj666+RkJCAvXv3wtvb2ziuuLgYHh4eJtu6u7ujpKSkxhrZ2dk2zWlQQKlN4yvKysqye9v6UlX//y/QvvuTQo+2PqZSx36kz9qeKj85J+dyasAFBARgy5YtxtujR4/G1q1bcerUKURGRhqXe3p6orTUNHhKS0vh6elZY43AwECrnlUJgoDs7Gx8c84ND0oNNnTxlLurEu/H+9m8HQDM2piDkie2Pet1d1Wid6cd0JbetbjeBWr4eQxDzsM0GFBmsq6pmzeOX3jdrpr29lgXyh8jax9TqWM/0ifHnhoTpwZcRkYGdDodIiIijMv0ej3c3d1Nxvn4+EAQBBQUFBg/jHL58mV07ty5xhpKpdKmHVNvUKBMUFg9vlyZQWH3L0CZHTXLDAqI0JuFV2UGlJmNEaG3u6YUfsltfUyljv1Inxx7agyc+j04nU6H1NRUZGVlQavVYv369dDpdOjRo4fJODc3N0RFRWHJkiUoLi7Gjz/+iP379yMmJsZJMyciIqlz6iu4iIgIFBQUYPLkydBqtQgICMDf//53uLq6AgD8/f2xYcMG9OrVCzNmzEBKSgrCwsLQsmVLzJ8/v8avFhARUePl8IDbuHGjye0xY8ZgzJgxFsdevHjR+HPz5s2xcuXKep0bERHJB0/VRUREssSAIyIiWWLAERGRLDHgiIhIlhhwREQkSww4IiKSJQYcERHJEgOOiIhkiQFHRESy5NRTdRFR7aVuvgTtY9uuDNG0iRKpcR3rtKbaRcT/C3x6dYwyg/mJvGtbk8hWDDiiBk77WECJjQFXHzXLL6Bb8kSw64ocRHWNb1ESEZEsMeCIiEiWGHBERCRLDDgiIpIlBhwREckSA46IiGSJAUdERLLEgCMiIlliwBERkSwx4IiISJYYcEREJEsMOCIikiUGHBERyRKvJkCS9+3x+XiiL4ELVHje7XXsODYHBuir3aZZE2/8lPOazZeRAWp3WRdbLl1T8fIyrhoVLyVTA2dcFogaNgYcSd4TfQmelJXABWrADXhS9ggGlFW7jUb1jGQuI1MVk8vLWLh+GplyxuNJDRvfoiQiIlliwBERkSw5POCmT5+OrVu3Gm//8MMPiImJQVBQEKKiorBr1y6L2507dw7+/v4m/7744gtHTZuIiBoYh/0N7siRIzh8+DB27tyJ7t27AwCKioowefJkvPvuuxgwYACOHTuGqVOnomPHjujUqZPJ9nl5eRg0aBAWL17sqCkTEVED5rCAO336NLRaLVq0aGFcdvLkSfj4+GD48OEAgIiICPj5+eHYsWNmAVdQUIB27drZXFcQrPujdPk4lYto/OO/LdQuotW1LG1ra021iwgFVE8/eGFB+XJL6xVQ2V3T3h5rw+VffVbXU2X29gg47rEsH6dWik7Zf2rzWFqqWbEfR9W0Zpva1Czf1tr7UCqVdteiuqcQRdH2I0AtxMfHY+DAgRg6dCgKCwtRWFgIX19fAEBhYSEiIyMxb948REdHm2yXkpKCy5cv48aNGxAEAbGxsUhOToZGo7FYRxAEZGVl1Xc7RERGoaGhzp4CVeDUrwl4eXnBy8sLwNNXc7NmzUJQUBD69+9vNlahUKB3794YPXo0ioqKMHXqVKxYsQLTpk2rtkZgYKBVz6oEQUB2dja+OeeGB6UGm3txd1Xi/Xg/m7cDnn4PquSJbc8y3V2V6N1pB7Sldy2ud4Eafh7DkPMwzewj9U3dvHH8wut21bS3x9rYcWwOnpQ9qranyuztEXDcY6lWiogL0WHzzxpoVCqH7z+1eSwt1azYT5lg/rWH+qhZk9rWLD8uWHscIWlx+vfgtFot5s2bh4MHD2LixIkYNWqUxR1pwYIFxp+9vLwwYcIErFy5ssaAUyqVNu2YeoPC4i9nTcoMCrt/AcrsqFlmUECEvsYDvQFlZmNE6O2u6YxfckOlPi31VJm9PQJOeCwFBRQujt9/avNYVlezTLC8rj5rVrdNXeyzth5HSBqc+jWBx48fIy4uDkVFRdi3bx/Gjh0Llco8c3U6HZYuXYri4mLjMr1eD3d3d0dOl4iIGhCnvoLbtWsXdDodPv300yr/lgYAGo0G6enp0Gq1mDRpEgoLC7FmzRqMGDHCgbMlIqKGxKmv4M6fP4+rV68iMDDQ5Ptt27dvBwD069fP+PPy5ctx5coVhIeHY9y4cYiOjsbIkSOdOX0iIpIwh7+C27hxo/Hn2bNnY/bs2VWOTU9PN/7ctm1brFu3rl7nRkRE8sFTdRERkSwx4IiISJYYcEREJEsMOCIikiUGHBERyRIDjoiIZIkBR0REssSAIyIiWWLAERGRLDHgiIhIlhhwREQkSww4IiKSJQYcERHJEgOOiIhkiQFHRESyxIAjIiJZsjrgOnfujIKCArPlly9fRrdu3ep0UkRERLVV4xW9Bw0aBIVCAVEUMWHCBKjVapP1hYWFaNasWb1NkIiIyB41Btwrr7wCADh//jx69+6N5s2bm6zXaDQIDw+vl8kRERHZq8aAmzRpkvHn//zP/4SHh0e9ToiIiKgu1Bhw5SZNmoQrV67g6NGjePz4sdn6QYMG1eW8iIjIAQwGA1xc5Pl5Q6sDbvXq1Vi6dCmUSiXc3NxM1ikUCgYcEZFEPXjwANOnT8f9+/fRoUMH5OfnIzc3F926dUNISAj+9Kc/Yf78+dDr9XB1dUVqaioUCgWmTZuGtLQ0AMCwYcOwZMkS7NixAzk5Obh//z7u37+PpKQkRERE4JNPPkFmZiZEUcRrr72GN954w8ld2xBwn332GRISEjBp0iQolcr6nBMREdWhNWvW4E9/+hPeeOMN/PTTT1ixYgXu3buHd955Bz4+PoiLi0NKSgpeeOEFpKen44MPPsDMmTOrvD+VSoWNGzfizp07eP311/HSSy9h27Zt+Mc//oGmTZvis88+c2B3VbP6dWlZWRmioqIYbkREDUxOTg5CQ0MBAIGBgQCAZ599Fj4+PgCA/Px8vPDCCwCAbt26IT8/3+w+DAaD8efu3bsDAP7whz/g2WefhVarxYcffoiFCxdi2rRpknnL0+pZvPrqq9i/f399zoWIiOrBc889hwsXLgAAjh49arbe29sbv/32GwDgxIkT6NKlCzQaDUpLSwEAjx49Mvke9Llz5wAAt2/fRmlpKURRxJEjR7B48WIsW7YMaWlpuHPnTn23VSOr36L09PTE2rVrcfz4cXTp0sXsldyMGTPqfHJERFR7CQkJmD59OrZs2YL27dubvcKaN28eZs6cCbVaDTc3N8yfPx8tW7ZEQEAAkpOTIYoivLy8jONv3bqFN998Ew8fPsTs2bPRokUL6PV6DBs2DHq9Hi+99BL+8Ic/OLpNM1YH3C+//IKgoCAAwMWLF03WKRSKup0VERHVmYKCAvz1r39FUFAQMjMz8c033+Dzzz83rg8KCsKWLVvMtlu0aJHF+3vllVcwcuRIk2WzZs2q20nXAasDbuPGjfU5DyIiqife3t7GV2hNmjTB+++/7+wpOYTVAXfo0KFq17/88stW3c/06dPRvXt3DB06FMDT93vnzp2L/Px8BAQEYMGCBWjfvr3ZdlqtFikpKTh8+DCaN2+Ot956y3gfRERUtQ4dOlh8hWaPxMTEOrkfR7A64CZOnGhxuVKpRPPmzWsMuCNHjuDw4cPYuXOn8RM4Wq0WkyZNQnJyMvr3749169ZhypQp2Llzp9n2CxcuhE6nw4EDB5Cbm4vx48ejc+fOxk/+EBERVWR1wJV/AqecTqfDb7/9hmXLlhnPV1md06dPQ6vVokWLFsZlhw4dQps2bYyvxCZOnIj169fj0qVL6Nixo0mt3bt3Y9u2bfD29oa3tzciIyOxZ8+eGgNOEASr+isfp3IRoVaKVm1TkdpFtLqWpW1tral2EaGACi5QW1xfvtzSegVUdte0t8facPlXn9X1VJm9PQKOeyzLx6mVolP2n9o8lpZqVuzHUTWt2aY2Ncu3tfY++DUqaVGIomj7EaCC4uJiDB48GAcOHLBqfHx8PAYOHIihQ4di8eLFKCkpwbx584zrX3vtNSQmJpqEZk5ODoYMGYLs7GzjsvXr1yMzMxMrV660WEcQBGRlZdnXFBGRHcq/a0bSYPUruKqUlZXZ/X2H4uJis6sTuLu7Q6vVmo2rfJJnd3d3lJSU1FgjMDDQqmdVgiAgOzsb35xzw4NSQ43jK3N3VeL9eD+btwOAWRtzUPLEtmeZ7q5K9O60A9rSuxbXu0ANP49hyHmYBgPKTNY1dfPG8Quv21XT3h5rY8exOXhS9qjaniqzt0eg5v/buqqpVoqIC9Fh888aaFQqh+8/tXksLdWs2E+ZYP7J6vqoWZPa1iw/Llh7HCFpsTrgFi5caLbs8ePHOH78uPFvarby8PAwO3FzaWkpPD09TZZ5enoav3BY3ThLlEqlTTum3qCw+MtZkzKDwu5fgDI7apYZFBChr/FAb0CZ2RgRertrOuOX3FCpT0s9VWZvj4D1/7d1VbNMUEDh4vj9pzaPZXU1ywTL6+qzZnXb1MU+a+txxF6pmy9B+7j+/gzQtIkSqXEdax4oE1YH3K+//mq2rPxacOPGjbOruK+vL7Zt22a8rdPpkJeXhy5dupiM8/HxgSAIKCgoQJs2bQA8vZJ4586d7apLRCRF2scCSuox4Gzl7++P7777Dm3btrV522vXruHll1/GuXPnoFLV+s1Cuzj1e3ARERH44IMPsG/fPoSFhWHFihUIDg5Gq1atTMa5ubkhKioKS5Yswbx583DmzBns378fX3/9dZ3PiYiI5MGmM2LevHkTH330Ed566y28/fbb+N///V88fPjQ7uLNmjXD8uXLsXTpUoSFhSEnJ8fkrVB/f3/89NNPAJ6eCuzJkycICwvDnDlzMH/+fOOrOSIiqlt9+/YFAPTv3x9nzpzBp59+ivDwcPTo0QMLFiyATqcD8PTdtBEjRiA4OBhRUVH47rvvoNfrjV8dCwgIQGFhoVN6sPoV3NmzZxEfHw8/Pz906tQJjx8/xpYtW7BmzRqsW7fOeIbqmlR+JdinTx/s27fP4tiKpwRr3rx5lZ+YJCKiuvXDDz8Y36L8/vvv8cMPP2DTpk0QRRFvv/021qxZg0mTJmHu3LkIDw/H559/jhMnTuCtt97Cjz/+iEOHDjn9LUqrX8F9+OGHiI+Px1dffYW5c+di8eLFOHToECIjI6s8XxkRETV8X331FaZMmYI//vGP8PHxQWJiIvbu3QsAcHFxwdmzZ3Ht2jWEhYXhn//8J5555hknz/gpqwPu7NmziImJMVs+YsQI46UTiIhIfq5fv47Ro0fD398f/v7+GDNmDG7dugUA+Pjjj/Fv//ZvSEhIQFhYGNasWePk2f4fq183Nm3aFLm5ufDzM/1Oya1bt+Dq6lrnEyMiImnw9vbG3/72N+MVZbRaLW7evAlBEHD06FFMnz4dM2fOxNWrVzFq1Cj06NEDnTp1cvKsbQi4IUOGIDU1FY8ePUKPHj2gUqmQlZWFRYsWITo6uj7nSETUKDRtUr/ftbP1/pVKJQoLCxEbG4v/+Z//wYIFC6DX6zFz5ky0bdsWs2bNwtKlS1FUVIRhw4ZBp9NBp9Ph2WefNf7drbCwEC1btqyPdmpkdcAlJiZCqVRi3rx5xi9dK5VKDBkyBO+++269TZCIqLGQ2pewIyIiEB8fj6+//holJSV49dVXIQgCXnnlFUybNg3A089nvP/++/jb3/6G5s2bY+zYsQgNDYXBYEBISAj69euHI0eOmJ21yhFqDDiDwYDdu3ejXbt2mDRpEsaNG4crV65gyZIlGDFiBPr16+eIeRIRkYMtW7bM+PPMmTMxc+ZMszE9evTAt99+a7bcxcWlzi7RY69qP2RSUlKC+Ph4pKSkGM83qdFo8O///u8oKirCxIkT8d577zlkokRERLaoNuCWLVuGu3fvYu/evSbXe3vmmWewdetWbNq0Cenp6SaXPiciIpKCagPu4MGDmDJlSpVnDOnWrRuSk5N5yiwiIpKcav8Gd/v2bZMLj1oSFBSEvLy8Op0UUWO09+THKH5s2yV6mjXxBmD+/VQiqiHgmjdvjtu3b8PX17fKMXfv3oWbm1udT4yosXmif4QnZTVf47AijUoaZ4wgkqJq36Ls27cvVq9eDYOh6guAfv7553jxxRfrfGJERES1UW3ATZ48Gb/99hvefPNNZGRkoKioCAaDAXfu3EFGRgbi4+Nx8uRJTJkyxUHTJSIisk61b1G2atUKX375JVJTUzF+/HgoFP93NV1RFNGjRw9s3rzZrovhERER1acav+jdpk0brF27Fjdv3sSvv/6KR48eoVmzZujcubPZhUmJiMh+3x6fjyd62/4OawtXlTte6+287y7Hx8dj4MCBGDp0qE3b2Xt1cKtHPvfcc3juuedsmhQREVnvib7E5g8ayYFer6+Xa8bZdEVvIiJqXI4dO4aBAwciODgYb7zxBnJzc3Hz5k2MHz8eoaGhiIyMxLZt24zj/f398eWXX6Jfv37o2rUrZsyYAVEUkZycjMzMTMyaNQtr167F9u3bER8fj+TkZAwYMAAAcPLkSQwePBghISEYNmwYsrKyajV3BhwREVl08+ZNJCYmYsaMGTh27BiCgoIwZ84cTJgwAX5+fvjnP/+JhQsX4pNPPsGRI0eM2x04cABbt27FV199hX/84x84deoUPv74Y/Ts2RPvv/8+xo4dCwA4ceIEevbsiV27duHWrVsYP348xo0bh8OHD2PEiBEYP3487t617buhFTHgiIjIot27d+PFF19Enz594Obmhrfeegs9evTAzZs3MXnyZHh4eKBbt24YMWIE9uzZY9wuMTERLVq0QKdOndChQwdcv37d4v23b98ew4YNg0ajwc6dO9GzZ09ERUWhadOmGDx4MDp27GgSnLZiwBERkUXXr19H69atjbebNWuG9u3bo3Xr1iZ/M2vRooXJKy0vLy/jz66urhAEweL9e3p6mtRq166dyXovLy/jif7twYAjIiKLKgfMrVu3sGnTJty6dQuiKBqXX7161SQI7dGiRQv8/vvvJstyc3Nrdb8MOCIiiXBVucNVXY//VO42zScqKgrp6ek4ceIEiouL8eGHH6JVq1bQaDRYtWoVtFotTp48iZ07d2Lw4ME13l/5FcItiY6Oxvfff4+MjAxotVqkpaXh7t27+POf/2zTnCuq+89lEhGRXZz5HTVL/Pz8MG/ePKSkpODevXt48cUXMX/+fNy9exdz5szBqlWr0KJFC/z1r39FYGBgjfcXHh6Ojz/+GM888wzc3U3D1tfXF4sXL8aiRYtw7do1dOzYEStWrKjVuY4ZcEREVKUBAwYYP8Zf7tlnn8XmzZstjr948aLJ7S+//NL486hRozBq1Cjj7cqv+vr374/+/fub3Wfr1q3N7tcaDLg6xMudVC918yVoH1v+Y3NVmjZRIrj9M43yy6/1zZ79FWh4+6y9fbqq3BHTI6UeZkSOwoCrQ7zcSfW0jwWU2BhwVH/s2V+BhrfP2tsnNXz8kAkREckSA46IiGRJEgG3fPly+Pv7m/2bMWOGybhz586Zjfniiy+cNGsiIpIySfwNLjExEYmJicbb9+7dw/Dhw43nKyuXl5eHQYMGYfHixY6eIhERNTCSCLjK3nvvPcTFxcHX19dkeUFBgdmpXGpS1SliqhqnchGhVoo1jDandhGhgAouUNu0nQIqqO2oWVO98uWW1temprX/n1VtX5s+q+upMnt7rFzTFrbWLB+nVoqS23/sqVmxn6pqOnP/sZULVMb5WjtvpVJpcx2qPwqx4vlWJCAjIwNz587Fvn37oNFoTNalpKTg8uXLuHHjBgRBQGxsLJKTk83GAU93yNpeaoGIyBahoaHOngJVILlXcKtWrUJSUpLF0FIoFOjduzdGjx6NoqIiTJ06FStWrMC0adOqvL/AwECrnlUJgoDs7Gx8c84ND0oNNs/b3VWJ3p12QFtq2/dtmrp54/iF11HyxLZntjXVc4Eafh7DkPMwDQaU1VnN9+P9bNqmolkbc2rVZ3U9VWZvj5Vr2sLWmmqliLgQHTb/rIFGpZLU/mNPzYr9lAkKizWduf/YylX9DAb2nI3s7GyrjyMkLZIKuDNnziA/P9/sW/PlFixYYPzZy8sLEyZMwMqVK6sNOKVSadOOqTcoLP5y1qTMoIAIfY0H3spE6FFmR01r6xlQZjamNjVr80teV31a6qkye3usqqY17P5/FRRQuEhz/7GnZplgeZ1U9h9rGaA3ztfW4whJgyQ+RVkuLS0N0dHRUKvN3y/X6XRYunQpiouLjcv0er3Z+cyIiIgAiQXc0aNHERYWZnGdRqNBeno6li1bhvv37+PKlStYs2YNYmNjHTxLIiJqCCQTcNeuXcP169cRFBRksrxfv37Yvn07gKffl7ty5QrCw8Mxbtw4REdHY+TIkc6YLhERSZxk/gZX1dmi09PTjT+3bdsW69atc+S0iIiogZLMKzgiIqK6JJlXcNQwNJZLrFD94P5DjsSAI5s0lkusUP3g/kOOxLcoiYhIlhhwREQkSww4IiKSJQYcERHJEgOOiIhkiQFHRESyxIAjIiJZYsAREZEsMeCIiEiWGHBERCRLDDgiIpIlBhwREckSA46IiGSJAUdERLLEgCMiIlliwBERkSwx4IiISJYYcEREJEsMOCIikiUGHBERyRIDjoiIZIkBR0REssSAIyIiWWLAERGRLDHgiIhIliQTcAkJCfD39zf+69Wrl9kYrVaLpKQkhISEIDw8HFu3bnXCTImIqCFQOXsC5fLz83Hw4EG0adOmyjELFy6ETqfDgQMHkJubi/Hjx6Nz58544YUXHDhTIiJqCCQRcKIo4vbt2/Dx8alyjE6nw+7du7Ft2zZ4e3vD29sbkZGR2LNnT7UBJwiCVXMoH6dyEaFWirY1AEDtIkIBFVygtmk7BVRQ21Gzpnrlyy2tr6+a1amLmtX1VFf1Kte0ha01y8eplaLk9h97albsx1E1a1Kbmi5QGY8L1h5HlEqlzXWo/ihEUbT9CFDHbt68iYEDB6JDhw64ePEi2rdvj5SUFISGhhrH5OTkYMiQIcjOzjYuW79+PTIzM7Fy5Uqz+xQEAVlZWY6YPhERAJgcs8j5JPEKrrCwEB06dMA777wDPz8/fP3110hISMDevXvh7e0NACguLoaHh4fJdu7u7igpKan2vgMDA616ViUIArKzs/HNOTc8KDXY3IO7qxK9O+2AtvSuTds1dfPG8Quvo+SJdc8Qra3nAjX8PIYh52EaDChzSM3q1EXN6nqqq3qVa9rC1ppqpYi4EB02/6yBRqWS1P5jT82K/ZQJCofUrEltarqqn8HAnrORnZ1t9XGEpEUSARcQEIAtW7YYb48ePRpbt27FqVOnEBkZCQDw9PREaWmpyXalpaXw9PSs9r6VSqVNO6beoLD4y1mTMoMCIvQ1HngrE6FHmR01ra1nQJnZmPquaUld1rTUU13Vq6qmNezuUVBA4SLN/ceemmWC5XVS2X+sZYDeeOyw9ThC0iCJT1FmZGTgwIEDJsv0ej3c3d2Nt318fCAIAgoKCozLLl++jM6dOztsnkRE1HBIIuB0Oh1SU1ORlZUFrVaL9evXQ6fToUePHsYxbm5uiIqKwpIlS1BcXIwff/wR+/fvR0xMjBNnTkREUiWJtygjIiJQUFCAyZMnQ6vVIiAgAH//+9/h6uoKf39/bNiwAb169cKMGTOQkpKCsLAwtGzZEvPnz6/2awVERNR4SSLgAGDMmDEYM2aM2fKLFy8af27evLnFT0wSERFVJom3KImIiOoaA46IiGSJAUdERLLEgCMiIlliwBERkSwx4IiISJYYcEREJEsMOCIikiUGHBERyRIDjoiIZIkBR0REssSAIyIiWWLAERGRLDHgiIhIlhhwREQkSww4IiKSJQYcERHJEgOOiIhkiQFHRESyxIAjIiJZYsAREZEsMeCIiEiWGHBERCRLDDgiIpIlBhwREckSA46IiGSJAUdERLLEgCMiIlmSTMD98MMPiImJQVBQEKKiorBr1y6zMefOnYO/v7/Jvy+++MIJsyUiIqlTOXsCAFBUVITJkyfj3XffxYABA3Ds2DFMnToVHTt2RKdOnYzj8vLyMGjQICxevNiJsyUiooZAEgF38uRJ+Pj4YPjw4QCAiIgI+Pn54dixYyYBV1BQgHbt2tl034Ig2DRO5SJCrRRtqgEAahcRCqjgArVN2ymggtqOmjXVK19uaX191axOXdSsrqe6qle5pi1srVk+Tq0UJbf/2FOzYj+OqlmT2tR0gcp4XLD2OKJUKm2uQ/VHIYqi7UeAOlZYWIjCwkL4+voab0dGRmLevHmIjo42jktJScHly5dx48YNCIKA2NhYJCcnQ6PRmN2nIAjIyspyVAtERAgNDXX2FKgCSbyC8/LygpeXF4Cnr+ZmzZqFoKAg9O/f32ScQqFA7969MXr0aBQVFWHq1KlYsWIFpk2bVuV9BwYGWvWsShAEZGdn45tzbnhQarC5B3dXJXp32gFt6V2btmvq5o3jF15HyRPrniFaW88Favh5DEPOwzQYUOaQmtWpi5rV9VRX9SrXtIWtNdVKEXEhOmz+WQONSiWp/ceemhX7KRMUDqlZk9rUdFU/g4E9ZyM7O9vq4whJiyQCDgC0Wi3mzZuHgwcPYuLEiRg1apTZDrVgwQLjz15eXpgwYQJWrlxZbcAplUqbdky9QWHxl7MmZQYFROhrPPBWJkKPMjtqWlvPgDKzMfVd05K6rGmpp7qqV1VNa9jdo6CAwkWa+489NcsEy+uksv9YywC98dhh63GEpEESn6J8/Pgx4uLiUFRUhH379mHs2LFQqUyzV6fTYenSpSguLjYu0+v1cHd3d/R0iYioAZDEK7hdu3ZBp9Ph008/tfj3NADQaDRIT0+HVqvFpEmTUFhYiDVr1mDEiBEOni0RETUEkngFd/78eVy9ehWBgYEm33Hbvn07+vXrh+3btwMAli9fjitXriA8PBzjxo1DdHQ0Ro4c6eTZExGRFEniFdzs2bMxe/Zsi+sGDx5s/Llt27ZYt26do6ZFREQNmCRewREREdU1BhwREckSA46IiGSJAUdERLLEgCMiIlliwBERkSwx4IiISJYYcEREJEsMOCIikiUGHBERyRIDjoiIZIkBR0REssSAIyIiWWLAERGRLDHgiIhIlhhwREQkSww4IiKSJQYcERHJEgOOiIhkiQFHRESyxIAjIiJZYsAREZEsMeCIiEiWGHBERCRLDDgiIpIlBhwREckSA46IiGRJMgF34sQJxMTEICgoCCNHjsSVK1fMxmi1WiQlJSEkJATh4eHYunWrE2ZKREQNgSQCTqvVYtKkSRg1ahQOHz6Mnj17YsqUKWbjFi5cCJ1OhwMHDuCjjz7CokWLcPbsWcdPmIiIJE8SAXfo0CG0adMGQ4cOhaenJyZOnIi8vDxcunTJOEan02H37t1ITk6Gt7c3unfvjsjISOzZs8eJMyciIqlSOXsCAHDhwgV06dLFeFuj0aBdu3bIy8tDx44dAQC5ubkwGAzw9fU1jvPz80NmZqbF+xRFEcDTYFQqlTXOQRAEAECLpgqoXBQ29+DeRAE3VXMIaoNN27mpmsO7qQJuKttq1lRP8a+H1k3tBRF6h9SsTl3UrK6nuqpXuaYtbK2p+tfTyxbuCrhqpLX/2FOzYj96g/l8nL3/2EqjcodOpwNg/XFEqVTCxcUFCoXt+x3VPUkEXHFxMZo3b26yzN3dHVqt1mSMh4eH2ZiSkhKL92kwPN2hf/31V5vmEv685fuzThi83WzfKqKDtuZBdtZr6xbt8JpVqauaVfVUd/XMa1rLnpoxnR87vGZt6tVU8//6cVzN6tlfs/z4YctxJDg42KowpPoniYDz8PDA48emvxSlpaXw9PQ03vb09ERpaWm1YypSqVQIDAzksykicigXF0n85YcgkYDz9fXFtm3bjLd1Oh3y8vJM3rb08fGBIAgoKChAmzZtAACXL19G586dLd6ni4sLNBpN/U6ciIgkSxJPNSIiInDx4kXs27cPxcXF+OSTTxAcHIxWrVoZx7i5uSEqKgpLlixBcXExfvzxR+zfvx8xMTFOnDkREUmVQiz/NIaTHTt2DHPnzsWNGzcQGhqKRYsWoVWrVvD398eGDRvQq1cv3L9/HykpKThy5AhatmyJd955B/3793f21ImISIIkE3BERER1SRJvURIREdW1RhNwCQkJ8Pf3N/7r1asXAOC7777Dyy+/jK5du2LcuHG4c+eOcZvNmzfjP/7jPxASEoLk5OQqv5LgLLdv38Z///d/G09dtnnzZgDVn/asun6dafny5SaPT/m/GTNmNMh+AGDbtm2IiIhAUFAQYmJicOjQIQAN8/EBgIyMDAwYMABdu3ZFXFwcrl69CqBh9jN9+nSTU/3Z24PUjxGNnthIREdHi/n5+SbL8vPzxeDgYPHQoUPivXv3xOnTp4sJCQmiKIriiRMnxJ49e4qnT58Wb9++LY4ZM0acP3++M6ZepTfeeEP86KOPxAcPHoinT58Wg4KCxPPnz4s9e/YU09LSxPv374tLliwRY2NjRVGsvl+puXv3rvjyyy832H5yc3PFwMBAMSsrS3zy5Im4Z88eMSAgQCwuLm6Q/dy4cUMMDg4WDx48KGq1WnH16tViTEyM+PDhwwbVz+HDh8UPPvhA7NSpk5iWliaKomj3Y9IQjhGNXaMIOIPBIIaGhoqCIJgs//TTT8WkpCTj7Rs3bohdunQRHzx4IKakpIiLFy82rjtx4oTYq1cvh825Jjk5OeJLL71k0tP58+fFHTt2iEOGDDEue/LkiRgUFCTm5ORU26/UTJgwQVy7dq34zTffNMh+yg+MP//8s/jkyRNx9+7dYq9evcTt27c3yH42bdok/uUvfzFZ1q1bN3H58uUNqp9ly5aJKSkpYlhYmDHg7N3HpH6MIFFsFG9R/v7771AoFIiLi0NISAiGDBmCU6dO4cKFCwgICDCOe+655+Dm5obr16/j/PnzJuv8/PxQVFSEhw8fOqMFM1lZWWjTpg2mTZuGrl274pVXXsGFCxdw8eLFKk97Vl2/UpKRkYELFy7gzTffrPY0blLup02bNvjLX/6C4cOHIzAwENOmTcOMGTOQk5PTIPspKysz+16pKIr49ttvG1Q/SUlJWLBgAZ5//nnjMnv3MakfI6iR/A2usLAQHTp0wDvvvIPDhw8jNjYWCQkJKCoqQrNmzUzGlp/+S6vVmqxzd3cHAMm8x15UVITMzEz07NkTR48exezZszF79mxcuHDB4inNtFotiouLq+xXSlatWoWkpCRoNJoqT9Em9X5OnjyJDRs2YMOGDTh9+jRmzZqFOXPmNNh++vTpg59++gmZmZl49OgRVq9ejZKSEgiC0CD7qcjex0TqxwhqJAEXEBCALVu2oFu3bmjatClGjx6Nli1b4vTp0xZPEebh4WF2+rDy04RV/kVwJj8/P8TFxcHd3R19+/ZFnz59kJmZWeVpz6o6JZqUejpz5gzy8/MxYMAAANWfxk3K/ezbtw+vvvoqevXqBXd3d8THx+OPf/wjjh492iD78fPzQ2pqKlJSUtCnTx/k5OTAz88PPj4+DbKfiuzdxxrCMaKxaxQBl5GRgQMHDpgs0+v1CAwMxPnz543Lbt++jbKyMrRr1w6+vr4m6y5duoS2bdsan6U5W+vWraHXm55RXxAEzJo1y2TeFU97Vrmniv1KRVpaGqKjo6FWqwHAbM4NpZ8mTZqYLVMqlfiv//qvBtnPnTt38Pzzz+PgwYP45ZdfkJqaiuvXryM6OrpB9lORvfuY1I8R1EgCTqfTITU1FVlZWdBqtVi/fj10Oh3mzp2LgwcP4uTJkygqKsIHH3yAmJgYaDQaDBo0CGlpabh06RJu3bqFTz75BIMHD3Z2K0Z9+/ZFYWEhNm3ahJKSEnz//ff45Zdf0Ldv3ypPexYbG1tlv1Jx9OhRhIWFGW9Xdxo3KffTr18/7N27Fz/99BNKS0uxfft2FBUVITY2tkH2c/36dYwZMwbnz59HUVERUlNTERsb22D7qcjefUzqxwhC4/mawNq1a8W+ffuK3bp1E+Pj48XLly+LoiiKu3btEv/85z+LXbt2FZOSksTi4mLjNp999pn44osviiEhIeKcOXNEnU7nrOlbdP78eTEuLk4MDg4WX3vtNfH48eOiKIri0aNHxcjISDEwMFAcPXq0+Pvvvxu3qa5fZysoKBD9/PzEu3fvmixvqP3s3r1bjI6OFoODg8Vhw4aJZ8+eFUWx4fazevVqMSwsTAwODhbffvtt8dGjR6IoNsx+3nzzTeOnKEXR/h6kfoxo7HiqLiIikqVG8RYlERE1Pgw4IiKSJQYcERHJEgOOiIhkiQFHRESyxIAjIiJZYsAREZEsMeCIiEiW/j+eV8/m9FY0PwAAAABJRU5ErkJggg==\n", "text/plain": [ "
" ] }, "metadata": {}, "output_type": "display_data" } ], "source": [ "average_check = lambda x: x['transaction_value'].mean()\n", "\n", "\n", "data.rete.compare(groups=(test, control),\n", " function=average_check,\n", " test='ks_2samp',\n", " group_names=('test','control'))" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "In this case we can see that there is no statistically significant difference in average checks between two groups (P-value is 0.55, selected threshold is 0.05). Note that for continuous variables like average check we used Kolmogorov-Smirnov test. While we can conclude that users in the test group converted to purchase more often than in the control group, there was no effect on the average check." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## More complex metrics" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Just to illustrate that metrics function can be any complex, let’s consider another example. Suppose we have a separate file, which has all transaction_id’s and their statuses (for example, whether a transaction was already confirmed by the bank or not). \n", "\n", "For the demonstration purpose let’s just create such dataframe with randomized data:" ] }, { "cell_type": "code", "execution_count": 11, "metadata": {}, "outputs": [ { "data": { "text/html": [ "
\n", "\n", "\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
transaction_IDconfirmed
07121884True
19641982True
29826287True
39647603True
48125650True
\n", "
" ], "text/plain": [ " transaction_ID confirmed\n", "0 7121884 True\n", "1 9641982 True\n", "2 9826287 True\n", "3 9647603 True\n", "4 8125650 True" ] }, "execution_count": 11, "metadata": {}, "output_type": "execute_result" } ], "source": [ "import pandas as pd\n", "import random\n", "\n", "all_ids = data['transaction_ID'].dropna().unique()\n", "status = pd.DataFrame({'transaction_ID': all_ids, \n", " 'confirmed': [random.random() > 0.2 \n", " for _ in all_ids]})\n", "\n", "status.head()" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Now, let’s write metrics function confirmed_purch, which will return 1 if user has confirmed transactions or 0 if has not:" ] }, { "cell_type": "code", "execution_count": 12, "metadata": {}, "outputs": [], "source": [ "def confirmed_purch(x):\n", " \n", " # get list of transactions for user x\n", " trans_list = x['transaction_ID'].unique()\n", " \n", " # get all status records for transactions from user x\n", " trans_status = status[status['transaction_ID'].isin(trans_list)]\n", " \n", " # True / False if user has conf transactions\n", " has_conf_trans = trans_status['confirmed'].sum() > 0\n", " \n", " # convert bool to int:\n", " return int(has_conf_trans)\n" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "It’s very straightforward. Again, function confirmed_purch() takes a single user trajectory as an argument (as pandas dataframe) and returns a single numerical value. Let’s compare our groups using confirmed_purch metric: " ] }, { "cell_type": "code", "execution_count": 13, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "test (mean ± SD): 0.190 ± 0.392, n = 554\n", "control (mean ± SD): 0.131 ± 0.337, n = 573\n", "'test' is greater than 'control' with P-value: 0.00363\n", "power of the test: 85.25%\n" ] }, { "data": { "image/png": "iVBORw0KGgoAAAANSUhEUgAAAbgAAAFOCAYAAAD0AQi3AAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjMuMSwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy/d3fzzAAAACXBIWXMAAAsTAAALEwEAmpwYAAAnsElEQVR4nO3df1iUdb7/8RcMYoiCQcay5MaqgaEgilpq2QE1rR3bytXjj1zduo5Z4o81jVXMVUqxH/46Vqc8tbVXqam5ttpuhJgn9WgltbaYgJqJZCa60sogBAzz/aPjfGNVmEGYGT4+H9fFdTH3/Zn7fr9Hxhf3cN+f28/hcDgEAIBh/L1dAAAAzYGAAwAYiYADABiJgAMAGMmogHM4HLLb7eK8GQCAUQFXW1ur/fv3q7a21uXn2O12ffrpp7Lb7c1YmeeY1o9ETy2Baf1IZvZ0tTEq4AAAuICAAwAYiYADABiJgAMAGImAAwAYiYADABiJgAMAGImAAwAYiYADABjJowE3efJkxcbGOr9uueUWSVJ2drYGDRqkHj16aNKkSTp9+rTzOWvXrtVtt92mnj17atasWSovL/dkyQCAFsqjAXf8+HHl5OSosLBQhYWF+vjjj1VcXKy0tDSlp6drx44dCgsL0/z58yVJubm5WrlypVatWqXs7GyVlpZq+fLlniwZANBCeSzgHA6HSkpKFBUVVWf51q1bNXDgQKWkpCgsLEzTp0/Xzp07de7cOW3evFkjRoxQz5491aFDBz3yyCN69913PVUyAKAFC/DUjr799lv5+flp7NixKiwsVKdOnTR37lwVFBSoe/fuznGRkZEKCgrSiRMnlJ+fr4ceesi5LiYmRqWlpTp37pxCQkIuuy93Jke9MNaUCVVN60eip5bAtH6kxvdksViaoxw0gscC7uzZs+rcubMef/xxxcTE6O2339bkyZPVtWtXtWvXrs7Y4OBglZeXy2az1VkXHBwsSSovL6834PLy8tyurzHP8WWm9SPRU0tgWj+S+z0lJSU1UyVwl8cCrlu3bnrrrbecjydOnKiNGzfqs88+U0pKSp2xFRUVCgkJUUhIiCorK+ssl1RvuElSfHy8y79F2e125eXlufycv+x7RlU1DZ/oEhgQrF/0edylGpqSu/20BPTk+0zrRzKzp6uNxwLuww8/VFVVlYYMGeJcVlNTo/j4eOXn5zuXlZSUqLq6WtHR0erSpYvy8/N15513SpIOHz6sG2+80XkkdzkWi8XtH0hXn1NZ8099X91wwNWqxqtvisa8Br6Onnyfaf1IZvZ0tfDYSSZVVVVasGCB9u/fL5vNptdff11VVVVauHChcnJylJubq9LSUi1evFhWq1WBgYG69957tWHDBh0+fFinTp3S0qVLdf/993uqZABAC+axI7ghQ4aouLhY06dPl81mU7du3fTKK6+oc+fOysjI0OzZs1VaWqo77rhDaWlpkqRbb71V//Ef/6GJEyeqoqJC99xzT52TTgAAuByPBZwkPfjgg3rwwQcvWm61WmW1Wi/5nIkTJ2rixInNXBkAwDRM1QUAMBIBBwAwEgEHADASAQcAMBIBBwAwEgEHADASAQcAMBIBBwAwEgEHADASAQcAMBIBBwAwEgEHADASAQcAMBIBBwAwEgEHADASAQcAMBIBBwAwEgEHADASAQcAMBIBBwAwEgEHADASAQcAMBIBBwAwUoC3C/AV8944pOpav3rHtL3GosRObfR9dbmHqgIANBYB93/Kv7er2l5/wAEAWg4+ogQAGImAAwAYiYADABiJgAMAGImAAwAYiYADABiJgAMAGImAAwAYiYADABiJgAMAGImAAwAYiYADABiJgAMAGImAAwAYiYADABiJgAMAGImAAwAYiYADABiJgAMAGImAAwAYiYADABiJgAMAGImAAwAYySsBV1JSoltuuUV79uyRJO3bt09Wq1UJCQkaM2aMjh496hybnZ2tQYMGqUePHpo0aZJOnz7tjZIBAC2MVwJu3rx5KisrkyTZbDalpqZqwoQJ2rVrl/r27asZM2ZIkoqLi5WWlqb09HTt2LFDYWFhmj9/vjdKBgC0MB4PuI0bN6pNmzb6yU9+Iknavn27OnbsqJEjRyo0NFRTpkxRUVGRDh8+rK1bt2rgwIFKSUlRWFiYpk+frp07d+rcuXOeLhsA0MIEeHJnX3/9tVavXq3169frV7/6lSSpoKBAcXFxzjGBgYGKjo5WUVGRCgoK1L17d+e6yMhIBQUF6cSJEwoJCbnsfux2u8s1XRjbyuJocGwrf4f8FCB/tWpwrL8C3KqjqVzYpzf23VzoyfeZ1o/U+J4sFktzlING8FjAORwOzZkzR7NmzVJYWJhzeVlZmdq3b19nbHBwsGw2m8rKytSuXbuL1pWXl9e7r7y8PLfrG9uzysWRQ+VCvkmS9u/f73YdTaUxr4GvoyffZ1o/kvs9JSUlNVMlcJfHAu7NN99UeHi4hg4dWmd5SEiIKisr6yyrqKhQaGjoZdfVd/QmSfHx8S7/FmW325WXl6e1fwtUtd2v3rHBrS26tetm2SrONLjd1q3a6L5+C12qoSld6Med18DX0ZPvM60fycyerjYeC7i9e/dq+/bteu+995zLfvOb36hdu3aKjY11LquqqlJRUZHi4uJ04MAB5efnO9eVlJSourpa0dHR9e7LYrG4/QNZbfdrMOCqa/3kUI1qVd3g9mpV49U3RWNeA19HT77PtH4kM3u6WnjsJJMXX3xRhYWFzq+oqCi99tpr2rFjhwoLC5WVlaWysjItXbpUiYmJioiI0PDhw5WTk6Pc3FyVlpZq8eLFslqtCgwM9FTZAIAWyusXerdr106rVq3SihUrNGDAAB06dEiZmZmSpOjoaGVkZGj27NlKTk6Ww+FQWlqalysGALQEHj2L8sc++OAD5/f9+vVTVlbWJcdZrVZZrVZPlQUAMITXj+AAAGgOBBwAwEgEHADASAQcAMBIBBwAwEgEHADASAQcAMBIBBwAwEgEHADASAQcAMBIBBwAwEgEHADASAQcAMBIBBwAwEgEHADASAQcAMBIBBwAwEgEHADASAQcAMBIBBwAwEgEHADASAQcAMBIBBwAwEgEHADASAQcAMBIBBwAwEgEHADASAQcAMBIBBwAwEgEHADASAQcAMBIBBwAwEgEHADASAQcAMBIBBwAwEgEHADASAQcAMBIBBwAwEgEHADASAQcAMBIBBwAwEgEHADASAQcAMBIBBwAwEgEHADASAQcAMBIBBwAwEgEHADASB4NuM2bNyslJUU9evTQiBEjlJubK0nat2+frFarEhISNGbMGB09etT5nOzsbA0aNEg9evTQpEmTdPr0aU+WDABooTwWcEePHlVGRoaeeOIJ7dmzR3fddZemTp0qm82m1NRUTZgwQbt27VLfvn01Y8YMSVJxcbHS0tKUnp6uHTt2KCwsTPPnz/dUyQCAFizAUzvau3evevfureTkZEnS6NGj9eyzzyonJ0cdO3bUyJEjJUlTpkzR66+/rsOHD2vbtm0aOHCgUlJSJEnTp0/X4MGDde7cOYWEhFx2X3a73eW6LoxtZXE0OLaVv0N+CpC/WjU41l8BbtXRVC7s0xv7bi705PtM60dqfE8Wi6U5ykEjeCzgRo8erVGjRsnhcKi8vFzr1q3Tz372Mx08eFBxcXHOcYGBgYqOjlZRUZEKCgrUvXt357rIyEgFBQXpxIkT9QZcXl6e2/WN7Vnl4sihciHfJEn79+93u46m0pjXwNfRk+8zrR/J/Z6SkpKaqRK4y2MBZ7FYZLFYtGfPHv3mN7+RJKWmpurUqVNq3759nbHBwcGy2WwqKytTu3btLlpXXl5e777i4+Nd/i3KbrcrLy9Pa/8WqGq7X71jg1tbdGvXzbJVnGlwu61btdF9/Ra6VENTutCPO6+Br6Mn32daP5KZPV1tPBZwF/Tv3195eXn6/PPPlZqaqvbt2+v222+vM6aiokKhoaEKCQlRZWXlRevqO3qT/n+YuqPa7tdgwFXX+smhGtWqusHt1arGq2+KxrwGvo6efJ9p/Uhm9nS18NhJJqtXr9Zbb70l6YePIfv06aOePXtq9OjRys/Pd46rqqpSUVGR4uLi1KVLlzrrSkpKVF1drejoaE+VDQBooTwWcOHh4Vq9erUOHjyoyspK7dq1S5999pn69eunwsJCZWVlqaysTEuXLlViYqIiIiI0fPhw5eTkKDc3V6WlpVq8eLGsVqsCAwM9VTYAoIXy2EeU9913n4qLi/Xoo4/qu+++U3R0tDIzM9W1a1etWrVKCxcu1OOPP66kpCQtWbJEkhQdHa2MjAzNnj1bpaWluuOOO5SWluapkgEALZjHAs7f318zZsxwXuP2Y/369VNWVtYln2e1WmW1Wpu5OgCAaVz+iPLmm29WcXHxRcuPHDmiXr16NWlRAABcqQaP4O699175+fnJ4XDokUceUatWdS8CO3v27EWn8gMA4G0NBtzgwYMlSfn5+br11lsvumYtMDDQOTsJAAC+osGAS01NdX7/61//usFr0AAA8AUun2SSmpqqo0ePas+ePRddfC398FEmAKBlqa2tlb+/mXdOczngXn75Za1YsUIWi0VBQUF11vn5+RFwAOCj/vnPfyotLU3fffedOnfurOPHj+vYsWPq1auXevbsqdtvv11PPvmkampq1Lp1ay1YsEB+fn6aOXOmNmzYIEkaNWqUli1bps2bN+vQoUP67rvv9N1332natGkaMmSIli5dqk8++UQOh0O//OUvNW7cOC937UbAvfbaa5o8ebJSU1OZtgYAWpDVq1fr9ttv17hx4/Txxx/r+eef1z/+8Q89/vjjioqK0tixYzV37lx1795dH3zwgRYvXqz09PTLbi8gIEBvvPGGTp8+rfvuu0933HGHNm3apL/+9a9q27atXnvtNQ92d3kuH5dWV1dr2LBhhBsAtDCHDh1y3uUgPj5eknTttdcqKipKknT8+HHnnVt69eql48ePX7SN2tpa5/e9e/eWJHXo0EHXXnutbDabnnnmGWVmZmrmzJk+85Gny1Xcfffdev/995uzFgBAM4iMjFRBQYEkac+ePRetv+666/Tll19Kkvbt26e4uDgFBgaqoqJCknT+/Pk610F/8cUXkn6YH7iiokIOh0O7d+/W008/rZUrV2rDhg06ffp0c7fVIJc/ogwNDdWrr76qjz76SHFxcRcdyc2ZM6fJiwMAXLnJkycrLS1Nb731ljp16nTREVZGRobS09PVqlUrBQUF6cknn9T111+vbt26adasWXI4HAoLC3OOP3XqlB544AGdO3dO8+fPV3h4uGpqajRq1CjV1NTojjvuUIcOHTzd5kVcDrjPP/9cCQkJkqTCwsI66/z86r/NDADAe4qLizV79mwlJCTok08+0TvvvKM//vGPzvUJCQnOu7382IV5gf/V4MGDNWbMmDrL5s2b17RFNwGXA+6NN95ozjoAAM3kuuuucx6hXXPNNXrqqae8XZJHuBxw27dvr3f9oEGDrrgYAEDT69y58yWP0Bpj6tSpTbIdT3A54KZMmXLJ5RaLRe3btyfgAAA+xeWAu3AGzgVVVVX68ssvtXLlSud8lQAA+IpGX6wQGBiom2++Wc8++6xefvnlpqwJAIArdsVX41VXV/vE9Q4AAPyYyx9RZmZmXrSssrJSH330kfOqdgBA4y1Ye1i2Snuzbb/tNRYtGHtTs23f17gccAcPHrxo2YV7wU2aNKlJiwKAq5Gt0q7yZgw4d8XGxio7O1s33nij28/9+uuvNWjQIH3xxRcKCHA5apoU18EBAIzk1t/gTp48qWeffVaPPvqoHnvsMf33f/+3zp0711y1AQC8ZODAgZKkO++8U3//+9/1wgsvKDk5WX369NGiRYtUVVUlSTpy5IhGjx6txMREDRs2TNnZ2aqpqXFeOtatWzedPXvWKz24HHAHDhzQ3XffrdzcXHXo0EEBAQF66623NGjQIOXl5TVnjQAAD9u5c6ckKTs7W3/729+0c+dOrVmzRu+8847y8vK0evVqSdLChQuVnJysjz/+WPPmzdOsWbNUUVHhnBzkiy++qDOPpSe5HHDPPPOMxo8fr/Xr12vhwoV6+umntX37dg0dOvSy85UBAFq+9evXa8aMGfrpT3+qqKgoTZ06Ve+9954kyd/fXwcOHNDXX3+tAQMGaMeOHWrTpo2XK/6BW0dwVqv1ouWjR4923joBAGCeEydOaOLEiYqNjVVsbKwefPBBnTp1SpL03HPP6Sc/+YkmT56sAQMGOI/sfIHLJ5m0bdtWx44dU0xMTJ3lp06dUuvWrZu8MACAb7juuuu0fPly5x1lbDabTp48Kbvdrj179igtLU3p6en66quvNGHCBPXp00ddu3b1ctVuBNyIESO0YMECnT9/Xn369FFAQID279+vJUuW6K677mrOGgHgqtD2GkvDgzy4fYvForNnz2r48OH6r//6Ly1atEg1NTVKT0/XjTfeqHnz5mnFihUqLS3VqFGjVFVVpaqqKl177bXOSwPOnj2r66+/vjnaaZDLATd16lRZLBZlZGQ47/JqsVg0YsQI/e53v2u2AgHgauFrF2EPGTJE48eP19tvv63y8nLdfffdstvtGjx4sGbOnCnph/MznnrqKS1fvlzt27fXQw89pKSkJNXW1qpnz55KSUnR7t271b59e4/X32DA1dbW6t1331V0dLRSU1M1adIkHT16VMuWLdPo0aOVkpLiiToBAB62cuVK5/fp6elKT0+/aEyfPn305z//+aLl/v7+TXaLnsaq9yST8vJyjR8/XnPnznXONxkYGKif/exnKi0t1ZQpU/TEE094pFAAANxRb8CtXLlSZ86c0XvvvVfnfm9t2rTRxo0btWbNGn3wwQd1bn0OAIAvqDfgcnJyNGPGDHXs2PGS63v16qVZs2bp7bffbpbiAABorHoDrqSkRDfdVP8fPRMSElRUVNSkRQEAcKXqDbj27durpKSk3g2cOXNGQUFBTVoUAABXqt6AGzhwoF5++WXV1tZedswf//hH9e/fv8kLAwDgStQbcNOnT9eXX36pBx54QB9++KFKS0tVW1ur06dP68MPP9T48eOVm5urGTNmeKhcAABcU+91cBEREVq3bp0WLFighx9+WH5+fs51DodDffr00dq1axt1MzwAAJpTgxd6d+zYUa+++qpOnjypgwcP6vz582rXrp1uvvlmRUREeKJGALgq/PmjJ/V9TXmzbb91QLB+eav3rl0eP3687rnnHo0cOdKt5zX27uAuj4yMjFRkZKRbRQEAXPd9Tbm+r26+gPNVNTU1bgWXq9y6ozcA4Oqyd+9e3XPPPUpMTNS4ceN07NgxnTx5Ug8//LCSkpI0dOhQbdq0yTk+NjZW69atU0pKinr06KE5c+bI4XBo1qxZ+uSTTzRv3jy9+uqr+tOf/qTx48dr1qxZ+sUvfiFJys3N1f3336+ePXtq1KhR2r9//xXVTsABAC7p5MmTmjp1qubMmaO9e/cqISFBv//97/XII48oJiZGO3bsUGZmppYuXardu3c7n7dt2zZt3LhR69ev11//+ld9+umneu6559S3b1899dRTeuihhyRJ+/btU9++fbV161adOnVKDz/8sCZNmqRdu3Zp9OjRevjhh3XmzJlG10/AAQAu6d1331X//v3Vr18/BQUF6dFHH1WfPn108uRJTZ8+XSEhIerVq5dGjx6tv/zlL87nTZ06VeHh4eratas6d+6sEydOXHL7nTp10qhRoxQYGKgtW7aob9++GjZsmNq2bav7779fN910U53gdBcBBwC4pBMnTuiGG25wPm7Xrp06deqkG264oc7fzMLDw+scaYWFhTm/b926tex2+yW3HxoaWmdf0dHRddaHhYU5J/pvDAIOAHBJ/xowp06d0po1a3Tq1Ck5HA7n8q+++qpOEDZGeHi4vv322zrLjh07dkXbJeAAwEe0DghW61bN+BUQ7FY9w4YN0wcffKB9+/aprKxMzzzzjCIiIhQYGKiXXnpJNptNubm52rJli+6///4Gt3fhDuGXctddd+l//ud/9OGHH8pms2nDhg06c+aM/u3f/s2tmn+s6c/LBAA0ijevUbuUmJgYZWRkaO7cufrHP/6h/v3768knn9SZM2f0+9//Xi+99JLCw8M1e/ZsxcfHN7i95ORkPffcc2rTpo2Cg+uGbZcuXfT0009ryZIl+vrrr3XTTTfp+eefv6K5jgk4AMBl/eIXv3Cexn/Btddeq7Vr115yfGFhYZ3H69atc34/YcIETZgwwfn4X4/67rzzTt15550XbfOGG264aLuu4CNKAICRCDgAgJE8GnA7d+6U1WpVQkKChg0bpq1bt0r64WK/C8vHjBmjo0ePOp+TnZ2tQYMGqUePHpo0adIVnTIKALh6eCzgSktLNX36dI0fP1579uzRY489pjlz5qigoECpqamaMGGCdu3apb59+zpvv1NcXKy0tDSlp6drx44dCgsL0/z58z1VMgCgBfPYSSa5ubmKiorSv//7v0uShgwZopiYGOXk5Khjx47O2aWnTJmi119/XYcPH9a2bds0cOBApaSkSPrh/nSDBw/WuXPnFBISctl9Xe6iwvrGtrI4GhgptfJ3yE8B8lerBsf6K8CtOprKhX16Y9/NhZ58n2n9SI3vyWKxNEc5aASPBVxSUpJWrFjhfHz27FkVFxfrnXfeqXNH8MDAQEVHR6uoqEgFBQXq3r27c11kZKSCgoJ04sSJegMuLy/P7frG9qxyceRQuZBvknTFE4Veica8Br6Onnyfaf1I7veUlJTUTJXAXR4LuLCwMOf0Lbm5uZo3b54SEhIUGRl5UVgFBwfLZrOprKxM7dq1u2hdeXn9t5OIj493+bcou92uvLw8rf1boKrtfvWODW5t0a1dN8tW0fDkn61btdF9/Ra6VENTutCPO6+Br6Mn32daP5KZPV1tPHodnM1mU0ZGhnJycjRlyhRNmDBBy5YtU2VlZZ1xFRUVCg0NVUhIyCXX1Xf0Jv3wEYG7P5DVdr8GA6661k8O1ahW1Q1ur1Y1Xn1TNOY18HX05PtM60cys6erhccCrrKyUmPHjlVERISysrJ0/fXXS/rh6vUf30uoqqpKRUVFiouL04EDB5Sfn+9cV1JSourq6osm5AQA4F957CzKrVu3qqqqSi+88IIz3KQfTjYpLCxUVlaWysrKtHTpUiUmJioiIkLDhw9XTk6OcnNzVVpaqsWLF8tqtSowMNBTZQMAWiiPBVx+fr6++uorxcfHKzY21vm1bds2rVq1SitWrNCAAQN06NAhZWZmSpKio6OVkZGh2bNnKzk5WQ6HQ2lpaZ4qGQDQgnnsI8r58+fXew1bVlbWJZdbrVZZrdbmKgsAYCim6gIAGImAAwAYiYADABiJgAMAGImAAwAYiYADABiJgAMAGImAAwAYiYADABiJgAMAGImAAwAYiYADABiJgAMAGImAAwAYiYADABiJgAMAGImAAwAYiYADABiJgAMAGImAAwAYiYADABiJgAMAGImAAwAYiYADABiJgAMAGImAAwAYiYADABiJgAMAGImAAwAYiYADABiJgAMAGImAAwAYiYADABiJgAMAGImAAwAYiYADABiJgAMAGImAAwAYiYADABiJgAMAGImAAwAYiYADABiJgAMAGImAAwAYiYADABiJgAMAGImAAwAYiYADABgpwNM7TEtLU+/evTVy5EhJ0r59+7Rw4UIdP35c3bp106JFi9SpUydJUnZ2tp5++mmdOXNGt9xyixYtWqQOHTp4umQAPm7B2sOyVdobHNf2GosWjL3JAxXBF3jsCG737t3KzMzUli1bnMtsNptSU1M1YcIE7dq1S3379tWMGTMkScXFxUpLS1N6erp27NihsLAwzZ8/31PlAmhBbJV2lbvw5UoIwhweO4L77LPPZLPZFB4e7ly2fft2dezY0Xk0N2XKFL3++us6fPiwtm3bpoEDByolJUWSNH36dA0ePFjnzp1TSEhIvfuy213/Ib4wtpXF0eDYVv4O+SlA/mrV4Fh/BbhVR1O5sE9v7Lu50JPv83Y/rfwdLr+HXa2xsT1ZLBa3xqP5eCzgpk2bJkkaP368c1lBQYHi4uKcjwMDAxUdHa2ioiIVFBSoe/fuznWRkZEKCgrSiRMnGgy4vLw8t+sb27PKxZFD5UK+SZL279/vdh1NpTGvga+jJ9/nrX5+Fe/6WHffl+72lJSU5NZ4NB+P/w3ux8rKytS+ffs6y4KDg2Wz2VRWVqZ27dpdtK68vLzB7cbHx7v8W5TdbldeXp7W/i1Q1Xa/escGt7bo1q6bZas40+B2W7dqo/v6LXSphqZ0oR93XgNfR0++z9v9zHvjkMq/b/hIK7i1RU+Nj3Fpm97uCVfOqwEXEhKiysrKOssqKioUGhp62XUNHb1JP3xE4O4PZLXdr8GAq671k0M1qlV1g9urVY1X3xSNeQ18HT35Pm/1U13b8Pv3wjh36zPt3+hq4tXLBLp06aL8/Hzn46qqKhUVFSkuLu6idSUlJaqurlZ0dLQXKgUAtDReDbghQ4aosLBQWVlZKisr09KlS5WYmKiIiAgNHz5cOTk5ys3NVWlpqRYvXiyr1arAwEBvlgwAaCG8GnDt2rXTqlWrtGLFCg0YMECHDh1SZmamJCk6OloZGRmaPXu2kpOT5XA4lJaW5s1yAQAtiMf/BvfGG2/UedyvXz9lZWVdcqzVapXVavVEWQAAwzBVFwDASAQcAMBIBBwAwEgEHADASAQcAMBIBBwAwEgEHADASAQcAMBIXp1sGQA87b3c51RW2fAdQa4JCFVUqzs9UBGaCwEH4Kryfc15fV/d8G23/BXg8r0f4Zv4iBIAYCQCDgBgJAIOAGAkAg4AYCQCDgBgJAIOAGAkAg4AYCQCDgBgJAIOAGAkAg4AYCQCDgBgJAIOAGAkAg4AYCQCDgBgJAIOAGAkAg4AYCQCDgBgJAIOAGAkAg4AYCQCDgBgJAIOAGAkAg4AYCQCDgBgJAIOAGAkAg4AYCQCDgBgJAIOAGAkAg4AYCQCDgBgJAIOAGAkAg4AYCQCDgBgJAIOAGAkAg4AYCQCDgBgJAIOAGAkAg4AYCSfD7h9+/bJarUqISFBY8aM0dGjR71dEgCgBfDpgLPZbEpNTdWECRO0a9cu9e3bVzNmzPB2WQCAFsCnA2779u3q2LGjRo4cqdDQUE2ZMkVFRUU6fPiwt0sDAPi4AG8XUJ+CggLFxcU5HwcGBio6OlpFRUW66aabLhrvcDgkSVVVVbJYLC7tw263S5LCg/1UU+tX79jga/wUFNBe9la1DW43MCBYVVVVLtXQlC70485r4Ovoyfd5u5/r2vopKKD+96/k3nu4dUCIJPd7slgs8vf3l59fw/Wgefl0wJWVlal9+/Z1lgUHB8tms11yfG3tDz+0Bw8edHtf1psrXRw5QNcFuTYyLy/P7TqaSmNeA19HT77PW/0M6ezOaNffw1LjekpMTDTiF5eWzqcDLiQkRJWVdYOnoqJCoaGhlxwfEBCg+Ph4fnsC4FX+/j7915+rhk8HXJcuXbRp0ybn46qqKhUVFdX52PLH/P39FRgY6KnyAAA+zKd/zRgyZIgKCwuVlZWlsrIyLV26VImJiYqIiPB2aQAAH+fnuHBmho/au3evFi5cqG+++UZJSUlasmQJAQcAaJDPBxwAAI3h0x9RAgDQWFdFwLky3ZfNZtO0adPUs2dPJScna+PGjV6o1HWu9HT27FmlpqaqZ8+e6t+/v5544gmdP3/eC9W6xt1p2WbOnKlZs2Z5qDr3udrPCy+8oP79+6tPnz6aPn26ysrKPFyp61zp6ejRo/r1r3/t/LlbtGiRampqvFCt69LS0i77nq+pqdH8+fOVlJSkAQMG6MUXX/RwdWgs4wPO1em+MjMzVVVVpW3btunZZ5/VkiVLdODAAc8X7AJXe3ryySdVXV2trKwsrV+/Xp9//rlWrVrl+YJd4O60bO+9956ysrI8V6CbXO1n06ZNys7O1po1a/T+++/r7NmzeuWVVzxfsAtc7emJJ55QfHy89u7dq/Xr12vnzp3605/+5PmCXbB7925lZmZqy5Ytlx2zevVqffHFF9q6datee+01rVu3Ttu2bfNglWgs4wPOlem+qqqq9O6772rWrFm67rrr1Lt3bw0dOlR/+ctfvFj55bk6hdnOnTuVmpqqiIgI5/jdu3d7qer6uTMt25kzZ7R8+XKNGDHCC5W6xtV+1q5dq8cff1w///nPFRYWpmXLlumee+7xUtX1c7Uni8UiPz8//fjP+/86YYOv+Oyzz2Sz2RQeHn7ZMZs3b9a0adP005/+VDExMRo9erS2bt3qwSrRWMYHXH3TfV1w7Ngx1dbWqkuXLs5lMTExdcb4Eld6kqQ//OEPio2NdT7+/PPPFRkZ6bE63eFqT5I0b948Pfroo7r++us9WaJbXOmnqqpK+fn5ysvL02233aY+ffpo+fLlLf7faO7cuVqzZo0SExM1ePBgRUVFaciQIZ4u1yXTpk3TokWL9POf//yS68vLy3X8+PE6fcfGxvrs/w2oy/iAKysrU0hISJ1l/zrd1+XGlJeXe6RGd7nSkyT16NFDgYGBOnv2rH73u99p586dmjlzpidLdZmrPV248P/ee+/1VGmN4ko///znP2W32/Xpp59q8+bN2rJliw4fPqzly5d7ulyXuNKT3W7Xb3/7W40bN06ffvqpNm3apKNHj+rNN9/0dLlN4kJvP+7bl/9vQF3GB5wr032FhoaqoqKi3jG+xJ0pzDZt2qRhw4bJZrPpz3/+s7p27eqpMt3iSk/ffvutXnjhBS1cuNDT5bnNnX+j6dOnq0OHDoqMjNRDDz2kjz76yFNlusWVngoLC/XNN9/ot7/9rdq2bavu3btr3Lhx2rVrl6fLbRIXgu3H/z+cP3/+oqCHbzI+4Lp06aL8/Hzn40tN9xUVFSW73a7i4mLnsiNHjujmm2/2aK2ucqUnSXrppZe0dOlSLVu2TM8//7zPfvQludbTgQMHdOLECQ0cOFCxsbF6/vnntXXrVvXu3dsbJdfLlX7Cw8PVpk0b50z80g8ThgcFuTETsAe50lPr1q0v+vtbQECA2rRp49Fam0pQUJCioqLq9H3kyJHLThcI32J8wLky3VdQUJCGDRumZcuWqaysTP/7v/+r999/X1ar1YuVX54rPZ0/f14vvviiVqxYodtuu82L1brGlZ4GDx6swsJC51dqaqqGDx+u3NxcL1Z+aa704+/vL6vVqpUrV6qkpETffPONXnnllRb9cxcdHa2oqCitWLFCNptNR44c0bp163TXXXd5sfIrc++99+rFF1/U2bNnlZeXpzfffFP33Xeft8uCKxxXgT179jiGDh3qiI+Pd0ycONHx7bffOhwOhyMmJsbx0UcfORwOh6O0tNTxyCOPOOLj4x2DBg1yvP/++94suUEN9fT3v//dERMTc9HXAw884OXKL8+Vf6cf+8///E/HY4895ukyXeZKP+fPn3dkZGQ4br31Vsftt9/uWL58uaOmpsabZdfLlZ6KioockyZNcvTq1cuRnJzs+MMf/uDNkl3ywAMPODZs2OB8nJyc7Ni0aZPD4XA4vv/+e8ecOXMciYmJjgEDBjjWrl3rrTLhJqbqAgAYyfiPKAEAVycCDgBgJAIOAGAkAg4AYCQCDgBgJAIOAGAkAg4AYCQCDgBgpP8HmC6XcfTkWgsAAAAASUVORK5CYII=\n", "text/plain": [ "
" ] }, "metadata": {}, "output_type": "display_data" } ], "source": [ "data.rete.compare(groups=(test, control),\n", " function=confirmed_purch,\n", " test='mannwhitneyu',\n", " group_names=('test','control'))" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "As we can see, a statistically significant difference in the conversion to confirmed purchases still holds with selected significance level 0.05. Note that because overall there are less confirmed transaction than total number of transactions, power for the test above is less than previously obtained value for all transactions (85% vs. 96%) " ] } ], "metadata": { "kernelspec": { "display_name": "Python 3", "language": "python", "name": "python3" }, "language_info": { "codemirror_mode": { "name": "ipython", "version": 3 }, "file_extension": ".py", "mimetype": "text/x-python", "name": "python", "nbconvert_exporter": "python", "pygments_lexer": "ipython3", "version": "3.8.5" }, "toc": { "base_numbering": 1, "nav_menu": {}, "number_sections": true, "sideBar": true, "skip_h1_title": false, "title_cell": "Table of Contents", "title_sidebar": "Contents", "toc_cell": false, "toc_position": {}, "toc_section_display": true, "toc_window_display": true } }, "nbformat": 4, "nbformat_minor": 4 }