personal-portfolio/notebooks/toronto/overview/income_safety_scatter.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Income vs Safety Scatter Plot\n",
    "\n",
    "Explores the correlation between median household income and safety score across Toronto neighbourhoods."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 1. Data Reference\n",
    "\n",
    "### Source Tables\n",
    "\n",
    "| Table | Grain | Key Columns |\n",
    "|-------|-------|-------------|\n",
    "| `mart_neighbourhood_overview` | neighbourhood × year | neighbourhood_name, median_household_income, safety_score, population |\n",
    "\n",
    "### SQL Query"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import os\n",
    "\n",
    "import pandas as pd\n",
    "from dotenv import load_dotenv\n",
    "from sqlalchemy import create_engine\n",
    "\n",
    "# Load .env from project root\n",
    "load_dotenv(\"../../.env\")\n",
    "\n",
    "engine = create_engine(os.environ[\"DATABASE_URL\"])\n",
    "\n",
    "query = \"\"\"\n",
    "SELECT\n",
    "    neighbourhood_name,\n",
    "    median_household_income,\n",
    "    safety_score,\n",
    "    population,\n",
    "    livability_score,\n",
    "    crime_rate_per_100k\n",
    "FROM public_marts.mart_neighbourhood_overview\n",
    "WHERE year = (SELECT MAX(year) FROM public_marts.mart_neighbourhood_overview)\n",
    "  AND median_household_income IS NOT NULL\n",
    "  AND safety_score IS NOT NULL\n",
    "ORDER BY median_household_income DESC\n",
    "\"\"\"\n",
    "\n",
    "df = pd.read_sql(query, engine)\n",
    "print(f\"Loaded {len(df)} neighbourhoods with income and safety data\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Transformation Steps\n",
    "\n",
    "1. Filter out null values for income and safety\n",
    "2. Optionally scale income to thousands for readability\n",
    "3. Pass to scatter figure factory with optional trendline"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Scale income to thousands for better axis readability\n",
    "df[\"income_thousands\"] = df[\"median_household_income\"] / 1000\n",
    "\n",
    "# Prepare data for figure factory\n",
    "data = df.to_dict(\"records\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Sample Output"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "df[\n",
    "    [\n",
    "        \"neighbourhood_name\",\n",
    "        \"median_household_income\",\n",
    "        \"safety_score\",\n",
    "        \"crime_rate_per_100k\",\n",
    "    ]\n",
    "].head(10)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 2. Data Visualization\n",
    "\n",
    "### Figure Factory\n",
    "\n",
    "Uses `create_scatter_figure` from `portfolio_app.figures.toronto.scatter`.\n",
    "\n",
    "**Key Parameters:**\n",
    "- `x_column`: 'income_thousands' (median household income in $K)\n",
    "- `y_column`: 'safety_score' (0-100 percentile rank)\n",
    "- `name_column`: 'neighbourhood_name' (hover label)\n",
    "- `size_column`: 'population' (optional, bubble size)\n",
    "- `trendline`: True (adds OLS regression line)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import sys\n",
    "\n",
    "sys.path.insert(0, \"../..\")\n",
    "\n",
    "from portfolio_app.figures.toronto.scatter import create_scatter_figure\n",
    "\n",
    "fig = create_scatter_figure(\n",
    "    data=data,\n",
    "    x_column=\"income_thousands\",\n",
    "    y_column=\"safety_score\",\n",
    "    name_column=\"neighbourhood_name\",\n",
    "    size_column=\"population\",\n",
    "    title=\"Income vs Safety by Neighbourhood\",\n",
    "    x_title=\"Median Household Income ($K)\",\n",
    "    y_title=\"Safety Score (0-100)\",\n",
    "    trendline=True,\n",
    ")\n",
    "\n",
    "fig.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Interpretation\n",
    "\n",
    "This scatter plot reveals the relationship between income and safety:\n",
    "\n",
    "- **Positive correlation**: Higher income neighbourhoods tend to have higher safety scores\n",
    "- **Bubble size**: Represents population (larger = more people)\n",
    "- **Trendline**: Orange dashed line shows the overall trend\n",
    "- **Outliers**: Neighbourhoods far from the trendline are interesting cases\n",
    "  - Above line: Safer than income would predict\n",
    "  - Below line: Less safe than income would predict"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Calculate correlation coefficient\n",
    "correlation = df[\"median_household_income\"].corr(df[\"safety_score\"])\n",
    "print(f\"Correlation coefficient (Income vs Safety): {correlation:.3f}\")"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "name": "python",
   "version": "3.11.0"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}