refactor: multi-dashboard structural migration
Some checks failed
CI / lint-and-test (pull_request) Has been cancelled
Some checks failed
CI / lint-and-test (pull_request) Has been cancelled
- Rename dbt project from toronto_housing to portfolio - Restructure dbt models into domain subdirectories: - shared/ for cross-domain dimensions (dim_time) - staging/toronto/, intermediate/toronto/, marts/toronto/ - Update SQLAlchemy models for raw_toronto schema - Add explicit cross-schema FK relationships for FactRentals - Namespace figure factories under figures/toronto/ - Namespace notebooks under notebooks/toronto/ - Update Makefile with domain-specific targets and env loading - Update all documentation for multi-dashboard structure This enables adding new dashboard projects (e.g., /football, /energy) without structural conflicts or naming collisions. Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
This commit is contained in:
0
notebooks/toronto/amenities/.gitkeep
Normal file
0
notebooks/toronto/amenities/.gitkeep
Normal file
182
notebooks/toronto/amenities/amenity_index_choropleth.ipynb
Normal file
182
notebooks/toronto/amenities/amenity_index_choropleth.ipynb
Normal file
@@ -0,0 +1,182 @@
|
||||
{
|
||||
"cells": [
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"# Amenity Index Choropleth Map\n",
|
||||
"\n",
|
||||
"Displays total amenities per 1,000 residents across Toronto's 158 neighbourhoods."
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"## 1. Data Reference\n",
|
||||
"\n",
|
||||
"### Source Tables\n",
|
||||
"\n",
|
||||
"| Table | Grain | Key Columns |\n",
|
||||
"|-------|-------|-------------|\n",
|
||||
"| `mart_neighbourhood_amenities` | neighbourhood × year | amenity_index, total_amenities_per_1000, amenity_tier, geometry |\n",
|
||||
"\n",
|
||||
"### SQL Query"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"import os\n",
|
||||
"\n",
|
||||
"import pandas as pd\n",
|
||||
"from dotenv import load_dotenv\n",
|
||||
"from sqlalchemy import create_engine\n",
|
||||
"\n",
|
||||
"# Load .env from project root\n",
|
||||
"load_dotenv(\"../../.env\")\n",
|
||||
"\n",
|
||||
"engine = create_engine(os.environ[\"DATABASE_URL\"])\n",
|
||||
"\n",
|
||||
"query = \"\"\"\n",
|
||||
"SELECT\n",
|
||||
" neighbourhood_id,\n",
|
||||
" neighbourhood_name,\n",
|
||||
" geometry,\n",
|
||||
" year,\n",
|
||||
" total_amenities_per_1000,\n",
|
||||
" amenity_index,\n",
|
||||
" amenity_tier,\n",
|
||||
" parks_per_1000,\n",
|
||||
" schools_per_1000,\n",
|
||||
" transit_per_1000,\n",
|
||||
" total_amenities,\n",
|
||||
" population\n",
|
||||
"FROM public_marts.mart_neighbourhood_amenities\n",
|
||||
"WHERE year = (SELECT MAX(year) FROM public_marts.mart_neighbourhood_amenities)\n",
|
||||
"ORDER BY total_amenities_per_1000 DESC\n",
|
||||
"\"\"\"\n",
|
||||
"\n",
|
||||
"df = pd.read_sql(query, engine)\n",
|
||||
"print(f\"Loaded {len(df)} neighbourhoods\")"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"### Transformation Steps\n",
|
||||
"\n",
|
||||
"1. Filter to most recent year\n",
|
||||
"2. Convert geometry to GeoJSON"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"import json\n",
|
||||
"\n",
|
||||
"import geopandas as gpd\n",
|
||||
"\n",
|
||||
"gdf = gpd.GeoDataFrame(\n",
|
||||
" df, geometry=gpd.GeoSeries.from_wkb(df[\"geometry\"]), crs=\"EPSG:4326\"\n",
|
||||
")\n",
|
||||
"\n",
|
||||
"geojson = json.loads(gdf.to_json())\n",
|
||||
"data = df.drop(columns=[\"geometry\"]).to_dict(\"records\")"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"### Sample Output"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"df[\n",
|
||||
" [\"neighbourhood_name\", \"total_amenities_per_1000\", \"amenity_index\", \"amenity_tier\"]\n",
|
||||
"].head(10)"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"## 2. Data Visualization\n",
|
||||
"\n",
|
||||
"### Figure Factory\n",
|
||||
"\n",
|
||||
"Uses `create_choropleth_figure` from `portfolio_app.figures.toronto.choropleth`."
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"import sys\n",
|
||||
"\n",
|
||||
"sys.path.insert(0, \"../..\")\n",
|
||||
"\n",
|
||||
"from portfolio_app.figures.toronto.choropleth import create_choropleth_figure\n",
|
||||
"\n",
|
||||
"fig = create_choropleth_figure(\n",
|
||||
" geojson=geojson,\n",
|
||||
" data=data,\n",
|
||||
" location_key=\"neighbourhood_id\",\n",
|
||||
" color_column=\"total_amenities_per_1000\",\n",
|
||||
" hover_data=[\n",
|
||||
" \"neighbourhood_name\",\n",
|
||||
" \"amenity_index\",\n",
|
||||
" \"parks_per_1000\",\n",
|
||||
" \"schools_per_1000\",\n",
|
||||
" ],\n",
|
||||
" color_scale=\"Greens\",\n",
|
||||
" title=\"Toronto Amenities per 1,000 Population\",\n",
|
||||
" zoom=10,\n",
|
||||
")\n",
|
||||
"\n",
|
||||
"fig.show()"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"### Amenity Tier Interpretation\n",
|
||||
"\n",
|
||||
"| Tier | Meaning |\n",
|
||||
"|------|--------|\n",
|
||||
"| 1 | Best served (top 20%) |\n",
|
||||
"| 2-4 | Middle tiers |\n",
|
||||
"| 5 | Underserved (bottom 20%) |"
|
||||
]
|
||||
}
|
||||
],
|
||||
"metadata": {
|
||||
"kernelspec": {
|
||||
"display_name": "Python 3",
|
||||
"language": "python",
|
||||
"name": "python3"
|
||||
},
|
||||
"language_info": {
|
||||
"name": "python",
|
||||
"version": "3.11.0"
|
||||
}
|
||||
},
|
||||
"nbformat": 4,
|
||||
"nbformat_minor": 4
|
||||
}
|
||||
191
notebooks/toronto/amenities/amenity_radar.ipynb
Normal file
191
notebooks/toronto/amenities/amenity_radar.ipynb
Normal file
@@ -0,0 +1,191 @@
|
||||
{
|
||||
"cells": [
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"# Amenity Radar Chart\n",
|
||||
"\n",
|
||||
"Spider/radar chart comparing amenity categories for selected neighbourhoods."
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"## 1. Data Reference\n",
|
||||
"\n",
|
||||
"### Source Tables\n",
|
||||
"\n",
|
||||
"| Table | Grain | Key Columns |\n",
|
||||
"|-------|-------|-------------|\n",
|
||||
"| `mart_neighbourhood_amenities` | neighbourhood × year | parks_index, schools_index, transit_index |\n",
|
||||
"\n",
|
||||
"### SQL Query"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"import os\n",
|
||||
"\n",
|
||||
"import pandas as pd\n",
|
||||
"from dotenv import load_dotenv\n",
|
||||
"from sqlalchemy import create_engine\n",
|
||||
"\n",
|
||||
"# Load .env from project root\n",
|
||||
"load_dotenv(\"../../.env\")\n",
|
||||
"\n",
|
||||
"engine = create_engine(os.environ[\"DATABASE_URL\"])\n",
|
||||
"\n",
|
||||
"query = \"\"\"\n",
|
||||
"SELECT\n",
|
||||
" neighbourhood_name,\n",
|
||||
" parks_index,\n",
|
||||
" schools_index,\n",
|
||||
" transit_index,\n",
|
||||
" amenity_index,\n",
|
||||
" amenity_tier\n",
|
||||
"FROM public_marts.mart_neighbourhood_amenities\n",
|
||||
"WHERE year = (SELECT MAX(year) FROM public_marts.mart_neighbourhood_amenities)\n",
|
||||
"ORDER BY amenity_index DESC\n",
|
||||
"\"\"\"\n",
|
||||
"\n",
|
||||
"df = pd.read_sql(query, engine)\n",
|
||||
"print(f\"Loaded {len(df)} neighbourhoods\")"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"### Transformation Steps\n",
|
||||
"\n",
|
||||
"1. Select top 5 and bottom 5 neighbourhoods by amenity index\n",
|
||||
"2. Reshape for radar chart format"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"# Select representative neighbourhoods\n",
|
||||
"top_5 = df.head(5)\n",
|
||||
"bottom_5 = df.tail(5)\n",
|
||||
"\n",
|
||||
"# Prepare radar data\n",
|
||||
"categories = [\"Parks\", \"Schools\", \"Transit\"]\n",
|
||||
"index_columns = [\"parks_index\", \"schools_index\", \"transit_index\"]"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"### Sample Output"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"print(\"Top 5 Amenity-Rich Neighbourhoods:\")\n",
|
||||
"display(\n",
|
||||
" top_5[\n",
|
||||
" [\n",
|
||||
" \"neighbourhood_name\",\n",
|
||||
" \"parks_index\",\n",
|
||||
" \"schools_index\",\n",
|
||||
" \"transit_index\",\n",
|
||||
" \"amenity_index\",\n",
|
||||
" ]\n",
|
||||
" ]\n",
|
||||
")\n",
|
||||
"print(\"\\nBottom 5 Underserved Neighbourhoods:\")\n",
|
||||
"display(\n",
|
||||
" bottom_5[\n",
|
||||
" [\n",
|
||||
" \"neighbourhood_name\",\n",
|
||||
" \"parks_index\",\n",
|
||||
" \"schools_index\",\n",
|
||||
" \"transit_index\",\n",
|
||||
" \"amenity_index\",\n",
|
||||
" ]\n",
|
||||
" ]\n",
|
||||
")"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"## 2. Data Visualization\n",
|
||||
"\n",
|
||||
"### Figure Factory\n",
|
||||
"\n",
|
||||
"Uses `create_radar` from `portfolio_app.figures.toronto.radar`."
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"import sys\n",
|
||||
"\n",
|
||||
"sys.path.insert(0, \"../..\")\n",
|
||||
"\n",
|
||||
"from portfolio_app.figures.toronto.radar import create_comparison_radar\n",
|
||||
"\n",
|
||||
"# Compare top neighbourhood vs city average (100)\n",
|
||||
"top_hood = top_5.iloc[0]\n",
|
||||
"metrics = [\"parks_index\", \"schools_index\", \"transit_index\"]\n",
|
||||
"\n",
|
||||
"fig = create_comparison_radar(\n",
|
||||
" selected_data=top_hood.to_dict(),\n",
|
||||
" average_data={\"parks_index\": 100, \"schools_index\": 100, \"transit_index\": 100},\n",
|
||||
" metrics=metrics,\n",
|
||||
" selected_name=top_hood[\"neighbourhood_name\"],\n",
|
||||
" average_name=\"City Average\",\n",
|
||||
" title=f\"Amenity Profile: {top_hood['neighbourhood_name']} vs City Average\",\n",
|
||||
")\n",
|
||||
"\n",
|
||||
"fig.show()"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"### Index Interpretation\n",
|
||||
"\n",
|
||||
"| Value | Meaning |\n",
|
||||
"|-------|--------|\n",
|
||||
"| < 100 | Below city average |\n",
|
||||
"| = 100 | City average |\n",
|
||||
"| > 100 | Above city average |"
|
||||
]
|
||||
}
|
||||
],
|
||||
"metadata": {
|
||||
"kernelspec": {
|
||||
"display_name": "Python 3",
|
||||
"language": "python",
|
||||
"name": "python3"
|
||||
},
|
||||
"language_info": {
|
||||
"name": "python",
|
||||
"version": "3.11.0"
|
||||
}
|
||||
},
|
||||
"nbformat": 4,
|
||||
"nbformat_minor": 4
|
||||
}
|
||||
169
notebooks/toronto/amenities/transit_accessibility_bar.ipynb
Normal file
169
notebooks/toronto/amenities/transit_accessibility_bar.ipynb
Normal file
@@ -0,0 +1,169 @@
|
||||
{
|
||||
"cells": [
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"# Transit Accessibility Bar Chart\n",
|
||||
"\n",
|
||||
"Shows transit stops per 1,000 residents across Toronto neighbourhoods."
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"## 1. Data Reference\n",
|
||||
"\n",
|
||||
"### Source Tables\n",
|
||||
"\n",
|
||||
"| Table | Grain | Key Columns |\n",
|
||||
"|-------|-------|-------------|\n",
|
||||
"| `mart_neighbourhood_amenities` | neighbourhood × year | transit_per_1000, transit_index, transit_count |\n",
|
||||
"\n",
|
||||
"### SQL Query"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"import os\n",
|
||||
"\n",
|
||||
"import pandas as pd\n",
|
||||
"from dotenv import load_dotenv\n",
|
||||
"from sqlalchemy import create_engine\n",
|
||||
"\n",
|
||||
"# Load .env from project root\n",
|
||||
"load_dotenv(\"../../.env\")\n",
|
||||
"\n",
|
||||
"engine = create_engine(os.environ[\"DATABASE_URL\"])\n",
|
||||
"\n",
|
||||
"query = \"\"\"\n",
|
||||
"SELECT\n",
|
||||
" neighbourhood_name,\n",
|
||||
" transit_per_1000,\n",
|
||||
" transit_index,\n",
|
||||
" transit_count,\n",
|
||||
" population,\n",
|
||||
" amenity_tier\n",
|
||||
"FROM public_marts.mart_neighbourhood_amenities\n",
|
||||
"WHERE year = (SELECT MAX(year) FROM public_marts.mart_neighbourhood_amenities)\n",
|
||||
" AND transit_per_1000 IS NOT NULL\n",
|
||||
"ORDER BY transit_per_1000 DESC\n",
|
||||
"\"\"\"\n",
|
||||
"\n",
|
||||
"df = pd.read_sql(query, engine)\n",
|
||||
"print(f\"Loaded {len(df)} neighbourhoods\")"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"### Transformation Steps\n",
|
||||
"\n",
|
||||
"1. Sort by transit accessibility\n",
|
||||
"2. Select top 20 for visualization"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"data = df.head(20).to_dict(\"records\")"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"### Sample Output"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"df[[\"neighbourhood_name\", \"transit_per_1000\", \"transit_index\", \"transit_count\"]].head(\n",
|
||||
" 10\n",
|
||||
")"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"## 2. Data Visualization\n",
|
||||
"\n",
|
||||
"### Figure Factory\n",
|
||||
"\n",
|
||||
"Uses `create_horizontal_bar` from `portfolio_app.figures.toronto.bar_charts`."
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"import sys\n",
|
||||
"\n",
|
||||
"sys.path.insert(0, \"../..\")\n",
|
||||
"\n",
|
||||
"from portfolio_app.figures.toronto.bar_charts import create_horizontal_bar\n",
|
||||
"\n",
|
||||
"fig = create_horizontal_bar(\n",
|
||||
" data=data,\n",
|
||||
" name_column=\"neighbourhood_name\",\n",
|
||||
" value_column=\"transit_per_1000\",\n",
|
||||
" title=\"Top 20 Neighbourhoods by Transit Accessibility\",\n",
|
||||
" color=\"#00BCD4\",\n",
|
||||
" value_format=\".2f\",\n",
|
||||
")\n",
|
||||
"\n",
|
||||
"fig.show()"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"### Transit Statistics"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"print(\"City-wide Transit Statistics:\")\n",
|
||||
"print(f\" Total Transit Stops: {df['transit_count'].sum():,.0f}\")\n",
|
||||
"print(f\" Average per 1,000 pop: {df['transit_per_1000'].mean():.2f}\")\n",
|
||||
"print(f\" Median per 1,000 pop: {df['transit_per_1000'].median():.2f}\")\n",
|
||||
"print(f\" Best Access: {df['transit_per_1000'].max():.2f} per 1,000\")\n",
|
||||
"print(f\" Worst Access: {df['transit_per_1000'].min():.2f} per 1,000\")"
|
||||
]
|
||||
}
|
||||
],
|
||||
"metadata": {
|
||||
"kernelspec": {
|
||||
"display_name": "Python 3",
|
||||
"language": "python",
|
||||
"name": "python3"
|
||||
},
|
||||
"language_info": {
|
||||
"name": "python",
|
||||
"version": "3.11.0"
|
||||
}
|
||||
},
|
||||
"nbformat": 4,
|
||||
"nbformat_minor": 4
|
||||
}
|
||||
Reference in New Issue
Block a user