personal-projects/personal-portfolio
Template
1
1
Fork 0
Code Issues 3 Pull Requests Actions Packages Projects 1 Releases Wiki Activity

docs: Streamline implementation plan to v2.0

Browse Source 
- Remove code snippets (will follow existing patterns during implementation)
- Remove appendices (risk, testing, file counts)
- Remove implementation timeline (handled by sprint planning)
- Keep Phase 1 cleanup detailed (actionable)
- Reduce Phases 3-5 to reference tables
- Keep notebook template, remove premature file listings
- Add CMHC zone mapping note

Trimmed from ~420 lines to ~180 lines for execution clarity.

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
This commit is contained in:
Leo Miranda
2026-01-16 09:39:40 -05:00
parent d3ca4ad4eb
commit 48b4eeeb62
1 changed files with 141 additions and 624 deletions
Download Patch File Download Diff File Expand all files Collapse all files

765
docs/changes/Change-Toronto-Analysis-Reviewed.md
View File

@@ -1,759 +1,276 @@
# Toronto Neighbourhood Dashboard — Implementation Plan # Toronto Neighbourhood Dashboard — Implementation Plan
**Document Type:** Change Implementation Plan **Document Type:** Execution Guide
**Target:** Transition from TRREB-based to Neighbourhood-based Dashboard **Target:** Transition from TRREB-based to Neighbourhood-based Dashboard
**Author:** Claude Code **Version:** 2.0 | January 2026
**Version:** 1.0 | January 2026
**Status:** Awaiting Approval
--- ---
## Executive Summary ## Overview
This plan details the transition from the current TRREB district-based housing dashboard to a comprehensive Toronto Neighbourhood Dashboard built around the city's 158 official neighbourhoods. The change simplifies geographic alignment, improves data availability through open APIs, and expands analytical scope to include housing, safety, demographics, and amenities. Transition from TRREB district-based housing dashboard to a comprehensive Toronto Neighbourhood Dashboard built around the city's 158 official neighbourhoods.
**Key Changes:** **Key Changes:**
- Geographic foundation shifts from TRREB districts (~35) to City Neighbourhoods (158) - Geographic foundation: TRREB districts (~35) → City Neighbourhoods (158)
- Data sources transition from PDF parsing to open APIs (Toronto Open Data, CMHC, Toronto Police) - Data sources: PDF parsing → Open APIs (Toronto Open Data, Toronto Police, CMHC)
- Dashboard expands from housing-only to 5 thematic tabs - Scope: Housing-only → 5 thematic tabs (Overview, Housing, Safety, Demographics, Amenities)
- Star schema redesigned around neighbourhood as the central dimension
---
## Table of Contents
1. [Phase 1: Repository Cleanup](#phase-1-repository-cleanup)
2. [Phase 2: Documentation Updates](#phase-2-documentation-updates)
3. [Phase 3: Data Pipeline Implementation](#phase-3-data-pipeline-implementation)
4. [Phase 4: dbt Model Restructuring](#phase-4-dbt-model-restructuring)
5. [Phase 5: Dashboard Implementation](#phase-5-dashboard-implementation)
6. [Phase 6: Jupyter Notebooks](#phase-6-jupyter-notebooks)
7. [Phase 7: Final Documentation Review](#phase-7-final-documentation-review)
8. [Phase 8: Commit and Merge Strategy](#phase-8-commit-and-merge-strategy)
--- ---
## Phase 1: Repository Cleanup ## Phase 1: Repository Cleanup
### 1.1 Files to DELETE (TRREB-Specific) ### Files to DELETE
These files are specific to the old TRREB district-based approach and will be completely removed: | File | Reason |
|------|--------|
| File | Reason for Deletion | | `portfolio_app/toronto/schemas/trreb.py` | TRREB schema obsolete |
|------|---------------------| | `portfolio_app/toronto/parsers/trreb.py` | PDF parsing no longer needed |
| `portfolio_app/toronto/schemas/trreb.py` | TRREB schema obsolete - replacing with neighbourhood-based |
| `portfolio_app/toronto/parsers/trreb.py` | PDF parsing no longer needed - using APIs |
| `portfolio_app/toronto/loaders/trreb.py` | TRREB loading logic obsolete | | `portfolio_app/toronto/loaders/trreb.py` | TRREB loading logic obsolete |
| `dbt/models/staging/stg_trreb__purchases.sql` | TRREB staging model obsolete | | `dbt/models/staging/stg_trreb__purchases.sql` | TRREB staging obsolete |
| `dbt/models/intermediate/int_purchases__monthly.sql` | TRREB-based intermediate obsolete | | `dbt/models/intermediate/int_purchases__monthly.sql` | TRREB intermediate obsolete |
| `dbt/models/marts/mart_toronto_purchases.sql` | Will be rebuilt for neighbourhood grain | | `dbt/models/marts/mart_toronto_purchases.sql` | Will rebuild for neighbourhood grain |
### 1.2 Files to MODIFY (Remove TRREB References) ### Files to MODIFY (Remove TRREB References)
| File | Changes Required | | File | Action |
|------|------------------| |------|--------|
| `portfolio_app/toronto/schemas/__init__.py` | Remove TRREB imports | | `portfolio_app/toronto/schemas/__init__.py` | Remove TRREB imports |
| `portfolio_app/toronto/parsers/__init__.py` | Remove TRREB parser imports | | `portfolio_app/toronto/parsers/__init__.py` | Remove TRREB parser imports |
| `portfolio_app/toronto/loaders/__init__.py` | Remove TRREB loader imports | | `portfolio_app/toronto/loaders/__init__.py` | Remove TRREB loader imports |
| `portfolio_app/toronto/models/facts.py` | Remove `FactPurchases` model (rebuild later) | | `portfolio_app/toronto/models/facts.py` | Remove `FactPurchases` model |
| `portfolio_app/toronto/models/dimensions.py` | Remove `DimTRREBDistrict` model | | `portfolio_app/toronto/models/dimensions.py` | Remove `DimTRREBDistrict` model |
| `portfolio_app/toronto/demo_data.py` | Remove TRREB demo districts, rebuild for neighbourhoods | | `portfolio_app/toronto/demo_data.py` | Remove TRREB demo data |
| `dbt/models/sources.yml` | Remove TRREB source definitions | | `dbt/models/sources.yml` | Remove TRREB source definitions |
| `dbt/models/schema.yml` | Remove TRREB model documentation | | `dbt/models/schema.yml` | Remove TRREB model documentation |
### 1.3 Files to KEEP (Reusable Infrastructure) ### Files to KEEP (Reusable)
| File | Why Keep | | File | Why |
|------|----------| |------|-----|
| `portfolio_app/toronto/schemas/cmhc.py` | CMHC data still used (requires zone-to-neighbourhood mapping) | | `portfolio_app/toronto/schemas/cmhc.py` | CMHC data still used |
| `portfolio_app/toronto/parsers/cmhc.py` | CMHC parser reusable with modifications | | `portfolio_app/toronto/parsers/cmhc.py` | Reusable with modifications |
| `portfolio_app/toronto/loaders/cmhc.py` | Loader patterns reusable |
| `portfolio_app/toronto/loaders/base.py` | Generic database utilities | | `portfolio_app/toronto/loaders/base.py` | Generic database utilities |
| `portfolio_app/toronto/loaders/dimensions.py` | Dimension loading patterns reusable | | `portfolio_app/toronto/loaders/dimensions.py` | Dimension loading patterns |
| `portfolio_app/toronto/models/base.py` | SQLAlchemy base class | | `portfolio_app/toronto/models/base.py` | SQLAlchemy base class |
| `portfolio_app/toronto/models/facts.py` | Keep `FactRentals`, refactor |
| `portfolio_app/toronto/models/dimensions.py` | Keep `DimTime`, `DimNeighbourhood`, refactor others |
| `portfolio_app/figures/*.py` | All chart factories reusable | | `portfolio_app/figures/*.py` | All chart factories reusable |
| `portfolio_app/components/*.py` | All UI components reusable | | `portfolio_app/components/*.py` | All UI components reusable |
### 1.4 Cleanup Commands
```bash
# Files to delete
rm portfolio_app/toronto/schemas/trreb.py
rm portfolio_app/toronto/parsers/trreb.py
rm portfolio_app/toronto/loaders/trreb.py
rm dbt/models/staging/stg_trreb__purchases.sql
rm dbt/models/intermediate/int_purchases__monthly.sql
rm dbt/models/marts/mart_toronto_purchases.sql
```
--- ---
## Phase 2: Documentation Updates ## Phase 2: Documentation Updates
### 2.1 Primary Documentation Files | Document | Action |
|----------|--------|
| Document | Current State | Required Updates | | `CLAUDE.md` | Update data model section, mark transition complete |
|----------|---------------|------------------| | `docs/PROJECT_REFERENCE.md` | Update architecture, data sources |
| `CLAUDE.md` (project) | References TRREB, old sprint structure | Complete rewrite of Data Model section | | `docs/toronto_housing_dashboard_spec_v5.md` | Archive or delete |
| `docs/PROJECT_REFERENCE.md` | Full spec references TRREB | Update architecture, data sources | | `docs/wbs_sprint_plan_v4.md` | Archive or delete |
| `docs/toronto_housing_dashboard_spec_v5.md` | TRREB/CMHC spec | Replace with neighbourhood spec |
| `docs/wbs_sprint_plan_v4.md` | Old sprint plan | New sprint plan for neighbourhood implementation |
### 2.2 CLAUDE.md Updates Required
**Section: Data Model Overview**
- Remove: TRREB Districts (~35) reference
- Remove: "These geographies do NOT align" note (now unified)
- Update: Star schema to neighbourhood-centric model
- Update: dbt layers description
**Section: Star Schema**
Replace with:
| Table | Type | Keys |
|-------|------|------|
| `dim_neighbourhood` | Central Dimension | neighbourhood_id (PK), geometry |
| `dim_time` | Dimension | date_key (PK) |
| `dim_cmhc_zone` | Bridge | zone_key (PK), neighbourhood mapping |
| `fact_census` | Fact | -> dim_neighbourhood, dim_time |
| `fact_crime` | Fact | -> dim_neighbourhood, dim_time |
| `fact_rentals` | Fact | -> dim_cmhc_zone, dim_time |
| `fact_amenities` | Fact | -> dim_neighbourhood |
**Section: DO NOT BUILD**
Update to reflect new scope constraints.
**Section: Module Responsibilities**
Update parsers description: "API extraction" instead of "PDF/CSV extraction"
### 2.3 New Reference Documents to Create
| Document | Purpose |
|----------|---------|
| `docs/neighbourhood_dashboard_spec_v1.md` | New dashboard specification (from Change-Toronto-Analysis.md) |
| `docs/data_source_inventory.md` | API endpoints, data dictionaries, refresh schedules |
| `docs/cmhc_neighbourhood_crosswalk.md` | CMHC zone to neighbourhood mapping methodology |
--- ---
## Phase 3: Data Pipeline Implementation ## Phase 3: New Data Model
### 3.1 New Schema Files ### Star Schema (Neighbourhood-Centric)
#### `portfolio_app/toronto/schemas/neighbourhood.py` | Table | Type | Description |
```python |-------|------|-------------|
"""Pydantic schemas for neighbourhood-level data.""" | `dim_neighbourhood` | Central Dimension | 158 neighbourhoods with geometry |
from pydantic import BaseModel, Field | `dim_time` | Dimension | Date dimension (keep existing) |
from datetime import date | `dim_cmhc_zone` | Bridge Dimension | 15 CMHC zones with neighbourhood mapping |
from typing import Optional | `bridge_cmhc_neighbourhood` | Bridge | Zone-to-neighbourhood area weights |
| `fact_census` | Fact | Census indicators by neighbourhood |
| `fact_crime` | Fact | Crime stats by neighbourhood |
| `fact_rentals` | Fact | Rental data by CMHC zone (keep existing) |
| `fact_amenities` | Fact | Amenity counts by neighbourhood |
class NeighbourhoodRecord(BaseModel): ### New Schema Files
"""Core neighbourhood dimension record."""
area_id: int = Field(..., description="Toronto Open Data AREA_ID")
name: str
population: Optional[int] = None
land_area_sqkm: Optional[float] = None
class CensusRecord(BaseModel): | File | Contains |
"""Census indicator for a neighbourhood.""" |------|----------|
neighbourhood_id: int | `toronto/schemas/neighbourhood.py` | NeighbourhoodRecord, CensusRecord, CrimeRecord |
census_year: int | `toronto/schemas/amenities.py` | AmenityType enum, AmenityRecord |
indicator_name: str
indicator_value: float
class CrimeRecord(BaseModel): ### New Parser Files
"""Crime statistics for a neighbourhood."""
neighbourhood_id: int
year: int
mci_category: str # Assault, B&E, Auto Theft, Robbery, Theft Over
count: int
rate_per_100k: float
```
#### `portfolio_app/toronto/schemas/amenities.py` | File | Data Source | API |
```python |------|-------------|-----|
"""Pydantic schemas for amenity data.""" | `toronto/parsers/toronto_open_data.py` | Neighbourhoods, Census, Parks, Schools, Childcare | Toronto Open Data Portal |
from pydantic import BaseModel | `toronto/parsers/toronto_police.py` | Crime Rates, MCI, Shootings | Toronto Police Portal |
from typing import Optional
from enum import Enum
class AmenityType(str, Enum): ### New Loader Files
PARK = "park"
SCHOOL = "school"
CHILDCARE = "childcare"
TTC_STOP = "ttc_stop"
class AmenityRecord(BaseModel):
"""Point amenity within a neighbourhood."""
neighbourhood_id: int
amenity_type: AmenityType
name: str
latitude: float
longitude: float
attributes: Optional[dict] = None # Type-specific attributes
```
### 3.2 New Parser Files
#### `portfolio_app/toronto/parsers/toronto_open_data.py`
```python
"""Parser for Toronto Open Data Portal APIs."""
# Endpoints:
# - Neighbourhoods GeoJSON
# - Neighbourhood Profiles CSV
# - Parks
# - Schools
# - Child Care Centres
class TorontoOpenDataParser:
BASE_URL = "https://ckan0.cf.opendata.inter.prod-toronto.ca"
def fetch_neighbourhoods_geojson(self) -> dict: ...
def fetch_neighbourhood_profiles(self) -> list[CensusRecord]: ...
def fetch_parks(self) -> list[AmenityRecord]: ...
def fetch_schools(self) -> list[AmenityRecord]: ...
def fetch_childcare(self) -> list[AmenityRecord]: ...
```
#### `portfolio_app/toronto/parsers/toronto_police.py`
```python
"""Parser for Toronto Police Service Open Data Portal."""
# Endpoints:
# - Neighbourhood Crime Rates
# - Major Crime Indicators
# - Shootings & Firearm Discharges
class TorontoPoliceParser:
BASE_URL = "https://data.torontopolice.on.ca"
def fetch_neighbourhood_crime_rates(self, year: int) -> list[CrimeRecord]: ...
def fetch_mci_details(self, year: int) -> list[CrimeRecord]: ...
def fetch_shootings(self, year: int) -> list[dict]: ...
```
### 3.3 New Model Files
#### `portfolio_app/toronto/models/dimensions.py` (Updated)
```python
"""Dimension models - neighbourhood as central dimension."""
class DimNeighbourhood(Base):
"""158 Toronto neighbourhoods - CENTRAL DIMENSION."""
__tablename__ = "dim_neighbourhood"
neighbourhood_id = Column(Integer, primary_key=True) # AREA_ID from GeoJSON
name = Column(String(100), nullable=False)
geometry = Column(Geometry("POLYGON", srid=4326))
population = Column(Integer)
land_area_sqkm = Column(Float)
pop_density_per_sqkm = Column(Float)
class DimCMHCZone(Base):
"""15 CMHC zones with neighbourhood mapping."""
__tablename__ = "dim_cmhc_zone"
zone_key = Column(Integer, primary_key=True, autoincrement=True)
zone_code = Column(String(10), unique=True, nullable=False)
zone_name = Column(String(100), nullable=False)
geometry = Column(Geometry("POLYGON", srid=4326))
class BridgeCMHCNeighbourhood(Base):
"""Many-to-many: CMHC zones to neighbourhoods with area weights."""
__tablename__ = "bridge_cmhc_neighbourhood"
zone_key = Column(Integer, ForeignKey("dim_cmhc_zone.zone_key"), primary_key=True)
neighbourhood_id = Column(Integer, ForeignKey("dim_neighbourhood.neighbourhood_id"), primary_key=True)
area_weight = Column(Float) # Proportion of neighbourhood in zone
```
#### `portfolio_app/toronto/models/facts.py` (Updated)
```python
"""Fact tables - all keyed to neighbourhood or CMHC zone."""
class FactCensus(Base):
"""Census indicators by neighbourhood."""
__tablename__ = "fact_census"
id = Column(Integer, primary_key=True, autoincrement=True)
neighbourhood_id = Column(Integer, ForeignKey("dim_neighbourhood.neighbourhood_id"))
date_key = Column(Integer, ForeignKey("dim_time.date_key"))
indicator_name = Column(String(100))
indicator_value = Column(Float)
class FactCrime(Base):
"""Crime statistics by neighbourhood."""
__tablename__ = "fact_crime"
id = Column(Integer, primary_key=True, autoincrement=True)
neighbourhood_id = Column(Integer, ForeignKey("dim_neighbourhood.neighbourhood_id"))
date_key = Column(Integer, ForeignKey("dim_time.date_key"))
mci_category = Column(String(50))
incident_count = Column(Integer)
rate_per_100k = Column(Float)
class FactAmenities(Base):
"""Amenity counts by neighbourhood (snapshot)."""
__tablename__ = "fact_amenities"
neighbourhood_id = Column(Integer, ForeignKey("dim_neighbourhood.neighbourhood_id"), primary_key=True)
parks_count = Column(Integer)
parks_area_sqm = Column(Float)
schools_count = Column(Integer)
childcare_spaces = Column(Integer)
ttc_stops_count = Column(Integer)
snapshot_date = Column(Date)
```
### 3.4 New Loader Files
| File | Purpose | | File | Purpose |
|------|---------| |------|---------|
| `portfolio_app/toronto/loaders/neighbourhoods.py` | Load GeoJSON boundaries | | `toronto/loaders/neighbourhoods.py` | Load GeoJSON boundaries |
| `portfolio_app/toronto/loaders/census.py` | Load neighbourhood profiles | | `toronto/loaders/census.py` | Load neighbourhood profiles |
| `portfolio_app/toronto/loaders/crime.py` | Load crime statistics | | `toronto/loaders/crime.py` | Load crime statistics |
| `portfolio_app/toronto/loaders/amenities.py` | Load parks, schools, childcare | | `toronto/loaders/amenities.py` | Load parks, schools, childcare |
| `portfolio_app/toronto/loaders/cmhc_crosswalk.py` | Build CMHC-neighbourhood bridge | | `toronto/loaders/cmhc_crosswalk.py` | Build CMHC-neighbourhood bridge |
--- ---
## Phase 4: dbt Model Restructuring ## Phase 4: dbt Restructuring
### 4.1 New Staging Models ### Staging Layer
| Model | Source | Purpose | | Model | Source |
|-------|--------|---------| |-------|--------|
| `stg_toronto__neighbourhoods` | dim_neighbourhood | Clean neighbourhood dimension | | `stg_toronto__neighbourhoods` | dim_neighbourhood |
| `stg_toronto__census` | fact_census | Pivoted census indicators | | `stg_toronto__census` | fact_census |
| `stg_toronto__crime` | fact_crime | Cleaned crime data | | `stg_toronto__crime` | fact_crime |
| `stg_toronto__amenities` | fact_amenities | Amenity counts | | `stg_toronto__amenities` | fact_amenities |
| `stg_cmhc__rentals` | fact_rentals | (Keep existing, modify) | | `stg_cmhc__rentals` | fact_rentals (modify existing) |
| `stg_cmhc__zone_crosswalk` | bridge_cmhc_neighbourhood | Zone-neighbourhood mapping | | `stg_cmhc__zone_crosswalk` | bridge_cmhc_neighbourhood |
### 4.2 New Intermediate Models ### Intermediate Layer
| Model | Purpose | | Model | Purpose |
|-------|---------| |-------|---------|
| `int_neighbourhood__demographics` | Combined census demographics | | `int_neighbourhood__demographics` | Combined census demographics |
| `int_neighbourhood__housing` | Housing indicators from census | | `int_neighbourhood__housing` | Housing indicators |
| `int_neighbourhood__crime_summary` | Aggregated crime by type | | `int_neighbourhood__crime_summary` | Aggregated crime by type |
| `int_neighbourhood__amenity_scores` | Normalized amenity metrics | | `int_neighbourhood__amenity_scores` | Normalized amenity metrics |
| `int_rentals__neighbourhood_allocated` | CMHC rentals allocated to neighbourhoods | | `int_rentals__neighbourhood_allocated` | CMHC rentals allocated to neighbourhoods |
### 4.3 New Mart Models ### Mart Layer (One per Tab)
| Model | Purpose | Tab | | Model | Tab | Key Metrics |
|-------|---------|-----| |-------|-----|-------------|
| `mart_neighbourhood_overview` | Composite livability scores | Overview | | `mart_neighbourhood_overview` | Overview | Composite livability score |
| `mart_neighbourhood_housing` | Affordability metrics | Housing | | `mart_neighbourhood_housing` | Housing | Affordability index, rent-to-income |
| `mart_neighbourhood_safety` | Crime rates and trends | Safety | | `mart_neighbourhood_safety` | Safety | Crime rates, YoY change |
| `mart_neighbourhood_demographics` | Population, income, diversity | Demographics | | `mart_neighbourhood_demographics` | Demographics | Income, age, diversity |
| `mart_neighbourhood_amenities` | Parks, schools, transit access | Amenities | | `mart_neighbourhood_amenities` | Amenities | Parks, schools, transit per capita |
| `mart_dashboard_kpis` | Pre-computed KPI values | All tabs |
### 4.4 dbt Sources Configuration
```yaml
# dbt/models/sources.yml
version: 2
sources:
- name: toronto
schema: public
tables:
- name: dim_neighbourhood
identifier: dim_neighbourhood
- name: dim_time
identifier: dim_time
- name: dim_cmhc_zone
identifier: dim_cmhc_zone
- name: bridge_cmhc_neighbourhood
identifier: bridge_cmhc_neighbourhood
- name: fact_census
identifier: fact_census
- name: fact_crime
identifier: fact_crime
- name: fact_rentals
identifier: fact_rentals
- name: fact_amenities
identifier: fact_amenities
```
--- ---
## Phase 5: Dashboard Implementation ## Phase 5: Dashboard Implementation
### 5.1 Tab Structure ### Tab Structure
``` ```
pages/toronto/ pages/toronto/
├── dashboard.py # Main layout with tab navigation ├── dashboard.py # Main layout with tab navigation
├── methodology.py # Keep existing
├── tabs/ ├── tabs/
│ ├── __init__.py │ ├── overview.py # Composite livability
│ ├── overview.py # Tab 1: Composite livability │ ├── housing.py # Affordability
│ ├── housing.py # Tab 2: Affordability │ ├── safety.py # Crime
│ ├── safety.py # Tab 3: Crime │ ├── demographics.py # Population
│ ├── demographics.py # Tab 4: Population │ └── amenities.py # Services
│ └── amenities.py # Tab 5: Services
└── callbacks/ └── callbacks/
├── __init__.py ├── map_callbacks.py
├── map_callbacks.py # Choropleth interactions ├── chart_callbacks.py
├── chart_callbacks.py # Supporting charts └── selection_callbacks.py
└── selection_callbacks.py # Neighbourhood selection
``` ```
### 5.2 Shared Components per Tab ### Layout Pattern (All Tabs)
Each tab follows the same layout pattern: Each tab follows the same structure:
1. **Choropleth Map** (left) — 158 neighbourhoods, click to select
2. **KPI Cards** (right) — 3-4 contextual metrics
3. **Supporting Charts** (bottom) — Trend + comparison visualizations
4. **Details Panel** (collapsible) — All metrics for selected neighbourhood
1. **Choropleth Map** (left, 60% width) ### Graphs by Tab
- 158 neighbourhoods
- Color by selected metric
- Click to select neighbourhood
2. **KPI Cards** (right, 40% width) | Tab | Choropleth Metric | Chart 1 | Chart 2 |
- 3-4 contextual KPIs |-----|-------------------|---------|---------|
- Update on neighbourhood selection | Overview | Livability score | Top/Bottom 10 bar | Income vs Crime scatter |
| Housing | Affordability index | Rent trend (5yr line) | Dwelling types (pie/bar) |
3. **Supporting Charts** (bottom row) | Safety | Crime rate per 100K | Crime breakdown (stacked bar) | Crime trend (5yr line) |
- Chart 1: Context/trend visualization | Demographics | Median income | Age pyramid | Top languages (bar) |
- Chart 2: Comparison/ranking visualization | Amenities | Park area per capita | Amenity radar | Transit accessibility (bar) |
4. **Details Panel** (collapsible)
- All metrics for selected neighbourhood
### 5.3 Graphs by Tab
#### Tab 1: Overview
| Graph ID | Type | Data Source |
|----------|------|-------------|
| `overview-choropleth` | Choropleth | mart_neighbourhood_overview |
| `overview-top-bottom` | Horizontal Bar | mart_neighbourhood_overview |
| `overview-income-crime-scatter` | Scatter | mart_neighbourhood_overview |
#### Tab 2: Housing & Affordability
| Graph ID | Type | Data Source |
|----------|------|-------------|
| `housing-choropleth` | Choropleth | mart_neighbourhood_housing |
| `housing-rent-trend` | Line | mart_neighbourhood_housing (historical) |
| `housing-dwelling-types` | Pie/Bar | mart_neighbourhood_housing |
#### Tab 3: Safety
| Graph ID | Type | Data Source |
|----------|------|-------------|
| `safety-choropleth` | Choropleth | mart_neighbourhood_safety |
| `safety-crime-breakdown` | Stacked Bar | mart_neighbourhood_safety |
| `safety-trend` | Line | mart_neighbourhood_safety (5-year) |
#### Tab 4: Demographics
| Graph ID | Type | Data Source |
|----------|------|-------------|
| `demographics-choropleth` | Choropleth | mart_neighbourhood_demographics |
| `demographics-age-pyramid` | Population Pyramid | mart_neighbourhood_demographics |
| `demographics-languages` | Horizontal Bar | mart_neighbourhood_demographics |
#### Tab 5: Amenities & Services
| Graph ID | Type | Data Source |
|----------|------|-------------|
| `amenities-choropleth` | Choropleth | mart_neighbourhood_amenities |
| `amenities-radar` | Radar | mart_neighbourhood_amenities |
| `amenities-transit` | Bar | mart_neighbourhood_amenities |
--- ---
## Phase 6: Jupyter Notebooks ## Phase 6: Jupyter Notebooks
### 6.1 Notebook Structure ### Purpose
Create one notebook per graph following this template: One notebook per graph to document:
1. **Data Reference** — How the data was built (query, transformation steps, sample output)
2. **Data Visualization** — Import figure factory, render the graph
### Directory Structure
``` ```
notebooks/ notebooks/
├── README.md # Notebook index and conventions ├── README.md
├── overview/ ├── overview/
│ ├── 01_overview_choropleth.ipynb
│ ├── 02_overview_top_bottom.ipynb
│ └── 03_overview_income_crime_scatter.ipynb
├── housing/ ├── housing/
│ ├── 01_housing_choropleth.ipynb
│ ├── 02_housing_rent_trend.ipynb
│ └── 03_housing_dwelling_types.ipynb
├── safety/ ├── safety/
│ ├── 01_safety_choropleth.ipynb
│ ├── 02_safety_crime_breakdown.ipynb
│ └── 03_safety_trend.ipynb
├── demographics/ ├── demographics/
│ ├── 01_demographics_choropleth.ipynb
│ ├── 02_demographics_age_pyramid.ipynb
│ └── 03_demographics_languages.ipynb
└── amenities/ └── amenities/
├── 01_amenities_choropleth.ipynb
├── 02_amenities_radar.ipynb
└── 03_amenities_transit.ipynb
``` ```
### 6.2 Notebook Template ### Notebook Template
Each notebook follows this structure:
```markdown ```markdown
# [Graph Name] — Data Reference & Visualization # [Graph Name]
## 1. Data Reference ## 1. Data Reference
### 1.1 Source Tables ### Source Tables
- List all source tables/marts used - List tables/marts used
- Explain the grain of each table - Grain of each table
### 1.2 Query ### Query
```sql ```sql
-- The exact query that feeds this visualization SELECT ... FROM ...
SELECT ...
FROM ...
``` ```
### 1.3 Data Pipeline Steps ### Transformation Steps
1. Step 1: Description of transformation 1. Step description
2. Step 2: Description of aggregation 2. Step description
3. Step 3: Description of final shaping
### 1.4 Sample Data ### Sample Data
```python ```python
import pandas as pd
from sqlalchemy import create_engine
engine = create_engine(DATABASE_URL)
df = pd.read_sql(query, engine) df = pd.read_sql(query, engine)
df.head(10) df.head(10)
``` ```
## 2. Data Visualization ## 2. Data Visualization
### 2.1 Import Graph Factory
```python ```python
from portfolio_app.figures.choropleth import create_choropleth_figure from portfolio_app.figures.choropleth import create_choropleth_figure
# or appropriate figure factory fig = create_choropleth_figure(...)
```
### 2.2 Create Visualization
```python
fig = create_choropleth_figure(
geojson=geojson_data,
data=df.to_dict('records'),
...
)
fig.show() fig.show()
``` ```
### 2.3 Interpretation Notes
- What this visualization shows
- Key insights from the data
- Caveats or limitations
``` ```
### 6.3 Notebooks to Create (15 Total) Create one notebook per graph as each is implemented (15 total across 5 tabs).
| # | Notebook | Graph | Tab |
|---|----------|-------|-----|
| 1 | `overview/01_overview_choropleth.ipynb` | Livability score map | Overview |
| 2 | `overview/02_overview_top_bottom.ipynb` | Top/Bottom 10 bar chart | Overview |
| 3 | `overview/03_overview_income_crime_scatter.ipynb` | Income vs Crime scatter | Overview |
| 4 | `housing/01_housing_choropleth.ipynb` | Affordability index map | Housing |
| 5 | `housing/02_housing_rent_trend.ipynb` | 5-year rent trend line | Housing |
| 6 | `housing/03_housing_dwelling_types.ipynb` | Dwelling type breakdown | Housing |
| 7 | `safety/01_safety_choropleth.ipynb` | Crime rate map | Safety |
| 8 | `safety/02_safety_crime_breakdown.ipynb` | Crime type stacked bar | Safety |
| 9 | `safety/03_safety_trend.ipynb` | 5-year crime trend line | Safety |
| 10 | `demographics/01_demographics_choropleth.ipynb` | Income distribution map | Demographics |
| 11 | `demographics/02_demographics_age_pyramid.ipynb` | Age distribution pyramid | Demographics |
| 12 | `demographics/03_demographics_languages.ipynb` | Top languages bar chart | Demographics |
| 13 | `amenities/01_amenities_choropleth.ipynb` | Park area per capita map | Amenities |
| 14 | `amenities/02_amenities_radar.ipynb` | Amenity density radar | Amenities |
| 15 | `amenities/03_amenities_transit.ipynb` | Transit accessibility bar | Amenities |
--- ---
## Phase 7: Final Documentation Review ## Phase 7: Final Documentation Review
### 7.1 Documentation Audit Checklist After all implementation, audit and update:
After all implementation is complete, perform a comprehensive audit: - [ ] `CLAUDE.md` — Project status, app structure, data model, URL routes
- [ ] `README.md` — Project description, installation, quick start
#### Project CLAUDE.md - [ ] `docs/PROJECT_REFERENCE.md` — Architecture matches implementation
- [ ] Project Status reflects current sprint - [ ] Remove or archive legacy spec documents
- [ ] Run Commands are accurate
- [ ] Code Conventions match actual code
- [ ] Application Structure matches filesystem
- [ ] URL Routing matches registered pages
- [ ] Tech Stack versions are accurate
- [ ] Data Model matches SQLAlchemy models
- [ ] dbt Layers match actual models
- [ ] Reference Documents section is current
#### docs/PROJECT_REFERENCE.md
- [ ] Architecture diagrams are current
- [ ] Data source inventory is complete
- [ ] API endpoints documented
- [ ] Refresh schedules documented
#### docs/neighbourhood_dashboard_spec_v1.md
- [ ] All tabs documented
- [ ] All graphs documented
- [ ] Data sources for each graph documented
- [ ] Colour palettes specified
#### README.md
- [ ] Project description accurate
- [ ] Installation instructions work
- [ ] Quick start guide is functional
### 7.2 App Structure Verification
Verify documentation matches actual app by running:
```bash
# Generate actual page routes
grep -r "dash.register_page" portfolio_app/pages/ --include="*.py"
# Generate actual model classes
grep -r "class.*Base" portfolio_app/toronto/models/ --include="*.py"
# Generate actual dbt models
ls -la dbt/models/**/*.sql
```
### 7.3 Final Documentation Updates
Based on audit findings, update:
1. All file path references
2. All URL route tables
3. All model/schema references
4. All dbt model references
5. Sprint number and status
--- ---
## Phase 8: Commit and Merge Strategy ## Data Source Reference
### 8.1 Branch Strategy | Source | Datasets | URL |
|--------|----------|-----|
``` | Toronto Open Data | Neighbourhoods, Census Profiles, Parks, Schools, Childcare, TTC | open.toronto.ca |
development (base) | Toronto Police | Crime Rates, MCI, Shootings | data.torontopolice.on.ca |
└── feature/neighbourhood-dashboard-transition | CMHC | Rental Market Survey | cmhc-schl.gc.ca |
├── Commit 1: Cleanup - Remove TRREB files
├── Commit 2: Schemas - New neighbourhood schemas
├── Commit 3: Models - Updated SQLAlchemy models
├── Commit 4: Parsers - API parsers implementation
├── Commit 5: Loaders - Data loading functions
├── Commit 6: dbt - New staging/intermediate/mart models
├── Commit 7: Dashboard - Tab implementations
├── Commit 8: Callbacks - Dashboard interactivity
├── Commit 9: Notebooks - All 15 Jupyter notebooks
├── Commit 10: Documentation - Updated docs
└── Commit 11: Final review - Documentation audit fixes
```
### 8.2 Commit Messages
Follow conventional commits format:
```
feat: Add neighbourhood-based schemas and models
fix: Remove obsolete TRREB pipeline
docs: Update CLAUDE.md for neighbourhood dashboard
refactor: Restructure dbt models for neighbourhood grain
test: Add tests for new parsers and loaders
```
### 8.3 Merge Process
1. Create feature branch from development
2. Implement all phases with atomic commits
3. Run full CI checks: `make ci`
4. Create PR to development
5. Squash merge with comprehensive message
6. Delete feature branch
7. Tag release: `v2.0.0-neighbourhood-dashboard`
--- ---
## Implementation Timeline ## CMHC Zone Mapping Note
### Sprint 9: Cleanup & Foundation CMHC uses 15 zones that don't align with 158 neighbourhoods. Strategy:
- [ ] Phase 1: Repository cleanup (delete TRREB files) - Create `bridge_cmhc_neighbourhood` with area weights
- [ ] Phase 2: Documentation updates (CLAUDE.md, specs) - Allocate rental metrics proportionally to overlapping neighbourhoods
- [ ] Phase 3.1: New schemas created - Document methodology in `/toronto/methodology` page
### Sprint 10: Data Pipeline
- [ ] Phase 3.2: Parsers implementation (API integrations)
- [ ] Phase 3.3: Models implementation (SQLAlchemy)
- [ ] Phase 3.4: Loaders implementation
### Sprint 11: dbt & Dashboard
- [ ] Phase 4: dbt model restructuring
- [ ] Phase 5.1: Dashboard layout with tabs
- [ ] Phase 5.2: Choropleth maps per tab
### Sprint 12: Interactivity & Charts
- [ ] Phase 5.3: Supporting charts implementation
- [ ] Phase 5.4: Callbacks and interactivity
- [ ] Phase 6.1-6.5: Jupyter notebooks (3 per sprint day)
### Sprint 13: Documentation & Release
- [ ] Phase 6.6-6.15: Remaining notebooks
- [ ] Phase 7: Final documentation review
- [ ] Phase 8: Commit, merge, and tag release
--- ---
## Appendix A: Files Inventory Summary *Document Version: 2.0*
*Trimmed from v1.0 for execution clarity*
### Files to DELETE: 6
### Files to MODIFY: 8
### Files to CREATE: ~45
- Schemas: 2
- Parsers: 2
- Models: 2 (modifications)
- Loaders: 5
- dbt models: 15
- Dashboard tabs: 5
- Callbacks: 3
- Notebooks: 15
- Documentation: 3
---
## Appendix B: Risk Mitigation
| Risk | Mitigation |
|------|------------|
| CMHC zone-neighbourhood mapping inaccuracy | Document methodology, use area-weighted allocation |
| API rate limits | Implement caching, respect rate limits, store locally |
| Census data staleness (5-year cycle) | Document data vintage, display last update prominently |
| Geographic boundary changes | Lock to 2024 158-neighbourhood boundaries |
---
## Appendix C: Testing Strategy
### Unit Tests
- Schema validation tests
- Parser output format tests
- Loader idempotency tests
### Integration Tests
- End-to-end: API -> Parse -> Load -> dbt -> Dashboard
- Database constraint tests
- dbt model tests (unique, not_null, relationships)
### Visual Regression Tests
- Screenshot comparison for each tab
- Choropleth rendering tests
---
*Document Version: 1.0*
*Created: January 2026*
*Author: Claude Code*