personal-projects/personal-portfolio
Template
1
1
Fork 0
Code Issues 3 Pull Requests Actions Packages Projects 1 Releases Wiki Activity
Files
c9cf744d84a4bf97890281ff50552ecb89aee109
personal-portfolio/docs/changes/Change-Toronto-Analysis-Reviewed.md
lmiranda 48b4eeeb62 docs: Streamline implementation plan to v2.0 
- Remove code snippets (will follow existing patterns during implementation)
- Remove appendices (risk, testing, file counts)
- Remove implementation timeline (handled by sprint planning)
- Keep Phase 1 cleanup detailed (actionable)
- Reduce Phases 3-5 to reference tables
- Keep notebook template, remove premature file listings
- Add CMHC zone mapping note

Trimmed from ~420 lines to ~180 lines for execution clarity.

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
2026-01-16 09:39:40 -05:00

277 lines
8.8 KiB
Markdown
Raw Blame History

# Toronto Neighbourhood Dashboard — Implementation Plan
**Document Type:** Execution Guide
**Target:** Transition from TRREB-based to Neighbourhood-based Dashboard
**Version:** 2.0 | January 2026
---
## Overview
Transition from TRREB district-based housing dashboard to a comprehensive Toronto Neighbourhood Dashboard built around the city's 158 official neighbourhoods.
**Key Changes:**
- Geographic foundation: TRREB districts (~35) → City Neighbourhoods (158)
- Data sources: PDF parsing → Open APIs (Toronto Open Data, Toronto Police, CMHC)
- Scope: Housing-only → 5 thematic tabs (Overview, Housing, Safety, Demographics, Amenities)
---
## Phase 1: Repository Cleanup
### Files to DELETE
| File | Reason |
|------|--------|
| `portfolio_app/toronto/schemas/trreb.py` | TRREB schema obsolete |
| `portfolio_app/toronto/parsers/trreb.py` | PDF parsing no longer needed |
| `portfolio_app/toronto/loaders/trreb.py` | TRREB loading logic obsolete |
| `dbt/models/staging/stg_trreb__purchases.sql` | TRREB staging obsolete |
| `dbt/models/intermediate/int_purchases__monthly.sql` | TRREB intermediate obsolete |
| `dbt/models/marts/mart_toronto_purchases.sql` | Will rebuild for neighbourhood grain |
### Files to MODIFY (Remove TRREB References)
| File | Action |
|------|--------|
| `portfolio_app/toronto/schemas/__init__.py` | Remove TRREB imports |
| `portfolio_app/toronto/parsers/__init__.py` | Remove TRREB parser imports |
| `portfolio_app/toronto/loaders/__init__.py` | Remove TRREB loader imports |
| `portfolio_app/toronto/models/facts.py` | Remove `FactPurchases` model |
| `portfolio_app/toronto/models/dimensions.py` | Remove `DimTRREBDistrict` model |
| `portfolio_app/toronto/demo_data.py` | Remove TRREB demo data |
| `dbt/models/sources.yml` | Remove TRREB source definitions |
| `dbt/models/schema.yml` | Remove TRREB model documentation |
### Files to KEEP (Reusable)
| File | Why |
|------|-----|
| `portfolio_app/toronto/schemas/cmhc.py` | CMHC data still used |
| `portfolio_app/toronto/parsers/cmhc.py` | Reusable with modifications |
| `portfolio_app/toronto/loaders/base.py` | Generic database utilities |
| `portfolio_app/toronto/loaders/dimensions.py` | Dimension loading patterns |
| `portfolio_app/toronto/models/base.py` | SQLAlchemy base class |
| `portfolio_app/figures/*.py` | All chart factories reusable |
| `portfolio_app/components/*.py` | All UI components reusable |
---
## Phase 2: Documentation Updates
| Document | Action |
|----------|--------|
| `CLAUDE.md` | Update data model section, mark transition complete |
| `docs/PROJECT_REFERENCE.md` | Update architecture, data sources |
| `docs/toronto_housing_dashboard_spec_v5.md` | Archive or delete |
| `docs/wbs_sprint_plan_v4.md` | Archive or delete |
---
## Phase 3: New Data Model
### Star Schema (Neighbourhood-Centric)
| Table | Type | Description |
|-------|------|-------------|
| `dim_neighbourhood` | Central Dimension | 158 neighbourhoods with geometry |
| `dim_time` | Dimension | Date dimension (keep existing) |
| `dim_cmhc_zone` | Bridge Dimension | 15 CMHC zones with neighbourhood mapping |
| `bridge_cmhc_neighbourhood` | Bridge | Zone-to-neighbourhood area weights |
| `fact_census` | Fact | Census indicators by neighbourhood |
| `fact_crime` | Fact | Crime stats by neighbourhood |
| `fact_rentals` | Fact | Rental data by CMHC zone (keep existing) |
| `fact_amenities` | Fact | Amenity counts by neighbourhood |
### New Schema Files
| File | Contains |
|------|----------|
| `toronto/schemas/neighbourhood.py` | NeighbourhoodRecord, CensusRecord, CrimeRecord |
| `toronto/schemas/amenities.py` | AmenityType enum, AmenityRecord |
### New Parser Files
| File | Data Source | API |
|------|-------------|-----|
| `toronto/parsers/toronto_open_data.py` | Neighbourhoods, Census, Parks, Schools, Childcare | Toronto Open Data Portal |
| `toronto/parsers/toronto_police.py` | Crime Rates, MCI, Shootings | Toronto Police Portal |
### New Loader Files
| File | Purpose |
|------|---------|
| `toronto/loaders/neighbourhoods.py` | Load GeoJSON boundaries |
| `toronto/loaders/census.py` | Load neighbourhood profiles |
| `toronto/loaders/crime.py` | Load crime statistics |
| `toronto/loaders/amenities.py` | Load parks, schools, childcare |
| `toronto/loaders/cmhc_crosswalk.py` | Build CMHC-neighbourhood bridge |
---
## Phase 4: dbt Restructuring
### Staging Layer
| Model | Source |
|-------|--------|
| `stg_toronto__neighbourhoods` | dim_neighbourhood |
| `stg_toronto__census` | fact_census |
| `stg_toronto__crime` | fact_crime |
| `stg_toronto__amenities` | fact_amenities |
| `stg_cmhc__rentals` | fact_rentals (modify existing) |
| `stg_cmhc__zone_crosswalk` | bridge_cmhc_neighbourhood |
### Intermediate Layer
| Model | Purpose |
|-------|---------|
| `int_neighbourhood__demographics` | Combined census demographics |
| `int_neighbourhood__housing` | Housing indicators |
| `int_neighbourhood__crime_summary` | Aggregated crime by type |
| `int_neighbourhood__amenity_scores` | Normalized amenity metrics |
| `int_rentals__neighbourhood_allocated` | CMHC rentals allocated to neighbourhoods |
### Mart Layer (One per Tab)
| Model | Tab | Key Metrics |
|-------|-----|-------------|
| `mart_neighbourhood_overview` | Overview | Composite livability score |
| `mart_neighbourhood_housing` | Housing | Affordability index, rent-to-income |
| `mart_neighbourhood_safety` | Safety | Crime rates, YoY change |
| `mart_neighbourhood_demographics` | Demographics | Income, age, diversity |
| `mart_neighbourhood_amenities` | Amenities | Parks, schools, transit per capita |
---
## Phase 5: Dashboard Implementation
### Tab Structure
```
pages/toronto/
├── dashboard.py # Main layout with tab navigation
├── tabs/
│ ├── overview.py # Composite livability
│ ├── housing.py # Affordability
│ ├── safety.py # Crime
│ ├── demographics.py # Population
│ └── amenities.py # Services
└── callbacks/
├── map_callbacks.py
├── chart_callbacks.py
└── selection_callbacks.py
```
### Layout Pattern (All Tabs)
Each tab follows the same structure:
1. **Choropleth Map** (left) — 158 neighbourhoods, click to select
2. **KPI Cards** (right) — 3-4 contextual metrics
3. **Supporting Charts** (bottom) — Trend + comparison visualizations
4. **Details Panel** (collapsible) — All metrics for selected neighbourhood
### Graphs by Tab
| Tab | Choropleth Metric | Chart 1 | Chart 2 |
|-----|-------------------|---------|---------|
| Overview | Livability score | Top/Bottom 10 bar | Income vs Crime scatter |
| Housing | Affordability index | Rent trend (5yr line) | Dwelling types (pie/bar) |
| Safety | Crime rate per 100K | Crime breakdown (stacked bar) | Crime trend (5yr line) |
| Demographics | Median income | Age pyramid | Top languages (bar) |
| Amenities | Park area per capita | Amenity radar | Transit accessibility (bar) |
---
## Phase 6: Jupyter Notebooks
### Purpose
One notebook per graph to document:
1. **Data Reference** — How the data was built (query, transformation steps, sample output)
2. **Data Visualization** — Import figure factory, render the graph
### Directory Structure
```
notebooks/
├── README.md
├── overview/
├── housing/
├── safety/
├── demographics/
└── amenities/
```
### Notebook Template
```markdown
# [Graph Name]
## 1. Data Reference
### Source Tables
- List tables/marts used
- Grain of each table
### Query
```sql
SELECT ... FROM ...
```
### Transformation Steps
1. Step description
2. Step description
### Sample Data
```python
df = pd.read_sql(query, engine)
df.head(10)
```
## 2. Data Visualization
```python
from portfolio_app.figures.choropleth import create_choropleth_figure
fig = create_choropleth_figure(...)
fig.show()
```
```
Create one notebook per graph as each is implemented (15 total across 5 tabs).
---
## Phase 7: Final Documentation Review
After all implementation, audit and update:
- [ ] `CLAUDE.md` — Project status, app structure, data model, URL routes
- [ ] `README.md` — Project description, installation, quick start
- [ ] `docs/PROJECT_REFERENCE.md` — Architecture matches implementation
- [ ] Remove or archive legacy spec documents
---
## Data Source Reference
| Source | Datasets | URL |
|--------|----------|-----|
| Toronto Open Data | Neighbourhoods, Census Profiles, Parks, Schools, Childcare, TTC | open.toronto.ca |
| Toronto Police | Crime Rates, MCI, Shootings | data.torontopolice.on.ca |
| CMHC | Rental Market Survey | cmhc-schl.gc.ca |
---
## CMHC Zone Mapping Note
CMHC uses 15 zones that don't align with 158 neighbourhoods. Strategy:
- Create `bridge_cmhc_neighbourhood` with area weights
- Allocate rental metrics proportionally to overlapping neighbourhoods
- Document methodology in `/toronto/methodology` page
---
*Document Version: 2.0*
*Trimmed from v1.0 for execution clarity*
Reference in New Issue View Git Blame Copy Permalink