personal-projects/personal-portfolio
Template
1
1
Fork 0
Code Issues 3 Pull Requests Actions Packages Projects 1 Releases Wiki Activity
65 Commits 3 Branches 1 Tag
7a16e6d121327c96bc5fddb320bff2cec9fb9bf6
Commit Graph

8 Commits

Author SHA1 Message Date
lmiranda
d0f32edba7 fix: Repair data pipeline with StatCan CMHC rental data 
- Add StatCan CMHC parser to fetch rental data from Statistics Canada API
- Create year spine (2014-2025) as time dimension driver instead of census
- Add CMA-level rental and income intermediate models
- Update mart_neighbourhood_overview to use rental years as base
- Fix neighbourhood_service queries to match dbt schema
- Add CMHC data loading to pipeline script

Data now flows correctly: 158 neighbourhoods × 12 years = 1,896 records
Rent data available 2019-2025, crime data 2014-2024

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
 2026-01-17 15:38:31 -05:00
lmiranda
c9cf744d84 feat: Complete Phase 5 dashboard implementation 
Implement full 5-tab Toronto Neighbourhood Dashboard with real data
connectivity:

Dashboard Structure:
- Overview tab with livability scores and rankings
- Housing tab with affordability metrics
- Safety tab with crime statistics
- Demographics tab with population/income data
- Amenities tab with parks, schools, transit

Figure Factories (portfolio_app/figures/):
- bar_charts.py: ranking, stacked, horizontal bars
- scatter.py: scatter plots, bubble charts
- radar.py: spider/radar charts
- demographics.py: donut, age pyramid, income distribution

Service Layer (portfolio_app/toronto/services/):
- neighbourhood_service.py: queries dbt marts for all tab data
- geometry_service.py: generates GeoJSON from PostGIS
- Graceful error handling when database unavailable

Callbacks (portfolio_app/pages/toronto/callbacks/):
- map_callbacks.py: choropleth updates, map click handling
- chart_callbacks.py: supporting chart updates
- selection_callbacks.py: dropdown handlers, KPI updates

Data Pipeline (scripts/data/):
- load_toronto_data.py: orchestration script with CLI flags

Lessons Learned:
- Graceful error handling in service layers
- Modular callback structure for multi-tab dashboards
- Figure factory pattern for reusable charts

Closes: #64, #65, #66, #67, #68, #69, #70

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
 2026-01-17 11:46:18 -05:00
lmiranda
053acf6436 feat: Implement Phase 3 neighbourhood data model 
Add schemas, parsers, loaders, and models for Toronto neighbourhood-centric
data including census profiles, crime statistics, and amenities.

Schemas:
- NeighbourhoodRecord, CensusRecord, CrimeRecord, CrimeType
- AmenityType, AmenityRecord, AmenityCount

Models:
- BridgeCMHCNeighbourhood (zone-to-neighbourhood mapping with weights)
- FactCensus, FactCrime, FactAmenities

Parsers:
- TorontoOpenDataParser (CKAN API for neighbourhoods, census, amenities)
- TorontoPoliceParser (crime rates, MCI data)

Loaders:
- load_census_data, load_crime_data, load_amenities
- build_cmhc_neighbourhood_crosswalk (PostGIS area weights)

Also updates CLAUDE.md with projman plugin workflow documentation.

Closes #53, #54, #55, #56, #57, #58, #59

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
 2026-01-16 11:07:13 -05:00
lmiranda
fcaefabce8 chore: Remove TRREB references from Python modules 
- Remove DimTRREBDistrict model and FactPurchases model
- Remove TRREBDistrict schema and AreaType enum
- Remove TRREBDistrictParser from geo parsers
- Remove load_trreb_districts from dimension loaders
- Remove create_district_map from choropleth figures
- Remove get_demo_districts and get_demo_purchase_data from demo_data
- Update summary metrics to remove purchase-related metrics
- Update callbacks to remove TRREB-related comments
- Update methodology page to remove TRREB data source section
- Update dashboard data notice to remove TRREB mention

Closes #49

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
 2026-01-16 10:11:11 -05:00
lmiranda
cb877df9e1 refactor: Delete legacy TRREB Python modules (#47) 
- Delete portfolio_app/toronto/schemas/trreb.py
- Delete portfolio_app/toronto/parsers/trreb.py
- Delete portfolio_app/toronto/loaders/trreb.py
- Remove TRREB imports from __init__.py files

Part of Sprint 9: Toronto Neighbourhood Dashboard transition
See docs/changes/Change-Toronto-Analysis-Reviewed.md

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
 2026-01-16 10:00:47 -05:00
lmiranda
9dfa24fb76 feat: add GeoJSON parsers and choropleth map visualization 
- Add geo.py parser module with CMHCZoneParser, TRREBDistrictParser,
  and NeighbourhoodParser for loading geographic boundaries
- Add coordinate reprojection support (EPSG:3857 to WGS84)
- Organize geo data in data/toronto/raw/geo/ directory
- Add CMHC zones GeoJSON (31 zones) for rental market choropleth
- Add Toronto neighbourhoods GeoJSON (158) as purchase market proxy
- Update callbacks with real CMHC 2024 rental data
- Add sample purchase data for all 158 neighbourhoods
- Update pre-commit config to exclude geo data files

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
 2026-01-14 17:58:13 -05:00
lmiranda
ead6d91a28 feat: add Pydantic schemas, SQLAlchemy models, and parser structure 
Sprint 3 implementation:
- Pydantic schemas for TRREB, CMHC, and dimension data validation
- SQLAlchemy models with PostGIS geometry for fact and dimension tables
- Parser structure (stubs) for TRREB PDF and CMHC CSV processing

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
 2026-01-11 14:58:31 -05:00
lmiranda
c7e9b88adb feat: project bootstrap and structure 
Sprint 1 initialization:
- Project directory structure (portfolio_app/, tests/, dbt/, data/, scripts/)
- CLAUDE.md with AI assistant context
- pyproject.toml with all dependencies
- docker-compose.yml for PostgreSQL 16 + PostGIS
- Makefile with standard targets
- Pre-commit configuration (ruff, mypy)
- Environment template (.env.example)
- Error handling foundation (PortfolioError hierarchy)
- Test configuration (conftest.py, pytest config)

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
 2026-01-11 13:49:28 -05:00