personal-projects/personal-portfolio
Template
1
1
Fork 0
Code Issues 3 Pull Requests Actions Packages Projects 1 Releases Wiki Activity
81 Commits 3 Branches 1 Tag
c3de98c4a5e9aca28c017b6cb0d9d0013c9886a9
Commit Graph

8 Commits

Author SHA1 Message Date
lmiranda
c3de98c4a5 feat: Add seed_amenity_data script for notebook testing
Some checks failed
CI / lint-and-test (push) Has been cancelled
Details 
Adds script to populate sample amenity data when Toronto Open Data
API doesn't return neighbourhood IDs (requires spatial join).

Run with: make seed-amenities

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
 2026-01-18 21:02:57 -05:00
lmiranda
8f3c5554f9 fix: Run dbt deps before dbt run to install packages 
dbt requires packages specified in packages.yml to be installed
before running models. Added dbt deps step to the pipeline.

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
 2026-01-18 12:20:26 -05:00
lmiranda
ebe48304d7 fix: Use venv dbt and show full error output 
- Use .venv/bin/dbt if available, fall back to system dbt
- Show both stdout and stderr on dbt failures for better debugging

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
 2026-01-18 12:18:26 -05:00
lmiranda
6872aa510b fix: Use explicit path for .env file loading 
load_dotenv() was searching from cwd, which may not be the project root.
Now explicitly passes PROJECT_ROOT / ".env" path.

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
 2026-01-18 12:14:48 -05:00
lmiranda
cf6e874961 fix: Load .env file for dbt database credentials 
dbt uses env_var() in profiles.yml to read POSTGRES_PASSWORD,
but subprocess.run() doesn't automatically load .env files.
Added python-dotenv to load credentials before dbt runs.

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
 2026-01-18 12:10:46 -05:00
lmiranda
193b9289b9 fix: Configure dbt to use local profiles.yml 
- Rename profiles.yml.example to profiles.yml (uses env vars, safe to commit)
- Add --profiles-dir flag to dbt commands in load_toronto_data.py
- Add --profiles-dir flag to dbt targets in Makefile

This fixes the "Path '~/.dbt' does not exist" error when running make load-data.

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
 2026-01-18 12:06:58 -05:00
lmiranda
d0f32edba7 fix: Repair data pipeline with StatCan CMHC rental data 
- Add StatCan CMHC parser to fetch rental data from Statistics Canada API
- Create year spine (2014-2025) as time dimension driver instead of census
- Add CMA-level rental and income intermediate models
- Update mart_neighbourhood_overview to use rental years as base
- Fix neighbourhood_service queries to match dbt schema
- Add CMHC data loading to pipeline script

Data now flows correctly: 158 neighbourhoods × 12 years = 1,896 records
Rent data available 2019-2025, crime data 2014-2024

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
 2026-01-17 15:38:31 -05:00
lmiranda
c9cf744d84 feat: Complete Phase 5 dashboard implementation 
Implement full 5-tab Toronto Neighbourhood Dashboard with real data
connectivity:

Dashboard Structure:
- Overview tab with livability scores and rankings
- Housing tab with affordability metrics
- Safety tab with crime statistics
- Demographics tab with population/income data
- Amenities tab with parks, schools, transit

Figure Factories (portfolio_app/figures/):
- bar_charts.py: ranking, stacked, horizontal bars
- scatter.py: scatter plots, bubble charts
- radar.py: spider/radar charts
- demographics.py: donut, age pyramid, income distribution

Service Layer (portfolio_app/toronto/services/):
- neighbourhood_service.py: queries dbt marts for all tab data
- geometry_service.py: generates GeoJSON from PostGIS
- Graceful error handling when database unavailable

Callbacks (portfolio_app/pages/toronto/callbacks/):
- map_callbacks.py: choropleth updates, map click handling
- chart_callbacks.py: supporting chart updates
- selection_callbacks.py: dropdown handlers, KPI updates

Data Pipeline (scripts/data/):
- load_toronto_data.py: orchestration script with CLI flags

Lessons Learned:
- Graceful error handling in service layers
- Modular callback structure for multi-tab dashboards
- Figure factory pattern for reusable charts

Closes: #64, #65, #66, #67, #68, #69, #70

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
 2026-01-17 11:46:18 -05:00