personal-projects/personal-portfolio
Template
1
1
Fork 0
Code Issues 3 Pull Requests Actions Packages Projects 1 Releases Wiki Activity
103 Commits 3 Branches 1 Tag
9e216962b1122ed40229bd644704e0c68db56cc0
Commit Graph

9 Commits

Author SHA1 Message Date
l3ocho
62d1a52eed refactor: multi-dashboard structural migration
Some checks failed
CI / lint-and-test (pull_request) Has been cancelled
Details 
- Rename dbt project from toronto_housing to portfolio
- Restructure dbt models into domain subdirectories:
  - shared/ for cross-domain dimensions (dim_time)
  - staging/toronto/, intermediate/toronto/, marts/toronto/
- Update SQLAlchemy models for raw_toronto schema
- Add explicit cross-schema FK relationships for FactRentals
- Namespace figure factories under figures/toronto/
- Namespace notebooks under notebooks/toronto/
- Update Makefile with domain-specific targets and env loading
- Update all documentation for multi-dashboard structure

This enables adding new dashboard projects (e.g., /football, /energy)
without structural conflicts or naming collisions.

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
 2026-02-01 19:08:20 -05:00
lmiranda
546ee1cc92 fix: Include seed-data in load-data target
Some checks failed
CI / lint-and-test (push) Has been cancelled
Details 
Now `make load-data` automatically seeds development data (amenities,
median_age) after loading Toronto data. Renamed seed-amenities to
seed-data to reflect broader scope.

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
 2026-01-18 22:52:31 -05:00
lmiranda
c3de98c4a5 feat: Add seed_amenity_data script for notebook testing
Some checks failed
CI / lint-and-test (push) Has been cancelled
Details 
Adds script to populate sample amenity data when Toronto Open Data
API doesn't return neighbourhood IDs (requires spatial join).

Run with: make seed-amenities

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
 2026-01-18 21:02:57 -05:00
lmiranda
193b9289b9 fix: Configure dbt to use local profiles.yml 
- Rename profiles.yml.example to profiles.yml (uses env vars, safe to commit)
- Add --profiles-dir flag to dbt commands in load_toronto_data.py
- Add --profiles-dir flag to dbt targets in Makefile

This fixes the "Path '~/.dbt' does not exist" error when running make load-data.

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
 2026-01-18 12:06:58 -05:00
lmiranda
ecc50e5d98 fix: Update db-init target to use Python script 
The Makefile was looking for scripts/db/init.sh which doesn't exist.
Updated to call scripts/db/init_schema.py instead.

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
 2026-01-18 11:59:19 -05:00
lmiranda
bf6e392002 feat: Sprint 10 - Architecture docs, CI/CD, operational scripts
Some checks failed
CI / lint-and-test (push) Has been cancelled
Details 
Phase 1 - Architecture Documentation:
- Add Architecture section with Mermaid flowchart to README
- Create docs/DATABASE_SCHEMA.md with full ERD

Phase 2 - CI/CD:
- Add CI badge to README
- Create .gitea/workflows/ci.yml for linting and tests
- Create .gitea/workflows/deploy-staging.yml
- Create .gitea/workflows/deploy-production.yml

Phase 3 - Operational Scripts:
- Create scripts/logs.sh for docker compose log following
- Create scripts/run-detached.sh with health check loop
- Create scripts/etl/toronto.sh for Toronto data pipeline
- Add Makefile targets: logs, run-detached, etl-toronto

Phase 4 - Runbooks:
- Create docs/runbooks/adding-dashboard.md
- Create docs/runbooks/deployment.md

Phase 5 - Hygiene:
- Create MIT LICENSE file

Phase 6 - Production:
- Add live demo link to README (leodata.science)

Closes #78, #79, #80, #81, #82, #83, #84, #85, #86, #87, #88, #89, #91

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
 2026-01-17 17:10:30 -05:00
lmiranda
d0f32edba7 fix: Repair data pipeline with StatCan CMHC rental data 
- Add StatCan CMHC parser to fetch rental data from Statistics Canada API
- Create year spine (2014-2025) as time dimension driver instead of census
- Add CMA-level rental and income intermediate models
- Update mart_neighbourhood_overview to use rental years as base
- Fix neighbourhood_service queries to match dbt schema
- Add CMHC data loading to pipeline script

Data now flows correctly: 158 neighbourhoods × 12 years = 1,896 records
Rent data available 2019-2025, crime data 2014-2024

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
 2026-01-17 15:38:31 -05:00
lmiranda
c9cf744d84 feat: Complete Phase 5 dashboard implementation 
Implement full 5-tab Toronto Neighbourhood Dashboard with real data
connectivity:

Dashboard Structure:
- Overview tab with livability scores and rankings
- Housing tab with affordability metrics
- Safety tab with crime statistics
- Demographics tab with population/income data
- Amenities tab with parks, schools, transit

Figure Factories (portfolio_app/figures/):
- bar_charts.py: ranking, stacked, horizontal bars
- scatter.py: scatter plots, bubble charts
- radar.py: spider/radar charts
- demographics.py: donut, age pyramid, income distribution

Service Layer (portfolio_app/toronto/services/):
- neighbourhood_service.py: queries dbt marts for all tab data
- geometry_service.py: generates GeoJSON from PostGIS
- Graceful error handling when database unavailable

Callbacks (portfolio_app/pages/toronto/callbacks/):
- map_callbacks.py: choropleth updates, map click handling
- chart_callbacks.py: supporting chart updates
- selection_callbacks.py: dropdown handlers, KPI updates

Data Pipeline (scripts/data/):
- load_toronto_data.py: orchestration script with CLI flags

Lessons Learned:
- Graceful error handling in service layers
- Modular callback structure for multi-tab dashboards
- Figure factory pattern for reusable charts

Closes: #64, #65, #66, #67, #68, #69, #70

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
 2026-01-17 11:46:18 -05:00
lmiranda
c7e9b88adb feat: project bootstrap and structure 
Sprint 1 initialization:
- Project directory structure (portfolio_app/, tests/, dbt/, data/, scripts/)
- CLAUDE.md with AI assistant context
- pyproject.toml with all dependencies
- docker-compose.yml for PostgreSQL 16 + PostGIS
- Makefile with standard targets
- Pre-commit configuration (ruff, mypy)
- Environment template (.env.example)
- Error handling foundation (PortfolioError hierarchy)
- Test configuration (conftest.py, pytest config)

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
 2026-01-11 13:49:28 -05:00