personal-projects/personal-portfolio
Template
1
1
Fork 0
Code Issues 3 Pull Requests Actions Packages Projects 1 Releases Wiki Activity
79 Commits 3 Branches 1 Tag
941305e71c661d96968baa35ea29582255765f9d
Commit Graph

10 Commits

Author SHA1 Message Date
lmiranda
8f3c5554f9 fix: Run dbt deps before dbt run to install packages 
dbt requires packages specified in packages.yml to be installed
before running models. Added dbt deps step to the pipeline.

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
 2026-01-18 12:20:26 -05:00
lmiranda
ebe48304d7 fix: Use venv dbt and show full error output 
- Use .venv/bin/dbt if available, fall back to system dbt
- Show both stdout and stderr on dbt failures for better debugging

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
 2026-01-18 12:18:26 -05:00
lmiranda
6872aa510b fix: Use explicit path for .env file loading 
load_dotenv() was searching from cwd, which may not be the project root.
Now explicitly passes PROJECT_ROOT / ".env" path.

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
 2026-01-18 12:14:48 -05:00
lmiranda
cf6e874961 fix: Load .env file for dbt database credentials 
dbt uses env_var() in profiles.yml to read POSTGRES_PASSWORD,
but subprocess.run() doesn't automatically load .env files.
Added python-dotenv to load credentials before dbt runs.

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
 2026-01-18 12:10:46 -05:00
lmiranda
193b9289b9 fix: Configure dbt to use local profiles.yml 
- Rename profiles.yml.example to profiles.yml (uses env vars, safe to commit)
- Add --profiles-dir flag to dbt commands in load_toronto_data.py
- Add --profiles-dir flag to dbt targets in Makefile

This fixes the "Path '~/.dbt' does not exist" error when running make load-data.

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
 2026-01-18 12:06:58 -05:00
lmiranda
bf6e392002 feat: Sprint 10 - Architecture docs, CI/CD, operational scripts
Some checks failed
CI / lint-and-test (push) Has been cancelled
Details 
Phase 1 - Architecture Documentation:
- Add Architecture section with Mermaid flowchart to README
- Create docs/DATABASE_SCHEMA.md with full ERD

Phase 2 - CI/CD:
- Add CI badge to README
- Create .gitea/workflows/ci.yml for linting and tests
- Create .gitea/workflows/deploy-staging.yml
- Create .gitea/workflows/deploy-production.yml

Phase 3 - Operational Scripts:
- Create scripts/logs.sh for docker compose log following
- Create scripts/run-detached.sh with health check loop
- Create scripts/etl/toronto.sh for Toronto data pipeline
- Add Makefile targets: logs, run-detached, etl-toronto

Phase 4 - Runbooks:
- Create docs/runbooks/adding-dashboard.md
- Create docs/runbooks/deployment.md

Phase 5 - Hygiene:
- Create MIT LICENSE file

Phase 6 - Production:
- Add live demo link to README (leodata.science)

Closes #78, #79, #80, #81, #82, #83, #84, #85, #86, #87, #88, #89, #91

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
 2026-01-17 17:10:30 -05:00
lmiranda
d0f32edba7 fix: Repair data pipeline with StatCan CMHC rental data 
- Add StatCan CMHC parser to fetch rental data from Statistics Canada API
- Create year spine (2014-2025) as time dimension driver instead of census
- Add CMA-level rental and income intermediate models
- Update mart_neighbourhood_overview to use rental years as base
- Fix neighbourhood_service queries to match dbt schema
- Add CMHC data loading to pipeline script

Data now flows correctly: 158 neighbourhoods × 12 years = 1,896 records
Rent data available 2019-2025, crime data 2014-2024

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
 2026-01-17 15:38:31 -05:00
lmiranda
c9cf744d84 feat: Complete Phase 5 dashboard implementation 
Implement full 5-tab Toronto Neighbourhood Dashboard with real data
connectivity:

Dashboard Structure:
- Overview tab with livability scores and rankings
- Housing tab with affordability metrics
- Safety tab with crime statistics
- Demographics tab with population/income data
- Amenities tab with parks, schools, transit

Figure Factories (portfolio_app/figures/):
- bar_charts.py: ranking, stacked, horizontal bars
- scatter.py: scatter plots, bubble charts
- radar.py: spider/radar charts
- demographics.py: donut, age pyramid, income distribution

Service Layer (portfolio_app/toronto/services/):
- neighbourhood_service.py: queries dbt marts for all tab data
- geometry_service.py: generates GeoJSON from PostGIS
- Graceful error handling when database unavailable

Callbacks (portfolio_app/pages/toronto/callbacks/):
- map_callbacks.py: choropleth updates, map click handling
- chart_callbacks.py: supporting chart updates
- selection_callbacks.py: dropdown handlers, KPI updates

Data Pipeline (scripts/data/):
- load_toronto_data.py: orchestration script with CLI flags

Lessons Learned:
- Graceful error handling in service layers
- Modular callback structure for multi-tab dashboards
- Figure factory pattern for reusable charts

Closes: #64, #65, #66, #67, #68, #69, #70

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
 2026-01-17 11:46:18 -05:00
lmiranda
140d3085bf feat: Sprint 6 polish - methodology, demo data, deployment prep 
- Add policy event markers to time series charts
- Create methodology page (/toronto/methodology) with data sources
- Add demo data module for testing without full pipeline
- Update README with project documentation
- Add health check endpoint (/health)
- Add database initialization script
- Export new figure factory functions

Closes #21

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
 2026-01-11 21:50:45 -05:00
lmiranda
c7e9b88adb feat: project bootstrap and structure 
Sprint 1 initialization:
- Project directory structure (portfolio_app/, tests/, dbt/, data/, scripts/)
- CLAUDE.md with AI assistant context
- pyproject.toml with all dependencies
- docker-compose.yml for PostgreSQL 16 + PostGIS
- Makefile with standard targets
- Pre-commit configuration (ruff, mypy)
- Environment template (.env.example)
- Error handling foundation (PortfolioError hierarchy)
- Test configuration (conftest.py, pytest config)

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
 2026-01-11 13:49:28 -05:00