- Rename dbt project from toronto_housing to portfolio
- Restructure dbt models into domain subdirectories:
- shared/ for cross-domain dimensions (dim_time)
- staging/toronto/, intermediate/toronto/, marts/toronto/
- Update SQLAlchemy models for raw_toronto schema
- Add explicit cross-schema FK relationships for FactRentals
- Namespace figure factories under figures/toronto/
- Namespace notebooks under notebooks/toronto/
- Update Makefile with domain-specific targets and env loading
- Update all documentation for multi-dashboard structure
This enables adding new dashboard projects (e.g., /football, /energy)
without structural conflicts or naming collisions.
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
Fixes identified by doc-guardian audit:
Critical fixes:
- DATABASE_SCHEMA.md: Fix staging model name stg_police__crimes → stg_toronto__crime
- DATABASE_SCHEMA.md: Update mart model names to match actual dbt models
- CLAUDE.md: Fix errors/ description (no handlers module exists)
- scripts/etl/toronto.sh: Fix parser module references to actual modules
Stale fixes:
- CONTRIBUTING.md: Add make typecheck, test-cov; fix make ci description
- PROJECT_REFERENCE.md: Document services/, callback modules, all Makefile targets
- CLAUDE.md: Expand Makefile commands, add plugin documentation
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
The seed script now inserts housing data for years 2019-2024 to
support rent trend line visualizations.
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
- Seed script now seeds: amenities, population, median_age, census
housing columns, housing mart (rent/affordability), overview mart
(safety_score, population)
- Add statsmodels dependency for scatter plot trendlines
- Add dbt/.user.yml to gitignore
All 15 notebooks now pass with valid data.
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
Updates seed_amenity_data.py to also seed median_age values in
fact_census where missing, ensuring demographics notebooks work.
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
Adds script to populate sample amenity data when Toronto Open Data
API doesn't return neighbourhood IDs (requires spatial join).
Run with: make seed-amenities
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
dbt requires packages specified in packages.yml to be installed
before running models. Added dbt deps step to the pipeline.
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
- Use .venv/bin/dbt if available, fall back to system dbt
- Show both stdout and stderr on dbt failures for better debugging
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
load_dotenv() was searching from cwd, which may not be the project root.
Now explicitly passes PROJECT_ROOT / ".env" path.
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
dbt uses env_var() in profiles.yml to read POSTGRES_PASSWORD,
but subprocess.run() doesn't automatically load .env files.
Added python-dotenv to load credentials before dbt runs.
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
- Rename profiles.yml.example to profiles.yml (uses env vars, safe to commit)
- Add --profiles-dir flag to dbt commands in load_toronto_data.py
- Add --profiles-dir flag to dbt targets in Makefile
This fixes the "Path '~/.dbt' does not exist" error when running make load-data.
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
- Add StatCan CMHC parser to fetch rental data from Statistics Canada API
- Create year spine (2014-2025) as time dimension driver instead of census
- Add CMA-level rental and income intermediate models
- Update mart_neighbourhood_overview to use rental years as base
- Fix neighbourhood_service queries to match dbt schema
- Add CMHC data loading to pipeline script
Data now flows correctly: 158 neighbourhoods × 12 years = 1,896 records
Rent data available 2019-2025, crime data 2014-2024
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
- Add policy event markers to time series charts
- Create methodology page (/toronto/methodology) with data sources
- Add demo data module for testing without full pipeline
- Update README with project documentation
- Add health check endpoint (/health)
- Add database initialization script
- Export new figure factory functions
Closes#21
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
