50 Commits

Author SHA1 Message Date
a0417182a7 Merge pull request 'development' (#107) from development into staging
Some checks failed
CI / lint-and-test (push) Has been cancelled
Deploy to Staging / deploy (push) Has been cancelled
Reviewed-on: #107
2026-02-02 22:03:06 +00:00
058d058975 Merge branch 'staging' into development
Some checks failed
CI / lint-and-test (push) Has been cancelled
2026-02-02 22:02:57 +00:00
0455ec69a0 Merge branch 'main' into development
Some checks failed
CI / lint-and-test (push) Has been cancelled
2026-02-02 22:02:26 +00:00
9e216962b1 Merge pull request 'refactor: domain-scoped schema migration for application code' (#104) from feature/domain-scoped-schema-migration into development
Some checks failed
CI / lint-and-test (push) Has been cancelled
Reviewed-on: #104
2026-02-02 22:01:48 +00:00
dfa5f92d8a refactor: update app code for domain-scoped schema migration
Some checks failed
CI / lint-and-test (pull_request) Has been cancelled
- Update dbt model references to use new schema naming (stg_toronto, int_toronto, mart_toronto)
- Refactor figure factories to use consistent column naming from new schema
- Update callbacks to work with refactored data structures
- Add centralized design tokens module for consistent styling
- Streamline CLAUDE.md documentation

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
2026-02-02 17:00:30 -05:00
3bd2005c9d Merge pull request 'Merge pull request 'development' (#99) from development into main' (#102) from main into staging
Some checks failed
CI / lint-and-test (push) Has been cancelled
Deploy to Staging / deploy (push) Has been cancelled
Reviewed-on: #102
2026-02-02 17:35:06 +00:00
0c9769fd27 Merge branch 'staging' into main
Some checks failed
CI / lint-and-test (push) Has been cancelled
Deploy to Production / deploy (push) Has been cancelled
CI / lint-and-test (pull_request) Has been cancelled
2026-02-02 17:34:58 +00:00
cb908a18c3 Merge pull request 'Merge pull request 'development' (#98) from development into staging' (#101) from staging into development
Some checks failed
CI / lint-and-test (push) Has been cancelled
Reviewed-on: #101
2026-02-02 17:34:27 +00:00
558022f26e Merge branch 'development' into staging
Some checks failed
CI / lint-and-test (push) Has been cancelled
Deploy to Staging / deploy (push) Has been cancelled
CI / lint-and-test (pull_request) Has been cancelled
2026-02-02 17:34:14 +00:00
9e27fb8011 Merge pull request 'refactor(dbt): migrate to domain-scoped schema names' (#100) from feature/domain-scoped-schema-migration into development
Some checks failed
CI / lint-and-test (push) Has been cancelled
Reviewed-on: #100
2026-02-02 17:33:40 +00:00
cda2a078d9 refactor(dbt): migrate to domain-scoped schema names
Some checks failed
CI / lint-and-test (pull_request) Has been cancelled
- Create generate_schema_name macro to use custom schema names directly
- Update dbt_project.yml schemas: staging→stg_toronto, intermediate→int_toronto, marts→mart_toronto
- Add dbt/macros/toronto/ directory for future domain-specific macros
- Fix documentation drift in PROJECT_REFERENCE.md (load-data-only→load-toronto-only)
- Update DATABASE_SCHEMA.md with new schema names
- Update CLAUDE.md database schemas table
- Update adding-dashboard.md runbook with domain-scoped pattern

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
2026-02-02 12:32:39 -05:00
dd8de9810d Merge pull request 'development' (#99) from development into main
Some checks failed
CI / lint-and-test (push) Has been cancelled
Deploy to Production / deploy (push) Has been cancelled
Reviewed-on: #99
2026-02-02 00:39:19 +00:00
56bcc1bb1d Merge branch 'main' into development
Some checks failed
CI / lint-and-test (push) Has been cancelled
2026-02-02 00:39:13 +00:00
ee0a7ef7ad Merge pull request 'development' (#98) from development into staging
Some checks failed
CI / lint-and-test (push) Has been cancelled
Deploy to Staging / deploy (push) Has been cancelled
CI / lint-and-test (pull_request) Has been cancelled
Reviewed-on: #98
2026-02-02 00:19:29 +00:00
fd9850778e Merge branch 'staging' into development
Some checks failed
CI / lint-and-test (push) Has been cancelled
2026-02-02 00:19:24 +00:00
01e98103c7 Merge pull request 'refactor: multi-dashboard structural migration' (#97) from feature/multi-dashboard-structure into development
Some checks failed
CI / lint-and-test (push) Has been cancelled
Reviewed-on: #97
2026-02-02 00:18:45 +00:00
62d1a52eed refactor: multi-dashboard structural migration
Some checks failed
CI / lint-and-test (pull_request) Has been cancelled
- Rename dbt project from toronto_housing to portfolio
- Restructure dbt models into domain subdirectories:
  - shared/ for cross-domain dimensions (dim_time)
  - staging/toronto/, intermediate/toronto/, marts/toronto/
- Update SQLAlchemy models for raw_toronto schema
- Add explicit cross-schema FK relationships for FactRentals
- Namespace figure factories under figures/toronto/
- Namespace notebooks under notebooks/toronto/
- Update Makefile with domain-specific targets and env loading
- Update all documentation for multi-dashboard structure

This enables adding new dashboard projects (e.g., /football, /energy)
without structural conflicts or naming collisions.

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
2026-02-01 19:08:20 -05:00
e37611673f Merge pull request 'staging' (#96) from staging into main
Some checks failed
CI / lint-and-test (push) Has been cancelled
Deploy to Production / deploy (push) Has been cancelled
Reviewed-on: #96
2026-02-01 21:33:12 +00:00
33306a911b Merge pull request 'development' (#95) from development into staging
Some checks failed
CI / lint-and-test (push) Has been cancelled
Deploy to Staging / deploy (push) Has been cancelled
Reviewed-on: #95
2026-02-01 21:32:41 +00:00
a5d6866d63 feat(contact): implement Formspree contact form submission
Some checks failed
CI / lint-and-test (push) Has been cancelled
- Enable contact form fields with component IDs
- Add callback for Formspree POST with JSON/AJAX
- Include honeypot spam protection (_gotcha field)
- Handle validation, loading, success/error states
- Clear form on successful submission
- Add lessons learned documentation

Closes #92, #93, #94

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
2026-02-01 15:00:04 -05:00
f58b2f70e2 chore(vscode): add workspace settings
Configure Python interpreter path for VSCode.

Co-Authored-By: Claude <noreply@anthropic.com>
2026-02-01 15:00:04 -05:00
263b52d5e4 docs: sync documentation with codebase
Some checks failed
CI / lint-and-test (push) Has been cancelled
Fixes identified by doc-guardian audit:

Critical fixes:
- DATABASE_SCHEMA.md: Fix staging model name stg_police__crimes → stg_toronto__crime
- DATABASE_SCHEMA.md: Update mart model names to match actual dbt models
- CLAUDE.md: Fix errors/ description (no handlers module exists)
- scripts/etl/toronto.sh: Fix parser module references to actual modules

Stale fixes:
- CONTRIBUTING.md: Add make typecheck, test-cov; fix make ci description
- PROJECT_REFERENCE.md: Document services/, callback modules, all Makefile targets
- CLAUDE.md: Expand Makefile commands, add plugin documentation

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
2026-01-22 16:25:29 -05:00
f345d41535 fix: Seed multi-year housing data for rent trend charts
Some checks failed
CI / lint-and-test (push) Has been cancelled
The seed script now inserts housing data for years 2019-2024 to
support rent trend line visualizations.

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
2026-01-18 23:44:08 -05:00
14701f334c fix: Complete seed script with all missing data + add statsmodels
Some checks failed
CI / lint-and-test (push) Has been cancelled
- Seed script now seeds: amenities, population, median_age, census
  housing columns, housing mart (rent/affordability), overview mart
  (safety_score, population)
- Add statsmodels dependency for scatter plot trendlines
- Add dbt/.user.yml to gitignore

All 15 notebooks now pass with valid data.

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
2026-01-18 23:21:14 -05:00
92763a17c4 fix: Use os.environ[] instead of .get() for DATABASE_URL
Some checks failed
CI / lint-and-test (push) Has been cancelled
Fixes Pylance type error - create_engine() expects str, not str | None.
Using direct access raises KeyError if not set, which is correct behavior.

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
2026-01-18 23:03:23 -05:00
546ee1cc92 fix: Include seed-data in load-data target
Some checks failed
CI / lint-and-test (push) Has been cancelled
Now `make load-data` automatically seeds development data (amenities,
median_age) after loading Toronto data. Renamed seed-amenities to
seed-data to reflect broader scope.

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
2026-01-18 22:52:31 -05:00
9cc2cf0e00 fix: Add median_age seeding to development data script
Some checks failed
CI / lint-and-test (push) Has been cancelled
Updates seed_amenity_data.py to also seed median_age values in
fact_census where missing, ensuring demographics notebooks work.

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
2026-01-18 22:49:57 -05:00
28f239e8cd fix: Update all notebooks to load .env for database credentials
Some checks failed
CI / lint-and-test (push) Has been cancelled
All 15 notebooks now use load_dotenv('../../.env') instead of
hardcoded fallback credentials.

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
2026-01-18 22:31:07 -05:00
c3de98c4a5 feat: Add seed_amenity_data script for notebook testing
Some checks failed
CI / lint-and-test (push) Has been cancelled
Adds script to populate sample amenity data when Toronto Open Data
API doesn't return neighbourhood IDs (requires spatial join).

Run with: make seed-amenities

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
2026-01-18 21:02:57 -05:00
eee015efac fix: Load .env in amenity_radar notebook for database credentials
Some checks failed
CI / lint-and-test (push) Has been cancelled
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
2026-01-18 20:43:43 -05:00
941305e71c fix: Update amenity_radar notebook to use correct radar API
Some checks failed
CI / lint-and-test (push) Has been cancelled
Use create_comparison_radar instead of create_radar_figure with
incorrect parameters.

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
2026-01-18 20:39:20 -05:00
54665bac63 revert: Remove unauthorized branch workflow instructions
Some checks failed
CI / lint-and-test (push) Has been cancelled
Removes instructions that were added without user authorization:
- Step about deleting feature branches after merge
- CRITICAL warning about development branch

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
2026-01-18 20:36:09 -05:00
3eb32a4766 Merge feature/fix-notebook-schema into development
Some checks failed
CI / lint-and-test (push) Has been cancelled
2026-01-18 19:45:25 -05:00
69c4216cd5 fix: Update notebooks to use public_marts schema
dbt creates mart tables in public_marts schema, not public.
Updated all notebook SQL queries to use the correct schema.

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
2026-01-18 19:45:23 -05:00
6e00a17c05 Merge feature/add-dbt-deps into development
Some checks failed
CI / lint-and-test (push) Has been cancelled
2026-01-18 12:20:38 -05:00
8f3c5554f9 fix: Run dbt deps before dbt run to install packages
dbt requires packages specified in packages.yml to be installed
before running models. Added dbt deps step to the pipeline.

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
2026-01-18 12:20:26 -05:00
5839eabf1e Merge feature/fix-dbt-venv-path into development
Some checks failed
CI / lint-and-test (push) Has been cancelled
2026-01-18 12:18:28 -05:00
ebe48304d7 fix: Use venv dbt and show full error output
- Use .venv/bin/dbt if available, fall back to system dbt
- Show both stdout and stderr on dbt failures for better debugging

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
2026-01-18 12:18:26 -05:00
2fc2a1bdb5 Merge feature/fix-dotenv-path into development
Some checks failed
CI / lint-and-test (push) Has been cancelled
2026-01-18 12:15:00 -05:00
6872aa510b fix: Use explicit path for .env file loading
load_dotenv() was searching from cwd, which may not be the project root.
Now explicitly passes PROJECT_ROOT / ".env" path.

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
2026-01-18 12:14:48 -05:00
9a1fc81f79 Merge feature/fix-dbt-env-vars into development
Some checks failed
CI / lint-and-test (push) Has been cancelled
2026-01-18 12:10:58 -05:00
cf6e874961 fix: Load .env file for dbt database credentials
dbt uses env_var() in profiles.yml to read POSTGRES_PASSWORD,
but subprocess.run() doesn't automatically load .env files.
Added python-dotenv to load credentials before dbt runs.

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
2026-01-18 12:10:46 -05:00
451dc10a10 Merge feature/fix-dbt-profiles into development
Some checks failed
CI / lint-and-test (push) Has been cancelled
2026-01-18 12:07:10 -05:00
193b9289b9 fix: Configure dbt to use local profiles.yml
- Rename profiles.yml.example to profiles.yml (uses env vars, safe to commit)
- Add --profiles-dir flag to dbt commands in load_toronto_data.py
- Add --profiles-dir flag to dbt targets in Makefile

This fixes the "Path '~/.dbt' does not exist" error when running make load-data.

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
2026-01-18 12:06:58 -05:00
7a16e6d121 Merge feature/fix-db-init-makefile into development
Some checks failed
CI / lint-and-test (push) Has been cancelled
2026-01-18 11:59:20 -05:00
ecc50e5d98 fix: Update db-init target to use Python script
The Makefile was looking for scripts/db/init.sh which doesn't exist.
Updated to call scripts/db/init_schema.py instead.

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
2026-01-18 11:59:19 -05:00
ae3742630e Merge feature/add-jupyter-dependency into development
Some checks failed
CI / lint-and-test (push) Has been cancelled
2026-01-18 11:25:20 -05:00
e70965b429 fix: Add jupyter and ipykernel to dev dependencies
Required to run the notebooks in notebooks/ directory.

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
2026-01-18 11:25:19 -05:00
25954f17bb Merge feature/add-pyproj-dependency into development
Some checks failed
CI / lint-and-test (push) Has been cancelled
2026-01-18 11:20:47 -05:00
bffd44a5a5 fix: Add pyproj as explicit dependency
pyproj is directly imported in portfolio_app/toronto/parsers/geo.py
but was only available as a transitive dependency of geopandas.
Adding it explicitly ensures reliable installation.

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
2026-01-18 11:20:40 -05:00
90 changed files with 2470 additions and 989 deletions

1
.gitignore vendored
View File

@@ -198,3 +198,4 @@ cython_debug/
# PyPI configuration file
.pypirc
dbt/.user.yml

3
.vscode/settings.json vendored Normal file
View File

@@ -0,0 +1,3 @@
{
"python.defaultInterpreterPath": "/home/leomiranda/WorkDev/personal/personal-portfolio/.venv/bin/python"
}

384
CLAUDE.md
View File

@@ -1,5 +1,48 @@
# CLAUDE.md
## ⛔ MANDATORY BEHAVIOR RULES - READ FIRST
**These rules are NON-NEGOTIABLE. Violating them wastes the user's time and money.**
### 1. WHEN USER ASKS YOU TO CHECK SOMETHING - CHECK EVERYTHING
- Search ALL locations, not just where you think it is
- Check cache directories: `~/.claude/plugins/cache/`
- Check installed: `~/.claude/plugins/marketplaces/`
- Check source directories
- **NEVER say "no" or "that's not the issue" without exhaustive verification**
### 2. WHEN USER SAYS SOMETHING IS WRONG - BELIEVE THEM
- The user knows their system better than you
- Investigate thoroughly before disagreeing
- **Your confidence is often wrong. User's instincts are often right.**
### 3. NEVER SAY "DONE" WITHOUT VERIFICATION
- Run the actual command/script to verify
- Show the output to the user
- **"Done" means VERIFIED WORKING, not "I made changes"**
### 4. SHOW EXACTLY WHAT USER ASKS FOR
- If user asks for messages, show the MESSAGES
- If user asks for code, show the CODE
- **Do not interpret or summarize unless asked**
**FAILURE TO FOLLOW THESE RULES = WASTED USER TIME = UNACCEPTABLE**
---
## Mandatory Behavior Rules
**These rules are NON-NEGOTIABLE. Violating them wastes the user's time and money.**
1. **CHECK EVERYTHING** - Search ALL locations before saying "no" (cache, installed, source directories)
2. **BELIEVE THE USER** - Investigate thoroughly before disagreeing; user instincts are often right
3. **VERIFY BEFORE "DONE"** - Run commands, show output; "done" means verified working
4. **SHOW EXACTLY WHAT'S ASKED** - Do not interpret or summarize unless requested
---
Working context for Claude Code on the Analytics Portfolio project.
---
@@ -17,15 +60,33 @@ Working context for Claude Code on the Analytics Portfolio project.
### Run Commands
```bash
# Setup & Database
make setup # Install deps, create .env, init pre-commit
make docker-up # Start PostgreSQL + PostGIS (auto-detects x86/ARM)
make docker-down # Stop containers
make db-init # Initialize database schema
make db-reset # Drop and recreate database (DESTRUCTIVE)
# Data Loading
make load-data # Load all project data (currently: Toronto)
make load-toronto # Load Toronto data from APIs
# Application
make run # Start Dash dev server
# Testing & Quality
make test # Run pytest
make lint # Run ruff linter
make format # Run ruff formatter
make ci # Run all checks
make typecheck # Run mypy type checker
make ci # Run all checks (lint, typecheck, test)
# dbt
make dbt-run # Run dbt models
make dbt-test # Run dbt tests
make dbt-docs # Generate and serve dbt documentation
# Run `make help` for full target list
```
### Branch Workflow
@@ -33,10 +94,7 @@ make ci # Run all checks
1. Create feature branch FROM `development`: `git checkout -b feature/{sprint}-{description}`
2. Work and commit on feature branch
3. Merge INTO `development` when complete
4. Delete the feature branch after merge (keep branches clean)
5. `development` -> `staging` -> `main` for releases
**CRITICAL: NEVER DELETE the `development` branch. It is the main integration branch.**
4. `development` -> `staging` -> `main` for releases
---
@@ -52,125 +110,44 @@ make ci # Run all checks
### Module Responsibilities
| Directory | Contains | Purpose |
|-----------|----------|---------|
| `schemas/` | Pydantic models | Data validation |
| `models/` | SQLAlchemy ORM | Database persistence |
| `parsers/` | API/CSV extraction | Raw data ingestion |
| `loaders/` | Database operations | Data loading |
| `figures/` | Chart factories | Plotly figure generation |
| `callbacks/` | Dash callbacks | In `pages/{dashboard}/callbacks/` |
| `errors/` | Exceptions + handlers | Error handling |
### Type Hints
Use Python 3.10+ style:
```python
def process(items: list[str], config: dict[str, int] | None = None) -> bool:
...
```
### Error Handling
```python
# errors/exceptions.py
class PortfolioError(Exception):
"""Base exception."""
class ParseError(PortfolioError):
"""PDF/CSV parsing failed."""
class ValidationError(PortfolioError):
"""Pydantic or business rule validation failed."""
class LoadError(PortfolioError):
"""Database load operation failed."""
```
| Directory | Purpose |
|-----------|---------|
| `schemas/` | Pydantic models for data validation |
| `models/` | SQLAlchemy ORM for database persistence |
| `parsers/` | API/CSV extraction for raw data ingestion |
| `loaders/` | Database operations for data loading |
| `services/` | Query functions for dbt mart queries |
| `figures/` | Chart factories for Plotly figure generation |
| `errors/` | Custom exception classes (see `errors/exceptions.py`) |
### Code Standards
- Python 3.10+ type hints: `list[str]`, `dict[str, int] | None`
- Single responsibility functions with verb naming
- Early returns over deep nesting
- Google-style docstrings only for non-obvious behavior
- Module-level constants for magic values
- Pydantic BaseSettings for runtime config
---
## Application Structure
```
portfolio_app/
├── app.py # Dash app factory with Pages routing
├── config.py # Pydantic BaseSettings
├── assets/ # CSS, images (auto-served)
│ └── sidebar.css # Navigation styling
├── callbacks/ # Global callbacks
│ ├── sidebar.py # Sidebar toggle
│ └── theme.py # Dark/light theme
├── pages/
│ ├── home.py # Bio landing page -> /
│ ├── about.py # About page -> /about
│ ├── contact.py # Contact form -> /contact
│ ├── health.py # Health endpoint -> /health
│ ├── projects.py # Project showcase -> /projects
│ ├── resume.py # Resume/CV -> /resume
│ ├── blog/
│ │ ├── index.py # Blog listing -> /blog
│ │ └── article.py # Blog article -> /blog/{slug}
│ └── toronto/
│ ├── dashboard.py # Dashboard -> /toronto
│ ├── methodology.py # Methodology -> /toronto/methodology
│ ├── tabs/ # 5 tab layouts (overview, housing, safety, demographics, amenities)
│ └── callbacks/ # Dashboard interactions
├── components/ # Shared UI (sidebar, cards, controls)
│ ├── metric_card.py # KPI card component
│ ├── map_controls.py # Map control panel
│ ├── sidebar.py # Navigation sidebar
│ └── time_slider.py # Time range selector
├── figures/ # Shared chart factories
│ ├── choropleth.py # Map visualizations
│ ├── bar_charts.py # Ranking, stacked, horizontal bars
│ ├── scatter.py # Scatter and bubble plots
│ ├── radar.py # Radar/spider charts
│ ├── demographics.py # Age pyramids, donut charts
│ ├── time_series.py # Trend lines
│ └── summary_cards.py # KPI figures
├── content/ # Markdown content
│ └── blog/ # Blog articles
├── toronto/ # Toronto data logic
│ ├── parsers/
│ ├── loaders/
│ ├── schemas/ # Pydantic
│ ├── models/ # SQLAlchemy
│ └── demo_data.py # Sample data
├── utils/ # Utilities
│ └── markdown_loader.py # Markdown processing
└── errors/
**Entry Point:** `portfolio_app/app.py` (Dash app factory with Pages routing)
notebooks/ # Data documentation (Phase 6)
├── README.md # Template and usage guide
├── overview/ # Overview tab notebooks (3)
├── housing/ # Housing tab notebooks (3)
├── safety/ # Safety tab notebooks (3)
├── demographics/ # Demographics tab notebooks (3)
└── amenities/ # Amenities tab notebooks (3)
```
| Directory | Purpose |
|-----------|---------|
| `pages/` | Dash Pages (file-based routing) |
| `pages/toronto/` | Toronto Dashboard (`tabs/` for layouts, `callbacks/` for interactions) |
| `components/` | Shared UI components |
| `figures/toronto/` | Toronto chart factories |
| `toronto/` | Toronto data logic (parsers, loaders, schemas, models) |
### URL Routing
**Key URLs:** `/` (home), `/toronto` (dashboard), `/blog` (listing), `/blog/{slug}` (articles), `/health` (status)
| URL | Page | Sprint |
|-----|------|--------|
| `/` | Bio landing page | 2 |
| `/about` | About page | 8 |
| `/contact` | Contact form | 8 |
| `/health` | Health endpoint | 8 |
| `/projects` | Project showcase | 8 |
| `/resume` | Resume/CV | 8 |
| `/blog` | Blog listing | 8 |
| `/blog/{slug}` | Blog article | 8 |
| `/toronto` | Toronto Dashboard | 6 |
| `/toronto/methodology` | Dashboard methodology | 6 |
### Multi-Dashboard Architecture
- **figures/**: Domain-namespaced (`figures/toronto/`, future: `figures/football/`)
- **dbt models**: Domain subdirectories (`staging/toronto/`, `marts/toronto/`)
- **Database schemas**: Domain-specific raw data (`raw_toronto`, future: `raw_football`)
---
@@ -182,44 +159,31 @@ notebooks/ # Data documentation (Phase 6)
| Validation | Pydantic | >=2.0 |
| ORM | SQLAlchemy | >=2.0 (2.0-style API only) |
| Transformation | dbt-postgres | >=1.7 |
| Data Processing | Pandas | >=2.1 |
| Visualization | Dash + Plotly + dash-mantine-components | >=2.14 |
| Geospatial | GeoPandas + Shapely | >=0.14 |
| Visualization | Dash + Plotly | >=2.14 |
| UI Components | dash-mantine-components | Latest stable |
| Testing | pytest | >=7.0 |
| Python | 3.11+ | Via pyenv |
**Notes**:
- SQLAlchemy 2.0 + Pydantic 2.0 only (never mix 1.x APIs)
- PostGIS extension required in database
- Docker Compose V2 format (no `version` field)
- **Multi-architecture support**: `make docker-up` auto-detects CPU architecture and uses the appropriate PostGIS image (x86_64: `postgis/postgis`, ARM64: `imresamu/postgis`)
**Notes**: SQLAlchemy 2.0 + Pydantic 2.0 only. Docker Compose V2 format (no `version` field).
---
## Data Model Overview
### Geographic Reality (Toronto Housing)
### Database Schemas
```
City Neighbourhoods (158) - Primary geographic unit for analysis
CMHC Zones (~20) - Rental data (Census Tract aligned)
```
| Schema | Purpose |
|--------|---------|
| `public` | Shared dimensions (dim_time) |
| `raw_toronto` | Toronto-specific raw/dimension tables |
| `stg_toronto` | Toronto dbt staging views |
| `int_toronto` | Toronto dbt intermediate views |
| `mart_toronto` | Toronto dbt mart tables |
### Star Schema
| Table | Type | Keys |
|-------|------|------|
| `fact_rentals` | Fact | -> dim_time, dim_cmhc_zone |
| `dim_time` | Dimension | date_key (PK) |
| `dim_cmhc_zone` | Dimension | zone_key (PK), geometry |
| `dim_neighbourhood` | Dimension | neighbourhood_id (PK), geometry |
| `dim_policy_event` | Dimension | event_id (PK) |
### dbt Layers
### dbt Project: `portfolio`
| Layer | Naming | Purpose |
|-------|--------|---------|
| Shared | `stg_dimensions__*` | Cross-domain dimensions |
| Staging | `stg_{source}__{entity}` | 1:1 source, cleaned, typed |
| Intermediate | `int_{domain}__{transform}` | Business logic |
| Marts | `mart_{domain}` | Final analytical tables |
@@ -228,13 +192,12 @@ CMHC Zones (~20) - Rental data (Census Tract aligned)
## Deferred Features
**Stop and flag if a task seems to require these**:
**Stop and flag if a task requires these**:
| Feature | Reason |
|---------|--------|
| Historical boundary reconciliation (140->158) | 2021+ data only for V1 |
| ML prediction models | Energy project scope (future phase) |
| Multi-project shared infrastructure | Build first, abstract second |
---
@@ -254,92 +217,123 @@ LOG_LEVEL=INFO
---
## Script Standards
All scripts in `scripts/`:
- Include usage comments at top
- Idempotent where possible
- Exit codes: 0 = success, 1 = error
- Use `set -euo pipefail` for bash
- Log to stdout, errors to stderr
---
## Reference Documents
| Document | Location | Use When |
|----------|----------|----------|
| Project reference | `docs/PROJECT_REFERENCE.md` | Architecture decisions, completed work |
| Developer guide | `docs/CONTRIBUTING.md` | How to add pages, blog posts, tabs |
| Project reference | `docs/PROJECT_REFERENCE.md` | Architecture decisions |
| Developer guide | `docs/CONTRIBUTING.md` | How to add pages, tabs |
| Lessons learned | `docs/project-lessons-learned/INDEX.md` | Past issues and solutions |
| Deployment runbook | `docs/runbooks/deployment.md` | Deploying to environments |
---
## Projman Plugin Workflow
## Plugin Reference
**CRITICAL: Always use the projman plugin for sprint and task management.**
### Sprint Management: projman
### When to Use Projman Skills
**CRITICAL: Always use projman for sprint and task management.**
| Skill | Trigger | Purpose |
|-------|---------|---------|
| `/projman:sprint-plan` | New sprint or phase implementation | Architecture analysis + Gitea issue creation |
| `/projman:sprint-start` | Beginning implementation work | Load lessons learned (Wiki.js or local), start execution |
| `/projman:sprint-status` | Check progress | Review blockers and completion status |
| `/projman:sprint-close` | Sprint completion | Capture lessons learned (Wiki.js or local backup) |
| `/projman:sprint-plan` | New sprint/feature | Architecture analysis + Gitea issue creation |
| `/projman:sprint-start` | Begin implementation | Load lessons learned, start execution |
| `/projman:sprint-status` | Check progress | Review blockers and completion |
| `/projman:sprint-close` | Sprint completion | Capture lessons learned |
### Default Behavior
**Default workflow**: `/projman:sprint-plan` before code -> create issues -> `/projman:sprint-start` -> track via Gitea -> `/projman:sprint-close`
When user requests implementation work:
**Gitea**: `personal-projects/personal-portfolio` at `gitea.hotserv.cloud`
1. **ALWAYS start with `/projman:sprint-plan`** before writing code
2. Create Gitea issues with proper labels and acceptance criteria
3. Use `/projman:sprint-start` to begin execution with lessons learned
4. Track progress via Gitea issue comments
5. Close sprint with `/projman:sprint-close` to document lessons
### Data Platform: data-platform
### Gitea Repository
Use for dbt, PostgreSQL, and PostGIS operations.
- **Repo**: `lmiranda/personal-portfolio`
- **Host**: `gitea.hotserv.cloud`
- **Note**: `lmiranda` is a user account (not org), so label lookup may require repo-level labels
| Skill | Purpose |
|-------|---------|
| `/data-platform:data-review` | Audit data integrity, schema validity, dbt compliance |
| `/data-platform:data-gate` | CI/CD data quality gate (pass/fail) |
### MCP Tools Available
**When to use:** Schema changes, dbt model development, data loading, before merging data PRs.
**Gitea**:
- `list_issues`, `get_issue`, `create_issue`, `update_issue`, `add_comment`
- `get_labels`, `suggest_labels`
**MCP tools available:** `pg_connect`, `pg_query`, `pg_tables`, `pg_columns`, `pg_schemas`, `st_*` (PostGIS), `dbt_*` operations.
**Wiki.js**:
- `search_lessons`, `create_lesson`, `search_pages`, `get_page`
### Visualization: viz-platform
### Lessons Learned (Backup Method)
Use for Dash/Mantine component validation and chart creation.
**When Wiki.js is unavailable**, use the local backup in `docs/project-lessons-learned/`:
| Skill | Purpose |
|-------|---------|
| `/viz-platform:component` | Inspect DMC component props and validation |
| `/viz-platform:chart` | Create themed Plotly charts |
| `/viz-platform:theme` | Apply/validate themes |
| `/viz-platform:dashboard` | Create dashboard layouts |
**At Sprint Start:**
1. Review `docs/project-lessons-learned/INDEX.md` for relevant past lessons
2. Search lesson files by tags/keywords before implementation
3. Apply prevention strategies from applicable lessons
**When to use:** Dashboard development, new visualizations, component prop lookup.
**At Sprint Close:**
1. Try Wiki.js `create_lesson` first
2. If Wiki.js fails, create lesson in `docs/project-lessons-learned/`
3. Use naming convention: `{phase-or-sprint}-{short-description}.md`
4. Update `INDEX.md` with new entry
5. Follow the lesson template in INDEX.md
### Code Quality: code-sentinel
**Migration:** Once Wiki.js is configured, lessons will be migrated there for better searchability.
Use for security scanning and refactoring analysis.
### Issue Structure
| Skill | Purpose |
|-------|---------|
| `/code-sentinel:security-scan` | Full security audit of codebase |
| `/code-sentinel:refactor` | Apply refactoring patterns |
| `/code-sentinel:refactor-dry` | Preview refactoring without applying |
Every Gitea issue should include:
- **Overview**: Brief description
- **Files to Create/Modify**: Explicit paths
- **Acceptance Criteria**: Checkboxes
- **Technical Notes**: Implementation hints
- **Labels**: Listed in body (workaround for label API issues)
**When to use:** Before major releases, after adding auth/data handling code, periodic audits.
### Documentation: doc-guardian
Use for documentation drift detection and synchronization.
| Skill | Purpose |
|-------|---------|
| `/doc-guardian:doc-audit` | Scan project for documentation drift |
| `/doc-guardian:doc-sync` | Synchronize pending documentation updates |
**When to use:** After significant code changes, before releases.
### Pull Requests: pr-review
Use for comprehensive PR review with multiple analysis perspectives.
| Skill | Purpose |
|-------|---------|
| `/pr-review:initial-setup` | Configure PR review for project |
| Triggered automatically | Security, performance, maintainability, test analysis |
**When to use:** Before merging significant PRs to `development` or `main`.
### Requirement Clarification: clarity-assist
Use when requirements are ambiguous or need decomposition.
**When to use:** Unclear specifications, complex feature requests, conflicting requirements.
### Contract Validation: contract-validator
Use for plugin interface validation.
| Skill | Purpose |
|-------|---------|
| `/contract-validator:agent-check` | Quick agent definition validation |
| `/contract-validator:full-validation` | Full plugin contract validation |
**When to use:** When modifying plugin integrations or agent definitions.
### Git Workflow: git-flow
Use for standardized git operations.
| Skill | Purpose |
|-------|---------|
| `/git-flow:commit` | Auto-generated conventional commit |
| `/git-flow:branch-start` | Create feature/fix/chore branch |
| `/git-flow:git-status` | Comprehensive status with recommendations |
**When to use:** Complex merge scenarios, branch management, standardized commits.
---
*Last Updated: January 2026 (Post-Sprint 9)*
*Last Updated: February 2026*

View File

@@ -1,11 +1,12 @@
.PHONY: setup docker-up docker-down db-init load-data run test dbt-run dbt-test lint format ci deploy clean help logs run-detached etl-toronto
.PHONY: setup docker-up docker-down db-init load-data load-all load-toronto load-toronto-only seed-data run test dbt-run dbt-test lint format ci deploy clean help logs run-detached etl-toronto
# Default target
.DEFAULT_GOAL := help
# Environment
PYTHON := python3
PIP := pip
VENV := .venv
PYTHON := $(VENV)/bin/python3
PIP := $(VENV)/bin/pip
DOCKER_COMPOSE := docker compose
# Architecture detection for Docker images
@@ -69,11 +70,7 @@ docker-logs: ## View container logs
db-init: ## Initialize database schema
@echo "$(GREEN)Initializing database schema...$(NC)"
@if [ -f scripts/db/init.sh ]; then \
bash scripts/db/init.sh; \
else \
echo "$(YELLOW)scripts/db/init.sh not found - skipping$(NC)"; \
fi
$(PYTHON) scripts/db/init_schema.py
db-reset: ## Drop and recreate database (DESTRUCTIVE)
@echo "$(YELLOW)WARNING: This will delete all data!$(NC)"
@@ -83,14 +80,27 @@ db-reset: ## Drop and recreate database (DESTRUCTIVE)
@sleep 3
$(MAKE) db-init
load-data: ## Load Toronto data from APIs and run dbt
# Domain-specific data loading
load-toronto: ## Load Toronto data from APIs
@echo "$(GREEN)Loading Toronto neighbourhood data...$(NC)"
$(PYTHON) scripts/data/load_toronto_data.py
@echo "$(GREEN)Seeding Toronto development data...$(NC)"
$(PYTHON) scripts/data/seed_amenity_data.py
load-data-only: ## Load Toronto data without running dbt
load-toronto-only: ## Load Toronto data without running dbt or seeding
@echo "$(GREEN)Loading Toronto data (skip dbt)...$(NC)"
$(PYTHON) scripts/data/load_toronto_data.py --skip-dbt
# Aggregate data loading
load-data: load-toronto ## Load all project data (currently: Toronto)
@echo "$(GREEN)All data loaded!$(NC)"
load-all: load-data ## Alias for load-data
seed-data: ## Seed sample development data (amenities, median_age)
@echo "$(GREEN)Seeding development data...$(NC)"
$(PYTHON) scripts/data/seed_amenity_data.py
# =============================================================================
# Application
# =============================================================================
@@ -117,15 +127,15 @@ test-cov: ## Run pytest with coverage
dbt-run: ## Run dbt models
@echo "$(GREEN)Running dbt models...$(NC)"
cd dbt && dbt run
@set -a && . ./.env && set +a && cd dbt && dbt run --profiles-dir .
dbt-test: ## Run dbt tests
@echo "$(GREEN)Running dbt tests...$(NC)"
cd dbt && dbt test
@set -a && . ./.env && set +a && cd dbt && dbt test --profiles-dir .
dbt-docs: ## Generate dbt documentation
@echo "$(GREEN)Generating dbt docs...$(NC)"
cd dbt && dbt docs generate && dbt docs serve
@set -a && . ./.env && set +a && cd dbt && dbt docs generate --profiles-dir . && dbt docs serve --profiles-dir .
# =============================================================================
# Code Quality

View File

@@ -115,28 +115,31 @@ portfolio_app/
│ ├── tabs/ # Tab layouts (5)
│ └── callbacks/ # Interaction logic
├── components/ # Shared UI components
├── figures/ # Plotly figure factories
├── figures/
│ └── toronto/ # Toronto figure factories
├── content/
│ └── blog/ # Markdown blog articles
├── toronto/ # Toronto data logic
│ ├── parsers/ # API data extraction
│ ├── loaders/ # Database operations
│ ├── schemas/ # Pydantic models
│ └── models/ # SQLAlchemy ORM
│ └── models/ # SQLAlchemy ORM (raw_toronto schema)
└── errors/ # Exception handling
dbt/
dbt/ # dbt project: portfolio
├── models/
│ ├── staging/ # 1:1 source tables
│ ├── intermediate/ # Business logic
── marts/ # Analytical tables
│ ├── shared/ # Cross-domain dimensions
│ ├── staging/toronto/ # Toronto staging models
── intermediate/toronto/ # Toronto intermediate models
│ └── marts/toronto/ # Toronto analytical tables
notebooks/ # Data documentation (15 notebooks)
── overview/ # Overview tab visualizations
├── housing/ # Housing tab visualizations
├── safety/ # Safety tab visualizations
├── demographics/ # Demographics tab visualizations
└── amenities/ # Amenities tab visualizations
notebooks/
── toronto/ # Toronto documentation (15 notebooks)
├── overview/ # Overview tab visualizations
├── housing/ # Housing tab visualizations
├── safety/ # Safety tab visualizations
├── demographics/ # Demographics tab visualizations
└── amenities/ # Amenities tab visualizations
docs/
├── PROJECT_REFERENCE.md # Architecture reference

View File

@@ -1,8 +1,7 @@
name: 'toronto_housing'
version: '1.0.0'
name: 'portfolio'
config-version: 2
profile: 'toronto_housing'
profile: 'portfolio'
model-paths: ["models"]
analysis-paths: ["analyses"]
@@ -16,13 +15,19 @@ clean-targets:
- "dbt_packages"
models:
toronto_housing:
portfolio:
shared:
+materialized: view
+schema: shared
staging:
+materialized: view
+schema: staging
toronto:
+materialized: view
+schema: stg_toronto
intermediate:
+materialized: view
+schema: intermediate
toronto:
+materialized: view
+schema: int_toronto
marts:
+materialized: table
+schema: marts
toronto:
+materialized: table
+schema: mart_toronto

View File

@@ -0,0 +1,11 @@
-- Override dbt default schema name generation.
-- Use the custom schema name directly instead of
-- concatenating with the target schema.
-- See: https://docs.getdbt.com/docs/build/custom-schemas
{% macro generate_schema_name(custom_schema_name, node) %}
{%- if custom_schema_name is none -%}
{{ target.schema }}
{%- else -%}
{{ custom_schema_name | trim }}
{%- endif -%}
{% endmacro %}

View File

@@ -5,11 +5,11 @@ models:
description: "Rental data enriched with time and zone dimensions"
columns:
- name: rental_id
tests:
data_tests:
- unique
- not_null
- name: zone_code
tests:
data_tests:
- not_null
- name: int_neighbourhood__demographics
@@ -17,11 +17,11 @@ models:
columns:
- name: neighbourhood_id
description: "Neighbourhood identifier"
tests:
data_tests:
- not_null
- name: census_year
description: "Census year"
tests:
data_tests:
- not_null
- name: income_quintile
description: "Income quintile (1-5, city-wide)"
@@ -31,7 +31,7 @@ models:
columns:
- name: neighbourhood_id
description: "Neighbourhood identifier"
tests:
data_tests:
- not_null
- name: year
description: "Reference year"
@@ -45,11 +45,11 @@ models:
columns:
- name: neighbourhood_id
description: "Neighbourhood identifier"
tests:
data_tests:
- not_null
- name: year
description: "Statistics year"
tests:
data_tests:
- not_null
- name: crime_rate_per_100k
description: "Total crime rate per 100K population"
@@ -61,7 +61,7 @@ models:
columns:
- name: neighbourhood_id
description: "Neighbourhood identifier"
tests:
data_tests:
- not_null
- name: year
description: "Reference year"
@@ -75,11 +75,11 @@ models:
columns:
- name: neighbourhood_id
description: "Neighbourhood identifier"
tests:
data_tests:
- not_null
- name: year
description: "Survey year"
tests:
data_tests:
- not_null
- name: avg_rent_2bed
description: "Weighted average 2-bedroom rent"

View File

@@ -16,12 +16,12 @@ crime_by_year as (
neighbourhood_id,
crime_year as year,
sum(incident_count) as total_incidents,
sum(case when crime_type = 'Assault' then incident_count else 0 end) as assault_count,
sum(case when crime_type = 'Auto Theft' then incident_count else 0 end) as auto_theft_count,
sum(case when crime_type = 'Break and Enter' then incident_count else 0 end) as break_enter_count,
sum(case when crime_type = 'Robbery' then incident_count else 0 end) as robbery_count,
sum(case when crime_type = 'Theft Over' then incident_count else 0 end) as theft_over_count,
sum(case when crime_type = 'Homicide' then incident_count else 0 end) as homicide_count,
sum(case when crime_type = 'assault' then incident_count else 0 end) as assault_count,
sum(case when crime_type = 'auto_theft' then incident_count else 0 end) as auto_theft_count,
sum(case when crime_type = 'break_and_enter' then incident_count else 0 end) as break_enter_count,
sum(case when crime_type = 'robbery' then incident_count else 0 end) as robbery_count,
sum(case when crime_type = 'theft_over' then incident_count else 0 end) as theft_over_count,
sum(case when crime_type = 'homicide' then incident_count else 0 end) as homicide_count,
avg(rate_per_100k) as avg_rate_per_100k
from crime
group by neighbourhood_id, crime_year

View File

@@ -42,10 +42,10 @@ pivoted as (
select
neighbourhood_id,
year,
max(case when bedroom_type = 'Two Bedroom' then weighted_avg_rent / nullif(total_weight, 0) end) as avg_rent_2bed,
max(case when bedroom_type = 'One Bedroom' then weighted_avg_rent / nullif(total_weight, 0) end) as avg_rent_1bed,
max(case when bedroom_type = 'Bachelor' then weighted_avg_rent / nullif(total_weight, 0) end) as avg_rent_bachelor,
max(case when bedroom_type = 'Three Bedroom +' then weighted_avg_rent / nullif(total_weight, 0) end) as avg_rent_3bed,
max(case when bedroom_type = '2bed' then weighted_avg_rent / nullif(total_weight, 0) end) as avg_rent_2bed,
max(case when bedroom_type = '1bed' then weighted_avg_rent / nullif(total_weight, 0) end) as avg_rent_1bed,
max(case when bedroom_type = 'bachelor' then weighted_avg_rent / nullif(total_weight, 0) end) as avg_rent_bachelor,
max(case when bedroom_type = '3bed' then weighted_avg_rent / nullif(total_weight, 0) end) as avg_rent_3bed,
avg(vacancy_rate) as vacancy_rate,
sum(rental_units_estimate) as total_rental_units
from allocated

View File

@@ -6,7 +6,7 @@ models:
columns:
- name: rental_id
description: "Unique rental record identifier"
tests:
data_tests:
- unique
- not_null
@@ -17,11 +17,11 @@ models:
columns:
- name: neighbourhood_id
description: "Neighbourhood identifier"
tests:
data_tests:
- not_null
- name: neighbourhood_name
description: "Official neighbourhood name"
tests:
data_tests:
- not_null
- name: geometry
description: "PostGIS geometry for mapping"
@@ -41,11 +41,11 @@ models:
columns:
- name: neighbourhood_id
description: "Neighbourhood identifier"
tests:
data_tests:
- not_null
- name: neighbourhood_name
description: "Official neighbourhood name"
tests:
data_tests:
- not_null
- name: geometry
description: "PostGIS geometry for mapping"
@@ -63,11 +63,11 @@ models:
columns:
- name: neighbourhood_id
description: "Neighbourhood identifier"
tests:
data_tests:
- not_null
- name: neighbourhood_name
description: "Official neighbourhood name"
tests:
data_tests:
- not_null
- name: geometry
description: "PostGIS geometry for mapping"
@@ -77,7 +77,7 @@ models:
description: "100 = city average crime rate"
- name: safety_tier
description: "Safety tier (1=safest, 5=highest crime)"
tests:
data_tests:
- accepted_values:
arguments:
values: [1, 2, 3, 4, 5]
@@ -89,11 +89,11 @@ models:
columns:
- name: neighbourhood_id
description: "Neighbourhood identifier"
tests:
data_tests:
- not_null
- name: neighbourhood_name
description: "Official neighbourhood name"
tests:
data_tests:
- not_null
- name: geometry
description: "PostGIS geometry for mapping"
@@ -103,7 +103,7 @@ models:
description: "100 = city average income"
- name: income_quintile
description: "Income quintile (1-5)"
tests:
data_tests:
- accepted_values:
arguments:
values: [1, 2, 3, 4, 5]
@@ -115,11 +115,11 @@ models:
columns:
- name: neighbourhood_id
description: "Neighbourhood identifier"
tests:
data_tests:
- not_null
- name: neighbourhood_name
description: "Official neighbourhood name"
tests:
data_tests:
- not_null
- name: geometry
description: "PostGIS geometry for mapping"
@@ -129,7 +129,7 @@ models:
description: "100 = city average amenities"
- name: amenity_tier
description: "Amenity tier (1=best, 5=lowest)"
tests:
data_tests:
- accepted_values:
arguments:
values: [1, 2, 3, 4, 5]

View File

@@ -128,7 +128,8 @@ final as (
-- Component scores (0-100)
round(safety_score::numeric, 1) as safety_score,
round(affordability_score::numeric, 1) as affordability_score,
-- Amenity score not available at this level, use placeholder
-- TODO: Replace with actual amenity score when fact_amenities is populated
-- Currently uses neutral placeholder (50.0) which affects livability_score accuracy
50.0 as amenity_score,
-- Composite livability score: safety (40%), affordability (40%), amenities (20%)

View File

@@ -0,0 +1,33 @@
version: 2
models:
- name: stg_dimensions__time
description: "Staged time dimension - shared across all projects"
columns:
- name: date_key
description: "Primary key (YYYYMM format)"
data_tests:
- unique
- not_null
- name: full_date
description: "First day of month"
data_tests:
- not_null
- name: year
description: "Calendar year"
data_tests:
- not_null
- name: month
description: "Month number (1-12)"
data_tests:
- not_null
- name: quarter
description: "Quarter (1-4)"
data_tests:
- not_null
- name: month_name
description: "Month name"
data_tests:
- not_null
- name: is_month_start
description: "Always true (monthly grain)"

View File

@@ -0,0 +1,25 @@
version: 2
sources:
- name: shared
description: "Shared dimension tables used across all dashboards"
database: portfolio
schema: public
tables:
- name: dim_time
description: "Time dimension (monthly grain) - shared across all projects"
columns:
- name: date_key
description: "Primary key (YYYYMM format)"
- name: full_date
description: "First day of month"
- name: year
description: "Calendar year"
- name: month
description: "Month number (1-12)"
- name: quarter
description: "Quarter (1-4)"
- name: month_name
description: "Month name"
- name: is_month_start
description: "Always true (monthly grain)"

View File

@@ -1,9 +1,10 @@
-- Staged time dimension
-- Source: dim_time table
-- Source: shared.dim_time table
-- Grain: One row per month
-- Note: Shared dimension used across all dashboard projects
with source as (
select * from {{ source('toronto_housing', 'dim_time') }}
select * from {{ source('shared', 'dim_time') }}
),
staged as (

View File

@@ -1,18 +0,0 @@
-- Staged CMHC zone dimension
-- Source: dim_cmhc_zone table
-- Grain: One row per zone
with source as (
select * from {{ source('toronto_housing', 'dim_cmhc_zone') }}
),
staged as (
select
zone_key,
zone_code,
zone_name,
geometry
from source
)
select * from staged

View File

@@ -1,10 +1,10 @@
version: 2
sources:
- name: toronto_housing
description: "Toronto housing data loaded from CMHC and City of Toronto sources"
- name: toronto
description: "Toronto data loaded from CMHC and City of Toronto sources"
database: portfolio
schema: public
schema: raw_toronto
tables:
- name: fact_rentals
description: "CMHC annual rental survey data by zone and bedroom type"
@@ -16,12 +16,6 @@ sources:
- name: zone_key
description: "Foreign key to dim_cmhc_zone"
- name: dim_time
description: "Time dimension (monthly grain)"
columns:
- name: date_key
description: "Primary key (YYYYMMDD format)"
- name: dim_cmhc_zone
description: "CMHC zone dimension with geometry"
columns:

View File

@@ -6,25 +6,16 @@ models:
columns:
- name: rental_id
description: "Unique identifier for rental record"
tests:
data_tests:
- unique
- not_null
- name: date_key
description: "Date dimension key (YYYYMMDD)"
tests:
data_tests:
- not_null
- name: zone_key
description: "CMHC zone dimension key"
tests:
- not_null
- name: stg_dimensions__time
description: "Staged time dimension"
columns:
- name: date_key
description: "Date dimension key (YYYYMMDD)"
tests:
- unique
data_tests:
- not_null
- name: stg_dimensions__cmhc_zones
@@ -32,12 +23,12 @@ models:
columns:
- name: zone_key
description: "Zone dimension key"
tests:
data_tests:
- unique
- not_null
- name: zone_code
description: "CMHC zone code"
tests:
data_tests:
- unique
- not_null
@@ -46,12 +37,12 @@ models:
columns:
- name: neighbourhood_id
description: "Neighbourhood primary key"
tests:
data_tests:
- unique
- not_null
- name: neighbourhood_name
description: "Official neighbourhood name"
tests:
data_tests:
- not_null
- name: geometry
description: "PostGIS geometry (POLYGON)"
@@ -61,16 +52,16 @@ models:
columns:
- name: census_id
description: "Census record identifier"
tests:
data_tests:
- unique
- not_null
- name: neighbourhood_id
description: "Neighbourhood foreign key"
tests:
data_tests:
- not_null
- name: census_year
description: "Census year (2016, 2021)"
tests:
data_tests:
- not_null
- name: stg_toronto__crime
@@ -78,16 +69,16 @@ models:
columns:
- name: crime_id
description: "Crime record identifier"
tests:
data_tests:
- unique
- not_null
- name: neighbourhood_id
description: "Neighbourhood foreign key"
tests:
data_tests:
- not_null
- name: crime_type
description: "Type of crime"
tests:
data_tests:
- not_null
- name: stg_toronto__amenities
@@ -95,16 +86,16 @@ models:
columns:
- name: amenity_id
description: "Amenity record identifier"
tests:
data_tests:
- unique
- not_null
- name: neighbourhood_id
description: "Neighbourhood foreign key"
tests:
data_tests:
- not_null
- name: amenity_type
description: "Type of amenity"
tests:
data_tests:
- not_null
- name: stg_cmhc__zone_crosswalk
@@ -112,18 +103,18 @@ models:
columns:
- name: crosswalk_id
description: "Crosswalk record identifier"
tests:
data_tests:
- unique
- not_null
- name: cmhc_zone_code
description: "CMHC zone code"
tests:
data_tests:
- not_null
- name: neighbourhood_id
description: "Neighbourhood foreign key"
tests:
data_tests:
- not_null
- name: area_weight
description: "Proportional area weight (0-1)"
tests:
data_tests:
- not_null

View File

@@ -6,8 +6,8 @@ with source as (
select
f.*,
t.year as survey_year
from {{ source('toronto_housing', 'fact_rentals') }} f
join {{ source('toronto_housing', 'dim_time') }} t on f.date_key = t.date_key
from {{ source('toronto', 'fact_rentals') }} f
join {{ source('shared', 'dim_time') }} t on f.date_key = t.date_key
),
staged as (

View File

@@ -3,7 +3,7 @@
-- Grain: One row per zone-neighbourhood intersection
with source as (
select * from {{ source('toronto_housing', 'bridge_cmhc_neighbourhood') }}
select * from {{ source('toronto', 'bridge_cmhc_neighbourhood') }}
),
staged as (

View File

@@ -0,0 +1,19 @@
-- Staged CMHC zone dimension
-- Source: dim_cmhc_zone table
-- Grain: One row per zone
with source as (
select * from {{ source('toronto', 'dim_cmhc_zone') }}
),
staged as (
select
zone_key,
zone_code,
zone_name
-- geometry column excluded: CMHC does not provide zone boundaries
-- Spatial analysis uses dim_neighbourhood geometry instead
from source
)
select * from staged

View File

@@ -3,7 +3,7 @@
-- Grain: One row per neighbourhood per amenity type per year
with source as (
select * from {{ source('toronto_housing', 'fact_amenities') }}
select * from {{ source('toronto', 'fact_amenities') }}
),
staged as (

View File

@@ -3,7 +3,7 @@
-- Grain: One row per neighbourhood per census year
with source as (
select * from {{ source('toronto_housing', 'fact_census') }}
select * from {{ source('toronto', 'fact_census') }}
),
staged as (

View File

@@ -3,7 +3,7 @@
-- Grain: One row per neighbourhood per year per crime type
with source as (
select * from {{ source('toronto_housing', 'fact_crime') }}
select * from {{ source('toronto', 'fact_crime') }}
),
staged as (

View File

@@ -3,7 +3,7 @@
-- Grain: One row per neighbourhood (158 total)
with source as (
select * from {{ source('toronto_housing', 'dim_neighbourhood') }}
select * from {{ source('toronto', 'dim_neighbourhood') }}
),
staged as (

View File

@@ -1,4 +1,4 @@
toronto_housing:
portfolio:
target: dev
outputs:
dev:

View File

@@ -50,9 +50,11 @@ The app runs at `http://localhost:8050`.
```bash
make test # Run tests
make test-cov # Run tests with coverage
make lint # Check code style
make format # Auto-format code
make ci # Run all checks (lint + test)
make typecheck # Run mypy type checker
make ci # Run all checks (lint, typecheck, test)
make dbt-run # Run dbt transformations
make dbt-test # Run dbt tests
```
@@ -247,13 +249,23 @@ def layout(slug: str = "") -> dmc.Container:
To add the page to the sidebar, edit `portfolio_app/components/sidebar.py`:
```python
NAV_ITEMS = [
{"label": "Home", "href": "/", "icon": "tabler:home"},
{"label": "Your Page", "href": "/your-page", "icon": "tabler:star"},
# For main pages (Home, About, Blog, etc.)
NAV_ITEMS_MAIN = [
{"path": "/", "icon": "tabler:home", "label": "Home"},
{"path": "/your-page", "icon": "tabler:star", "label": "Your Page"},
# ...
]
# For project/dashboard pages
NAV_ITEMS_PROJECTS = [
{"path": "/projects", "icon": "tabler:folder", "label": "Projects"},
{"path": "/your-dashboard", "icon": "tabler:chart-bar", "label": "Your Dashboard"},
# ...
]
```
The sidebar uses icon buttons with tooltips. Each item needs `path`, `icon` (Tabler icon name), and `label` (tooltip text).
### URL Routing Summary
| File Location | URL |
@@ -278,7 +290,7 @@ Dashboard tabs are in `portfolio_app/pages/toronto/tabs/`.
import dash_mantine_components as dmc
from portfolio_app.figures.choropleth import create_choropleth
from portfolio_app.figures.toronto.choropleth import create_choropleth
from portfolio_app.toronto.demo_data import get_demo_data
@@ -327,13 +339,13 @@ dmc.TabsPanel(create_your_tab_layout(), value="your-tab"),
## Creating Figure Factories
Figure factories are in `portfolio_app/figures/`. They create reusable Plotly figures.
Figure factories are organized by dashboard domain under `portfolio_app/figures/{domain}/`.
### Pattern
```python
# figures/your_chart.py
"""Your chart type factory."""
# figures/toronto/your_chart.py
"""Your chart type factory for Toronto dashboard."""
import plotly.express as px
import plotly.graph_objects as go
@@ -370,7 +382,7 @@ def create_your_chart(
### Export from `__init__.py`
```python
# figures/__init__.py
# figures/toronto/__init__.py
from .your_chart import create_your_chart
__all__ = [
@@ -379,6 +391,14 @@ __all__ = [
]
```
### Importing Figure Factories
```python
# In callbacks or tabs
from portfolio_app.figures.toronto import create_choropleth_figure
from portfolio_app.figures.toronto.bar_charts import create_ranking_bar
```
---
## Branch Workflow

View File

@@ -116,18 +116,40 @@ erDiagram
## Schema Layers
### Raw Schema
### Database Schemas
Raw data is loaded directly from external sources without transformation:
| Schema | Purpose | Managed By |
|--------|---------|------------|
| `public` | Shared dimensions (dim_time) | SQLAlchemy |
| `raw_toronto` | Toronto dimension and fact tables | SQLAlchemy |
| `stg_toronto` | Toronto staging models | dbt |
| `int_toronto` | Toronto intermediate models | dbt |
| `mart_toronto` | Toronto analytical tables | dbt |
### Raw Toronto Schema (raw_toronto)
Toronto-specific tables loaded by SQLAlchemy:
| Table | Source | Description |
|-------|--------|-------------|
| `raw.neighbourhoods` | City of Toronto API | GeoJSON neighbourhood boundaries |
| `raw.census_profiles` | City of Toronto API | Census profile data |
| `raw.crime_data` | Toronto Police API | Crime statistics by neighbourhood |
| `raw.cmhc_rentals` | CMHC Data Files | Rental market survey data |
| `dim_neighbourhood` | City of Toronto API | 158 neighbourhood boundaries |
| `dim_cmhc_zone` | CMHC | ~20 rental market zones |
| `dim_policy_event` | Manual | Policy events for annotation |
| `fact_census` | City of Toronto API | Census profile data |
| `fact_crime` | Toronto Police API | Crime statistics |
| `fact_amenities` | City of Toronto API | Amenity counts |
| `fact_rentals` | CMHC Data Files | Rental market survey data |
| `bridge_cmhc_neighbourhood` | Computed | Zone-neighbourhood mapping |
### Staging Schema (dbt)
### Public Schema
Shared dimensions used across all projects:
| Table | Description |
|-------|-------------|
| `dim_time` | Time dimension (monthly grain) |
### Staging Schema - stg_toronto (dbt)
Staging models provide 1:1 cleaned representations of source data:
@@ -136,18 +158,24 @@ Staging models provide 1:1 cleaned representations of source data:
| `stg_toronto__neighbourhoods` | raw.neighbourhoods | Cleaned boundaries with standardized names |
| `stg_toronto__census` | raw.census_profiles | Typed census metrics |
| `stg_cmhc__rentals` | raw.cmhc_rentals | Validated rental data |
| `stg_police__crimes` | raw.crime_data | Standardized crime categories |
| `stg_toronto__crime` | raw.crime_data | Standardized crime categories |
| `stg_toronto__amenities` | raw.amenities | Typed amenity counts |
| `stg_dimensions__time` | generated | Time dimension |
| `stg_dimensions__cmhc_zones` | raw.cmhc_zones | CMHC zone boundaries |
| `stg_cmhc__zone_crosswalk` | raw.crosswalk | Zone-neighbourhood mapping |
### Marts Schema (dbt)
### Marts Schema - mart_toronto (dbt)
Analytical tables ready for dashboard consumption:
| Model | Grain | Purpose |
|-------|-------|---------|
| `mart_neighbourhood_summary` | neighbourhood | Composite livability scores |
| `mart_rental_trends` | zone × month | Time-series rental analysis |
| `mart_crime_rates` | neighbourhood × year | Crime rate calculations |
| `mart_amenity_density` | neighbourhood | Amenity accessibility scores |
| `mart_neighbourhood_overview` | neighbourhood | Composite livability scores |
| `mart_neighbourhood_housing` | neighbourhood | Housing and rent metrics |
| `mart_neighbourhood_safety` | neighbourhood × year | Crime rate calculations |
| `mart_neighbourhood_demographics` | neighbourhood | Income, age, population metrics |
| `mart_neighbourhood_amenities` | neighbourhood | Amenity accessibility scores |
| `mart_toronto_rentals` | zone × month | Time-series rental analysis |
## Table Details

View File

@@ -76,7 +76,8 @@ portfolio_app/
├── components/ # Shared UI components
├── content/blog/ # Markdown blog articles
├── errors/ # Exception handling
├── figures/ # Plotly figure factories
├── figures/
│ └── toronto/ # Toronto figure factories
├── pages/
│ ├── home.py
│ ├── about.py
@@ -91,15 +92,26 @@ portfolio_app/
│ ├── dashboard.py
│ ├── methodology.py
│ ├── tabs/ # 5 tab layouts
│ └── callbacks/ # Dashboard interactions
│ └── callbacks/ # Dashboard interactions (map_callbacks, chart_callbacks, selection_callbacks)
├── toronto/ # Data logic
│ ├── parsers/ # API extraction
│ ├── loaders/ # Database operations
│ ├── parsers/ # API extraction (geo, toronto_open_data, toronto_police, cmhc)
│ ├── loaders/ # Database operations (base, cmhc, cmhc_crosswalk)
│ ├── schemas/ # Pydantic models
│ ├── models/ # SQLAlchemy ORM
│ ├── models/ # SQLAlchemy ORM (raw_toronto schema)
│ ├── services/ # Query functions (neighbourhood_service, geometry_service)
│ └── demo_data.py # Sample data
└── utils/
└── markdown_loader.py # Blog article loading
dbt/ # dbt project: portfolio
├── models/
│ ├── shared/ # Cross-domain dimensions
│ ├── staging/toronto/ # Toronto staging models
│ ├── intermediate/toronto/ # Toronto intermediate models
│ └── marts/toronto/ # Toronto mart tables
notebooks/
└── toronto/ # Toronto documentation notebooks
```
---
@@ -143,10 +155,20 @@ CMHC Zones (~20) ← Rental data (Census Tract aligned)
| `fact_rentals` | Fact | Rental data by CMHC zone |
| `fact_amenities` | Fact | Amenity counts by neighbourhood |
### dbt Layers
### dbt Project: `portfolio`
**Model Structure:**
```
dbt/models/
├── shared/ # Cross-domain dimensions (stg_dimensions__time)
├── staging/toronto/ # Toronto staging models
├── intermediate/toronto/ # Toronto intermediate models
└── marts/toronto/ # Toronto mart tables
```
| Layer | Naming | Example |
|-------|--------|---------|
| Shared | `stg_dimensions__*` | `stg_dimensions__time` |
| Staging | `stg_{source}__{entity}` | `stg_toronto__neighbourhoods` |
| Intermediate | `int_{domain}__{transform}` | `int_neighbourhood__demographics` |
| Marts | `mart_{domain}` | `mart_neighbourhood_overview` |
@@ -241,16 +263,25 @@ LOG_LEVEL=INFO
| Target | Purpose |
|--------|---------|
| `setup` | Install deps, create .env, init pre-commit |
| `docker-up` | Start PostgreSQL + PostGIS |
| `docker-up` | Start PostgreSQL + PostGIS (auto-detects x86/ARM) |
| `docker-down` | Stop containers |
| `docker-logs` | View container logs |
| `db-init` | Initialize database schema |
| `db-reset` | Drop and recreate database (DESTRUCTIVE) |
| `load-data` | Load Toronto data from APIs, seed dev data |
| `load-toronto-only` | Load Toronto data without dbt or seeding |
| `seed-data` | Seed sample development data |
| `run` | Start Dash dev server |
| `test` | Run pytest |
| `dbt-run` | Run dbt models |
| `dbt-test` | Run dbt tests |
| `test-cov` | Run pytest with coverage |
| `lint` | Run ruff linter |
| `format` | Run ruff formatter |
| `ci` | Run all checks |
| `typecheck` | Run mypy type checker |
| `ci` | Run all checks (lint, typecheck, test) |
| `dbt-run` | Run dbt models |
| `dbt-test` | Run dbt tests |
| `dbt-docs` | Generate and serve dbt documentation |
| `clean` | Remove build artifacts and caches |
---

View File

@@ -10,6 +10,7 @@ This folder contains lessons learned from sprints and development work. These le
| Date | Sprint/Phase | Title | Tags |
|------|--------------|-------|------|
| 2026-02-01 | Sprint 10 | [Formspree Integration with Dash Callbacks](./sprint-10-formspree-dash-integration.md) | formspree, dash, callbacks, forms, spam-protection, honeypot, ajax |
| 2026-01-17 | Sprint 9 | [Gitea Labels API Requires Org Context](./sprint-9-gitea-labels-user-repos.md) | gitea, mcp, api, labels, projman, configuration |
| 2026-01-17 | Sprint 9 | [Always Read CLAUDE.md Before Asking Questions](./sprint-9-read-claude-md-first.md) | projman, claude-code, context, documentation, workflow |
| 2026-01-17 | Sprint 9-10 | [Graceful Error Handling in Service Layers](./sprint-9-10-graceful-error-handling.md) | python, postgresql, error-handling, dash, graceful-degradation, arm64 |

View File

@@ -0,0 +1,70 @@
# Sprint 10 - Formspree Integration with Dash Callbacks
## Context
Implementing a contact form on a Dash portfolio site that submits to Formspree, a third-party form handling service.
## Insights
### Formspree AJAX Submission
Formspree supports AJAX submissions (no page redirect) when you:
1. POST with `Content-Type: application/json`
2. Include `Accept: application/json` header
3. Send form data as JSON body
This returns a JSON response instead of redirecting to a thank-you page, which is ideal for single-page Dash applications.
### Dash Multi-Output Callbacks for Forms
When handling form submission with validation and feedback, use a multi-output callback pattern:
```python
@callback(
Output("feedback-container", "children"), # Success/error alert
Output("submit-button", "loading"), # Button loading state
Output("field-1", "value"), # Clear on success
Output("field-2", "value"), # Clear on success
Output("field-1", "error"), # Field-level errors
Output("field-2", "error"), # Field-level errors
Input("submit-button", "n_clicks"),
State("field-1", "value"),
State("field-2", "value"),
prevent_initial_call=True,
)
```
Use `no_update` for outputs you don't want to change (e.g., keep form values on validation error, only clear on success).
### Honeypot Spam Protection
Simple and effective bot protection without CAPTCHA:
1. Add a hidden text input field (CSS: `position: absolute; left: -9999px`)
2. Set `tabIndex=-1` and `autoComplete="off"` to prevent accidental filling
3. In callback, check if honeypot has value - if yes, it's a bot
4. For bots: return fake success (don't reveal detection)
5. For humans: proceed with real submission
Formspree also accepts `_gotcha` as a honeypot field name in the JSON payload.
## Code Pattern
```python
# Honeypot check - bots fill hidden fields
if honeypot_value:
# Fake success - don't let bots know they were caught
return (_create_success_alert(), False, "", "", None, None)
# Real submission for humans
response = requests.post(
FORMSPREE_ENDPOINT,
json=form_data,
headers={"Accept": "application/json", "Content-Type": "application/json"},
timeout=10,
)
```
## Prevention/Best Practices
- Always use `timeout` parameter with `requests.post()` to avoid hanging
- Wrap external API calls in try/except for network errors
- Return user-friendly error messages, not technical details
- Use DMC's `required=True` and `error` props for form validation feedback
## Tags
formspree, dash, callbacks, forms, spam-protection, honeypot, ajax, python, requests, validation

View File

@@ -10,7 +10,9 @@ This runbook describes how to add a new data dashboard to the portfolio applicat
## Directory Structure
Create the following structure under `portfolio_app/`:
Create the following structure:
### Application Code (`portfolio_app/`)
```
portfolio_app/
@@ -33,8 +35,40 @@ portfolio_app/
│ │ └── __init__.py
│ ├── schemas/ # Pydantic models
│ │ └── __init__.py
│ └── models/ # SQLAlchemy ORM
│ └── models/ # SQLAlchemy ORM (schema: raw_{dashboard_name})
│ └── __init__.py
└── figures/
└── {dashboard_name}/ # Figure factories for this dashboard
├── __init__.py
└── ... # Chart modules
```
### dbt Models (`dbt/models/`)
```
dbt/models/
├── staging/
│ └── {dashboard_name}/ # Staging models
│ ├── _sources.yml # Source definitions (schema: raw_{dashboard_name})
│ ├── _staging.yml # Model tests/docs
│ └── stg_*.sql # Staging models
├── intermediate/
│ └── {dashboard_name}/ # Intermediate models
│ ├── _intermediate.yml
│ └── int_*.sql
└── marts/
└── {dashboard_name}/ # Mart tables
├── _marts.yml
└── mart_*.sql
```
### Documentation (`notebooks/`)
```
notebooks/
└── {dashboard_name}/ # Domain subdirectories
├── overview/
├── ...
```
## Step-by-Step Checklist
@@ -47,24 +81,55 @@ portfolio_app/
- [ ] Create loaders in `{dashboard_name}/loaders/`
- [ ] Add database migrations if needed
### 2. dbt Models
### 2. Database Schema
- [ ] Define schema constant in models (e.g., `RAW_FOOTBALL_SCHEMA = "raw_football"`)
- [ ] Add `__table_args__ = {"schema": RAW_FOOTBALL_SCHEMA}` to all models
- [ ] Update `scripts/db/init_schema.py` to create the new schema
### 3. dbt Models
Create dbt models in `dbt/models/`:
- [ ] `staging/stg_{source}__{entity}.sql` - Raw data cleaning
- [ ] `intermediate/int_{domain}__{transform}.sql` - Business logic
- [ ] `marts/mart_{domain}.sql` - Final analytical tables
- [ ] `staging/{dashboard_name}/_sources.yml` - Source definitions pointing to `raw_{dashboard_name}` schema
- [ ] `staging/{dashboard_name}/stg_{source}__{entity}.sql` - Raw data cleaning
- [ ] `intermediate/{dashboard_name}/int_{domain}__{transform}.sql` - Business logic
- [ ] `marts/{dashboard_name}/mart_{domain}.sql` - Final analytical tables
Update `dbt/dbt_project.yml` with new subdirectory config:
```yaml
models:
portfolio:
staging:
{dashboard_name}:
+materialized: view
+schema: stg_{dashboard_name}
intermediate:
{dashboard_name}:
+materialized: view
+schema: int_{dashboard_name}
marts:
{dashboard_name}:
+materialized: table
+schema: mart_{dashboard_name}
```
Follow naming conventions:
- Staging: `stg_{source}__{entity}`
- Intermediate: `int_{domain}__{transform}`
- Marts: `mart_{domain}`
### 3. Visualization Layer
### 4. Visualization Layer
- [ ] Create figure factories in `figures/` (or reuse existing)
- [ ] Create figure factories in `figures/{dashboard_name}/`
- [ ] Create `figures/{dashboard_name}/__init__.py` with exports
- [ ] Follow the factory pattern: `create_{chart_type}_figure(data, **kwargs)`
Import pattern:
```python
from portfolio_app.figures.{dashboard_name} import create_choropleth_figure
```
### 4. Dashboard Pages
#### Main Dashboard (`pages/{dashboard_name}/dashboard.py`)

View File

@@ -1,17 +1,18 @@
# Toronto Neighbourhood Dashboard - Notebooks
# Dashboard Documentation Notebooks
Documentation notebooks for the Toronto Neighbourhood Dashboard visualizations. Each notebook documents how data is queried, transformed, and visualized using the figure factory pattern.
Documentation notebooks organized by dashboard project. Each notebook documents how data is queried, transformed, and visualized using the figure factory pattern.
## Directory Structure
```
notebooks/
├── README.md # This file
── overview/ # Overview tab visualizations
├── housing/ # Housing tab visualizations
├── safety/ # Safety tab visualizations
├── demographics/ # Demographics tab visualizations
└── amenities/ # Amenities tab visualizations
── toronto/ # Toronto Neighbourhood Dashboard
├── overview/ # Overview tab visualizations
├── housing/ # Housing tab visualizations
├── safety/ # Safety tab visualizations
├── demographics/ # Demographics tab visualizations
└── amenities/ # Amenities tab visualizations
```
## Notebook Template

View File

@@ -30,11 +30,16 @@
"metadata": {},
"outputs": [],
"source": [
"import pandas as pd\n",
"from sqlalchemy import create_engine\n",
"import os\n",
"\n",
"engine = create_engine(os.environ.get('DATABASE_URL', 'postgresql://portfolio:portfolio@localhost:5432/portfolio'))\n",
"import pandas as pd\n",
"from dotenv import load_dotenv\n",
"from sqlalchemy import create_engine\n",
"\n",
"# Load .env from project root\n",
"load_dotenv(\"../../.env\")\n",
"\n",
"engine = create_engine(os.environ[\"DATABASE_URL\"])\n",
"\n",
"query = \"\"\"\n",
"SELECT\n",
@@ -50,8 +55,8 @@
" transit_per_1000,\n",
" total_amenities,\n",
" population\n",
"FROM mart_neighbourhood_amenities\n",
"WHERE year = (SELECT MAX(year) FROM mart_neighbourhood_amenities)\n",
"FROM public_marts.mart_neighbourhood_amenities\n",
"WHERE year = (SELECT MAX(year) FROM public_marts.mart_neighbourhood_amenities)\n",
"ORDER BY total_amenities_per_1000 DESC\n",
"\"\"\"\n",
"\n",
@@ -75,17 +80,16 @@
"metadata": {},
"outputs": [],
"source": [
"import geopandas as gpd\n",
"import json\n",
"\n",
"import geopandas as gpd\n",
"\n",
"gdf = gpd.GeoDataFrame(\n",
" df,\n",
" geometry=gpd.GeoSeries.from_wkb(df['geometry']),\n",
" crs='EPSG:4326'\n",
" df, geometry=gpd.GeoSeries.from_wkb(df[\"geometry\"]), crs=\"EPSG:4326\"\n",
")\n",
"\n",
"geojson = json.loads(gdf.to_json())\n",
"data = df.drop(columns=['geometry']).to_dict('records')"
"data = df.drop(columns=[\"geometry\"]).to_dict(\"records\")"
]
},
{
@@ -101,7 +105,9 @@
"metadata": {},
"outputs": [],
"source": [
"df[['neighbourhood_name', 'total_amenities_per_1000', 'amenity_index', 'amenity_tier']].head(10)"
"df[\n",
" [\"neighbourhood_name\", \"total_amenities_per_1000\", \"amenity_index\", \"amenity_tier\"]\n",
"].head(10)"
]
},
{
@@ -112,7 +118,7 @@
"\n",
"### Figure Factory\n",
"\n",
"Uses `create_choropleth_figure` from `portfolio_app.figures.choropleth`."
"Uses `create_choropleth_figure` from `portfolio_app.figures.toronto.choropleth`."
]
},
{
@@ -122,18 +128,24 @@
"outputs": [],
"source": [
"import sys\n",
"sys.path.insert(0, '../..')\n",
"\n",
"from portfolio_app.figures.choropleth import create_choropleth_figure\n",
"sys.path.insert(0, \"../..\")\n",
"\n",
"from portfolio_app.figures.toronto.choropleth import create_choropleth_figure\n",
"\n",
"fig = create_choropleth_figure(\n",
" geojson=geojson,\n",
" data=data,\n",
" location_key='neighbourhood_id',\n",
" color_column='total_amenities_per_1000',\n",
" hover_data=['neighbourhood_name', 'amenity_index', 'parks_per_1000', 'schools_per_1000'],\n",
" color_scale='Greens',\n",
" title='Toronto Amenities per 1,000 Population',\n",
" location_key=\"neighbourhood_id\",\n",
" color_column=\"total_amenities_per_1000\",\n",
" hover_data=[\n",
" \"neighbourhood_name\",\n",
" \"amenity_index\",\n",
" \"parks_per_1000\",\n",
" \"schools_per_1000\",\n",
" ],\n",
" color_scale=\"Greens\",\n",
" title=\"Toronto Amenities per 1,000 Population\",\n",
" zoom=10,\n",
")\n",
"\n",

View File

@@ -30,11 +30,16 @@
"metadata": {},
"outputs": [],
"source": [
"import pandas as pd\n",
"from sqlalchemy import create_engine\n",
"import os\n",
"\n",
"engine = create_engine(os.environ.get('DATABASE_URL', 'postgresql://portfolio:portfolio@localhost:5432/portfolio'))\n",
"import pandas as pd\n",
"from dotenv import load_dotenv\n",
"from sqlalchemy import create_engine\n",
"\n",
"# Load .env from project root\n",
"load_dotenv(\"../../.env\")\n",
"\n",
"engine = create_engine(os.environ[\"DATABASE_URL\"])\n",
"\n",
"query = \"\"\"\n",
"SELECT\n",
@@ -44,8 +49,8 @@
" transit_index,\n",
" amenity_index,\n",
" amenity_tier\n",
"FROM mart_neighbourhood_amenities\n",
"WHERE year = (SELECT MAX(year) FROM mart_neighbourhood_amenities)\n",
"FROM public_marts.mart_neighbourhood_amenities\n",
"WHERE year = (SELECT MAX(year) FROM public_marts.mart_neighbourhood_amenities)\n",
"ORDER BY amenity_index DESC\n",
"\"\"\"\n",
"\n",
@@ -74,8 +79,8 @@
"bottom_5 = df.tail(5)\n",
"\n",
"# Prepare radar data\n",
"categories = ['Parks', 'Schools', 'Transit']\n",
"index_columns = ['parks_index', 'schools_index', 'transit_index']"
"categories = [\"Parks\", \"Schools\", \"Transit\"]\n",
"index_columns = [\"parks_index\", \"schools_index\", \"transit_index\"]"
]
},
{
@@ -92,9 +97,29 @@
"outputs": [],
"source": [
"print(\"Top 5 Amenity-Rich Neighbourhoods:\")\n",
"display(top_5[['neighbourhood_name', 'parks_index', 'schools_index', 'transit_index', 'amenity_index']])\n",
"display(\n",
" top_5[\n",
" [\n",
" \"neighbourhood_name\",\n",
" \"parks_index\",\n",
" \"schools_index\",\n",
" \"transit_index\",\n",
" \"amenity_index\",\n",
" ]\n",
" ]\n",
")\n",
"print(\"\\nBottom 5 Underserved Neighbourhoods:\")\n",
"display(bottom_5[['neighbourhood_name', 'parks_index', 'schools_index', 'transit_index', 'amenity_index']])"
"display(\n",
" bottom_5[\n",
" [\n",
" \"neighbourhood_name\",\n",
" \"parks_index\",\n",
" \"schools_index\",\n",
" \"transit_index\",\n",
" \"amenity_index\",\n",
" ]\n",
" ]\n",
")"
]
},
{
@@ -105,7 +130,7 @@
"\n",
"### Figure Factory\n",
"\n",
"Uses `create_radar` from `portfolio_app.figures.radar`."
"Uses `create_radar` from `portfolio_app.figures.toronto.radar`."
]
},
{
@@ -115,28 +140,21 @@
"outputs": [],
"source": [
"import sys\n",
"sys.path.insert(0, '../..')\n",
"\n",
"from portfolio_app.figures.radar import create_radar_figure\n",
"sys.path.insert(0, \"../..\")\n",
"\n",
"from portfolio_app.figures.toronto.radar import create_comparison_radar\n",
"\n",
"# Compare top neighbourhood vs city average (100)\n",
"top_hood = top_5.iloc[0]\n",
"metrics = [\"parks_index\", \"schools_index\", \"transit_index\"]\n",
"\n",
"data = [\n",
" {\n",
" 'name': top_hood['neighbourhood_name'],\n",
" 'values': [top_hood['parks_index'], top_hood['schools_index'], top_hood['transit_index']],\n",
" 'categories': categories\n",
" },\n",
" {\n",
" 'name': 'City Average',\n",
" 'values': [100, 100, 100],\n",
" 'categories': categories\n",
" }\n",
"]\n",
"\n",
"fig = create_radar_figure(\n",
" data=data,\n",
"fig = create_comparison_radar(\n",
" selected_data=top_hood.to_dict(),\n",
" average_data={\"parks_index\": 100, \"schools_index\": 100, \"transit_index\": 100},\n",
" metrics=metrics,\n",
" selected_name=top_hood[\"neighbourhood_name\"],\n",
" average_name=\"City Average\",\n",
" title=f\"Amenity Profile: {top_hood['neighbourhood_name']} vs City Average\",\n",
")\n",
"\n",

View File

@@ -30,11 +30,16 @@
"metadata": {},
"outputs": [],
"source": [
"import pandas as pd\n",
"from sqlalchemy import create_engine\n",
"import os\n",
"\n",
"engine = create_engine(os.environ.get('DATABASE_URL', 'postgresql://portfolio:portfolio@localhost:5432/portfolio'))\n",
"import pandas as pd\n",
"from dotenv import load_dotenv\n",
"from sqlalchemy import create_engine\n",
"\n",
"# Load .env from project root\n",
"load_dotenv(\"../../.env\")\n",
"\n",
"engine = create_engine(os.environ[\"DATABASE_URL\"])\n",
"\n",
"query = \"\"\"\n",
"SELECT\n",
@@ -44,8 +49,8 @@
" transit_count,\n",
" population,\n",
" amenity_tier\n",
"FROM mart_neighbourhood_amenities\n",
"WHERE year = (SELECT MAX(year) FROM mart_neighbourhood_amenities)\n",
"FROM public_marts.mart_neighbourhood_amenities\n",
"WHERE year = (SELECT MAX(year) FROM public_marts.mart_neighbourhood_amenities)\n",
" AND transit_per_1000 IS NOT NULL\n",
"ORDER BY transit_per_1000 DESC\n",
"\"\"\"\n",
@@ -70,7 +75,7 @@
"metadata": {},
"outputs": [],
"source": [
"data = df.head(20).to_dict('records')"
"data = df.head(20).to_dict(\"records\")"
]
},
{
@@ -86,7 +91,9 @@
"metadata": {},
"outputs": [],
"source": [
"df[['neighbourhood_name', 'transit_per_1000', 'transit_index', 'transit_count']].head(10)"
"df[[\"neighbourhood_name\", \"transit_per_1000\", \"transit_index\", \"transit_count\"]].head(\n",
" 10\n",
")"
]
},
{
@@ -97,7 +104,7 @@
"\n",
"### Figure Factory\n",
"\n",
"Uses `create_horizontal_bar` from `portfolio_app.figures.bar_charts`."
"Uses `create_horizontal_bar` from `portfolio_app.figures.toronto.bar_charts`."
]
},
{
@@ -107,17 +114,18 @@
"outputs": [],
"source": [
"import sys\n",
"sys.path.insert(0, '../..')\n",
"\n",
"from portfolio_app.figures.bar_charts import create_horizontal_bar\n",
"sys.path.insert(0, \"../..\")\n",
"\n",
"from portfolio_app.figures.toronto.bar_charts import create_horizontal_bar\n",
"\n",
"fig = create_horizontal_bar(\n",
" data=data,\n",
" name_column='neighbourhood_name',\n",
" value_column='transit_per_1000',\n",
" title='Top 20 Neighbourhoods by Transit Accessibility',\n",
" color='#00BCD4',\n",
" value_format='.2f',\n",
" name_column=\"neighbourhood_name\",\n",
" value_column=\"transit_per_1000\",\n",
" title=\"Top 20 Neighbourhoods by Transit Accessibility\",\n",
" color=\"#00BCD4\",\n",
" value_format=\".2f\",\n",
")\n",
"\n",
"fig.show()"
@@ -136,7 +144,7 @@
"metadata": {},
"outputs": [],
"source": [
"print(f\"City-wide Transit Statistics:\")\n",
"print(\"City-wide Transit Statistics:\")\n",
"print(f\" Total Transit Stops: {df['transit_count'].sum():,.0f}\")\n",
"print(f\" Average per 1,000 pop: {df['transit_per_1000'].mean():.2f}\")\n",
"print(f\" Median per 1,000 pop: {df['transit_per_1000'].median():.2f}\")\n",

View File

@@ -30,11 +30,16 @@
"metadata": {},
"outputs": [],
"source": [
"import pandas as pd\n",
"from sqlalchemy import create_engine\n",
"import os\n",
"\n",
"engine = create_engine(os.environ.get('DATABASE_URL', 'postgresql://portfolio:portfolio@localhost:5432/portfolio'))\n",
"import pandas as pd\n",
"from dotenv import load_dotenv\n",
"from sqlalchemy import create_engine\n",
"\n",
"# Load .env from project root\n",
"load_dotenv(\"../../.env\")\n",
"\n",
"engine = create_engine(os.environ[\"DATABASE_URL\"])\n",
"\n",
"query = \"\"\"\n",
"SELECT\n",
@@ -45,8 +50,8 @@
" population,\n",
" income_quintile,\n",
" pct_renter_occupied\n",
"FROM mart_neighbourhood_demographics\n",
"WHERE year = (SELECT MAX(year) FROM mart_neighbourhood_demographics)\n",
"FROM public_marts.mart_neighbourhood_demographics\n",
"WHERE year = (SELECT MAX(year) FROM public_marts.mart_neighbourhood_demographics)\n",
" AND median_age IS NOT NULL\n",
"ORDER BY median_age DESC\n",
"\"\"\"\n",
@@ -72,13 +77,13 @@
"metadata": {},
"outputs": [],
"source": [
"city_avg = df['city_avg_age'].iloc[0]\n",
"df['age_category'] = df['median_age'].apply(\n",
" lambda x: 'Younger' if x < city_avg else 'Older'\n",
"city_avg = df[\"city_avg_age\"].iloc[0]\n",
"df[\"age_category\"] = df[\"median_age\"].apply(\n",
" lambda x: \"Younger\" if x < city_avg else \"Older\"\n",
")\n",
"df['age_deviation'] = df['median_age'] - city_avg\n",
"df[\"age_deviation\"] = df[\"median_age\"] - city_avg\n",
"\n",
"data = df.to_dict('records')"
"data = df.to_dict(\"records\")"
]
},
{
@@ -96,9 +101,13 @@
"source": [
"print(f\"City Average Age: {city_avg:.1f}\")\n",
"print(\"\\nYoungest Neighbourhoods:\")\n",
"display(df.tail(5)[['neighbourhood_name', 'median_age', 'age_index', 'pct_renter_occupied']])\n",
"display(\n",
" df.tail(5)[[\"neighbourhood_name\", \"median_age\", \"age_index\", \"pct_renter_occupied\"]]\n",
")\n",
"print(\"\\nOldest Neighbourhoods:\")\n",
"display(df.head(5)[['neighbourhood_name', 'median_age', 'age_index', 'pct_renter_occupied']])"
"display(\n",
" df.head(5)[[\"neighbourhood_name\", \"median_age\", \"age_index\", \"pct_renter_occupied\"]]\n",
")"
]
},
{
@@ -109,7 +118,7 @@
"\n",
"### Figure Factory\n",
"\n",
"Uses `create_ranking_bar` from `portfolio_app.figures.bar_charts`."
"Uses `create_ranking_bar` from `portfolio_app.figures.toronto.bar_charts`."
]
},
{
@@ -119,20 +128,21 @@
"outputs": [],
"source": [
"import sys\n",
"sys.path.insert(0, '../..')\n",
"\n",
"from portfolio_app.figures.bar_charts import create_ranking_bar\n",
"sys.path.insert(0, \"../..\")\n",
"\n",
"from portfolio_app.figures.toronto.bar_charts import create_ranking_bar\n",
"\n",
"fig = create_ranking_bar(\n",
" data=data,\n",
" name_column='neighbourhood_name',\n",
" value_column='median_age',\n",
" title='Youngest & Oldest Neighbourhoods (Median Age)',\n",
" name_column=\"neighbourhood_name\",\n",
" value_column=\"median_age\",\n",
" title=\"Youngest & Oldest Neighbourhoods (Median Age)\",\n",
" top_n=10,\n",
" bottom_n=10,\n",
" color_top='#FF9800', # Orange for older\n",
" color_bottom='#2196F3', # Blue for younger\n",
" value_format='.1f',\n",
" color_top=\"#FF9800\", # Orange for older\n",
" color_bottom=\"#2196F3\", # Blue for younger\n",
" value_format=\".1f\",\n",
")\n",
"\n",
"fig.show()"
@@ -153,7 +163,7 @@
"source": [
"# Age by income quintile\n",
"print(\"Median Age by Income Quintile:\")\n",
"df.groupby('income_quintile')['median_age'].mean().round(1)"
"df.groupby(\"income_quintile\")[\"median_age\"].mean().round(1)"
]
}
],

View File

@@ -30,11 +30,16 @@
"metadata": {},
"outputs": [],
"source": [
"import pandas as pd\n",
"from sqlalchemy import create_engine\n",
"import os\n",
"\n",
"engine = create_engine(os.environ.get('DATABASE_URL', 'postgresql://portfolio:portfolio@localhost:5432/portfolio'))\n",
"import pandas as pd\n",
"from dotenv import load_dotenv\n",
"from sqlalchemy import create_engine\n",
"\n",
"# Load .env from project root\n",
"load_dotenv(\"../../.env\")\n",
"\n",
"engine = create_engine(os.environ[\"DATABASE_URL\"])\n",
"\n",
"query = \"\"\"\n",
"SELECT\n",
@@ -47,8 +52,8 @@
" income_quintile,\n",
" population,\n",
" unemployment_rate\n",
"FROM mart_neighbourhood_demographics\n",
"WHERE year = (SELECT MAX(year) FROM mart_neighbourhood_demographics)\n",
"FROM public_marts.mart_neighbourhood_demographics\n",
"WHERE year = (SELECT MAX(year) FROM public_marts.mart_neighbourhood_demographics)\n",
"ORDER BY median_household_income DESC\n",
"\"\"\"\n",
"\n",
@@ -73,19 +78,18 @@
"metadata": {},
"outputs": [],
"source": [
"import geopandas as gpd\n",
"import json\n",
"\n",
"df['income_thousands'] = df['median_household_income'] / 1000\n",
"import geopandas as gpd\n",
"\n",
"df[\"income_thousands\"] = df[\"median_household_income\"] / 1000\n",
"\n",
"gdf = gpd.GeoDataFrame(\n",
" df,\n",
" geometry=gpd.GeoSeries.from_wkb(df['geometry']),\n",
" crs='EPSG:4326'\n",
" df, geometry=gpd.GeoSeries.from_wkb(df[\"geometry\"]), crs=\"EPSG:4326\"\n",
")\n",
"\n",
"geojson = json.loads(gdf.to_json())\n",
"data = df.drop(columns=['geometry']).to_dict('records')"
"data = df.drop(columns=[\"geometry\"]).to_dict(\"records\")"
]
},
{
@@ -101,7 +105,9 @@
"metadata": {},
"outputs": [],
"source": [
"df[['neighbourhood_name', 'median_household_income', 'income_index', 'income_quintile']].head(10)"
"df[\n",
" [\"neighbourhood_name\", \"median_household_income\", \"income_index\", \"income_quintile\"]\n",
"].head(10)"
]
},
{
@@ -112,7 +118,7 @@
"\n",
"### Figure Factory\n",
"\n",
"Uses `create_choropleth_figure` from `portfolio_app.figures.choropleth`."
"Uses `create_choropleth_figure` from `portfolio_app.figures.toronto.choropleth`."
]
},
{
@@ -122,18 +128,19 @@
"outputs": [],
"source": [
"import sys\n",
"sys.path.insert(0, '../..')\n",
"\n",
"from portfolio_app.figures.choropleth import create_choropleth_figure\n",
"sys.path.insert(0, \"../..\")\n",
"\n",
"from portfolio_app.figures.toronto.choropleth import create_choropleth_figure\n",
"\n",
"fig = create_choropleth_figure(\n",
" geojson=geojson,\n",
" data=data,\n",
" location_key='neighbourhood_id',\n",
" color_column='median_household_income',\n",
" hover_data=['neighbourhood_name', 'income_index', 'income_quintile'],\n",
" color_scale='Viridis',\n",
" title='Toronto Median Household Income by Neighbourhood',\n",
" location_key=\"neighbourhood_id\",\n",
" color_column=\"median_household_income\",\n",
" hover_data=[\"neighbourhood_name\", \"income_index\", \"income_quintile\"],\n",
" color_scale=\"Viridis\",\n",
" title=\"Toronto Median Household Income by Neighbourhood\",\n",
" zoom=10,\n",
")\n",
"\n",
@@ -153,7 +160,9 @@
"metadata": {},
"outputs": [],
"source": [
"df.groupby('income_quintile')['median_household_income'].agg(['count', 'mean', 'min', 'max']).round(0)"
"df.groupby(\"income_quintile\")[\"median_household_income\"].agg(\n",
" [\"count\", \"mean\", \"min\", \"max\"]\n",
").round(0)"
]
}
],

View File

@@ -30,11 +30,16 @@
"metadata": {},
"outputs": [],
"source": [
"import pandas as pd\n",
"from sqlalchemy import create_engine\n",
"import os\n",
"\n",
"engine = create_engine(os.environ.get('DATABASE_URL', 'postgresql://portfolio:portfolio@localhost:5432/portfolio'))\n",
"import pandas as pd\n",
"from dotenv import load_dotenv\n",
"from sqlalchemy import create_engine\n",
"\n",
"# Load .env from project root\n",
"load_dotenv(\"../../.env\")\n",
"\n",
"engine = create_engine(os.environ[\"DATABASE_URL\"])\n",
"\n",
"query = \"\"\"\n",
"SELECT\n",
@@ -44,8 +49,8 @@
" land_area_sqkm,\n",
" median_household_income,\n",
" pct_renter_occupied\n",
"FROM mart_neighbourhood_demographics\n",
"WHERE year = (SELECT MAX(year) FROM mart_neighbourhood_demographics)\n",
"FROM public_marts.mart_neighbourhood_demographics\n",
"WHERE year = (SELECT MAX(year) FROM public_marts.mart_neighbourhood_demographics)\n",
" AND population_density IS NOT NULL\n",
"ORDER BY population_density DESC\n",
"\"\"\"\n",
@@ -70,7 +75,7 @@
"metadata": {},
"outputs": [],
"source": [
"data = df.head(20).to_dict('records')"
"data = df.head(20).to_dict(\"records\")"
]
},
{
@@ -86,7 +91,9 @@
"metadata": {},
"outputs": [],
"source": [
"df[['neighbourhood_name', 'population_density', 'population', 'land_area_sqkm']].head(10)"
"df[[\"neighbourhood_name\", \"population_density\", \"population\", \"land_area_sqkm\"]].head(\n",
" 10\n",
")"
]
},
{
@@ -97,7 +104,7 @@
"\n",
"### Figure Factory\n",
"\n",
"Uses `create_horizontal_bar` from `portfolio_app.figures.bar_charts`."
"Uses `create_horizontal_bar` from `portfolio_app.figures.toronto.bar_charts`."
]
},
{
@@ -107,17 +114,18 @@
"outputs": [],
"source": [
"import sys\n",
"sys.path.insert(0, '../..')\n",
"\n",
"from portfolio_app.figures.bar_charts import create_horizontal_bar\n",
"sys.path.insert(0, \"../..\")\n",
"\n",
"from portfolio_app.figures.toronto.bar_charts import create_horizontal_bar\n",
"\n",
"fig = create_horizontal_bar(\n",
" data=data,\n",
" name_column='neighbourhood_name',\n",
" value_column='population_density',\n",
" title='Top 20 Most Dense Neighbourhoods',\n",
" color='#9C27B0',\n",
" value_format=',.0f',\n",
" name_column=\"neighbourhood_name\",\n",
" value_column=\"population_density\",\n",
" title=\"Top 20 Most Dense Neighbourhoods\",\n",
" color=\"#9C27B0\",\n",
" value_format=\",.0f\",\n",
")\n",
"\n",
"fig.show()"
@@ -136,7 +144,7 @@
"metadata": {},
"outputs": [],
"source": [
"print(f\"City-wide Statistics:\")\n",
"print(\"City-wide Statistics:\")\n",
"print(f\" Total Population: {df['population'].sum():,.0f}\")\n",
"print(f\" Total Area: {df['land_area_sqkm'].sum():,.1f} sq km\")\n",
"print(f\" Average Density: {df['population_density'].mean():,.0f} per sq km\")\n",

View File

@@ -30,11 +30,16 @@
"metadata": {},
"outputs": [],
"source": [
"import pandas as pd\n",
"from sqlalchemy import create_engine\n",
"import os\n",
"\n",
"engine = create_engine(os.environ.get('DATABASE_URL', 'postgresql://portfolio:portfolio@localhost:5432/portfolio'))\n",
"import pandas as pd\n",
"from dotenv import load_dotenv\n",
"from sqlalchemy import create_engine\n",
"\n",
"# Load .env from project root\n",
"load_dotenv(\"../../.env\")\n",
"\n",
"engine = create_engine(os.environ[\"DATABASE_URL\"])\n",
"\n",
"query = \"\"\"\n",
"SELECT\n",
@@ -47,8 +52,8 @@
" avg_rent_2bed,\n",
" median_household_income,\n",
" is_affordable\n",
"FROM mart_neighbourhood_housing\n",
"WHERE year = (SELECT MAX(year) FROM mart_neighbourhood_housing)\n",
"FROM public_marts.mart_neighbourhood_housing\n",
"WHERE year = (SELECT MAX(year) FROM public_marts.mart_neighbourhood_housing)\n",
"ORDER BY affordability_index ASC\n",
"\"\"\"\n",
"\n",
@@ -73,17 +78,16 @@
"metadata": {},
"outputs": [],
"source": [
"import geopandas as gpd\n",
"import json\n",
"\n",
"import geopandas as gpd\n",
"\n",
"gdf = gpd.GeoDataFrame(\n",
" df,\n",
" geometry=gpd.GeoSeries.from_wkb(df['geometry']),\n",
" crs='EPSG:4326'\n",
" df, geometry=gpd.GeoSeries.from_wkb(df[\"geometry\"]), crs=\"EPSG:4326\"\n",
")\n",
"\n",
"geojson = json.loads(gdf.to_json())\n",
"data = df.drop(columns=['geometry']).to_dict('records')"
"data = df.drop(columns=[\"geometry\"]).to_dict(\"records\")"
]
},
{
@@ -99,7 +103,15 @@
"metadata": {},
"outputs": [],
"source": [
"df[['neighbourhood_name', 'affordability_index', 'rent_to_income_pct', 'avg_rent_2bed', 'is_affordable']].head(10)"
"df[\n",
" [\n",
" \"neighbourhood_name\",\n",
" \"affordability_index\",\n",
" \"rent_to_income_pct\",\n",
" \"avg_rent_2bed\",\n",
" \"is_affordable\",\n",
" ]\n",
"].head(10)"
]
},
{
@@ -110,7 +122,7 @@
"\n",
"### Figure Factory\n",
"\n",
"Uses `create_choropleth_figure` from `portfolio_app.figures.choropleth`.\n",
"Uses `create_choropleth_figure` from `portfolio_app.figures.toronto.choropleth`.\n",
"\n",
"**Key Parameters:**\n",
"- `color_column`: 'affordability_index'\n",
@@ -124,18 +136,19 @@
"outputs": [],
"source": [
"import sys\n",
"sys.path.insert(0, '../..')\n",
"\n",
"from portfolio_app.figures.choropleth import create_choropleth_figure\n",
"sys.path.insert(0, \"../..\")\n",
"\n",
"from portfolio_app.figures.toronto.choropleth import create_choropleth_figure\n",
"\n",
"fig = create_choropleth_figure(\n",
" geojson=geojson,\n",
" data=data,\n",
" location_key='neighbourhood_id',\n",
" color_column='affordability_index',\n",
" hover_data=['neighbourhood_name', 'rent_to_income_pct', 'avg_rent_2bed'],\n",
" color_scale='RdYlGn_r', # Reversed: lower index (affordable) = green\n",
" title='Toronto Housing Affordability Index',\n",
" location_key=\"neighbourhood_id\",\n",
" color_column=\"affordability_index\",\n",
" hover_data=[\"neighbourhood_name\", \"rent_to_income_pct\", \"avg_rent_2bed\"],\n",
" color_scale=\"RdYlGn_r\", # Reversed: lower index (affordable) = green\n",
" title=\"Toronto Housing Affordability Index\",\n",
" zoom=10,\n",
")\n",
"\n",

View File

@@ -30,11 +30,16 @@
"metadata": {},
"outputs": [],
"source": [
"import pandas as pd\n",
"from sqlalchemy import create_engine\n",
"import os\n",
"\n",
"engine = create_engine(os.environ.get('DATABASE_URL', 'postgresql://portfolio:portfolio@localhost:5432/portfolio'))\n",
"import pandas as pd\n",
"from dotenv import load_dotenv\n",
"from sqlalchemy import create_engine\n",
"\n",
"# Load .env from project root\n",
"load_dotenv(\"../../.env\")\n",
"\n",
"engine = create_engine(os.environ[\"DATABASE_URL\"])\n",
"\n",
"# City-wide average rent by year\n",
"query = \"\"\"\n",
@@ -45,8 +50,8 @@
" AVG(avg_rent_2bed) as avg_rent_2bed,\n",
" AVG(avg_rent_3bed) as avg_rent_3bed,\n",
" AVG(rent_yoy_change_pct) as avg_yoy_change\n",
"FROM mart_neighbourhood_housing\n",
"WHERE year >= (SELECT MAX(year) - 5 FROM mart_neighbourhood_housing)\n",
"FROM public_marts.mart_neighbourhood_housing\n",
"WHERE year >= (SELECT MAX(year) - 5 FROM public_marts.mart_neighbourhood_housing)\n",
"GROUP BY year\n",
"ORDER BY year\n",
"\"\"\"\n",
@@ -73,23 +78,25 @@
"outputs": [],
"source": [
"# Create date column from year\n",
"df['date'] = pd.to_datetime(df['year'].astype(str) + '-01-01')\n",
"df[\"date\"] = pd.to_datetime(df[\"year\"].astype(str) + \"-01-01\")\n",
"\n",
"# Melt for multi-line chart\n",
"df_melted = df.melt(\n",
" id_vars=['year', 'date'],\n",
" value_vars=['avg_rent_bachelor', 'avg_rent_1bed', 'avg_rent_2bed', 'avg_rent_3bed'],\n",
" var_name='bedroom_type',\n",
" value_name='avg_rent'\n",
" id_vars=[\"year\", \"date\"],\n",
" value_vars=[\"avg_rent_bachelor\", \"avg_rent_1bed\", \"avg_rent_2bed\", \"avg_rent_3bed\"],\n",
" var_name=\"bedroom_type\",\n",
" value_name=\"avg_rent\",\n",
")\n",
"\n",
"# Clean labels\n",
"df_melted['bedroom_type'] = df_melted['bedroom_type'].map({\n",
" 'avg_rent_bachelor': 'Bachelor',\n",
" 'avg_rent_1bed': '1 Bedroom',\n",
" 'avg_rent_2bed': '2 Bedroom',\n",
" 'avg_rent_3bed': '3 Bedroom'\n",
"})"
"df_melted[\"bedroom_type\"] = df_melted[\"bedroom_type\"].map(\n",
" {\n",
" \"avg_rent_bachelor\": \"Bachelor\",\n",
" \"avg_rent_1bed\": \"1 Bedroom\",\n",
" \"avg_rent_2bed\": \"2 Bedroom\",\n",
" \"avg_rent_3bed\": \"3 Bedroom\",\n",
" }\n",
")"
]
},
{
@@ -105,7 +112,16 @@
"metadata": {},
"outputs": [],
"source": [
"df[['year', 'avg_rent_bachelor', 'avg_rent_1bed', 'avg_rent_2bed', 'avg_rent_3bed', 'avg_yoy_change']]"
"df[\n",
" [\n",
" \"year\",\n",
" \"avg_rent_bachelor\",\n",
" \"avg_rent_1bed\",\n",
" \"avg_rent_2bed\",\n",
" \"avg_rent_3bed\",\n",
" \"avg_yoy_change\",\n",
" ]\n",
"]"
]
},
{
@@ -116,7 +132,7 @@
"\n",
"### Figure Factory\n",
"\n",
"Uses `create_price_time_series` from `portfolio_app.figures.time_series`.\n",
"Uses `create_price_time_series` from `portfolio_app.figures.toronto.time_series`.\n",
"\n",
"**Key Parameters:**\n",
"- `date_column`: 'date'\n",
@@ -131,18 +147,19 @@
"outputs": [],
"source": [
"import sys\n",
"sys.path.insert(0, '../..')\n",
"\n",
"from portfolio_app.figures.time_series import create_price_time_series\n",
"sys.path.insert(0, \"../..\")\n",
"\n",
"data = df_melted.to_dict('records')\n",
"from portfolio_app.figures.toronto.time_series import create_price_time_series\n",
"\n",
"data = df_melted.to_dict(\"records\")\n",
"\n",
"fig = create_price_time_series(\n",
" data=data,\n",
" date_column='date',\n",
" price_column='avg_rent',\n",
" group_column='bedroom_type',\n",
" title='Toronto Average Rent Trend (5 Years)',\n",
" date_column=\"date\",\n",
" price_column=\"avg_rent\",\n",
" group_column=\"bedroom_type\",\n",
" title=\"Toronto Average Rent Trend (5 Years)\",\n",
")\n",
"\n",
"fig.show()"
@@ -163,7 +180,7 @@
"source": [
"# Show year-over-year changes\n",
"print(\"Year-over-Year Rent Change (%)\")\n",
"df[['year', 'avg_yoy_change']].dropna()"
"df[[\"year\", \"avg_yoy_change\"]].dropna()"
]
}
],

View File

@@ -30,11 +30,16 @@
"metadata": {},
"outputs": [],
"source": [
"import pandas as pd\n",
"from sqlalchemy import create_engine\n",
"import os\n",
"\n",
"engine = create_engine(os.environ.get('DATABASE_URL', 'postgresql://portfolio:portfolio@localhost:5432/portfolio'))\n",
"import pandas as pd\n",
"from dotenv import load_dotenv\n",
"from sqlalchemy import create_engine\n",
"\n",
"# Load .env from project root\n",
"load_dotenv(\"../../.env\")\n",
"\n",
"engine = create_engine(os.environ[\"DATABASE_URL\"])\n",
"\n",
"query = \"\"\"\n",
"SELECT\n",
@@ -44,8 +49,8 @@
" income_quintile,\n",
" total_rental_units,\n",
" average_dwelling_value\n",
"FROM mart_neighbourhood_housing\n",
"WHERE year = (SELECT MAX(year) FROM mart_neighbourhood_housing)\n",
"FROM public_marts.mart_neighbourhood_housing\n",
"WHERE year = (SELECT MAX(year) FROM public_marts.mart_neighbourhood_housing)\n",
" AND pct_owner_occupied IS NOT NULL\n",
"ORDER BY pct_renter_occupied DESC\n",
"\"\"\"\n",
@@ -73,18 +78,17 @@
"source": [
"# Prepare for stacked bar\n",
"df_stacked = df.melt(\n",
" id_vars=['neighbourhood_name', 'income_quintile'],\n",
" value_vars=['pct_owner_occupied', 'pct_renter_occupied'],\n",
" var_name='tenure_type',\n",
" value_name='percentage'\n",
" id_vars=[\"neighbourhood_name\", \"income_quintile\"],\n",
" value_vars=[\"pct_owner_occupied\", \"pct_renter_occupied\"],\n",
" var_name=\"tenure_type\",\n",
" value_name=\"percentage\",\n",
")\n",
"\n",
"df_stacked['tenure_type'] = df_stacked['tenure_type'].map({\n",
" 'pct_owner_occupied': 'Owner',\n",
" 'pct_renter_occupied': 'Renter'\n",
"})\n",
"df_stacked[\"tenure_type\"] = df_stacked[\"tenure_type\"].map(\n",
" {\"pct_owner_occupied\": \"Owner\", \"pct_renter_occupied\": \"Renter\"}\n",
")\n",
"\n",
"data = df_stacked.to_dict('records')"
"data = df_stacked.to_dict(\"records\")"
]
},
{
@@ -101,7 +105,14 @@
"outputs": [],
"source": [
"print(\"Highest Renter Neighbourhoods:\")\n",
"df[['neighbourhood_name', 'pct_renter_occupied', 'pct_owner_occupied', 'income_quintile']].head(10)"
"df[\n",
" [\n",
" \"neighbourhood_name\",\n",
" \"pct_renter_occupied\",\n",
" \"pct_owner_occupied\",\n",
" \"income_quintile\",\n",
" ]\n",
"].head(10)"
]
},
{
@@ -112,7 +123,7 @@
"\n",
"### Figure Factory\n",
"\n",
"Uses `create_stacked_bar` from `portfolio_app.figures.bar_charts`.\n",
"Uses `create_stacked_bar` from `portfolio_app.figures.toronto.bar_charts`.\n",
"\n",
"**Key Parameters:**\n",
"- `x_column`: 'neighbourhood_name'\n",
@@ -128,21 +139,22 @@
"outputs": [],
"source": [
"import sys\n",
"sys.path.insert(0, '../..')\n",
"\n",
"from portfolio_app.figures.bar_charts import create_stacked_bar\n",
"sys.path.insert(0, \"../..\")\n",
"\n",
"from portfolio_app.figures.toronto.bar_charts import create_stacked_bar\n",
"\n",
"# Show top 20 by renter percentage\n",
"top_20_names = df.head(20)['neighbourhood_name'].tolist()\n",
"data_filtered = [d for d in data if d['neighbourhood_name'] in top_20_names]\n",
"top_20_names = df.head(20)[\"neighbourhood_name\"].tolist()\n",
"data_filtered = [d for d in data if d[\"neighbourhood_name\"] in top_20_names]\n",
"\n",
"fig = create_stacked_bar(\n",
" data=data_filtered,\n",
" x_column='neighbourhood_name',\n",
" value_column='percentage',\n",
" category_column='tenure_type',\n",
" title='Housing Tenure Mix - Top 20 Renter Neighbourhoods',\n",
" color_map={'Owner': '#4CAF50', 'Renter': '#2196F3'},\n",
" x_column=\"neighbourhood_name\",\n",
" value_column=\"percentage\",\n",
" category_column=\"tenure_type\",\n",
" title=\"Housing Tenure Mix - Top 20 Renter Neighbourhoods\",\n",
" color_map={\"Owner\": \"#4CAF50\", \"Renter\": \"#2196F3\"},\n",
" show_percentages=True,\n",
")\n",
"\n",
@@ -168,7 +180,9 @@
"\n",
"# By income quintile\n",
"print(\"\\nTenure by Income Quintile:\")\n",
"df.groupby('income_quintile')[['pct_owner_occupied', 'pct_renter_occupied']].mean().round(1)"
"df.groupby(\"income_quintile\")[\n",
" [\"pct_owner_occupied\", \"pct_renter_occupied\"]\n",
"].mean().round(1)"
]
}
],

View File

@@ -30,11 +30,16 @@
"metadata": {},
"outputs": [],
"source": [
"import pandas as pd\n",
"from sqlalchemy import create_engine\n",
"import os\n",
"\n",
"engine = create_engine(os.environ.get('DATABASE_URL', 'postgresql://portfolio:portfolio@localhost:5432/portfolio'))\n",
"import pandas as pd\n",
"from dotenv import load_dotenv\n",
"from sqlalchemy import create_engine\n",
"\n",
"# Load .env from project root\n",
"load_dotenv(\"../../.env\")\n",
"\n",
"engine = create_engine(os.environ[\"DATABASE_URL\"])\n",
"\n",
"query = \"\"\"\n",
"SELECT\n",
@@ -44,8 +49,8 @@
" population,\n",
" livability_score,\n",
" crime_rate_per_100k\n",
"FROM mart_neighbourhood_overview\n",
"WHERE year = (SELECT MAX(year) FROM mart_neighbourhood_overview)\n",
"FROM public_marts.mart_neighbourhood_overview\n",
"WHERE year = (SELECT MAX(year) FROM public_marts.mart_neighbourhood_overview)\n",
" AND median_household_income IS NOT NULL\n",
" AND safety_score IS NOT NULL\n",
"ORDER BY median_household_income DESC\n",
@@ -73,10 +78,10 @@
"outputs": [],
"source": [
"# Scale income to thousands for better axis readability\n",
"df['income_thousands'] = df['median_household_income'] / 1000\n",
"df[\"income_thousands\"] = df[\"median_household_income\"] / 1000\n",
"\n",
"# Prepare data for figure factory\n",
"data = df.to_dict('records')"
"data = df.to_dict(\"records\")"
]
},
{
@@ -92,7 +97,14 @@
"metadata": {},
"outputs": [],
"source": [
"df[['neighbourhood_name', 'median_household_income', 'safety_score', 'crime_rate_per_100k']].head(10)"
"df[\n",
" [\n",
" \"neighbourhood_name\",\n",
" \"median_household_income\",\n",
" \"safety_score\",\n",
" \"crime_rate_per_100k\",\n",
" ]\n",
"].head(10)"
]
},
{
@@ -103,7 +115,7 @@
"\n",
"### Figure Factory\n",
"\n",
"Uses `create_scatter_figure` from `portfolio_app.figures.scatter`.\n",
"Uses `create_scatter_figure` from `portfolio_app.figures.toronto.scatter`.\n",
"\n",
"**Key Parameters:**\n",
"- `x_column`: 'income_thousands' (median household income in $K)\n",
@@ -120,19 +132,20 @@
"outputs": [],
"source": [
"import sys\n",
"sys.path.insert(0, '../..')\n",
"\n",
"from portfolio_app.figures.scatter import create_scatter_figure\n",
"sys.path.insert(0, \"../..\")\n",
"\n",
"from portfolio_app.figures.toronto.scatter import create_scatter_figure\n",
"\n",
"fig = create_scatter_figure(\n",
" data=data,\n",
" x_column='income_thousands',\n",
" y_column='safety_score',\n",
" name_column='neighbourhood_name',\n",
" size_column='population',\n",
" title='Income vs Safety by Neighbourhood',\n",
" x_title='Median Household Income ($K)',\n",
" y_title='Safety Score (0-100)',\n",
" x_column=\"income_thousands\",\n",
" y_column=\"safety_score\",\n",
" name_column=\"neighbourhood_name\",\n",
" size_column=\"population\",\n",
" title=\"Income vs Safety by Neighbourhood\",\n",
" x_title=\"Median Household Income ($K)\",\n",
" y_title=\"Safety Score (0-100)\",\n",
" trendline=True,\n",
")\n",
"\n",
@@ -162,7 +175,7 @@
"outputs": [],
"source": [
"# Calculate correlation coefficient\n",
"correlation = df['median_household_income'].corr(df['safety_score'])\n",
"correlation = df[\"median_household_income\"].corr(df[\"safety_score\"])\n",
"print(f\"Correlation coefficient (Income vs Safety): {correlation:.3f}\")"
]
}

View File

@@ -30,12 +30,16 @@
"metadata": {},
"outputs": [],
"source": [
"import pandas as pd\n",
"from sqlalchemy import create_engine\n",
"import os\n",
"\n",
"# Connect to database\n",
"engine = create_engine(os.environ.get('DATABASE_URL', 'postgresql://portfolio:portfolio@localhost:5432/portfolio'))\n",
"import pandas as pd\n",
"from dotenv import load_dotenv\n",
"from sqlalchemy import create_engine\n",
"\n",
"# Load .env from project root\n",
"load_dotenv(\"../../.env\")\n",
"\n",
"engine = create_engine(os.environ[\"DATABASE_URL\"])\n",
"\n",
"query = \"\"\"\n",
"SELECT\n",
@@ -49,8 +53,8 @@
" amenity_score,\n",
" population,\n",
" median_household_income\n",
"FROM mart_neighbourhood_overview\n",
"WHERE year = (SELECT MAX(year) FROM mart_neighbourhood_overview)\n",
"FROM public_marts.mart_neighbourhood_overview\n",
"WHERE year = (SELECT MAX(year) FROM public_marts.mart_neighbourhood_overview)\n",
"ORDER BY livability_score DESC\n",
"\"\"\"\n",
"\n",
@@ -76,21 +80,20 @@
"outputs": [],
"source": [
"# Transform geometry to GeoJSON\n",
"import geopandas as gpd\n",
"import json\n",
"\n",
"import geopandas as gpd\n",
"\n",
"# Convert WKB geometry to GeoDataFrame\n",
"gdf = gpd.GeoDataFrame(\n",
" df,\n",
" geometry=gpd.GeoSeries.from_wkb(df['geometry']),\n",
" crs='EPSG:4326'\n",
" df, geometry=gpd.GeoSeries.from_wkb(df[\"geometry\"]), crs=\"EPSG:4326\"\n",
")\n",
"\n",
"# Create GeoJSON FeatureCollection\n",
"geojson = json.loads(gdf.to_json())\n",
"\n",
"# Prepare data for figure factory\n",
"data = df.drop(columns=['geometry']).to_dict('records')"
"data = df.drop(columns=[\"geometry\"]).to_dict(\"records\")"
]
},
{
@@ -106,7 +109,15 @@
"metadata": {},
"outputs": [],
"source": [
"df[['neighbourhood_name', 'livability_score', 'safety_score', 'affordability_score', 'amenity_score']].head(10)"
"df[\n",
" [\n",
" \"neighbourhood_name\",\n",
" \"livability_score\",\n",
" \"safety_score\",\n",
" \"affordability_score\",\n",
" \"amenity_score\",\n",
" ]\n",
"].head(10)"
]
},
{
@@ -117,7 +128,7 @@
"\n",
"### Figure Factory\n",
"\n",
"Uses `create_choropleth_figure` from `portfolio_app.figures.choropleth`.\n",
"Uses `create_choropleth_figure` from `portfolio_app.figures.toronto.choropleth`.\n",
"\n",
"**Key Parameters:**\n",
"- `geojson`: GeoJSON FeatureCollection with neighbourhood boundaries\n",
@@ -134,18 +145,24 @@
"outputs": [],
"source": [
"import sys\n",
"sys.path.insert(0, '../..')\n",
"\n",
"from portfolio_app.figures.choropleth import create_choropleth_figure\n",
"sys.path.insert(0, \"../..\")\n",
"\n",
"from portfolio_app.figures.toronto.choropleth import create_choropleth_figure\n",
"\n",
"fig = create_choropleth_figure(\n",
" geojson=geojson,\n",
" data=data,\n",
" location_key='neighbourhood_id',\n",
" color_column='livability_score',\n",
" hover_data=['neighbourhood_name', 'safety_score', 'affordability_score', 'amenity_score'],\n",
" color_scale='RdYlGn',\n",
" title='Toronto Neighbourhood Livability Score',\n",
" location_key=\"neighbourhood_id\",\n",
" color_column=\"livability_score\",\n",
" hover_data=[\n",
" \"neighbourhood_name\",\n",
" \"safety_score\",\n",
" \"affordability_score\",\n",
" \"amenity_score\",\n",
" ],\n",
" color_scale=\"RdYlGn\",\n",
" title=\"Toronto Neighbourhood Livability Score\",\n",
" zoom=10,\n",
")\n",
"\n",

View File

@@ -30,11 +30,16 @@
"metadata": {},
"outputs": [],
"source": [
"import pandas as pd\n",
"from sqlalchemy import create_engine\n",
"import os\n",
"\n",
"engine = create_engine(os.environ.get('DATABASE_URL', 'postgresql://portfolio:portfolio@localhost:5432/portfolio'))\n",
"import pandas as pd\n",
"from dotenv import load_dotenv\n",
"from sqlalchemy import create_engine\n",
"\n",
"# Load .env from project root\n",
"load_dotenv(\"../../.env\")\n",
"\n",
"engine = create_engine(os.environ[\"DATABASE_URL\"])\n",
"\n",
"query = \"\"\"\n",
"SELECT\n",
@@ -43,8 +48,8 @@
" safety_score,\n",
" affordability_score,\n",
" amenity_score\n",
"FROM mart_neighbourhood_overview\n",
"WHERE year = (SELECT MAX(year) FROM mart_neighbourhood_overview)\n",
"FROM public_marts.mart_neighbourhood_overview\n",
"WHERE year = (SELECT MAX(year) FROM public_marts.mart_neighbourhood_overview)\n",
" AND livability_score IS NOT NULL\n",
"ORDER BY livability_score DESC\n",
"\"\"\"\n",
@@ -72,7 +77,7 @@
"source": [
"# The figure factory handles top/bottom selection internally\n",
"# Just prepare as list of dicts\n",
"data = df.to_dict('records')"
"data = df.to_dict(\"records\")"
]
},
{
@@ -102,7 +107,7 @@
"\n",
"### Figure Factory\n",
"\n",
"Uses `create_ranking_bar` from `portfolio_app.figures.bar_charts`.\n",
"Uses `create_ranking_bar` from `portfolio_app.figures.toronto.bar_charts`.\n",
"\n",
"**Key Parameters:**\n",
"- `data`: List of dicts with all neighbourhoods\n",
@@ -119,20 +124,21 @@
"outputs": [],
"source": [
"import sys\n",
"sys.path.insert(0, '../..')\n",
"\n",
"from portfolio_app.figures.bar_charts import create_ranking_bar\n",
"sys.path.insert(0, \"../..\")\n",
"\n",
"from portfolio_app.figures.toronto.bar_charts import create_ranking_bar\n",
"\n",
"fig = create_ranking_bar(\n",
" data=data,\n",
" name_column='neighbourhood_name',\n",
" value_column='livability_score',\n",
" title='Top & Bottom 10 Neighbourhoods by Livability',\n",
" name_column=\"neighbourhood_name\",\n",
" value_column=\"livability_score\",\n",
" title=\"Top & Bottom 10 Neighbourhoods by Livability\",\n",
" top_n=10,\n",
" bottom_n=10,\n",
" color_top='#4CAF50', # Green for top performers\n",
" color_bottom='#F44336', # Red for bottom performers\n",
" value_format='.1f',\n",
" color_top=\"#4CAF50\", # Green for top performers\n",
" color_bottom=\"#F44336\", # Red for bottom performers\n",
" value_format=\".1f\",\n",
")\n",
"\n",
"fig.show()"

View File

View File

@@ -30,11 +30,16 @@
"metadata": {},
"outputs": [],
"source": [
"import pandas as pd\n",
"from sqlalchemy import create_engine\n",
"import os\n",
"\n",
"engine = create_engine(os.environ.get('DATABASE_URL', 'postgresql://portfolio:portfolio@localhost:5432/portfolio'))\n",
"import pandas as pd\n",
"from dotenv import load_dotenv\n",
"from sqlalchemy import create_engine\n",
"\n",
"# Load .env from project root\n",
"load_dotenv(\"../../.env\")\n",
"\n",
"engine = create_engine(os.environ[\"DATABASE_URL\"])\n",
"\n",
"query = \"\"\"\n",
"SELECT\n",
@@ -47,8 +52,8 @@
" homicide_count,\n",
" total_incidents,\n",
" crime_rate_per_100k\n",
"FROM mart_neighbourhood_safety\n",
"WHERE year = (SELECT MAX(year) FROM mart_neighbourhood_safety)\n",
"FROM public_marts.mart_neighbourhood_safety\n",
"WHERE year = (SELECT MAX(year) FROM public_marts.mart_neighbourhood_safety)\n",
"ORDER BY total_incidents DESC\n",
"LIMIT 15\n",
"\"\"\"\n",
@@ -75,17 +80,25 @@
"outputs": [],
"source": [
"df_melted = df.melt(\n",
" id_vars=['neighbourhood_name', 'total_incidents'],\n",
" value_vars=['assault_count', 'auto_theft_count', 'break_enter_count', \n",
" 'robbery_count', 'theft_over_count', 'homicide_count'],\n",
" var_name='crime_type',\n",
" value_name='count'\n",
" id_vars=[\"neighbourhood_name\", \"total_incidents\"],\n",
" value_vars=[\n",
" \"assault_count\",\n",
" \"auto_theft_count\",\n",
" \"break_enter_count\",\n",
" \"robbery_count\",\n",
" \"theft_over_count\",\n",
" \"homicide_count\",\n",
" ],\n",
" var_name=\"crime_type\",\n",
" value_name=\"count\",\n",
")\n",
"\n",
"# Clean labels\n",
"df_melted['crime_type'] = df_melted['crime_type'].str.replace('_count', '').str.replace('_', ' ').str.title()\n",
"df_melted[\"crime_type\"] = (\n",
" df_melted[\"crime_type\"].str.replace(\"_count\", \"\").str.replace(\"_\", \" \").str.title()\n",
")\n",
"\n",
"data = df_melted.to_dict('records')"
"data = df_melted.to_dict(\"records\")"
]
},
{
@@ -101,7 +114,15 @@
"metadata": {},
"outputs": [],
"source": [
"df[['neighbourhood_name', 'assault_count', 'auto_theft_count', 'break_enter_count', 'total_incidents']].head(10)"
"df[\n",
" [\n",
" \"neighbourhood_name\",\n",
" \"assault_count\",\n",
" \"auto_theft_count\",\n",
" \"break_enter_count\",\n",
" \"total_incidents\",\n",
" ]\n",
"].head(10)"
]
},
{
@@ -112,7 +133,7 @@
"\n",
"### Figure Factory\n",
"\n",
"Uses `create_stacked_bar` from `portfolio_app.figures.bar_charts`."
"Uses `create_stacked_bar` from `portfolio_app.figures.toronto.bar_charts`."
]
},
{
@@ -122,23 +143,24 @@
"outputs": [],
"source": [
"import sys\n",
"sys.path.insert(0, '../..')\n",
"\n",
"from portfolio_app.figures.bar_charts import create_stacked_bar\n",
"sys.path.insert(0, \"../..\")\n",
"\n",
"from portfolio_app.figures.toronto.bar_charts import create_stacked_bar\n",
"\n",
"fig = create_stacked_bar(\n",
" data=data,\n",
" x_column='neighbourhood_name',\n",
" value_column='count',\n",
" category_column='crime_type',\n",
" title='Crime Type Breakdown - Top 15 Neighbourhoods',\n",
" x_column=\"neighbourhood_name\",\n",
" value_column=\"count\",\n",
" category_column=\"crime_type\",\n",
" title=\"Crime Type Breakdown - Top 15 Neighbourhoods\",\n",
" color_map={\n",
" 'Assault': '#d62728',\n",
" 'Auto Theft': '#ff7f0e',\n",
" 'Break Enter': '#9467bd',\n",
" 'Robbery': '#8c564b',\n",
" 'Theft Over': '#e377c2',\n",
" 'Homicide': '#1f77b4'\n",
" \"Assault\": \"#d62728\",\n",
" \"Auto Theft\": \"#ff7f0e\",\n",
" \"Break Enter\": \"#9467bd\",\n",
" \"Robbery\": \"#8c564b\",\n",
" \"Theft Over\": \"#e377c2\",\n",
" \"Homicide\": \"#1f77b4\",\n",
" },\n",
")\n",
"\n",

View File

@@ -30,11 +30,16 @@
"metadata": {},
"outputs": [],
"source": [
"import pandas as pd\n",
"from sqlalchemy import create_engine\n",
"import os\n",
"\n",
"engine = create_engine(os.environ.get('DATABASE_URL', 'postgresql://portfolio:portfolio@localhost:5432/portfolio'))\n",
"import pandas as pd\n",
"from dotenv import load_dotenv\n",
"from sqlalchemy import create_engine\n",
"\n",
"# Load .env from project root\n",
"load_dotenv(\"../../.env\")\n",
"\n",
"engine = create_engine(os.environ[\"DATABASE_URL\"])\n",
"\n",
"query = \"\"\"\n",
"SELECT\n",
@@ -47,8 +52,8 @@
" safety_tier,\n",
" total_incidents,\n",
" population\n",
"FROM mart_neighbourhood_safety\n",
"WHERE year = (SELECT MAX(year) FROM mart_neighbourhood_safety)\n",
"FROM public_marts.mart_neighbourhood_safety\n",
"WHERE year = (SELECT MAX(year) FROM public_marts.mart_neighbourhood_safety)\n",
"ORDER BY crime_rate_per_100k DESC\n",
"\"\"\"\n",
"\n",
@@ -73,17 +78,16 @@
"metadata": {},
"outputs": [],
"source": [
"import geopandas as gpd\n",
"import json\n",
"\n",
"import geopandas as gpd\n",
"\n",
"gdf = gpd.GeoDataFrame(\n",
" df,\n",
" geometry=gpd.GeoSeries.from_wkb(df['geometry']),\n",
" crs='EPSG:4326'\n",
" df, geometry=gpd.GeoSeries.from_wkb(df[\"geometry\"]), crs=\"EPSG:4326\"\n",
")\n",
"\n",
"geojson = json.loads(gdf.to_json())\n",
"data = df.drop(columns=['geometry']).to_dict('records')"
"data = df.drop(columns=[\"geometry\"]).to_dict(\"records\")"
]
},
{
@@ -99,7 +103,15 @@
"metadata": {},
"outputs": [],
"source": [
"df[['neighbourhood_name', 'crime_rate_per_100k', 'crime_index', 'safety_tier', 'total_incidents']].head(10)"
"df[\n",
" [\n",
" \"neighbourhood_name\",\n",
" \"crime_rate_per_100k\",\n",
" \"crime_index\",\n",
" \"safety_tier\",\n",
" \"total_incidents\",\n",
" ]\n",
"].head(10)"
]
},
{
@@ -110,7 +122,7 @@
"\n",
"### Figure Factory\n",
"\n",
"Uses `create_choropleth_figure` from `portfolio_app.figures.choropleth`.\n",
"Uses `create_choropleth_figure` from `portfolio_app.figures.toronto.choropleth`.\n",
"\n",
"**Key Parameters:**\n",
"- `color_column`: 'crime_rate_per_100k'\n",
@@ -124,18 +136,19 @@
"outputs": [],
"source": [
"import sys\n",
"sys.path.insert(0, '../..')\n",
"\n",
"from portfolio_app.figures.choropleth import create_choropleth_figure\n",
"sys.path.insert(0, \"../..\")\n",
"\n",
"from portfolio_app.figures.toronto.choropleth import create_choropleth_figure\n",
"\n",
"fig = create_choropleth_figure(\n",
" geojson=geojson,\n",
" data=data,\n",
" location_key='neighbourhood_id',\n",
" color_column='crime_rate_per_100k',\n",
" hover_data=['neighbourhood_name', 'crime_index', 'total_incidents'],\n",
" color_scale='RdYlGn_r',\n",
" title='Toronto Crime Rate per 100,000 Population',\n",
" location_key=\"neighbourhood_id\",\n",
" color_column=\"crime_rate_per_100k\",\n",
" hover_data=[\"neighbourhood_name\", \"crime_index\", \"total_incidents\"],\n",
" color_scale=\"RdYlGn_r\",\n",
" title=\"Toronto Crime Rate per 100,000 Population\",\n",
" zoom=10,\n",
")\n",
"\n",

View File

@@ -30,11 +30,16 @@
"metadata": {},
"outputs": [],
"source": [
"import pandas as pd\n",
"from sqlalchemy import create_engine\n",
"import os\n",
"\n",
"engine = create_engine(os.environ.get('DATABASE_URL', 'postgresql://portfolio:portfolio@localhost:5432/portfolio'))\n",
"import pandas as pd\n",
"from dotenv import load_dotenv\n",
"from sqlalchemy import create_engine\n",
"\n",
"# Load .env from project root\n",
"load_dotenv(\"../../.env\")\n",
"\n",
"engine = create_engine(os.environ[\"DATABASE_URL\"])\n",
"\n",
"query = \"\"\"\n",
"SELECT\n",
@@ -45,8 +50,8 @@
" AVG(break_enter_rate_per_100k) as avg_break_enter_rate,\n",
" SUM(total_incidents) as total_city_incidents,\n",
" AVG(crime_yoy_change_pct) as avg_yoy_change\n",
"FROM mart_neighbourhood_safety\n",
"WHERE year >= (SELECT MAX(year) - 5 FROM mart_neighbourhood_safety)\n",
"FROM public_marts.mart_neighbourhood_safety\n",
"WHERE year >= (SELECT MAX(year) - 5 FROM public_marts.mart_neighbourhood_safety)\n",
"GROUP BY year\n",
"ORDER BY year\n",
"\"\"\"\n",
@@ -72,21 +77,23 @@
"metadata": {},
"outputs": [],
"source": [
"df['date'] = pd.to_datetime(df['year'].astype(str) + '-01-01')\n",
"df[\"date\"] = pd.to_datetime(df[\"year\"].astype(str) + \"-01-01\")\n",
"\n",
"# Melt for multi-line\n",
"df_melted = df.melt(\n",
" id_vars=['year', 'date'],\n",
" value_vars=['avg_assault_rate', 'avg_auto_theft_rate', 'avg_break_enter_rate'],\n",
" var_name='crime_type',\n",
" value_name='rate_per_100k'\n",
" id_vars=[\"year\", \"date\"],\n",
" value_vars=[\"avg_assault_rate\", \"avg_auto_theft_rate\", \"avg_break_enter_rate\"],\n",
" var_name=\"crime_type\",\n",
" value_name=\"rate_per_100k\",\n",
")\n",
"\n",
"df_melted['crime_type'] = df_melted['crime_type'].map({\n",
" 'avg_assault_rate': 'Assault',\n",
" 'avg_auto_theft_rate': 'Auto Theft',\n",
" 'avg_break_enter_rate': 'Break & Enter'\n",
"})"
"df_melted[\"crime_type\"] = df_melted[\"crime_type\"].map(\n",
" {\n",
" \"avg_assault_rate\": \"Assault\",\n",
" \"avg_auto_theft_rate\": \"Auto Theft\",\n",
" \"avg_break_enter_rate\": \"Break & Enter\",\n",
" }\n",
")"
]
},
{
@@ -102,7 +109,7 @@
"metadata": {},
"outputs": [],
"source": [
"df[['year', 'avg_crime_rate', 'total_city_incidents', 'avg_yoy_change']]"
"df[[\"year\", \"avg_crime_rate\", \"total_city_incidents\", \"avg_yoy_change\"]]"
]
},
{
@@ -123,22 +130,23 @@
"outputs": [],
"source": [
"import sys\n",
"sys.path.insert(0, '../..')\n",
"\n",
"from portfolio_app.figures.time_series import create_price_time_series\n",
"sys.path.insert(0, \"../..\")\n",
"\n",
"data = df_melted.to_dict('records')\n",
"from portfolio_app.figures.toronto.time_series import create_price_time_series\n",
"\n",
"data = df_melted.to_dict(\"records\")\n",
"\n",
"fig = create_price_time_series(\n",
" data=data,\n",
" date_column='date',\n",
" price_column='rate_per_100k',\n",
" group_column='crime_type',\n",
" title='Toronto Crime Trends by Type (5 Years)',\n",
" date_column=\"date\",\n",
" price_column=\"rate_per_100k\",\n",
" group_column=\"crime_type\",\n",
" title=\"Toronto Crime Trends by Type (5 Years)\",\n",
")\n",
"\n",
"# Remove dollar sign formatting since this is rate data\n",
"fig.update_layout(yaxis_tickprefix='', yaxis_title='Rate per 100K')\n",
"fig.update_layout(yaxis_tickprefix=\"\", yaxis_title=\"Rate per 100K\")\n",
"\n",
"fig.show()"
]
@@ -157,15 +165,19 @@
"outputs": [],
"source": [
"# Total crime rate trend\n",
"total_data = df[['date', 'avg_crime_rate']].rename(columns={'avg_crime_rate': 'total_rate'}).to_dict('records')\n",
"total_data = (\n",
" df[[\"date\", \"avg_crime_rate\"]]\n",
" .rename(columns={\"avg_crime_rate\": \"total_rate\"})\n",
" .to_dict(\"records\")\n",
")\n",
"\n",
"fig2 = create_price_time_series(\n",
" data=total_data,\n",
" date_column='date',\n",
" price_column='total_rate',\n",
" title='Toronto Overall Crime Rate Trend',\n",
" date_column=\"date\",\n",
" price_column=\"total_rate\",\n",
" title=\"Toronto Overall Crime Rate Trend\",\n",
")\n",
"fig2.update_layout(yaxis_tickprefix='', yaxis_title='Rate per 100K')\n",
"fig2.update_layout(yaxis_tickprefix=\"\", yaxis_title=\"Rate per 100K\")\n",
"fig2.show()"
]
}

View File

@@ -1,5 +1,5 @@
"""Application-level callbacks for the portfolio app."""
from . import sidebar, theme
from . import contact, sidebar, theme
__all__ = ["sidebar", "theme"]
__all__ = ["contact", "sidebar", "theme"]

View File

@@ -0,0 +1,214 @@
"""Contact form submission callback with Formspree integration."""
import re
from typing import Any
import dash_mantine_components as dmc
import requests
from dash import Input, Output, State, callback, no_update
from dash_iconify import DashIconify
FORMSPREE_ENDPOINT = "https://formspree.io/f/mqelqzpd"
EMAIL_REGEX = re.compile(r"^[a-zA-Z0-9._%+-]+@[a-zA-Z0-9.-]+\.[a-zA-Z]{2,}$")
def _validate_form(
name: str | None, email: str | None, message: str | None
) -> str | None:
"""Validate form fields and return error message if invalid."""
if not name or not name.strip():
return "Please enter your name."
if not email or not email.strip():
return "Please enter your email address."
if not EMAIL_REGEX.match(email.strip()):
return "Please enter a valid email address."
if not message or not message.strip():
return "Please enter a message."
return None
def _create_success_alert() -> dmc.Alert:
"""Create success feedback alert."""
return dmc.Alert(
"Thank you for your message! I'll get back to you soon.",
title="Message Sent",
color="green",
variant="light",
icon=DashIconify(icon="tabler:check", width=20),
withCloseButton=True,
)
def _create_error_alert(message: str) -> dmc.Alert:
"""Create error feedback alert."""
return dmc.Alert(
message,
title="Error",
color="red",
variant="light",
icon=DashIconify(icon="tabler:alert-circle", width=20),
withCloseButton=True,
)
@callback( # type: ignore[misc]
Output("contact-feedback", "children"),
Output("contact-submit", "loading"),
Output("contact-name", "value"),
Output("contact-email", "value"),
Output("contact-subject", "value"),
Output("contact-message", "value"),
Output("contact-name", "error"),
Output("contact-email", "error"),
Output("contact-message", "error"),
Input("contact-submit", "n_clicks"),
State("contact-name", "value"),
State("contact-email", "value"),
State("contact-subject", "value"),
State("contact-message", "value"),
State("contact-gotcha", "value"),
prevent_initial_call=True,
)
def submit_contact_form(
n_clicks: int | None,
name: str | None,
email: str | None,
subject: str | None,
message: str | None,
gotcha: str | None,
) -> tuple[Any, ...]:
"""Submit contact form to Formspree.
Args:
n_clicks: Button click count.
name: User's name.
email: User's email address.
subject: Message subject (optional).
message: Message content.
gotcha: Honeypot field value (should be empty for real users).
Returns:
Tuple of (feedback, loading, name, email, subject, message,
name_error, email_error, message_error).
"""
if not n_clicks:
return (no_update,) * 9
# Check honeypot - if filled, silently "succeed" (it's a bot)
if gotcha:
return (
_create_success_alert(),
False,
"",
"",
None,
"",
None,
None,
None,
)
# Validate form
validation_error = _validate_form(name, email, message)
if validation_error:
# Determine which field has the error
name_error = "Required" if not name or not name.strip() else None
email_error = None
message_error = "Required" if not message or not message.strip() else None
if not email or not email.strip():
email_error = "Required"
elif not EMAIL_REGEX.match(email.strip()):
email_error = "Invalid email format"
return (
_create_error_alert(validation_error),
False,
no_update,
no_update,
no_update,
no_update,
name_error,
email_error,
message_error,
)
# Prepare form data (validation passed, so name/email/message are not None)
assert name is not None
assert email is not None
assert message is not None
form_data = {
"name": name.strip(),
"email": email.strip(),
"subject": subject or "General Inquiry",
"message": message.strip(),
"_gotcha": "", # Formspree honeypot
}
# Submit to Formspree
try:
response = requests.post(
FORMSPREE_ENDPOINT,
json=form_data,
headers={
"Accept": "application/json",
"Content-Type": "application/json",
},
timeout=10,
)
if response.status_code == 200:
# Success - clear form
return (
_create_success_alert(),
False,
"",
"",
None,
"",
None,
None,
None,
)
else:
# Formspree returned an error
return (
_create_error_alert(
"Failed to send message. Please try again or use direct contact."
),
False,
no_update,
no_update,
no_update,
no_update,
None,
None,
None,
)
except requests.exceptions.Timeout:
return (
_create_error_alert("Request timed out. Please try again."),
False,
no_update,
no_update,
no_update,
no_update,
None,
None,
None,
)
except requests.exceptions.RequestException:
return (
_create_error_alert(
"Network error. Please check your connection and try again."
),
False,
no_update,
no_update,
no_update,
no_update,
None,
None,
None,
)

View File

@@ -28,7 +28,7 @@ def create_metric_selector(
label=label,
data=options,
value=default_value or (options[0]["value"] if options else None),
style={"width": "200px"},
w=200,
)
@@ -64,7 +64,7 @@ def create_map_controls(
id=f"{id_prefix}-layer-toggle",
label="Show Boundaries",
checked=True,
style={"marginTop": "10px"},
mt="sm",
)
)

View File

@@ -5,7 +5,7 @@ from typing import Any
import dash_mantine_components as dmc
from dash import dcc
from portfolio_app.figures.summary_cards import create_metric_card_figure
from portfolio_app.figures.toronto.summary_cards import create_metric_card_figure
class MetricCard:

View File

@@ -38,7 +38,7 @@ def create_year_selector(
label=label,
data=options,
value=str(default_year),
style={"width": "120px"},
w=120,
)
@@ -83,7 +83,8 @@ def create_time_slider(
marks=marks,
step=1,
minRange=1,
style={"marginTop": "20px", "marginBottom": "10px"},
mt="md",
mb="sm",
),
],
p="md",
@@ -131,5 +132,5 @@ def create_month_selector(
label=label,
data=options,
value=str(default_month),
style={"width": "140px"},
w=140,
)

View File

@@ -0,0 +1,48 @@
"""Design system tokens and utilities."""
from .tokens import (
CHART_PALETTE,
COLOR_ACCENT,
COLOR_NEGATIVE,
COLOR_POSITIVE,
COLOR_WARNING,
GRID_COLOR,
GRID_COLOR_DARK,
PALETTE_COMPARISON,
PALETTE_GENDER,
PALETTE_TREND,
PAPER_BG,
PLOT_BG,
POLICY_COLORS,
TEXT_MUTED,
TEXT_PRIMARY,
TEXT_SECONDARY,
get_colorbar_defaults,
get_default_layout,
)
__all__ = [
# Text colors
"TEXT_PRIMARY",
"TEXT_SECONDARY",
"TEXT_MUTED",
# Chart backgrounds
"GRID_COLOR",
"GRID_COLOR_DARK",
"PAPER_BG",
"PLOT_BG",
# Semantic colors
"COLOR_POSITIVE",
"COLOR_NEGATIVE",
"COLOR_WARNING",
"COLOR_ACCENT",
# Palettes
"CHART_PALETTE",
"PALETTE_COMPARISON",
"PALETTE_GENDER",
"PALETTE_TREND",
"POLICY_COLORS",
# Utility functions
"get_default_layout",
"get_colorbar_defaults",
]

View File

@@ -0,0 +1,162 @@
"""Centralized design tokens for consistent styling across the application.
This module provides a single source of truth for colors, ensuring:
- Consistent styling across all Plotly figures and components
- Accessibility compliance (WCAG color contrast)
- Easy theme updates without hunting through multiple files
Usage:
from portfolio_app.design import TEXT_PRIMARY, CHART_PALETTE
fig.update_layout(font_color=TEXT_PRIMARY)
"""
from typing import Any
# =============================================================================
# TEXT COLORS (Dark Theme)
# =============================================================================
TEXT_PRIMARY = "#c9c9c9"
"""Primary text color for labels, titles, and body text."""
TEXT_SECONDARY = "#888888"
"""Secondary text color for subtitles, captions, and muted text."""
TEXT_MUTED = "#666666"
"""Muted text color for disabled states and placeholders."""
# =============================================================================
# CHART BACKGROUND & GRID
# =============================================================================
GRID_COLOR = "rgba(128, 128, 128, 0.2)"
"""Standard grid line color with transparency."""
GRID_COLOR_DARK = "rgba(128, 128, 128, 0.3)"
"""Darker grid for radar charts and polar plots."""
PAPER_BG = "rgba(0, 0, 0, 0)"
"""Transparent paper background for charts."""
PLOT_BG = "rgba(0, 0, 0, 0)"
"""Transparent plot background for charts."""
# =============================================================================
# SEMANTIC COLORS
# =============================================================================
COLOR_POSITIVE = "#40c057"
"""Positive/success indicator (Mantine green-6)."""
COLOR_NEGATIVE = "#fa5252"
"""Negative/error indicator (Mantine red-6)."""
COLOR_WARNING = "#fab005"
"""Warning indicator (Mantine yellow-6)."""
COLOR_ACCENT = "#228be6"
"""Primary accent color (Mantine blue-6)."""
# =============================================================================
# ACCESSIBLE CHART PALETTE
# =============================================================================
# Okabe-Ito palette - optimized for all color vision deficiencies
# Reference: https://jfly.uni-koeln.de/color/
CHART_PALETTE = [
"#0072B2", # Blue (primary data series)
"#E69F00", # Orange
"#56B4E9", # Sky blue
"#009E73", # Teal/green
"#F0E442", # Yellow
"#D55E00", # Vermillion
"#CC79A7", # Pink
"#000000", # Black (use sparingly)
]
"""
Accessible categorical palette (Okabe-Ito).
Distinguishable for deuteranopia, protanopia, and tritanopia.
Use indices 0-6 for most charts; index 7 (black) for emphasis only.
"""
# Semantic subsets for specific use cases
PALETTE_COMPARISON = [CHART_PALETTE[0], CHART_PALETTE[1]]
"""Two-color palette for A/B comparisons."""
PALETTE_GENDER = {
"male": "#56B4E9", # Sky blue
"female": "#CC79A7", # Pink
}
"""Gender-specific colors (accessible contrast)."""
PALETTE_TREND = {
"positive": COLOR_POSITIVE,
"negative": COLOR_NEGATIVE,
"neutral": TEXT_SECONDARY,
}
"""Trend indicator colors for sparklines and deltas."""
# =============================================================================
# POLICY/EVENT MARKERS (Time Series)
# =============================================================================
POLICY_COLORS = {
"policy_change": "#E69F00", # Orange - policy changes
"major_event": "#D55E00", # Vermillion - major events
"data_note": "#56B4E9", # Sky blue - data annotations
"forecast": "#009E73", # Teal - forecast periods
"highlight": "#F0E442", # Yellow - highlighted regions
}
"""Colors for policy markers and event annotations on time series."""
# =============================================================================
# CHART LAYOUT DEFAULTS
# =============================================================================
def get_default_layout() -> dict[str, Any]:
"""Return default Plotly layout settings with design tokens.
Returns:
dict: Layout configuration for fig.update_layout()
Example:
fig.update_layout(**get_default_layout())
"""
return {
"paper_bgcolor": PAPER_BG,
"plot_bgcolor": PLOT_BG,
"font": {"color": TEXT_PRIMARY},
"title": {"font": {"color": TEXT_PRIMARY}},
"legend": {"font": {"color": TEXT_PRIMARY}},
"xaxis": {
"gridcolor": GRID_COLOR,
"linecolor": GRID_COLOR,
"tickfont": {"color": TEXT_PRIMARY},
"title": {"font": {"color": TEXT_PRIMARY}},
},
"yaxis": {
"gridcolor": GRID_COLOR,
"linecolor": GRID_COLOR,
"tickfont": {"color": TEXT_PRIMARY},
"title": {"font": {"color": TEXT_PRIMARY}},
},
}
def get_colorbar_defaults() -> dict[str, Any]:
"""Return default colorbar settings with design tokens.
Returns:
dict: Colorbar configuration for choropleth/heatmap traces
"""
return {
"tickfont": {"color": TEXT_PRIMARY},
"title": {"font": {"color": TEXT_PRIMARY}},
}

View File

@@ -1,61 +1,15 @@
"""Plotly figure factories for data visualization."""
"""Plotly figure factories for data visualization.
from .bar_charts import (
create_horizontal_bar,
create_ranking_bar,
create_stacked_bar,
)
from .choropleth import (
create_choropleth_figure,
create_zone_map,
)
from .demographics import (
create_age_pyramid,
create_donut_chart,
create_income_distribution,
)
from .radar import (
create_comparison_radar,
create_radar_figure,
)
from .scatter import (
create_bubble_chart,
create_scatter_figure,
)
from .summary_cards import create_metric_card_figure, create_summary_metrics
from .time_series import (
add_policy_markers,
create_market_comparison_chart,
create_price_time_series,
create_time_series_with_events,
create_volume_time_series,
)
Figure factories are organized by dashboard domain:
- toronto/ : Toronto Neighbourhood Dashboard figures
Usage:
from portfolio_app.figures.toronto import create_choropleth_figure
from portfolio_app.figures.toronto import create_ranking_bar
"""
from . import toronto
__all__ = [
# Choropleth
"create_choropleth_figure",
"create_zone_map",
# Time series
"create_price_time_series",
"create_volume_time_series",
"create_market_comparison_chart",
"create_time_series_with_events",
"add_policy_markers",
# Summary
"create_metric_card_figure",
"create_summary_metrics",
# Bar charts
"create_ranking_bar",
"create_stacked_bar",
"create_horizontal_bar",
# Scatter plots
"create_scatter_figure",
"create_bubble_chart",
# Radar charts
"create_radar_figure",
"create_comparison_radar",
# Demographics
"create_age_pyramid",
"create_donut_chart",
"create_income_distribution",
"toronto",
]

View File

@@ -0,0 +1,61 @@
"""Plotly figure factories for Toronto dashboard visualizations."""
from .bar_charts import (
create_horizontal_bar,
create_ranking_bar,
create_stacked_bar,
)
from .choropleth import (
create_choropleth_figure,
create_zone_map,
)
from .demographics import (
create_age_pyramid,
create_donut_chart,
create_income_distribution,
)
from .radar import (
create_comparison_radar,
create_radar_figure,
)
from .scatter import (
create_bubble_chart,
create_scatter_figure,
)
from .summary_cards import create_metric_card_figure, create_summary_metrics
from .time_series import (
add_policy_markers,
create_market_comparison_chart,
create_price_time_series,
create_time_series_with_events,
create_volume_time_series,
)
__all__ = [
# Choropleth
"create_choropleth_figure",
"create_zone_map",
# Time series
"create_price_time_series",
"create_volume_time_series",
"create_market_comparison_chart",
"create_time_series_with_events",
"add_policy_markers",
# Summary
"create_metric_card_figure",
"create_summary_metrics",
# Bar charts
"create_ranking_bar",
"create_stacked_bar",
"create_horizontal_bar",
# Scatter plots
"create_scatter_figure",
"create_bubble_chart",
# Radar charts
"create_radar_figure",
"create_comparison_radar",
# Demographics
"create_age_pyramid",
"create_donut_chart",
"create_income_distribution",
]

View File

@@ -6,6 +6,17 @@ import pandas as pd
import plotly.express as px
import plotly.graph_objects as go
from portfolio_app.design import (
CHART_PALETTE,
COLOR_NEGATIVE,
COLOR_POSITIVE,
GRID_COLOR,
PAPER_BG,
PLOT_BG,
TEXT_PRIMARY,
TEXT_SECONDARY,
)
def create_ranking_bar(
data: list[dict[str, Any]],
@@ -14,8 +25,8 @@ def create_ranking_bar(
title: str | None = None,
top_n: int = 10,
bottom_n: int = 10,
color_top: str = "#4CAF50",
color_bottom: str = "#F44336",
color_top: str = COLOR_POSITIVE,
color_bottom: str = COLOR_NEGATIVE,
value_format: str = ",.0f",
) -> go.Figure:
"""Create horizontal bar chart showing top and bottom rankings.
@@ -87,10 +98,10 @@ def create_ranking_bar(
barmode="group",
showlegend=True,
legend={"orientation": "h", "yanchor": "bottom", "y": 1.02},
paper_bgcolor="rgba(0,0,0,0)",
plot_bgcolor="rgba(0,0,0,0)",
font_color="#c9c9c9",
xaxis={"gridcolor": "rgba(128,128,128,0.2)", "title": None},
paper_bgcolor=PAPER_BG,
plot_bgcolor=PLOT_BG,
font_color=TEXT_PRIMARY,
xaxis={"gridcolor": GRID_COLOR, "title": None},
yaxis={"autorange": "reversed", "title": None},
margin={"l": 10, "r": 10, "t": 40, "b": 10},
)
@@ -126,10 +137,10 @@ def create_stacked_bar(
df = pd.DataFrame(data)
# Default color scheme
# Default color scheme using accessible palette
if color_map is None:
categories = df[category_column].unique()
colors = px.colors.qualitative.Set2[: len(categories)]
colors = CHART_PALETTE[: len(categories)]
color_map = dict(zip(categories, colors, strict=False))
fig = px.bar(
@@ -147,11 +158,11 @@ def create_stacked_bar(
fig.update_layout(
title=title,
paper_bgcolor="rgba(0,0,0,0)",
plot_bgcolor="rgba(0,0,0,0)",
font_color="#c9c9c9",
xaxis={"gridcolor": "rgba(128,128,128,0.2)", "title": None},
yaxis={"gridcolor": "rgba(128,128,128,0.2)", "title": None},
paper_bgcolor=PAPER_BG,
plot_bgcolor=PLOT_BG,
font_color=TEXT_PRIMARY,
xaxis={"gridcolor": GRID_COLOR, "title": None},
yaxis={"gridcolor": GRID_COLOR, "title": None},
legend={"orientation": "h", "yanchor": "bottom", "y": 1.02},
margin={"l": 10, "r": 10, "t": 60, "b": 10},
)
@@ -164,7 +175,7 @@ def create_horizontal_bar(
name_column: str,
value_column: str,
title: str | None = None,
color: str = "#2196F3",
color: str = CHART_PALETTE[0],
value_format: str = ",.0f",
sort: bool = True,
) -> go.Figure:
@@ -204,10 +215,10 @@ def create_horizontal_bar(
fig.update_layout(
title=title,
paper_bgcolor="rgba(0,0,0,0)",
plot_bgcolor="rgba(0,0,0,0)",
font_color="#c9c9c9",
xaxis={"gridcolor": "rgba(128,128,128,0.2)", "title": None},
paper_bgcolor=PAPER_BG,
plot_bgcolor=PLOT_BG,
font_color=TEXT_PRIMARY,
xaxis={"gridcolor": GRID_COLOR, "title": None},
yaxis={"title": None},
margin={"l": 10, "r": 10, "t": 40, "b": 10},
)
@@ -225,13 +236,13 @@ def _create_empty_figure(title: str) -> go.Figure:
x=0.5,
y=0.5,
showarrow=False,
font={"size": 14, "color": "#888888"},
font={"size": 14, "color": TEXT_SECONDARY},
)
fig.update_layout(
title=title,
paper_bgcolor="rgba(0,0,0,0)",
plot_bgcolor="rgba(0,0,0,0)",
font_color="#c9c9c9",
paper_bgcolor=PAPER_BG,
plot_bgcolor=PLOT_BG,
font_color=TEXT_PRIMARY,
xaxis={"visible": False},
yaxis={"visible": False},
)

View File

@@ -5,6 +5,13 @@ from typing import Any
import plotly.express as px
import plotly.graph_objects as go
from portfolio_app.design import (
PAPER_BG,
PLOT_BG,
TEXT_PRIMARY,
TEXT_SECONDARY,
)
def create_choropleth_figure(
geojson: dict[str, Any] | None,
@@ -55,9 +62,9 @@ def create_choropleth_figure(
margin={"l": 0, "r": 0, "t": 40, "b": 0},
title=title or "Toronto Housing Map",
height=500,
paper_bgcolor="rgba(0,0,0,0)",
plot_bgcolor="rgba(0,0,0,0)",
font_color="#c9c9c9",
paper_bgcolor=PAPER_BG,
plot_bgcolor=PLOT_BG,
font_color=TEXT_PRIMARY,
)
fig.add_annotation(
text="No geometry data available. Complete QGIS digitization to enable map.",
@@ -66,7 +73,7 @@ def create_choropleth_figure(
x=0.5,
y=0.5,
showarrow=False,
font={"size": 14, "color": "#888888"},
font={"size": 14, "color": TEXT_SECONDARY},
)
return fig
@@ -98,17 +105,17 @@ def create_choropleth_figure(
margin={"l": 0, "r": 0, "t": 40, "b": 0},
title=title,
height=500,
paper_bgcolor="rgba(0,0,0,0)",
plot_bgcolor="rgba(0,0,0,0)",
font_color="#c9c9c9",
paper_bgcolor=PAPER_BG,
plot_bgcolor=PLOT_BG,
font_color=TEXT_PRIMARY,
coloraxis_colorbar={
"title": {
"text": color_column.replace("_", " ").title(),
"font": {"color": "#c9c9c9"},
"font": {"color": TEXT_PRIMARY},
},
"thickness": 15,
"len": 0.7,
"tickfont": {"color": "#c9c9c9"},
"tickfont": {"color": TEXT_PRIMARY},
},
)

View File

@@ -5,6 +5,16 @@ from typing import Any
import pandas as pd
import plotly.graph_objects as go
from portfolio_app.design import (
CHART_PALETTE,
GRID_COLOR,
PALETTE_GENDER,
PAPER_BG,
PLOT_BG,
TEXT_PRIMARY,
TEXT_SECONDARY,
)
def create_age_pyramid(
data: list[dict[str, Any]],
@@ -52,7 +62,7 @@ def create_age_pyramid(
x=male_values_neg,
orientation="h",
name="Male",
marker_color="#2196F3",
marker_color=PALETTE_GENDER["male"],
hovertemplate="%{y}<br>Male: %{customdata:,}<extra></extra>",
customdata=male_values,
)
@@ -65,7 +75,7 @@ def create_age_pyramid(
x=female_values,
orientation="h",
name="Female",
marker_color="#E91E63",
marker_color=PALETTE_GENDER["female"],
hovertemplate="%{y}<br>Female: %{x:,}<extra></extra>",
)
)
@@ -77,12 +87,12 @@ def create_age_pyramid(
title=title,
barmode="overlay",
bargap=0.1,
paper_bgcolor="rgba(0,0,0,0)",
plot_bgcolor="rgba(0,0,0,0)",
font_color="#c9c9c9",
paper_bgcolor=PAPER_BG,
plot_bgcolor=PLOT_BG,
font_color=TEXT_PRIMARY,
xaxis={
"title": "Population",
"gridcolor": "rgba(128,128,128,0.2)",
"gridcolor": GRID_COLOR,
"range": [-max_val * 1.1, max_val * 1.1],
"tickvals": [-max_val, -max_val / 2, 0, max_val / 2, max_val],
"ticktext": [
@@ -93,7 +103,7 @@ def create_age_pyramid(
f"{max_val:,.0f}",
],
},
yaxis={"title": None, "gridcolor": "rgba(128,128,128,0.2)"},
yaxis={"title": None, "gridcolor": GRID_COLOR},
legend={"orientation": "h", "yanchor": "bottom", "y": 1.02},
margin={"l": 10, "r": 10, "t": 60, "b": 10},
)
@@ -127,17 +137,9 @@ def create_donut_chart(
df = pd.DataFrame(data)
# Use accessible palette by default
if colors is None:
colors = [
"#2196F3",
"#4CAF50",
"#FF9800",
"#E91E63",
"#9C27B0",
"#00BCD4",
"#FFC107",
"#795548",
]
colors = CHART_PALETTE
fig = go.Figure(
go.Pie(
@@ -153,8 +155,8 @@ def create_donut_chart(
fig.update_layout(
title=title,
paper_bgcolor="rgba(0,0,0,0)",
font_color="#c9c9c9",
paper_bgcolor=PAPER_BG,
font_color=TEXT_PRIMARY,
showlegend=False,
margin={"l": 10, "r": 10, "t": 60, "b": 10},
)
@@ -167,7 +169,7 @@ def create_income_distribution(
bracket_column: str,
count_column: str,
title: str | None = None,
color: str = "#4CAF50",
color: str = CHART_PALETTE[3], # Teal
) -> go.Figure:
"""Create histogram-style bar chart for income distribution.
@@ -199,17 +201,17 @@ def create_income_distribution(
fig.update_layout(
title=title,
paper_bgcolor="rgba(0,0,0,0)",
plot_bgcolor="rgba(0,0,0,0)",
font_color="#c9c9c9",
paper_bgcolor=PAPER_BG,
plot_bgcolor=PLOT_BG,
font_color=TEXT_PRIMARY,
xaxis={
"title": "Income Bracket",
"gridcolor": "rgba(128,128,128,0.2)",
"gridcolor": GRID_COLOR,
"tickangle": -45,
},
yaxis={
"title": "Households",
"gridcolor": "rgba(128,128,128,0.2)",
"gridcolor": GRID_COLOR,
},
margin={"l": 10, "r": 10, "t": 60, "b": 80},
)
@@ -227,13 +229,13 @@ def _create_empty_figure(title: str) -> go.Figure:
x=0.5,
y=0.5,
showarrow=False,
font={"size": 14, "color": "#888888"},
font={"size": 14, "color": TEXT_SECONDARY},
)
fig.update_layout(
title=title,
paper_bgcolor="rgba(0,0,0,0)",
plot_bgcolor="rgba(0,0,0,0)",
font_color="#c9c9c9",
paper_bgcolor=PAPER_BG,
plot_bgcolor=PLOT_BG,
font_color=TEXT_PRIMARY,
xaxis={"visible": False},
yaxis={"visible": False},
)

View File

@@ -4,6 +4,14 @@ from typing import Any
import plotly.graph_objects as go
from portfolio_app.design import (
CHART_PALETTE,
GRID_COLOR_DARK,
PAPER_BG,
TEXT_PRIMARY,
TEXT_SECONDARY,
)
def create_radar_figure(
data: list[dict[str, Any]],
@@ -32,16 +40,9 @@ def create_radar_figure(
if not data or not metrics:
return _create_empty_figure(title or "Radar Chart")
# Default colors
# Use accessible palette by default
if colors is None:
colors = [
"#2196F3",
"#4CAF50",
"#FF9800",
"#E91E63",
"#9C27B0",
"#00BCD4",
]
colors = CHART_PALETTE
fig = go.Figure()
@@ -78,19 +79,19 @@ def create_radar_figure(
polar={
"radialaxis": {
"visible": True,
"gridcolor": "rgba(128,128,128,0.3)",
"linecolor": "rgba(128,128,128,0.3)",
"tickfont": {"color": "#c9c9c9"},
"gridcolor": GRID_COLOR_DARK,
"linecolor": GRID_COLOR_DARK,
"tickfont": {"color": TEXT_PRIMARY},
},
"angularaxis": {
"gridcolor": "rgba(128,128,128,0.3)",
"linecolor": "rgba(128,128,128,0.3)",
"tickfont": {"color": "#c9c9c9"},
"gridcolor": GRID_COLOR_DARK,
"linecolor": GRID_COLOR_DARK,
"tickfont": {"color": TEXT_PRIMARY},
},
"bgcolor": "rgba(0,0,0,0)",
"bgcolor": PAPER_BG,
},
paper_bgcolor="rgba(0,0,0,0)",
font_color="#c9c9c9",
paper_bgcolor=PAPER_BG,
font_color=TEXT_PRIMARY,
showlegend=len(data) > 1,
legend={"orientation": "h", "yanchor": "bottom", "y": -0.2},
margin={"l": 40, "r": 40, "t": 60, "b": 40},
@@ -133,7 +134,7 @@ def create_comparison_radar(
metrics=metrics,
name_column="__name__",
title=title,
colors=["#4CAF50", "#9E9E9E"],
colors=[CHART_PALETTE[3], TEXT_SECONDARY], # Teal for selected, gray for avg
)
@@ -156,11 +157,11 @@ def _create_empty_figure(title: str) -> go.Figure:
x=0.5,
y=0.5,
showarrow=False,
font={"size": 14, "color": "#888888"},
font={"size": 14, "color": TEXT_SECONDARY},
)
fig.update_layout(
title=title,
paper_bgcolor="rgba(0,0,0,0)",
font_color="#c9c9c9",
paper_bgcolor=PAPER_BG,
font_color=TEXT_PRIMARY,
)
return fig

View File

@@ -6,6 +6,15 @@ import pandas as pd
import plotly.express as px
import plotly.graph_objects as go
from portfolio_app.design import (
CHART_PALETTE,
GRID_COLOR,
PAPER_BG,
PLOT_BG,
TEXT_PRIMARY,
TEXT_SECONDARY,
)
def create_scatter_figure(
data: list[dict[str, Any]],
@@ -72,21 +81,21 @@ def create_scatter_figure(
if trendline:
fig.update_traces(
selector={"mode": "lines"},
line={"color": "#FF9800", "dash": "dash", "width": 2},
line={"color": CHART_PALETTE[1], "dash": "dash", "width": 2},
)
fig.update_layout(
title=title,
paper_bgcolor="rgba(0,0,0,0)",
plot_bgcolor="rgba(0,0,0,0)",
font_color="#c9c9c9",
paper_bgcolor=PAPER_BG,
plot_bgcolor=PLOT_BG,
font_color=TEXT_PRIMARY,
xaxis={
"gridcolor": "rgba(128,128,128,0.2)",
"gridcolor": GRID_COLOR,
"title": x_title or x_column.replace("_", " ").title(),
"zeroline": False,
},
yaxis={
"gridcolor": "rgba(128,128,128,0.2)",
"gridcolor": GRID_COLOR,
"title": y_title or y_column.replace("_", " ").title(),
"zeroline": False,
},
@@ -140,19 +149,20 @@ def create_bubble_chart(
hover_name=name_column,
size_max=size_max,
opacity=0.7,
color_discrete_sequence=CHART_PALETTE,
)
fig.update_layout(
title=title,
paper_bgcolor="rgba(0,0,0,0)",
plot_bgcolor="rgba(0,0,0,0)",
font_color="#c9c9c9",
paper_bgcolor=PAPER_BG,
plot_bgcolor=PLOT_BG,
font_color=TEXT_PRIMARY,
xaxis={
"gridcolor": "rgba(128,128,128,0.2)",
"gridcolor": GRID_COLOR,
"title": x_title or x_column.replace("_", " ").title(),
},
yaxis={
"gridcolor": "rgba(128,128,128,0.2)",
"gridcolor": GRID_COLOR,
"title": y_title or y_column.replace("_", " ").title(),
},
margin={"l": 10, "r": 10, "t": 40, "b": 10},
@@ -171,13 +181,13 @@ def _create_empty_figure(title: str) -> go.Figure:
x=0.5,
y=0.5,
showarrow=False,
font={"size": 14, "color": "#888888"},
font={"size": 14, "color": TEXT_SECONDARY},
)
fig.update_layout(
title=title,
paper_bgcolor="rgba(0,0,0,0)",
plot_bgcolor="rgba(0,0,0,0)",
font_color="#c9c9c9",
paper_bgcolor=PAPER_BG,
plot_bgcolor=PLOT_BG,
font_color=TEXT_PRIMARY,
xaxis={"visible": False},
yaxis={"visible": False},
)

View File

@@ -4,6 +4,14 @@ from typing import Any
import plotly.graph_objects as go
from portfolio_app.design import (
COLOR_NEGATIVE,
COLOR_POSITIVE,
PAPER_BG,
PLOT_BG,
TEXT_PRIMARY,
)
def create_metric_card_figure(
value: float | int | str,
@@ -59,8 +67,12 @@ def create_metric_card_figure(
"relative": False,
"valueformat": ".1f",
"suffix": delta_suffix,
"increasing": {"color": "green" if positive_is_good else "red"},
"decreasing": {"color": "red" if positive_is_good else "green"},
"increasing": {
"color": COLOR_POSITIVE if positive_is_good else COLOR_NEGATIVE
},
"decreasing": {
"color": COLOR_NEGATIVE if positive_is_good else COLOR_POSITIVE
},
}
fig.add_trace(go.Indicator(**indicator_config))
@@ -68,9 +80,9 @@ def create_metric_card_figure(
fig.update_layout(
height=120,
margin={"l": 20, "r": 20, "t": 40, "b": 20},
paper_bgcolor="rgba(0,0,0,0)",
plot_bgcolor="rgba(0,0,0,0)",
font={"family": "Inter, sans-serif", "color": "#c9c9c9"},
paper_bgcolor=PAPER_BG,
plot_bgcolor=PLOT_BG,
font={"family": "Inter, sans-serif", "color": TEXT_PRIMARY},
)
return fig

View File

@@ -5,6 +5,15 @@ from typing import Any
import plotly.express as px
import plotly.graph_objects as go
from portfolio_app.design import (
CHART_PALETTE,
GRID_COLOR,
PAPER_BG,
PLOT_BG,
TEXT_PRIMARY,
TEXT_SECONDARY,
)
def create_price_time_series(
data: list[dict[str, Any]],
@@ -38,14 +47,14 @@ def create_price_time_series(
x=0.5,
y=0.5,
showarrow=False,
font={"color": "#888888"},
font={"color": TEXT_SECONDARY},
)
fig.update_layout(
title=title,
height=350,
paper_bgcolor="rgba(0,0,0,0)",
plot_bgcolor="rgba(0,0,0,0)",
font_color="#c9c9c9",
paper_bgcolor=PAPER_BG,
plot_bgcolor=PLOT_BG,
font_color=TEXT_PRIMARY,
)
return fig
@@ -59,6 +68,7 @@ def create_price_time_series(
y=price_column,
color=group_column,
title=title,
color_discrete_sequence=CHART_PALETTE,
)
else:
fig = px.line(
@@ -67,6 +77,7 @@ def create_price_time_series(
y=price_column,
title=title,
)
fig.update_traces(line_color=CHART_PALETTE[0])
fig.update_layout(
height=350,
@@ -76,11 +87,11 @@ def create_price_time_series(
yaxis_tickprefix="$",
yaxis_tickformat=",",
hovermode="x unified",
paper_bgcolor="rgba(0,0,0,0)",
plot_bgcolor="rgba(0,0,0,0)",
font_color="#c9c9c9",
xaxis={"gridcolor": "#333333", "linecolor": "#444444"},
yaxis={"gridcolor": "#333333", "linecolor": "#444444"},
paper_bgcolor=PAPER_BG,
plot_bgcolor=PLOT_BG,
font_color=TEXT_PRIMARY,
xaxis={"gridcolor": GRID_COLOR, "linecolor": GRID_COLOR},
yaxis={"gridcolor": GRID_COLOR, "linecolor": GRID_COLOR},
)
return fig
@@ -118,14 +129,14 @@ def create_volume_time_series(
x=0.5,
y=0.5,
showarrow=False,
font={"color": "#888888"},
font={"color": TEXT_SECONDARY},
)
fig.update_layout(
title=title,
height=350,
paper_bgcolor="rgba(0,0,0,0)",
plot_bgcolor="rgba(0,0,0,0)",
font_color="#c9c9c9",
paper_bgcolor=PAPER_BG,
plot_bgcolor=PLOT_BG,
font_color=TEXT_PRIMARY,
)
return fig
@@ -140,6 +151,7 @@ def create_volume_time_series(
y=volume_column,
color=group_column,
title=title,
color_discrete_sequence=CHART_PALETTE,
)
else:
fig = px.bar(
@@ -148,6 +160,7 @@ def create_volume_time_series(
y=volume_column,
title=title,
)
fig.update_traces(marker_color=CHART_PALETTE[0])
else:
if group_column and group_column in df.columns:
fig = px.line(
@@ -156,6 +169,7 @@ def create_volume_time_series(
y=volume_column,
color=group_column,
title=title,
color_discrete_sequence=CHART_PALETTE,
)
else:
fig = px.line(
@@ -164,6 +178,7 @@ def create_volume_time_series(
y=volume_column,
title=title,
)
fig.update_traces(line_color=CHART_PALETTE[0])
fig.update_layout(
height=350,
@@ -172,11 +187,11 @@ def create_volume_time_series(
yaxis_title=volume_column.replace("_", " ").title(),
yaxis_tickformat=",",
hovermode="x unified",
paper_bgcolor="rgba(0,0,0,0)",
plot_bgcolor="rgba(0,0,0,0)",
font_color="#c9c9c9",
xaxis={"gridcolor": "#333333", "linecolor": "#444444"},
yaxis={"gridcolor": "#333333", "linecolor": "#444444"},
paper_bgcolor=PAPER_BG,
plot_bgcolor=PLOT_BG,
font_color=TEXT_PRIMARY,
xaxis={"gridcolor": GRID_COLOR, "linecolor": GRID_COLOR},
yaxis={"gridcolor": GRID_COLOR, "linecolor": GRID_COLOR},
)
return fig
@@ -211,14 +226,14 @@ def create_market_comparison_chart(
x=0.5,
y=0.5,
showarrow=False,
font={"color": "#888888"},
font={"color": TEXT_SECONDARY},
)
fig.update_layout(
title=title,
height=400,
paper_bgcolor="rgba(0,0,0,0)",
plot_bgcolor="rgba(0,0,0,0)",
font_color="#c9c9c9",
paper_bgcolor=PAPER_BG,
plot_bgcolor=PLOT_BG,
font_color=TEXT_PRIMARY,
)
return fig
@@ -230,8 +245,6 @@ def create_market_comparison_chart(
fig = make_subplots(specs=[[{"secondary_y": True}]])
colors = ["#1f77b4", "#ff7f0e", "#2ca02c", "#d62728"]
for i, metric in enumerate(metrics[:4]):
if metric not in df.columns:
continue
@@ -242,7 +255,7 @@ def create_market_comparison_chart(
x=df[date_column],
y=df[metric],
name=metric.replace("_", " ").title(),
line={"color": colors[i % len(colors)]},
line={"color": CHART_PALETTE[i % len(CHART_PALETTE)]},
),
secondary_y=secondary,
)
@@ -252,18 +265,18 @@ def create_market_comparison_chart(
height=400,
margin={"l": 40, "r": 40, "t": 50, "b": 40},
hovermode="x unified",
paper_bgcolor="rgba(0,0,0,0)",
plot_bgcolor="rgba(0,0,0,0)",
font_color="#c9c9c9",
xaxis={"gridcolor": "#333333", "linecolor": "#444444"},
yaxis={"gridcolor": "#333333", "linecolor": "#444444"},
paper_bgcolor=PAPER_BG,
plot_bgcolor=PLOT_BG,
font_color=TEXT_PRIMARY,
xaxis={"gridcolor": GRID_COLOR, "linecolor": GRID_COLOR},
yaxis={"gridcolor": GRID_COLOR, "linecolor": GRID_COLOR},
legend={
"orientation": "h",
"yanchor": "bottom",
"y": 1.02,
"xanchor": "right",
"x": 1,
"font": {"color": "#c9c9c9"},
"font": {"color": TEXT_PRIMARY},
},
)
@@ -290,13 +303,13 @@ def add_policy_markers(
if not policy_events:
return fig
# Color mapping for policy categories
# Color mapping for policy categories using design tokens
category_colors = {
"monetary": "#1f77b4", # Blue
"tax": "#2ca02c", # Green
"regulatory": "#ff7f0e", # Orange
"supply": "#9467bd", # Purple
"economic": "#d62728", # Red
"monetary": CHART_PALETTE[0], # Blue
"tax": CHART_PALETTE[3], # Teal/green
"regulatory": CHART_PALETTE[1], # Orange
"supply": CHART_PALETTE[6], # Pink
"economic": CHART_PALETTE[5], # Vermillion
}
# Symbol mapping for expected direction
@@ -313,7 +326,7 @@ def add_policy_markers(
title = event.get("title", "Policy Event")
level = event.get("level", "federal")
color = category_colors.get(category, "#666666")
color = category_colors.get(category, TEXT_SECONDARY)
symbol = direction_symbols.get(direction, "circle")
# Add vertical line for the event
@@ -335,7 +348,7 @@ def add_policy_markers(
"symbol": symbol,
"size": 12,
"color": color,
"line": {"width": 1, "color": "white"},
"line": {"width": 1, "color": TEXT_PRIMARY},
},
name=title,
hovertemplate=(

View File

@@ -2,6 +2,7 @@
import dash
import dash_mantine_components as dmc
from dash import html
from dash_iconify import DashIconify
dash.register_page(__name__, path="/contact", name="Contact")
@@ -51,51 +52,57 @@ def create_intro_section() -> dmc.Stack:
def create_contact_form() -> dmc.Paper:
"""Create the contact form (disabled in Phase 1)."""
"""Create the contact form with Formspree integration."""
return dmc.Paper(
dmc.Stack(
[
dmc.Title("Send a Message", order=2, size="h4"),
dmc.Alert(
"Contact form submission is coming soon. Please use the direct contact "
"methods below for now.",
title="Form Coming Soon",
color="blue",
variant="light",
),
# Feedback container for success/error messages
html.Div(id="contact-feedback"),
dmc.TextInput(
id="contact-name",
label="Name",
placeholder="Your name",
leftSection=DashIconify(icon="tabler:user", width=18),
disabled=True,
required=True,
),
dmc.TextInput(
id="contact-email",
label="Email",
placeholder="your.email@example.com",
leftSection=DashIconify(icon="tabler:mail", width=18),
disabled=True,
required=True,
),
dmc.Select(
id="contact-subject",
label="Subject",
placeholder="Select a subject",
data=SUBJECT_OPTIONS,
leftSection=DashIconify(icon="tabler:tag", width=18),
disabled=True,
),
dmc.Textarea(
id="contact-message",
label="Message",
placeholder="Your message...",
minRows=4,
disabled=True,
required=True,
),
# Honeypot field for spam protection (hidden from users)
dmc.TextInput(
id="contact-gotcha",
style={"position": "absolute", "left": "-9999px"},
tabIndex=-1,
autoComplete="off",
),
dmc.Button(
"Send Message",
id="contact-submit",
fullWidth=True,
leftSection=DashIconify(icon="tabler:send", width=18),
disabled=True,
),
],
gap="md",
style={"position": "relative"},
),
p="xl",
radius="md",

View File

@@ -5,7 +5,15 @@ import pandas as pd
import plotly.graph_objects as go
from dash import Input, Output, callback
from portfolio_app.figures import (
from portfolio_app.design import (
CHART_PALETTE,
GRID_COLOR,
PAPER_BG,
PLOT_BG,
TEXT_PRIMARY,
TEXT_SECONDARY,
)
from portfolio_app.figures.toronto import (
create_donut_chart,
create_horizontal_bar,
create_radar_figure,
@@ -109,18 +117,18 @@ def update_housing_trend(year: str, neighbourhood_id: int | None) -> go.Figure:
x=[d["year"] for d in data],
y=[d["avg_rent"] for d in data],
mode="lines+markers",
line={"color": "#2196F3", "width": 2},
line={"color": CHART_PALETTE[0], "width": 2},
marker={"size": 8},
name="City Average",
)
)
fig.update_layout(
paper_bgcolor="rgba(0,0,0,0)",
plot_bgcolor="rgba(0,0,0,0)",
font_color="#c9c9c9",
xaxis={"gridcolor": "rgba(128,128,128,0.2)"},
yaxis={"gridcolor": "rgba(128,128,128,0.2)", "title": "Avg Rent (2BR)"},
paper_bgcolor=PAPER_BG,
plot_bgcolor=PLOT_BG,
font_color=TEXT_PRIMARY,
xaxis={"gridcolor": GRID_COLOR},
yaxis={"gridcolor": GRID_COLOR, "title": "Avg Rent (2BR)"},
showlegend=False,
margin={"l": 40, "r": 10, "t": 10, "b": 30},
)
@@ -153,7 +161,7 @@ def update_housing_types(year: str) -> go.Figure:
data=data,
name_column="type",
value_column="percentage",
colors=["#4CAF50", "#2196F3"],
colors=[CHART_PALETTE[3], CHART_PALETTE[0]], # Teal for owner, blue for renter
)
@@ -178,19 +186,19 @@ def update_safety_trend(year: str) -> go.Figure:
x=[d["year"] for d in data],
y=[d["crime_rate"] for d in data],
mode="lines+markers",
line={"color": "#FF5722", "width": 2},
line={"color": CHART_PALETTE[5], "width": 2}, # Vermillion
marker={"size": 8},
fill="tozeroy",
fillcolor="rgba(255,87,34,0.1)",
fillcolor="rgba(213, 94, 0, 0.1)", # Vermillion with opacity
)
)
fig.update_layout(
paper_bgcolor="rgba(0,0,0,0)",
plot_bgcolor="rgba(0,0,0,0)",
font_color="#c9c9c9",
xaxis={"gridcolor": "rgba(128,128,128,0.2)"},
yaxis={"gridcolor": "rgba(128,128,128,0.2)", "title": "Crime Rate per 100K"},
paper_bgcolor=PAPER_BG,
plot_bgcolor=PLOT_BG,
font_color=TEXT_PRIMARY,
xaxis={"gridcolor": GRID_COLOR},
yaxis={"gridcolor": GRID_COLOR, "title": "Crime Rate per 100K"},
showlegend=False,
margin={"l": 40, "r": 10, "t": 10, "b": 30},
)
@@ -233,7 +241,7 @@ def update_safety_types(year: str) -> go.Figure:
data=data,
name_column="category",
value_column="count",
color="#FF5722",
color=CHART_PALETTE[5], # Vermillion for crime
)
@@ -264,7 +272,11 @@ def update_demographics_age(year: str) -> go.Figure:
data=data,
name_column="age_group",
value_column="percentage",
colors=["#9C27B0", "#673AB7", "#3F51B5"],
colors=[
CHART_PALETTE[2],
CHART_PALETTE[0],
CHART_PALETTE[4],
], # Sky, Blue, Yellow
)
@@ -301,7 +313,7 @@ def update_demographics_income(year: str) -> go.Figure:
data=data,
name_column="bracket",
value_column="count",
color="#4CAF50",
color=CHART_PALETTE[3], # Teal
sort=False,
)
@@ -333,7 +345,7 @@ def update_amenities_breakdown(year: str) -> go.Figure:
data=data,
name_column="type",
value_column="count",
color="#4CAF50",
color=CHART_PALETTE[3], # Teal
)
@@ -387,9 +399,9 @@ def _empty_chart(message: str) -> go.Figure:
"""Create an empty chart with a message."""
fig = go.Figure()
fig.update_layout(
paper_bgcolor="rgba(0,0,0,0)",
plot_bgcolor="rgba(0,0,0,0)",
font_color="#c9c9c9",
paper_bgcolor=PAPER_BG,
plot_bgcolor=PLOT_BG,
font_color=TEXT_PRIMARY,
xaxis={"visible": False},
yaxis={"visible": False},
)
@@ -400,6 +412,6 @@ def _empty_chart(message: str) -> go.Figure:
x=0.5,
y=0.5,
showarrow=False,
font={"size": 14, "color": "#888888"},
font={"size": 14, "color": TEXT_SECONDARY},
)
return fig

View File

@@ -4,7 +4,13 @@
import plotly.graph_objects as go
from dash import Input, Output, State, callback, no_update
from portfolio_app.figures import create_choropleth_figure, create_ranking_bar
from portfolio_app.design import (
PAPER_BG,
PLOT_BG,
TEXT_PRIMARY,
TEXT_SECONDARY,
)
from portfolio_app.figures.toronto import create_choropleth_figure, create_ranking_bar
from portfolio_app.toronto.services import (
get_amenities_data,
get_demographics_data,
@@ -267,8 +273,8 @@ def _empty_map(message: str) -> go.Figure:
"zoom": 9.5,
},
margin={"l": 0, "r": 0, "t": 0, "b": 0},
paper_bgcolor="rgba(0,0,0,0)",
font_color="#c9c9c9",
paper_bgcolor=PAPER_BG,
font_color=TEXT_PRIMARY,
)
fig.add_annotation(
text=message,
@@ -277,7 +283,7 @@ def _empty_map(message: str) -> go.Figure:
x=0.5,
y=0.5,
showarrow=False,
font={"size": 14, "color": "#888888"},
font={"size": 14, "color": TEXT_SECONDARY},
)
return fig
@@ -286,9 +292,9 @@ def _empty_chart(message: str) -> go.Figure:
"""Create an empty chart with a message."""
fig = go.Figure()
fig.update_layout(
paper_bgcolor="rgba(0,0,0,0)",
plot_bgcolor="rgba(0,0,0,0)",
font_color="#c9c9c9",
paper_bgcolor=PAPER_BG,
plot_bgcolor=PLOT_BG,
font_color=TEXT_PRIMARY,
xaxis={"visible": False},
yaxis={"visible": False},
)
@@ -299,6 +305,6 @@ def _empty_chart(message: str) -> go.Figure:
x=0.5,
y=0.5,
showarrow=False,
font={"size": 14, "color": "#888888"},
font={"size": 14, "color": TEXT_SECONDARY},
)
return fig

View File

@@ -8,11 +8,18 @@ from sqlalchemy.orm import Mapped, mapped_column
from .base import Base
# Schema constants
RAW_TORONTO_SCHEMA = "raw_toronto"
class DimTime(Base):
"""Time dimension table."""
"""Time dimension table (shared across all projects).
Note: Stays in public schema as it's a shared dimension.
"""
__tablename__ = "dim_time"
__table_args__ = {"schema": "public"}
date_key: Mapped[int] = mapped_column(Integer, primary_key=True)
full_date: Mapped[date] = mapped_column(Date, nullable=False, unique=True)
@@ -27,6 +34,7 @@ class DimCMHCZone(Base):
"""CMHC zone dimension table with PostGIS geometry."""
__tablename__ = "dim_cmhc_zone"
__table_args__ = {"schema": RAW_TORONTO_SCHEMA}
zone_key: Mapped[int] = mapped_column(Integer, primary_key=True, autoincrement=True)
zone_code: Mapped[str] = mapped_column(String(10), nullable=False, unique=True)
@@ -41,6 +49,7 @@ class DimNeighbourhood(Base):
"""
__tablename__ = "dim_neighbourhood"
__table_args__ = {"schema": RAW_TORONTO_SCHEMA}
neighbourhood_id: Mapped[int] = mapped_column(Integer, primary_key=True)
name: Mapped[str] = mapped_column(String(100), nullable=False)
@@ -69,6 +78,7 @@ class DimPolicyEvent(Base):
"""Policy event dimension for time-series annotation."""
__tablename__ = "dim_policy_event"
__table_args__ = {"schema": RAW_TORONTO_SCHEMA}
event_id: Mapped[int] = mapped_column(Integer, primary_key=True, autoincrement=True)
event_date: Mapped[date] = mapped_column(Date, nullable=False)

View File

@@ -4,6 +4,7 @@ from sqlalchemy import ForeignKey, Index, Integer, Numeric, String
from sqlalchemy.orm import Mapped, mapped_column, relationship
from .base import Base
from .dimensions import RAW_TORONTO_SCHEMA
class BridgeCMHCNeighbourhood(Base):
@@ -14,6 +15,11 @@ class BridgeCMHCNeighbourhood(Base):
"""
__tablename__ = "bridge_cmhc_neighbourhood"
__table_args__ = (
Index("ix_bridge_cmhc_zone", "cmhc_zone_code"),
Index("ix_bridge_neighbourhood", "neighbourhood_id"),
{"schema": RAW_TORONTO_SCHEMA},
)
id: Mapped[int] = mapped_column(Integer, primary_key=True, autoincrement=True)
cmhc_zone_code: Mapped[str] = mapped_column(String(10), nullable=False)
@@ -22,11 +28,6 @@ class BridgeCMHCNeighbourhood(Base):
Numeric(5, 4), nullable=False
) # 0.0000 to 1.0000
__table_args__ = (
Index("ix_bridge_cmhc_zone", "cmhc_zone_code"),
Index("ix_bridge_neighbourhood", "neighbourhood_id"),
)
class FactCensus(Base):
"""Census statistics by neighbourhood and year.
@@ -35,6 +36,10 @@ class FactCensus(Base):
"""
__tablename__ = "fact_census"
__table_args__ = (
Index("ix_fact_census_neighbourhood_year", "neighbourhood_id", "census_year"),
{"schema": RAW_TORONTO_SCHEMA},
)
id: Mapped[int] = mapped_column(Integer, primary_key=True, autoincrement=True)
neighbourhood_id: Mapped[int] = mapped_column(Integer, nullable=False)
@@ -66,10 +71,6 @@ class FactCensus(Base):
Numeric(12, 2), nullable=True
)
__table_args__ = (
Index("ix_fact_census_neighbourhood_year", "neighbourhood_id", "census_year"),
)
class FactCrime(Base):
"""Crime statistics by neighbourhood and year.
@@ -78,6 +79,11 @@ class FactCrime(Base):
"""
__tablename__ = "fact_crime"
__table_args__ = (
Index("ix_fact_crime_neighbourhood_year", "neighbourhood_id", "year"),
Index("ix_fact_crime_type", "crime_type"),
{"schema": RAW_TORONTO_SCHEMA},
)
id: Mapped[int] = mapped_column(Integer, primary_key=True, autoincrement=True)
neighbourhood_id: Mapped[int] = mapped_column(Integer, nullable=False)
@@ -86,11 +92,6 @@ class FactCrime(Base):
count: Mapped[int] = mapped_column(Integer, nullable=False)
rate_per_100k: Mapped[float | None] = mapped_column(Numeric(10, 2), nullable=True)
__table_args__ = (
Index("ix_fact_crime_neighbourhood_year", "neighbourhood_id", "year"),
Index("ix_fact_crime_type", "crime_type"),
)
class FactAmenities(Base):
"""Amenity counts by neighbourhood.
@@ -99,6 +100,11 @@ class FactAmenities(Base):
"""
__tablename__ = "fact_amenities"
__table_args__ = (
Index("ix_fact_amenities_neighbourhood_year", "neighbourhood_id", "year"),
Index("ix_fact_amenities_type", "amenity_type"),
{"schema": RAW_TORONTO_SCHEMA},
)
id: Mapped[int] = mapped_column(Integer, primary_key=True, autoincrement=True)
neighbourhood_id: Mapped[int] = mapped_column(Integer, nullable=False)
@@ -106,11 +112,6 @@ class FactAmenities(Base):
count: Mapped[int] = mapped_column(Integer, nullable=False)
year: Mapped[int] = mapped_column(Integer, nullable=False)
__table_args__ = (
Index("ix_fact_amenities_neighbourhood_year", "neighbourhood_id", "year"),
Index("ix_fact_amenities_type", "amenity_type"),
)
class FactRentals(Base):
"""Fact table for CMHC rental market data.
@@ -119,13 +120,16 @@ class FactRentals(Base):
"""
__tablename__ = "fact_rentals"
__table_args__ = {"schema": RAW_TORONTO_SCHEMA}
id: Mapped[int] = mapped_column(Integer, primary_key=True, autoincrement=True)
date_key: Mapped[int] = mapped_column(
Integer, ForeignKey("dim_time.date_key"), nullable=False
Integer, ForeignKey("public.dim_time.date_key"), nullable=False
)
zone_key: Mapped[int] = mapped_column(
Integer, ForeignKey("dim_cmhc_zone.zone_key"), nullable=False
Integer,
ForeignKey(f"{RAW_TORONTO_SCHEMA}.dim_cmhc_zone.zone_key"),
nullable=False,
)
bedroom_type: Mapped[str] = mapped_column(String(20), nullable=False)
universe: Mapped[int | None] = mapped_column(Integer, nullable=True)
@@ -139,6 +143,6 @@ class FactRentals(Base):
rent_change_pct: Mapped[float | None] = mapped_column(Numeric(5, 2), nullable=True)
reliability_code: Mapped[str | None] = mapped_column(String(2), nullable=True)
# Relationships
time = relationship("DimTime", backref="rentals")
zone = relationship("DimCMHCZone", backref="rentals")
# Relationships - explicit foreign_keys needed for cross-schema joins
time = relationship("DimTime", foreign_keys=[date_key], backref="rentals")
zone = relationship("DimCMHCZone", foreign_keys=[zone_key], backref="rentals")

View File

@@ -34,6 +34,8 @@ dependencies = [
"pandas>=2.3",
"geopandas>=1.1",
"shapely>=2.0",
"pyproj>=3.6",
"statsmodels>=0.14",
# Visualization
"dash>=3.3",
@@ -69,6 +71,10 @@ dev = [
# Pre-commit
"pre-commit>=4.0",
# Jupyter
"jupyter>=1.0",
"ipykernel>=6.29",
# Type stubs
"pandas-stubs",
"types-requests",

View File

@@ -28,8 +28,13 @@ from datetime import date
from pathlib import Path
from typing import Any
from dotenv import load_dotenv
# Add project root to path
PROJECT_ROOT = Path(__file__).parent.parent.parent
# Load .env file so dbt can access POSTGRES_* environment variables
load_dotenv(PROJECT_ROOT / ".env")
sys.path.insert(0, str(PROJECT_ROOT))
from portfolio_app.toronto.loaders import ( # noqa: E402
@@ -283,30 +288,46 @@ class DataPipeline:
logger.info("Running dbt transformations...")
dbt_project_dir = PROJECT_ROOT / "dbt"
venv_dbt = PROJECT_ROOT / ".venv" / "bin" / "dbt"
# Use venv dbt if available, otherwise fall back to system dbt
dbt_cmd = str(venv_dbt) if venv_dbt.exists() else "dbt"
if not dbt_project_dir.exists():
logger.error(f"dbt project directory not found: {dbt_project_dir}")
return False
if self.dry_run:
logger.info(" [DRY RUN] Would run: dbt deps")
logger.info(" [DRY RUN] Would run: dbt run")
logger.info(" [DRY RUN] Would run: dbt test")
return True
try:
# Run dbt models
logger.info(" Running dbt run...")
# Install dbt packages if needed
logger.info(" Running dbt deps...")
result = subprocess.run(
["dbt", "run"],
[dbt_cmd, "deps", "--profiles-dir", str(dbt_project_dir)],
cwd=dbt_project_dir,
capture_output=True,
text=True,
)
if result.returncode != 0:
logger.error(f"dbt run failed:\n{result.stderr}")
if self.verbose:
logger.debug(f"dbt output:\n{result.stdout}")
logger.error(f"dbt deps failed:\n{result.stdout}\n{result.stderr}")
return False
# Run dbt models
logger.info(" Running dbt run...")
result = subprocess.run(
[dbt_cmd, "run", "--profiles-dir", str(dbt_project_dir)],
cwd=dbt_project_dir,
capture_output=True,
text=True,
)
if result.returncode != 0:
logger.error(f"dbt run failed:\n{result.stdout}\n{result.stderr}")
return False
logger.info(" dbt run completed successfully")
@@ -314,14 +335,16 @@ class DataPipeline:
# Run dbt tests
logger.info(" Running dbt test...")
result = subprocess.run(
["dbt", "test"],
[dbt_cmd, "test", "--profiles-dir", str(dbt_project_dir)],
cwd=dbt_project_dir,
capture_output=True,
text=True,
)
if result.returncode != 0:
logger.warning(f"dbt test had failures:\n{result.stderr}")
logger.warning(
f"dbt test had failures:\n{result.stdout}\n{result.stderr}"
)
# Don't fail on test failures, just warn
else:
logger.info(" dbt test completed successfully")

View File

@@ -0,0 +1,421 @@
#!/usr/bin/env python3
"""Seed sample data for development/testing.
This script:
- Populates fact_amenities with sample data
- Updates dim_neighbourhood with population from fact_census
- Seeds median_age in fact_census where missing
- Seeds census housing columns (tenure, income, dwelling value)
- Seeds housing mart data (rent, affordability)
- Seeds overview mart data (safety_score, population)
- Runs dbt to rebuild the marts
Usage:
python scripts/data/seed_amenity_data.py
Run this after load_toronto_data.py to ensure notebooks have data.
"""
import os
import random
import subprocess
import sys
from pathlib import Path
from dotenv import load_dotenv
from sqlalchemy import create_engine, text
PROJECT_ROOT = Path(__file__).parent.parent.parent
load_dotenv(PROJECT_ROOT / ".env")
DATABASE_URL = os.environ.get("DATABASE_URL")
if not DATABASE_URL:
print("ERROR: DATABASE_URL not set in .env")
sys.exit(1)
def seed_amenities() -> int:
"""Insert sample amenity data for all neighbourhoods."""
engine = create_engine(DATABASE_URL)
with engine.connect() as conn:
result = conn.execute(
text("SELECT neighbourhood_id FROM public.dim_neighbourhood")
)
neighbourhood_ids = [row[0] for row in result]
print(f"Found {len(neighbourhood_ids)} neighbourhoods")
amenity_types = [
"Parks",
"Schools",
"Transit Stops",
"Libraries",
"Community Centres",
"Recreation",
]
year = 2024
with engine.begin() as conn:
conn.execute(text("DELETE FROM public.fact_amenities"))
total = 0
for n_id in neighbourhood_ids:
for amenity_type in amenity_types:
count = random.randint(1, 50)
conn.execute(
text(
"""
INSERT INTO public.fact_amenities
(neighbourhood_id, amenity_type, count, year)
VALUES (:neighbourhood_id, :amenity_type, :count, :year)
"""
),
{
"neighbourhood_id": n_id,
"amenity_type": amenity_type,
"count": count,
"year": year,
},
)
total += 1
print(f"Inserted {total} amenity records")
return total
def update_population() -> int:
"""Update dim_neighbourhood with population from fact_census."""
engine = create_engine(DATABASE_URL)
with engine.begin() as conn:
result = conn.execute(
text(
"""
UPDATE public.dim_neighbourhood dn
SET population = fc.population
FROM public.fact_census fc
WHERE dn.neighbourhood_id = fc.neighbourhood_id
AND fc.census_year = 2021
"""
)
)
count = int(result.rowcount)
print(f"Updated {count} neighbourhoods with population")
return count
def seed_median_age() -> int:
"""Seed median_age in fact_census where missing."""
engine = create_engine(DATABASE_URL)
with engine.begin() as conn:
result = conn.execute(
text("SELECT id FROM public.fact_census WHERE median_age IS NULL")
)
null_ids = [row[0] for row in result]
if not null_ids:
print("No NULL median_age values found")
return 0
for census_id in null_ids:
age = random.randint(30, 50)
conn.execute(
text("UPDATE public.fact_census SET median_age = :age WHERE id = :id"),
{"age": age, "id": census_id},
)
print(f"Seeded median_age for {len(null_ids)} census records")
return len(null_ids)
def seed_census_housing() -> int:
"""Seed housing columns in fact_census where missing."""
engine = create_engine(DATABASE_URL)
with engine.begin() as conn:
result = conn.execute(
text("SELECT id FROM public.fact_census WHERE pct_owner_occupied IS NULL")
)
null_ids = [row[0] for row in result]
if not null_ids:
print("No NULL census housing values found")
return 0
for census_id in null_ids:
conn.execute(
text(
"""
UPDATE public.fact_census SET
pct_owner_occupied = :owner,
pct_renter_occupied = :renter,
average_dwelling_value = :dwelling,
median_household_income = :income
WHERE id = :id
"""
),
{
"id": census_id,
"owner": round(random.uniform(30, 80), 1),
"renter": round(random.uniform(20, 70), 1),
"dwelling": random.randint(400000, 1500000),
"income": random.randint(50000, 180000),
},
)
print(f"Seeded census housing data for {len(null_ids)} records")
return len(null_ids)
def seed_housing_mart() -> int:
"""Seed housing mart with rental and affordability data for multiple years."""
engine = create_engine(DATABASE_URL)
total = 0
# First update existing NULL records
with engine.begin() as conn:
result = conn.execute(
text(
"""
SELECT neighbourhood_id, year
FROM public_marts.mart_neighbourhood_housing
WHERE avg_rent_2bed IS NULL
"""
)
)
rows = [dict(row._mapping) for row in result]
for row in rows:
avg_rent = random.randint(1800, 3200)
income = random.randint(55000, 180000)
rent_to_income = round((avg_rent * 12 / income) * 100, 2)
affordability = round(rent_to_income / 30 * 100, 1)
conn.execute(
text(
"""
UPDATE public_marts.mart_neighbourhood_housing SET
avg_rent_bachelor = :bachelor,
avg_rent_1bed = :onebed,
avg_rent_2bed = :twobed,
avg_rent_3bed = :threebed,
vacancy_rate = :vacancy,
rent_to_income_pct = :rent_income,
affordability_index = :afford_idx,
is_affordable = :is_afford,
median_household_income = :income,
pct_owner_occupied = :owner,
pct_renter_occupied = :renter
WHERE neighbourhood_id = :nid AND year = :year
"""
),
{
"nid": row["neighbourhood_id"],
"year": row["year"],
"bachelor": avg_rent - 500,
"onebed": avg_rent - 300,
"twobed": avg_rent,
"threebed": avg_rent + 400,
"vacancy": round(random.uniform(0.5, 4.5), 1),
"rent_income": rent_to_income,
"afford_idx": affordability,
"is_afford": rent_to_income <= 30,
"income": income,
"owner": round(random.uniform(30, 75), 1),
"renter": round(random.uniform(25, 70), 1),
},
)
total += 1
# Then insert multi-year data for trend charts
years = [2019, 2020, 2022, 2023, 2024]
with engine.begin() as conn:
result = conn.execute(
text(
"SELECT neighbourhood_id, name, geometry FROM public.dim_neighbourhood"
)
)
neighbourhoods = [dict(row._mapping) for row in result]
for n in neighbourhoods:
for year in years:
# Check if exists
exists = conn.execute(
text(
"""
SELECT 1 FROM public_marts.mart_neighbourhood_housing
WHERE neighbourhood_id = :nid AND year = :year
"""
),
{"nid": n["neighbourhood_id"], "year": year},
).fetchone()
if exists:
continue
base_rent = random.randint(1800, 2800)
year_factor = (year - 2019) * random.randint(50, 150)
avg_rent = base_rent + year_factor
income = random.randint(55000, 180000)
rent_to_income = round((avg_rent * 12 / income) * 100, 2)
conn.execute(
text(
"""
INSERT INTO public_marts.mart_neighbourhood_housing
(neighbourhood_id, neighbourhood_name, geometry, year,
avg_rent_bachelor, avg_rent_1bed, avg_rent_2bed, avg_rent_3bed,
vacancy_rate, rent_to_income_pct, affordability_index, is_affordable,
median_household_income, pct_owner_occupied, pct_renter_occupied)
VALUES
(:nid, :name, :geom, :year,
:bachelor, :onebed, :twobed, :threebed,
:vacancy, :rent_income, :afford_idx, :is_afford,
:income, :owner, :renter)
"""
),
{
"nid": n["neighbourhood_id"],
"name": n["name"],
"geom": n["geometry"],
"year": year,
"bachelor": avg_rent - 500,
"onebed": avg_rent - 300,
"twobed": avg_rent,
"threebed": avg_rent + 400,
"vacancy": round(random.uniform(0.5, 4.5), 1),
"rent_income": rent_to_income,
"afford_idx": round(rent_to_income / 30 * 100, 1),
"is_afford": rent_to_income <= 30,
"income": income,
"owner": round(random.uniform(30, 75), 1),
"renter": round(random.uniform(25, 70), 1),
},
)
total += 1
print(f"Seeded housing mart data for {total} records")
return total
def seed_overview_mart() -> int:
"""Seed overview mart with safety_score and population."""
engine = create_engine(DATABASE_URL)
total = 0
with engine.begin() as conn:
# Seed safety_score
result = conn.execute(
text(
"""
SELECT neighbourhood_id, year
FROM public_marts.mart_neighbourhood_overview
WHERE safety_score IS NULL
"""
)
)
rows = [dict(row._mapping) for row in result]
for row in rows:
conn.execute(
text(
"""
UPDATE public_marts.mart_neighbourhood_overview
SET safety_score = :score
WHERE neighbourhood_id = :nid AND year = :year
"""
),
{
"nid": row["neighbourhood_id"],
"year": row["year"],
"score": round(random.uniform(40, 95), 1),
},
)
total += 1
# Seed population
result = conn.execute(
text(
"""
SELECT neighbourhood_id, year
FROM public_marts.mart_neighbourhood_overview
WHERE population IS NULL
"""
)
)
rows = [dict(row._mapping) for row in result]
for row in rows:
conn.execute(
text(
"""
UPDATE public_marts.mart_neighbourhood_overview
SET population = :pop
WHERE neighbourhood_id = :nid AND year = :year
"""
),
{
"nid": row["neighbourhood_id"],
"year": row["year"],
"pop": random.randint(8000, 45000),
},
)
total += 1
print(f"Seeded overview mart data for {total} records")
return total
def run_dbt() -> bool:
"""Run dbt to rebuild marts."""
dbt_dir = PROJECT_ROOT / "dbt"
venv_dbt = PROJECT_ROOT / ".venv" / "bin" / "dbt"
dbt_cmd = str(venv_dbt) if venv_dbt.exists() else "dbt"
print("Running dbt to rebuild marts...")
env = os.environ.copy()
env["POSTGRES_PASSWORD"] = os.environ.get("POSTGRES_PASSWORD", "")
result = subprocess.run(
[dbt_cmd, "run", "--profiles-dir", str(dbt_dir)],
cwd=dbt_dir,
capture_output=True,
text=True,
env=env,
)
if result.returncode != 0:
print(f"dbt failed:\n{result.stdout}\n{result.stderr}")
return False
print("dbt completed successfully")
return True
def main() -> int:
"""Main entry point."""
print("Seeding development data...")
seed_amenities()
update_population()
seed_median_age()
seed_census_housing()
if not run_dbt():
return 1
# Seed mart tables after dbt rebuild
seed_housing_mart()
seed_overview_mart()
print("\nDone! Development data is ready.")
return 0
if __name__ == "__main__":
result = main()
sys.exit(result)

View File

@@ -15,6 +15,7 @@ from pathlib import Path
sys.path.insert(0, str(Path(__file__).parent.parent.parent))
from portfolio_app.toronto.models import create_tables, get_engine # noqa: E402
from portfolio_app.toronto.models.dimensions import RAW_TORONTO_SCHEMA # noqa: E402
def main() -> int:
@@ -32,16 +33,30 @@ def main() -> int:
result.fetchone()
print("Database connection successful")
# Create domain-specific schemas
with engine.connect() as conn:
conn.execute(text(f"CREATE SCHEMA IF NOT EXISTS {RAW_TORONTO_SCHEMA}"))
conn.commit()
print(f"Created schema: {RAW_TORONTO_SCHEMA}")
# Create all tables
create_tables()
print("Schema created successfully")
# List created tables
# List created tables by schema
from sqlalchemy import inspect
inspector = inspect(engine)
tables = inspector.get_table_names()
print(f"Created tables: {', '.join(tables)}")
# Public schema tables
public_tables = inspector.get_table_names(schema="public")
if public_tables:
print(f"Public schema tables: {', '.join(public_tables)}")
# raw_toronto schema tables
toronto_tables = inspector.get_table_names(schema=RAW_TORONTO_SCHEMA)
if toronto_tables:
print(f"{RAW_TORONTO_SCHEMA} schema tables: {', '.join(toronto_tables)}")
return 0

View File

@@ -39,14 +39,14 @@ case "$MODE" in
--full)
log "Running FULL data reload..."
log "Step 1/4: Parsing neighbourhood data..."
python -m portfolio_app.toronto.parsers.neighbourhoods 2>&1 | tee -a "$LOG_FILE"
log "Step 1/4: Parsing neighbourhood/geographic data..."
python -m portfolio_app.toronto.parsers.geo 2>&1 | tee -a "$LOG_FILE"
log "Step 2/4: Parsing census data..."
python -m portfolio_app.toronto.parsers.census 2>&1 | tee -a "$LOG_FILE"
log "Step 2/4: Parsing Toronto Open Data (census, amenities)..."
python -m portfolio_app.toronto.parsers.toronto_open_data 2>&1 | tee -a "$LOG_FILE"
log "Step 3/4: Parsing crime data..."
python -m portfolio_app.toronto.parsers.crime 2>&1 | tee -a "$LOG_FILE"
python -m portfolio_app.toronto.parsers.toronto_police 2>&1 | tee -a "$LOG_FILE"
log "Step 4/4: Running dbt transformations..."
cd dbt && dbt run --full-refresh --profiles-dir . 2>&1 | tee -a "$LOG_FILE" && cd ..