refactor: multi-dashboard structural migration
Some checks failed
CI / lint-and-test (pull_request) Has been cancelled

- Rename dbt project from toronto_housing to portfolio
- Restructure dbt models into domain subdirectories:
  - shared/ for cross-domain dimensions (dim_time)
  - staging/toronto/, intermediate/toronto/, marts/toronto/
- Update SQLAlchemy models for raw_toronto schema
- Add explicit cross-schema FK relationships for FactRentals
- Namespace figure factories under figures/toronto/
- Namespace notebooks under notebooks/toronto/
- Update Makefile with domain-specific targets and env loading
- Update all documentation for multi-dashboard structure

This enables adding new dashboard projects (e.g., /football, /energy)
without structural conflicts or naming collisions.

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
This commit is contained in:
2026-02-01 19:08:20 -05:00
parent a5d6866d63
commit 62d1a52eed
73 changed files with 1114 additions and 623 deletions

View File

@@ -290,7 +290,7 @@ Dashboard tabs are in `portfolio_app/pages/toronto/tabs/`.
import dash_mantine_components as dmc
from portfolio_app.figures.choropleth import create_choropleth
from portfolio_app.figures.toronto.choropleth import create_choropleth
from portfolio_app.toronto.demo_data import get_demo_data
@@ -339,13 +339,13 @@ dmc.TabsPanel(create_your_tab_layout(), value="your-tab"),
## Creating Figure Factories
Figure factories are in `portfolio_app/figures/`. They create reusable Plotly figures.
Figure factories are organized by dashboard domain under `portfolio_app/figures/{domain}/`.
### Pattern
```python
# figures/your_chart.py
"""Your chart type factory."""
# figures/toronto/your_chart.py
"""Your chart type factory for Toronto dashboard."""
import plotly.express as px
import plotly.graph_objects as go
@@ -382,7 +382,7 @@ def create_your_chart(
### Export from `__init__.py`
```python
# figures/__init__.py
# figures/toronto/__init__.py
from .your_chart import create_your_chart
__all__ = [
@@ -391,6 +391,14 @@ __all__ = [
]
```
### Importing Figure Factories
```python
# In callbacks or tabs
from portfolio_app.figures.toronto import create_choropleth_figure
from portfolio_app.figures.toronto.bar_charts import create_ranking_bar
```
---
## Branch Workflow

View File

@@ -116,16 +116,38 @@ erDiagram
## Schema Layers
### Raw Schema
### Database Schemas
Raw data is loaded directly from external sources without transformation:
| Schema | Purpose | Managed By |
|--------|---------|------------|
| `public` | Shared dimensions (dim_time) | SQLAlchemy |
| `raw_toronto` | Toronto dimension and fact tables | SQLAlchemy |
| `staging` | Staging models | dbt |
| `intermediate` | Intermediate models | dbt |
| `marts` | Analytical tables | dbt |
### Raw Toronto Schema (raw_toronto)
Toronto-specific tables loaded by SQLAlchemy:
| Table | Source | Description |
|-------|--------|-------------|
| `raw.neighbourhoods` | City of Toronto API | GeoJSON neighbourhood boundaries |
| `raw.census_profiles` | City of Toronto API | Census profile data |
| `raw.crime_data` | Toronto Police API | Crime statistics by neighbourhood |
| `raw.cmhc_rentals` | CMHC Data Files | Rental market survey data |
| `dim_neighbourhood` | City of Toronto API | 158 neighbourhood boundaries |
| `dim_cmhc_zone` | CMHC | ~20 rental market zones |
| `dim_policy_event` | Manual | Policy events for annotation |
| `fact_census` | City of Toronto API | Census profile data |
| `fact_crime` | Toronto Police API | Crime statistics |
| `fact_amenities` | City of Toronto API | Amenity counts |
| `fact_rentals` | CMHC Data Files | Rental market survey data |
| `bridge_cmhc_neighbourhood` | Computed | Zone-neighbourhood mapping |
### Public Schema
Shared dimensions used across all projects:
| Table | Description |
|-------|-------------|
| `dim_time` | Time dimension (monthly grain) |
### Staging Schema (dbt)

View File

@@ -76,7 +76,8 @@ portfolio_app/
├── components/ # Shared UI components
├── content/blog/ # Markdown blog articles
├── errors/ # Exception handling
├── figures/ # Plotly figure factories
├── figures/
│ └── toronto/ # Toronto figure factories
├── pages/
│ ├── home.py
│ ├── about.py
@@ -96,11 +97,21 @@ portfolio_app/
│ ├── parsers/ # API extraction (geo, toronto_open_data, toronto_police, cmhc)
│ ├── loaders/ # Database operations (base, cmhc, cmhc_crosswalk)
│ ├── schemas/ # Pydantic models
│ ├── models/ # SQLAlchemy ORM
│ ├── models/ # SQLAlchemy ORM (raw_toronto schema)
│ ├── services/ # Query functions (neighbourhood_service, geometry_service)
│ └── demo_data.py # Sample data
└── utils/
└── markdown_loader.py # Blog article loading
dbt/ # dbt project: portfolio
├── models/
│ ├── shared/ # Cross-domain dimensions
│ ├── staging/toronto/ # Toronto staging models
│ ├── intermediate/toronto/ # Toronto intermediate models
│ └── marts/toronto/ # Toronto mart tables
notebooks/
└── toronto/ # Toronto documentation notebooks
```
---
@@ -144,10 +155,20 @@ CMHC Zones (~20) ← Rental data (Census Tract aligned)
| `fact_rentals` | Fact | Rental data by CMHC zone |
| `fact_amenities` | Fact | Amenity counts by neighbourhood |
### dbt Layers
### dbt Project: `portfolio`
**Model Structure:**
```
dbt/models/
├── shared/ # Cross-domain dimensions (stg_dimensions__time)
├── staging/toronto/ # Toronto staging models
├── intermediate/toronto/ # Toronto intermediate models
└── marts/toronto/ # Toronto mart tables
```
| Layer | Naming | Example |
|-------|--------|---------|
| Shared | `stg_dimensions__*` | `stg_dimensions__time` |
| Staging | `stg_{source}__{entity}` | `stg_toronto__neighbourhoods` |
| Intermediate | `int_{domain}__{transform}` | `int_neighbourhood__demographics` |
| Marts | `mart_{domain}` | `mart_neighbourhood_overview` |

View File

@@ -10,7 +10,9 @@ This runbook describes how to add a new data dashboard to the portfolio applicat
## Directory Structure
Create the following structure under `portfolio_app/`:
Create the following structure:
### Application Code (`portfolio_app/`)
```
portfolio_app/
@@ -33,8 +35,40 @@ portfolio_app/
│ │ └── __init__.py
│ ├── schemas/ # Pydantic models
│ │ └── __init__.py
│ └── models/ # SQLAlchemy ORM
│ └── models/ # SQLAlchemy ORM (schema: raw_{dashboard_name})
│ └── __init__.py
└── figures/
└── {dashboard_name}/ # Figure factories for this dashboard
├── __init__.py
└── ... # Chart modules
```
### dbt Models (`dbt/models/`)
```
dbt/models/
├── staging/
│ └── {dashboard_name}/ # Staging models
│ ├── _sources.yml # Source definitions (schema: raw_{dashboard_name})
│ ├── _staging.yml # Model tests/docs
│ └── stg_*.sql # Staging models
├── intermediate/
│ └── {dashboard_name}/ # Intermediate models
│ ├── _intermediate.yml
│ └── int_*.sql
└── marts/
└── {dashboard_name}/ # Mart tables
├── _marts.yml
└── mart_*.sql
```
### Documentation (`notebooks/`)
```
notebooks/
└── {dashboard_name}/ # Domain subdirectories
├── overview/
├── ...
```
## Step-by-Step Checklist
@@ -47,24 +81,47 @@ portfolio_app/
- [ ] Create loaders in `{dashboard_name}/loaders/`
- [ ] Add database migrations if needed
### 2. dbt Models
### 2. Database Schema
- [ ] Define schema constant in models (e.g., `RAW_FOOTBALL_SCHEMA = "raw_football"`)
- [ ] Add `__table_args__ = {"schema": RAW_FOOTBALL_SCHEMA}` to all models
- [ ] Update `scripts/db/init_schema.py` to create the new schema
### 3. dbt Models
Create dbt models in `dbt/models/`:
- [ ] `staging/stg_{source}__{entity}.sql` - Raw data cleaning
- [ ] `intermediate/int_{domain}__{transform}.sql` - Business logic
- [ ] `marts/mart_{domain}.sql` - Final analytical tables
- [ ] `staging/{dashboard_name}/_sources.yml` - Source definitions pointing to `raw_{dashboard_name}` schema
- [ ] `staging/{dashboard_name}/stg_{source}__{entity}.sql` - Raw data cleaning
- [ ] `intermediate/{dashboard_name}/int_{domain}__{transform}.sql` - Business logic
- [ ] `marts/{dashboard_name}/mart_{domain}.sql` - Final analytical tables
Update `dbt/dbt_project.yml` with new subdirectory config:
```yaml
models:
portfolio:
staging:
{dashboard_name}:
+materialized: view
+schema: staging
```
Follow naming conventions:
- Staging: `stg_{source}__{entity}`
- Intermediate: `int_{domain}__{transform}`
- Marts: `mart_{domain}`
### 3. Visualization Layer
### 4. Visualization Layer
- [ ] Create figure factories in `figures/` (or reuse existing)
- [ ] Create figure factories in `figures/{dashboard_name}/`
- [ ] Create `figures/{dashboard_name}/__init__.py` with exports
- [ ] Follow the factory pattern: `create_{chart_type}_figure(data, **kwargs)`
Import pattern:
```python
from portfolio_app.figures.{dashboard_name} import create_choropleth_figure
```
### 4. Dashboard Pages
#### Main Dashboard (`pages/{dashboard_name}/dashboard.py`)