refactor: multi-dashboard structural migration
Some checks failed
CI / lint-and-test (pull_request) Has been cancelled
Some checks failed
CI / lint-and-test (pull_request) Has been cancelled
- Rename dbt project from toronto_housing to portfolio - Restructure dbt models into domain subdirectories: - shared/ for cross-domain dimensions (dim_time) - staging/toronto/, intermediate/toronto/, marts/toronto/ - Update SQLAlchemy models for raw_toronto schema - Add explicit cross-schema FK relationships for FactRentals - Namespace figure factories under figures/toronto/ - Namespace notebooks under notebooks/toronto/ - Update Makefile with domain-specific targets and env loading - Update all documentation for multi-dashboard structure This enables adding new dashboard projects (e.g., /football, /energy) without structural conflicts or naming collisions. Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
This commit is contained in:
@@ -10,7 +10,9 @@ This runbook describes how to add a new data dashboard to the portfolio applicat
|
||||
|
||||
## Directory Structure
|
||||
|
||||
Create the following structure under `portfolio_app/`:
|
||||
Create the following structure:
|
||||
|
||||
### Application Code (`portfolio_app/`)
|
||||
|
||||
```
|
||||
portfolio_app/
|
||||
@@ -33,8 +35,40 @@ portfolio_app/
|
||||
│ │ └── __init__.py
|
||||
│ ├── schemas/ # Pydantic models
|
||||
│ │ └── __init__.py
|
||||
│ └── models/ # SQLAlchemy ORM
|
||||
│ └── models/ # SQLAlchemy ORM (schema: raw_{dashboard_name})
|
||||
│ └── __init__.py
|
||||
└── figures/
|
||||
└── {dashboard_name}/ # Figure factories for this dashboard
|
||||
├── __init__.py
|
||||
└── ... # Chart modules
|
||||
```
|
||||
|
||||
### dbt Models (`dbt/models/`)
|
||||
|
||||
```
|
||||
dbt/models/
|
||||
├── staging/
|
||||
│ └── {dashboard_name}/ # Staging models
|
||||
│ ├── _sources.yml # Source definitions (schema: raw_{dashboard_name})
|
||||
│ ├── _staging.yml # Model tests/docs
|
||||
│ └── stg_*.sql # Staging models
|
||||
├── intermediate/
|
||||
│ └── {dashboard_name}/ # Intermediate models
|
||||
│ ├── _intermediate.yml
|
||||
│ └── int_*.sql
|
||||
└── marts/
|
||||
└── {dashboard_name}/ # Mart tables
|
||||
├── _marts.yml
|
||||
└── mart_*.sql
|
||||
```
|
||||
|
||||
### Documentation (`notebooks/`)
|
||||
|
||||
```
|
||||
notebooks/
|
||||
└── {dashboard_name}/ # Domain subdirectories
|
||||
├── overview/
|
||||
├── ...
|
||||
```
|
||||
|
||||
## Step-by-Step Checklist
|
||||
@@ -47,24 +81,47 @@ portfolio_app/
|
||||
- [ ] Create loaders in `{dashboard_name}/loaders/`
|
||||
- [ ] Add database migrations if needed
|
||||
|
||||
### 2. dbt Models
|
||||
### 2. Database Schema
|
||||
|
||||
- [ ] Define schema constant in models (e.g., `RAW_FOOTBALL_SCHEMA = "raw_football"`)
|
||||
- [ ] Add `__table_args__ = {"schema": RAW_FOOTBALL_SCHEMA}` to all models
|
||||
- [ ] Update `scripts/db/init_schema.py` to create the new schema
|
||||
|
||||
### 3. dbt Models
|
||||
|
||||
Create dbt models in `dbt/models/`:
|
||||
|
||||
- [ ] `staging/stg_{source}__{entity}.sql` - Raw data cleaning
|
||||
- [ ] `intermediate/int_{domain}__{transform}.sql` - Business logic
|
||||
- [ ] `marts/mart_{domain}.sql` - Final analytical tables
|
||||
- [ ] `staging/{dashboard_name}/_sources.yml` - Source definitions pointing to `raw_{dashboard_name}` schema
|
||||
- [ ] `staging/{dashboard_name}/stg_{source}__{entity}.sql` - Raw data cleaning
|
||||
- [ ] `intermediate/{dashboard_name}/int_{domain}__{transform}.sql` - Business logic
|
||||
- [ ] `marts/{dashboard_name}/mart_{domain}.sql` - Final analytical tables
|
||||
|
||||
Update `dbt/dbt_project.yml` with new subdirectory config:
|
||||
```yaml
|
||||
models:
|
||||
portfolio:
|
||||
staging:
|
||||
{dashboard_name}:
|
||||
+materialized: view
|
||||
+schema: staging
|
||||
```
|
||||
|
||||
Follow naming conventions:
|
||||
- Staging: `stg_{source}__{entity}`
|
||||
- Intermediate: `int_{domain}__{transform}`
|
||||
- Marts: `mart_{domain}`
|
||||
|
||||
### 3. Visualization Layer
|
||||
### 4. Visualization Layer
|
||||
|
||||
- [ ] Create figure factories in `figures/` (or reuse existing)
|
||||
- [ ] Create figure factories in `figures/{dashboard_name}/`
|
||||
- [ ] Create `figures/{dashboard_name}/__init__.py` with exports
|
||||
- [ ] Follow the factory pattern: `create_{chart_type}_figure(data, **kwargs)`
|
||||
|
||||
Import pattern:
|
||||
```python
|
||||
from portfolio_app.figures.{dashboard_name} import create_choropleth_figure
|
||||
```
|
||||
|
||||
### 4. Dashboard Pages
|
||||
|
||||
#### Main Dashboard (`pages/{dashboard_name}/dashboard.py`)
|
||||
|
||||
Reference in New Issue
Block a user