Some checks failed
CI / lint-and-test (pull_request) Has been cancelled
- Rename dbt project from toronto_housing to portfolio - Restructure dbt models into domain subdirectories: - shared/ for cross-domain dimensions (dim_time) - staging/toronto/, intermediate/toronto/, marts/toronto/ - Update SQLAlchemy models for raw_toronto schema - Add explicit cross-schema FK relationships for FactRentals - Namespace figure factories under figures/toronto/ - Namespace notebooks under notebooks/toronto/ - Update Makefile with domain-specific targets and env loading - Update all documentation for multi-dashboard structure This enables adding new dashboard projects (e.g., /football, /energy) without structural conflicts or naming collisions. Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
6.7 KiB
6.7 KiB
Runbook: Adding a New Dashboard
This runbook describes how to add a new data dashboard to the portfolio application.
Prerequisites
- Data sources identified and accessible
- Database schema designed
- Basic Dash/Plotly familiarity
Directory Structure
Create the following structure:
Application Code (portfolio_app/)
portfolio_app/
├── pages/
│ └── {dashboard_name}/
│ ├── dashboard.py # Main layout with tabs
│ ├── methodology.py # Data sources and methods page
│ ├── tabs/
│ │ ├── __init__.py
│ │ ├── overview.py # Overview tab layout
│ │ └── ... # Additional tab layouts
│ └── callbacks/
│ ├── __init__.py
│ └── ... # Callback modules
├── {dashboard_name}/ # Data logic (outside pages/)
│ ├── __init__.py
│ ├── parsers/ # API/CSV extraction
│ │ └── __init__.py
│ ├── loaders/ # Database operations
│ │ └── __init__.py
│ ├── schemas/ # Pydantic models
│ │ └── __init__.py
│ └── models/ # SQLAlchemy ORM (schema: raw_{dashboard_name})
│ └── __init__.py
└── figures/
└── {dashboard_name}/ # Figure factories for this dashboard
├── __init__.py
└── ... # Chart modules
dbt Models (dbt/models/)
dbt/models/
├── staging/
│ └── {dashboard_name}/ # Staging models
│ ├── _sources.yml # Source definitions (schema: raw_{dashboard_name})
│ ├── _staging.yml # Model tests/docs
│ └── stg_*.sql # Staging models
├── intermediate/
│ └── {dashboard_name}/ # Intermediate models
│ ├── _intermediate.yml
│ └── int_*.sql
└── marts/
└── {dashboard_name}/ # Mart tables
├── _marts.yml
└── mart_*.sql
Documentation (notebooks/)
notebooks/
└── {dashboard_name}/ # Domain subdirectories
├── overview/
├── ...
Step-by-Step Checklist
1. Data Layer
- Create Pydantic schemas in
{dashboard_name}/schemas/ - Create SQLAlchemy models in
{dashboard_name}/models/ - Create parsers in
{dashboard_name}/parsers/ - Create loaders in
{dashboard_name}/loaders/ - Add database migrations if needed
2. Database Schema
- Define schema constant in models (e.g.,
RAW_FOOTBALL_SCHEMA = "raw_football") - Add
__table_args__ = {"schema": RAW_FOOTBALL_SCHEMA}to all models - Update
scripts/db/init_schema.pyto create the new schema
3. dbt Models
Create dbt models in dbt/models/:
staging/{dashboard_name}/_sources.yml- Source definitions pointing toraw_{dashboard_name}schemastaging/{dashboard_name}/stg_{source}__{entity}.sql- Raw data cleaningintermediate/{dashboard_name}/int_{domain}__{transform}.sql- Business logicmarts/{dashboard_name}/mart_{domain}.sql- Final analytical tables
Update dbt/dbt_project.yml with new subdirectory config:
models:
portfolio:
staging:
{dashboard_name}:
+materialized: view
+schema: staging
Follow naming conventions:
- Staging:
stg_{source}__{entity} - Intermediate:
int_{domain}__{transform} - Marts:
mart_{domain}
4. Visualization Layer
- Create figure factories in
figures/{dashboard_name}/ - Create
figures/{dashboard_name}/__init__.pywith exports - Follow the factory pattern:
create_{chart_type}_figure(data, **kwargs)
Import pattern:
from portfolio_app.figures.{dashboard_name} import create_choropleth_figure
4. Dashboard Pages
Main Dashboard (pages/{dashboard_name}/dashboard.py)
import dash
from dash import html, dcc
import dash_mantine_components as dmc
dash.register_page(
__name__,
path="/{dashboard_name}",
title="{Dashboard Title}",
description="{Description}"
)
def layout():
return dmc.Container([
# Header
dmc.Title("{Dashboard Title}", order=1),
# Tabs
dmc.Tabs([
dmc.TabsList([
dmc.TabsTab("Overview", value="overview"),
# Add more tabs
]),
dmc.TabsPanel(overview_tab(), value="overview"),
# Add more panels
], value="overview"),
])
Tab Layouts (pages/{dashboard_name}/tabs/)
- Create one file per tab
- Export layout function from each
Callbacks (pages/{dashboard_name}/callbacks/)
- Create callback modules for interactivity
- Import and register in dashboard.py
5. Navigation
Add to sidebar in components/sidebar.py:
dmc.NavLink(
label="{Dashboard Name}",
href="/{dashboard_name}",
icon=DashIconify(icon="..."),
)
6. Documentation
- Create methodology page (
pages/{dashboard_name}/methodology.py) - Document data sources
- Document transformation logic
- Add notebooks to
notebooks/{dashboard_name}/if needed
7. Testing
- Add unit tests for parsers
- Add unit tests for loaders
- Add integration tests for callbacks
- Run
make test
8. Final Verification
- All pages render without errors
- All callbacks respond correctly
- Data loads successfully
- dbt models run cleanly (
make dbt-run) - Linting passes (
make lint) - Tests pass (
make test)
Example: Toronto Dashboard
Reference implementation: portfolio_app/pages/toronto/
Key files:
dashboard.py- Main layout with 5 tabstabs/overview.py- Livability scores, scatter plotscallbacks/map_callbacks.py- Choropleth interactionstoronto/models/dimensions.py- Dimension tablestoronto/models/facts.py- Fact tables
Common Patterns
Figure Factories
# figures/choropleth.py
def create_choropleth_figure(
gdf: gpd.GeoDataFrame,
value_column: str,
title: str,
**kwargs
) -> go.Figure:
...
Callbacks
# callbacks/map_callbacks.py
@callback(
Output("neighbourhood-details", "children"),
Input("choropleth-map", "clickData"),
)
def update_details(click_data):
...
Data Loading
# {dashboard_name}/loaders/load.py
def load_data(session: Session) -> None:
# Parse from source
records = parse_source_data()
# Validate with Pydantic
validated = [Schema(**r) for r in records]
# Load to database
for record in validated:
session.add(Model(**record.model_dump()))
session.commit()