personal-portfolio/docs/runbooks/adding-dashboard.md

Runbook: Adding a New Dashboard

This runbook describes how to add a new data dashboard to the portfolio application.

Prerequisites

• Data sources identified and accessible
• Database schema designed
• Basic Dash/Plotly familiarity

Directory Structure

Create the following structure under portfolio_app/:

portfolio_app/
├── pages/
│   └── {dashboard_name}/
│       ├── dashboard.py      # Main layout with tabs
│       ├── methodology.py    # Data sources and methods page
│       ├── tabs/
│       │   ├── __init__.py
│       │   ├── overview.py   # Overview tab layout
│       │   └── ...           # Additional tab layouts
│       └── callbacks/
│           ├── __init__.py
│           └── ...           # Callback modules
├── {dashboard_name}/         # Data logic (outside pages/)
│   ├── __init__.py
│   ├── parsers/              # API/CSV extraction
│   │   └── __init__.py
│   ├── loaders/              # Database operations
│   │   └── __init__.py
│   ├── schemas/              # Pydantic models
│   │   └── __init__.py
│   └── models/               # SQLAlchemy ORM
│       └── __init__.py

Step-by-Step Checklist

1. Data Layer

• Create Pydantic schemas in {dashboard_name}/schemas/
• Create SQLAlchemy models in {dashboard_name}/models/
• Create parsers in {dashboard_name}/parsers/
• Create loaders in {dashboard_name}/loaders/
• Add database migrations if needed

2. dbt Models

Create dbt models in dbt/models/:

• staging/stg_{source}__{entity}.sql - Raw data cleaning
• intermediate/int_{domain}__{transform}.sql - Business logic
• marts/mart_{domain}.sql - Final analytical tables

Follow naming conventions:

• Staging: stg_{source}__{entity}
• Intermediate: int_{domain}__{transform}
• Marts: mart_{domain}

3. Visualization Layer

• Create figure factories in figures/ (or reuse existing)
• Follow the factory pattern: create_{chart_type}_figure(data, **kwargs)

4. Dashboard Pages

Main Dashboard (pages/{dashboard_name}/dashboard.py)

import dash
from dash import html, dcc
import dash_mantine_components as dmc

dash.register_page(
    __name__,
    path="/{dashboard_name}",
    title="{Dashboard Title}",
    description="{Description}"
)

def layout():
    return dmc.Container([
        # Header
        dmc.Title("{Dashboard Title}", order=1),

        # Tabs
        dmc.Tabs([
            dmc.TabsList([
                dmc.TabsTab("Overview", value="overview"),
                # Add more tabs
            ]),
            dmc.TabsPanel(overview_tab(), value="overview"),
            # Add more panels
        ], value="overview"),
    ])

Tab Layouts (pages/{dashboard_name}/tabs/)

• Create one file per tab
• Export layout function from each

Callbacks (pages/{dashboard_name}/callbacks/)

• Create callback modules for interactivity
• Import and register in dashboard.py

5. Navigation

Add to sidebar in components/sidebar.py:

dmc.NavLink(
    label="{Dashboard Name}",
    href="/{dashboard_name}",
    icon=DashIconify(icon="..."),
)

6. Documentation

• Create methodology page (pages/{dashboard_name}/methodology.py)
• Document data sources
• Document transformation logic
• Add notebooks to notebooks/{dashboard_name}/ if needed

7. Testing

• Add unit tests for parsers
• Add unit tests for loaders
• Add integration tests for callbacks
• Run make test

8. Final Verification

• All pages render without errors
• All callbacks respond correctly
• Data loads successfully
• dbt models run cleanly (make dbt-run)
• Linting passes (make lint)
• Tests pass (make test)

Example: Toronto Dashboard

Reference implementation: portfolio_app/pages/toronto/

Key files:

• dashboard.py - Main layout with 5 tabs
• tabs/overview.py - Livability scores, scatter plots
• callbacks/map_callbacks.py - Choropleth interactions
• toronto/models/dimensions.py - Dimension tables
• toronto/models/facts.py - Fact tables

Common Patterns

Figure Factories

# figures/choropleth.py
def create_choropleth_figure(
    gdf: gpd.GeoDataFrame,
    value_column: str,
    title: str,
    **kwargs
) -> go.Figure:
    ...

Callbacks

# callbacks/map_callbacks.py
@callback(
    Output("neighbourhood-details", "children"),
    Input("choropleth-map", "clickData"),
)
def update_details(click_data):
    ...

Data Loading

# {dashboard_name}/loaders/load.py
def load_data(session: Session) -> None:
    # Parse from source
    records = parse_source_data()

    # Validate with Pydantic
    validated = [Schema(**r) for r in records]

    # Load to database
    for record in validated:
        session.add(Model(**record.model_dump()))

    session.commit()