docs: sync documentation with codebase

Fixes identified by doc-guardian audit: Critical fixes: - DATABASE_SCHEMA.md: Fix staging model name stg_police__crimes → stg_toronto__crime - DATABASE_SCHEMA.md: Update mart model names to match actual dbt models - CLAUDE.md: Fix errors/ description (no handlers module exists) - scripts/etl/toronto.sh: Fix parser module references to actual modules Stale fixes: - CONTRIBUTING.md: Add make typecheck, test-cov; fix make ci description - PROJECT_REFERENCE.md: Document services/, callback modules, all Makefile targets - CLAUDE.md: Expand Makefile commands, add plugin documentation Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
2026-01-22 16:25:29 -05:00
parent f345d41535
commit 263b52d5e4
5 changed files with 135 additions and 95 deletions
--- a/CLAUDE.md
+++ b/CLAUDE.md
@@ -17,15 +17,37 @@ Working context for Claude Code on the Analytics Portfolio project.
 ### Run Commands

 ```bash
+# Setup & Database
 make setup          # Install deps, create .env, init pre-commit
 make docker-up      # Start PostgreSQL + PostGIS (auto-detects x86/ARM)
 make docker-down    # Stop containers
+make docker-logs    # View container logs
 make db-init        # Initialize database schema
+make db-reset       # Drop and recreate database (DESTRUCTIVE)
+
+# Data Loading
+make load-data      # Load Toronto data from APIs, seed dev data
+make load-data-only # Load Toronto data without dbt or seeding
+make seed-data      # Seed sample development data
+
+# Application
 make run            # Start Dash dev server
+
+# Testing & Quality
 make test           # Run pytest
+make test-cov       # Run pytest with coverage
 make lint           # Run ruff linter
 make format         # Run ruff formatter
-make ci             # Run all checks
+make typecheck      # Run mypy type checker
+make ci             # Run all checks (lint, typecheck, test)
+
+# dbt
+make dbt-run        # Run dbt models
+make dbt-test       # Run dbt tests
+make dbt-docs       # Generate and serve dbt documentation
+
+# Maintenance
+make clean          # Remove build artifacts and caches
 ```

 ### Branch Workflow
@@ -55,9 +77,11 @@ make ci             # Run all checks
 | `models/` | SQLAlchemy ORM | Database persistence |
 | `parsers/` | API/CSV extraction | Raw data ingestion |
 | `loaders/` | Database operations | Data loading |
+| `services/` | Query functions | dbt mart queries, business logic |
 | `figures/` | Chart factories | Plotly figure generation |
 | `callbacks/` | Dash callbacks | In `pages/{dashboard}/callbacks/` |
-| `errors/` | Exceptions + handlers | Error handling |
+| `errors/` | Exception classes | Custom exceptions |
+| `utils/` | Helper modules | Markdown loading, shared utilities |

 ### Type Hints

@@ -96,78 +120,19 @@ class LoadError(PortfolioError):

 ## Application Structure

-```
-portfolio_app/
-├── app.py                    # Dash app factory with Pages routing
-├── config.py                 # Pydantic BaseSettings
-├── assets/                   # CSS, images (auto-served)
-│   └── sidebar.css          # Navigation styling
-├── callbacks/               # Global callbacks
-│   ├── sidebar.py           # Sidebar toggle
-│   └── theme.py             # Dark/light theme
-├── pages/
-│   ├── home.py              # Bio landing page -> /
-│   ├── about.py             # About page -> /about
-│   ├── contact.py           # Contact form -> /contact
-│   ├── health.py            # Health endpoint -> /health
-│   ├── projects.py          # Project showcase -> /projects
-│   ├── resume.py            # Resume/CV -> /resume
-│   ├── blog/
-│   │   ├── index.py         # Blog listing -> /blog
-│   │   └── article.py       # Blog article -> /blog/{slug}
-│   └── toronto/
-│       ├── dashboard.py     # Dashboard -> /toronto
-│       ├── methodology.py   # Methodology -> /toronto/methodology
-│       ├── tabs/            # 5 tab layouts (overview, housing, safety, demographics, amenities)
-│       └── callbacks/       # Dashboard interactions
-├── components/              # Shared UI (sidebar, cards, controls)
-│   ├── metric_card.py       # KPI card component
-│   ├── map_controls.py      # Map control panel
-│   ├── sidebar.py           # Navigation sidebar
-│   └── time_slider.py       # Time range selector
-├── figures/                 # Shared chart factories
-│   ├── choropleth.py        # Map visualizations
-│   ├── bar_charts.py        # Ranking, stacked, horizontal bars
-│   ├── scatter.py           # Scatter and bubble plots
-│   ├── radar.py             # Radar/spider charts
-│   ├── demographics.py      # Age pyramids, donut charts
-│   ├── time_series.py       # Trend lines
-│   └── summary_cards.py     # KPI figures
-├── content/                 # Markdown content
-│   └── blog/                # Blog articles
-├── toronto/                 # Toronto data logic
-│   ├── parsers/
-│   ├── loaders/
-│   ├── schemas/             # Pydantic
-│   ├── models/              # SQLAlchemy
-│   └── demo_data.py         # Sample data
-├── utils/                   # Utilities
-│   └── markdown_loader.py   # Markdown processing
-└── errors/
+**Entry Point:** `portfolio_app/app.py` (Dash app factory with Pages routing)

-notebooks/                   # Data documentation (Phase 6)
-├── README.md                # Template and usage guide
-├── overview/                # Overview tab notebooks (3)
-├── housing/                 # Housing tab notebooks (3)
-├── safety/                  # Safety tab notebooks (3)
-├── demographics/            # Demographics tab notebooks (3)
-└── amenities/               # Amenities tab notebooks (3)
-```
+| Directory | Purpose | Notes |
+|-----------|---------|-------|
+| `pages/` | Dash Pages (file-based routing) | URLs match file paths |
+| `pages/toronto/` | Toronto Dashboard | `tabs/` for layouts, `callbacks/` for interactions |
+| `components/` | Shared UI components | metric_card, sidebar, map_controls, time_slider |
+| `figures/` | Plotly chart factories | choropleth, bar_charts, scatter, radar, time_series |
+| `toronto/` | Toronto data logic | parsers/, loaders/, schemas/, models/ |
+| `content/blog/` | Markdown blog articles | Processed by `utils/markdown_loader.py` |
+| `notebooks/` | Data documentation | 5 domains: overview, housing, safety, demographics, amenities |

-### URL Routing
-
-| URL | Page | Sprint |
-|-----|------|--------|
-| `/` | Bio landing page | 2 |
-| `/about` | About page | 8 |
-| `/contact` | Contact form | 8 |
-| `/health` | Health endpoint | 8 |
-| `/projects` | Project showcase | 8 |
-| `/resume` | Resume/CV | 8 |
-| `/blog` | Blog listing | 8 |
-| `/blog/{slug}` | Blog article | 8 |
-| `/toronto` | Toronto Dashboard | 6 |
-| `/toronto/methodology` | Dashboard methodology | 6 |
+**Key URLs:** `/` (home), `/toronto` (dashboard), `/blog` (listing), `/blog/{slug}` (articles)

 ---

@@ -269,6 +234,8 @@ All scripts in `scripts/`:
 | Project reference | `docs/PROJECT_REFERENCE.md` | Architecture decisions, completed work |
 | Developer guide | `docs/CONTRIBUTING.md` | How to add pages, blog posts, tabs |
 | Lessons learned | `docs/project-lessons-learned/INDEX.md` | Past issues and solutions |
+| Deployment runbook | `docs/runbooks/deployment.md` | Deploying to staging/production |
+| Dashboard runbook | `docs/runbooks/adding-dashboard.md` | Adding new data dashboards |

 ---

@@ -297,9 +264,10 @@ When user requests implementation work:

 ### Gitea Repository

- **Repo**: `lmiranda/personal-portfolio`
+- **Repo**: `personal-projects/personal-portfolio`
 - **Host**: `gitea.hotserv.cloud`
- **Note**: `lmiranda` is a user account (not org), so label lookup may require repo-level labels
+- **SSH**: `ssh://git@hotserv.tailc9b278.ts.net:2222/personal-projects/personal-portfolio.git`
+- **Labels**: 18 repository-level labels configured (Type, Priority, Complexity, Effort)

 ### MCP Tools Available

@@ -339,4 +307,48 @@ Every Gitea issue should include:

 ---

+## Other Available Plugins
+
+### Code Quality: code-sentinel
+
+Use for security scanning and refactoring analysis.
+
+| Command | Purpose |
+|---------|---------|
+| `/code-sentinel:security-scan` | Full security audit of codebase |
+| `/code-sentinel:refactor` | Apply refactoring patterns |
+| `/code-sentinel:refactor-dry` | Preview refactoring without applying |
+
+**When to use:** Before major releases, after adding authentication/data handling code, periodic audits.
+
+### Documentation: doc-guardian
+
+Use for documentation drift detection and synchronization.
+
+| Command | Purpose |
+|---------|---------|
+| `/doc-guardian:doc-audit` | Scan project for documentation drift |
+| `/doc-guardian:doc-sync` | Synchronize pending documentation updates |
+
+**When to use:** After significant code changes, before releases, when docs feel stale.
+
+### Pull Requests: pr-review
+
+Use for comprehensive PR review with multiple analysis perspectives.
+
+| Command | Purpose |
+|---------|---------|
+| `/pr-review:initial-setup` | Configure PR review for this project |
+| `/pr-review:project-init` | Quick project-level setup |
+
+**When to use:** Before merging significant PRs to `development` or `main`.
+
+### Git Workflow: git-flow
+
+Use for git operations assistance.
+
+**When to use:** Complex merge scenarios, branch management questions.
+
+---
+
 *Last Updated: January 2026 (Post-Sprint 9)*
--- a/docs/CONTRIBUTING.md
+++ b/docs/CONTRIBUTING.md
@@ -50,9 +50,11 @@ The app runs at `http://localhost:8050`.

 ```bash
 make test       # Run tests
+make test-cov   # Run tests with coverage
 make lint       # Check code style
 make format     # Auto-format code
-make ci         # Run all checks (lint + test)
+make typecheck  # Run mypy type checker
+make ci         # Run all checks (lint, typecheck, test)
 make dbt-run    # Run dbt transformations
 make dbt-test   # Run dbt tests
 ```
@@ -247,13 +249,23 @@ def layout(slug: str = "") -> dmc.Container:
 To add the page to the sidebar, edit `portfolio_app/components/sidebar.py`:

 ```python
-NAV_ITEMS = [
-    {"label": "Home", "href": "/", "icon": "tabler:home"},
-    {"label": "Your Page", "href": "/your-page", "icon": "tabler:star"},
+# For main pages (Home, About, Blog, etc.)
+NAV_ITEMS_MAIN = [
+    {"path": "/", "icon": "tabler:home", "label": "Home"},
+    {"path": "/your-page", "icon": "tabler:star", "label": "Your Page"},
+    # ...
+]
+
+# For project/dashboard pages
+NAV_ITEMS_PROJECTS = [
+    {"path": "/projects", "icon": "tabler:folder", "label": "Projects"},
+    {"path": "/your-dashboard", "icon": "tabler:chart-bar", "label": "Your Dashboard"},
    # ...
 ]
 ```

+The sidebar uses icon buttons with tooltips. Each item needs `path`, `icon` (Tabler icon name), and `label` (tooltip text).
+
 ### URL Routing Summary

 | File Location | URL |
--- a/docs/DATABASE_SCHEMA.md
+++ b/docs/DATABASE_SCHEMA.md
@@ -136,7 +136,11 @@ Staging models provide 1:1 cleaned representations of source data:
 | `stg_toronto__neighbourhoods` | raw.neighbourhoods | Cleaned boundaries with standardized names |
 | `stg_toronto__census` | raw.census_profiles | Typed census metrics |
 | `stg_cmhc__rentals` | raw.cmhc_rentals | Validated rental data |
-| `stg_police__crimes` | raw.crime_data | Standardized crime categories |
+| `stg_toronto__crime` | raw.crime_data | Standardized crime categories |
+| `stg_toronto__amenities` | raw.amenities | Typed amenity counts |
+| `stg_dimensions__time` | generated | Time dimension |
+| `stg_dimensions__cmhc_zones` | raw.cmhc_zones | CMHC zone boundaries |
+| `stg_cmhc__zone_crosswalk` | raw.crosswalk | Zone-neighbourhood mapping |

 ### Marts Schema (dbt)

@@ -144,10 +148,12 @@ Analytical tables ready for dashboard consumption:

 | Model | Grain | Purpose |
 |-------|-------|---------|
-| `mart_neighbourhood_summary` | neighbourhood | Composite livability scores |
-| `mart_rental_trends` | zone × month | Time-series rental analysis |
-| `mart_crime_rates` | neighbourhood × year | Crime rate calculations |
-| `mart_amenity_density` | neighbourhood | Amenity accessibility scores |
+| `mart_neighbourhood_overview` | neighbourhood | Composite livability scores |
+| `mart_neighbourhood_housing` | neighbourhood | Housing and rent metrics |
+| `mart_neighbourhood_safety` | neighbourhood × year | Crime rate calculations |
+| `mart_neighbourhood_demographics` | neighbourhood | Income, age, population metrics |
+| `mart_neighbourhood_amenities` | neighbourhood | Amenity accessibility scores |
+| `mart_toronto_rentals` | zone × month | Time-series rental analysis |

 ## Table Details

--- a/docs/PROJECT_REFERENCE.md
+++ b/docs/PROJECT_REFERENCE.md
@@ -91,12 +91,13 @@ portfolio_app/
 │       ├── dashboard.py
 │       ├── methodology.py
 │       ├── tabs/             # 5 tab layouts
-│       └── callbacks/        # Dashboard interactions
+│       └── callbacks/        # Dashboard interactions (map_callbacks, chart_callbacks, selection_callbacks)
 ├── toronto/                  # Data logic
-│   ├── parsers/              # API extraction
-│   ├── loaders/              # Database operations
+│   ├── parsers/              # API extraction (geo, toronto_open_data, toronto_police, cmhc)
+│   ├── loaders/              # Database operations (base, cmhc, cmhc_crosswalk)
 │   ├── schemas/              # Pydantic models
 │   ├── models/               # SQLAlchemy ORM
+│   ├── services/             # Query functions (neighbourhood_service, geometry_service)
 │   └── demo_data.py          # Sample data
 └── utils/
    └── markdown_loader.py    # Blog article loading
@@ -241,16 +242,25 @@ LOG_LEVEL=INFO
 | Target | Purpose |
 |--------|---------|
 | `setup` | Install deps, create .env, init pre-commit |
-| `docker-up` | Start PostgreSQL + PostGIS |
+| `docker-up` | Start PostgreSQL + PostGIS (auto-detects x86/ARM) |
 | `docker-down` | Stop containers |
+| `docker-logs` | View container logs |
 | `db-init` | Initialize database schema |
+| `db-reset` | Drop and recreate database (DESTRUCTIVE) |
+| `load-data` | Load Toronto data from APIs, seed dev data |
+| `load-data-only` | Load Toronto data without dbt or seeding |
+| `seed-data` | Seed sample development data |
 | `run` | Start Dash dev server |
 | `test` | Run pytest |
-| `dbt-run` | Run dbt models |
-| `dbt-test` | Run dbt tests |
+| `test-cov` | Run pytest with coverage |
 | `lint` | Run ruff linter |
 | `format` | Run ruff formatter |
-| `ci` | Run all checks |
+| `typecheck` | Run mypy type checker |
+| `ci` | Run all checks (lint, typecheck, test) |
+| `dbt-run` | Run dbt models |
+| `dbt-test` | Run dbt tests |
+| `dbt-docs` | Generate and serve dbt documentation |
+| `clean` | Remove build artifacts and caches |

 ---

--- a/scripts/etl/toronto.sh
+++ b/scripts/etl/toronto.sh
@@ -39,14 +39,14 @@ case "$MODE" in
    --full)
        log "Running FULL data reload..."

-        log "Step 1/4: Parsing neighbourhood data..."
-        python -m portfolio_app.toronto.parsers.neighbourhoods 2>&1 | tee -a "$LOG_FILE"
+        log "Step 1/4: Parsing neighbourhood/geographic data..."
+        python -m portfolio_app.toronto.parsers.geo 2>&1 | tee -a "$LOG_FILE"

-        log "Step 2/4: Parsing census data..."
-        python -m portfolio_app.toronto.parsers.census 2>&1 | tee -a "$LOG_FILE"
+        log "Step 2/4: Parsing Toronto Open Data (census, amenities)..."
+        python -m portfolio_app.toronto.parsers.toronto_open_data 2>&1 | tee -a "$LOG_FILE"

        log "Step 3/4: Parsing crime data..."
-        python -m portfolio_app.toronto.parsers.crime 2>&1 | tee -a "$LOG_FILE"
+        python -m portfolio_app.toronto.parsers.toronto_police 2>&1 | tee -a "$LOG_FILE"

        log "Step 4/4: Running dbt transformations..."
        cd dbt && dbt run --full-refresh --profiles-dir . 2>&1 | tee -a "$LOG_FILE" && cd ..