docs: sync documentation with codebase
Some checks failed
CI / lint-and-test (push) Has been cancelled
Some checks failed
CI / lint-and-test (push) Has been cancelled
Fixes identified by doc-guardian audit: Critical fixes: - DATABASE_SCHEMA.md: Fix staging model name stg_police__crimes → stg_toronto__crime - DATABASE_SCHEMA.md: Update mart model names to match actual dbt models - CLAUDE.md: Fix errors/ description (no handlers module exists) - scripts/etl/toronto.sh: Fix parser module references to actual modules Stale fixes: - CONTRIBUTING.md: Add make typecheck, test-cov; fix make ci description - PROJECT_REFERENCE.md: Document services/, callback modules, all Makefile targets - CLAUDE.md: Expand Makefile commands, add plugin documentation Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
This commit is contained in:
160
CLAUDE.md
160
CLAUDE.md
@@ -17,15 +17,37 @@ Working context for Claude Code on the Analytics Portfolio project.
|
||||
### Run Commands
|
||||
|
||||
```bash
|
||||
# Setup & Database
|
||||
make setup # Install deps, create .env, init pre-commit
|
||||
make docker-up # Start PostgreSQL + PostGIS (auto-detects x86/ARM)
|
||||
make docker-down # Stop containers
|
||||
make docker-logs # View container logs
|
||||
make db-init # Initialize database schema
|
||||
make db-reset # Drop and recreate database (DESTRUCTIVE)
|
||||
|
||||
# Data Loading
|
||||
make load-data # Load Toronto data from APIs, seed dev data
|
||||
make load-data-only # Load Toronto data without dbt or seeding
|
||||
make seed-data # Seed sample development data
|
||||
|
||||
# Application
|
||||
make run # Start Dash dev server
|
||||
|
||||
# Testing & Quality
|
||||
make test # Run pytest
|
||||
make test-cov # Run pytest with coverage
|
||||
make lint # Run ruff linter
|
||||
make format # Run ruff formatter
|
||||
make ci # Run all checks
|
||||
make typecheck # Run mypy type checker
|
||||
make ci # Run all checks (lint, typecheck, test)
|
||||
|
||||
# dbt
|
||||
make dbt-run # Run dbt models
|
||||
make dbt-test # Run dbt tests
|
||||
make dbt-docs # Generate and serve dbt documentation
|
||||
|
||||
# Maintenance
|
||||
make clean # Remove build artifacts and caches
|
||||
```
|
||||
|
||||
### Branch Workflow
|
||||
@@ -55,9 +77,11 @@ make ci # Run all checks
|
||||
| `models/` | SQLAlchemy ORM | Database persistence |
|
||||
| `parsers/` | API/CSV extraction | Raw data ingestion |
|
||||
| `loaders/` | Database operations | Data loading |
|
||||
| `services/` | Query functions | dbt mart queries, business logic |
|
||||
| `figures/` | Chart factories | Plotly figure generation |
|
||||
| `callbacks/` | Dash callbacks | In `pages/{dashboard}/callbacks/` |
|
||||
| `errors/` | Exceptions + handlers | Error handling |
|
||||
| `errors/` | Exception classes | Custom exceptions |
|
||||
| `utils/` | Helper modules | Markdown loading, shared utilities |
|
||||
|
||||
### Type Hints
|
||||
|
||||
@@ -96,78 +120,19 @@ class LoadError(PortfolioError):
|
||||
|
||||
## Application Structure
|
||||
|
||||
```
|
||||
portfolio_app/
|
||||
├── app.py # Dash app factory with Pages routing
|
||||
├── config.py # Pydantic BaseSettings
|
||||
├── assets/ # CSS, images (auto-served)
|
||||
│ └── sidebar.css # Navigation styling
|
||||
├── callbacks/ # Global callbacks
|
||||
│ ├── sidebar.py # Sidebar toggle
|
||||
│ └── theme.py # Dark/light theme
|
||||
├── pages/
|
||||
│ ├── home.py # Bio landing page -> /
|
||||
│ ├── about.py # About page -> /about
|
||||
│ ├── contact.py # Contact form -> /contact
|
||||
│ ├── health.py # Health endpoint -> /health
|
||||
│ ├── projects.py # Project showcase -> /projects
|
||||
│ ├── resume.py # Resume/CV -> /resume
|
||||
│ ├── blog/
|
||||
│ │ ├── index.py # Blog listing -> /blog
|
||||
│ │ └── article.py # Blog article -> /blog/{slug}
|
||||
│ └── toronto/
|
||||
│ ├── dashboard.py # Dashboard -> /toronto
|
||||
│ ├── methodology.py # Methodology -> /toronto/methodology
|
||||
│ ├── tabs/ # 5 tab layouts (overview, housing, safety, demographics, amenities)
|
||||
│ └── callbacks/ # Dashboard interactions
|
||||
├── components/ # Shared UI (sidebar, cards, controls)
|
||||
│ ├── metric_card.py # KPI card component
|
||||
│ ├── map_controls.py # Map control panel
|
||||
│ ├── sidebar.py # Navigation sidebar
|
||||
│ └── time_slider.py # Time range selector
|
||||
├── figures/ # Shared chart factories
|
||||
│ ├── choropleth.py # Map visualizations
|
||||
│ ├── bar_charts.py # Ranking, stacked, horizontal bars
|
||||
│ ├── scatter.py # Scatter and bubble plots
|
||||
│ ├── radar.py # Radar/spider charts
|
||||
│ ├── demographics.py # Age pyramids, donut charts
|
||||
│ ├── time_series.py # Trend lines
|
||||
│ └── summary_cards.py # KPI figures
|
||||
├── content/ # Markdown content
|
||||
│ └── blog/ # Blog articles
|
||||
├── toronto/ # Toronto data logic
|
||||
│ ├── parsers/
|
||||
│ ├── loaders/
|
||||
│ ├── schemas/ # Pydantic
|
||||
│ ├── models/ # SQLAlchemy
|
||||
│ └── demo_data.py # Sample data
|
||||
├── utils/ # Utilities
|
||||
│ └── markdown_loader.py # Markdown processing
|
||||
└── errors/
|
||||
**Entry Point:** `portfolio_app/app.py` (Dash app factory with Pages routing)
|
||||
|
||||
notebooks/ # Data documentation (Phase 6)
|
||||
├── README.md # Template and usage guide
|
||||
├── overview/ # Overview tab notebooks (3)
|
||||
├── housing/ # Housing tab notebooks (3)
|
||||
├── safety/ # Safety tab notebooks (3)
|
||||
├── demographics/ # Demographics tab notebooks (3)
|
||||
└── amenities/ # Amenities tab notebooks (3)
|
||||
```
|
||||
| Directory | Purpose | Notes |
|
||||
|-----------|---------|-------|
|
||||
| `pages/` | Dash Pages (file-based routing) | URLs match file paths |
|
||||
| `pages/toronto/` | Toronto Dashboard | `tabs/` for layouts, `callbacks/` for interactions |
|
||||
| `components/` | Shared UI components | metric_card, sidebar, map_controls, time_slider |
|
||||
| `figures/` | Plotly chart factories | choropleth, bar_charts, scatter, radar, time_series |
|
||||
| `toronto/` | Toronto data logic | parsers/, loaders/, schemas/, models/ |
|
||||
| `content/blog/` | Markdown blog articles | Processed by `utils/markdown_loader.py` |
|
||||
| `notebooks/` | Data documentation | 5 domains: overview, housing, safety, demographics, amenities |
|
||||
|
||||
### URL Routing
|
||||
|
||||
| URL | Page | Sprint |
|
||||
|-----|------|--------|
|
||||
| `/` | Bio landing page | 2 |
|
||||
| `/about` | About page | 8 |
|
||||
| `/contact` | Contact form | 8 |
|
||||
| `/health` | Health endpoint | 8 |
|
||||
| `/projects` | Project showcase | 8 |
|
||||
| `/resume` | Resume/CV | 8 |
|
||||
| `/blog` | Blog listing | 8 |
|
||||
| `/blog/{slug}` | Blog article | 8 |
|
||||
| `/toronto` | Toronto Dashboard | 6 |
|
||||
| `/toronto/methodology` | Dashboard methodology | 6 |
|
||||
**Key URLs:** `/` (home), `/toronto` (dashboard), `/blog` (listing), `/blog/{slug}` (articles)
|
||||
|
||||
---
|
||||
|
||||
@@ -269,6 +234,8 @@ All scripts in `scripts/`:
|
||||
| Project reference | `docs/PROJECT_REFERENCE.md` | Architecture decisions, completed work |
|
||||
| Developer guide | `docs/CONTRIBUTING.md` | How to add pages, blog posts, tabs |
|
||||
| Lessons learned | `docs/project-lessons-learned/INDEX.md` | Past issues and solutions |
|
||||
| Deployment runbook | `docs/runbooks/deployment.md` | Deploying to staging/production |
|
||||
| Dashboard runbook | `docs/runbooks/adding-dashboard.md` | Adding new data dashboards |
|
||||
|
||||
---
|
||||
|
||||
@@ -297,9 +264,10 @@ When user requests implementation work:
|
||||
|
||||
### Gitea Repository
|
||||
|
||||
- **Repo**: `lmiranda/personal-portfolio`
|
||||
- **Repo**: `personal-projects/personal-portfolio`
|
||||
- **Host**: `gitea.hotserv.cloud`
|
||||
- **Note**: `lmiranda` is a user account (not org), so label lookup may require repo-level labels
|
||||
- **SSH**: `ssh://git@hotserv.tailc9b278.ts.net:2222/personal-projects/personal-portfolio.git`
|
||||
- **Labels**: 18 repository-level labels configured (Type, Priority, Complexity, Effort)
|
||||
|
||||
### MCP Tools Available
|
||||
|
||||
@@ -339,4 +307,48 @@ Every Gitea issue should include:
|
||||
|
||||
---
|
||||
|
||||
## Other Available Plugins
|
||||
|
||||
### Code Quality: code-sentinel
|
||||
|
||||
Use for security scanning and refactoring analysis.
|
||||
|
||||
| Command | Purpose |
|
||||
|---------|---------|
|
||||
| `/code-sentinel:security-scan` | Full security audit of codebase |
|
||||
| `/code-sentinel:refactor` | Apply refactoring patterns |
|
||||
| `/code-sentinel:refactor-dry` | Preview refactoring without applying |
|
||||
|
||||
**When to use:** Before major releases, after adding authentication/data handling code, periodic audits.
|
||||
|
||||
### Documentation: doc-guardian
|
||||
|
||||
Use for documentation drift detection and synchronization.
|
||||
|
||||
| Command | Purpose |
|
||||
|---------|---------|
|
||||
| `/doc-guardian:doc-audit` | Scan project for documentation drift |
|
||||
| `/doc-guardian:doc-sync` | Synchronize pending documentation updates |
|
||||
|
||||
**When to use:** After significant code changes, before releases, when docs feel stale.
|
||||
|
||||
### Pull Requests: pr-review
|
||||
|
||||
Use for comprehensive PR review with multiple analysis perspectives.
|
||||
|
||||
| Command | Purpose |
|
||||
|---------|---------|
|
||||
| `/pr-review:initial-setup` | Configure PR review for this project |
|
||||
| `/pr-review:project-init` | Quick project-level setup |
|
||||
|
||||
**When to use:** Before merging significant PRs to `development` or `main`.
|
||||
|
||||
### Git Workflow: git-flow
|
||||
|
||||
Use for git operations assistance.
|
||||
|
||||
**When to use:** Complex merge scenarios, branch management questions.
|
||||
|
||||
---
|
||||
|
||||
*Last Updated: January 2026 (Post-Sprint 9)*
|
||||
|
||||
@@ -50,9 +50,11 @@ The app runs at `http://localhost:8050`.
|
||||
|
||||
```bash
|
||||
make test # Run tests
|
||||
make test-cov # Run tests with coverage
|
||||
make lint # Check code style
|
||||
make format # Auto-format code
|
||||
make ci # Run all checks (lint + test)
|
||||
make typecheck # Run mypy type checker
|
||||
make ci # Run all checks (lint, typecheck, test)
|
||||
make dbt-run # Run dbt transformations
|
||||
make dbt-test # Run dbt tests
|
||||
```
|
||||
@@ -247,13 +249,23 @@ def layout(slug: str = "") -> dmc.Container:
|
||||
To add the page to the sidebar, edit `portfolio_app/components/sidebar.py`:
|
||||
|
||||
```python
|
||||
NAV_ITEMS = [
|
||||
{"label": "Home", "href": "/", "icon": "tabler:home"},
|
||||
{"label": "Your Page", "href": "/your-page", "icon": "tabler:star"},
|
||||
# For main pages (Home, About, Blog, etc.)
|
||||
NAV_ITEMS_MAIN = [
|
||||
{"path": "/", "icon": "tabler:home", "label": "Home"},
|
||||
{"path": "/your-page", "icon": "tabler:star", "label": "Your Page"},
|
||||
# ...
|
||||
]
|
||||
|
||||
# For project/dashboard pages
|
||||
NAV_ITEMS_PROJECTS = [
|
||||
{"path": "/projects", "icon": "tabler:folder", "label": "Projects"},
|
||||
{"path": "/your-dashboard", "icon": "tabler:chart-bar", "label": "Your Dashboard"},
|
||||
# ...
|
||||
]
|
||||
```
|
||||
|
||||
The sidebar uses icon buttons with tooltips. Each item needs `path`, `icon` (Tabler icon name), and `label` (tooltip text).
|
||||
|
||||
### URL Routing Summary
|
||||
|
||||
| File Location | URL |
|
||||
|
||||
@@ -136,7 +136,11 @@ Staging models provide 1:1 cleaned representations of source data:
|
||||
| `stg_toronto__neighbourhoods` | raw.neighbourhoods | Cleaned boundaries with standardized names |
|
||||
| `stg_toronto__census` | raw.census_profiles | Typed census metrics |
|
||||
| `stg_cmhc__rentals` | raw.cmhc_rentals | Validated rental data |
|
||||
| `stg_police__crimes` | raw.crime_data | Standardized crime categories |
|
||||
| `stg_toronto__crime` | raw.crime_data | Standardized crime categories |
|
||||
| `stg_toronto__amenities` | raw.amenities | Typed amenity counts |
|
||||
| `stg_dimensions__time` | generated | Time dimension |
|
||||
| `stg_dimensions__cmhc_zones` | raw.cmhc_zones | CMHC zone boundaries |
|
||||
| `stg_cmhc__zone_crosswalk` | raw.crosswalk | Zone-neighbourhood mapping |
|
||||
|
||||
### Marts Schema (dbt)
|
||||
|
||||
@@ -144,10 +148,12 @@ Analytical tables ready for dashboard consumption:
|
||||
|
||||
| Model | Grain | Purpose |
|
||||
|-------|-------|---------|
|
||||
| `mart_neighbourhood_summary` | neighbourhood | Composite livability scores |
|
||||
| `mart_rental_trends` | zone × month | Time-series rental analysis |
|
||||
| `mart_crime_rates` | neighbourhood × year | Crime rate calculations |
|
||||
| `mart_amenity_density` | neighbourhood | Amenity accessibility scores |
|
||||
| `mart_neighbourhood_overview` | neighbourhood | Composite livability scores |
|
||||
| `mart_neighbourhood_housing` | neighbourhood | Housing and rent metrics |
|
||||
| `mart_neighbourhood_safety` | neighbourhood × year | Crime rate calculations |
|
||||
| `mart_neighbourhood_demographics` | neighbourhood | Income, age, population metrics |
|
||||
| `mart_neighbourhood_amenities` | neighbourhood | Amenity accessibility scores |
|
||||
| `mart_toronto_rentals` | zone × month | Time-series rental analysis |
|
||||
|
||||
## Table Details
|
||||
|
||||
|
||||
@@ -91,12 +91,13 @@ portfolio_app/
|
||||
│ ├── dashboard.py
|
||||
│ ├── methodology.py
|
||||
│ ├── tabs/ # 5 tab layouts
|
||||
│ └── callbacks/ # Dashboard interactions
|
||||
│ └── callbacks/ # Dashboard interactions (map_callbacks, chart_callbacks, selection_callbacks)
|
||||
├── toronto/ # Data logic
|
||||
│ ├── parsers/ # API extraction
|
||||
│ ├── loaders/ # Database operations
|
||||
│ ├── parsers/ # API extraction (geo, toronto_open_data, toronto_police, cmhc)
|
||||
│ ├── loaders/ # Database operations (base, cmhc, cmhc_crosswalk)
|
||||
│ ├── schemas/ # Pydantic models
|
||||
│ ├── models/ # SQLAlchemy ORM
|
||||
│ ├── services/ # Query functions (neighbourhood_service, geometry_service)
|
||||
│ └── demo_data.py # Sample data
|
||||
└── utils/
|
||||
└── markdown_loader.py # Blog article loading
|
||||
@@ -241,16 +242,25 @@ LOG_LEVEL=INFO
|
||||
| Target | Purpose |
|
||||
|--------|---------|
|
||||
| `setup` | Install deps, create .env, init pre-commit |
|
||||
| `docker-up` | Start PostgreSQL + PostGIS |
|
||||
| `docker-up` | Start PostgreSQL + PostGIS (auto-detects x86/ARM) |
|
||||
| `docker-down` | Stop containers |
|
||||
| `docker-logs` | View container logs |
|
||||
| `db-init` | Initialize database schema |
|
||||
| `db-reset` | Drop and recreate database (DESTRUCTIVE) |
|
||||
| `load-data` | Load Toronto data from APIs, seed dev data |
|
||||
| `load-data-only` | Load Toronto data without dbt or seeding |
|
||||
| `seed-data` | Seed sample development data |
|
||||
| `run` | Start Dash dev server |
|
||||
| `test` | Run pytest |
|
||||
| `dbt-run` | Run dbt models |
|
||||
| `dbt-test` | Run dbt tests |
|
||||
| `test-cov` | Run pytest with coverage |
|
||||
| `lint` | Run ruff linter |
|
||||
| `format` | Run ruff formatter |
|
||||
| `ci` | Run all checks |
|
||||
| `typecheck` | Run mypy type checker |
|
||||
| `ci` | Run all checks (lint, typecheck, test) |
|
||||
| `dbt-run` | Run dbt models |
|
||||
| `dbt-test` | Run dbt tests |
|
||||
| `dbt-docs` | Generate and serve dbt documentation |
|
||||
| `clean` | Remove build artifacts and caches |
|
||||
|
||||
---
|
||||
|
||||
|
||||
@@ -39,14 +39,14 @@ case "$MODE" in
|
||||
--full)
|
||||
log "Running FULL data reload..."
|
||||
|
||||
log "Step 1/4: Parsing neighbourhood data..."
|
||||
python -m portfolio_app.toronto.parsers.neighbourhoods 2>&1 | tee -a "$LOG_FILE"
|
||||
log "Step 1/4: Parsing neighbourhood/geographic data..."
|
||||
python -m portfolio_app.toronto.parsers.geo 2>&1 | tee -a "$LOG_FILE"
|
||||
|
||||
log "Step 2/4: Parsing census data..."
|
||||
python -m portfolio_app.toronto.parsers.census 2>&1 | tee -a "$LOG_FILE"
|
||||
log "Step 2/4: Parsing Toronto Open Data (census, amenities)..."
|
||||
python -m portfolio_app.toronto.parsers.toronto_open_data 2>&1 | tee -a "$LOG_FILE"
|
||||
|
||||
log "Step 3/4: Parsing crime data..."
|
||||
python -m portfolio_app.toronto.parsers.crime 2>&1 | tee -a "$LOG_FILE"
|
||||
python -m portfolio_app.toronto.parsers.toronto_police 2>&1 | tee -a "$LOG_FILE"
|
||||
|
||||
log "Step 4/4: Running dbt transformations..."
|
||||
cd dbt && dbt run --full-refresh --profiles-dir . 2>&1 | tee -a "$LOG_FILE" && cd ..
|
||||
|
||||
Reference in New Issue
Block a user