refactor: update app code for domain-scoped schema migration
Some checks failed
CI / lint-and-test (pull_request) Has been cancelled
Some checks failed
CI / lint-and-test (pull_request) Has been cancelled
- Update dbt model references to use new schema naming (stg_toronto, int_toronto, mart_toronto) - Refactor figure factories to use consistent column naming from new schema - Update callbacks to work with refactored data structures - Add centralized design tokens module for consistent styling - Streamline CLAUDE.md documentation Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
This commit is contained in:
274
CLAUDE.md
274
CLAUDE.md
@@ -32,6 +32,17 @@
|
||||
|
||||
|
||||
|
||||
## Mandatory Behavior Rules
|
||||
|
||||
**These rules are NON-NEGOTIABLE. Violating them wastes the user's time and money.**
|
||||
|
||||
1. **CHECK EVERYTHING** - Search ALL locations before saying "no" (cache, installed, source directories)
|
||||
2. **BELIEVE THE USER** - Investigate thoroughly before disagreeing; user instincts are often right
|
||||
3. **VERIFY BEFORE "DONE"** - Run commands, show output; "done" means verified working
|
||||
4. **SHOW EXACTLY WHAT'S ASKED** - Do not interpret or summarize unless requested
|
||||
|
||||
---
|
||||
|
||||
Working context for Claude Code on the Analytics Portfolio project.
|
||||
|
||||
---
|
||||
@@ -53,22 +64,18 @@ Working context for Claude Code on the Analytics Portfolio project.
|
||||
make setup # Install deps, create .env, init pre-commit
|
||||
make docker-up # Start PostgreSQL + PostGIS (auto-detects x86/ARM)
|
||||
make docker-down # Stop containers
|
||||
make docker-logs # View container logs
|
||||
make db-init # Initialize database schema
|
||||
make db-reset # Drop and recreate database (DESTRUCTIVE)
|
||||
|
||||
# Data Loading
|
||||
make load-data # Load all project data (currently: Toronto)
|
||||
make load-toronto # Load Toronto data from APIs
|
||||
make load-toronto-only # Load Toronto data without dbt or seeding
|
||||
make seed-data # Seed sample development data
|
||||
|
||||
# Application
|
||||
make run # Start Dash dev server
|
||||
|
||||
# Testing & Quality
|
||||
make test # Run pytest
|
||||
make test-cov # Run pytest with coverage
|
||||
make lint # Run ruff linter
|
||||
make format # Run ruff formatter
|
||||
make typecheck # Run mypy type checker
|
||||
@@ -79,8 +86,7 @@ make dbt-run # Run dbt models
|
||||
make dbt-test # Run dbt tests
|
||||
make dbt-docs # Generate and serve dbt documentation
|
||||
|
||||
# Maintenance
|
||||
make clean # Remove build artifacts and caches
|
||||
# Run `make help` for full target list
|
||||
```
|
||||
|
||||
### Branch Workflow
|
||||
@@ -104,50 +110,22 @@ make clean # Remove build artifacts and caches
|
||||
|
||||
### Module Responsibilities
|
||||
|
||||
| Directory | Contains | Purpose |
|
||||
|-----------|----------|---------|
|
||||
| `schemas/` | Pydantic models | Data validation |
|
||||
| `models/` | SQLAlchemy ORM | Database persistence |
|
||||
| `parsers/` | API/CSV extraction | Raw data ingestion |
|
||||
| `loaders/` | Database operations | Data loading |
|
||||
| `services/` | Query functions | dbt mart queries, business logic |
|
||||
| `figures/` | Chart factories | Plotly figure generation |
|
||||
| `callbacks/` | Dash callbacks | In `pages/{dashboard}/callbacks/` |
|
||||
| `errors/` | Exception classes | Custom exceptions |
|
||||
| `utils/` | Helper modules | Markdown loading, shared utilities |
|
||||
|
||||
### Type Hints
|
||||
|
||||
Use Python 3.10+ style:
|
||||
```python
|
||||
def process(items: list[str], config: dict[str, int] | None = None) -> bool:
|
||||
...
|
||||
```
|
||||
|
||||
### Error Handling
|
||||
|
||||
```python
|
||||
# errors/exceptions.py
|
||||
class PortfolioError(Exception):
|
||||
"""Base exception."""
|
||||
|
||||
class ParseError(PortfolioError):
|
||||
"""PDF/CSV parsing failed."""
|
||||
|
||||
class ValidationError(PortfolioError):
|
||||
"""Pydantic or business rule validation failed."""
|
||||
|
||||
class LoadError(PortfolioError):
|
||||
"""Database load operation failed."""
|
||||
```
|
||||
| Directory | Purpose |
|
||||
|-----------|---------|
|
||||
| `schemas/` | Pydantic models for data validation |
|
||||
| `models/` | SQLAlchemy ORM for database persistence |
|
||||
| `parsers/` | API/CSV extraction for raw data ingestion |
|
||||
| `loaders/` | Database operations for data loading |
|
||||
| `services/` | Query functions for dbt mart queries |
|
||||
| `figures/` | Chart factories for Plotly figure generation |
|
||||
| `errors/` | Custom exception classes (see `errors/exceptions.py`) |
|
||||
|
||||
### Code Standards
|
||||
|
||||
- Python 3.10+ type hints: `list[str]`, `dict[str, int] | None`
|
||||
- Single responsibility functions with verb naming
|
||||
- Early returns over deep nesting
|
||||
- Google-style docstrings only for non-obvious behavior
|
||||
- Module-level constants for magic values
|
||||
- Pydantic BaseSettings for runtime config
|
||||
|
||||
---
|
||||
|
||||
@@ -155,23 +133,19 @@ class LoadError(PortfolioError):
|
||||
|
||||
**Entry Point:** `portfolio_app/app.py` (Dash app factory with Pages routing)
|
||||
|
||||
| Directory | Purpose | Notes |
|
||||
|-----------|---------|-------|
|
||||
| `pages/` | Dash Pages (file-based routing) | URLs match file paths |
|
||||
| `pages/toronto/` | Toronto Dashboard | `tabs/` for layouts, `callbacks/` for interactions |
|
||||
| `components/` | Shared UI components | metric_card, sidebar, map_controls, time_slider |
|
||||
| `figures/toronto/` | Toronto chart factories | choropleth, bar_charts, scatter, radar, time_series |
|
||||
| `toronto/` | Toronto data logic | parsers/, loaders/, schemas/, models/ |
|
||||
| `content/blog/` | Markdown blog articles | Processed by `utils/markdown_loader.py` |
|
||||
| `notebooks/toronto/` | Toronto documentation | 5 domains: overview, housing, safety, demographics, amenities |
|
||||
| Directory | Purpose |
|
||||
|-----------|---------|
|
||||
| `pages/` | Dash Pages (file-based routing) |
|
||||
| `pages/toronto/` | Toronto Dashboard (`tabs/` for layouts, `callbacks/` for interactions) |
|
||||
| `components/` | Shared UI components |
|
||||
| `figures/toronto/` | Toronto chart factories |
|
||||
| `toronto/` | Toronto data logic (parsers, loaders, schemas, models) |
|
||||
|
||||
**Key URLs:** `/` (home), `/toronto` (dashboard), `/blog` (listing), `/blog/{slug}` (articles)
|
||||
**Key URLs:** `/` (home), `/toronto` (dashboard), `/blog` (listing), `/blog/{slug}` (articles), `/health` (status)
|
||||
|
||||
### Multi-Dashboard Architecture
|
||||
|
||||
The codebase is structured to support multiple dashboard projects:
|
||||
- **figures/**: Domain-namespaced figure factories (`figures/toronto/`, future: `figures/football/`)
|
||||
- **notebooks/**: Domain-namespaced documentation (`notebooks/toronto/`, future: `notebooks/football/`)
|
||||
- **figures/**: Domain-namespaced (`figures/toronto/`, future: `figures/football/`)
|
||||
- **dbt models**: Domain subdirectories (`staging/toronto/`, `marts/toronto/`)
|
||||
- **Database schemas**: Domain-specific raw data (`raw_toronto`, future: `raw_football`)
|
||||
|
||||
@@ -185,18 +159,11 @@ The codebase is structured to support multiple dashboard projects:
|
||||
| Validation | Pydantic | >=2.0 |
|
||||
| ORM | SQLAlchemy | >=2.0 (2.0-style API only) |
|
||||
| Transformation | dbt-postgres | >=1.7 |
|
||||
| Data Processing | Pandas | >=2.1 |
|
||||
| Visualization | Dash + Plotly + dash-mantine-components | >=2.14 |
|
||||
| Geospatial | GeoPandas + Shapely | >=0.14 |
|
||||
| Visualization | Dash + Plotly | >=2.14 |
|
||||
| UI Components | dash-mantine-components | Latest stable |
|
||||
| Testing | pytest | >=7.0 |
|
||||
| Python | 3.11+ | Via pyenv |
|
||||
|
||||
**Notes**:
|
||||
- SQLAlchemy 2.0 + Pydantic 2.0 only (never mix 1.x APIs)
|
||||
- PostGIS extension required in database
|
||||
- Docker Compose V2 format (no `version` field)
|
||||
- **Multi-architecture support**: `make docker-up` auto-detects CPU architecture and uses the appropriate PostGIS image (x86_64: `postgis/postgis`, ARM64: `imresamu/postgis`)
|
||||
**Notes**: SQLAlchemy 2.0 + Pydantic 2.0 only. Docker Compose V2 format (no `version` field).
|
||||
|
||||
---
|
||||
|
||||
@@ -212,35 +179,8 @@ The codebase is structured to support multiple dashboard projects:
|
||||
| `int_toronto` | Toronto dbt intermediate views |
|
||||
| `mart_toronto` | Toronto dbt mart tables |
|
||||
|
||||
### Geographic Reality (Toronto Housing)
|
||||
|
||||
```
|
||||
City Neighbourhoods (158) - Primary geographic unit for analysis
|
||||
CMHC Zones (~20) - Rental data (Census Tract aligned)
|
||||
```
|
||||
|
||||
### Star Schema (raw_toronto)
|
||||
|
||||
| Table | Type | Keys |
|
||||
|-------|------|------|
|
||||
| `fact_rentals` | Fact | -> dim_time, dim_cmhc_zone |
|
||||
| `dim_time` | Dimension (public) | date_key (PK) - shared |
|
||||
| `dim_cmhc_zone` | Dimension | zone_key (PK), geometry |
|
||||
| `dim_neighbourhood` | Dimension | neighbourhood_id (PK), geometry |
|
||||
| `dim_policy_event` | Dimension | event_id (PK) |
|
||||
|
||||
### dbt Project: `portfolio`
|
||||
|
||||
**Model Structure:**
|
||||
```
|
||||
dbt/models/
|
||||
├── shared/ # Cross-domain dimensions
|
||||
│ └── stg_dimensions__time.sql
|
||||
├── staging/toronto/ # Toronto staging models
|
||||
├── intermediate/toronto/ # Toronto intermediate models
|
||||
└── marts/toronto/ # Toronto mart tables
|
||||
```
|
||||
|
||||
| Layer | Naming | Purpose |
|
||||
|-------|--------|---------|
|
||||
| Shared | `stg_dimensions__*` | Cross-domain dimensions |
|
||||
@@ -252,7 +192,7 @@ dbt/models/
|
||||
|
||||
## Deferred Features
|
||||
|
||||
**Stop and flag if a task seems to require these**:
|
||||
**Stop and flag if a task requires these**:
|
||||
|
||||
| Feature | Reason |
|
||||
|---------|--------|
|
||||
@@ -277,139 +217,123 @@ LOG_LEVEL=INFO
|
||||
|
||||
---
|
||||
|
||||
## Script Standards
|
||||
|
||||
All scripts in `scripts/`:
|
||||
- Include usage comments at top
|
||||
- Idempotent where possible
|
||||
- Exit codes: 0 = success, 1 = error
|
||||
- Use `set -euo pipefail` for bash
|
||||
- Log to stdout, errors to stderr
|
||||
|
||||
---
|
||||
|
||||
## Reference Documents
|
||||
|
||||
| Document | Location | Use When |
|
||||
|----------|----------|----------|
|
||||
| Project reference | `docs/PROJECT_REFERENCE.md` | Architecture decisions, completed work |
|
||||
| Developer guide | `docs/CONTRIBUTING.md` | How to add pages, blog posts, tabs |
|
||||
| Project reference | `docs/PROJECT_REFERENCE.md` | Architecture decisions |
|
||||
| Developer guide | `docs/CONTRIBUTING.md` | How to add pages, tabs |
|
||||
| Lessons learned | `docs/project-lessons-learned/INDEX.md` | Past issues and solutions |
|
||||
| Deployment runbook | `docs/runbooks/deployment.md` | Deploying to staging/production |
|
||||
| Dashboard runbook | `docs/runbooks/adding-dashboard.md` | Adding new data dashboards |
|
||||
| Deployment runbook | `docs/runbooks/deployment.md` | Deploying to environments |
|
||||
|
||||
---
|
||||
|
||||
## Projman Plugin Workflow
|
||||
## Plugin Reference
|
||||
|
||||
**CRITICAL: Always use the projman plugin for sprint and task management.**
|
||||
### Sprint Management: projman
|
||||
|
||||
### When to Use Projman Skills
|
||||
**CRITICAL: Always use projman for sprint and task management.**
|
||||
|
||||
| Skill | Trigger | Purpose |
|
||||
|-------|---------|---------|
|
||||
| `/projman:sprint-plan` | New sprint or phase implementation | Architecture analysis + Gitea issue creation |
|
||||
| `/projman:sprint-start` | Beginning implementation work | Load lessons learned (Wiki.js or local), start execution |
|
||||
| `/projman:sprint-status` | Check progress | Review blockers and completion status |
|
||||
| `/projman:sprint-close` | Sprint completion | Capture lessons learned (Wiki.js or local backup) |
|
||||
| `/projman:sprint-plan` | New sprint/feature | Architecture analysis + Gitea issue creation |
|
||||
| `/projman:sprint-start` | Begin implementation | Load lessons learned, start execution |
|
||||
| `/projman:sprint-status` | Check progress | Review blockers and completion |
|
||||
| `/projman:sprint-close` | Sprint completion | Capture lessons learned |
|
||||
|
||||
### Default Behavior
|
||||
**Default workflow**: `/projman:sprint-plan` before code -> create issues -> `/projman:sprint-start` -> track via Gitea -> `/projman:sprint-close`
|
||||
|
||||
When user requests implementation work:
|
||||
**Gitea**: `personal-projects/personal-portfolio` at `gitea.hotserv.cloud`
|
||||
|
||||
1. **ALWAYS start with `/projman:sprint-plan`** before writing code
|
||||
2. Create Gitea issues with proper labels and acceptance criteria
|
||||
3. Use `/projman:sprint-start` to begin execution with lessons learned
|
||||
4. Track progress via Gitea issue comments
|
||||
5. Close sprint with `/projman:sprint-close` to document lessons
|
||||
### Data Platform: data-platform
|
||||
|
||||
### Gitea Repository
|
||||
Use for dbt, PostgreSQL, and PostGIS operations.
|
||||
|
||||
- **Repo**: `personal-projects/personal-portfolio`
|
||||
- **Host**: `gitea.hotserv.cloud`
|
||||
- **SSH**: `ssh://git@hotserv.tailc9b278.ts.net:2222/personal-projects/personal-portfolio.git`
|
||||
- **Labels**: 18 repository-level labels configured (Type, Priority, Complexity, Effort)
|
||||
| Skill | Purpose |
|
||||
|-------|---------|
|
||||
| `/data-platform:data-review` | Audit data integrity, schema validity, dbt compliance |
|
||||
| `/data-platform:data-gate` | CI/CD data quality gate (pass/fail) |
|
||||
|
||||
### MCP Tools Available
|
||||
**When to use:** Schema changes, dbt model development, data loading, before merging data PRs.
|
||||
|
||||
**Gitea**:
|
||||
- `list_issues`, `get_issue`, `create_issue`, `update_issue`, `add_comment`
|
||||
- `get_labels`, `suggest_labels`
|
||||
**MCP tools available:** `pg_connect`, `pg_query`, `pg_tables`, `pg_columns`, `pg_schemas`, `st_*` (PostGIS), `dbt_*` operations.
|
||||
|
||||
**Wiki.js**:
|
||||
- `search_lessons`, `create_lesson`, `search_pages`, `get_page`
|
||||
### Visualization: viz-platform
|
||||
|
||||
### Lessons Learned (Backup Method)
|
||||
Use for Dash/Mantine component validation and chart creation.
|
||||
|
||||
**When Wiki.js is unavailable**, use the local backup in `docs/project-lessons-learned/`:
|
||||
| Skill | Purpose |
|
||||
|-------|---------|
|
||||
| `/viz-platform:component` | Inspect DMC component props and validation |
|
||||
| `/viz-platform:chart` | Create themed Plotly charts |
|
||||
| `/viz-platform:theme` | Apply/validate themes |
|
||||
| `/viz-platform:dashboard` | Create dashboard layouts |
|
||||
|
||||
**At Sprint Start:**
|
||||
1. Review `docs/project-lessons-learned/INDEX.md` for relevant past lessons
|
||||
2. Search lesson files by tags/keywords before implementation
|
||||
3. Apply prevention strategies from applicable lessons
|
||||
|
||||
**At Sprint Close:**
|
||||
1. Try Wiki.js `create_lesson` first
|
||||
2. If Wiki.js fails, create lesson in `docs/project-lessons-learned/`
|
||||
3. Use naming convention: `{phase-or-sprint}-{short-description}.md`
|
||||
4. Update `INDEX.md` with new entry
|
||||
5. Follow the lesson template in INDEX.md
|
||||
|
||||
**Migration:** Once Wiki.js is configured, lessons will be migrated there for better searchability.
|
||||
|
||||
### Issue Structure
|
||||
|
||||
Every Gitea issue should include:
|
||||
- **Overview**: Brief description
|
||||
- **Files to Create/Modify**: Explicit paths
|
||||
- **Acceptance Criteria**: Checkboxes
|
||||
- **Technical Notes**: Implementation hints
|
||||
- **Labels**: Listed in body (workaround for label API issues)
|
||||
|
||||
---
|
||||
|
||||
## Other Available Plugins
|
||||
**When to use:** Dashboard development, new visualizations, component prop lookup.
|
||||
|
||||
### Code Quality: code-sentinel
|
||||
|
||||
Use for security scanning and refactoring analysis.
|
||||
|
||||
| Command | Purpose |
|
||||
|---------|---------|
|
||||
| Skill | Purpose |
|
||||
|-------|---------|
|
||||
| `/code-sentinel:security-scan` | Full security audit of codebase |
|
||||
| `/code-sentinel:refactor` | Apply refactoring patterns |
|
||||
| `/code-sentinel:refactor-dry` | Preview refactoring without applying |
|
||||
|
||||
**When to use:** Before major releases, after adding authentication/data handling code, periodic audits.
|
||||
**When to use:** Before major releases, after adding auth/data handling code, periodic audits.
|
||||
|
||||
### Documentation: doc-guardian
|
||||
|
||||
Use for documentation drift detection and synchronization.
|
||||
|
||||
| Command | Purpose |
|
||||
|---------|---------|
|
||||
| Skill | Purpose |
|
||||
|-------|---------|
|
||||
| `/doc-guardian:doc-audit` | Scan project for documentation drift |
|
||||
| `/doc-guardian:doc-sync` | Synchronize pending documentation updates |
|
||||
|
||||
**When to use:** After significant code changes, before releases, when docs feel stale.
|
||||
**When to use:** After significant code changes, before releases.
|
||||
|
||||
### Pull Requests: pr-review
|
||||
|
||||
Use for comprehensive PR review with multiple analysis perspectives.
|
||||
|
||||
| Command | Purpose |
|
||||
|---------|---------|
|
||||
| `/pr-review:initial-setup` | Configure PR review for this project |
|
||||
| `/pr-review:project-init` | Quick project-level setup |
|
||||
| Skill | Purpose |
|
||||
|-------|---------|
|
||||
| `/pr-review:initial-setup` | Configure PR review for project |
|
||||
| Triggered automatically | Security, performance, maintainability, test analysis |
|
||||
|
||||
**When to use:** Before merging significant PRs to `development` or `main`.
|
||||
|
||||
### Requirement Clarification: clarity-assist
|
||||
|
||||
Use when requirements are ambiguous or need decomposition.
|
||||
|
||||
**When to use:** Unclear specifications, complex feature requests, conflicting requirements.
|
||||
|
||||
### Contract Validation: contract-validator
|
||||
|
||||
Use for plugin interface validation.
|
||||
|
||||
| Skill | Purpose |
|
||||
|-------|---------|
|
||||
| `/contract-validator:agent-check` | Quick agent definition validation |
|
||||
| `/contract-validator:full-validation` | Full plugin contract validation |
|
||||
|
||||
**When to use:** When modifying plugin integrations or agent definitions.
|
||||
|
||||
### Git Workflow: git-flow
|
||||
|
||||
Use for git operations assistance.
|
||||
Use for standardized git operations.
|
||||
|
||||
**When to use:** Complex merge scenarios, branch management questions.
|
||||
| Skill | Purpose |
|
||||
|-------|---------|
|
||||
| `/git-flow:commit` | Auto-generated conventional commit |
|
||||
| `/git-flow:branch-start` | Create feature/fix/chore branch |
|
||||
| `/git-flow:git-status` | Comprehensive status with recommendations |
|
||||
|
||||
**When to use:** Complex merge scenarios, branch management, standardized commits.
|
||||
|
||||
---
|
||||
|
||||
*Last Updated: February 2026 (Multi-Dashboard Architecture)*
|
||||
*Last Updated: February 2026*
|
||||
|
||||
Reference in New Issue
Block a user