diff --git a/CLAUDE.md b/CLAUDE.md index 2ebf7ca..78e5da7 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -17,15 +17,37 @@ Working context for Claude Code on the Analytics Portfolio project. ### Run Commands ```bash +# Setup & Database make setup # Install deps, create .env, init pre-commit make docker-up # Start PostgreSQL + PostGIS (auto-detects x86/ARM) make docker-down # Stop containers +make docker-logs # View container logs make db-init # Initialize database schema +make db-reset # Drop and recreate database (DESTRUCTIVE) + +# Data Loading +make load-data # Load Toronto data from APIs, seed dev data +make load-data-only # Load Toronto data without dbt or seeding +make seed-data # Seed sample development data + +# Application make run # Start Dash dev server + +# Testing & Quality make test # Run pytest +make test-cov # Run pytest with coverage make lint # Run ruff linter make format # Run ruff formatter -make ci # Run all checks +make typecheck # Run mypy type checker +make ci # Run all checks (lint, typecheck, test) + +# dbt +make dbt-run # Run dbt models +make dbt-test # Run dbt tests +make dbt-docs # Generate and serve dbt documentation + +# Maintenance +make clean # Remove build artifacts and caches ``` ### Branch Workflow @@ -55,9 +77,11 @@ make ci # Run all checks | `models/` | SQLAlchemy ORM | Database persistence | | `parsers/` | API/CSV extraction | Raw data ingestion | | `loaders/` | Database operations | Data loading | +| `services/` | Query functions | dbt mart queries, business logic | | `figures/` | Chart factories | Plotly figure generation | | `callbacks/` | Dash callbacks | In `pages/{dashboard}/callbacks/` | -| `errors/` | Exceptions + handlers | Error handling | +| `errors/` | Exception classes | Custom exceptions | +| `utils/` | Helper modules | Markdown loading, shared utilities | ### Type Hints @@ -96,78 +120,19 @@ class LoadError(PortfolioError): ## Application Structure -``` -portfolio_app/ -├── app.py # Dash app factory with Pages routing -├── config.py # Pydantic BaseSettings -├── assets/ # CSS, images (auto-served) -│ └── sidebar.css # Navigation styling -├── callbacks/ # Global callbacks -│ ├── sidebar.py # Sidebar toggle -│ └── theme.py # Dark/light theme -├── pages/ -│ ├── home.py # Bio landing page -> / -│ ├── about.py # About page -> /about -│ ├── contact.py # Contact form -> /contact -│ ├── health.py # Health endpoint -> /health -│ ├── projects.py # Project showcase -> /projects -│ ├── resume.py # Resume/CV -> /resume -│ ├── blog/ -│ │ ├── index.py # Blog listing -> /blog -│ │ └── article.py # Blog article -> /blog/{slug} -│ └── toronto/ -│ ├── dashboard.py # Dashboard -> /toronto -│ ├── methodology.py # Methodology -> /toronto/methodology -│ ├── tabs/ # 5 tab layouts (overview, housing, safety, demographics, amenities) -│ └── callbacks/ # Dashboard interactions -├── components/ # Shared UI (sidebar, cards, controls) -│ ├── metric_card.py # KPI card component -│ ├── map_controls.py # Map control panel -│ ├── sidebar.py # Navigation sidebar -│ └── time_slider.py # Time range selector -├── figures/ # Shared chart factories -│ ├── choropleth.py # Map visualizations -│ ├── bar_charts.py # Ranking, stacked, horizontal bars -│ ├── scatter.py # Scatter and bubble plots -│ ├── radar.py # Radar/spider charts -│ ├── demographics.py # Age pyramids, donut charts -│ ├── time_series.py # Trend lines -│ └── summary_cards.py # KPI figures -├── content/ # Markdown content -│ └── blog/ # Blog articles -├── toronto/ # Toronto data logic -│ ├── parsers/ -│ ├── loaders/ -│ ├── schemas/ # Pydantic -│ ├── models/ # SQLAlchemy -│ └── demo_data.py # Sample data -├── utils/ # Utilities -│ └── markdown_loader.py # Markdown processing -└── errors/ +**Entry Point:** `portfolio_app/app.py` (Dash app factory with Pages routing) -notebooks/ # Data documentation (Phase 6) -├── README.md # Template and usage guide -├── overview/ # Overview tab notebooks (3) -├── housing/ # Housing tab notebooks (3) -├── safety/ # Safety tab notebooks (3) -├── demographics/ # Demographics tab notebooks (3) -└── amenities/ # Amenities tab notebooks (3) -``` +| Directory | Purpose | Notes | +|-----------|---------|-------| +| `pages/` | Dash Pages (file-based routing) | URLs match file paths | +| `pages/toronto/` | Toronto Dashboard | `tabs/` for layouts, `callbacks/` for interactions | +| `components/` | Shared UI components | metric_card, sidebar, map_controls, time_slider | +| `figures/` | Plotly chart factories | choropleth, bar_charts, scatter, radar, time_series | +| `toronto/` | Toronto data logic | parsers/, loaders/, schemas/, models/ | +| `content/blog/` | Markdown blog articles | Processed by `utils/markdown_loader.py` | +| `notebooks/` | Data documentation | 5 domains: overview, housing, safety, demographics, amenities | -### URL Routing - -| URL | Page | Sprint | -|-----|------|--------| -| `/` | Bio landing page | 2 | -| `/about` | About page | 8 | -| `/contact` | Contact form | 8 | -| `/health` | Health endpoint | 8 | -| `/projects` | Project showcase | 8 | -| `/resume` | Resume/CV | 8 | -| `/blog` | Blog listing | 8 | -| `/blog/{slug}` | Blog article | 8 | -| `/toronto` | Toronto Dashboard | 6 | -| `/toronto/methodology` | Dashboard methodology | 6 | +**Key URLs:** `/` (home), `/toronto` (dashboard), `/blog` (listing), `/blog/{slug}` (articles) --- @@ -269,6 +234,8 @@ All scripts in `scripts/`: | Project reference | `docs/PROJECT_REFERENCE.md` | Architecture decisions, completed work | | Developer guide | `docs/CONTRIBUTING.md` | How to add pages, blog posts, tabs | | Lessons learned | `docs/project-lessons-learned/INDEX.md` | Past issues and solutions | +| Deployment runbook | `docs/runbooks/deployment.md` | Deploying to staging/production | +| Dashboard runbook | `docs/runbooks/adding-dashboard.md` | Adding new data dashboards | --- @@ -297,9 +264,10 @@ When user requests implementation work: ### Gitea Repository -- **Repo**: `lmiranda/personal-portfolio` +- **Repo**: `personal-projects/personal-portfolio` - **Host**: `gitea.hotserv.cloud` -- **Note**: `lmiranda` is a user account (not org), so label lookup may require repo-level labels +- **SSH**: `ssh://git@hotserv.tailc9b278.ts.net:2222/personal-projects/personal-portfolio.git` +- **Labels**: 18 repository-level labels configured (Type, Priority, Complexity, Effort) ### MCP Tools Available @@ -339,4 +307,48 @@ Every Gitea issue should include: --- +## Other Available Plugins + +### Code Quality: code-sentinel + +Use for security scanning and refactoring analysis. + +| Command | Purpose | +|---------|---------| +| `/code-sentinel:security-scan` | Full security audit of codebase | +| `/code-sentinel:refactor` | Apply refactoring patterns | +| `/code-sentinel:refactor-dry` | Preview refactoring without applying | + +**When to use:** Before major releases, after adding authentication/data handling code, periodic audits. + +### Documentation: doc-guardian + +Use for documentation drift detection and synchronization. + +| Command | Purpose | +|---------|---------| +| `/doc-guardian:doc-audit` | Scan project for documentation drift | +| `/doc-guardian:doc-sync` | Synchronize pending documentation updates | + +**When to use:** After significant code changes, before releases, when docs feel stale. + +### Pull Requests: pr-review + +Use for comprehensive PR review with multiple analysis perspectives. + +| Command | Purpose | +|---------|---------| +| `/pr-review:initial-setup` | Configure PR review for this project | +| `/pr-review:project-init` | Quick project-level setup | + +**When to use:** Before merging significant PRs to `development` or `main`. + +### Git Workflow: git-flow + +Use for git operations assistance. + +**When to use:** Complex merge scenarios, branch management questions. + +--- + *Last Updated: January 2026 (Post-Sprint 9)* diff --git a/docs/CONTRIBUTING.md b/docs/CONTRIBUTING.md index d4a0816..cfe3507 100644 --- a/docs/CONTRIBUTING.md +++ b/docs/CONTRIBUTING.md @@ -50,9 +50,11 @@ The app runs at `http://localhost:8050`. ```bash make test # Run tests +make test-cov # Run tests with coverage make lint # Check code style make format # Auto-format code -make ci # Run all checks (lint + test) +make typecheck # Run mypy type checker +make ci # Run all checks (lint, typecheck, test) make dbt-run # Run dbt transformations make dbt-test # Run dbt tests ``` @@ -247,13 +249,23 @@ def layout(slug: str = "") -> dmc.Container: To add the page to the sidebar, edit `portfolio_app/components/sidebar.py`: ```python -NAV_ITEMS = [ - {"label": "Home", "href": "/", "icon": "tabler:home"}, - {"label": "Your Page", "href": "/your-page", "icon": "tabler:star"}, +# For main pages (Home, About, Blog, etc.) +NAV_ITEMS_MAIN = [ + {"path": "/", "icon": "tabler:home", "label": "Home"}, + {"path": "/your-page", "icon": "tabler:star", "label": "Your Page"}, + # ... +] + +# For project/dashboard pages +NAV_ITEMS_PROJECTS = [ + {"path": "/projects", "icon": "tabler:folder", "label": "Projects"}, + {"path": "/your-dashboard", "icon": "tabler:chart-bar", "label": "Your Dashboard"}, # ... ] ``` +The sidebar uses icon buttons with tooltips. Each item needs `path`, `icon` (Tabler icon name), and `label` (tooltip text). + ### URL Routing Summary | File Location | URL | diff --git a/docs/DATABASE_SCHEMA.md b/docs/DATABASE_SCHEMA.md index 7336820..4ecf5f3 100644 --- a/docs/DATABASE_SCHEMA.md +++ b/docs/DATABASE_SCHEMA.md @@ -136,7 +136,11 @@ Staging models provide 1:1 cleaned representations of source data: | `stg_toronto__neighbourhoods` | raw.neighbourhoods | Cleaned boundaries with standardized names | | `stg_toronto__census` | raw.census_profiles | Typed census metrics | | `stg_cmhc__rentals` | raw.cmhc_rentals | Validated rental data | -| `stg_police__crimes` | raw.crime_data | Standardized crime categories | +| `stg_toronto__crime` | raw.crime_data | Standardized crime categories | +| `stg_toronto__amenities` | raw.amenities | Typed amenity counts | +| `stg_dimensions__time` | generated | Time dimension | +| `stg_dimensions__cmhc_zones` | raw.cmhc_zones | CMHC zone boundaries | +| `stg_cmhc__zone_crosswalk` | raw.crosswalk | Zone-neighbourhood mapping | ### Marts Schema (dbt) @@ -144,10 +148,12 @@ Analytical tables ready for dashboard consumption: | Model | Grain | Purpose | |-------|-------|---------| -| `mart_neighbourhood_summary` | neighbourhood | Composite livability scores | -| `mart_rental_trends` | zone × month | Time-series rental analysis | -| `mart_crime_rates` | neighbourhood × year | Crime rate calculations | -| `mart_amenity_density` | neighbourhood | Amenity accessibility scores | +| `mart_neighbourhood_overview` | neighbourhood | Composite livability scores | +| `mart_neighbourhood_housing` | neighbourhood | Housing and rent metrics | +| `mart_neighbourhood_safety` | neighbourhood × year | Crime rate calculations | +| `mart_neighbourhood_demographics` | neighbourhood | Income, age, population metrics | +| `mart_neighbourhood_amenities` | neighbourhood | Amenity accessibility scores | +| `mart_toronto_rentals` | zone × month | Time-series rental analysis | ## Table Details diff --git a/docs/PROJECT_REFERENCE.md b/docs/PROJECT_REFERENCE.md index 92ca37f..d9bec1b 100644 --- a/docs/PROJECT_REFERENCE.md +++ b/docs/PROJECT_REFERENCE.md @@ -91,12 +91,13 @@ portfolio_app/ │ ├── dashboard.py │ ├── methodology.py │ ├── tabs/ # 5 tab layouts -│ └── callbacks/ # Dashboard interactions +│ └── callbacks/ # Dashboard interactions (map_callbacks, chart_callbacks, selection_callbacks) ├── toronto/ # Data logic -│ ├── parsers/ # API extraction -│ ├── loaders/ # Database operations +│ ├── parsers/ # API extraction (geo, toronto_open_data, toronto_police, cmhc) +│ ├── loaders/ # Database operations (base, cmhc, cmhc_crosswalk) │ ├── schemas/ # Pydantic models │ ├── models/ # SQLAlchemy ORM +│ ├── services/ # Query functions (neighbourhood_service, geometry_service) │ └── demo_data.py # Sample data └── utils/ └── markdown_loader.py # Blog article loading @@ -241,16 +242,25 @@ LOG_LEVEL=INFO | Target | Purpose | |--------|---------| | `setup` | Install deps, create .env, init pre-commit | -| `docker-up` | Start PostgreSQL + PostGIS | +| `docker-up` | Start PostgreSQL + PostGIS (auto-detects x86/ARM) | | `docker-down` | Stop containers | +| `docker-logs` | View container logs | | `db-init` | Initialize database schema | +| `db-reset` | Drop and recreate database (DESTRUCTIVE) | +| `load-data` | Load Toronto data from APIs, seed dev data | +| `load-data-only` | Load Toronto data without dbt or seeding | +| `seed-data` | Seed sample development data | | `run` | Start Dash dev server | | `test` | Run pytest | -| `dbt-run` | Run dbt models | -| `dbt-test` | Run dbt tests | +| `test-cov` | Run pytest with coverage | | `lint` | Run ruff linter | | `format` | Run ruff formatter | -| `ci` | Run all checks | +| `typecheck` | Run mypy type checker | +| `ci` | Run all checks (lint, typecheck, test) | +| `dbt-run` | Run dbt models | +| `dbt-test` | Run dbt tests | +| `dbt-docs` | Generate and serve dbt documentation | +| `clean` | Remove build artifacts and caches | --- diff --git a/scripts/etl/toronto.sh b/scripts/etl/toronto.sh index 6019693..b770691 100755 --- a/scripts/etl/toronto.sh +++ b/scripts/etl/toronto.sh @@ -39,14 +39,14 @@ case "$MODE" in --full) log "Running FULL data reload..." - log "Step 1/4: Parsing neighbourhood data..." - python -m portfolio_app.toronto.parsers.neighbourhoods 2>&1 | tee -a "$LOG_FILE" + log "Step 1/4: Parsing neighbourhood/geographic data..." + python -m portfolio_app.toronto.parsers.geo 2>&1 | tee -a "$LOG_FILE" - log "Step 2/4: Parsing census data..." - python -m portfolio_app.toronto.parsers.census 2>&1 | tee -a "$LOG_FILE" + log "Step 2/4: Parsing Toronto Open Data (census, amenities)..." + python -m portfolio_app.toronto.parsers.toronto_open_data 2>&1 | tee -a "$LOG_FILE" log "Step 3/4: Parsing crime data..." - python -m portfolio_app.toronto.parsers.crime 2>&1 | tee -a "$LOG_FILE" + python -m portfolio_app.toronto.parsers.toronto_police 2>&1 | tee -a "$LOG_FILE" log "Step 4/4: Running dbt transformations..." cd dbt && dbt run --full-refresh --profiles-dir . 2>&1 | tee -a "$LOG_FILE" && cd ..