feat: Sprint 10 - Architecture docs, CI/CD, operational scripts
Some checks failed
CI / lint-and-test (push) Has been cancelled
Some checks failed
CI / lint-and-test (push) Has been cancelled
Phase 1 - Architecture Documentation: - Add Architecture section with Mermaid flowchart to README - Create docs/DATABASE_SCHEMA.md with full ERD Phase 2 - CI/CD: - Add CI badge to README - Create .gitea/workflows/ci.yml for linting and tests - Create .gitea/workflows/deploy-staging.yml - Create .gitea/workflows/deploy-production.yml Phase 3 - Operational Scripts: - Create scripts/logs.sh for docker compose log following - Create scripts/run-detached.sh with health check loop - Create scripts/etl/toronto.sh for Toronto data pipeline - Add Makefile targets: logs, run-detached, etl-toronto Phase 4 - Runbooks: - Create docs/runbooks/adding-dashboard.md - Create docs/runbooks/deployment.md Phase 5 - Hygiene: - Create MIT LICENSE file Phase 6 - Production: - Add live demo link to README (leodata.science) Closes #78, #79, #80, #81, #82, #83, #84, #85, #86, #87, #88, #89, #91 Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
This commit is contained in:
40
README.md
40
README.md
@@ -1,5 +1,9 @@
|
||||
# Analytics Portfolio
|
||||
|
||||
[](https://gitea.hotserv.cloud/lmiranda/personal-portfolio/actions)
|
||||
|
||||
**Live Demo:** [leodata.science](https://leodata.science)
|
||||
|
||||
A personal portfolio website showcasing data engineering and visualization capabilities, featuring an interactive Toronto Neighbourhood Dashboard.
|
||||
|
||||
## Live Pages
|
||||
@@ -32,6 +36,42 @@ An interactive choropleth dashboard analyzing Toronto's 158 official neighbourho
|
||||
- Toronto Police Service (crime statistics)
|
||||
- CMHC Rental Market Survey (rental data by zone)
|
||||
|
||||
## Architecture
|
||||
|
||||
```mermaid
|
||||
flowchart LR
|
||||
subgraph Sources
|
||||
A1[City of Toronto API]
|
||||
A2[Toronto Police API]
|
||||
A3[CMHC Data]
|
||||
end
|
||||
|
||||
subgraph ETL
|
||||
B1[Parsers]
|
||||
B2[Loaders]
|
||||
end
|
||||
|
||||
subgraph Database
|
||||
C1[(PostgreSQL/PostGIS)]
|
||||
C2[dbt Models]
|
||||
end
|
||||
|
||||
subgraph Application
|
||||
D1[Dash App]
|
||||
D2[Plotly Figures]
|
||||
end
|
||||
|
||||
A1 & A2 & A3 --> B1 --> B2 --> C1 --> C2 --> D1 --> D2
|
||||
```
|
||||
|
||||
**Pipeline Stages:**
|
||||
- **Sources**: External APIs and data files (City of Toronto, Toronto Police, CMHC)
|
||||
- **ETL**: Python parsers extract and validate data; loaders persist to database
|
||||
- **Database**: PostgreSQL with PostGIS for geospatial; dbt transforms raw → staging → marts
|
||||
- **Application**: Dash serves interactive dashboards with Plotly visualizations
|
||||
|
||||
For detailed database schema, see [docs/DATABASE_SCHEMA.md](docs/DATABASE_SCHEMA.md).
|
||||
|
||||
## Quick Start
|
||||
|
||||
```bash
|
||||
|
||||
Reference in New Issue
Block a user