docs: Rewrite documentation with accurate project state
- Delete obsolete change proposals and bio content source - Rewrite README.md with correct features, data sources, structure - Update PROJECT_REFERENCE.md with accurate status and completed work - Update CLAUDE.md references and sprint status - Add docs/CONTRIBUTING.md developer guide with: - How to add blog posts (frontmatter, markdown) - How to add new pages (Dash routing) - How to add dashboard tabs - How to create figure factories - Branch workflow and code standards Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
This commit is contained in:
139
README.md
139
README.md
@@ -1,36 +1,42 @@
|
||||
# Analytics Portfolio
|
||||
|
||||
A data analytics portfolio showcasing end-to-end data engineering, visualization, and analysis capabilities.
|
||||
A personal portfolio website showcasing data engineering and visualization capabilities, featuring an interactive Toronto Neighbourhood Dashboard.
|
||||
|
||||
## Projects
|
||||
## Live Pages
|
||||
|
||||
### Toronto Housing Dashboard
|
||||
| Route | Page | Description |
|
||||
|-------|------|-------------|
|
||||
| `/` | Home | Bio landing page |
|
||||
| `/about` | About | Background and experience |
|
||||
| `/projects` | Projects | Portfolio project showcase |
|
||||
| `/resume` | Resume | Professional CV |
|
||||
| `/contact` | Contact | Contact form |
|
||||
| `/blog` | Blog | Technical articles |
|
||||
| `/blog/{slug}` | Article | Individual blog posts |
|
||||
| `/toronto` | Toronto Dashboard | Neighbourhood analysis (5 tabs) |
|
||||
| `/toronto/methodology` | Methodology | Dashboard data sources and methods |
|
||||
| `/health` | Health | API health check endpoint |
|
||||
|
||||
An interactive choropleth dashboard analyzing Toronto's housing market using multi-source data integration.
|
||||
## Toronto Neighbourhood Dashboard
|
||||
|
||||
**Features:**
|
||||
- Purchase market analysis from TRREB monthly reports
|
||||
- Rental market analysis from CMHC annual surveys
|
||||
- Interactive choropleth maps by district/zone
|
||||
- Time series visualization with policy event annotations
|
||||
- Purchase/Rental mode toggle
|
||||
An interactive choropleth dashboard analyzing Toronto's 158 official neighbourhoods across five dimensions:
|
||||
|
||||
**Data Sources:**
|
||||
- [TRREB Market Watch](https://trreb.ca/market-data/market-watch/) - Monthly purchase statistics
|
||||
- [CMHC Rental Market Survey](https://www.cmhc-schl.gc.ca/professionals/housing-markets-data-and-research/housing-data/data-tables/rental-market) - Annual rental data
|
||||
- **Overview**: Composite livability scores, income vs safety scatter
|
||||
- **Housing**: Affordability index, rent trends, dwelling types
|
||||
- **Safety**: Crime rates, breakdowns by type, trend analysis
|
||||
- **Demographics**: Income distribution, age pyramids, population density
|
||||
- **Amenities**: Parks, schools, transit accessibility
|
||||
|
||||
**Tech Stack:**
|
||||
- Python 3.11+ / Dash / Plotly
|
||||
- PostgreSQL + PostGIS
|
||||
- dbt for data transformation
|
||||
- Pydantic for validation
|
||||
- SQLAlchemy 2.0
|
||||
**Data Sources**:
|
||||
- City of Toronto Open Data Portal (neighbourhoods, census profiles, amenities)
|
||||
- Toronto Police Service (crime statistics)
|
||||
- CMHC Rental Market Survey (rental data by zone)
|
||||
|
||||
## Quick Start
|
||||
|
||||
```bash
|
||||
# Clone and setup
|
||||
git clone https://github.com/lmiranda/personal-portfolio.git
|
||||
git clone https://gitea.hotserv.cloud/lmiranda/personal-portfolio.git
|
||||
cd personal-portfolio
|
||||
|
||||
# Install dependencies and configure environment
|
||||
@@ -55,48 +61,72 @@ portfolio_app/
|
||||
├── app.py # Dash app factory
|
||||
├── config.py # Pydantic settings
|
||||
├── pages/
|
||||
│ ├── home.py # Bio landing page (/)
|
||||
│ └── toronto/ # Toronto dashboard (/toronto)
|
||||
│ ├── home.py # Bio landing (/)
|
||||
│ ├── about.py # About page
|
||||
│ ├── contact.py # Contact form
|
||||
│ ├── projects.py # Project showcase
|
||||
│ ├── resume.py # Resume/CV
|
||||
│ ├── blog/ # Blog system
|
||||
│ │ ├── index.py # Article listing
|
||||
│ │ └── article.py # Article renderer
|
||||
│ └── toronto/ # Toronto dashboard
|
||||
│ ├── dashboard.py # Main layout with tabs
|
||||
│ ├── methodology.py # Data documentation
|
||||
│ ├── tabs/ # Tab layouts (5)
|
||||
│ └── callbacks/ # Interaction logic
|
||||
├── components/ # Shared UI components
|
||||
├── figures/ # Plotly figure factories
|
||||
└── toronto/ # Toronto data logic
|
||||
├── parsers/ # PDF/CSV extraction
|
||||
├── loaders/ # Database operations
|
||||
├── schemas/ # Pydantic models
|
||||
└── models/ # SQLAlchemy ORM
|
||||
├── content/
|
||||
│ └── blog/ # Markdown blog articles
|
||||
├── toronto/ # Toronto data logic
|
||||
│ ├── parsers/ # API data extraction
|
||||
│ ├── loaders/ # Database operations
|
||||
│ ├── schemas/ # Pydantic models
|
||||
│ └── models/ # SQLAlchemy ORM
|
||||
└── errors/ # Exception handling
|
||||
|
||||
dbt/
|
||||
├── models/
|
||||
│ ├── staging/ # 1:1 source tables
|
||||
│ ├── intermediate/ # Business logic
|
||||
│ └── marts/ # Analytical tables
|
||||
│ ├── staging/ # 1:1 source tables
|
||||
│ ├── intermediate/ # Business logic
|
||||
│ └── marts/ # Analytical tables
|
||||
|
||||
notebooks/ # Data documentation (15 notebooks)
|
||||
├── overview/ # Overview tab visualizations
|
||||
├── housing/ # Housing tab visualizations
|
||||
├── safety/ # Safety tab visualizations
|
||||
├── demographics/ # Demographics tab visualizations
|
||||
└── amenities/ # Amenities tab visualizations
|
||||
|
||||
docs/
|
||||
├── PROJECT_REFERENCE.md # Architecture reference
|
||||
├── CONTRIBUTING.md # Developer guide
|
||||
└── project-lessons-learned/
|
||||
```
|
||||
|
||||
## Tech Stack
|
||||
|
||||
| Layer | Technology |
|
||||
|-------|------------|
|
||||
| Database | PostgreSQL 16 + PostGIS |
|
||||
| Validation | Pydantic 2.x |
|
||||
| ORM | SQLAlchemy 2.x |
|
||||
| Transformation | dbt-postgres |
|
||||
| Data Processing | Pandas, GeoPandas |
|
||||
| Visualization | Dash + Plotly |
|
||||
| UI Components | dash-mantine-components |
|
||||
| Testing | pytest |
|
||||
| Python | 3.11+ |
|
||||
|
||||
## Development
|
||||
|
||||
```bash
|
||||
make test # Run tests
|
||||
make lint # Run linter
|
||||
make test # Run pytest
|
||||
make lint # Run ruff linter
|
||||
make format # Format code
|
||||
make ci # Run all checks
|
||||
```
|
||||
|
||||
## Data Pipeline
|
||||
|
||||
```
|
||||
Raw Files (PDF/Excel)
|
||||
↓
|
||||
Parsers (pdfplumber, pandas)
|
||||
↓
|
||||
Pydantic Validation
|
||||
↓
|
||||
SQLAlchemy Loaders
|
||||
↓
|
||||
PostgreSQL + PostGIS
|
||||
↓
|
||||
dbt Transformations
|
||||
↓
|
||||
Dash Visualization
|
||||
make dbt-run # Run dbt models
|
||||
make dbt-test # Run dbt tests
|
||||
```
|
||||
|
||||
## Environment Variables
|
||||
@@ -109,12 +139,19 @@ POSTGRES_USER=portfolio
|
||||
POSTGRES_PASSWORD=<secure>
|
||||
POSTGRES_DB=portfolio
|
||||
DASH_DEBUG=true
|
||||
SECRET_KEY=<random>
|
||||
```
|
||||
|
||||
## Documentation
|
||||
|
||||
- **For developers**: See `docs/CONTRIBUTING.md` for setup and contribution guidelines
|
||||
- **For Claude Code**: See `CLAUDE.md` for AI assistant context
|
||||
- **Architecture**: See `docs/PROJECT_REFERENCE.md` for technical details
|
||||
|
||||
## License
|
||||
|
||||
MIT
|
||||
|
||||
## Author
|
||||
|
||||
Leo Miranda - [GitHub](https://github.com/lmiranda) | [LinkedIn](https://linkedin.com/in/yourprofile)
|
||||
Leo Miranda
|
||||
|
||||
Reference in New Issue
Block a user