personal-projects/personal-portfolio
Template
1
1
Fork 0
Code Issues 3 Pull Requests Actions Packages Projects 1 Releases Wiki Activity

Create data loading pipeline script #65

New Issue
Closed
opened 2026-01-17 16:07:39 +00:00 by lmiranda · 0 comments
lmiranda commented 2026-01-17 16:07:39 +00:00
Owner
Copy Link

Summary

Create an orchestration script that fetches data from Toronto Open Data and Toronto Police APIs, loads it into PostgreSQL, and runs dbt to populate the mart tables.

Files to Create

File Purpose
scripts/data/load_toronto_data.py Main orchestration script
scripts/data/__init__.py Package init

Script Workflow

  1. Fetch from APIs:

    • Neighbourhoods (boundaries)
    • Census profiles (demographics)
    • Crime rates (safety)
    • Parks, Schools, Childcare (amenities)
    • CMHC rentals (housing)

  2. Load to Database:

    • Use existing loaders (census.py, crime.py, amenities.py, etc.)
    • Upsert to avoid duplicates

  3. Run dbt:

    • Execute dbt run to transform staging → intermediate → marts
    • Validate with dbt test

Acceptance Criteria

  • Script can be run via python scripts/data/load_toronto_data.py
  • Supports --skip-fetch flag to only run dbt
  • Supports --skip-dbt flag to only load data
  • Logs progress and errors clearly
  • Handles API failures gracefully with retries
  • Makefile target added: make load-data

Technical Notes

  • Use existing parsers and loaders from portfolio_app/toronto/
  • Consider adding progress bars for long-running operations
  • Cache API responses to avoid re-fetching during development

Labels: type:feature, component:backend, priority:high, tech:python, tech:dbt

## Summary Create an orchestration script that fetches data from Toronto Open Data and Toronto Police APIs, loads it into PostgreSQL, and runs dbt to populate the mart tables. ## Files to Create | File | Purpose | |------|---------| | `scripts/data/load_toronto_data.py` | Main orchestration script | | `scripts/data/__init__.py` | Package init | ## Script Workflow 1. **Fetch from APIs:** - Neighbourhoods (boundaries) - Census profiles (demographics) - Crime rates (safety) - Parks, Schools, Childcare (amenities) - CMHC rentals (housing) 2. **Load to Database:** - Use existing loaders (census.py, crime.py, amenities.py, etc.) - Upsert to avoid duplicates 3. **Run dbt:** - Execute `dbt run` to transform staging → intermediate → marts - Validate with `dbt test` ## Acceptance Criteria - [ ] Script can be run via `python scripts/data/load_toronto_data.py` - [ ] Supports `--skip-fetch` flag to only run dbt - [ ] Supports `--skip-dbt` flag to only load data - [ ] Logs progress and errors clearly - [ ] Handles API failures gracefully with retries - [ ] Makefile target added: `make load-data` ## Technical Notes - Use existing parsers and loaders from `portfolio_app/toronto/` - Consider adding progress bars for long-running operations - Cache API responses to avoid re-fetching during development **Labels:** type:feature, component:backend, priority:high, tech:python, tech:dbt
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
Labels
Clear labels
Complexity/Complex
High uncertainty or many dependencies
 Complexity/Medium
Some unknowns or dependencies
 Complexity/Simple
Straightforward, well-understood
 Effort/L
3-5 days
 Effort/M
1-2 days
 Effort/S
2-4 hours
 Effort/XL
> 1 week
 Effort/XS
< 2 hours
 Priority/Critical
Immediate attention required
 Priority/High
Important, needs attention soon
 Priority/Low
Nice to have, can wait
 Priority/Medium
Normal priority
 Type/Bug
Something isn't working
 Type/Chore
Maintenance tasks
 Type/Documentation
Documentation improvements
 Type/Feature
New feature or request
 Type/Refactor
Code refactoring without changing behavior
 Type/Test
Testing related
No Label
Milestone
No items
No Milestone
Projects
Clear projects
No project
Assignees
Clear assignees
lmiranda
No Assignees
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: personal-projects/personal-portfolio#65
Delete Branch "%!s()"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?