feat: Sprint 10 - Architecture docs, CI/CD, operational scripts
Some checks failed
CI / lint-and-test (push) Has been cancelled
Some checks failed
CI / lint-and-test (push) Has been cancelled
Phase 1 - Architecture Documentation: - Add Architecture section with Mermaid flowchart to README - Create docs/DATABASE_SCHEMA.md with full ERD Phase 2 - CI/CD: - Add CI badge to README - Create .gitea/workflows/ci.yml for linting and tests - Create .gitea/workflows/deploy-staging.yml - Create .gitea/workflows/deploy-production.yml Phase 3 - Operational Scripts: - Create scripts/logs.sh for docker compose log following - Create scripts/run-detached.sh with health check loop - Create scripts/etl/toronto.sh for Toronto data pipeline - Add Makefile targets: logs, run-detached, etl-toronto Phase 4 - Runbooks: - Create docs/runbooks/adding-dashboard.md - Create docs/runbooks/deployment.md Phase 5 - Hygiene: - Create MIT LICENSE file Phase 6 - Production: - Add live demo link to README (leodata.science) Closes #78, #79, #80, #81, #82, #83, #84, #85, #86, #87, #88, #89, #91 Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
This commit is contained in:
232
docs/runbooks/deployment.md
Normal file
232
docs/runbooks/deployment.md
Normal file
@@ -0,0 +1,232 @@
|
||||
# Runbook: Deployment
|
||||
|
||||
This runbook covers deployment procedures for the Analytics Portfolio application.
|
||||
|
||||
## Environments
|
||||
|
||||
| Environment | Branch | Server | URL |
|
||||
|-------------|--------|--------|-----|
|
||||
| Development | `development` | Local | http://localhost:8050 |
|
||||
| Staging | `staging` | Homelab (hotserv) | Internal |
|
||||
| Production | `main` | Bandit Labs VPS | https://leodata.science |
|
||||
|
||||
## CI/CD Pipeline
|
||||
|
||||
### Automatic Deployment
|
||||
|
||||
Deployments are triggered automatically via Gitea Actions:
|
||||
|
||||
1. **Push to `staging`** → Deploys to staging server
|
||||
2. **Push to `main`** → Deploys to production server
|
||||
|
||||
### Workflow Files
|
||||
|
||||
- `.gitea/workflows/ci.yml` - Runs linting and tests on all branches
|
||||
- `.gitea/workflows/deploy-staging.yml` - Staging deployment
|
||||
- `.gitea/workflows/deploy-production.yml` - Production deployment
|
||||
|
||||
### Required Secrets
|
||||
|
||||
Configure these in Gitea repository settings:
|
||||
|
||||
| Secret | Description |
|
||||
|--------|-------------|
|
||||
| `STAGING_HOST` | Staging server hostname/IP |
|
||||
| `STAGING_USER` | SSH username for staging |
|
||||
| `STAGING_SSH_KEY` | Private key for staging SSH |
|
||||
| `PROD_HOST` | Production server hostname/IP |
|
||||
| `PROD_USER` | SSH username for production |
|
||||
| `PROD_SSH_KEY` | Private key for production SSH |
|
||||
|
||||
## Manual Deployment
|
||||
|
||||
### Prerequisites
|
||||
|
||||
- SSH access to target server
|
||||
- Repository cloned at `~/apps/personal-portfolio`
|
||||
- Virtual environment created at `.venv`
|
||||
- Docker and Docker Compose installed
|
||||
- PostgreSQL container running
|
||||
|
||||
### Steps
|
||||
|
||||
```bash
|
||||
# 1. SSH to server
|
||||
ssh user@server
|
||||
|
||||
# 2. Navigate to app directory
|
||||
cd ~/apps/personal-portfolio
|
||||
|
||||
# 3. Pull latest changes
|
||||
git fetch origin {branch}
|
||||
git reset --hard origin/{branch}
|
||||
|
||||
# 4. Activate virtual environment
|
||||
source .venv/bin/activate
|
||||
|
||||
# 5. Install dependencies
|
||||
pip install -r requirements.txt
|
||||
|
||||
# 6. Run database migrations (if any)
|
||||
# python -m alembic upgrade head
|
||||
|
||||
# 7. Run dbt models
|
||||
cd dbt && dbt run --profiles-dir . && cd ..
|
||||
|
||||
# 8. Restart application
|
||||
docker compose down
|
||||
docker compose up -d
|
||||
|
||||
# 9. Verify health
|
||||
curl http://localhost:8050/health
|
||||
```
|
||||
|
||||
## Rollback Procedure
|
||||
|
||||
### Quick Rollback
|
||||
|
||||
If deployment fails, rollback to previous commit:
|
||||
|
||||
```bash
|
||||
# 1. Find previous working commit
|
||||
git log --oneline -10
|
||||
|
||||
# 2. Reset to that commit
|
||||
git reset --hard {commit_hash}
|
||||
|
||||
# 3. Restart services
|
||||
docker compose down
|
||||
docker compose up -d
|
||||
|
||||
# 4. Verify
|
||||
curl http://localhost:8050/health
|
||||
```
|
||||
|
||||
### Full Rollback (Database)
|
||||
|
||||
If database changes need to be reverted:
|
||||
|
||||
```bash
|
||||
# 1. Stop application
|
||||
docker compose down
|
||||
|
||||
# 2. Restore database backup
|
||||
pg_restore -h localhost -U portfolio -d portfolio backup.dump
|
||||
|
||||
# 3. Revert code
|
||||
git reset --hard {commit_hash}
|
||||
|
||||
# 4. Run dbt at that version
|
||||
cd dbt && dbt run --profiles-dir . && cd ..
|
||||
|
||||
# 5. Restart
|
||||
docker compose up -d
|
||||
```
|
||||
|
||||
## Health Checks
|
||||
|
||||
### Application Health
|
||||
|
||||
```bash
|
||||
curl http://localhost:8050/health
|
||||
```
|
||||
|
||||
Expected response:
|
||||
```json
|
||||
{"status": "healthy"}
|
||||
```
|
||||
|
||||
### Database Health
|
||||
|
||||
```bash
|
||||
docker compose exec postgres pg_isready -U portfolio
|
||||
```
|
||||
|
||||
### Container Status
|
||||
|
||||
```bash
|
||||
docker compose ps
|
||||
```
|
||||
|
||||
## Monitoring
|
||||
|
||||
### View Logs
|
||||
|
||||
```bash
|
||||
# All services
|
||||
make logs
|
||||
|
||||
# Specific service
|
||||
make logs SERVICE=postgres
|
||||
|
||||
# Or directly
|
||||
docker compose logs -f
|
||||
```
|
||||
|
||||
### Check Resource Usage
|
||||
|
||||
```bash
|
||||
docker stats
|
||||
```
|
||||
|
||||
## Troubleshooting
|
||||
|
||||
### Application Won't Start
|
||||
|
||||
1. Check container logs: `docker compose logs app`
|
||||
2. Verify environment variables: `cat .env`
|
||||
3. Check database connectivity: `docker compose exec postgres pg_isready`
|
||||
4. Verify port availability: `lsof -i :8050`
|
||||
|
||||
### Database Connection Errors
|
||||
|
||||
1. Check postgres container: `docker compose ps postgres`
|
||||
2. Verify DATABASE_URL in `.env`
|
||||
3. Check postgres logs: `docker compose logs postgres`
|
||||
4. Test connection: `docker compose exec postgres psql -U portfolio -c '\l'`
|
||||
|
||||
### dbt Failures
|
||||
|
||||
1. Check dbt logs: `cd dbt && dbt debug`
|
||||
2. Verify profiles.yml: `cat dbt/profiles.yml`
|
||||
3. Run with verbose output: `dbt run --debug`
|
||||
|
||||
### Out of Memory
|
||||
|
||||
1. Check memory usage: `free -h`
|
||||
2. Review container limits in docker-compose.yml
|
||||
3. Consider increasing swap or server resources
|
||||
|
||||
## Backup Procedures
|
||||
|
||||
### Database Backup
|
||||
|
||||
```bash
|
||||
# Create backup
|
||||
docker compose exec postgres pg_dump -U portfolio portfolio > backup_$(date +%Y%m%d).sql
|
||||
|
||||
# Compressed backup
|
||||
docker compose exec postgres pg_dump -U portfolio -Fc portfolio > backup_$(date +%Y%m%d).dump
|
||||
```
|
||||
|
||||
### Restore from Backup
|
||||
|
||||
```bash
|
||||
# From SQL file
|
||||
docker compose exec -T postgres psql -U portfolio portfolio < backup.sql
|
||||
|
||||
# From dump file
|
||||
docker compose exec -T postgres pg_restore -U portfolio -d portfolio < backup.dump
|
||||
```
|
||||
|
||||
## Deployment Checklist
|
||||
|
||||
Before deploying to production:
|
||||
|
||||
- [ ] All tests pass (`make test`)
|
||||
- [ ] Linting passes (`make lint`)
|
||||
- [ ] Staging deployment successful
|
||||
- [ ] Manual testing on staging complete
|
||||
- [ ] Database backup taken
|
||||
- [ ] Rollback plan confirmed
|
||||
- [ ] Team notified of deployment window
|
||||
Reference in New Issue
Block a user