lmiranda/job-forge
0
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Major documentation overhaul: Transform to Python/FastAPI web application

Browse Source 
This comprehensive update transforms Job Forge from a generic MVP concept to a
production-ready Python/FastAPI web application prototype with complete documentation,
testing infrastructure, and deployment procedures.

## 🏗️ Architecture Changes
- Updated all documentation to reflect Python/FastAPI + Dash + PostgreSQL stack
- Transformed from MVP concept to deployable web application prototype
- Added comprehensive multi-tenant architecture with Row Level Security (RLS)
- Integrated Claude API and OpenAI API for AI-powered document generation

## 📚 Documentation Overhaul
- **CLAUDE.md**: Complete rewrite as project orchestrator for 4 specialized agents
- **README.md**: New centralized documentation hub with organized navigation
- **API Specification**: Updated with comprehensive FastAPI endpoint documentation
- **Database Design**: Enhanced schema with RLS policies and performance optimization
- **Architecture Guide**: Transformed to web application focus with deployment strategy

## 🏗️ New Documentation Structure
- **docs/development/**: Python/FastAPI coding standards and development guidelines
- **docs/infrastructure/**: Docker setup and server deployment procedures
- **docs/testing/**: Comprehensive QA procedures with pytest integration
- **docs/ai/**: AI prompt templates and examples (preserved from original)

## 🎯 Team Structure Updates
- **.claude/agents/**: 4 new Python/FastAPI specialized agents
  - simplified_technical_lead.md: Architecture and technical guidance
  - fullstack_developer.md: FastAPI backend + Dash frontend implementation
  - simplified_qa.md: pytest testing and quality assurance
  - simplified_devops.md: Docker deployment and server infrastructure

## 🧪 Testing Infrastructure
- **pytest.ini**: Complete pytest configuration with coverage requirements
- **tests/conftest.py**: Comprehensive test fixtures and database setup
- **tests/unit/**: Example unit tests for auth and application services
- **tests/integration/**: API integration test examples
- Support for async testing, AI service mocking, and database testing

## 🧹 Cleanup
- Removed 9 duplicate/outdated documentation files
- Eliminated conflicting technology references (Node.js/TypeScript)
- Consolidated overlapping content into comprehensive guides
- Cleaned up project structure for professional development workflow

## 🚀 Production Ready Features
- Docker containerization for development and production
- Server deployment procedures for prototype hosting
- Security best practices with JWT authentication and RLS
- Performance optimization with database indexing and caching
- Comprehensive testing strategy with quality gates

This update establishes Job Forge as a professional Python/FastAPI web application
prototype ready for development and deployment.

🤖 Generated with Claude Code (https://claude.ai/code)

Co-Authored-By: Claude <noreply@anthropic.com>
This commit is contained in:
Leo Miranda
2025-08-02 11:33:32 -04:00
parent d9a8b13c16
commit b646e2f5df
41 changed files with 10237 additions and 5499 deletions
Download Patch File Download Diff File Expand all files Collapse all files

601
docs/infrastructure/docker_setup.md Normal file
View File

@@ -0,0 +1,601 @@
# Docker Setup Guide - Job Forge
## Overview
Job Forge uses Docker for containerization to ensure consistent environments across development, testing, and production. This guide covers Docker configuration, best practices, and troubleshooting.
## Docker Architecture
### Container Structure
```
Job Forge Docker Setup
├── jobforge-app # FastAPI + Dash application
├── postgres # PostgreSQL 16 with pgvector
└── nginx # Reverse proxy and SSL termination
```
### Network Configuration
- **Internal Network**: Docker compose creates isolated network
- **External Access**: Only nginx container exposes ports 80/443
- **Service Discovery**: Containers communicate via service names
## Development Setup
### 1. Prerequisites
```bash
# Install Docker and Docker Compose
curl -fsSL https://get.docker.com -o get-docker.sh
sudo sh get-docker.sh
# Install Docker Compose
sudo curl -L "https://github.com/docker/compose/releases/latest/download/docker-compose-$(uname -s)-$(uname -m)" -o /usr/local/bin/docker-compose
sudo chmod +x /usr/local/bin/docker-compose
# Add user to docker group
sudo usermod -aG docker $USER
newgrp docker
```
### 2. Environment Configuration
```bash
# Copy environment template
cp .env.example .env
# Edit for development
nano .env
```
```bash
# Development Environment Variables
DATABASE_URL="postgresql://jobforge:jobforge123@postgres:5432/jobforge"
CLAUDE_API_KEY="your-claude-api-key"
OPENAI_API_KEY="your-openai-api-key"
JWT_SECRET="development-secret-key-change-in-production"
DEBUG=true
LOG_LEVEL="DEBUG"
```
### 3. Docker Compose Configuration
#### docker-compose.yml
```yaml
version: '3.8'
services:
# FastAPI + Dash Application
jobforge-app:
build:
context: .
dockerfile: Dockerfile
container_name: jobforge-app
ports:
- "8000:8000"
environment:
- DATABASE_URL=postgresql://jobforge:jobforge123@postgres:5432/jobforge
- CLAUDE_API_KEY=${CLAUDE_API_KEY}
- OPENAI_API_KEY=${OPENAI_API_KEY}
- JWT_SECRET=${JWT_SECRET}
- DEBUG=${DEBUG:-false}
- LOG_LEVEL=${LOG_LEVEL:-INFO}
depends_on:
postgres:
condition: service_healthy
volumes:
# Development: mount source for hot reload
- ./app:/app/app:ro
- ./uploads:/app/uploads
- ./logs:/var/log/jobforge
networks:
- jobforge-network
restart: unless-stopped
healthcheck:
test: ["CMD", "curl", "-f", "http://localhost:8000/health"]
interval: 30s
timeout: 10s
retries: 3
start_period: 40s
# PostgreSQL with pgvector
postgres:
image: pgvector/pgvector:pg16
container_name: jobforge-postgres
environment:
- POSTGRES_DB=jobforge
- POSTGRES_USER=jobforge
- POSTGRES_PASSWORD=jobforge123
- POSTGRES_INITDB_ARGS="--encoding=UTF8 --lc-collate=en_US.UTF-8 --lc-ctype=en_US.UTF-8"
ports:
- "5432:5432" # Expose for development access
volumes:
- postgres_data:/var/lib/postgresql/data
- ./init_db.sql:/docker-entrypoint-initdb.d/init_db.sql:ro
- ./backups:/backups
networks:
- jobforge-network
restart: unless-stopped
healthcheck:
test: ["CMD-SHELL", "pg_isready -U jobforge -d jobforge"]
interval: 10s
timeout: 5s
retries: 5
start_period: 30s
# Nginx Reverse Proxy
nginx:
image: nginx:alpine
container_name: jobforge-nginx
ports:
- "80:80"
- "443:443"
volumes:
- ./nginx/nginx.conf:/etc/nginx/nginx.conf:ro
- ./nginx/conf.d:/etc/nginx/conf.d:ro
- ./ssl:/etc/nginx/ssl:ro
- ./static:/var/www/static:ro
depends_on:
- jobforge-app
networks:
- jobforge-network
restart: unless-stopped
healthcheck:
test: ["CMD", "nginx", "-t"]
interval: 30s
timeout: 10s
retries: 3
networks:
jobforge-network:
driver: bridge
ipam:
config:
- subnet: 172.20.0.0/16
volumes:
postgres_data:
driver: local
```
#### docker-compose.override.yml (Development)
```yaml
# Development overrides
version: '3.8'
services:
jobforge-app:
build:
target: development # Multi-stage build target
environment:
- DEBUG=true
- LOG_LEVEL=DEBUG
- RELOAD=true
volumes:
# Enable hot reload in development
- ./app:/app/app
- ./tests:/app/tests
command: ["uvicorn", "app.main:app", "--host", "0.0.0.0", "--port", "8000", "--reload"]
postgres:
environment:
- POSTGRES_PASSWORD=jobforge123 # Simpler password for dev
ports:
- "5432:5432" # Expose port for development tools
```
### 4. Dockerfile Configuration
#### Multi-stage Dockerfile
```dockerfile
# Multi-stage Dockerfile for Job Forge
FROM python:3.12-slim as base
# Set environment variables
ENV PYTHONDONTWRITEBYTECODE=1 \
PYTHONUNBUFFERED=1 \
PYTHONPATH="/app" \
PIP_NO_CACHE_DIR=1 \
PIP_DISABLE_PIP_VERSION_CHECK=1
# Install system dependencies
RUN apt-get update && apt-get install -y \
curl \
postgresql-client \
build-essential \
&& rm -rf /var/lib/apt/lists/*
WORKDIR /app
# Copy requirements and install Python dependencies
COPY requirements.txt .
RUN pip install --no-cache-dir -r requirements.txt
# Development target
FROM base as development
# Install development dependencies
COPY requirements-dev.txt .
RUN pip install --no-cache-dir -r requirements-dev.txt
# Copy source code
COPY . .
# Create non-root user
RUN adduser --disabled-password --gecos '' jobforge && \
chown -R jobforge:jobforge /app
USER jobforge
# Health check
HEALTHCHECK --interval=30s --timeout=30s --start-period=5s --retries=3 \
CMD curl -f http://localhost:8000/health || exit 1
EXPOSE 8000
# Development command with hot reload
CMD ["uvicorn", "app.main:app", "--host", "0.0.0.0", "--port", "8000", "--reload"]
# Production target
FROM base as production
# Copy source code
COPY app/ ./app/
COPY alembic/ ./alembic/
COPY alembic.ini .
# Create non-root user
RUN adduser --disabled-password --gecos '' jobforge && \
chown -R jobforge:jobforge /app && \
mkdir -p /var/log/jobforge && \
chown jobforge:jobforge /var/log/jobforge
USER jobforge
# Health check
HEALTHCHECK --interval=30s --timeout=30s --start-period=5s --retries=3 \
CMD curl -f http://localhost:8000/health || exit 1
EXPOSE 8000
# Production command
CMD ["uvicorn", "app.main:app", "--host", "0.0.0.0", "--port", "8000", "--workers", "2"]
```
#### .dockerignore
```
# .dockerignore for Job Forge
__pycache__/
*.pyc
*.pyo
*.pyd
.Python
env/
venv/
.venv/
pip-log.txt
pip-delete-this-directory.txt
.git/
.gitignore
README.md
.env
.env.*
docker-compose*.yml
Dockerfile*
.pytest_cache/
htmlcov/
.coverage
*.log
logs/
backups/
uploads/
!uploads/.gitkeep
.vscode/
.idea/
*.swp
*.swo
*~
```
## Production Setup
### 1. Production Docker Compose
```yaml
# docker-compose.prod.yml
version: '3.8'
services:
jobforge-app:
build:
context: .
dockerfile: Dockerfile
target: production
environment:
- DATABASE_URL=${DATABASE_URL}
- CLAUDE_API_KEY=${CLAUDE_API_KEY}
- OPENAI_API_KEY=${OPENAI_API_KEY}
- JWT_SECRET=${JWT_SECRET}
- DEBUG=false
- LOG_LEVEL=INFO
- WORKERS=4
volumes:
- ./uploads:/app/uploads
- ./logs:/var/log/jobforge
deploy:
resources:
limits:
cpus: '2.0'
memory: 2G
reservations:
cpus: '0.5'
memory: 512M
restart_policy:
condition: on-failure
delay: 5s
max_attempts: 3
postgres:
image: pgvector/pgvector:pg16
environment:
- POSTGRES_DB=${DB_NAME}
- POSTGRES_USER=${DB_USER}
- POSTGRES_PASSWORD=${DB_PASSWORD}
volumes:
- postgres_data:/var/lib/postgresql/data
- ./backups:/backups:ro
deploy:
resources:
limits:
cpus: '1.0'
memory: 1G
reservations:
cpus: '0.25'
memory: 256M
# Don't expose port in production
# ports:
# - "5432:5432"
nginx:
image: nginx:alpine
ports:
- "80:80"
- "443:443"
volumes:
- ./nginx/prod.conf:/etc/nginx/nginx.conf:ro
- ./ssl:/etc/nginx/ssl:ro
deploy:
resources:
limits:
cpus: '0.5'
memory: 128M
```
### 2. Production Commands
```bash
# Build production images
docker-compose -f docker-compose.prod.yml build
# Start production services
docker-compose -f docker-compose.prod.yml up -d
# Scale application containers
docker-compose -f docker-compose.prod.yml up -d --scale jobforge-app=3
```
## Container Management
### 1. Basic Operations
```bash
# Start all services
docker-compose up -d
# Stop all services
docker-compose down
# Restart specific service
docker-compose restart jobforge-app
# View logs
docker-compose logs -f jobforge-app
docker-compose logs --tail=100 postgres
# Execute commands in container
docker-compose exec jobforge-app bash
docker-compose exec postgres psql -U jobforge -d jobforge
```
### 2. Database Operations
```bash
# Run database migrations
docker-compose exec jobforge-app alembic upgrade head
# Create new migration
docker-compose exec jobforge-app alembic revision --autogenerate -m "description"
# Database backup
docker-compose exec postgres pg_dump -U jobforge jobforge > backup.sql
# Database restore
docker-compose exec -T postgres psql -U jobforge -d jobforge < backup.sql
```
### 3. Application Management
```bash
# Update application code (development)
docker-compose restart jobforge-app
# Rebuild containers
docker-compose build
docker-compose up -d
# View container resource usage
docker stats
# Clean up unused resources
docker system prune -a -f
```
## Monitoring and Debugging
### 1. Health Checks
```bash
# Check container health
docker-compose ps
# Manual health check
curl http://localhost:8000/health
# Check individual services
docker-compose exec jobforge-app python -c "import requests; print(requests.get('http://localhost:8000/health').json())"
```
### 2. Log Management
```bash
# View application logs
docker-compose logs -f jobforge-app
# View specific timeframe
docker-compose logs --since="2024-01-01T00:00:00" jobforge-app
# Export logs
docker-compose logs jobforge-app > app.log
# Follow logs with timestamps
docker-compose logs -f -t jobforge-app
```
### 3. Performance Monitoring
```bash
# Container resource usage
docker stats --format "table {{.Container}}\t{{.CPUPerc}}\t{{.MemUsage}}\t{{.NetIO}}\t{{.BlockIO}}"
# Container processes
docker-compose exec jobforge-app ps aux
# Database performance
docker-compose exec postgres psql -U jobforge -d jobforge -c "SELECT * FROM pg_stat_activity;"
```
## Troubleshooting
### 1. Common Issues
#### Container Won't Start
```bash
# Check logs for errors
docker-compose logs jobforge-app
# Check if ports are in use
sudo netstat -tulpn | grep :8000
# Check Docker daemon
sudo systemctl status docker
```
#### Database Connection Issues
```bash
# Test database connectivity
docker-compose exec jobforge-app python -c "
import asyncio
import asyncpg
asyncio.run(asyncpg.connect('postgresql://jobforge:jobforge123@postgres:5432/jobforge'))
print('Database connection successful')
"
# Check database is ready
docker-compose exec postgres pg_isready -U jobforge
```
#### Volume Mount Issues
```bash
# Check volume permissions
ls -la uploads/ logs/
# Fix permissions
sudo chown -R $USER:$USER uploads logs
chmod 755 uploads logs
```
### 2. Debug Mode
```bash
# Run container in debug mode
docker-compose run --rm jobforge-app bash
# Run with environment override
docker-compose run -e DEBUG=true jobforge-app
# Attach to running container
docker-compose exec jobforge-app bash
```
### 3. Network Issues
```bash
# Check network connectivity
docker network ls
docker network inspect jobforge_jobforge-network
# Test inter-container communication
docker-compose exec jobforge-app ping postgres
docker-compose exec nginx ping jobforge-app
```
## Optimization
### 1. Image Optimization
```dockerfile
# Use multi-stage builds to reduce image size
# Use .dockerignore to exclude unnecessary files
# Use specific base image versions
# Combine RUN commands to reduce layers
# Clean up package caches
```
### 2. Resource Limits
```yaml
# Set appropriate resource limits
deploy:
resources:
limits:
cpus: '1.0'
memory: 1G
reservations:
cpus: '0.5'
memory: 512M
```
### 3. Volume Optimization
```bash
# Use bind mounts for development
# Use named volumes for persistent data
# Regular cleanup of unused volumes
docker volume prune
```
## Security Best Practices
### 1. Container Security
```dockerfile
# Run as non-root user
USER jobforge
# Use specific image versions
FROM python:3.12-slim
# Don't expose unnecessary ports
# Use secrets for sensitive data
```
### 2. Network Security
```yaml
# Use custom networks
networks:
jobforge-network:
driver: bridge
internal: true # For internal services
```
### 3. Environment Security
```bash
# Use .env files for secrets
# Don't commit secrets to version control
# Use Docker secrets in production
# Regular security updates
```
This Docker setup provides a robust, scalable, and maintainable containerization solution for Job Forge, suitable for both development and production environments.