Major documentation overhaul: Transform to Python/FastAPI web application
This comprehensive update transforms Job Forge from a generic MVP concept to a production-ready Python/FastAPI web application prototype with complete documentation, testing infrastructure, and deployment procedures. ## 🏗️ Architecture Changes - Updated all documentation to reflect Python/FastAPI + Dash + PostgreSQL stack - Transformed from MVP concept to deployable web application prototype - Added comprehensive multi-tenant architecture with Row Level Security (RLS) - Integrated Claude API and OpenAI API for AI-powered document generation ## 📚 Documentation Overhaul - **CLAUDE.md**: Complete rewrite as project orchestrator for 4 specialized agents - **README.md**: New centralized documentation hub with organized navigation - **API Specification**: Updated with comprehensive FastAPI endpoint documentation - **Database Design**: Enhanced schema with RLS policies and performance optimization - **Architecture Guide**: Transformed to web application focus with deployment strategy ## 🏗️ New Documentation Structure - **docs/development/**: Python/FastAPI coding standards and development guidelines - **docs/infrastructure/**: Docker setup and server deployment procedures - **docs/testing/**: Comprehensive QA procedures with pytest integration - **docs/ai/**: AI prompt templates and examples (preserved from original) ## 🎯 Team Structure Updates - **.claude/agents/**: 4 new Python/FastAPI specialized agents - simplified_technical_lead.md: Architecture and technical guidance - fullstack_developer.md: FastAPI backend + Dash frontend implementation - simplified_qa.md: pytest testing and quality assurance - simplified_devops.md: Docker deployment and server infrastructure ## 🧪 Testing Infrastructure - **pytest.ini**: Complete pytest configuration with coverage requirements - **tests/conftest.py**: Comprehensive test fixtures and database setup - **tests/unit/**: Example unit tests for auth and application services - **tests/integration/**: API integration test examples - Support for async testing, AI service mocking, and database testing ## 🧹 Cleanup - Removed 9 duplicate/outdated documentation files - Eliminated conflicting technology references (Node.js/TypeScript) - Consolidated overlapping content into comprehensive guides - Cleaned up project structure for professional development workflow ## 🚀 Production Ready Features - Docker containerization for development and production - Server deployment procedures for prototype hosting - Security best practices with JWT authentication and RLS - Performance optimization with database indexing and caching - Comprehensive testing strategy with quality gates This update establishes Job Forge as a professional Python/FastAPI web application prototype ready for development and deployment. 🤖 Generated with Claude Code (https://claude.ai/code) Co-Authored-By: Claude <noreply@anthropic.com>
This commit is contained in:
601
docs/infrastructure/docker_setup.md
Normal file
601
docs/infrastructure/docker_setup.md
Normal file
@@ -0,0 +1,601 @@
|
||||
# Docker Setup Guide - Job Forge
|
||||
|
||||
## Overview
|
||||
|
||||
Job Forge uses Docker for containerization to ensure consistent environments across development, testing, and production. This guide covers Docker configuration, best practices, and troubleshooting.
|
||||
|
||||
## Docker Architecture
|
||||
|
||||
### Container Structure
|
||||
```
|
||||
Job Forge Docker Setup
|
||||
├── jobforge-app # FastAPI + Dash application
|
||||
├── postgres # PostgreSQL 16 with pgvector
|
||||
└── nginx # Reverse proxy and SSL termination
|
||||
```
|
||||
|
||||
### Network Configuration
|
||||
- **Internal Network**: Docker compose creates isolated network
|
||||
- **External Access**: Only nginx container exposes ports 80/443
|
||||
- **Service Discovery**: Containers communicate via service names
|
||||
|
||||
## Development Setup
|
||||
|
||||
### 1. Prerequisites
|
||||
```bash
|
||||
# Install Docker and Docker Compose
|
||||
curl -fsSL https://get.docker.com -o get-docker.sh
|
||||
sudo sh get-docker.sh
|
||||
|
||||
# Install Docker Compose
|
||||
sudo curl -L "https://github.com/docker/compose/releases/latest/download/docker-compose-$(uname -s)-$(uname -m)" -o /usr/local/bin/docker-compose
|
||||
sudo chmod +x /usr/local/bin/docker-compose
|
||||
|
||||
# Add user to docker group
|
||||
sudo usermod -aG docker $USER
|
||||
newgrp docker
|
||||
```
|
||||
|
||||
### 2. Environment Configuration
|
||||
```bash
|
||||
# Copy environment template
|
||||
cp .env.example .env
|
||||
|
||||
# Edit for development
|
||||
nano .env
|
||||
```
|
||||
|
||||
```bash
|
||||
# Development Environment Variables
|
||||
DATABASE_URL="postgresql://jobforge:jobforge123@postgres:5432/jobforge"
|
||||
CLAUDE_API_KEY="your-claude-api-key"
|
||||
OPENAI_API_KEY="your-openai-api-key"
|
||||
JWT_SECRET="development-secret-key-change-in-production"
|
||||
DEBUG=true
|
||||
LOG_LEVEL="DEBUG"
|
||||
```
|
||||
|
||||
### 3. Docker Compose Configuration
|
||||
|
||||
#### docker-compose.yml
|
||||
```yaml
|
||||
version: '3.8'
|
||||
|
||||
services:
|
||||
# FastAPI + Dash Application
|
||||
jobforge-app:
|
||||
build:
|
||||
context: .
|
||||
dockerfile: Dockerfile
|
||||
container_name: jobforge-app
|
||||
ports:
|
||||
- "8000:8000"
|
||||
environment:
|
||||
- DATABASE_URL=postgresql://jobforge:jobforge123@postgres:5432/jobforge
|
||||
- CLAUDE_API_KEY=${CLAUDE_API_KEY}
|
||||
- OPENAI_API_KEY=${OPENAI_API_KEY}
|
||||
- JWT_SECRET=${JWT_SECRET}
|
||||
- DEBUG=${DEBUG:-false}
|
||||
- LOG_LEVEL=${LOG_LEVEL:-INFO}
|
||||
depends_on:
|
||||
postgres:
|
||||
condition: service_healthy
|
||||
volumes:
|
||||
# Development: mount source for hot reload
|
||||
- ./app:/app/app:ro
|
||||
- ./uploads:/app/uploads
|
||||
- ./logs:/var/log/jobforge
|
||||
networks:
|
||||
- jobforge-network
|
||||
restart: unless-stopped
|
||||
healthcheck:
|
||||
test: ["CMD", "curl", "-f", "http://localhost:8000/health"]
|
||||
interval: 30s
|
||||
timeout: 10s
|
||||
retries: 3
|
||||
start_period: 40s
|
||||
|
||||
# PostgreSQL with pgvector
|
||||
postgres:
|
||||
image: pgvector/pgvector:pg16
|
||||
container_name: jobforge-postgres
|
||||
environment:
|
||||
- POSTGRES_DB=jobforge
|
||||
- POSTGRES_USER=jobforge
|
||||
- POSTGRES_PASSWORD=jobforge123
|
||||
- POSTGRES_INITDB_ARGS="--encoding=UTF8 --lc-collate=en_US.UTF-8 --lc-ctype=en_US.UTF-8"
|
||||
ports:
|
||||
- "5432:5432" # Expose for development access
|
||||
volumes:
|
||||
- postgres_data:/var/lib/postgresql/data
|
||||
- ./init_db.sql:/docker-entrypoint-initdb.d/init_db.sql:ro
|
||||
- ./backups:/backups
|
||||
networks:
|
||||
- jobforge-network
|
||||
restart: unless-stopped
|
||||
healthcheck:
|
||||
test: ["CMD-SHELL", "pg_isready -U jobforge -d jobforge"]
|
||||
interval: 10s
|
||||
timeout: 5s
|
||||
retries: 5
|
||||
start_period: 30s
|
||||
|
||||
# Nginx Reverse Proxy
|
||||
nginx:
|
||||
image: nginx:alpine
|
||||
container_name: jobforge-nginx
|
||||
ports:
|
||||
- "80:80"
|
||||
- "443:443"
|
||||
volumes:
|
||||
- ./nginx/nginx.conf:/etc/nginx/nginx.conf:ro
|
||||
- ./nginx/conf.d:/etc/nginx/conf.d:ro
|
||||
- ./ssl:/etc/nginx/ssl:ro
|
||||
- ./static:/var/www/static:ro
|
||||
depends_on:
|
||||
- jobforge-app
|
||||
networks:
|
||||
- jobforge-network
|
||||
restart: unless-stopped
|
||||
healthcheck:
|
||||
test: ["CMD", "nginx", "-t"]
|
||||
interval: 30s
|
||||
timeout: 10s
|
||||
retries: 3
|
||||
|
||||
networks:
|
||||
jobforge-network:
|
||||
driver: bridge
|
||||
ipam:
|
||||
config:
|
||||
- subnet: 172.20.0.0/16
|
||||
|
||||
volumes:
|
||||
postgres_data:
|
||||
driver: local
|
||||
```
|
||||
|
||||
#### docker-compose.override.yml (Development)
|
||||
```yaml
|
||||
# Development overrides
|
||||
version: '3.8'
|
||||
|
||||
services:
|
||||
jobforge-app:
|
||||
build:
|
||||
target: development # Multi-stage build target
|
||||
environment:
|
||||
- DEBUG=true
|
||||
- LOG_LEVEL=DEBUG
|
||||
- RELOAD=true
|
||||
volumes:
|
||||
# Enable hot reload in development
|
||||
- ./app:/app/app
|
||||
- ./tests:/app/tests
|
||||
command: ["uvicorn", "app.main:app", "--host", "0.0.0.0", "--port", "8000", "--reload"]
|
||||
|
||||
postgres:
|
||||
environment:
|
||||
- POSTGRES_PASSWORD=jobforge123 # Simpler password for dev
|
||||
ports:
|
||||
- "5432:5432" # Expose port for development tools
|
||||
```
|
||||
|
||||
### 4. Dockerfile Configuration
|
||||
|
||||
#### Multi-stage Dockerfile
|
||||
```dockerfile
|
||||
# Multi-stage Dockerfile for Job Forge
|
||||
FROM python:3.12-slim as base
|
||||
|
||||
# Set environment variables
|
||||
ENV PYTHONDONTWRITEBYTECODE=1 \
|
||||
PYTHONUNBUFFERED=1 \
|
||||
PYTHONPATH="/app" \
|
||||
PIP_NO_CACHE_DIR=1 \
|
||||
PIP_DISABLE_PIP_VERSION_CHECK=1
|
||||
|
||||
# Install system dependencies
|
||||
RUN apt-get update && apt-get install -y \
|
||||
curl \
|
||||
postgresql-client \
|
||||
build-essential \
|
||||
&& rm -rf /var/lib/apt/lists/*
|
||||
|
||||
WORKDIR /app
|
||||
|
||||
# Copy requirements and install Python dependencies
|
||||
COPY requirements.txt .
|
||||
RUN pip install --no-cache-dir -r requirements.txt
|
||||
|
||||
# Development target
|
||||
FROM base as development
|
||||
|
||||
# Install development dependencies
|
||||
COPY requirements-dev.txt .
|
||||
RUN pip install --no-cache-dir -r requirements-dev.txt
|
||||
|
||||
# Copy source code
|
||||
COPY . .
|
||||
|
||||
# Create non-root user
|
||||
RUN adduser --disabled-password --gecos '' jobforge && \
|
||||
chown -R jobforge:jobforge /app
|
||||
USER jobforge
|
||||
|
||||
# Health check
|
||||
HEALTHCHECK --interval=30s --timeout=30s --start-period=5s --retries=3 \
|
||||
CMD curl -f http://localhost:8000/health || exit 1
|
||||
|
||||
EXPOSE 8000
|
||||
|
||||
# Development command with hot reload
|
||||
CMD ["uvicorn", "app.main:app", "--host", "0.0.0.0", "--port", "8000", "--reload"]
|
||||
|
||||
# Production target
|
||||
FROM base as production
|
||||
|
||||
# Copy source code
|
||||
COPY app/ ./app/
|
||||
COPY alembic/ ./alembic/
|
||||
COPY alembic.ini .
|
||||
|
||||
# Create non-root user
|
||||
RUN adduser --disabled-password --gecos '' jobforge && \
|
||||
chown -R jobforge:jobforge /app && \
|
||||
mkdir -p /var/log/jobforge && \
|
||||
chown jobforge:jobforge /var/log/jobforge
|
||||
|
||||
USER jobforge
|
||||
|
||||
# Health check
|
||||
HEALTHCHECK --interval=30s --timeout=30s --start-period=5s --retries=3 \
|
||||
CMD curl -f http://localhost:8000/health || exit 1
|
||||
|
||||
EXPOSE 8000
|
||||
|
||||
# Production command
|
||||
CMD ["uvicorn", "app.main:app", "--host", "0.0.0.0", "--port", "8000", "--workers", "2"]
|
||||
```
|
||||
|
||||
#### .dockerignore
|
||||
```
|
||||
# .dockerignore for Job Forge
|
||||
__pycache__/
|
||||
*.pyc
|
||||
*.pyo
|
||||
*.pyd
|
||||
.Python
|
||||
env/
|
||||
venv/
|
||||
.venv/
|
||||
pip-log.txt
|
||||
pip-delete-this-directory.txt
|
||||
.git/
|
||||
.gitignore
|
||||
README.md
|
||||
.env
|
||||
.env.*
|
||||
docker-compose*.yml
|
||||
Dockerfile*
|
||||
.pytest_cache/
|
||||
htmlcov/
|
||||
.coverage
|
||||
*.log
|
||||
logs/
|
||||
backups/
|
||||
uploads/
|
||||
!uploads/.gitkeep
|
||||
.vscode/
|
||||
.idea/
|
||||
*.swp
|
||||
*.swo
|
||||
*~
|
||||
```
|
||||
|
||||
## Production Setup
|
||||
|
||||
### 1. Production Docker Compose
|
||||
```yaml
|
||||
# docker-compose.prod.yml
|
||||
version: '3.8'
|
||||
|
||||
services:
|
||||
jobforge-app:
|
||||
build:
|
||||
context: .
|
||||
dockerfile: Dockerfile
|
||||
target: production
|
||||
environment:
|
||||
- DATABASE_URL=${DATABASE_URL}
|
||||
- CLAUDE_API_KEY=${CLAUDE_API_KEY}
|
||||
- OPENAI_API_KEY=${OPENAI_API_KEY}
|
||||
- JWT_SECRET=${JWT_SECRET}
|
||||
- DEBUG=false
|
||||
- LOG_LEVEL=INFO
|
||||
- WORKERS=4
|
||||
volumes:
|
||||
- ./uploads:/app/uploads
|
||||
- ./logs:/var/log/jobforge
|
||||
deploy:
|
||||
resources:
|
||||
limits:
|
||||
cpus: '2.0'
|
||||
memory: 2G
|
||||
reservations:
|
||||
cpus: '0.5'
|
||||
memory: 512M
|
||||
restart_policy:
|
||||
condition: on-failure
|
||||
delay: 5s
|
||||
max_attempts: 3
|
||||
|
||||
postgres:
|
||||
image: pgvector/pgvector:pg16
|
||||
environment:
|
||||
- POSTGRES_DB=${DB_NAME}
|
||||
- POSTGRES_USER=${DB_USER}
|
||||
- POSTGRES_PASSWORD=${DB_PASSWORD}
|
||||
volumes:
|
||||
- postgres_data:/var/lib/postgresql/data
|
||||
- ./backups:/backups:ro
|
||||
deploy:
|
||||
resources:
|
||||
limits:
|
||||
cpus: '1.0'
|
||||
memory: 1G
|
||||
reservations:
|
||||
cpus: '0.25'
|
||||
memory: 256M
|
||||
# Don't expose port in production
|
||||
# ports:
|
||||
# - "5432:5432"
|
||||
|
||||
nginx:
|
||||
image: nginx:alpine
|
||||
ports:
|
||||
- "80:80"
|
||||
- "443:443"
|
||||
volumes:
|
||||
- ./nginx/prod.conf:/etc/nginx/nginx.conf:ro
|
||||
- ./ssl:/etc/nginx/ssl:ro
|
||||
deploy:
|
||||
resources:
|
||||
limits:
|
||||
cpus: '0.5'
|
||||
memory: 128M
|
||||
```
|
||||
|
||||
### 2. Production Commands
|
||||
```bash
|
||||
# Build production images
|
||||
docker-compose -f docker-compose.prod.yml build
|
||||
|
||||
# Start production services
|
||||
docker-compose -f docker-compose.prod.yml up -d
|
||||
|
||||
# Scale application containers
|
||||
docker-compose -f docker-compose.prod.yml up -d --scale jobforge-app=3
|
||||
```
|
||||
|
||||
## Container Management
|
||||
|
||||
### 1. Basic Operations
|
||||
```bash
|
||||
# Start all services
|
||||
docker-compose up -d
|
||||
|
||||
# Stop all services
|
||||
docker-compose down
|
||||
|
||||
# Restart specific service
|
||||
docker-compose restart jobforge-app
|
||||
|
||||
# View logs
|
||||
docker-compose logs -f jobforge-app
|
||||
docker-compose logs --tail=100 postgres
|
||||
|
||||
# Execute commands in container
|
||||
docker-compose exec jobforge-app bash
|
||||
docker-compose exec postgres psql -U jobforge -d jobforge
|
||||
```
|
||||
|
||||
### 2. Database Operations
|
||||
```bash
|
||||
# Run database migrations
|
||||
docker-compose exec jobforge-app alembic upgrade head
|
||||
|
||||
# Create new migration
|
||||
docker-compose exec jobforge-app alembic revision --autogenerate -m "description"
|
||||
|
||||
# Database backup
|
||||
docker-compose exec postgres pg_dump -U jobforge jobforge > backup.sql
|
||||
|
||||
# Database restore
|
||||
docker-compose exec -T postgres psql -U jobforge -d jobforge < backup.sql
|
||||
```
|
||||
|
||||
### 3. Application Management
|
||||
```bash
|
||||
# Update application code (development)
|
||||
docker-compose restart jobforge-app
|
||||
|
||||
# Rebuild containers
|
||||
docker-compose build
|
||||
docker-compose up -d
|
||||
|
||||
# View container resource usage
|
||||
docker stats
|
||||
|
||||
# Clean up unused resources
|
||||
docker system prune -a -f
|
||||
```
|
||||
|
||||
## Monitoring and Debugging
|
||||
|
||||
### 1. Health Checks
|
||||
```bash
|
||||
# Check container health
|
||||
docker-compose ps
|
||||
|
||||
# Manual health check
|
||||
curl http://localhost:8000/health
|
||||
|
||||
# Check individual services
|
||||
docker-compose exec jobforge-app python -c "import requests; print(requests.get('http://localhost:8000/health').json())"
|
||||
```
|
||||
|
||||
### 2. Log Management
|
||||
```bash
|
||||
# View application logs
|
||||
docker-compose logs -f jobforge-app
|
||||
|
||||
# View specific timeframe
|
||||
docker-compose logs --since="2024-01-01T00:00:00" jobforge-app
|
||||
|
||||
# Export logs
|
||||
docker-compose logs jobforge-app > app.log
|
||||
|
||||
# Follow logs with timestamps
|
||||
docker-compose logs -f -t jobforge-app
|
||||
```
|
||||
|
||||
### 3. Performance Monitoring
|
||||
```bash
|
||||
# Container resource usage
|
||||
docker stats --format "table {{.Container}}\t{{.CPUPerc}}\t{{.MemUsage}}\t{{.NetIO}}\t{{.BlockIO}}"
|
||||
|
||||
# Container processes
|
||||
docker-compose exec jobforge-app ps aux
|
||||
|
||||
# Database performance
|
||||
docker-compose exec postgres psql -U jobforge -d jobforge -c "SELECT * FROM pg_stat_activity;"
|
||||
```
|
||||
|
||||
## Troubleshooting
|
||||
|
||||
### 1. Common Issues
|
||||
|
||||
#### Container Won't Start
|
||||
```bash
|
||||
# Check logs for errors
|
||||
docker-compose logs jobforge-app
|
||||
|
||||
# Check if ports are in use
|
||||
sudo netstat -tulpn | grep :8000
|
||||
|
||||
# Check Docker daemon
|
||||
sudo systemctl status docker
|
||||
```
|
||||
|
||||
#### Database Connection Issues
|
||||
```bash
|
||||
# Test database connectivity
|
||||
docker-compose exec jobforge-app python -c "
|
||||
import asyncio
|
||||
import asyncpg
|
||||
asyncio.run(asyncpg.connect('postgresql://jobforge:jobforge123@postgres:5432/jobforge'))
|
||||
print('Database connection successful')
|
||||
"
|
||||
|
||||
# Check database is ready
|
||||
docker-compose exec postgres pg_isready -U jobforge
|
||||
```
|
||||
|
||||
#### Volume Mount Issues
|
||||
```bash
|
||||
# Check volume permissions
|
||||
ls -la uploads/ logs/
|
||||
|
||||
# Fix permissions
|
||||
sudo chown -R $USER:$USER uploads logs
|
||||
chmod 755 uploads logs
|
||||
```
|
||||
|
||||
### 2. Debug Mode
|
||||
```bash
|
||||
# Run container in debug mode
|
||||
docker-compose run --rm jobforge-app bash
|
||||
|
||||
# Run with environment override
|
||||
docker-compose run -e DEBUG=true jobforge-app
|
||||
|
||||
# Attach to running container
|
||||
docker-compose exec jobforge-app bash
|
||||
```
|
||||
|
||||
### 3. Network Issues
|
||||
```bash
|
||||
# Check network connectivity
|
||||
docker network ls
|
||||
docker network inspect jobforge_jobforge-network
|
||||
|
||||
# Test inter-container communication
|
||||
docker-compose exec jobforge-app ping postgres
|
||||
docker-compose exec nginx ping jobforge-app
|
||||
```
|
||||
|
||||
## Optimization
|
||||
|
||||
### 1. Image Optimization
|
||||
```dockerfile
|
||||
# Use multi-stage builds to reduce image size
|
||||
# Use .dockerignore to exclude unnecessary files
|
||||
# Use specific base image versions
|
||||
# Combine RUN commands to reduce layers
|
||||
# Clean up package caches
|
||||
```
|
||||
|
||||
### 2. Resource Limits
|
||||
```yaml
|
||||
# Set appropriate resource limits
|
||||
deploy:
|
||||
resources:
|
||||
limits:
|
||||
cpus: '1.0'
|
||||
memory: 1G
|
||||
reservations:
|
||||
cpus: '0.5'
|
||||
memory: 512M
|
||||
```
|
||||
|
||||
### 3. Volume Optimization
|
||||
```bash
|
||||
# Use bind mounts for development
|
||||
# Use named volumes for persistent data
|
||||
# Regular cleanup of unused volumes
|
||||
docker volume prune
|
||||
```
|
||||
|
||||
## Security Best Practices
|
||||
|
||||
### 1. Container Security
|
||||
```dockerfile
|
||||
# Run as non-root user
|
||||
USER jobforge
|
||||
|
||||
# Use specific image versions
|
||||
FROM python:3.12-slim
|
||||
|
||||
# Don't expose unnecessary ports
|
||||
# Use secrets for sensitive data
|
||||
```
|
||||
|
||||
### 2. Network Security
|
||||
```yaml
|
||||
# Use custom networks
|
||||
networks:
|
||||
jobforge-network:
|
||||
driver: bridge
|
||||
internal: true # For internal services
|
||||
```
|
||||
|
||||
### 3. Environment Security
|
||||
```bash
|
||||
# Use .env files for secrets
|
||||
# Don't commit secrets to version control
|
||||
# Use Docker secrets in production
|
||||
# Regular security updates
|
||||
```
|
||||
|
||||
This Docker setup provides a robust, scalable, and maintainable containerization solution for Job Forge, suitable for both development and production environments.
|
||||
530
docs/infrastructure/server_deployment.md
Normal file
530
docs/infrastructure/server_deployment.md
Normal file
@@ -0,0 +1,530 @@
|
||||
# Server Deployment Guide - Job Forge
|
||||
|
||||
## Overview
|
||||
|
||||
This guide covers deploying Job Forge to your own server for prototype development and testing. The deployment uses Docker containers for easy management and isolation.
|
||||
|
||||
## Prerequisites
|
||||
|
||||
### Server Requirements
|
||||
- **OS**: Ubuntu 20.04+ or CentOS 8+
|
||||
- **RAM**: Minimum 2GB, recommended 4GB
|
||||
- **Storage**: Minimum 20GB available disk space
|
||||
- **CPU**: 2 cores recommended
|
||||
- **Network**: Public IP address and domain name (optional)
|
||||
|
||||
### Required Software
|
||||
- Docker 20.10+
|
||||
- Docker Compose 2.0+
|
||||
- Git
|
||||
- Nginx (for reverse proxy)
|
||||
- Certbot (for SSL certificates)
|
||||
|
||||
## Initial Server Setup
|
||||
|
||||
### 1. Update System
|
||||
```bash
|
||||
# Ubuntu/Debian
|
||||
sudo apt update && sudo apt upgrade -y
|
||||
|
||||
# CentOS/RHEL
|
||||
sudo yum update -y
|
||||
```
|
||||
|
||||
### 2. Install Docker
|
||||
```bash
|
||||
# Ubuntu/Debian
|
||||
curl -fsSL https://get.docker.com -o get-docker.sh
|
||||
sudo sh get-docker.sh
|
||||
sudo usermod -aG docker $USER
|
||||
|
||||
# CentOS/RHEL
|
||||
sudo yum install -y docker
|
||||
sudo systemctl start docker
|
||||
sudo systemctl enable docker
|
||||
sudo usermod -aG docker $USER
|
||||
```
|
||||
|
||||
### 3. Install Docker Compose
|
||||
```bash
|
||||
sudo curl -L "https://github.com/docker/compose/releases/latest/download/docker-compose-$(uname -s)-$(uname -m)" -o /usr/local/bin/docker-compose
|
||||
sudo chmod +x /usr/local/bin/docker-compose
|
||||
```
|
||||
|
||||
### 4. Install Additional Tools
|
||||
```bash
|
||||
# Ubuntu/Debian
|
||||
sudo apt install -y git nginx certbot python3-certbot-nginx
|
||||
|
||||
# CentOS/RHEL
|
||||
sudo yum install -y git nginx certbot python3-certbot-nginx
|
||||
```
|
||||
|
||||
## Application Deployment
|
||||
|
||||
### 1. Clone Repository
|
||||
```bash
|
||||
cd /opt
|
||||
sudo git clone https://github.com/yourusername/job-forge.git
|
||||
sudo chown -R $USER:$USER job-forge
|
||||
cd job-forge
|
||||
```
|
||||
|
||||
### 2. Environment Configuration
|
||||
```bash
|
||||
# Copy environment template
|
||||
cp .env.example .env
|
||||
|
||||
# Edit environment variables
|
||||
nano .env
|
||||
```
|
||||
|
||||
#### Required Environment Variables
|
||||
```bash
|
||||
# Database Configuration
|
||||
DATABASE_URL="postgresql://jobforge:CHANGE_THIS_PASSWORD@postgres:5432/jobforge"
|
||||
DATABASE_POOL_SIZE=10
|
||||
|
||||
# AI Service API Keys (obtain from providers)
|
||||
CLAUDE_API_KEY="your-claude-api-key-here"
|
||||
OPENAI_API_KEY="your-openai-api-key-here"
|
||||
|
||||
# Security Settings
|
||||
JWT_SECRET="your-super-secret-jwt-key-minimum-32-characters"
|
||||
JWT_ALGORITHM="HS256"
|
||||
JWT_EXPIRE_MINUTES=1440
|
||||
|
||||
# Application Settings
|
||||
APP_NAME="Job Forge"
|
||||
DEBUG=false
|
||||
LOG_LEVEL="INFO"
|
||||
|
||||
# Server Configuration
|
||||
SERVER_HOST="0.0.0.0"
|
||||
SERVER_PORT=8000
|
||||
WORKERS=2
|
||||
|
||||
# Security
|
||||
ALLOWED_HOSTS=["yourdomain.com", "www.yourdomain.com", "localhost"]
|
||||
CORS_ORIGINS=["https://yourdomain.com", "https://www.yourdomain.com"]
|
||||
```
|
||||
|
||||
### 3. Create Required Directories
|
||||
```bash
|
||||
mkdir -p uploads backups logs ssl
|
||||
chmod 755 uploads backups logs
|
||||
chmod 700 ssl
|
||||
```
|
||||
|
||||
### 4. Build and Start Services
|
||||
```bash
|
||||
# Build Docker images
|
||||
docker-compose build
|
||||
|
||||
# Start services in background
|
||||
docker-compose up -d
|
||||
|
||||
# Check status
|
||||
docker-compose ps
|
||||
```
|
||||
|
||||
### 5. Initialize Database
|
||||
```bash
|
||||
# Run database migrations
|
||||
docker-compose exec jobforge-app alembic upgrade head
|
||||
|
||||
# Verify database setup
|
||||
docker-compose exec postgres psql -U jobforge -d jobforge -c "\dt"
|
||||
```
|
||||
|
||||
### 6. Verify Application
|
||||
```bash
|
||||
# Check application health
|
||||
curl http://localhost:8000/health
|
||||
|
||||
# Check logs
|
||||
docker-compose logs jobforge-app
|
||||
```
|
||||
|
||||
## Nginx Configuration
|
||||
|
||||
### 1. Create Nginx Configuration
|
||||
```bash
|
||||
sudo nano /etc/nginx/sites-available/jobforge
|
||||
```
|
||||
|
||||
```nginx
|
||||
server {
|
||||
listen 80;
|
||||
server_name yourdomain.com www.yourdomain.com;
|
||||
|
||||
# Redirect to HTTPS
|
||||
return 301 https://$server_name$request_uri;
|
||||
}
|
||||
|
||||
server {
|
||||
listen 443 ssl http2;
|
||||
server_name yourdomain.com www.yourdomain.com;
|
||||
|
||||
# SSL Configuration (will be added by certbot)
|
||||
ssl_certificate /etc/letsencrypt/live/yourdomain.com/fullchain.pem;
|
||||
ssl_certificate_key /etc/letsencrypt/live/yourdomain.com/privkey.pem;
|
||||
|
||||
# Security headers
|
||||
add_header X-Frame-Options "SAMEORIGIN" always;
|
||||
add_header X-XSS-Protection "1; mode=block" always;
|
||||
add_header X-Content-Type-Options "nosniff" always;
|
||||
add_header Referrer-Policy "no-referrer-when-downgrade" always;
|
||||
add_header Content-Security-Policy "default-src 'self' http: https: data: blob: 'unsafe-inline'" always;
|
||||
|
||||
# File upload size limit
|
||||
client_max_body_size 10M;
|
||||
|
||||
# Main application
|
||||
location / {
|
||||
proxy_pass http://localhost:8000;
|
||||
proxy_set_header Host $host;
|
||||
proxy_set_header X-Real-IP $remote_addr;
|
||||
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
|
||||
proxy_set_header X-Forwarded-Proto $scheme;
|
||||
proxy_redirect off;
|
||||
|
||||
# Timeout settings for AI operations
|
||||
proxy_connect_timeout 60s;
|
||||
proxy_send_timeout 60s;
|
||||
proxy_read_timeout 120s;
|
||||
}
|
||||
|
||||
# Health check endpoint
|
||||
location /health {
|
||||
proxy_pass http://localhost:8000/health;
|
||||
access_log off;
|
||||
}
|
||||
|
||||
# Static files (if any)
|
||||
location /static/ {
|
||||
alias /opt/job-forge/static/;
|
||||
expires 30d;
|
||||
add_header Cache-Control "public, immutable";
|
||||
}
|
||||
|
||||
# API documentation
|
||||
location /docs {
|
||||
proxy_pass http://localhost:8000/docs;
|
||||
}
|
||||
|
||||
# Deny access to sensitive files
|
||||
location ~ /\. {
|
||||
deny all;
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
### 2. Enable Site and Test Configuration
|
||||
```bash
|
||||
# Enable site
|
||||
sudo ln -s /etc/nginx/sites-available/jobforge /etc/nginx/sites-enabled/
|
||||
|
||||
# Test configuration
|
||||
sudo nginx -t
|
||||
|
||||
# Restart nginx
|
||||
sudo systemctl restart nginx
|
||||
```
|
||||
|
||||
## SSL Certificate Setup
|
||||
|
||||
### 1. Obtain SSL Certificate
|
||||
```bash
|
||||
# Using Let's Encrypt Certbot
|
||||
sudo certbot --nginx -d yourdomain.com -d www.yourdomain.com
|
||||
|
||||
# Follow prompts to configure SSL
|
||||
```
|
||||
|
||||
### 2. Configure Auto-Renewal
|
||||
```bash
|
||||
# Test renewal
|
||||
sudo certbot renew --dry-run
|
||||
|
||||
# Add cron job for auto-renewal
|
||||
sudo crontab -e
|
||||
|
||||
# Add this line:
|
||||
0 12 * * * /usr/bin/certbot renew --quiet
|
||||
```
|
||||
|
||||
## Firewall Configuration
|
||||
|
||||
### 1. Configure UFW (Ubuntu)
|
||||
```bash
|
||||
sudo ufw allow ssh
|
||||
sudo ufw allow 'Nginx Full'
|
||||
sudo ufw --force enable
|
||||
sudo ufw status
|
||||
```
|
||||
|
||||
### 2. Configure Firewalld (CentOS)
|
||||
```bash
|
||||
sudo firewall-cmd --permanent --add-service=ssh
|
||||
sudo firewall-cmd --permanent --add-service=http
|
||||
sudo firewall-cmd --permanent --add-service=https
|
||||
sudo firewall-cmd --reload
|
||||
```
|
||||
|
||||
## Monitoring and Maintenance
|
||||
|
||||
### 1. Log Management
|
||||
```bash
|
||||
# View application logs
|
||||
docker-compose logs -f jobforge-app
|
||||
|
||||
# View nginx logs
|
||||
sudo tail -f /var/log/nginx/access.log
|
||||
sudo tail -f /var/log/nginx/error.log
|
||||
|
||||
# Set up log rotation
|
||||
sudo nano /etc/logrotate.d/jobforge
|
||||
```
|
||||
|
||||
```
|
||||
/opt/job-forge/logs/*.log {
|
||||
daily
|
||||
missingok
|
||||
rotate 30
|
||||
compress
|
||||
delaycompress
|
||||
notifempty
|
||||
copytruncate
|
||||
}
|
||||
```
|
||||
|
||||
### 2. Backup Configuration
|
||||
```bash
|
||||
# Create backup script
|
||||
sudo nano /opt/job-forge/backup.sh
|
||||
```
|
||||
|
||||
```bash
|
||||
#!/bin/bash
|
||||
# Job Forge backup script
|
||||
|
||||
BACKUP_DIR="/opt/backups/jobforge"
|
||||
DATE=$(date +%Y%m%d_%H%M%S)
|
||||
|
||||
# Create backup directory
|
||||
mkdir -p "$BACKUP_DIR"
|
||||
|
||||
echo "Starting Job Forge backup - $DATE"
|
||||
|
||||
# Database backup
|
||||
docker-compose exec -T postgres pg_dump -U jobforge jobforge | gzip > "$BACKUP_DIR/database_$DATE.sql.gz"
|
||||
|
||||
# Application files backup
|
||||
tar -czf "$BACKUP_DIR/app_$DATE.tar.gz" \
|
||||
--exclude="logs/*" \
|
||||
--exclude="uploads/*" \
|
||||
/opt/job-forge
|
||||
|
||||
# Keep only last 30 days
|
||||
find "$BACKUP_DIR" -name "*.gz" -mtime +30 -delete
|
||||
|
||||
echo "Backup completed successfully"
|
||||
```
|
||||
|
||||
```bash
|
||||
# Make executable
|
||||
chmod +x /opt/job-forge/backup.sh
|
||||
|
||||
# Add to crontab (daily backup at 2 AM)
|
||||
sudo crontab -e
|
||||
0 2 * * * /opt/job-forge/backup.sh
|
||||
```
|
||||
|
||||
### 3. Health Monitoring
|
||||
```bash
|
||||
# Create health check script
|
||||
nano /opt/job-forge/health-check.sh
|
||||
```
|
||||
|
||||
```bash
|
||||
#!/bin/bash
|
||||
# Job Forge health check
|
||||
|
||||
HEALTH_URL="http://localhost:8000/health"
|
||||
LOG_FILE="/opt/job-forge/logs/health-check.log"
|
||||
|
||||
response=$(curl -s -o /dev/null -w "%{http_code}" "$HEALTH_URL")
|
||||
|
||||
if [ "$response" = "200" ]; then
|
||||
echo "$(date): Health check passed" >> "$LOG_FILE"
|
||||
else
|
||||
echo "$(date): Health check failed - HTTP $response" >> "$LOG_FILE"
|
||||
# Send alert (email, Slack, etc.)
|
||||
# systemctl restart docker-compose@job-forge
|
||||
fi
|
||||
```
|
||||
|
||||
```bash
|
||||
# Make executable and add to cron (every 5 minutes)
|
||||
chmod +x /opt/job-forge/health-check.sh
|
||||
crontab -e
|
||||
*/5 * * * * /opt/job-forge/health-check.sh
|
||||
```
|
||||
|
||||
## Application Updates
|
||||
|
||||
### 1. Update Process
|
||||
```bash
|
||||
cd /opt/job-forge
|
||||
|
||||
# Create backup before update
|
||||
./backup.sh
|
||||
|
||||
# Pull latest changes
|
||||
git pull origin main
|
||||
|
||||
# Rebuild and restart services
|
||||
docker-compose build
|
||||
docker-compose up -d
|
||||
|
||||
# Run database migrations if needed
|
||||
docker-compose exec jobforge-app alembic upgrade head
|
||||
|
||||
# Verify application is working
|
||||
curl http://localhost:8000/health
|
||||
```
|
||||
|
||||
### 2. Rollback Process
|
||||
```bash
|
||||
# If update fails, rollback to previous version
|
||||
git log --oneline -10 # Find previous commit
|
||||
git checkout <previous-commit-hash>
|
||||
|
||||
# Rebuild and restart
|
||||
docker-compose build
|
||||
docker-compose up -d
|
||||
|
||||
# Or restore from backup if needed
|
||||
```
|
||||
|
||||
## Troubleshooting
|
||||
|
||||
### Common Issues
|
||||
|
||||
#### 1. Application Won't Start
|
||||
```bash
|
||||
# Check logs
|
||||
docker-compose logs jobforge-app
|
||||
|
||||
# Check database connection
|
||||
docker-compose exec postgres pg_isready -U jobforge
|
||||
|
||||
# Check environment variables
|
||||
docker-compose exec jobforge-app env | grep -E "(DATABASE|CLAUDE|OPENAI)"
|
||||
```
|
||||
|
||||
#### 2. Database Connection Issues
|
||||
```bash
|
||||
# Restart postgres
|
||||
docker-compose restart postgres
|
||||
|
||||
# Check database logs
|
||||
docker-compose logs postgres
|
||||
|
||||
# Connect to database manually
|
||||
docker-compose exec postgres psql -U jobforge -d jobforge
|
||||
```
|
||||
|
||||
#### 3. SSL Certificate Issues
|
||||
```bash
|
||||
# Check certificate status
|
||||
sudo certbot certificates
|
||||
|
||||
# Renew certificate manually
|
||||
sudo certbot renew
|
||||
|
||||
# Check nginx configuration
|
||||
sudo nginx -t
|
||||
```
|
||||
|
||||
#### 4. Permission Issues
|
||||
```bash
|
||||
# Fix file permissions
|
||||
sudo chown -R $USER:$USER /opt/job-forge
|
||||
chmod 755 /opt/job-forge/uploads
|
||||
chmod 700 /opt/job-forge/ssl
|
||||
```
|
||||
|
||||
### Performance Optimization
|
||||
|
||||
#### 1. Database Optimization
|
||||
```bash
|
||||
# Connect to database
|
||||
docker-compose exec postgres psql -U jobforge -d jobforge
|
||||
|
||||
# Check slow queries
|
||||
SELECT query, mean_time, calls FROM pg_stat_statements ORDER BY mean_time DESC LIMIT 10;
|
||||
|
||||
# Analyze table statistics
|
||||
ANALYZE;
|
||||
```
|
||||
|
||||
#### 2. Container Resource Limits
|
||||
```yaml
|
||||
# In docker-compose.yml, add resource limits
|
||||
services:
|
||||
jobforge-app:
|
||||
deploy:
|
||||
resources:
|
||||
limits:
|
||||
cpus: '1.0'
|
||||
memory: 1G
|
||||
reservations:
|
||||
cpus: '0.5'
|
||||
memory: 512M
|
||||
```
|
||||
|
||||
#### 3. Nginx Caching
|
||||
```nginx
|
||||
# Add to nginx configuration
|
||||
location ~* \.(jpg|jpeg|png|gif|ico|css|js)$ {
|
||||
expires 1y;
|
||||
add_header Cache-Control "public, immutable";
|
||||
}
|
||||
```
|
||||
|
||||
## Security Hardening
|
||||
|
||||
### 1. System Updates
|
||||
```bash
|
||||
# Enable automatic security updates
|
||||
sudo apt install unattended-upgrades
|
||||
sudo dpkg-reconfigure -plow unattended-upgrades
|
||||
```
|
||||
|
||||
### 2. Fail2Ban Setup
|
||||
```bash
|
||||
# Install fail2ban
|
||||
sudo apt install fail2ban
|
||||
|
||||
# Configure for nginx
|
||||
sudo nano /etc/fail2ban/jail.local
|
||||
```
|
||||
|
||||
```ini
|
||||
[nginx-http-auth]
|
||||
enabled = true
|
||||
|
||||
[nginx-limit-req]
|
||||
enabled = true
|
||||
```
|
||||
|
||||
### 3. Docker Security
|
||||
```bash
|
||||
# Run containers as non-root user (already configured in Dockerfile)
|
||||
# Limit container capabilities
|
||||
# Use secrets for sensitive data
|
||||
```
|
||||
|
||||
This deployment guide provides a comprehensive setup for Job Forge on your own server. Adjust configurations based on your specific requirements and security policies.
|
||||
Reference in New Issue
Block a user