job-forge/docs/infrastructure/server_deployment.md

# Server Deployment Guide - Job Forge

## Overview

This guide covers deploying Job Forge to your own server for prototype development and testing. The deployment uses Docker containers for easy management and isolation.

## Prerequisites

### Server Requirements
- **OS**: Ubuntu 20.04+ or CentOS 8+
- **RAM**: Minimum 2GB, recommended 4GB
- **Storage**: Minimum 20GB available disk space
- **CPU**: 2 cores recommended
- **Network**: Public IP address and domain name (optional)

### Required Software
- Docker 20.10+
- Docker Compose 2.0+
- Git
- Nginx (for reverse proxy)
- Certbot (for SSL certificates)

## Initial Server Setup

### 1. Update System
```bash
# Ubuntu/Debian
sudo apt update && sudo apt upgrade -y

# CentOS/RHEL
sudo yum update -y
```

### 2. Install Docker
```bash
# Ubuntu/Debian
curl -fsSL https://get.docker.com -o get-docker.sh
sudo sh get-docker.sh
sudo usermod -aG docker $USER

# CentOS/RHEL
sudo yum install -y docker
sudo systemctl start docker
sudo systemctl enable docker
sudo usermod -aG docker $USER
```

### 3. Install Docker Compose
```bash
sudo curl -L "https://github.com/docker/compose/releases/latest/download/docker-compose-$(uname -s)-$(uname -m)" -o /usr/local/bin/docker-compose
sudo chmod +x /usr/local/bin/docker-compose
```

### 4. Install Additional Tools
```bash
# Ubuntu/Debian
sudo apt install -y git nginx certbot python3-certbot-nginx

# CentOS/RHEL
sudo yum install -y git nginx certbot python3-certbot-nginx
```

## Application Deployment

### 1. Clone Repository
```bash
cd /opt
sudo git clone https://github.com/yourusername/job-forge.git
sudo chown -R $USER:$USER job-forge
cd job-forge
```

### 2. Environment Configuration
```bash
# Copy environment template
cp .env.example .env

# Edit environment variables
nano .env
```

#### Required Environment Variables
```bash
# Database Configuration
DATABASE_URL="postgresql://jobforge:CHANGE_THIS_PASSWORD@postgres:5432/jobforge"
DATABASE_POOL_SIZE=10

# AI Service API Keys (obtain from providers)
CLAUDE_API_KEY="your-claude-api-key-here"
OPENAI_API_KEY="your-openai-api-key-here"

# Security Settings
JWT_SECRET="your-super-secret-jwt-key-minimum-32-characters"
JWT_ALGORITHM="HS256"
JWT_EXPIRE_MINUTES=1440

# Application Settings
APP_NAME="Job Forge"
DEBUG=false
LOG_LEVEL="INFO"

# Server Configuration
SERVER_HOST="0.0.0.0"
SERVER_PORT=8000
WORKERS=2

# Security
ALLOWED_HOSTS=["yourdomain.com", "www.yourdomain.com", "localhost"]
CORS_ORIGINS=["https://yourdomain.com", "https://www.yourdomain.com"]
```

### 3. Create Required Directories
```bash
mkdir -p uploads backups logs ssl
chmod 755 uploads backups logs
chmod 700 ssl
```

### 4. Build and Start Services
```bash
# Build Docker images
docker-compose build

# Start services in background
docker-compose up -d

# Check status
docker-compose ps
```

### 5. Initialize Database
```bash
# Run database migrations
docker-compose exec jobforge-app alembic upgrade head

# Verify database setup
docker-compose exec postgres psql -U jobforge -d jobforge -c "\dt"
```

### 6. Verify Application
```bash
# Check application health
curl http://localhost:8000/health

# Check logs
docker-compose logs jobforge-app
```

## Nginx Configuration

### 1. Create Nginx Configuration
```bash
sudo nano /etc/nginx/sites-available/jobforge
```

```nginx
server {
    listen 80;
    server_name yourdomain.com www.yourdomain.com;
    
    # Redirect to HTTPS
    return 301 https://$server_name$request_uri;
}

server {
    listen 443 ssl http2;
    server_name yourdomain.com www.yourdomain.com;

    # SSL Configuration (will be added by certbot)
    ssl_certificate /etc/letsencrypt/live/yourdomain.com/fullchain.pem;
    ssl_certificate_key /etc/letsencrypt/live/yourdomain.com/privkey.pem;
    
    # Security headers
    add_header X-Frame-Options "SAMEORIGIN" always;
    add_header X-XSS-Protection "1; mode=block" always;
    add_header X-Content-Type-Options "nosniff" always;
    add_header Referrer-Policy "no-referrer-when-downgrade" always;
    add_header Content-Security-Policy "default-src 'self' http: https: data: blob: 'unsafe-inline'" always;

    # File upload size limit
    client_max_body_size 10M;

    # Main application
    location / {
        proxy_pass http://localhost:8000;
        proxy_set_header Host $host;
        proxy_set_header X-Real-IP $remote_addr;
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
        proxy_set_header X-Forwarded-Proto $scheme;
        proxy_redirect off;
        
        # Timeout settings for AI operations
        proxy_connect_timeout 60s;
        proxy_send_timeout 60s;
        proxy_read_timeout 120s;
    }

    # Health check endpoint
    location /health {
        proxy_pass http://localhost:8000/health;
        access_log off;
    }

    # Static files (if any)
    location /static/ {
        alias /opt/job-forge/static/;
        expires 30d;
        add_header Cache-Control "public, immutable";
    }

    # API documentation
    location /docs {
        proxy_pass http://localhost:8000/docs;
    }

    # Deny access to sensitive files
    location ~ /\. {
        deny all;
    }
}
```

### 2. Enable Site and Test Configuration
```bash
# Enable site
sudo ln -s /etc/nginx/sites-available/jobforge /etc/nginx/sites-enabled/

# Test configuration
sudo nginx -t

# Restart nginx
sudo systemctl restart nginx
```

## SSL Certificate Setup

### 1. Obtain SSL Certificate
```bash
# Using Let's Encrypt Certbot
sudo certbot --nginx -d yourdomain.com -d www.yourdomain.com

# Follow prompts to configure SSL
```

### 2. Configure Auto-Renewal
```bash
# Test renewal
sudo certbot renew --dry-run

# Add cron job for auto-renewal
sudo crontab -e

# Add this line:
0 12 * * * /usr/bin/certbot renew --quiet
```

## Firewall Configuration

### 1. Configure UFW (Ubuntu)
```bash
sudo ufw allow ssh
sudo ufw allow 'Nginx Full'
sudo ufw --force enable
sudo ufw status
```

### 2. Configure Firewalld (CentOS)
```bash
sudo firewall-cmd --permanent --add-service=ssh
sudo firewall-cmd --permanent --add-service=http
sudo firewall-cmd --permanent --add-service=https
sudo firewall-cmd --reload
```

## Monitoring and Maintenance

### 1. Log Management
```bash
# View application logs
docker-compose logs -f jobforge-app

# View nginx logs
sudo tail -f /var/log/nginx/access.log
sudo tail -f /var/log/nginx/error.log

# Set up log rotation
sudo nano /etc/logrotate.d/jobforge
```

```
/opt/job-forge/logs/*.log {
    daily
    missingok
    rotate 30
    compress
    delaycompress
    notifempty
    copytruncate
}
```

### 2. Backup Configuration
```bash
# Create backup script
sudo nano /opt/job-forge/backup.sh
```

```bash
#!/bin/bash
# Job Forge backup script

BACKUP_DIR="/opt/backups/jobforge"
DATE=$(date +%Y%m%d_%H%M%S)

# Create backup directory
mkdir -p "$BACKUP_DIR"

echo "Starting Job Forge backup - $DATE"

# Database backup
docker-compose exec -T postgres pg_dump -U jobforge jobforge | gzip > "$BACKUP_DIR/database_$DATE.sql.gz"

# Application files backup
tar -czf "$BACKUP_DIR/app_$DATE.tar.gz" \
    --exclude="logs/*" \
    --exclude="uploads/*" \
    /opt/job-forge

# Keep only last 30 days
find "$BACKUP_DIR" -name "*.gz" -mtime +30 -delete

echo "Backup completed successfully"
```

```bash
# Make executable
chmod +x /opt/job-forge/backup.sh

# Add to crontab (daily backup at 2 AM)
sudo crontab -e
0 2 * * * /opt/job-forge/backup.sh
```

### 3. Health Monitoring
```bash
# Create health check script
nano /opt/job-forge/health-check.sh
```

```bash
#!/bin/bash
# Job Forge health check

HEALTH_URL="http://localhost:8000/health"
LOG_FILE="/opt/job-forge/logs/health-check.log"

response=$(curl -s -o /dev/null -w "%{http_code}" "$HEALTH_URL")

if [ "$response" = "200" ]; then
    echo "$(date): Health check passed" >> "$LOG_FILE"
else
    echo "$(date): Health check failed - HTTP $response" >> "$LOG_FILE"
    # Send alert (email, Slack, etc.)
    # systemctl restart docker-compose@job-forge
fi
```

```bash
# Make executable and add to cron (every 5 minutes)
chmod +x /opt/job-forge/health-check.sh
crontab -e
*/5 * * * * /opt/job-forge/health-check.sh
```

## Application Updates

### 1. Update Process
```bash
cd /opt/job-forge

# Create backup before update
./backup.sh

# Pull latest changes
git pull origin main

# Rebuild and restart services
docker-compose build
docker-compose up -d

# Run database migrations if needed
docker-compose exec jobforge-app alembic upgrade head

# Verify application is working
curl http://localhost:8000/health
```

### 2. Rollback Process
```bash
# If update fails, rollback to previous version
git log --oneline -10  # Find previous commit
git checkout <previous-commit-hash>

# Rebuild and restart
docker-compose build
docker-compose up -d

# Or restore from backup if needed
```

## Troubleshooting

### Common Issues

#### 1. Application Won't Start
```bash
# Check logs
docker-compose logs jobforge-app

# Check database connection
docker-compose exec postgres pg_isready -U jobforge

# Check environment variables
docker-compose exec jobforge-app env | grep -E "(DATABASE|CLAUDE|OPENAI)"
```

#### 2. Database Connection Issues
```bash
# Restart postgres
docker-compose restart postgres

# Check database logs
docker-compose logs postgres

# Connect to database manually
docker-compose exec postgres psql -U jobforge -d jobforge
```

#### 3. SSL Certificate Issues
```bash
# Check certificate status
sudo certbot certificates

# Renew certificate manually
sudo certbot renew

# Check nginx configuration
sudo nginx -t
```

#### 4. Permission Issues
```bash
# Fix file permissions
sudo chown -R $USER:$USER /opt/job-forge
chmod 755 /opt/job-forge/uploads
chmod 700 /opt/job-forge/ssl
```

### Performance Optimization

#### 1. Database Optimization
```bash
# Connect to database
docker-compose exec postgres psql -U jobforge -d jobforge

# Check slow queries
SELECT query, mean_time, calls FROM pg_stat_statements ORDER BY mean_time DESC LIMIT 10;

# Analyze table statistics
ANALYZE;
```

#### 2. Container Resource Limits
```yaml
# In docker-compose.yml, add resource limits
services:
  jobforge-app:
    deploy:
      resources:
        limits:
          cpus: '1.0'
          memory: 1G
        reservations:
          cpus: '0.5'
          memory: 512M
```

#### 3. Nginx Caching
```nginx
# Add to nginx configuration
location ~* \.(jpg|jpeg|png|gif|ico|css|js)$ {
    expires 1y;
    add_header Cache-Control "public, immutable";
}
```

## Security Hardening

### 1. System Updates
```bash
# Enable automatic security updates
sudo apt install unattended-upgrades
sudo dpkg-reconfigure -plow unattended-upgrades
```

### 2. Fail2Ban Setup
```bash
# Install fail2ban
sudo apt install fail2ban

# Configure for nginx
sudo nano /etc/fail2ban/jail.local
```

```ini
[nginx-http-auth]
enabled = true

[nginx-limit-req]
enabled = true
```

### 3. Docker Security
```bash
# Run containers as non-root user (already configured in Dockerfile)
# Limit container capabilities
# Use secrets for sensitive data
```

This deployment guide provides a comprehensive setup for Job Forge on your own server. Adjust configurations based on your specific requirements and security policies.