job-forge/docs/infrastructure/server_deployment.md

Server Deployment Guide - Job Forge

Overview

This guide covers deploying Job Forge to your own server for prototype development and testing. The deployment uses Docker containers for easy management and isolation.

Prerequisites

Server Requirements

• OS: Ubuntu 20.04+ or CentOS 8+
• RAM: Minimum 2GB, recommended 4GB
• Storage: Minimum 20GB available disk space
• CPU: 2 cores recommended
• Network: Public IP address and domain name (optional)

Required Software

• Docker 20.10+
• Docker Compose 2.0+
• Git
• Nginx (for reverse proxy)
• Certbot (for SSL certificates)

Initial Server Setup

1. Update System

# Ubuntu/Debian
sudo apt update && sudo apt upgrade -y

# CentOS/RHEL
sudo yum update -y

2. Install Docker

# Ubuntu/Debian
curl -fsSL https://get.docker.com -o get-docker.sh
sudo sh get-docker.sh
sudo usermod -aG docker $USER

# CentOS/RHEL
sudo yum install -y docker
sudo systemctl start docker
sudo systemctl enable docker
sudo usermod -aG docker $USER

3. Install Docker Compose

sudo curl -L "https://github.com/docker/compose/releases/latest/download/docker-compose-$(uname -s)-$(uname -m)" -o /usr/local/bin/docker-compose
sudo chmod +x /usr/local/bin/docker-compose

4. Install Additional Tools

# Ubuntu/Debian
sudo apt install -y git nginx certbot python3-certbot-nginx

# CentOS/RHEL
sudo yum install -y git nginx certbot python3-certbot-nginx

Application Deployment

1. Clone Repository

cd /opt
sudo git clone https://github.com/yourusername/job-forge.git
sudo chown -R $USER:$USER job-forge
cd job-forge

2. Environment Configuration

# Copy environment template
cp .env.example .env

# Edit environment variables
nano .env

Required Environment Variables

# Database Configuration
DATABASE_URL="postgresql://jobforge:CHANGE_THIS_PASSWORD@postgres:5432/jobforge"
DATABASE_POOL_SIZE=10

# AI Service API Keys (obtain from providers)
CLAUDE_API_KEY="your-claude-api-key-here"
OPENAI_API_KEY="your-openai-api-key-here"

# Security Settings
JWT_SECRET="your-super-secret-jwt-key-minimum-32-characters"
JWT_ALGORITHM="HS256"
JWT_EXPIRE_MINUTES=1440

# Application Settings
APP_NAME="Job Forge"
DEBUG=false
LOG_LEVEL="INFO"

# Server Configuration
SERVER_HOST="0.0.0.0"
SERVER_PORT=8000
WORKERS=2

# Security
ALLOWED_HOSTS=["yourdomain.com", "www.yourdomain.com", "localhost"]
CORS_ORIGINS=["https://yourdomain.com", "https://www.yourdomain.com"]

3. Create Required Directories

mkdir -p uploads backups logs ssl
chmod 755 uploads backups logs
chmod 700 ssl

4. Build and Start Services

# Build Docker images
docker-compose build

# Start services in background
docker-compose up -d

# Check status
docker-compose ps

5. Initialize Database

# Run database migrations
docker-compose exec jobforge-app alembic upgrade head

# Verify database setup
docker-compose exec postgres psql -U jobforge -d jobforge -c "\dt"

6. Verify Application

# Check application health
curl http://localhost:8000/health

# Check logs
docker-compose logs jobforge-app

Nginx Configuration

1. Create Nginx Configuration

sudo nano /etc/nginx/sites-available/jobforge

server {
    listen 80;
    server_name yourdomain.com www.yourdomain.com;
    
    # Redirect to HTTPS
    return 301 https://$server_name$request_uri;
}

server {
    listen 443 ssl http2;
    server_name yourdomain.com www.yourdomain.com;

    # SSL Configuration (will be added by certbot)
    ssl_certificate /etc/letsencrypt/live/yourdomain.com/fullchain.pem;
    ssl_certificate_key /etc/letsencrypt/live/yourdomain.com/privkey.pem;
    
    # Security headers
    add_header X-Frame-Options "SAMEORIGIN" always;
    add_header X-XSS-Protection "1; mode=block" always;
    add_header X-Content-Type-Options "nosniff" always;
    add_header Referrer-Policy "no-referrer-when-downgrade" always;
    add_header Content-Security-Policy "default-src 'self' http: https: data: blob: 'unsafe-inline'" always;

    # File upload size limit
    client_max_body_size 10M;

    # Main application
    location / {
        proxy_pass http://localhost:8000;
        proxy_set_header Host $host;
        proxy_set_header X-Real-IP $remote_addr;
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
        proxy_set_header X-Forwarded-Proto $scheme;
        proxy_redirect off;
        
        # Timeout settings for AI operations
        proxy_connect_timeout 60s;
        proxy_send_timeout 60s;
        proxy_read_timeout 120s;
    }

    # Health check endpoint
    location /health {
        proxy_pass http://localhost:8000/health;
        access_log off;
    }

    # Static files (if any)
    location /static/ {
        alias /opt/job-forge/static/;
        expires 30d;
        add_header Cache-Control "public, immutable";
    }

    # API documentation
    location /docs {
        proxy_pass http://localhost:8000/docs;
    }

    # Deny access to sensitive files
    location ~ /\. {
        deny all;
    }
}

2. Enable Site and Test Configuration

# Enable site
sudo ln -s /etc/nginx/sites-available/jobforge /etc/nginx/sites-enabled/

# Test configuration
sudo nginx -t

# Restart nginx
sudo systemctl restart nginx

SSL Certificate Setup

1. Obtain SSL Certificate

# Using Let's Encrypt Certbot
sudo certbot --nginx -d yourdomain.com -d www.yourdomain.com

# Follow prompts to configure SSL

2. Configure Auto-Renewal

# Test renewal
sudo certbot renew --dry-run

# Add cron job for auto-renewal
sudo crontab -e

# Add this line:
0 12 * * * /usr/bin/certbot renew --quiet

Firewall Configuration

1. Configure UFW (Ubuntu)

sudo ufw allow ssh
sudo ufw allow 'Nginx Full'
sudo ufw --force enable
sudo ufw status

2. Configure Firewalld (CentOS)

sudo firewall-cmd --permanent --add-service=ssh
sudo firewall-cmd --permanent --add-service=http
sudo firewall-cmd --permanent --add-service=https
sudo firewall-cmd --reload

Monitoring and Maintenance

1. Log Management

# View application logs
docker-compose logs -f jobforge-app

# View nginx logs
sudo tail -f /var/log/nginx/access.log
sudo tail -f /var/log/nginx/error.log

# Set up log rotation
sudo nano /etc/logrotate.d/jobforge

/opt/job-forge/logs/*.log {
    daily
    missingok
    rotate 30
    compress
    delaycompress
    notifempty
    copytruncate
}

2. Backup Configuration

# Create backup script
sudo nano /opt/job-forge/backup.sh

#!/bin/bash
# Job Forge backup script

BACKUP_DIR="/opt/backups/jobforge"
DATE=$(date +%Y%m%d_%H%M%S)

# Create backup directory
mkdir -p "$BACKUP_DIR"

echo "Starting Job Forge backup - $DATE"

# Database backup
docker-compose exec -T postgres pg_dump -U jobforge jobforge | gzip > "$BACKUP_DIR/database_$DATE.sql.gz"

# Application files backup
tar -czf "$BACKUP_DIR/app_$DATE.tar.gz" \
    --exclude="logs/*" \
    --exclude="uploads/*" \
    /opt/job-forge

# Keep only last 30 days
find "$BACKUP_DIR" -name "*.gz" -mtime +30 -delete

echo "Backup completed successfully"

# Make executable
chmod +x /opt/job-forge/backup.sh

# Add to crontab (daily backup at 2 AM)
sudo crontab -e
0 2 * * * /opt/job-forge/backup.sh

3. Health Monitoring

# Create health check script
nano /opt/job-forge/health-check.sh

#!/bin/bash
# Job Forge health check

HEALTH_URL="http://localhost:8000/health"
LOG_FILE="/opt/job-forge/logs/health-check.log"

response=$(curl -s -o /dev/null -w "%{http_code}" "$HEALTH_URL")

if [ "$response" = "200" ]; then
    echo "$(date): Health check passed" >> "$LOG_FILE"
else
    echo "$(date): Health check failed - HTTP $response" >> "$LOG_FILE"
    # Send alert (email, Slack, etc.)
    # systemctl restart docker-compose@job-forge
fi

# Make executable and add to cron (every 5 minutes)
chmod +x /opt/job-forge/health-check.sh
crontab -e
*/5 * * * * /opt/job-forge/health-check.sh

Application Updates

1. Update Process

cd /opt/job-forge

# Create backup before update
./backup.sh

# Pull latest changes
git pull origin main

# Rebuild and restart services
docker-compose build
docker-compose up -d

# Run database migrations if needed
docker-compose exec jobforge-app alembic upgrade head

# Verify application is working
curl http://localhost:8000/health

2. Rollback Process

# If update fails, rollback to previous version
git log --oneline -10  # Find previous commit
git checkout <previous-commit-hash>

# Rebuild and restart
docker-compose build
docker-compose up -d

# Or restore from backup if needed

Troubleshooting

Common Issues

1. Application Won't Start

# Check logs
docker-compose logs jobforge-app

# Check database connection
docker-compose exec postgres pg_isready -U jobforge

# Check environment variables
docker-compose exec jobforge-app env | grep -E "(DATABASE|CLAUDE|OPENAI)"

2. Database Connection Issues

# Restart postgres
docker-compose restart postgres

# Check database logs
docker-compose logs postgres

# Connect to database manually
docker-compose exec postgres psql -U jobforge -d jobforge

3. SSL Certificate Issues

# Check certificate status
sudo certbot certificates

# Renew certificate manually
sudo certbot renew

# Check nginx configuration
sudo nginx -t

4. Permission Issues

# Fix file permissions
sudo chown -R $USER:$USER /opt/job-forge
chmod 755 /opt/job-forge/uploads
chmod 700 /opt/job-forge/ssl

Performance Optimization

1. Database Optimization

# Connect to database
docker-compose exec postgres psql -U jobforge -d jobforge

# Check slow queries
SELECT query, mean_time, calls FROM pg_stat_statements ORDER BY mean_time DESC LIMIT 10;

# Analyze table statistics
ANALYZE;

2. Container Resource Limits

# In docker-compose.yml, add resource limits
services:
  jobforge-app:
    deploy:
      resources:
        limits:
          cpus: '1.0'
          memory: 1G
        reservations:
          cpus: '0.5'
          memory: 512M

3. Nginx Caching

# Add to nginx configuration
location ~* \.(jpg|jpeg|png|gif|ico|css|js)$ {
    expires 1y;
    add_header Cache-Control "public, immutable";
}

Security Hardening

1. System Updates

# Enable automatic security updates
sudo apt install unattended-upgrades
sudo dpkg-reconfigure -plow unattended-upgrades

2. Fail2Ban Setup

# Install fail2ban
sudo apt install fail2ban

# Configure for nginx
sudo nano /etc/fail2ban/jail.local

[nginx-http-auth]
enabled = true

[nginx-limit-req]
enabled = true

3. Docker Security

# Run containers as non-root user (already configured in Dockerfile)
# Limit container capabilities
# Use secrets for sensitive data

This deployment guide provides a comprehensive setup for Job Forge on your own server. Adjust configurations based on your specific requirements and security policies.