job-forge/docs/testing/qa_procedures.md

QA Procedures - Job Forge

Overview

This document outlines the Quality Assurance procedures for Job Forge, including testing strategies, quality gates, bug reporting, and release validation processes.

Testing Strategy

Test Pyramid for Job Forge

    /\     E2E Tests (10%)
   /  \    - Critical user workflows
  /____\   - Cross-browser testing
 /      \  Integration Tests (20%)
/        \ - API endpoint testing
\        / - Database RLS validation
 \______/  - AI service integration
  \    /   Unit Tests (70%)
   \  /    - Business logic
    \/     - Authentication
           - Data validation

1. Unit Testing (70% of tests)

Test Categories

• Authentication & Security: Login, JWT tokens, password hashing
• Business Logic: Application CRUD operations, status transitions
• Data Validation: Pydantic model validation, input sanitization
• AI Integration: Service mocking, error handling, rate limiting
• Database Operations: RLS policies, query optimization

Running Unit Tests

# Run all unit tests
pytest tests/unit/ -v

# Run specific test file
pytest tests/unit/test_auth_service.py -v

# Run with coverage
pytest tests/unit/ --cov=app --cov-report=html

# Run tests matching pattern
pytest -k "test_auth" -v

Unit Test Example

# tests/unit/test_application_service.py
import pytest
from unittest.mock import AsyncMock

@pytest.mark.asyncio
async def test_create_application_with_ai_generation(test_db, test_user, mock_claude_service):
    """Test application creation with AI cover letter generation."""
    
    # Arrange
    mock_claude_service.generate_cover_letter.return_value = "Generated cover letter"
    
    app_data = ApplicationCreate(
        company_name="AI Corp",
        role_title="ML Engineer",
        job_description="Python ML position",
        status="draft"
    )
    
    # Act
    with patch('app.services.ai.claude_service.ClaudeService', return_value=mock_claude_service):
        application = await create_application(test_db, app_data, test_user.id)
    
    # Assert
    assert application.company_name == "AI Corp"
    assert application.cover_letter == "Generated cover letter"
    mock_claude_service.generate_cover_letter.assert_called_once()

2. Integration Testing (20% of tests)

Test Categories

• API Integration: Full request/response testing with authentication
• Database Integration: Multi-tenant isolation, RLS policy validation
• AI Service Integration: Real API calls with mocking strategies
• Service Layer Integration: Complete workflow testing

Running Integration Tests

# Run integration tests
pytest tests/integration/ -v

# Run with test database
pytest tests/integration/ --db-url=postgresql://test:test@localhost:5432/jobforge_test

# Run specific integration test
pytest tests/integration/test_api_auth.py::TestAuthenticationEndpoints::test_complete_registration_flow -v

Integration Test Example

# tests/integration/test_api_applications.py
@pytest.mark.asyncio
async def test_complete_application_workflow(async_client, test_user_token):
    """Test complete application workflow from creation to update."""
    
    headers = {"Authorization": f"Bearer {test_user_token}"}
    
    # 1. Create application
    app_data = {
        "company_name": "Integration Test Corp",
        "role_title": "Software Engineer",
        "job_description": "Full-stack developer position",
        "status": "draft"
    }
    
    create_response = await async_client.post(
        "/api/v1/applications/", 
        json=app_data, 
        headers=headers
    )
    assert create_response.status_code == 201
    
    app_id = create_response.json()["id"]
    
    # 2. Get application
    get_response = await async_client.get(
        f"/api/v1/applications/{app_id}", 
        headers=headers
    )
    assert get_response.status_code == 200
    
    # 3. Update application status
    update_response = await async_client.put(
        f"/api/v1/applications/{app_id}",
        json={"status": "applied"},
        headers=headers
    )
    assert update_response.status_code == 200
    assert update_response.json()["status"] == "applied"

3. End-to-End Testing (10% of tests)

Test Categories

• Critical User Journeys: Registration → Login → Create Application → Generate Cover Letter
• Cross-browser Compatibility: Chrome, Firefox, Safari, Edge
• Performance Testing: Response times, concurrent users
• Error Scenario Testing: Network failures, service outages

E2E Test Tools Setup

# Install Playwright for E2E testing
pip install playwright
playwright install

# Run E2E tests
pytest tests/e2e/ -v --headed  # With browser UI
pytest tests/e2e/ -v           # Headless mode

E2E Test Example

# tests/e2e/test_user_workflows.py
import pytest
from playwright.async_api import async_playwright

@pytest.mark.asyncio
async def test_complete_user_journey():
    """Test complete user journey from registration to application creation."""
    
    async with async_playwright() as p:
        browser = await p.chromium.launch()
        page = await browser.new_page()
        
        try:
            # 1. Navigate to registration
            await page.goto("http://localhost:8000/register")
            
            # 2. Fill registration form
            await page.fill('[data-testid="email-input"]', 'e2e@test.com')
            await page.fill('[data-testid="password-input"]', 'E2EPassword123!')
            await page.fill('[data-testid="first-name-input"]', 'E2E')
            await page.fill('[data-testid="last-name-input"]', 'User')
            
            # 3. Submit registration
            await page.click('[data-testid="register-button"]')
            
            # 4. Verify redirect to dashboard
            await page.wait_for_url("**/dashboard")
            
            # 5. Create application
            await page.click('[data-testid="new-application-button"]')
            await page.fill('[data-testid="company-input"]', 'E2E Test Corp')
            await page.fill('[data-testid="role-input"]', 'Test Engineer')
            
            # 6. Submit application
            await page.click('[data-testid="save-application-button"]')
            
            # 7. Verify application appears
            await page.wait_for_selector('[data-testid="application-card"]')
            
            # 8. Verify application details
            company_text = await page.text_content('[data-testid="company-name"]')
            assert company_text == "E2E Test Corp"
            
        finally:
            await browser.close()

Quality Gates

1. Code Quality Gates

Pre-commit Hooks

# Install pre-commit hooks
pip install pre-commit
pre-commit install

# Run hooks manually
pre-commit run --all-files

.pre-commit-config.yaml

repos:
  - repo: https://github.com/psf/black
    rev: 23.7.0
    hooks:
      - id: black
        language_version: python3.12

  - repo: https://github.com/charliermarsh/ruff-pre-commit
    rev: v0.0.284
    hooks:
      - id: ruff
        args: [--fix, --exit-non-zero-on-fix]

  - repo: https://github.com/pre-commit/mirrors-mypy
    rev: v1.5.1
    hooks:
      - id: mypy
        additional_dependencies: [pydantic, sqlalchemy]

  - repo: https://github.com/pre-commit/pre-commit-hooks
    rev: v4.4.0
    hooks:
      - id: trailing-whitespace
      - id: end-of-file-fixer
      - id: check-yaml
      - id: check-added-large-files

Quality Metrics Thresholds

# Code coverage minimum: 80%
pytest --cov=app --cov-fail-under=80

# Complexity maximum: 10
ruff check --select=C901

# Type coverage minimum: 90%
mypy app/ --strict

2. Functional Quality Gates

API Response Time Requirements

• Authentication endpoints: < 200ms
• CRUD operations: < 500ms
• AI generation endpoints: < 30 seconds
• Dashboard loading: < 2 seconds

Reliability Requirements

• Uptime: > 99% during testing
• Error rate: < 1% for non-AI operations
• AI service fallback: Must handle service failures gracefully

3. Security Quality Gates

Security Testing Checklist

authentication_security:
  - [ ] JWT tokens expire correctly
  - [ ] Password hashing is secure (bcrypt)
  - [ ] Session management is stateless
  - [ ] Rate limiting prevents brute force

authorization_security:
  - [ ] RLS policies enforce user isolation
  - [ ] API endpoints require proper authentication
  - [ ] Users cannot access other users' data
  - [ ] Admin endpoints are properly protected

input_validation:
  - [ ] All API inputs are validated
  - [ ] SQL injection prevention works
  - [ ] XSS prevention is implemented
  - [ ] File upload validation is secure

data_protection:
  - [ ] Sensitive data is encrypted
  - [ ] API keys are properly secured
  - [ ] Environment variables contain no secrets
  - [ ] Database connections are secure

Bug Reporting and Management

1. Bug Classification

Severity Levels

• Critical: Application crashes, data loss, security vulnerabilities
• High: Major features not working, authentication failures
• Medium: Minor features broken, UI issues, performance problems
• Low: Cosmetic issues, minor improvements, documentation errors

Priority Levels

• P0: Fix immediately (< 2 hours)
• P1: Fix within 24 hours
• P2: Fix within 1 week
• P3: Fix in next release cycle

2. Bug Report Template

GitHub Issue Template

## Bug Report

### Summary
Brief description of the bug

### Environment
- **OS**: macOS 14.0 / Windows 11 / Ubuntu 22.04
- **Browser**: Chrome 118.0 / Firefox 119.0 / Safari 17.0
- **Python Version**: 3.12.0
- **FastAPI Version**: 0.104.1

### Steps to Reproduce
1. Go to '...'
2. Click on '...'
3. Enter data '...'
4. See error

### Expected Behavior
What should happen

### Actual Behavior
What actually happens

### Screenshots/Logs

Error logs or screenshots

### Additional Context
Any other context about the problem

### Severity/Priority
- [ ] Critical
- [ ] High
- [ ] Medium
- [ ] Low

3. Bug Triage Process

Weekly Bug Triage Meeting

1. Review new bugs: Assign severity and priority
2. Update existing bugs: Check progress and blockers
3. Close resolved bugs: Verify fixes and close tickets
4. Plan bug fixes: Assign to sprints based on priority

Bug Assignment Criteria

• Critical/P0: Technical Lead + DevOps
• High/P1: Full-stack Developer
• Medium/P2: QA Engineer + Developer collaboration
• Low/P3: Next available developer

Test Data Management

1. Test Data Strategy

Test Database Setup

# Create test database
createdb jobforge_test

# Run test migrations
DATABASE_URL=postgresql://test:test@localhost/jobforge_test alembic upgrade head

# Seed test data
python scripts/seed_test_data.py

Test Data Factory

# tests/factories.py
import factory
from app.models.user import User
from app.models.application import Application

class UserFactory(factory.Factory):
    class Meta:
        model = User
    
    email = factory.Sequence(lambda n: f"user{n}@test.com")
    password_hash = "$2b$12$hash"
    first_name = "Test"
    last_name = factory.Sequence(lambda n: f"User{n}")

class ApplicationFactory(factory.Factory):
    class Meta:
        model = Application
    
    company_name = factory.Faker('company')
    role_title = factory.Faker('job')
    job_description = factory.Faker('text', max_nb_chars=500)
    status = "draft"
    user = factory.SubFactory(UserFactory)

2. Test Environment Management

Environment Isolation

# docker-compose.test.yml
version: '3.8'

services:
  test-db:
    image: pgvector/pgvector:pg16
    environment:
      - POSTGRES_DB=jobforge_test
      - POSTGRES_USER=test
      - POSTGRES_PASSWORD=test
    ports:
      - "5433:5432"
    tmpfs:
      - /var/lib/postgresql/data  # In-memory for speed

  test-app:
    build: .
    environment:
      - DATABASE_URL=postgresql://test:test@test-db:5432/jobforge_test
      - TESTING=true
    depends_on:
      - test-db

Test Data Cleanup

# tests/conftest.py
@pytest.fixture(autouse=True)
async def cleanup_test_data(test_db):
    """Clean up test data after each test."""
    yield
    
    # Truncate all tables
    await test_db.execute("TRUNCATE TABLE applications CASCADE")
    await test_db.execute("TRUNCATE TABLE users CASCADE")
    await test_db.commit()

Performance Testing

1. Load Testing with Locust

Installation and Setup

# Install locust
pip install locust

# Run load tests
locust -f tests/performance/locustfile.py --host=http://localhost:8000

Load Test Example

# tests/performance/locustfile.py
from locust import HttpUser, task, between
import json

class JobForgeUser(HttpUser):
    wait_time = between(1, 3)
    
    def on_start(self):
        """Login user on start."""
        response = self.client.post("/api/auth/login", data={
            "username": "test@example.com",
            "password": "testpass123"
        })
        
        if response.status_code == 200:
            self.token = response.json()["access_token"]
            self.headers = {"Authorization": f"Bearer {self.token}"}
    
    @task(3)
    def get_applications(self):
        """Get user applications."""
        self.client.get("/api/v1/applications/", headers=self.headers)
    
    @task(1)
    def create_application(self):
        """Create new application."""
        app_data = {
            "company_name": "Load Test Corp",
            "role_title": "Test Engineer",
            "job_description": "Performance testing position",
            "status": "draft"
        }
        
        self.client.post(
            "/api/v1/applications/",
            json=app_data,
            headers=self.headers
        )
    
    @task(1)
    def generate_cover_letter(self):
        """Generate AI cover letter (expensive operation)."""
        # Get first application
        response = self.client.get("/api/v1/applications/", headers=self.headers)
        
        if response.status_code == 200:
            applications = response.json()
            if applications:
                app_id = applications[0]["id"]
                self.client.post(
                    f"/api/v1/applications/{app_id}/generate-cover-letter",
                    headers=self.headers
                )

2. Performance Benchmarks

Response Time Targets

# tests/performance/test_benchmarks.py
import pytest
import time
import statistics

@pytest.mark.performance
@pytest.mark.asyncio
async def test_api_response_times(async_client, test_user_token):
    """Test API response time benchmarks."""
    
    headers = {"Authorization": f"Bearer {test_user_token}"}
    
    # Test multiple requests
    response_times = []
    for _ in range(50):
        start_time = time.time()
        
        response = await async_client.get("/api/v1/applications/", headers=headers)
        assert response.status_code == 200
        
        response_time = (time.time() - start_time) * 1000  # Convert to ms
        response_times.append(response_time)
    
    # Analyze results
    avg_time = statistics.mean(response_times)
    p95_time = statistics.quantiles(response_times, n=20)[18]  # 95th percentile
    
    # Assert performance requirements
    assert avg_time < 200, f"Average response time {avg_time}ms exceeds 200ms limit"
    assert p95_time < 500, f"95th percentile {p95_time}ms exceeds 500ms limit"
    
    print(f"Average response time: {avg_time:.2f}ms")
    print(f"95th percentile: {p95_time:.2f}ms")

Release Testing Procedures

1. Pre-Release Testing Checklist

Functional Testing

authentication_testing:
  - [ ] User registration works
  - [ ] User login/logout works
  - [ ] JWT token validation works
  - [ ] Password reset works (if implemented)

application_management:
  - [ ] Create application works
  - [ ] View applications works
  - [ ] Update application works
  - [ ] Delete application works
  - [ ] Application status transitions work

ai_integration:
  - [ ] Cover letter generation works
  - [ ] AI service error handling works
  - [ ] Rate limiting is enforced
  - [ ] Fallback mechanisms work

data_security:
  - [ ] User data isolation works
  - [ ] RLS policies are enforced
  - [ ] No data leakage between users
  - [ ] Sensitive data is protected

Cross-Browser Testing

browsers_to_test:
  chrome:
    - [ ] Latest version
    - [ ] Previous major version
  firefox:
    - [ ] Latest version
    - [ ] ESR version
  safari:
    - [ ] Latest version (macOS/iOS)
  edge:
    - [ ] Latest version

mobile_testing:
  - [ ] iOS Safari
  - [ ] Android Chrome
  - [ ] Responsive design works
  - [ ] Touch interactions work

2. Release Validation Process

Staging Environment Testing

# Deploy to staging
docker-compose -f docker-compose.staging.yml up -d

# Run full test suite against staging
pytest tests/ --base-url=https://staging.jobforge.com

# Run smoke tests
pytest tests/smoke/ -v

# Performance testing
locust -f tests/performance/locustfile.py --host=https://staging.jobforge.com --users=50 --spawn-rate=5 --run-time=5m

Production Deployment Checklist

pre_deployment:
  - [ ] All tests passing in CI/CD
  - [ ] Code review completed
  - [ ] Database migrations tested
  - [ ] Environment variables updated
  - [ ] SSL certificates valid
  - [ ] Backup created

deployment:
  - [ ] Deploy with zero downtime
  - [ ] Health checks passing
  - [ ] Database migrations applied
  - [ ] Cache cleared if needed
  - [ ] CDN updated if needed

post_deployment:
  - [ ] Smoke tests passing
  - [ ] Performance metrics normal
  - [ ] Error rates acceptable
  - [ ] User workflows tested
  - [ ] Rollback plan ready

Continuous Testing Integration

1. CI/CD Pipeline Testing

GitHub Actions Workflow

# .github/workflows/test.yml
name: Test Suite

on:
  push:
    branches: [main, develop]
  pull_request:
    branches: [main]

jobs:
  test:
    runs-on: ubuntu-latest
    
    services:
      postgres:
        image: pgvector/pgvector:pg16
        env:
          POSTGRES_PASSWORD: test
          POSTGRES_DB: jobforge_test
        options: >-
          --health-cmd pg_isready
          --health-interval 10s
          --health-timeout 5s
          --health-retries 5
    
    steps:
    - uses: actions/checkout@v4
    
    - name: Set up Python
      uses: actions/setup-python@v4
      with:
        python-version: '3.12'
    
    - name: Install dependencies
      run: |
        pip install -r requirements.txt
        pip install -r requirements-dev.txt
    
    - name: Run linting
      run: |
        black --check .
        ruff check .
        mypy app/
    
    - name: Run tests
      run: |
        pytest tests/unit/ tests/integration/ --cov=app --cov-report=xml
      env:
        DATABASE_URL: postgresql://postgres:test@localhost:5432/jobforge_test
    
    - name: Upload coverage
      uses: codecov/codecov-action@v3
      with:
        file: ./coverage.xml

2. Quality Metrics Dashboard

Test Results Tracking

# scripts/generate_test_report.py
import json
import subprocess
from datetime import datetime

def generate_test_report():
    """Generate comprehensive test report."""
    
    # Run tests with JSON output
    result = subprocess.run([
        'pytest', 'tests/', '--json-report', '--json-report-file=test_report.json'
    ], capture_output=True, text=True)
    
    # Load test results
    with open('test_report.json') as f:
        test_data = json.load(f)
    
    # Generate summary
    summary = {
        'timestamp': datetime.now().isoformat(),
        'total_tests': test_data['summary']['total'],
        'passed': test_data['summary']['passed'],
        'failed': test_data['summary']['failed'],
        'skipped': test_data['summary']['skipped'],
        'duration': test_data['duration'],
        'pass_rate': test_data['summary']['passed'] / test_data['summary']['total'] * 100
    }
    
    print(f"Test Summary: {summary['passed']}/{summary['total']} passed ({summary['pass_rate']:.1f}%)")
    return summary

if __name__ == "__main__":
    generate_test_report()

This comprehensive QA procedure ensures that Job Forge maintains high quality through systematic testing, monitoring, and validation processes.