job-forge/docs/testing/qa_procedures.md

# QA Procedures - Job Forge

## Overview

This document outlines the Quality Assurance procedures for Job Forge, including testing strategies, quality gates, bug reporting, and release validation processes.

## Testing Strategy

### Test Pyramid for Job Forge
```
    /\     E2E Tests (10%)
   /  \    - Critical user workflows
  /____\   - Cross-browser testing
 /      \  Integration Tests (20%)
/        \ - API endpoint testing
\        / - Database RLS validation
 \______/  - AI service integration
  \    /   Unit Tests (70%)
   \  /    - Business logic
    \/     - Authentication
           - Data validation
```

### 1. Unit Testing (70% of tests)

#### Test Categories
- **Authentication & Security**: Login, JWT tokens, password hashing
- **Business Logic**: Application CRUD operations, status transitions
- **Data Validation**: Pydantic model validation, input sanitization
- **AI Integration**: Service mocking, error handling, rate limiting
- **Database Operations**: RLS policies, query optimization

#### Running Unit Tests
```bash
# Run all unit tests
pytest tests/unit/ -v

# Run specific test file
pytest tests/unit/test_auth_service.py -v

# Run with coverage
pytest tests/unit/ --cov=app --cov-report=html

# Run tests matching pattern
pytest -k "test_auth" -v
```

#### Unit Test Example
```python
# tests/unit/test_application_service.py
import pytest
from unittest.mock import AsyncMock

@pytest.mark.asyncio
async def test_create_application_with_ai_generation(test_db, test_user, mock_claude_service):
    """Test application creation with AI cover letter generation."""
    
    # Arrange
    mock_claude_service.generate_cover_letter.return_value = "Generated cover letter"
    
    app_data = ApplicationCreate(
        company_name="AI Corp",
        role_title="ML Engineer",
        job_description="Python ML position",
        status="draft"
    )
    
    # Act
    with patch('app.services.ai.claude_service.ClaudeService', return_value=mock_claude_service):
        application = await create_application(test_db, app_data, test_user.id)
    
    # Assert
    assert application.company_name == "AI Corp"
    assert application.cover_letter == "Generated cover letter"
    mock_claude_service.generate_cover_letter.assert_called_once()
```

### 2. Integration Testing (20% of tests)

#### Test Categories
- **API Integration**: Full request/response testing with authentication
- **Database Integration**: Multi-tenant isolation, RLS policy validation
- **AI Service Integration**: Real API calls with mocking strategies
- **Service Layer Integration**: Complete workflow testing

#### Running Integration Tests
```bash
# Run integration tests
pytest tests/integration/ -v

# Run with test database
pytest tests/integration/ --db-url=postgresql://test:test@localhost:5432/jobforge_test

# Run specific integration test
pytest tests/integration/test_api_auth.py::TestAuthenticationEndpoints::test_complete_registration_flow -v
```

#### Integration Test Example
```python
# tests/integration/test_api_applications.py
@pytest.mark.asyncio
async def test_complete_application_workflow(async_client, test_user_token):
    """Test complete application workflow from creation to update."""
    
    headers = {"Authorization": f"Bearer {test_user_token}"}
    
    # 1. Create application
    app_data = {
        "company_name": "Integration Test Corp",
        "role_title": "Software Engineer",
        "job_description": "Full-stack developer position",
        "status": "draft"
    }
    
    create_response = await async_client.post(
        "/api/v1/applications/", 
        json=app_data, 
        headers=headers
    )
    assert create_response.status_code == 201
    
    app_id = create_response.json()["id"]
    
    # 2. Get application
    get_response = await async_client.get(
        f"/api/v1/applications/{app_id}", 
        headers=headers
    )
    assert get_response.status_code == 200
    
    # 3. Update application status
    update_response = await async_client.put(
        f"/api/v1/applications/{app_id}",
        json={"status": "applied"},
        headers=headers
    )
    assert update_response.status_code == 200
    assert update_response.json()["status"] == "applied"
```

### 3. End-to-End Testing (10% of tests)

#### Test Categories
- **Critical User Journeys**: Registration → Login → Create Application → Generate Cover Letter
- **Cross-browser Compatibility**: Chrome, Firefox, Safari, Edge
- **Performance Testing**: Response times, concurrent users
- **Error Scenario Testing**: Network failures, service outages

#### E2E Test Tools Setup
```bash
# Install Playwright for E2E testing
pip install playwright
playwright install

# Run E2E tests
pytest tests/e2e/ -v --headed  # With browser UI
pytest tests/e2e/ -v           # Headless mode
```

#### E2E Test Example
```python
# tests/e2e/test_user_workflows.py
import pytest
from playwright.async_api import async_playwright

@pytest.mark.asyncio
async def test_complete_user_journey():
    """Test complete user journey from registration to application creation."""
    
    async with async_playwright() as p:
        browser = await p.chromium.launch()
        page = await browser.new_page()
        
        try:
            # 1. Navigate to registration
            await page.goto("http://localhost:8000/register")
            
            # 2. Fill registration form
            await page.fill('[data-testid="email-input"]', 'e2e@test.com')
            await page.fill('[data-testid="password-input"]', 'E2EPassword123!')
            await page.fill('[data-testid="first-name-input"]', 'E2E')
            await page.fill('[data-testid="last-name-input"]', 'User')
            
            # 3. Submit registration
            await page.click('[data-testid="register-button"]')
            
            # 4. Verify redirect to dashboard
            await page.wait_for_url("**/dashboard")
            
            # 5. Create application
            await page.click('[data-testid="new-application-button"]')
            await page.fill('[data-testid="company-input"]', 'E2E Test Corp')
            await page.fill('[data-testid="role-input"]', 'Test Engineer')
            
            # 6. Submit application
            await page.click('[data-testid="save-application-button"]')
            
            # 7. Verify application appears
            await page.wait_for_selector('[data-testid="application-card"]')
            
            # 8. Verify application details
            company_text = await page.text_content('[data-testid="company-name"]')
            assert company_text == "E2E Test Corp"
            
        finally:
            await browser.close()
```

## Quality Gates

### 1. Code Quality Gates

#### Pre-commit Hooks
```bash
# Install pre-commit hooks
pip install pre-commit
pre-commit install

# Run hooks manually
pre-commit run --all-files
```

#### .pre-commit-config.yaml
```yaml
repos:
  - repo: https://github.com/psf/black
    rev: 23.7.0
    hooks:
      - id: black
        language_version: python3.12

  - repo: https://github.com/charliermarsh/ruff-pre-commit
    rev: v0.0.284
    hooks:
      - id: ruff
        args: [--fix, --exit-non-zero-on-fix]

  - repo: https://github.com/pre-commit/mirrors-mypy
    rev: v1.5.1
    hooks:
      - id: mypy
        additional_dependencies: [pydantic, sqlalchemy]

  - repo: https://github.com/pre-commit/pre-commit-hooks
    rev: v4.4.0
    hooks:
      - id: trailing-whitespace
      - id: end-of-file-fixer
      - id: check-yaml
      - id: check-added-large-files
```

#### Quality Metrics Thresholds
```bash
# Code coverage minimum: 80%
pytest --cov=app --cov-fail-under=80

# Complexity maximum: 10
ruff check --select=C901

# Type coverage minimum: 90%
mypy app/ --strict
```

### 2. Functional Quality Gates

#### API Response Time Requirements
- **Authentication endpoints**: < 200ms
- **CRUD operations**: < 500ms
- **AI generation endpoints**: < 30 seconds
- **Dashboard loading**: < 2 seconds

#### Reliability Requirements
- **Uptime**: > 99% during testing
- **Error rate**: < 1% for non-AI operations
- **AI service fallback**: Must handle service failures gracefully

### 3. Security Quality Gates

#### Security Testing Checklist
```yaml
authentication_security:
  - [ ] JWT tokens expire correctly
  - [ ] Password hashing is secure (bcrypt)
  - [ ] Session management is stateless
  - [ ] Rate limiting prevents brute force

authorization_security:
  - [ ] RLS policies enforce user isolation
  - [ ] API endpoints require proper authentication
  - [ ] Users cannot access other users' data
  - [ ] Admin endpoints are properly protected

input_validation:
  - [ ] All API inputs are validated
  - [ ] SQL injection prevention works
  - [ ] XSS prevention is implemented
  - [ ] File upload validation is secure

data_protection:
  - [ ] Sensitive data is encrypted
  - [ ] API keys are properly secured
  - [ ] Environment variables contain no secrets
  - [ ] Database connections are secure
```

## Bug Reporting and Management

### 1. Bug Classification

#### Severity Levels
- **Critical**: Application crashes, data loss, security vulnerabilities
- **High**: Major features not working, authentication failures
- **Medium**: Minor features broken, UI issues, performance problems
- **Low**: Cosmetic issues, minor improvements, documentation errors

#### Priority Levels
- **P0**: Fix immediately (< 2 hours)
- **P1**: Fix within 24 hours
- **P2**: Fix within 1 week
- **P3**: Fix in next release cycle

### 2. Bug Report Template

#### GitHub Issue Template
```markdown
## Bug Report

### Summary
Brief description of the bug

### Environment
- **OS**: macOS 14.0 / Windows 11 / Ubuntu 22.04
- **Browser**: Chrome 118.0 / Firefox 119.0 / Safari 17.0
- **Python Version**: 3.12.0
- **FastAPI Version**: 0.104.1

### Steps to Reproduce
1. Go to '...'
2. Click on '...'
3. Enter data '...'
4. See error

### Expected Behavior
What should happen

### Actual Behavior
What actually happens

### Screenshots/Logs
```
Error logs or screenshots
```

### Additional Context
Any other context about the problem

### Severity/Priority
- [ ] Critical
- [ ] High
- [ ] Medium
- [ ] Low
```

### 3. Bug Triage Process

#### Weekly Bug Triage Meeting
1. **Review new bugs**: Assign severity and priority
2. **Update existing bugs**: Check progress and blockers
3. **Close resolved bugs**: Verify fixes and close tickets
4. **Plan bug fixes**: Assign to sprints based on priority

#### Bug Assignment Criteria
- **Critical/P0**: Technical Lead + DevOps
- **High/P1**: Full-stack Developer
- **Medium/P2**: QA Engineer + Developer collaboration
- **Low/P3**: Next available developer

## Test Data Management

### 1. Test Data Strategy

#### Test Database Setup
```bash
# Create test database
createdb jobforge_test

# Run test migrations
DATABASE_URL=postgresql://test:test@localhost/jobforge_test alembic upgrade head

# Seed test data
python scripts/seed_test_data.py
```

#### Test Data Factory
```python
# tests/factories.py
import factory
from app.models.user import User
from app.models.application import Application

class UserFactory(factory.Factory):
    class Meta:
        model = User
    
    email = factory.Sequence(lambda n: f"user{n}@test.com")
    password_hash = "$2b$12$hash"
    first_name = "Test"
    last_name = factory.Sequence(lambda n: f"User{n}")

class ApplicationFactory(factory.Factory):
    class Meta:
        model = Application
    
    company_name = factory.Faker('company')
    role_title = factory.Faker('job')
    job_description = factory.Faker('text', max_nb_chars=500)
    status = "draft"
    user = factory.SubFactory(UserFactory)
```

### 2. Test Environment Management

#### Environment Isolation
```yaml
# docker-compose.test.yml
version: '3.8'

services:
  test-db:
    image: pgvector/pgvector:pg16
    environment:
      - POSTGRES_DB=jobforge_test
      - POSTGRES_USER=test
      - POSTGRES_PASSWORD=test
    ports:
      - "5433:5432"
    tmpfs:
      - /var/lib/postgresql/data  # In-memory for speed

  test-app:
    build: .
    environment:
      - DATABASE_URL=postgresql://test:test@test-db:5432/jobforge_test
      - TESTING=true
    depends_on:
      - test-db
```

#### Test Data Cleanup
```python
# tests/conftest.py
@pytest.fixture(autouse=True)
async def cleanup_test_data(test_db):
    """Clean up test data after each test."""
    yield
    
    # Truncate all tables
    await test_db.execute("TRUNCATE TABLE applications CASCADE")
    await test_db.execute("TRUNCATE TABLE users CASCADE")
    await test_db.commit()
```

## Performance Testing

### 1. Load Testing with Locust

#### Installation and Setup
```bash
# Install locust
pip install locust

# Run load tests
locust -f tests/performance/locustfile.py --host=http://localhost:8000
```

#### Load Test Example
```python
# tests/performance/locustfile.py
from locust import HttpUser, task, between
import json

class JobForgeUser(HttpUser):
    wait_time = between(1, 3)
    
    def on_start(self):
        """Login user on start."""
        response = self.client.post("/api/auth/login", data={
            "username": "test@example.com",
            "password": "testpass123"
        })
        
        if response.status_code == 200:
            self.token = response.json()["access_token"]
            self.headers = {"Authorization": f"Bearer {self.token}"}
    
    @task(3)
    def get_applications(self):
        """Get user applications."""
        self.client.get("/api/v1/applications/", headers=self.headers)
    
    @task(1)
    def create_application(self):
        """Create new application."""
        app_data = {
            "company_name": "Load Test Corp",
            "role_title": "Test Engineer",
            "job_description": "Performance testing position",
            "status": "draft"
        }
        
        self.client.post(
            "/api/v1/applications/",
            json=app_data,
            headers=self.headers
        )
    
    @task(1)
    def generate_cover_letter(self):
        """Generate AI cover letter (expensive operation)."""
        # Get first application
        response = self.client.get("/api/v1/applications/", headers=self.headers)
        
        if response.status_code == 200:
            applications = response.json()
            if applications:
                app_id = applications[0]["id"]
                self.client.post(
                    f"/api/v1/applications/{app_id}/generate-cover-letter",
                    headers=self.headers
                )
```

### 2. Performance Benchmarks

#### Response Time Targets
```python
# tests/performance/test_benchmarks.py
import pytest
import time
import statistics

@pytest.mark.performance
@pytest.mark.asyncio
async def test_api_response_times(async_client, test_user_token):
    """Test API response time benchmarks."""
    
    headers = {"Authorization": f"Bearer {test_user_token}"}
    
    # Test multiple requests
    response_times = []
    for _ in range(50):
        start_time = time.time()
        
        response = await async_client.get("/api/v1/applications/", headers=headers)
        assert response.status_code == 200
        
        response_time = (time.time() - start_time) * 1000  # Convert to ms
        response_times.append(response_time)
    
    # Analyze results
    avg_time = statistics.mean(response_times)
    p95_time = statistics.quantiles(response_times, n=20)[18]  # 95th percentile
    
    # Assert performance requirements
    assert avg_time < 200, f"Average response time {avg_time}ms exceeds 200ms limit"
    assert p95_time < 500, f"95th percentile {p95_time}ms exceeds 500ms limit"
    
    print(f"Average response time: {avg_time:.2f}ms")
    print(f"95th percentile: {p95_time:.2f}ms")
```

## Release Testing Procedures

### 1. Pre-Release Testing Checklist

#### Functional Testing
```yaml
authentication_testing:
  - [ ] User registration works
  - [ ] User login/logout works
  - [ ] JWT token validation works
  - [ ] Password reset works (if implemented)

application_management:
  - [ ] Create application works
  - [ ] View applications works
  - [ ] Update application works
  - [ ] Delete application works
  - [ ] Application status transitions work

ai_integration:
  - [ ] Cover letter generation works
  - [ ] AI service error handling works
  - [ ] Rate limiting is enforced
  - [ ] Fallback mechanisms work

data_security:
  - [ ] User data isolation works
  - [ ] RLS policies are enforced
  - [ ] No data leakage between users
  - [ ] Sensitive data is protected
```

#### Cross-Browser Testing
```yaml
browsers_to_test:
  chrome:
    - [ ] Latest version
    - [ ] Previous major version
  firefox:
    - [ ] Latest version
    - [ ] ESR version
  safari:
    - [ ] Latest version (macOS/iOS)
  edge:
    - [ ] Latest version

mobile_testing:
  - [ ] iOS Safari
  - [ ] Android Chrome
  - [ ] Responsive design works
  - [ ] Touch interactions work
```

### 2. Release Validation Process

#### Staging Environment Testing
```bash
# Deploy to staging
docker-compose -f docker-compose.staging.yml up -d

# Run full test suite against staging
pytest tests/ --base-url=https://staging.jobforge.com

# Run smoke tests
pytest tests/smoke/ -v

# Performance testing
locust -f tests/performance/locustfile.py --host=https://staging.jobforge.com --users=50 --spawn-rate=5 --run-time=5m
```

#### Production Deployment Checklist
```yaml
pre_deployment:
  - [ ] All tests passing in CI/CD
  - [ ] Code review completed
  - [ ] Database migrations tested
  - [ ] Environment variables updated
  - [ ] SSL certificates valid
  - [ ] Backup created

deployment:
  - [ ] Deploy with zero downtime
  - [ ] Health checks passing
  - [ ] Database migrations applied
  - [ ] Cache cleared if needed
  - [ ] CDN updated if needed

post_deployment:
  - [ ] Smoke tests passing
  - [ ] Performance metrics normal
  - [ ] Error rates acceptable
  - [ ] User workflows tested
  - [ ] Rollback plan ready
```

## Continuous Testing Integration

### 1. CI/CD Pipeline Testing

#### GitHub Actions Workflow
```yaml
# .github/workflows/test.yml
name: Test Suite

on:
  push:
    branches: [main, develop]
  pull_request:
    branches: [main]

jobs:
  test:
    runs-on: ubuntu-latest
    
    services:
      postgres:
        image: pgvector/pgvector:pg16
        env:
          POSTGRES_PASSWORD: test
          POSTGRES_DB: jobforge_test
        options: >-
          --health-cmd pg_isready
          --health-interval 10s
          --health-timeout 5s
          --health-retries 5
    
    steps:
    - uses: actions/checkout@v4
    
    - name: Set up Python
      uses: actions/setup-python@v4
      with:
        python-version: '3.12'
    
    - name: Install dependencies
      run: |
        pip install -r requirements.txt
        pip install -r requirements-dev.txt
    
    - name: Run linting
      run: |
        black --check .
        ruff check .
        mypy app/
    
    - name: Run tests
      run: |
        pytest tests/unit/ tests/integration/ --cov=app --cov-report=xml
      env:
        DATABASE_URL: postgresql://postgres:test@localhost:5432/jobforge_test
    
    - name: Upload coverage
      uses: codecov/codecov-action@v3
      with:
        file: ./coverage.xml
```

### 2. Quality Metrics Dashboard

#### Test Results Tracking
```python
# scripts/generate_test_report.py
import json
import subprocess
from datetime import datetime

def generate_test_report():
    """Generate comprehensive test report."""
    
    # Run tests with JSON output
    result = subprocess.run([
        'pytest', 'tests/', '--json-report', '--json-report-file=test_report.json'
    ], capture_output=True, text=True)
    
    # Load test results
    with open('test_report.json') as f:
        test_data = json.load(f)
    
    # Generate summary
    summary = {
        'timestamp': datetime.now().isoformat(),
        'total_tests': test_data['summary']['total'],
        'passed': test_data['summary']['passed'],
        'failed': test_data['summary']['failed'],
        'skipped': test_data['summary']['skipped'],
        'duration': test_data['duration'],
        'pass_rate': test_data['summary']['passed'] / test_data['summary']['total'] * 100
    }
    
    print(f"Test Summary: {summary['passed']}/{summary['total']} passed ({summary['pass_rate']:.1f}%)")
    return summary

if __name__ == "__main__":
    generate_test_report()
```

This comprehensive QA procedure ensures that Job Forge maintains high quality through systematic testing, monitoring, and validation processes.