job-forge/docs/lessons-learned/003-requirements-file-organization.md

Lesson Learned #003: Requirements File Organization

Issue Name

Missing main requirements.txt for local development and testing

Description

After implementing comprehensive test coverage, tests could not be run locally due to missing dependencies. The project had separate requirements-backend.txt and requirements-frontend.txt files for Docker containers, but no unified requirements file for local development.

Error Messages

ModuleNotFoundError: No module named 'structlog'
ModuleNotFoundError: No module named 'pytest'

Root Cause

1. Fragmented Dependencies: Backend and frontend requirements were split into separate files for Docker optimization
2. Missing Local Setup: No unified requirements file for local development and testing
3. Documentation Gap: README didn't clearly explain how to install dependencies for local testing

Solution Implemented

1. Created Main Requirements File

• File: requirements.txt
• Purpose: Combined all dependencies for local development
• Content: Merged backend and frontend requirements

2. Created Development Requirements File

• File: dev-requirements.txt
• Purpose: Testing and development dependencies only
• Content: pytest, black, flake8, mypy, and core dependencies needed for testing

3. Updated Documentation

• File: README.md
• Section: Quick Start
• Addition: Local development setup instructions with proper pip install commands

4. Maintained Docker Optimization

• Approach: Kept separate requirements-backend.txt and requirements-frontend.txt for Docker containers
• Benefit: Smaller container images with only necessary dependencies

File Structure Created

job-forge/
├── requirements.txt              # All dependencies for local development
├── dev-requirements.txt          # Development and testing dependencies only
├── requirements-backend.txt      # Backend container dependencies (existing)
├── requirements-frontend.txt     # Frontend container dependencies (existing)
└── README.md                    # Updated with local setup instructions

Prevention Strategy

1. Requirements File Standards

• Main Requirements: Always maintain a unified requirements.txt for local development
• Development Requirements: Separate dev-requirements.txt for testing tools
• Container Requirements: Keep optimized files for Docker containers

2. Documentation Requirements

• Installation Instructions: Clear pip install commands in README
• Testing Setup: Document how to run tests locally vs in containers
• Dependencies Explanation: Explain the purpose of each requirements file

3. Testing Integration

• Local Testing: Ensure tests can run with local pip-installed dependencies
• Container Testing: Maintain ability to test within Docker environment
• CI/CD Integration: Use appropriate requirements file for each environment

Implementation Details

Requirements.txt Content

# Combined requirements for local development
fastapi==0.109.2
uvicorn[standard]==0.27.1
# ... (all backend and frontend dependencies)
pytest==8.0.2
pytest-asyncio==0.23.5
# ... (all testing dependencies)

Dev-Requirements.txt Content

# Development and testing only
pytest==8.0.2
pytest-asyncio==0.23.5
pytest-cov==4.0.0
black==24.2.0
# ... (minimal set for testing)

README Update

# For local development and testing
pip install -r requirements.txt

# For development dependencies only  
pip install -r dev-requirements.txt

# Run tests locally
python validate_tests.py
python run_tests.py
pytest

Key Takeaways

1. Multiple Requirements Files: Different environments need different dependency sets
2. Local Development Priority: Always provide easy local setup for developers
3. Documentation Clarity: Clear installation instructions prevent frustration
4. Container Optimization: Keep container-specific requirements minimal and focused

Status

✅ RESOLVED - Created unified requirements files and updated documentation

Related Files

• requirements.txt (new)
• dev-requirements.txt (new)
• README.md (updated)
• requirements-backend.txt (existing, unchanged)
• requirements-frontend.txt (existing, unchanged)