4406f545fa063f81165b3b286c637ac6e9be1511
5 Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
| 167ab803a6 |
Phase 2 Step 7: Implement Error Handling & Logging
Comprehensive error handling and logging system with production-ready features for monitoring, debugging, and user experience. Enhanced Logging System: - Implemented RotatingFileHandler (10MB per file, 10 backups, 100MB total) - Separate error log file for ERROR level messages with detailed tracebacks - Structured logging with request IDs, timestamps, and module names - RequestIDLogFilter for automatic request context injection - Console logging in debug mode with simplified format Request/Response Middleware: - Request ID generation using UUID (8-character prefix for readability) - Request timing with millisecond precision - User authentication context in all logs - Response duration tracking and headers (X-Request-ID, X-Request-Duration-Ms) - Security headers: X-Content-Type-Options, X-Frame-Options, X-XSS-Protection Database Error Handling: - Enabled SQLite WAL mode for better concurrency with background jobs - Busy timeout configuration (15 seconds) for lock handling - Automatic rollback on request exceptions via teardown handler - Dedicated SQLAlchemyError handler with explicit rollback - Connection pooling with pre-ping validation Comprehensive Error Handlers: - Content negotiation: JSON responses for API, HTML for web requests - Error handlers for 400, 401, 403, 404, 405, 500 - Database rollback in all error handlers - Full exception logging with traceback for debugging Custom Error Templates: - Created web/templates/errors/ directory with 6 templates - Dark theme matching application design (slate colors) - User-friendly error messages with navigation - Templates: 400, 401, 403, 404, 405, 500 Testing: - Comprehensive test suite (320+ lines) in tests/test_error_handling.py - Tests for JSON vs HTML error responses - Request ID and duration header verification - Security header validation - Log rotation configuration tests - Structured logging tests Bug Fix: - Fixed pagination bug in scans API endpoint - Changed paginated_result.total_pages to paginated_result.pages - Resolves AttributeError when listing scans Files Added: - tests/test_error_handling.py - web/templates/errors/400.html - web/templates/errors/401.html - web/templates/errors/403.html - web/templates/errors/404.html - web/templates/errors/405.html - web/templates/errors/500.html Files Modified: - web/app.py (logging, error handlers, request handlers, database config) - web/api/scans.py (pagination bug fix) - docs/ai/PHASE2.md (mark Step 7 complete, update progress to 86%) Phase 2 Progress: 12/14 days complete (86%) |
|||
| abc682a634 |
Phase 2 Step 4: Implement Authentication System
Implemented comprehensive Flask-Login authentication with single-user support. New Features: - Flask-Login integration with User model - Bcrypt password hashing via PasswordManager - Login, logout, and initial password setup routes - @login_required and @api_auth_required decorators - All API endpoints now require authentication - Bootstrap 5 dark theme UI templates - Dashboard with navigation - Remember me and next parameter redirect support Files Created (12): - web/auth/__init__.py, models.py, decorators.py, routes.py - web/routes/__init__.py, main.py - web/templates/login.html, setup.html, dashboard.html, scans.html, scan_detail.html - tests/test_authentication.py (30+ tests) Files Modified (6): - web/app.py: Added Flask-Login initialization and main routes - web/api/scans.py: Protected all endpoints with @api_auth_required - web/api/settings.py: Protected all endpoints with @api_auth_required - web/api/schedules.py: Protected all endpoints with @api_auth_required - web/api/alerts.py: Protected all endpoints with @api_auth_required - tests/conftest.py: Added authentication test fixtures Security: - Session-based authentication for both web UI and API - Secure password storage with bcrypt - Protected routes redirect to login page - Protected API endpoints return 401 Unauthorized - Health check endpoints remain accessible for monitoring Testing: - User model authentication and properties - Login success/failure flows - Logout and session management - Password setup workflow - API endpoint authentication requirements - Session persistence and remember me functionality - Next parameter redirect behavior Total: ~1,200 lines of code added 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude <noreply@anthropic.com> |
|||
| ee0c5a2c3c |
Phase 2 Step 3: Implement Background Job Queue
Implemented APScheduler integration for background scan execution,
enabling async job processing without blocking HTTP requests.
## Changes
### Background Jobs (web/jobs/)
- scan_job.py - Execute scans in background threads
- execute_scan() with isolated database sessions
- Comprehensive error handling and logging
- Scan status lifecycle tracking
- Timing and error message storage
### Scheduler Service (web/services/scheduler_service.py)
- SchedulerService class for job management
- APScheduler BackgroundScheduler integration
- ThreadPoolExecutor for concurrent jobs (max 3 workers)
- queue_scan() - Immediate job execution
- Job monitoring: list_jobs(), get_job_status()
- Graceful shutdown handling
### Flask Integration (web/app.py)
- init_scheduler() function
- Scheduler initialization in app factory
- Stored scheduler in app context (app.scheduler)
### Database Schema (migration 003)
- Added scan timing fields:
- started_at - Scan execution start time
- completed_at - Scan execution completion time
- error_message - Error details for failed scans
### Service Layer Updates (web/services/scan_service.py)
- trigger_scan() accepts scheduler parameter
- Queues background jobs after creating scan record
- get_scan_status() includes new timing and error fields
- _save_scan_to_db() sets completed_at timestamp
### API Updates (web/api/scans.py)
- POST /api/scans passes scheduler to trigger_scan()
- Scans now execute in background automatically
### Model Updates (web/models.py)
- Added started_at, completed_at, error_message to Scan model
### Testing (tests/test_background_jobs.py)
- 13 unit tests for background job execution
- Scheduler initialization and configuration tests
- Job queuing and status tracking tests
- Scan timing field tests
- Error handling and storage tests
- Integration test for full workflow (skipped by default)
## Features
- Async scan execution without blocking HTTP requests
- Concurrent scan support (configurable max workers)
- Isolated database sessions per background thread
- Scan lifecycle tracking: created → running → completed/failed
- Error messages captured and stored in database
- Job monitoring and management capabilities
- Graceful shutdown waits for running jobs
## Implementation Notes
- Scanner runs in subprocess from background thread
- Docker provides necessary privileges (--privileged, --network host)
- Each job gets isolated SQLAlchemy session (avoid locking)
- Job IDs follow pattern: scan_{scan_id}
- Background jobs survive across requests
- Failed jobs store error messages in database
## Documentation (docs/ai/PHASE2.md)
- Updated progress: 6/14 days complete (43%)
- Marked Step 3 as complete
- Added detailed implementation notes
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-Authored-By: Claude <noreply@anthropic.com>
|
|||
| 6c4905d6c1 |
Phase 2 Step 2: Implement Scan API Endpoints
Implemented all 5 scan management endpoints with comprehensive error handling, logging, and integration tests. ## Changes ### API Endpoints (web/api/scans.py) - POST /api/scans - Trigger new scan with config file validation - GET /api/scans - List scans with pagination and status filtering - GET /api/scans/<id> - Retrieve scan details with all relationships - DELETE /api/scans/<id> - Delete scan and associated files - GET /api/scans/<id>/status - Poll scan status for long-running scans ### Features - Comprehensive error handling (400, 404, 500) - Structured logging with appropriate levels - Input validation via validators - Consistent JSON error format - SQLAlchemy error handling with graceful degradation - HTTP status codes following REST conventions ### Testing (tests/test_scan_api.py) - 24 integration tests covering all endpoints - Empty/populated scan lists - Pagination with multiple pages - Status filtering - Error scenarios (invalid input, not found, etc.) - Complete workflow integration test ### Test Infrastructure (tests/conftest.py) - Flask app fixture with test database - Flask test client fixture - Database session fixture compatible with app context - Sample scan fixture for testing ### Documentation (docs/ai/PHASE2.md) - Updated progress: 4/14 days complete (29%) - Marked Step 2 as complete - Added implementation details and testing results ## Implementation Notes - All endpoints use ScanService for business logic separation - Scan triggering returns immediately; client polls status endpoint - Background job execution will be added in Step 3 - Authentication will be added in Step 4 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude <noreply@anthropic.com> |
|||
| d7c68a2be8 |
Phase 2 Step 1: Implement database and service layer
Complete the foundation for Phase 2 by implementing the service layer, utilities, and comprehensive test suite. This establishes the core business logic for scan management. Service Layer: - Add ScanService class with complete scan lifecycle management * trigger_scan() - Create scan record and prepare for execution * get_scan() - Retrieve scan with all related data (eager loading) * list_scans() - Paginated scan list with status filtering * delete_scan() - Remove scan from DB and delete all files * get_scan_status() - Poll current scan status and progress * _save_scan_to_db() - Persist scan results to database * _map_report_to_models() - Complex JSON-to-DB mapping logic Database Mapping: - Comprehensive mapping from scanner JSON output to normalized schema - Handles nested relationships: sites → IPs → ports → services → certs → TLS - Processes both TCP and UDP ports with expected/actual tracking - Maps service detection results with HTTP/HTTPS information - Stores SSL/TLS certificates with expiration tracking - Records TLS version support and cipher suites - Links screenshots to services Utilities: - Add pagination.py with PaginatedResult class * paginate() function for SQLAlchemy queries * validate_page_params() for input sanitization * Metadata: total, pages, has_prev, has_next, etc. - Add validators.py with comprehensive validation functions * validate_config_file() - YAML structure and required fields * validate_scan_status() - Enum validation (running/completed/failed) * validate_scan_id() - Positive integer validation * validate_port() - Port range validation (1-65535) * validate_ip_address() - Basic IPv4 format validation * sanitize_filename() - Path traversal prevention Database Migration: - Add migration 002 for scan status index - Optimizes queries filtering by scan status - Timestamp index already exists from migration 001 Testing: - Add pytest infrastructure with conftest.py * test_db fixture - Temporary SQLite database per test * sample_scan_report fixture - Realistic scanner output * sample_config_file fixture - Valid YAML config * sample_invalid_config_file fixture - For validation tests - Add comprehensive test_scan_service.py (15 tests) * Test scan trigger with valid/invalid configs * Test scan retrieval (found/not found cases) * Test scan listing with pagination and filtering * Test scan deletion with cascade cleanup * Test scan status retrieval * Test database mapping from JSON to models * Test expected vs actual port flagging * Test certificate and TLS data mapping * Test full scan retrieval with all relationships * All tests passing Files Added: - web/services/__init__.py - web/services/scan_service.py (545 lines) - web/utils/pagination.py (153 lines) - web/utils/validators.py (245 lines) - migrations/versions/002_add_scan_indexes.py - tests/__init__.py - tests/conftest.py (142 lines) - tests/test_scan_service.py (374 lines) Next Steps (Step 2): - Implement scan API endpoints in web/api/scans.py - Add authentication decorators - Integrate ScanService with API routes - Test API endpoints with integration tests Phase 2 Step 1 Complete ✓ |
