cbc3ff0f51c3b6bbe7e15142bcbc1221deac524a
8 Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
| cbc3ff0f51 |
Phase 3 Step 1: Fix Styling Issues & CSS Refactor
Extracted inline CSS to external stylesheet and fixed white row bug affecting dynamically created table rows across all scan views. Changes: - Created web/static/css/styles.css with extracted CSS from base.html - Added CSS variables for consistent theming and maintainability - Added Bootstrap 5 CSS variable overrides to fix table styling - Integrated Chart.js 4.4.0 for future dashboard visualizations - Added Bootstrap Icons for enhanced UI components Template Updates: - Updated base.html to use external CSS instead of inline styles - Added Chart.js dark theme configuration - Fixed white row bug in dashboard.html (added .scan-row class) - Fixed white row bug in scans.html (added .scan-row class) - Fixed white row bug in scan_detail.html port tables (added .scan-row class) The white row bug was caused by Bootstrap 5's CSS variables overriding custom styles. Fixed by setting --bs-table-bg and related variables. Phase 3 Documentation: - Added PHASE3.md with complete implementation plan (2204 lines) - Includes 8 implementation steps, file changes, and success criteria This completes Phase 3 Step 1 (Day 1 of 14). |
|||
| 167ab803a6 |
Phase 2 Step 7: Implement Error Handling & Logging
Comprehensive error handling and logging system with production-ready features for monitoring, debugging, and user experience. Enhanced Logging System: - Implemented RotatingFileHandler (10MB per file, 10 backups, 100MB total) - Separate error log file for ERROR level messages with detailed tracebacks - Structured logging with request IDs, timestamps, and module names - RequestIDLogFilter for automatic request context injection - Console logging in debug mode with simplified format Request/Response Middleware: - Request ID generation using UUID (8-character prefix for readability) - Request timing with millisecond precision - User authentication context in all logs - Response duration tracking and headers (X-Request-ID, X-Request-Duration-Ms) - Security headers: X-Content-Type-Options, X-Frame-Options, X-XSS-Protection Database Error Handling: - Enabled SQLite WAL mode for better concurrency with background jobs - Busy timeout configuration (15 seconds) for lock handling - Automatic rollback on request exceptions via teardown handler - Dedicated SQLAlchemyError handler with explicit rollback - Connection pooling with pre-ping validation Comprehensive Error Handlers: - Content negotiation: JSON responses for API, HTML for web requests - Error handlers for 400, 401, 403, 404, 405, 500 - Database rollback in all error handlers - Full exception logging with traceback for debugging Custom Error Templates: - Created web/templates/errors/ directory with 6 templates - Dark theme matching application design (slate colors) - User-friendly error messages with navigation - Templates: 400, 401, 403, 404, 405, 500 Testing: - Comprehensive test suite (320+ lines) in tests/test_error_handling.py - Tests for JSON vs HTML error responses - Request ID and duration header verification - Security header validation - Log rotation configuration tests - Structured logging tests Bug Fix: - Fixed pagination bug in scans API endpoint - Changed paginated_result.total_pages to paginated_result.pages - Resolves AttributeError when listing scans Files Added: - tests/test_error_handling.py - web/templates/errors/400.html - web/templates/errors/401.html - web/templates/errors/403.html - web/templates/errors/404.html - web/templates/errors/405.html - web/templates/errors/500.html Files Modified: - web/app.py (logging, error handlers, request handlers, database config) - web/api/scans.py (pagination bug fix) - docs/ai/PHASE2.md (mark Step 7 complete, update progress to 86%) Phase 2 Progress: 12/14 days complete (86%) |
|||
| a64096ece3 |
Phase 2 Step 5: Implement Basic UI Templates
Implement comprehensive web UI with dark slate theme matching HTML reports: Templates: - Create base.html with navigation, dark theme (#0f172a background) - Update dashboard.html with stats cards and recent scans table - Update scans.html with pagination, filtering, and status badges - Update scan_detail.html with comprehensive scan results display - Update login.html to extend base template with centered design Features: - AJAX-powered dynamic data loading from API endpoints - Auto-refresh for running scans (10-15 second intervals) - Responsive Bootstrap 5 grid layout - Color scheme matches report_mockup.html (slate dark theme) - Status badges (success/danger/warning/info) with proper colors - Modal dialogs for triggering scans - Pagination with ellipsis for large result sets - Delete confirmation dialogs - Loading spinners for async operations Bug Fixes: - Fix scanner.py imports to use 'src.' prefix for module imports - Fix scans.py to import validate_page_params from pagination module All templates use consistent color palette: - Background: #0f172a, Cards: #1e293b, Accent: #60a5fa - Success: #065f46/#6ee7b7, Danger: #7f1d1d/#fca5a5 - Warning: #78350f/#fcd34d, Info: #1e3a8a/#93c5fd |
|||
| abc682a634 |
Phase 2 Step 4: Implement Authentication System
Implemented comprehensive Flask-Login authentication with single-user support. New Features: - Flask-Login integration with User model - Bcrypt password hashing via PasswordManager - Login, logout, and initial password setup routes - @login_required and @api_auth_required decorators - All API endpoints now require authentication - Bootstrap 5 dark theme UI templates - Dashboard with navigation - Remember me and next parameter redirect support Files Created (12): - web/auth/__init__.py, models.py, decorators.py, routes.py - web/routes/__init__.py, main.py - web/templates/login.html, setup.html, dashboard.html, scans.html, scan_detail.html - tests/test_authentication.py (30+ tests) Files Modified (6): - web/app.py: Added Flask-Login initialization and main routes - web/api/scans.py: Protected all endpoints with @api_auth_required - web/api/settings.py: Protected all endpoints with @api_auth_required - web/api/schedules.py: Protected all endpoints with @api_auth_required - web/api/alerts.py: Protected all endpoints with @api_auth_required - tests/conftest.py: Added authentication test fixtures Security: - Session-based authentication for both web UI and API - Secure password storage with bcrypt - Protected routes redirect to login page - Protected API endpoints return 401 Unauthorized - Health check endpoints remain accessible for monitoring Testing: - User model authentication and properties - Login success/failure flows - Logout and session management - Password setup workflow - API endpoint authentication requirements - Session persistence and remember me functionality - Next parameter redirect behavior Total: ~1,200 lines of code added 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude <noreply@anthropic.com> |
|||
| ee0c5a2c3c |
Phase 2 Step 3: Implement Background Job Queue
Implemented APScheduler integration for background scan execution,
enabling async job processing without blocking HTTP requests.
## Changes
### Background Jobs (web/jobs/)
- scan_job.py - Execute scans in background threads
- execute_scan() with isolated database sessions
- Comprehensive error handling and logging
- Scan status lifecycle tracking
- Timing and error message storage
### Scheduler Service (web/services/scheduler_service.py)
- SchedulerService class for job management
- APScheduler BackgroundScheduler integration
- ThreadPoolExecutor for concurrent jobs (max 3 workers)
- queue_scan() - Immediate job execution
- Job monitoring: list_jobs(), get_job_status()
- Graceful shutdown handling
### Flask Integration (web/app.py)
- init_scheduler() function
- Scheduler initialization in app factory
- Stored scheduler in app context (app.scheduler)
### Database Schema (migration 003)
- Added scan timing fields:
- started_at - Scan execution start time
- completed_at - Scan execution completion time
- error_message - Error details for failed scans
### Service Layer Updates (web/services/scan_service.py)
- trigger_scan() accepts scheduler parameter
- Queues background jobs after creating scan record
- get_scan_status() includes new timing and error fields
- _save_scan_to_db() sets completed_at timestamp
### API Updates (web/api/scans.py)
- POST /api/scans passes scheduler to trigger_scan()
- Scans now execute in background automatically
### Model Updates (web/models.py)
- Added started_at, completed_at, error_message to Scan model
### Testing (tests/test_background_jobs.py)
- 13 unit tests for background job execution
- Scheduler initialization and configuration tests
- Job queuing and status tracking tests
- Scan timing field tests
- Error handling and storage tests
- Integration test for full workflow (skipped by default)
## Features
- Async scan execution without blocking HTTP requests
- Concurrent scan support (configurable max workers)
- Isolated database sessions per background thread
- Scan lifecycle tracking: created → running → completed/failed
- Error messages captured and stored in database
- Job monitoring and management capabilities
- Graceful shutdown waits for running jobs
## Implementation Notes
- Scanner runs in subprocess from background thread
- Docker provides necessary privileges (--privileged, --network host)
- Each job gets isolated SQLAlchemy session (avoid locking)
- Job IDs follow pattern: scan_{scan_id}
- Background jobs survive across requests
- Failed jobs store error messages in database
## Documentation (docs/ai/PHASE2.md)
- Updated progress: 6/14 days complete (43%)
- Marked Step 3 as complete
- Added detailed implementation notes
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-Authored-By: Claude <noreply@anthropic.com>
|
|||
| 6c4905d6c1 |
Phase 2 Step 2: Implement Scan API Endpoints
Implemented all 5 scan management endpoints with comprehensive error handling, logging, and integration tests. ## Changes ### API Endpoints (web/api/scans.py) - POST /api/scans - Trigger new scan with config file validation - GET /api/scans - List scans with pagination and status filtering - GET /api/scans/<id> - Retrieve scan details with all relationships - DELETE /api/scans/<id> - Delete scan and associated files - GET /api/scans/<id>/status - Poll scan status for long-running scans ### Features - Comprehensive error handling (400, 404, 500) - Structured logging with appropriate levels - Input validation via validators - Consistent JSON error format - SQLAlchemy error handling with graceful degradation - HTTP status codes following REST conventions ### Testing (tests/test_scan_api.py) - 24 integration tests covering all endpoints - Empty/populated scan lists - Pagination with multiple pages - Status filtering - Error scenarios (invalid input, not found, etc.) - Complete workflow integration test ### Test Infrastructure (tests/conftest.py) - Flask app fixture with test database - Flask test client fixture - Database session fixture compatible with app context - Sample scan fixture for testing ### Documentation (docs/ai/PHASE2.md) - Updated progress: 4/14 days complete (29%) - Marked Step 2 as complete - Added implementation details and testing results ## Implementation Notes - All endpoints use ScanService for business logic separation - Scan triggering returns immediately; client polls status endpoint - Background job execution will be added in Step 3 - Authentication will be added in Step 4 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude <noreply@anthropic.com> |
|||
| d7c68a2be8 |
Phase 2 Step 1: Implement database and service layer
Complete the foundation for Phase 2 by implementing the service layer, utilities, and comprehensive test suite. This establishes the core business logic for scan management. Service Layer: - Add ScanService class with complete scan lifecycle management * trigger_scan() - Create scan record and prepare for execution * get_scan() - Retrieve scan with all related data (eager loading) * list_scans() - Paginated scan list with status filtering * delete_scan() - Remove scan from DB and delete all files * get_scan_status() - Poll current scan status and progress * _save_scan_to_db() - Persist scan results to database * _map_report_to_models() - Complex JSON-to-DB mapping logic Database Mapping: - Comprehensive mapping from scanner JSON output to normalized schema - Handles nested relationships: sites → IPs → ports → services → certs → TLS - Processes both TCP and UDP ports with expected/actual tracking - Maps service detection results with HTTP/HTTPS information - Stores SSL/TLS certificates with expiration tracking - Records TLS version support and cipher suites - Links screenshots to services Utilities: - Add pagination.py with PaginatedResult class * paginate() function for SQLAlchemy queries * validate_page_params() for input sanitization * Metadata: total, pages, has_prev, has_next, etc. - Add validators.py with comprehensive validation functions * validate_config_file() - YAML structure and required fields * validate_scan_status() - Enum validation (running/completed/failed) * validate_scan_id() - Positive integer validation * validate_port() - Port range validation (1-65535) * validate_ip_address() - Basic IPv4 format validation * sanitize_filename() - Path traversal prevention Database Migration: - Add migration 002 for scan status index - Optimizes queries filtering by scan status - Timestamp index already exists from migration 001 Testing: - Add pytest infrastructure with conftest.py * test_db fixture - Temporary SQLite database per test * sample_scan_report fixture - Realistic scanner output * sample_config_file fixture - Valid YAML config * sample_invalid_config_file fixture - For validation tests - Add comprehensive test_scan_service.py (15 tests) * Test scan trigger with valid/invalid configs * Test scan retrieval (found/not found cases) * Test scan listing with pagination and filtering * Test scan deletion with cascade cleanup * Test scan status retrieval * Test database mapping from JSON to models * Test expected vs actual port flagging * Test certificate and TLS data mapping * Test full scan retrieval with all relationships * All tests passing Files Added: - web/services/__init__.py - web/services/scan_service.py (545 lines) - web/utils/pagination.py (153 lines) - web/utils/validators.py (245 lines) - migrations/versions/002_add_scan_indexes.py - tests/__init__.py - tests/conftest.py (142 lines) - tests/test_scan_service.py (374 lines) Next Steps (Step 2): - Implement scan API endpoints in web/api/scans.py - Add authentication decorators - Integrate ScanService with API routes - Test API endpoints with integration tests Phase 2 Step 1 Complete ✓ |
|||
| 986c0d3d17 |
Complete Phase 1: Foundation - Flask web application infrastructure
Implement complete database schema and Flask application structure for SneakyScan web interface. This establishes the foundation for web-based scan management, scheduling, and visualization. Database & ORM: - Add 11 SQLAlchemy models for comprehensive scan data storage (Scan, ScanSite, ScanIP, ScanPort, ScanService, ScanCertificate, ScanTLSVersion, Schedule, Alert, AlertRule, Setting) - Configure Alembic migrations system with initial schema migration - Add init_db.py script for database initialization and password setup - Support both migration-based and direct table creation Settings System: - Implement SettingsManager with automatic encryption for sensitive values - Add Fernet encryption for SMTP passwords and API tokens - Implement PasswordManager with bcrypt password hashing (work factor 12) - Initialize default settings for SMTP, authentication, and retention Flask Application: - Create Flask app factory pattern with scoped session management - Add 4 API blueprints: scans, schedules, alerts, settings - Implement functional Settings API (GET/PUT/DELETE endpoints) - Add CORS support, error handlers, and request/response logging - Configure development and production logging to file and console Docker & Deployment: - Update Dockerfile to install Flask dependencies - Add docker-compose-web.yml for web application deployment - Configure volume mounts for database, output, and logs persistence - Expose port 5000 for Flask web server Testing & Validation: - Add validate_phase1.py script to verify all deliverables - Validate directory structure, Python syntax, models, and endpoints - All validation checks passing Documentation: - Add PHASE1_COMPLETE.md with comprehensive Phase 1 summary - Update ROADMAP.md with Phase 1 completion status - Update .gitignore to exclude database files and documentation Files changed: 21 files - New: web/ directory with complete Flask app structure - New: migrations/ with Alembic configuration - New: requirements-web.txt with Flask dependencies - Modified: Dockerfile, ROADMAP.md, .gitignore |
