Implemented comprehensive scan comparison functionality with historical
analysis and improved user experience for scan triggering.
Features Added:
- Scan comparison engine with ports, services, and certificates analysis
- Drift score calculation (0.0-1.0 scale) for infrastructure changes
- Side-by-side comparison UI with color-coded changes (added/removed/changed)
- Historical trend charts showing port counts over time
- "Compare with Previous" button on scan detail pages
- Scan history API endpoint for trending data
API Endpoints:
- GET /api/scans/<id1>/compare/<id2> - Compare two scans
- GET /api/stats/scan-history/<id> - Historical scan data for charts
UI Improvements:
- Replaced config file text inputs with dropdown selectors
- Added config file selection to dashboard and scans pages
- Improved delete scan confirmation with proper async handling
- Enhanced error messages with detailed validation feedback
- Added 2-second delay before redirect to ensure deletion completes
Comparison Features:
- Port changes: tracks added, removed, and unchanged ports
- Service changes: detects version updates and service modifications
- Certificate changes: monitors SSL/TLS certificate updates
- Interactive historical charts with clickable data points
- Automatic detection of previous scan for comparison
Bug Fixes:
- Fixed scan deletion UI alert appearing on successful deletion
- Prevented config file path duplication (configs/configs/...)
- Improved error handling for failed API responses
- Added proper JSON response parsing with fallback handling
Testing:
- Created comprehensive test suite for comparison functionality
- Tests cover comparison API, service methods, and drift scoring
- Added edge case tests for identical scans and missing data
Implemented complete scheduler integration with automatic schedule loading,
orphaned scan cleanup, and conversion to local timezone for better UX.
Backend Changes:
- Added load_schedules_on_startup() to load enabled schedules on app start
- Implemented cleanup_orphaned_scans() to handle crashed/interrupted scans
- Converted scheduler from UTC to local system timezone throughout
- Enhanced scheduler service with robust error handling and logging
Frontend Changes:
- Updated all schedule UI templates to display local time instead of UTC
- Improved timezone indicators and user messaging
- Removed confusing timezone converter (no longer needed)
- Updated quick templates and help text for local time
Bug Fixes:
- Fixed critical timezone bug causing cron expressions to run at wrong times
- Fixed orphaned scans stuck in 'running' status after system crashes
- Improved time display clarity across all schedule pages
All schedules now use local system time for intuitive scheduling.
Implemented full schedule management system with API endpoints and
user interface for creating, editing, and managing scheduled scans.
API Implementation:
- Implemented all 6 schedules API endpoints (list, get, create, update, delete, trigger)
- Added comprehensive error handling and validation
- Integrated with ScheduleService and SchedulerService
- Added manual trigger endpoint for on-demand execution
Schedule Management UI:
- Created schedules list page with stats cards and enable/disable toggles
- Built schedule creation form with cron expression builder and quick templates
- Implemented schedule edit page with execution history
- Added "Schedules" navigation link to main menu
- Real-time validation and human-readable cron descriptions
Config File Path Resolution:
- Fixed config file path handling to support relative filenames
- Updated validators.py to resolve relative paths to /app/configs/
- Modified schedule_service.py, scan_service.py, and scan_job.py for consistency
- Ensures UI can use simple filenames while backend uses absolute paths
Scheduler Integration:
- Completed scheduled scan execution in scheduler_service.py
- Added cron job management with APScheduler
- Implemented automatic schedule loading on startup
- Updated run times after each execution
Testing:
- Added comprehensive API integration tests (test_schedule_api.py)
- 22+ test cases covering all endpoints and workflows
Progress: Phase 3 Steps 1-4 complete (36% - 5/14 days)
Next: Step 5 - Enhanced Dashboard with Charts
Implement comprehensive schedule management service for automated scans:
New Files:
- web/services/schedule_service.py (470 lines)
* Complete CRUD operations for schedules
* Cron expression validation using croniter
* Next run time calculation
* Execution history tracking
* Human-readable relative time formatting
- tests/test_schedule_service.py (671 lines, 40+ tests)
* Create/get/list/update/delete schedule tests
* Cron validation and next run calculation tests
* Pagination and filtering tests
* Schedule history and serialization tests
Changes:
- requirements-web.txt: Add croniter==2.0.1 dependency
- docs/ai/PHASE3.md: Mark Step 1 complete, Step 2 next
Key Features:
- Validates cron expressions before saving
- Automatically calculates next execution time
- Preserves historical scans when schedules deleted
- Supports pagination and filtering by enabled status
- Provides relative time display (e.g., "in 2 hours")
Complete the foundation for Phase 2 by implementing the service layer,
utilities, and comprehensive test suite. This establishes the core
business logic for scan management.
Service Layer:
- Add ScanService class with complete scan lifecycle management
* trigger_scan() - Create scan record and prepare for execution
* get_scan() - Retrieve scan with all related data (eager loading)
* list_scans() - Paginated scan list with status filtering
* delete_scan() - Remove scan from DB and delete all files
* get_scan_status() - Poll current scan status and progress
* _save_scan_to_db() - Persist scan results to database
* _map_report_to_models() - Complex JSON-to-DB mapping logic
Database Mapping:
- Comprehensive mapping from scanner JSON output to normalized schema
- Handles nested relationships: sites → IPs → ports → services → certs → TLS
- Processes both TCP and UDP ports with expected/actual tracking
- Maps service detection results with HTTP/HTTPS information
- Stores SSL/TLS certificates with expiration tracking
- Records TLS version support and cipher suites
- Links screenshots to services
Utilities:
- Add pagination.py with PaginatedResult class
* paginate() function for SQLAlchemy queries
* validate_page_params() for input sanitization
* Metadata: total, pages, has_prev, has_next, etc.
- Add validators.py with comprehensive validation functions
* validate_config_file() - YAML structure and required fields
* validate_scan_status() - Enum validation (running/completed/failed)
* validate_scan_id() - Positive integer validation
* validate_port() - Port range validation (1-65535)
* validate_ip_address() - Basic IPv4 format validation
* sanitize_filename() - Path traversal prevention
Database Migration:
- Add migration 002 for scan status index
- Optimizes queries filtering by scan status
- Timestamp index already exists from migration 001
Testing:
- Add pytest infrastructure with conftest.py
* test_db fixture - Temporary SQLite database per test
* sample_scan_report fixture - Realistic scanner output
* sample_config_file fixture - Valid YAML config
* sample_invalid_config_file fixture - For validation tests
- Add comprehensive test_scan_service.py (15 tests)
* Test scan trigger with valid/invalid configs
* Test scan retrieval (found/not found cases)
* Test scan listing with pagination and filtering
* Test scan deletion with cascade cleanup
* Test scan status retrieval
* Test database mapping from JSON to models
* Test expected vs actual port flagging
* Test certificate and TLS data mapping
* Test full scan retrieval with all relationships
* All tests passing
Files Added:
- web/services/__init__.py
- web/services/scan_service.py (545 lines)
- web/utils/pagination.py (153 lines)
- web/utils/validators.py (245 lines)
- migrations/versions/002_add_scan_indexes.py
- tests/__init__.py
- tests/conftest.py (142 lines)
- tests/test_scan_service.py (374 lines)
Next Steps (Step 2):
- Implement scan API endpoints in web/api/scans.py
- Add authentication decorators
- Integrate ScanService with API routes
- Test API endpoints with integration tests
Phase 2 Step 1 Complete ✓
