This commit addresses multiple issues with schedule management and updates
documentation to reflect the transition from YAML-based to database-backed
configuration system.
**Documentation Updates:**
- Update DEPLOYMENT.md to remove all references to YAML config files
- Document that all configurations are now stored in SQLite database
- Update API examples to use config IDs instead of YAML filenames
- Remove configs directory from backup/restore procedures
- Update volume management section to reflect database-only storage
**Cron Expression Handling:**
- Add comprehensive documentation for APScheduler cron format conversion
- Document that from_crontab() accepts standard format (Sunday=0) and converts automatically
- Add validate_cron_expression() helper method with detailed error messages
- Include helpful hints for day-of-week field errors in validation
- Fix all deprecated datetime.utcnow() calls, replace with datetime.now(timezone.utc)
**Timezone-Aware DateTime Fixes:**
- Fix "can't subtract offset-naive and offset-aware datetimes" error
- Add timezone awareness to croniter.get_next() return values
- Make _get_relative_time() defensive to handle both naive and aware datetimes
- Ensure all datetime comparisons use timezone-aware objects
**Schedule Edit UI Fixes:**
- Fix JavaScript error "Cannot set properties of null (setting 'value')"
- Change reference from non-existent 'config-id' to correct 'config-file' element
- Add config_name field to schedule API responses for better UX
- Eagerly load Schedule.config relationship using joinedload()
- Fix AttributeError: use schedule.config.title instead of .name
- Display config title and ID in schedule edit form
**Technical Details:**
- app/web/services/schedule_service.py: 6 datetime.utcnow() fixes, validation enhancements
- app/web/services/scheduler_service.py: Documentation, validation, timezone fixes
- app/web/templates/schedule_edit.html: JavaScript element reference fix
- docs/DEPLOYMENT.md: Complete rewrite of config management sections
Fixes scheduling for Sunday at midnight (cron: 0 0 * * 0)
Fixes schedule edit page JavaScript errors
Improves user experience with config title display
The sites page previously showed total IP count which included duplicates
across multiple sites, leading to inflated numbers. Now displays unique
IP count as the primary metric with duplicate count shown when present.
- Add get_global_ip_stats() method to SiteService for unique/duplicate counts
- Update /api/sites?all=true endpoint to include IP statistics
- Update sites.html to display unique IPs with optional duplicate indicator
- Update API documentation with new response fields
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-Authored-By: Claude <noreply@anthropic.com>
- Add API endpoint GET /api/scans/by-ip/{ip_address} to retrieve
last 10 scans containing a specific IP
- Add ScanService.get_scans_by_ip() method with ScanIP join query
- Add search box to global navigation header
- Create dedicated search results page at /search/ip
- Update API documentation with new endpoint
- Fix config_id references to use integers instead of file paths
- Update scan delete response format to include scan_id field
- Add missing read_only field to Settings API responses
- Add missing template fields to Webhook responses
- Correct endpoint count from 80+ to 65+
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-Authored-By: Claude <noreply@anthropic.com>
Updated init_db.py to use config_id field after database migration,
fixing container startup error on new systems.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-Authored-By: Claude <noreply@anthropic.com>
Implemented complete scheduler integration with automatic schedule loading,
orphaned scan cleanup, and conversion to local timezone for better UX.
Backend Changes:
- Added load_schedules_on_startup() to load enabled schedules on app start
- Implemented cleanup_orphaned_scans() to handle crashed/interrupted scans
- Converted scheduler from UTC to local system timezone throughout
- Enhanced scheduler service with robust error handling and logging
Frontend Changes:
- Updated all schedule UI templates to display local time instead of UTC
- Improved timezone indicators and user messaging
- Removed confusing timezone converter (no longer needed)
- Updated quick templates and help text for local time
Bug Fixes:
- Fixed critical timezone bug causing cron expressions to run at wrong times
- Fixed orphaned scans stuck in 'running' status after system crashes
- Improved time display clarity across all schedule pages
All schedules now use local system time for intuitive scheduling.
Implemented dashboard visualizations and statistics API endpoints:
New Features:
- Stats API endpoints (/api/stats/scan-trend, /api/stats/summary)
- Chart.js trending chart showing 30-day scan activity
- Schedules widget displaying next 3 upcoming scheduled scans
- Enhanced Quick Actions with Manage Schedules button
Stats API (web/api/stats.py):
- scan-trend endpoint with configurable days (1-365)
- Summary endpoint for dashboard statistics
- Automatic date range filling with zeros for missing days
- Proper authentication and validation
Dashboard Enhancements (web/templates/dashboard.html):
- Chart.js line chart with dark theme styling
- Real-time schedules widget with human-readable time display
- Auto-refresh for schedules every 30 seconds
- Responsive 8-4 column layout for chart and schedules
Tests (tests/test_stats_api.py):
- 18 comprehensive test cases for stats API
- Coverage for date validation, authentication, edge cases
- Tests for empty data handling and date formatting
Progress: 64% complete (9/14 days)
Next: Step 6 - Scheduler Integration
Implemented full schedule management system with API endpoints and
user interface for creating, editing, and managing scheduled scans.
API Implementation:
- Implemented all 6 schedules API endpoints (list, get, create, update, delete, trigger)
- Added comprehensive error handling and validation
- Integrated with ScheduleService and SchedulerService
- Added manual trigger endpoint for on-demand execution
Schedule Management UI:
- Created schedules list page with stats cards and enable/disable toggles
- Built schedule creation form with cron expression builder and quick templates
- Implemented schedule edit page with execution history
- Added "Schedules" navigation link to main menu
- Real-time validation and human-readable cron descriptions
Config File Path Resolution:
- Fixed config file path handling to support relative filenames
- Updated validators.py to resolve relative paths to /app/configs/
- Modified schedule_service.py, scan_service.py, and scan_job.py for consistency
- Ensures UI can use simple filenames while backend uses absolute paths
Scheduler Integration:
- Completed scheduled scan execution in scheduler_service.py
- Added cron job management with APScheduler
- Implemented automatic schedule loading on startup
- Updated run times after each execution
Testing:
- Added comprehensive API integration tests (test_schedule_api.py)
- 22+ test cases covering all endpoints and workflows
Progress: Phase 3 Steps 1-4 complete (36% - 5/14 days)
Next: Step 5 - Enhanced Dashboard with Charts
Implement comprehensive schedule management service for automated scans:
New Files:
- web/services/schedule_service.py (470 lines)
* Complete CRUD operations for schedules
* Cron expression validation using croniter
* Next run time calculation
* Execution history tracking
* Human-readable relative time formatting
- tests/test_schedule_service.py (671 lines, 40+ tests)
* Create/get/list/update/delete schedule tests
* Cron validation and next run calculation tests
* Pagination and filtering tests
* Schedule history and serialization tests
Changes:
- requirements-web.txt: Add croniter==2.0.1 dependency
- docs/ai/PHASE3.md: Mark Step 1 complete, Step 2 next
Key Features:
- Validates cron expressions before saving
- Automatically calculates next execution time
- Preserves historical scans when schedules deleted
- Supports pagination and filtering by enabled status
- Provides relative time display (e.g., "in 2 hours")
Extracted inline CSS to external stylesheet and fixed white row bug
affecting dynamically created table rows across all scan views.
Changes:
- Created web/static/css/styles.css with extracted CSS from base.html
- Added CSS variables for consistent theming and maintainability
- Added Bootstrap 5 CSS variable overrides to fix table styling
- Integrated Chart.js 4.4.0 for future dashboard visualizations
- Added Bootstrap Icons for enhanced UI components
Template Updates:
- Updated base.html to use external CSS instead of inline styles
- Added Chart.js dark theme configuration
- Fixed white row bug in dashboard.html (added .scan-row class)
- Fixed white row bug in scans.html (added .scan-row class)
- Fixed white row bug in scan_detail.html port tables (added .scan-row class)
The white row bug was caused by Bootstrap 5's CSS variables overriding
custom styles. Fixed by setting --bs-table-bg and related variables.
Phase 3 Documentation:
- Added PHASE3.md with complete implementation plan (2204 lines)
- Includes 8 implementation steps, file changes, and success criteria
This completes Phase 3 Step 1 (Day 1 of 14).
Implement production-ready Docker deployment with comprehensive configuration
and documentation for SneakyScanner web application.
Changes:
- Update docker-compose-web.yml with production configuration
- Add scheduler environment variables (SCHEDULER_EXECUTORS, SCHEDULER_JOB_DEFAULTS_MAX_INSTANCES)
- Enable privileged mode and host networking for scanner operations
- Configure health check endpoint monitoring (30s interval, 40s start period)
- Set production defaults (FLASK_ENV=production, FLASK_DEBUG=false)
- Add SNEAKYSCANNER_ENCRYPTION_KEY support
- Create .env.example configuration template
- Flask, database, and security settings
- Scheduler configuration options
- Detailed comments with key generation examples
- Production deployment guidance
- Create comprehensive deployment documentation (docs/ai/DEPLOYMENT.md)
- Quick start guide and prerequisites
- Detailed configuration instructions
- Volume management and backup procedures
- Health monitoring and troubleshooting
- Security considerations and best practices
- Upgrade/rollback and backup/restore procedures
- Update PHASE2.md progress tracker
- Mark Step 6 as complete
- Update progress to 11/14 days (79%)
- Document deliverables and implementation details
Deployment is now production-ready with proper security defaults, health
monitoring, and comprehensive documentation for system administrators.
Implement comprehensive web UI with dark slate theme matching HTML reports:
Templates:
- Create base.html with navigation, dark theme (#0f172a background)
- Update dashboard.html with stats cards and recent scans table
- Update scans.html with pagination, filtering, and status badges
- Update scan_detail.html with comprehensive scan results display
- Update login.html to extend base template with centered design
Features:
- AJAX-powered dynamic data loading from API endpoints
- Auto-refresh for running scans (10-15 second intervals)
- Responsive Bootstrap 5 grid layout
- Color scheme matches report_mockup.html (slate dark theme)
- Status badges (success/danger/warning/info) with proper colors
- Modal dialogs for triggering scans
- Pagination with ellipsis for large result sets
- Delete confirmation dialogs
- Loading spinners for async operations
Bug Fixes:
- Fix scanner.py imports to use 'src.' prefix for module imports
- Fix scans.py to import validate_page_params from pagination module
All templates use consistent color palette:
- Background: #0f172a, Cards: #1e293b, Accent: #60a5fa
- Success: #065f46/#6ee7b7, Danger: #7f1d1d/#fca5a5
- Warning: #78350f/#fcd34d, Info: #1e3a8a/#93c5fd
Implemented all 5 scan management endpoints with comprehensive error
handling, logging, and integration tests.
## Changes
### API Endpoints (web/api/scans.py)
- POST /api/scans - Trigger new scan with config file validation
- GET /api/scans - List scans with pagination and status filtering
- GET /api/scans/<id> - Retrieve scan details with all relationships
- DELETE /api/scans/<id> - Delete scan and associated files
- GET /api/scans/<id>/status - Poll scan status for long-running scans
### Features
- Comprehensive error handling (400, 404, 500)
- Structured logging with appropriate levels
- Input validation via validators
- Consistent JSON error format
- SQLAlchemy error handling with graceful degradation
- HTTP status codes following REST conventions
### Testing (tests/test_scan_api.py)
- 24 integration tests covering all endpoints
- Empty/populated scan lists
- Pagination with multiple pages
- Status filtering
- Error scenarios (invalid input, not found, etc.)
- Complete workflow integration test
### Test Infrastructure (tests/conftest.py)
- Flask app fixture with test database
- Flask test client fixture
- Database session fixture compatible with app context
- Sample scan fixture for testing
### Documentation (docs/ai/PHASE2.md)
- Updated progress: 4/14 days complete (29%)
- Marked Step 2 as complete
- Added implementation details and testing results
## Implementation Notes
- All endpoints use ScanService for business logic separation
- Scan triggering returns immediately; client polls status endpoint
- Background job execution will be added in Step 3
- Authentication will be added in Step 4
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-Authored-By: Claude <noreply@anthropic.com>
