Commit Graph

28 Commits

Author SHA1 Message Date
41ba4c47b5 refactor to remove config_files in favor of db 2025-11-19 20:29:14 -06:00
e7dd207a62 Fix AlertRule initialization to use config_id instead of config_file
Updated init_db.py to use config_id field after database migration,
fixing container startup error on new systems.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude <noreply@anthropic.com>
2025-11-19 19:56:28 -06:00
4a4c33a10b doc changes 2025-11-19 10:42:49 -06:00
21254c3522 added webhooks and templates to alerting, email is next 2025-11-18 19:26:12 -06:00
28b32a2049 added webhooks, moved app name and verison to simple config file 2025-11-18 15:05:57 -06:00
1d076a467a added webhooks, moved app name and verison to simple config file 2025-11-18 15:05:39 -06:00
3c740268c4 updated API docs 2025-11-18 13:23:06 -06:00
131e1f5a61 adding phase 5 init framework, added deployment ease scripts 2025-11-18 13:10:53 -06:00
cd840cb8ca restructure of dirs, huge docs update 2025-11-17 16:29:14 -06:00
456e052389 updating docs 2025-11-17 15:50:15 -06:00
5f2314a532 phase 4 complete 2025-11-17 14:54:31 -06:00
6fe24c3907 adding Phase4 2025-11-17 12:05:11 -06:00
489284bde1 updating Phase3.md 2025-11-14 16:31:35 -06:00
9b88f42297 Phase 3 Step 6: Complete Scheduler Integration with Bug Fixes
Implemented complete scheduler integration with automatic schedule loading,
orphaned scan cleanup, and conversion to local timezone for better UX.

Backend Changes:
- Added load_schedules_on_startup() to load enabled schedules on app start
- Implemented cleanup_orphaned_scans() to handle crashed/interrupted scans
- Converted scheduler from UTC to local system timezone throughout
- Enhanced scheduler service with robust error handling and logging

Frontend Changes:
- Updated all schedule UI templates to display local time instead of UTC
- Improved timezone indicators and user messaging
- Removed confusing timezone converter (no longer needed)
- Updated quick templates and help text for local time

Bug Fixes:
- Fixed critical timezone bug causing cron expressions to run at wrong times
- Fixed orphaned scans stuck in 'running' status after system crashes
- Improved time display clarity across all schedule pages

All schedules now use local system time for intuitive scheduling.
2025-11-14 15:44:13 -06:00
effce42f21 Phase 3 Step 5: Enhanced Dashboard with Charts & Analytics
Implemented dashboard visualizations and statistics API endpoints:

New Features:
- Stats API endpoints (/api/stats/scan-trend, /api/stats/summary)
- Chart.js trending chart showing 30-day scan activity
- Schedules widget displaying next 3 upcoming scheduled scans
- Enhanced Quick Actions with Manage Schedules button

Stats API (web/api/stats.py):
- scan-trend endpoint with configurable days (1-365)
- Summary endpoint for dashboard statistics
- Automatic date range filling with zeros for missing days
- Proper authentication and validation

Dashboard Enhancements (web/templates/dashboard.html):
- Chart.js line chart with dark theme styling
- Real-time schedules widget with human-readable time display
- Auto-refresh for schedules every 30 seconds
- Responsive 8-4 column layout for chart and schedules

Tests (tests/test_stats_api.py):
- 18 comprehensive test cases for stats API
- Coverage for date validation, authentication, edge cases
- Tests for empty data handling and date formatting

Progress: 64% complete (9/14 days)
Next: Step 6 - Scheduler Integration
2025-11-14 14:50:20 -06:00
d68d9133c1 Phase 3 Steps 3 & 4: Complete Schedules API & Management UI
Implemented full schedule management system with API endpoints and
user interface for creating, editing, and managing scheduled scans.

API Implementation:
- Implemented all 6 schedules API endpoints (list, get, create, update, delete, trigger)
- Added comprehensive error handling and validation
- Integrated with ScheduleService and SchedulerService
- Added manual trigger endpoint for on-demand execution

Schedule Management UI:
- Created schedules list page with stats cards and enable/disable toggles
- Built schedule creation form with cron expression builder and quick templates
- Implemented schedule edit page with execution history
- Added "Schedules" navigation link to main menu
- Real-time validation and human-readable cron descriptions

Config File Path Resolution:
- Fixed config file path handling to support relative filenames
- Updated validators.py to resolve relative paths to /app/configs/
- Modified schedule_service.py, scan_service.py, and scan_job.py for consistency
- Ensures UI can use simple filenames while backend uses absolute paths

Scheduler Integration:
- Completed scheduled scan execution in scheduler_service.py
- Added cron job management with APScheduler
- Implemented automatic schedule loading on startup
- Updated run times after each execution

Testing:
- Added comprehensive API integration tests (test_schedule_api.py)
- 22+ test cases covering all endpoints and workflows

Progress: Phase 3 Steps 1-4 complete (36% - 5/14 days)
Next: Step 5 - Enhanced Dashboard with Charts
2025-11-14 14:33:48 -06:00
7969068c36 Phase 3 Step 2: Implement ScheduleService with cron support
Implement comprehensive schedule management service for automated scans:

New Files:
- web/services/schedule_service.py (470 lines)
  * Complete CRUD operations for schedules
  * Cron expression validation using croniter
  * Next run time calculation
  * Execution history tracking
  * Human-readable relative time formatting

- tests/test_schedule_service.py (671 lines, 40+ tests)
  * Create/get/list/update/delete schedule tests
  * Cron validation and next run calculation tests
  * Pagination and filtering tests
  * Schedule history and serialization tests

Changes:
- requirements-web.txt: Add croniter==2.0.1 dependency
- docs/ai/PHASE3.md: Mark Step 1 complete, Step 2 next

Key Features:
- Validates cron expressions before saving
- Automatically calculates next execution time
- Preserves historical scans when schedules deleted
- Supports pagination and filtering by enabled status
- Provides relative time display (e.g., "in 2 hours")
2025-11-14 13:41:49 -06:00
cbc3ff0f51 Phase 3 Step 1: Fix Styling Issues & CSS Refactor
Extracted inline CSS to external stylesheet and fixed white row bug
affecting dynamically created table rows across all scan views.

Changes:
- Created web/static/css/styles.css with extracted CSS from base.html
- Added CSS variables for consistent theming and maintainability
- Added Bootstrap 5 CSS variable overrides to fix table styling
- Integrated Chart.js 4.4.0 for future dashboard visualizations
- Added Bootstrap Icons for enhanced UI components

Template Updates:
- Updated base.html to use external CSS instead of inline styles
- Added Chart.js dark theme configuration
- Fixed white row bug in dashboard.html (added .scan-row class)
- Fixed white row bug in scans.html (added .scan-row class)
- Fixed white row bug in scan_detail.html port tables (added .scan-row class)

The white row bug was caused by Bootstrap 5's CSS variables overriding
custom styles. Fixed by setting --bs-table-bg and related variables.

Phase 3 Documentation:
- Added PHASE3.md with complete implementation plan (2204 lines)
- Includes 8 implementation steps, file changes, and success criteria

This completes Phase 3 Step 1 (Day 1 of 14).
2025-11-14 13:21:48 -06:00
4febdd23a5 Phase 2 Step 8: Testing & Documentation
Complete Phase 2 with comprehensive testing and documentation suite.

Testing:
- Reviewed existing test suite: 100 test functions, 1,825 lines of test code
- All tests passing across 6 test files
- Coverage: service layer, API endpoints, authentication, background jobs, error handling

Documentation Created:
- API_REFERENCE.md (17KB): Complete REST API documentation with examples
  * All 5 scan endpoints documented
  * Settings API reference
  * Authentication flow examples
  * Request/response examples with curl commands
  * Error handling and status codes

- PHASE2_COMPLETE.md (29KB): Comprehensive Phase 2 summary
  * All success criteria met (100%)
  * Deliverables by step (7 steps completed)
  * Code metrics: 34 files created, ~7,500+ lines
  * Technical implementation details
  * Lessons learned and key accomplishments

- MANUAL_TESTING.md (24KB): Manual testing checklist
  * 38 comprehensive tests across 10 categories
  * Step-by-step test procedures
  * Expected results for each test
  * Critical tests highlighted

- README.md: Major update with Phase 2 features
  * Quick start for web application
  * Complete web application section
  * API endpoints reference
  * Deployment instructions
  * Development section with testing guide

- ROADMAP.md: Updated with Phase 2 completion
  * Marked Phase 2 as COMPLETE 
  * Updated progress overview
  * Phase 2 success criteria achieved
  * Changelog updated

Phase 2 Final Metrics:
- Files Created: 34
- Lines of Code: ~7,500+
- Test Functions: 100 (all passing)
- Documentation: 2,000+ lines across 5 documents

Features Delivered:
- REST API (5 scan endpoints, 3 settings endpoints)
- Background job queue with APScheduler
- Session-based authentication
- Web UI (dashboard, scans, login, error pages)
- Comprehensive error handling and logging
- Docker deployment with healthcheck
- Complete documentation suite

Status: Phase 2 COMPLETE  - Production ready
Next: Phase 3 - Dashboard & Scheduling

🤖 Generated with SneakyScanner Development Tools
2025-11-14 12:38:58 -06:00
167ab803a6 Phase 2 Step 7: Implement Error Handling & Logging
Comprehensive error handling and logging system with production-ready
features for monitoring, debugging, and user experience.

Enhanced Logging System:
- Implemented RotatingFileHandler (10MB per file, 10 backups, 100MB total)
- Separate error log file for ERROR level messages with detailed tracebacks
- Structured logging with request IDs, timestamps, and module names
- RequestIDLogFilter for automatic request context injection
- Console logging in debug mode with simplified format

Request/Response Middleware:
- Request ID generation using UUID (8-character prefix for readability)
- Request timing with millisecond precision
- User authentication context in all logs
- Response duration tracking and headers (X-Request-ID, X-Request-Duration-Ms)
- Security headers: X-Content-Type-Options, X-Frame-Options, X-XSS-Protection

Database Error Handling:
- Enabled SQLite WAL mode for better concurrency with background jobs
- Busy timeout configuration (15 seconds) for lock handling
- Automatic rollback on request exceptions via teardown handler
- Dedicated SQLAlchemyError handler with explicit rollback
- Connection pooling with pre-ping validation

Comprehensive Error Handlers:
- Content negotiation: JSON responses for API, HTML for web requests
- Error handlers for 400, 401, 403, 404, 405, 500
- Database rollback in all error handlers
- Full exception logging with traceback for debugging

Custom Error Templates:
- Created web/templates/errors/ directory with 6 templates
- Dark theme matching application design (slate colors)
- User-friendly error messages with navigation
- Templates: 400, 401, 403, 404, 405, 500

Testing:
- Comprehensive test suite (320+ lines) in tests/test_error_handling.py
- Tests for JSON vs HTML error responses
- Request ID and duration header verification
- Security header validation
- Log rotation configuration tests
- Structured logging tests

Bug Fix:
- Fixed pagination bug in scans API endpoint
- Changed paginated_result.total_pages to paginated_result.pages
- Resolves AttributeError when listing scans

Files Added:
- tests/test_error_handling.py
- web/templates/errors/400.html
- web/templates/errors/401.html
- web/templates/errors/403.html
- web/templates/errors/404.html
- web/templates/errors/405.html
- web/templates/errors/500.html

Files Modified:
- web/app.py (logging, error handlers, request handlers, database config)
- web/api/scans.py (pagination bug fix)
- docs/ai/PHASE2.md (mark Step 7 complete, update progress to 86%)

Phase 2 Progress: 12/14 days complete (86%)
2025-11-14 12:19:07 -06:00
ebfefa9df3 Phase 2 Step 6: Docker & Deployment Configuration
Implement production-ready Docker deployment with comprehensive configuration
and documentation for SneakyScanner web application.

Changes:
- Update docker-compose-web.yml with production configuration
  - Add scheduler environment variables (SCHEDULER_EXECUTORS, SCHEDULER_JOB_DEFAULTS_MAX_INSTANCES)
  - Enable privileged mode and host networking for scanner operations
  - Configure health check endpoint monitoring (30s interval, 40s start period)
  - Set production defaults (FLASK_ENV=production, FLASK_DEBUG=false)
  - Add SNEAKYSCANNER_ENCRYPTION_KEY support

- Create .env.example configuration template
  - Flask, database, and security settings
  - Scheduler configuration options
  - Detailed comments with key generation examples
  - Production deployment guidance

- Create comprehensive deployment documentation (docs/ai/DEPLOYMENT.md)
  - Quick start guide and prerequisites
  - Detailed configuration instructions
  - Volume management and backup procedures
  - Health monitoring and troubleshooting
  - Security considerations and best practices
  - Upgrade/rollback and backup/restore procedures

- Update PHASE2.md progress tracker
  - Mark Step 6 as complete
  - Update progress to 11/14 days (79%)
  - Document deliverables and implementation details

Deployment is now production-ready with proper security defaults, health
monitoring, and comprehensive documentation for system administrators.
2025-11-14 12:01:21 -06:00
a64096ece3 Phase 2 Step 5: Implement Basic UI Templates
Implement comprehensive web UI with dark slate theme matching HTML reports:

Templates:
- Create base.html with navigation, dark theme (#0f172a background)
- Update dashboard.html with stats cards and recent scans table
- Update scans.html with pagination, filtering, and status badges
- Update scan_detail.html with comprehensive scan results display
- Update login.html to extend base template with centered design

Features:
- AJAX-powered dynamic data loading from API endpoints
- Auto-refresh for running scans (10-15 second intervals)
- Responsive Bootstrap 5 grid layout
- Color scheme matches report_mockup.html (slate dark theme)
- Status badges (success/danger/warning/info) with proper colors
- Modal dialogs for triggering scans
- Pagination with ellipsis for large result sets
- Delete confirmation dialogs
- Loading spinners for async operations

Bug Fixes:
- Fix scanner.py imports to use 'src.' prefix for module imports
- Fix scans.py to import validate_page_params from pagination module

All templates use consistent color palette:
- Background: #0f172a, Cards: #1e293b, Accent: #60a5fa
- Success: #065f46/#6ee7b7, Danger: #7f1d1d/#fca5a5
- Warning: #78350f/#fcd34d, Info: #1e3a8a/#93c5fd
2025-11-14 11:51:27 -06:00
0791c60f60 Fix duplicate line in PHASE2.md 2025-11-14 11:24:14 -06:00
ebe0a08b24 Update PHASE2.md: Mark Step 4 (Authentication System) as complete
Progress: 8/14 days (57%)

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude <noreply@anthropic.com>
2025-11-14 11:24:06 -06:00
ee0c5a2c3c Phase 2 Step 3: Implement Background Job Queue
Implemented APScheduler integration for background scan execution,
enabling async job processing without blocking HTTP requests.

## Changes

### Background Jobs (web/jobs/)
- scan_job.py - Execute scans in background threads
  - execute_scan() with isolated database sessions
  - Comprehensive error handling and logging
  - Scan status lifecycle tracking
  - Timing and error message storage

### Scheduler Service (web/services/scheduler_service.py)
- SchedulerService class for job management
- APScheduler BackgroundScheduler integration
- ThreadPoolExecutor for concurrent jobs (max 3 workers)
- queue_scan() - Immediate job execution
- Job monitoring: list_jobs(), get_job_status()
- Graceful shutdown handling

### Flask Integration (web/app.py)
- init_scheduler() function
- Scheduler initialization in app factory
- Stored scheduler in app context (app.scheduler)

### Database Schema (migration 003)
- Added scan timing fields:
  - started_at - Scan execution start time
  - completed_at - Scan execution completion time
  - error_message - Error details for failed scans

### Service Layer Updates (web/services/scan_service.py)
- trigger_scan() accepts scheduler parameter
- Queues background jobs after creating scan record
- get_scan_status() includes new timing and error fields
- _save_scan_to_db() sets completed_at timestamp

### API Updates (web/api/scans.py)
- POST /api/scans passes scheduler to trigger_scan()
- Scans now execute in background automatically

### Model Updates (web/models.py)
- Added started_at, completed_at, error_message to Scan model

### Testing (tests/test_background_jobs.py)
- 13 unit tests for background job execution
- Scheduler initialization and configuration tests
- Job queuing and status tracking tests
- Scan timing field tests
- Error handling and storage tests
- Integration test for full workflow (skipped by default)

## Features

- Async scan execution without blocking HTTP requests
- Concurrent scan support (configurable max workers)
- Isolated database sessions per background thread
- Scan lifecycle tracking: created → running → completed/failed
- Error messages captured and stored in database
- Job monitoring and management capabilities
- Graceful shutdown waits for running jobs

## Implementation Notes

- Scanner runs in subprocess from background thread
- Docker provides necessary privileges (--privileged, --network host)
- Each job gets isolated SQLAlchemy session (avoid locking)
- Job IDs follow pattern: scan_{scan_id}
- Background jobs survive across requests
- Failed jobs store error messages in database

## Documentation (docs/ai/PHASE2.md)
- Updated progress: 6/14 days complete (43%)
- Marked Step 3 as complete
- Added detailed implementation notes

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude <noreply@anthropic.com>
2025-11-14 09:24:00 -06:00
6c4905d6c1 Phase 2 Step 2: Implement Scan API Endpoints
Implemented all 5 scan management endpoints with comprehensive error
handling, logging, and integration tests.

## Changes

### API Endpoints (web/api/scans.py)
- POST /api/scans - Trigger new scan with config file validation
- GET /api/scans - List scans with pagination and status filtering
- GET /api/scans/<id> - Retrieve scan details with all relationships
- DELETE /api/scans/<id> - Delete scan and associated files
- GET /api/scans/<id>/status - Poll scan status for long-running scans

### Features
- Comprehensive error handling (400, 404, 500)
- Structured logging with appropriate levels
- Input validation via validators
- Consistent JSON error format
- SQLAlchemy error handling with graceful degradation
- HTTP status codes following REST conventions

### Testing (tests/test_scan_api.py)
- 24 integration tests covering all endpoints
- Empty/populated scan lists
- Pagination with multiple pages
- Status filtering
- Error scenarios (invalid input, not found, etc.)
- Complete workflow integration test

### Test Infrastructure (tests/conftest.py)
- Flask app fixture with test database
- Flask test client fixture
- Database session fixture compatible with app context
- Sample scan fixture for testing

### Documentation (docs/ai/PHASE2.md)
- Updated progress: 4/14 days complete (29%)
- Marked Step 2 as complete
- Added implementation details and testing results

## Implementation Notes

- All endpoints use ScanService for business logic separation
- Scan triggering returns immediately; client polls status endpoint
- Background job execution will be added in Step 3
- Authentication will be added in Step 4

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude <noreply@anthropic.com>
2025-11-14 09:13:30 -06:00
9255233a74 Add comprehensive Phase 2 implementation plan
Create detailed 14-day implementation roadmap for Flask Web App Core.

Plan includes:
- 21 new files to create (services, auth, templates, tests)
- 7 files to modify (app.py, scans.py, scanner.py, etc.)
- 8-step implementation guide with daily breakdown
- 7 key challenges with solutions
- Complete testing strategy (unit, integration, e2e)
- API documentation examples
- Success criteria (14 checkpoints)
- Design decisions documented

Key features planned:
- REST API for scan management (trigger, list, view, delete, status)
- Background job queue with APScheduler
- Authentication system with Flask-Login
- Scanner integration with database
- Basic UI templates (login, dashboard)
- Production Docker deployment

Ready to begin Phase 2 implementation.
2025-11-14 00:18:34 -06:00
b4360487b9 Reorganize documentation into docs/ folder structure
Move documentation files into organized folder structure:
- docs/ai/ - Documentation generated by development tools
- docs/human/ - Manual notes and testing documentation

Files moved:
- PHASE1_COMPLETE.md -> docs/ai/PHASE1_COMPLETE.md
- ROADMAP.md -> docs/ai/ROADMAP.md
2025-11-14 00:09:54 -06:00