` to prevent infinite resize loop **Files Modified:** - `web/templates/scan_detail.html` **Status:** ✅ Fixed and tested ## Vision & Goals SneakyScanner is evolving from a CLI-based network scanning tool into a comprehensive **Flask web application** for infrastructure monitoring and security auditing. The web application will provide: - **Centralized dashboard** for viewing scan history and trends - **Scheduled scanning** for continuous infrastructure monitoring - **Email notifications** for critical changes and certificate expirations - **Historical analysis** with charts, graphs, and comparison reports - **RESTful API** for integration and automation - **Simple deployment** using SQLite3 (single-user, self-hosted) The CLI scanner will evolve into an **API client**, maintaining backward compatibility while enabling web-based management and visualization. ## Target Users - **Infrastructure teams** monitoring on-premises networks - **Security professionals** performing periodic security audits - **DevOps engineers** tracking infrastructure drift - **Single users or small teams** (not enterprise multi-tenant) ## Technology Stack ### Backend - **Flask** 3.x - Lightweight Python web framework - **SQLAlchemy** 2.x - ORM for database abstraction - **SQLite3** - Embedded database (easy deployment, sufficient for single-user) - **APScheduler** 3.x - Background job scheduling for periodic scans - **Flask-Login** - Simple session-based authentication - **Flask-CORS** - API access control (optional for CLI API client) - **Marshmallow** - API serialization/deserialization ### Frontend - **Jinja2** - Server-side templating (already in use) - **Bootstrap 5** - Responsive UI framework with dark theme support - **Chart.js** 4.x - Lightweight charting library for trends - **DataTables.js** - Interactive sortable/filterable tables (Phase 6) - **Vanilla JavaScript** - Keep dependencies minimal ### Infrastructure - **Docker Compose** - Multi-container orchestration (Flask app + Scanner) - **Gunicorn** - WSGI server for production - **Nginx** - Reverse proxy (optional, for production deployments) ### Existing Components (Keep) - **Masscan** - Fast port discovery - **Nmap** - Service detection - **Playwright** - Screenshot capture - **sslyze** - SSL/TLS analysis ## Architecture Overview ``` ┌─────────────────────────────────────────────────────────────┐ │ Flask Web Application │ │ ┌──────────────┐ ┌──────────────┐ ┌──────────────────┐ │ │ │ Web UI │ │ REST API │ │ Scheduler │ │ │ │ (Dashboard) │ │ (JSON/CRUD) │ │ (APScheduler) │ │ │ └──────┬───────┘ └──────┬───────┘ └────────┬─────────┘ │ │ │ │ │ │ │ └─────────────────┴────────────────────┘ │ │ │ │ │ ┌────────▼────────┐ │ │ │ SQLAlchemy │ │ │ │ (ORM Layer) │ │ │ └────────┬────────┘ │ │ │ │ │ ┌────────▼────────┐ │ │ │ SQLite3 DB │ │ │ │ (scan history) │ │ │ └─────────────────┘ │ └───────────────────────────┬─────────────────────────────────┘ │ ┌──────────▼──────────┐ │ Scanner Engine │ │ (scanner.py) │ │ ┌────────────────┐ │ │ │ Masscan/Nmap │ │ │ │ Playwright │ │ │ │ sslyze │ │ │ └────────────────┘ │ └─────────────────────┘ │ ┌──────────▼──────────┐ │ CLI API Client │ │ (optional future) │ └─────────────────────┘ ``` ### Component Interaction 1. **Web UI** - User interacts with dashboard, triggers scans, views history 2. **REST API** - Handles requests from web UI and CLI client 3. **Scheduler (APScheduler)** - Triggers scans based on cron-like schedules 4. **SQLAlchemy ORM** - Abstracts database operations 5. **SQLite3 Database** - Stores scan results, schedules, settings, alerts 6. **Scanner Engine** - Performs actual network scanning (masscan, nmap, etc.) 7. **CLI API Client** - Future: thin client that calls Flask API ## Database Schema Design ### Core Tables #### `scans` Stores metadata about each scan execution. | Column | Type | Description | |--------|------|-------------| | `id` | INTEGER PRIMARY KEY | Unique scan ID | | `timestamp` | DATETIME | Scan start time (UTC) | | `duration` | FLOAT | Total scan duration (seconds) | | `status` | VARCHAR(20) | `running`, `completed`, `failed` | | `config_file` | TEXT | Path to YAML config used | | `title` | TEXT | Scan title from config | | `json_path` | TEXT | Path to JSON report | | `html_path` | TEXT | Path to HTML report | | `zip_path` | TEXT | Path to ZIP archive | | `screenshot_dir` | TEXT | Path to screenshot directory | | `created_at` | DATETIME | Record creation time | | `triggered_by` | VARCHAR(50) | `manual`, `scheduled`, `api` | | `schedule_id` | INTEGER | FK to schedules (if triggered by schedule) | #### `scan_sites` Logical grouping of IPs by site. | Column | Type | Description | |--------|------|-------------| | `id` | INTEGER PRIMARY KEY | Unique site record ID | | `scan_id` | INTEGER | FK to scans | | `site_name` | VARCHAR(255) | Site name from config | #### `scan_ips` IP addresses scanned in each scan. | Column | Type | Description | |--------|------|-------------| | `id` | INTEGER PRIMARY KEY | Unique IP record ID | | `scan_id` | INTEGER | FK to scans | | `site_id` | INTEGER | FK to scan_sites | | `ip_address` | VARCHAR(45) | IPv4 or IPv6 address | | `ping_expected` | BOOLEAN | Expected ping response | | `ping_actual` | BOOLEAN | Actual ping response | #### `scan_ports` Discovered TCP/UDP ports. | Column | Type | Description | |--------|------|-------------| | `id` | INTEGER PRIMARY KEY | Unique port record ID | | `scan_id` | INTEGER | FK to scans | | `ip_id` | INTEGER | FK to scan_ips | | `port` | INTEGER | Port number (1-65535) | | `protocol` | VARCHAR(10) | `tcp` or `udp` | | `expected` | BOOLEAN | Was this port expected? | | `state` | VARCHAR(20) | `open`, `closed`, `filtered` | #### `scan_services` Detected services on open ports. | Column | Type | Description | |--------|------|-------------| | `id` | INTEGER PRIMARY KEY | Unique service record ID | | `scan_id` | INTEGER | FK to scans | | `port_id` | INTEGER | FK to scan_ports | | `service_name` | VARCHAR(100) | Service name (e.g., `ssh`, `http`) | | `product` | VARCHAR(255) | Product name (e.g., `OpenSSH`) | | `version` | VARCHAR(100) | Version string | | `extrainfo` | TEXT | Additional nmap info | | `ostype` | VARCHAR(100) | OS type if detected | | `http_protocol` | VARCHAR(10) | `http` or `https` (if web service) | | `screenshot_path` | TEXT | Relative path to screenshot | #### `scan_certificates` SSL/TLS certificates discovered on HTTPS services. | Column | Type | Description | |--------|------|-------------| | `id` | INTEGER PRIMARY KEY | Unique certificate record ID | | `scan_id` | INTEGER | FK to scans | | `service_id` | INTEGER | FK to scan_services | | `subject` | TEXT | Certificate subject (CN) | | `issuer` | TEXT | Certificate issuer | | `serial_number` | TEXT | Serial number | | `not_valid_before` | DATETIME | Validity start date | | `not_valid_after` | DATETIME | Validity end date | | `days_until_expiry` | INTEGER | Days until expiration | | `sans` | TEXT | JSON array of SANs | | `is_self_signed` | BOOLEAN | Self-signed certificate flag | #### `scan_tls_versions` TLS version support and cipher suites. | Column | Type | Description | |--------|------|-------------| | `id` | INTEGER PRIMARY KEY | Unique TLS version record ID | | `scan_id` | INTEGER | FK to scans | | `certificate_id` | INTEGER | FK to scan_certificates | | `tls_version` | VARCHAR(20) | `TLS 1.0`, `TLS 1.1`, `TLS 1.2`, `TLS 1.3` | | `supported` | BOOLEAN | Is this version supported? | | `cipher_suites` | TEXT | JSON array of cipher suites | ### Scheduling & Notifications Tables #### `schedules` Scheduled scan configurations. | Column | Type | Description | |--------|------|-------------| | `id` | INTEGER PRIMARY KEY | Unique schedule ID | | `name` | VARCHAR(255) | Schedule name (e.g., "Daily prod scan") | | `config_file` | TEXT | Path to YAML config | | `cron_expression` | VARCHAR(100) | Cron-like schedule (e.g., `0 2 * * *`) | | `enabled` | BOOLEAN | Is schedule active? | | `last_run` | DATETIME | Last execution time | | `next_run` | DATETIME | Next scheduled execution | | `created_at` | DATETIME | Schedule creation time | | `updated_at` | DATETIME | Last modification time | #### `alerts` Alert history and notifications sent. | Column | Type | Description | |--------|------|-------------| | `id` | INTEGER PRIMARY KEY | Unique alert ID | | `scan_id` | INTEGER | FK to scans | | `alert_type` | VARCHAR(50) | `new_port`, `cert_expiry`, `service_change`, `ping_failed` | | `severity` | VARCHAR(20) | `info`, `warning`, `critical` | | `message` | TEXT | Human-readable alert message | | `ip_address` | VARCHAR(45) | Related IP (optional) | | `port` | INTEGER | Related port (optional) | | `email_sent` | BOOLEAN | Was email notification sent? | | `email_sent_at` | DATETIME | Email send timestamp | | `created_at` | DATETIME | Alert creation time | #### `alert_rules` User-defined alert rules. | Column | Type | Description | |--------|------|-------------| | `id` | INTEGER PRIMARY KEY | Unique rule ID | | `rule_type` | VARCHAR(50) | `unexpected_port`, `cert_expiry`, `service_down`, etc. | | `enabled` | BOOLEAN | Is rule active? | | `threshold` | INTEGER | Threshold value (e.g., days for cert expiry) | | `email_enabled` | BOOLEAN | Send email for this rule? | | `created_at` | DATETIME | Rule creation time | ### Settings Table #### `settings` Application configuration key-value store. | Column | Type | Description | |--------|------|-------------| | `id` | INTEGER PRIMARY KEY | Unique setting ID | | `key` | VARCHAR(255) UNIQUE | Setting key (e.g., `smtp_server`) | | `value` | TEXT | Setting value (JSON for complex values) | | `updated_at` | DATETIME | Last modification time | **Example Settings:** - `smtp_server` - SMTP server hostname - `smtp_port` - SMTP port (587, 465, 25) - `smtp_username` - SMTP username - `smtp_password` - SMTP password (encrypted) - `smtp_from_email` - From email address - `smtp_to_emails` - JSON array of recipient emails - `app_password` - Single-user password hash (bcrypt) - `retention_days` - How long to keep old scans (0 = forever) ## API Design ### REST API Endpoints All API endpoints return JSON and follow RESTful conventions. #### Scans | Method | Endpoint | Description | Request Body | Response | |--------|----------|-------------|--------------|----------| | `GET` | `/api/scans` | List all scans (paginated) | - | `{ "scans": [...], "total": N, "page": 1 }` | | `GET` | `/api/scans/{id}` | Get scan details | - | `{ "scan": {...} }` | | `POST` | `/api/scans` | Trigger new scan | `{ "config_file": "path" }` | `{ "scan_id": N, "status": "running" }` | | `DELETE` | `/api/scans/{id}` | Delete scan and files | - | `{ "status": "deleted" }` | | `GET` | `/api/scans/{id}/status` | Get scan status | - | `{ "status": "running", "progress": "45%" }` | | `GET` | `/api/scans/{id1}/compare/{id2}` | Compare two scans | - | `{ "diff": {...} }` | #### Schedules | Method | Endpoint | Description | Request Body | Response | |--------|----------|-------------|--------------|----------| | `GET` | `/api/schedules` | List all schedules | - | `{ "schedules": [...] }` | | `GET` | `/api/schedules/{id}` | Get schedule details | - | `{ "schedule": {...} }` | | `POST` | `/api/schedules` | Create new schedule | `{ "name": "...", "config_file": "...", "cron_expression": "..." }` | `{ "schedule_id": N }` | | `PUT` | `/api/schedules/{id}` | Update schedule | `{ "enabled": true, "cron_expression": "..." }` | `{ "status": "updated" }` | | `DELETE` | `/api/schedules/{id}` | Delete schedule | - | `{ "status": "deleted" }` | | `POST` | `/api/schedules/{id}/trigger` | Manually trigger scheduled scan | - | `{ "scan_id": N }` | #### Alerts | Method | Endpoint | Description | Request Body | Response | |--------|----------|-------------|--------------|----------| | `GET` | `/api/alerts` | List recent alerts | - | `{ "alerts": [...] }` | | `GET` | `/api/alerts/rules` | List alert rules | - | `{ "rules": [...] }` | | `POST` | `/api/alerts/rules` | Create alert rule | `{ "rule_type": "...", "threshold": N }` | `{ "rule_id": N }` | | `PUT` | `/api/alerts/rules/{id}` | Update alert rule | `{ "enabled": false }` | `{ "status": "updated" }` | | `DELETE` | `/api/alerts/rules/{id}` | Delete alert rule | - | `{ "status": "deleted" }` | #### Settings | Method | Endpoint | Description | Request Body | Response | |--------|----------|-------------|--------------|----------| | `GET` | `/api/settings` | Get all settings (sanitized) | - | `{ "settings": {...} }` | | `PUT` | `/api/settings` | Update settings | `{ "smtp_server": "...", ... }` | `{ "status": "updated" }` | | `POST` | `/api/settings/test-email` | Test email configuration | - | `{ "status": "sent" }` | #### Statistics & Trends | Method | Endpoint | Description | Request Body | Response | |--------|----------|-------------|--------------|----------| | `GET` | `/api/stats/summary` | Dashboard summary stats | - | `{ "total_scans": N, "last_scan": "...", ... }` | | `GET` | `/api/stats/trends` | Trend data for charts | `?days=30&metric=port_count` | `{ "data": [...] }` | | `GET` | `/api/stats/certificates` | Certificate expiry overview | - | `{ "expiring_soon": [...], "expired": [...] }` | ### Authentication **Phase 2-3:** Simple session-based authentication (single-user) - Login endpoint: `POST /api/auth/login` (username/password) - Logout endpoint: `POST /api/auth/logout` - Session cookies with Flask-Login - Password stored as bcrypt hash in settings table **Phase 5:** API token authentication for CLI client - Generate API token: `POST /api/auth/token` - Revoke token: `DELETE /api/auth/token` - CLI sends token in `Authorization: Bearer ` header ## Phased Roadmap ### Phase 1: Foundation ✅ COMPLETE **Completed:** 2025-11-13 **Priority: CRITICAL** - Database and settings infrastructure **Goals:** - ✅ Establish database schema - ✅ Create settings system - ✅ Set up Flask project structure **Tasks:** 1. ✅ Create SQLite database schema (use Alembic for migrations) 2. ✅ Implement SQLAlchemy models for all tables (11 models) 3. ✅ Create database initialization script (`init_db.py`) 4. ✅ Implement settings system: - ✅ Settings model with get/set methods - ✅ Default settings initialization - ✅ Encrypted storage for passwords (cryptography library + bcrypt) - ✅ PasswordManager for bcrypt password hashing 5. ✅ Set up Flask project structure: ``` SneakyScanner/ ├── src/ │ ├── scanner.py (existing) │ ├── screenshot_capture.py (existing) │ └── report_generator.py (existing) ├── web/ ✅ CREATED │ ├── __init__.py ✅ │ ├── app.py (Flask app factory) ✅ │ ├── models.py (SQLAlchemy models) ✅ │ ├── api/ (API blueprints) ✅ │ │ ├── __init__.py ✅ │ │ ├── scans.py ✅ │ │ ├── schedules.py ✅ │ │ ├── alerts.py ✅ │ │ └── settings.py ✅ (Fully functional!) │ ├── templates/ (Jinja2 templates) ✅ │ ├── static/ (CSS, JS, images) ✅ │ │ ├── css/ ✅ │ │ ├── js/ ✅ │ │ └── images/ ✅ │ └── utils/ (helpers, decorators) ✅ │ ├── __init__.py ✅ │ └── settings.py ✅ ├── migrations/ (Alembic migrations) ✅ │ ├── env.py ✅ │ ├── script.py.mako ✅ │ └── versions/ ✅ │ └── 001_initial_schema.py ✅ ├── alembic.ini ✅ ├── configs/ (existing) ├── output/ (existing) └── templates/ (existing - for reports) ``` 6. ✅ Create `requirements-web.txt` for Flask dependencies 7. ✅ Update Dockerfile to support Flask app 8. ✅ Create `docker-compose-web.yml` for web deployment 9. ✅ Create `validate_phase1.py` for verification **Deliverables:** - ✅ Working database with schema (SQLite3 + Alembic migrations) - ✅ Settings CRUD functionality (with encryption for sensitive values) - ✅ Flask app skeleton with functional Settings API - ✅ Database migration system (Alembic) - ✅ API blueprint stubs (scans, schedules, alerts, settings) - ✅ Docker support (Dockerfile updated, docker-compose-web.yml created) **Testing:** - ✅ Database creates successfully (`init_db.py` works) - ✅ Settings can be stored/retrieved (encryption working) - ✅ Flask app starts without errors (`python3 -m web.app` works) - ✅ All validation checks pass (`validate_phase1.py` ✓) - ✅ All 11 database models defined correctly - ✅ Settings API endpoints functional and tested **Documentation:** - ✅ `PHASE1_COMPLETE.md` - Complete Phase 1 summary with API reference and deployment guide - ✅ `validate_phase1.py` - Automated validation script --- ### Phase 2: Flask Web App Core ✅ COMPLETE **Completed:** 2025-11-14 **Duration:** 14 days (Weeks 3-4) **Priority:** HIGH **Goals:** - ✅ Implement REST API for scans - ✅ Add background job queue - ✅ Create simple authentication - ✅ Integrate scanner with database **Deliverables Completed:** - ✅ **REST API** - 5 scan endpoints (trigger, list, get, status, delete) + 3 settings endpoints - ✅ **Background Jobs** - APScheduler with ThreadPoolExecutor (up to 3 concurrent scans) - ✅ **Authentication** - Flask-Login session-based auth (login, logout, decorators) - ✅ **Database Integration** - Complete scan results saved to normalized schema - ✅ **Web UI** - Dashboard, scans list/detail, login, error templates - ✅ **Error Handling** - Content negotiation (JSON/HTML), custom error pages, request IDs - ✅ **Logging** - Rotating file handlers (10MB max), request timing, structured logs - ✅ **Docker Deployment** - Production-ready docker-compose with healthcheck - ✅ **Testing** - 100 test functions, 1,825 lines of test code, all passing - ✅ **Documentation** - API_REFERENCE.md, DEPLOYMENT.md, PHASE2_COMPLETE.md **Files Created:** 34 files, ~7,500+ lines of code **Key Features:** - Scans execute in background without blocking HTTP requests - Status tracking: `running` → `completed`/`failed` - Pagination and filtering for scan lists - Complete scan details with all relationships (sites, IPs, ports, services, certs, TLS) - Secure password hashing with bcrypt - SQLite WAL mode for better concurrency - Request IDs for debugging and correlation - Comprehensive error handling for all HTTP status codes **Testing Results:** - ✅ All API endpoints tested (24 integration tests) - ✅ Service layer tested (15 unit tests) - ✅ Authentication tested (30+ tests) - ✅ Background jobs tested (13 tests) - ✅ Error handling tested (18+ tests) - ✅ All 100 tests passing **Documentation:** - [PHASE2_COMPLETE.md](PHASE2_COMPLETE.md) - Complete Phase 2 summary - [API_REFERENCE.md](API_REFERENCE.md) - Comprehensive API documentation - [DEPLOYMENT.md](DEPLOYMENT.md) - Production deployment guide - README.md updated with Phase 2 features --- ### Phase 3: Dashboard & Scheduling (Weeks 5-6) **Priority: HIGH** - User's top requested features **Goals:** - Build web dashboard with scan history - Implement trend charts - Add scheduled scan management UI - Real-time scan progress **Tasks:** 1. Dashboard implementation: - Summary cards (total scans, last scan, total IPs, open ports) - Recent scans table (clickable to view details) - Security warnings section (expiring certs, weak TLS) - Drift alerts section (unexpected ports, new services) 2. Scan detail page: - Display full scan results (sites, IPs, services) - Embedded screenshots or links - Download buttons (JSON, HTML, ZIP) - Delete scan button 3. Trend charts with Chart.js: - Port count over time (line chart) - Service distribution (bar chart) - Certificate expiration timeline (timeline chart) - Charts update based on date range selector 4. Scheduled scans UI: - List all schedules (table with enable/disable toggle) - Create schedule form (name, config file, cron expression) - Edit schedule form - Delete schedule button - Next run time display 5. APScheduler integration: - Schedule manager class - Load schedules from DB on app start - Add/remove jobs dynamically - Cron expression validation - Update `next_run` and `last_run` in DB 6. Manual scan trigger: - "Run Scan Now" button on dashboard - Config file selector - Show real-time progress (polling or WebSocket) 7. Navigation menu: - Dashboard - Scans - Schedules - Alerts (placeholder) - Settings (placeholder) **Deliverables:** - Functional dashboard with charts - Scan history browser - Scheduled scan management UI - Background scheduler running scans **Testing:** - Dashboard displays accurate summary stats - Charts render correctly with real data - Scheduled scans execute at specified times - Manual scan trigger works - Real-time progress updates --- ### Phase 4: Email & Comparisons (Weeks 7-8) **Priority: MEDIUM** - Monitoring and analysis features **Goals:** - Implement email notification system - Create scan comparison reports - Add alert rule configuration **Tasks:** 1. Email notification system: - SMTP integration (using `smtplib` or `Flask-Mail`) - Email template for alerts (Jinja2 HTML email) - Settings page for SMTP configuration - Test email button 2. Alert rule engine: - Define alert rule types: - Unexpected TCP/UDP port opened - Expected port missing - Certificate expiring in < N days - Certificate expired - Service version changed - Ping failed (host down) - Weak TLS version detected (1.0/1.1) - Alert rule creation UI - Alert rule evaluation after each scan - Store alerts in `alerts` table 3. Alert history page: - List all alerts (filterable by type, severity, date) - Mark alerts as "acknowledged" - Alert detail view 4. Scan comparison: - Compare two scans API endpoint - Comparison algorithm: - New ports/services - Removed ports/services - Service version changes - Certificate changes - TLS configuration changes - Comparison report UI: - Side-by-side view - Diff highlighting (green = added, red = removed, yellow = changed) - "Compare" button on scan list (select 2 scans) 5. Email notification triggers: - Send email when alert rule triggered - Daily digest email (summary of all alerts) - Weekly scan summary email 6. Settings page: - SMTP configuration form - Alert rule management - Email recipient list - Test email button **Deliverables:** - Working email notification system - Alert rules with email triggers - Scan comparison functionality - Settings UI for configuration **Testing:** - Email sends successfully with SMTP config - Alert rules trigger correctly - Comparison shows accurate diffs - Settings persist correctly --- ### Phase 5: CLI as API Client (Week 9) **Priority: MEDIUM** - Backward compatibility and automation **Goals:** - Refactor CLI to optionally call Flask API - Maintain standalone mode for testing - API token authentication **Tasks:** 1. API client mode for `scanner.py`: - Add `--api-mode` flag - Add `--api-url` and `--api-token` arguments - When `--api-mode` enabled: - Send scan request to `POST /api/scans` - Poll `GET /api/scans/{id}/status` for progress - Download results when complete - When `--api-mode` disabled (default): - Run standalone as currently works 2. API token generation: - UI to generate API tokens (settings page) - Store tokens in `api_tokens` table (hashed) - API token authentication middleware - Token expiration and revocation 3. CLI documentation: - Update README.md with API mode usage - Example commands for API mode - Token generation instructions 4. Benefits of API mode: - Scans stored centrally in database - No need to mount volumes for output - Scheduled scans managed through web UI - Scan history accessible via web dashboard **Deliverables:** - CLI with `--api-mode` flag - API token system - Updated documentation **Testing:** - CLI can trigger scan via API - API token authentication works - Standalone mode still functional - Token revocation works --- ### Phase 6: Advanced Features (Weeks 10+) **Priority: LOW** - Nice-to-have enhancements **Goals:** - Enhanced interactive reports - Vulnerability detection - PDF export - Timeline view **Tasks:** 1. Enhanced HTML reports: - Sortable/filterable tables (DataTables.js) - Inline screenshot thumbnails (lightbox on click) - Export to PDF button (WeasyPrint or pdfkit) - Print-friendly CSS 2. Vulnerability detection: - Integrate with CVE databases (NVD API or Vulners API) - Match detected services/versions to known CVEs - Display CVE list with severity scores (CVSS) - CVE detail page with description, remediation - Alert rule for new critical CVEs 3. Timeline view: - Visual timeline of all scans - Filter by site or IP - Click on timeline event to view scan - Annotations for important events (cert renewals, config changes) 4. Advanced charts: - Heatmap of port activity - Service version tracking over time - Certificate expiration forecast - Top 10 services pie chart 5. Export/Import: - Export scan data to CSV - Import scan configs from CSV - Bulk schedule creation 6. Additional integrations: - Slack notifications (in addition to email) - Webhook support (POST to custom URL on events) - Prometheus metrics export **Deliverables:** - Interactive sortable tables - CVE integration - PDF export - Timeline view - Additional integrations **Testing:** - DataTables work with large datasets - CVE data fetches correctly - PDF exports render properly - Timeline view performs well with many scans --- ## Migration Strategy ### From Current CLI to Web App **Current State:** - CLI tool (`scanner.py`) runs standalone - Outputs JSON, HTML, ZIP files - No database, no web UI **Migration Path:** #### Step 1: Add Database Layer (Phase 1) - Database runs alongside CLI - CLI can optionally save to DB (flag: `--save-to-db`) - No breaking changes #### Step 2: Launch Web App (Phase 2-3) - Web app reads from DB - Users can trigger scans via web UI - CLI still works standalone #### Step 3: Transition Period (Phase 4-5) - Users gradually adopt web UI - CLI used for scripting/automation - Both modes fully supported #### Step 4: API Client Mode (Phase 5) - CLI becomes thin API client - All scans stored in central DB - Standalone mode remains for testing #### Step 5: Full Web App (Phase 6+) - Primary interface is web UI - CLI optional for power users ### Backward Compatibility **Maintained:** - Existing YAML config format - JSON/HTML/ZIP output files - Screenshot capture - Docker deployment **Deprecated (eventually):** - Standalone CLI mode (Phase 6+) - Direct file output (replaced by DB + API) ## Prioritized Feature List ### Must-Have (Phases 1-3) 1. **Database foundation** (SQLite3 + SQLAlchemy) 2. **Flask web app core** (REST API, authentication) 3. **Dashboard with scan history** (list, detail, delete) 4. **Trend charts** (Chart.js - port counts, service distribution) 5. **Scheduled scans** (APScheduler + cron expressions) 6. **Manual scan trigger** (web UI button) ### Should-Have (Phase 4) 7. **Email notifications** (SMTP integration) 8. **Alert rules** (cert expiry, unexpected ports, etc.) 9. **Scan comparison reports** (diff view) 10. **Settings UI** (SMTP, alerts, retention) ### Nice-to-Have (Phases 5-6) 11. **CLI as API client** (token auth, backward compat) 12. **Sortable/filterable tables** (DataTables.js) 13. **PDF export** (WeasyPrint) 14. **Vulnerability detection** (CVE integration) 15. **Timeline view** (visual scan history) 16. **Embedded screenshot thumbnails** (lightbox) ### Future/Deferred 17. **Multi-user support** (if requirements change) 18. **Slack/webhook integrations** 19. **Prometheus metrics** 20. **Mobile-responsive dashboard** (Bootstrap handles basics) ## Development Workflow ### Iteration Cycle 1. **Plan** - Define features for phase 2. **Implement** - Code backend + frontend 3. **Test** - Unit tests + manual testing 4. **Deploy** - Update Docker Compose 5. **Document** - Update README.md, ROADMAP.md 6. **Review** - Get user feedback 7. **Iterate** - Adjust priorities based on feedback ### Git Workflow - **main branch** - Stable releases - **develop branch** - Active development - **feature branches** - Individual features (`feature/dashboard`, `feature/scheduler`) - **Pull requests** - Review before merge ### Testing Strategy - **Unit tests** - pytest for models, API endpoints - **Integration tests** - Full scan → DB → API workflow - **Manual testing** - UI/UX testing in browser - **Performance tests** - Large scans, database queries ### Documentation - **README.md** - User-facing documentation (updated each phase) - **ROADMAP.md** - This file (updated as priorities shift) - **CLAUDE.md** - Developer documentation (architecture, code references) - **API.md** - API documentation (OpenAPI/Swagger in Phase 4) ## Success Metrics ### Phase 1 Success ✅ ACHIEVED - [x] Database creates successfully with all 11 tables - [x] Settings can be stored/retrieved with encryption - [x] Flask app starts without errors - [x] API blueprints load correctly - [x] All Python modules have valid syntax - [x] Docker deployment configured ### Phase 2 Success ✅ ACHIEVED - [x] Database stores scan results correctly - [x] REST API functional with all endpoints - [x] Background scans execute asynchronously - [x] Authentication protects all routes - [x] Web UI is intuitive and responsive - [x] 100 tests passing with comprehensive coverage - [x] Docker deployment production-ready ### Phase 3 Success ✅ ACHIEVED - [x] Dashboard displays scans and trends with charts - [x] Scheduled scans execute automatically - [x] Historical trend charts show scan history - [x] Real-time progress updates for running scans ### Phase 4 Success ✅ ACHIEVED - [x] Users can create configs from CIDR ranges via web UI - [x] YAML editor with syntax highlighting works correctly - [x] Config management UI provides list/view/edit/download/delete operations - [x] Direct YAML upload works for advanced users - [x] Configs immediately usable in scan triggers and schedules - [x] Delete protection prevents removal of configs used by schedules - [x] All tests passing (25+ unit and integration tests) ### Phase 5 Success (Email & Comparisons) - [ ] Email notifications sent for critical alerts - [ ] Comparison reports show meaningful diffs - [ ] Settings UI allows SMTP configuration without editing files ### Phase 6 Success (CLI as API Client) - [ ] CLI can trigger scans via API - [ ] API tokens work for authentication - [ ] Standalone CLI mode still functional ### Phase 6+ Success - [ ] CVE integration provides actionable vulnerability data - [ ] Timeline view helps track infrastructure changes - [ ] PDF exports are shareable and professional ## Open Questions ### Technical Decisions - **Flask vs. FastAPI?** - Sticking with Flask for simplicity, but FastAPI offers async and auto-docs - **APScheduler vs. Celery?** - APScheduler for simplicity (no Redis/RabbitMQ needed), but Celery scales better - **Bootstrap vs. Tailwind?** - Bootstrap for speed (pre-built components), Tailwind for customization - **Chart.js vs. Plotly?** - Chart.js for lightweight, Plotly for interactive (decide in Phase 3) ### Product Questions - **Should we support multiple configs per schedule?** - Start with 1:1, add later if needed - **How many scans to keep in DB?** - Add retention setting (default: keep all) - **Support for multi-tenancy?** - Not in scope (single-user), but database schema allows future expansion - **Mobile app?** - Out of scope, but responsive web UI covers basics ## Resources & References ### Documentation - [Flask Documentation](https://flask.palletsprojects.com/) - [SQLAlchemy ORM](https://docs.sqlalchemy.org/) - [APScheduler](https://apscheduler.readthedocs.io/) - [Chart.js](https://www.chartjs.org/docs/) - [Bootstrap 5](https://getbootstrap.com/docs/5.3/) ### Tutorials - [Flask Mega-Tutorial](https://blog.miguelgrinberg.com/post/the-flask-mega-tutorial-part-i-hello-world) - [SQLAlchemy Tutorial](https://docs.sqlalchemy.org/en/20/tutorial/) - [APScheduler with Flask](https://github.com/viniciuschiele/flask-apscheduler) ### Similar Projects (Inspiration) - [OpenVAS](https://www.openvas.org/) - Vulnerability scanner with web UI - [Nessus](https://www.tenable.com/products/nessus) - Commercial scanner (inspiration for UI/UX) - [OWASP ZAP](https://www.zaproxy.org/) - Web app scanner (comparison reports, alerts) ## Changelog | Date | Version | Changes | |------|---------|---------| | 2025-11-14 | 1.0 | Initial roadmap created based on user requirements | | 2025-11-13 | 1.1 | **Phase 1 COMPLETE** - Database schema, SQLAlchemy models, Flask app structure, settings system with encryption, Alembic migrations, API blueprints, Docker support, validation script | | 2025-11-14 | 1.2 | **Phase 2 COMPLETE** - REST API (5 scan endpoints, 3 settings endpoints), background jobs (APScheduler), authentication (Flask-Login), web UI (dashboard, scans, login, errors), error handling (content negotiation, request IDs, logging), 100 tests passing, comprehensive documentation (API_REFERENCE.md, DEPLOYMENT.md, PHASE2_COMPLETE.md) | | 2025-11-17 | 1.3 | **Bug Fix** - Fixed Chart.js infinite canvas growth issue in scan detail page (duplicate initialization, missing chart.destroy(), missing fixed-height container) | | 2025-11-17 | 1.4 | **Phase 4 COMPLETE** - Config Creator with CIDR-based creation, YAML editor (CodeMirror), config management UI (list/edit/delete), REST API (7 endpoints), Docker volume permissions fix, comprehensive testing and documentation | --- **Last Updated:** 2025-11-17 **Next Review:** Before Phase 5 kickoff (Email & Comparisons)