From e29c839d80e7a1b47de27d4e89cb9a31e318b93b Mon Sep 17 00:00:00 2001 From: Phillip Tarrant Date: Fri, 14 Nov 2025 02:40:49 +0000 Subject: [PATCH] adding Roadmap --- ROADMAP.md | 851 +++++++++++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 851 insertions(+) create mode 100644 ROADMAP.md diff --git a/ROADMAP.md b/ROADMAP.md new file mode 100644 index 0000000..462a783 --- /dev/null +++ b/ROADMAP.md @@ -0,0 +1,851 @@ +# SneakyScanner Roadmap + +## Vision & Goals + +SneakyScanner is evolving from a CLI-based network scanning tool into a comprehensive **Flask web application** for infrastructure monitoring and security auditing. The web application will provide: + +- **Centralized dashboard** for viewing scan history and trends +- **Scheduled scanning** for continuous infrastructure monitoring +- **Email notifications** for critical changes and certificate expirations +- **Historical analysis** with charts, graphs, and comparison reports +- **RESTful API** for integration and automation +- **Simple deployment** using SQLite3 (single-user, self-hosted) + +The CLI scanner will evolve into an **API client**, maintaining backward compatibility while enabling web-based management and visualization. + +## Target Users + +- **Infrastructure teams** monitoring on-premises networks +- **Security professionals** performing periodic security audits +- **DevOps engineers** tracking infrastructure drift +- **Single users or small teams** (not enterprise multi-tenant) + +## Technology Stack + +### Backend +- **Flask** 3.x - Lightweight Python web framework +- **SQLAlchemy** 2.x - ORM for database abstraction +- **SQLite3** - Embedded database (easy deployment, sufficient for single-user) +- **APScheduler** 3.x - Background job scheduling for periodic scans +- **Flask-Login** - Simple session-based authentication +- **Flask-CORS** - API access control (optional for CLI API client) +- **Marshmallow** - API serialization/deserialization + +### Frontend +- **Jinja2** - Server-side templating (already in use) +- **Bootstrap 5** - Responsive UI framework with dark theme support +- **Chart.js** 4.x - Lightweight charting library for trends +- **DataTables.js** - Interactive sortable/filterable tables (Phase 6) +- **Vanilla JavaScript** - Keep dependencies minimal + +### Infrastructure +- **Docker Compose** - Multi-container orchestration (Flask app + Scanner) +- **Gunicorn** - WSGI server for production +- **Nginx** - Reverse proxy (optional, for production deployments) + +### Existing Components (Keep) +- **Masscan** - Fast port discovery +- **Nmap** - Service detection +- **Playwright** - Screenshot capture +- **sslyze** - SSL/TLS analysis + +## Architecture Overview + +``` +┌─────────────────────────────────────────────────────────────┐ +│ Flask Web Application │ +│ ┌──────────────┐ ┌──────────────┐ ┌──────────────────┐ │ +│ │ Web UI │ │ REST API │ │ Scheduler │ │ +│ │ (Dashboard) │ │ (JSON/CRUD) │ │ (APScheduler) │ │ +│ └──────┬───────┘ └──────┬───────┘ └────────┬─────────┘ │ +│ │ │ │ │ +│ └─────────────────┴────────────────────┘ │ +│ │ │ +│ ┌────────▼────────┐ │ +│ │ SQLAlchemy │ │ +│ │ (ORM Layer) │ │ +│ └────────┬────────┘ │ +│ │ │ +│ ┌────────▼────────┐ │ +│ │ SQLite3 DB │ │ +│ │ (scan history) │ │ +│ └─────────────────┘ │ +└───────────────────────────┬─────────────────────────────────┘ + │ + ┌──────────▼──────────┐ + │ Scanner Engine │ + │ (scanner.py) │ + │ ┌────────────────┐ │ + │ │ Masscan/Nmap │ │ + │ │ Playwright │ │ + │ │ sslyze │ │ + │ └────────────────┘ │ + └─────────────────────┘ + │ + ┌──────────▼──────────┐ + │ CLI API Client │ + │ (optional future) │ + └─────────────────────┘ +``` + +### Component Interaction + +1. **Web UI** - User interacts with dashboard, triggers scans, views history +2. **REST API** - Handles requests from web UI and CLI client +3. **Scheduler (APScheduler)** - Triggers scans based on cron-like schedules +4. **SQLAlchemy ORM** - Abstracts database operations +5. **SQLite3 Database** - Stores scan results, schedules, settings, alerts +6. **Scanner Engine** - Performs actual network scanning (masscan, nmap, etc.) +7. **CLI API Client** - Future: thin client that calls Flask API + +## Database Schema Design + +### Core Tables + +#### `scans` +Stores metadata about each scan execution. + +| Column | Type | Description | +|--------|------|-------------| +| `id` | INTEGER PRIMARY KEY | Unique scan ID | +| `timestamp` | DATETIME | Scan start time (UTC) | +| `duration` | FLOAT | Total scan duration (seconds) | +| `status` | VARCHAR(20) | `running`, `completed`, `failed` | +| `config_file` | TEXT | Path to YAML config used | +| `title` | TEXT | Scan title from config | +| `json_path` | TEXT | Path to JSON report | +| `html_path` | TEXT | Path to HTML report | +| `zip_path` | TEXT | Path to ZIP archive | +| `screenshot_dir` | TEXT | Path to screenshot directory | +| `created_at` | DATETIME | Record creation time | +| `triggered_by` | VARCHAR(50) | `manual`, `scheduled`, `api` | +| `schedule_id` | INTEGER | FK to schedules (if triggered by schedule) | + +#### `scan_sites` +Logical grouping of IPs by site. + +| Column | Type | Description | +|--------|------|-------------| +| `id` | INTEGER PRIMARY KEY | Unique site record ID | +| `scan_id` | INTEGER | FK to scans | +| `site_name` | VARCHAR(255) | Site name from config | + +#### `scan_ips` +IP addresses scanned in each scan. + +| Column | Type | Description | +|--------|------|-------------| +| `id` | INTEGER PRIMARY KEY | Unique IP record ID | +| `scan_id` | INTEGER | FK to scans | +| `site_id` | INTEGER | FK to scan_sites | +| `ip_address` | VARCHAR(45) | IPv4 or IPv6 address | +| `ping_expected` | BOOLEAN | Expected ping response | +| `ping_actual` | BOOLEAN | Actual ping response | + +#### `scan_ports` +Discovered TCP/UDP ports. + +| Column | Type | Description | +|--------|------|-------------| +| `id` | INTEGER PRIMARY KEY | Unique port record ID | +| `scan_id` | INTEGER | FK to scans | +| `ip_id` | INTEGER | FK to scan_ips | +| `port` | INTEGER | Port number (1-65535) | +| `protocol` | VARCHAR(10) | `tcp` or `udp` | +| `expected` | BOOLEAN | Was this port expected? | +| `state` | VARCHAR(20) | `open`, `closed`, `filtered` | + +#### `scan_services` +Detected services on open ports. + +| Column | Type | Description | +|--------|------|-------------| +| `id` | INTEGER PRIMARY KEY | Unique service record ID | +| `scan_id` | INTEGER | FK to scans | +| `port_id` | INTEGER | FK to scan_ports | +| `service_name` | VARCHAR(100) | Service name (e.g., `ssh`, `http`) | +| `product` | VARCHAR(255) | Product name (e.g., `OpenSSH`) | +| `version` | VARCHAR(100) | Version string | +| `extrainfo` | TEXT | Additional nmap info | +| `ostype` | VARCHAR(100) | OS type if detected | +| `http_protocol` | VARCHAR(10) | `http` or `https` (if web service) | +| `screenshot_path` | TEXT | Relative path to screenshot | + +#### `scan_certificates` +SSL/TLS certificates discovered on HTTPS services. + +| Column | Type | Description | +|--------|------|-------------| +| `id` | INTEGER PRIMARY KEY | Unique certificate record ID | +| `scan_id` | INTEGER | FK to scans | +| `service_id` | INTEGER | FK to scan_services | +| `subject` | TEXT | Certificate subject (CN) | +| `issuer` | TEXT | Certificate issuer | +| `serial_number` | TEXT | Serial number | +| `not_valid_before` | DATETIME | Validity start date | +| `not_valid_after` | DATETIME | Validity end date | +| `days_until_expiry` | INTEGER | Days until expiration | +| `sans` | TEXT | JSON array of SANs | +| `is_self_signed` | BOOLEAN | Self-signed certificate flag | + +#### `scan_tls_versions` +TLS version support and cipher suites. + +| Column | Type | Description | +|--------|------|-------------| +| `id` | INTEGER PRIMARY KEY | Unique TLS version record ID | +| `scan_id` | INTEGER | FK to scans | +| `certificate_id` | INTEGER | FK to scan_certificates | +| `tls_version` | VARCHAR(20) | `TLS 1.0`, `TLS 1.1`, `TLS 1.2`, `TLS 1.3` | +| `supported` | BOOLEAN | Is this version supported? | +| `cipher_suites` | TEXT | JSON array of cipher suites | + +### Scheduling & Notifications Tables + +#### `schedules` +Scheduled scan configurations. + +| Column | Type | Description | +|--------|------|-------------| +| `id` | INTEGER PRIMARY KEY | Unique schedule ID | +| `name` | VARCHAR(255) | Schedule name (e.g., "Daily prod scan") | +| `config_file` | TEXT | Path to YAML config | +| `cron_expression` | VARCHAR(100) | Cron-like schedule (e.g., `0 2 * * *`) | +| `enabled` | BOOLEAN | Is schedule active? | +| `last_run` | DATETIME | Last execution time | +| `next_run` | DATETIME | Next scheduled execution | +| `created_at` | DATETIME | Schedule creation time | +| `updated_at` | DATETIME | Last modification time | + +#### `alerts` +Alert history and notifications sent. + +| Column | Type | Description | +|--------|------|-------------| +| `id` | INTEGER PRIMARY KEY | Unique alert ID | +| `scan_id` | INTEGER | FK to scans | +| `alert_type` | VARCHAR(50) | `new_port`, `cert_expiry`, `service_change`, `ping_failed` | +| `severity` | VARCHAR(20) | `info`, `warning`, `critical` | +| `message` | TEXT | Human-readable alert message | +| `ip_address` | VARCHAR(45) | Related IP (optional) | +| `port` | INTEGER | Related port (optional) | +| `email_sent` | BOOLEAN | Was email notification sent? | +| `email_sent_at` | DATETIME | Email send timestamp | +| `created_at` | DATETIME | Alert creation time | + +#### `alert_rules` +User-defined alert rules. + +| Column | Type | Description | +|--------|------|-------------| +| `id` | INTEGER PRIMARY KEY | Unique rule ID | +| `rule_type` | VARCHAR(50) | `unexpected_port`, `cert_expiry`, `service_down`, etc. | +| `enabled` | BOOLEAN | Is rule active? | +| `threshold` | INTEGER | Threshold value (e.g., days for cert expiry) | +| `email_enabled` | BOOLEAN | Send email for this rule? | +| `created_at` | DATETIME | Rule creation time | + +### Settings Table + +#### `settings` +Application configuration key-value store. + +| Column | Type | Description | +|--------|------|-------------| +| `id` | INTEGER PRIMARY KEY | Unique setting ID | +| `key` | VARCHAR(255) UNIQUE | Setting key (e.g., `smtp_server`) | +| `value` | TEXT | Setting value (JSON for complex values) | +| `updated_at` | DATETIME | Last modification time | + +**Example Settings:** +- `smtp_server` - SMTP server hostname +- `smtp_port` - SMTP port (587, 465, 25) +- `smtp_username` - SMTP username +- `smtp_password` - SMTP password (encrypted) +- `smtp_from_email` - From email address +- `smtp_to_emails` - JSON array of recipient emails +- `app_password` - Single-user password hash (bcrypt) +- `retention_days` - How long to keep old scans (0 = forever) + +## API Design + +### REST API Endpoints + +All API endpoints return JSON and follow RESTful conventions. + +#### Scans + +| Method | Endpoint | Description | Request Body | Response | +|--------|----------|-------------|--------------|----------| +| `GET` | `/api/scans` | List all scans (paginated) | - | `{ "scans": [...], "total": N, "page": 1 }` | +| `GET` | `/api/scans/{id}` | Get scan details | - | `{ "scan": {...} }` | +| `POST` | `/api/scans` | Trigger new scan | `{ "config_file": "path" }` | `{ "scan_id": N, "status": "running" }` | +| `DELETE` | `/api/scans/{id}` | Delete scan and files | - | `{ "status": "deleted" }` | +| `GET` | `/api/scans/{id}/status` | Get scan status | - | `{ "status": "running", "progress": "45%" }` | +| `GET` | `/api/scans/{id1}/compare/{id2}` | Compare two scans | - | `{ "diff": {...} }` | + +#### Schedules + +| Method | Endpoint | Description | Request Body | Response | +|--------|----------|-------------|--------------|----------| +| `GET` | `/api/schedules` | List all schedules | - | `{ "schedules": [...] }` | +| `GET` | `/api/schedules/{id}` | Get schedule details | - | `{ "schedule": {...} }` | +| `POST` | `/api/schedules` | Create new schedule | `{ "name": "...", "config_file": "...", "cron_expression": "..." }` | `{ "schedule_id": N }` | +| `PUT` | `/api/schedules/{id}` | Update schedule | `{ "enabled": true, "cron_expression": "..." }` | `{ "status": "updated" }` | +| `DELETE` | `/api/schedules/{id}` | Delete schedule | - | `{ "status": "deleted" }` | +| `POST` | `/api/schedules/{id}/trigger` | Manually trigger scheduled scan | - | `{ "scan_id": N }` | + +#### Alerts + +| Method | Endpoint | Description | Request Body | Response | +|--------|----------|-------------|--------------|----------| +| `GET` | `/api/alerts` | List recent alerts | - | `{ "alerts": [...] }` | +| `GET` | `/api/alerts/rules` | List alert rules | - | `{ "rules": [...] }` | +| `POST` | `/api/alerts/rules` | Create alert rule | `{ "rule_type": "...", "threshold": N }` | `{ "rule_id": N }` | +| `PUT` | `/api/alerts/rules/{id}` | Update alert rule | `{ "enabled": false }` | `{ "status": "updated" }` | +| `DELETE` | `/api/alerts/rules/{id}` | Delete alert rule | - | `{ "status": "deleted" }` | + +#### Settings + +| Method | Endpoint | Description | Request Body | Response | +|--------|----------|-------------|--------------|----------| +| `GET` | `/api/settings` | Get all settings (sanitized) | - | `{ "settings": {...} }` | +| `PUT` | `/api/settings` | Update settings | `{ "smtp_server": "...", ... }` | `{ "status": "updated" }` | +| `POST` | `/api/settings/test-email` | Test email configuration | - | `{ "status": "sent" }` | + +#### Statistics & Trends + +| Method | Endpoint | Description | Request Body | Response | +|--------|----------|-------------|--------------|----------| +| `GET` | `/api/stats/summary` | Dashboard summary stats | - | `{ "total_scans": N, "last_scan": "...", ... }` | +| `GET` | `/api/stats/trends` | Trend data for charts | `?days=30&metric=port_count` | `{ "data": [...] }` | +| `GET` | `/api/stats/certificates` | Certificate expiry overview | - | `{ "expiring_soon": [...], "expired": [...] }` | + +### Authentication + +**Phase 2-3:** Simple session-based authentication (single-user) +- Login endpoint: `POST /api/auth/login` (username/password) +- Logout endpoint: `POST /api/auth/logout` +- Session cookies with Flask-Login +- Password stored as bcrypt hash in settings table + +**Phase 5:** API token authentication for CLI client +- Generate API token: `POST /api/auth/token` +- Revoke token: `DELETE /api/auth/token` +- CLI sends token in `Authorization: Bearer ` header + +## Phased Roadmap + +### Phase 1: Foundation (Weeks 1-2) +**Priority: CRITICAL** - Database and settings infrastructure + +**Goals:** +- Establish database schema +- Create settings system +- Set up Flask project structure + +**Tasks:** +1. Create SQLite database schema (use Alembic for migrations) +2. Implement SQLAlchemy models for all tables +3. Create database initialization script (`init_db.py`) +4. Implement settings system: + - Settings model with get/set methods + - Default settings initialization + - Encrypted storage for passwords (cryptography library) +5. Set up Flask project structure: + ``` + SneakyScanner/ + ├── src/ + │ ├── scanner.py (existing) + │ ├── screenshot_capture.py (existing) + │ └── report_generator.py (existing) + ├── web/ + │ ├── app.py (Flask app factory) + │ ├── models.py (SQLAlchemy models) + │ ├── api/ (API blueprints) + │ │ ├── scans.py + │ │ ├── schedules.py + │ │ ├── alerts.py + │ │ └── settings.py + │ ├── templates/ (Jinja2 templates) + │ ├── static/ (CSS, JS, images) + │ └── utils/ (helpers, decorators) + ├── migrations/ (Alembic migrations) + ├── configs/ (existing) + ├── output/ (existing) + └── templates/ (existing - for reports) + ``` +6. Create `requirements-web.txt` for Flask dependencies +7. Update Dockerfile to support Flask app + +**Deliverables:** +- Working database with schema +- Settings CRUD functionality +- Flask app skeleton (no UI yet) +- Database migration system + +**Testing:** +- Database creates successfully +- Settings can be stored/retrieved +- Flask app starts without errors + +--- + +### Phase 2: Flask Web App Core (Weeks 3-4) +**Priority: HIGH** - Basic web application with API + +**Goals:** +- Implement REST API for scans +- Add background job queue +- Create simple authentication +- Integrate scanner with database + +**Tasks:** +1. Implement scan API endpoints: + - `POST /api/scans` - trigger scan, save to DB + - `GET /api/scans` - list scans with pagination + - `GET /api/scans/{id}` - get scan details from DB + - `DELETE /api/scans/{id}` - delete scan +2. Integrate scanner with database: + - Modify `scanner.py` to save results to DB after scan + - Create `ScanService` class to handle scan → DB logic + - Maintain JSON/HTML/ZIP file generation +3. Set up background job queue: + - Install APScheduler + - Create job executor for scans + - Implement scan status tracking (`running`, `completed`, `failed`) +4. Implement authentication: + - Flask-Login for session management + - Login page (`/login`) + - Password verification against settings table + - Protect all routes with `@login_required` decorator +5. Create basic templates: + - `base.html` - Base layout with Bootstrap 5 dark theme + - `login.html` - Login page + - `dashboard.html` - Placeholder dashboard +6. Error handling and logging: + - API error responses (JSON format) + - Logging configuration (file + console) +7. Docker Compose setup: + - Flask container (Gunicorn) + - Volume mounts for DB, configs, output + - Port mapping (5000 for Flask) + +**Deliverables:** +- Working REST API for scans +- Background scan execution +- Simple login system +- Scanner integrated with database +- Docker Compose deployment + +**Testing:** +- API can trigger scan and return scan_id +- Scan results saved to database +- Pagination works for scan list +- Authentication protects routes +- Docker Compose brings up Flask app + +--- + +### Phase 3: Dashboard & Scheduling (Weeks 5-6) +**Priority: HIGH** - User's top requested features + +**Goals:** +- Build web dashboard with scan history +- Implement trend charts +- Add scheduled scan management UI +- Real-time scan progress + +**Tasks:** +1. Dashboard implementation: + - Summary cards (total scans, last scan, total IPs, open ports) + - Recent scans table (clickable to view details) + - Security warnings section (expiring certs, weak TLS) + - Drift alerts section (unexpected ports, new services) +2. Scan detail page: + - Display full scan results (sites, IPs, services) + - Embedded screenshots or links + - Download buttons (JSON, HTML, ZIP) + - Delete scan button +3. Trend charts with Chart.js: + - Port count over time (line chart) + - Service distribution (bar chart) + - Certificate expiration timeline (timeline chart) + - Charts update based on date range selector +4. Scheduled scans UI: + - List all schedules (table with enable/disable toggle) + - Create schedule form (name, config file, cron expression) + - Edit schedule form + - Delete schedule button + - Next run time display +5. APScheduler integration: + - Schedule manager class + - Load schedules from DB on app start + - Add/remove jobs dynamically + - Cron expression validation + - Update `next_run` and `last_run` in DB +6. Manual scan trigger: + - "Run Scan Now" button on dashboard + - Config file selector + - Show real-time progress (polling or WebSocket) +7. Navigation menu: + - Dashboard + - Scans + - Schedules + - Alerts (placeholder) + - Settings (placeholder) + +**Deliverables:** +- Functional dashboard with charts +- Scan history browser +- Scheduled scan management UI +- Background scheduler running scans + +**Testing:** +- Dashboard displays accurate summary stats +- Charts render correctly with real data +- Scheduled scans execute at specified times +- Manual scan trigger works +- Real-time progress updates + +--- + +### Phase 4: Email & Comparisons (Weeks 7-8) +**Priority: MEDIUM** - Monitoring and analysis features + +**Goals:** +- Implement email notification system +- Create scan comparison reports +- Add alert rule configuration + +**Tasks:** +1. Email notification system: + - SMTP integration (using `smtplib` or `Flask-Mail`) + - Email template for alerts (Jinja2 HTML email) + - Settings page for SMTP configuration + - Test email button +2. Alert rule engine: + - Define alert rule types: + - Unexpected TCP/UDP port opened + - Expected port missing + - Certificate expiring in < N days + - Certificate expired + - Service version changed + - Ping failed (host down) + - Weak TLS version detected (1.0/1.1) + - Alert rule creation UI + - Alert rule evaluation after each scan + - Store alerts in `alerts` table +3. Alert history page: + - List all alerts (filterable by type, severity, date) + - Mark alerts as "acknowledged" + - Alert detail view +4. Scan comparison: + - Compare two scans API endpoint + - Comparison algorithm: + - New ports/services + - Removed ports/services + - Service version changes + - Certificate changes + - TLS configuration changes + - Comparison report UI: + - Side-by-side view + - Diff highlighting (green = added, red = removed, yellow = changed) + - "Compare" button on scan list (select 2 scans) +5. Email notification triggers: + - Send email when alert rule triggered + - Daily digest email (summary of all alerts) + - Weekly scan summary email +6. Settings page: + - SMTP configuration form + - Alert rule management + - Email recipient list + - Test email button + +**Deliverables:** +- Working email notification system +- Alert rules with email triggers +- Scan comparison functionality +- Settings UI for configuration + +**Testing:** +- Email sends successfully with SMTP config +- Alert rules trigger correctly +- Comparison shows accurate diffs +- Settings persist correctly + +--- + +### Phase 5: CLI as API Client (Week 9) +**Priority: MEDIUM** - Backward compatibility and automation + +**Goals:** +- Refactor CLI to optionally call Flask API +- Maintain standalone mode for testing +- API token authentication + +**Tasks:** +1. API client mode for `scanner.py`: + - Add `--api-mode` flag + - Add `--api-url` and `--api-token` arguments + - When `--api-mode` enabled: + - Send scan request to `POST /api/scans` + - Poll `GET /api/scans/{id}/status` for progress + - Download results when complete + - When `--api-mode` disabled (default): + - Run standalone as currently works +2. API token generation: + - UI to generate API tokens (settings page) + - Store tokens in `api_tokens` table (hashed) + - API token authentication middleware + - Token expiration and revocation +3. CLI documentation: + - Update README.md with API mode usage + - Example commands for API mode + - Token generation instructions +4. Benefits of API mode: + - Scans stored centrally in database + - No need to mount volumes for output + - Scheduled scans managed through web UI + - Scan history accessible via web dashboard + +**Deliverables:** +- CLI with `--api-mode` flag +- API token system +- Updated documentation + +**Testing:** +- CLI can trigger scan via API +- API token authentication works +- Standalone mode still functional +- Token revocation works + +--- + +### Phase 6: Advanced Features (Weeks 10+) +**Priority: LOW** - Nice-to-have enhancements + +**Goals:** +- Enhanced interactive reports +- Vulnerability detection +- PDF export +- Timeline view + +**Tasks:** +1. Enhanced HTML reports: + - Sortable/filterable tables (DataTables.js) + - Inline screenshot thumbnails (lightbox on click) + - Export to PDF button (WeasyPrint or pdfkit) + - Print-friendly CSS +2. Vulnerability detection: + - Integrate with CVE databases (NVD API or Vulners API) + - Match detected services/versions to known CVEs + - Display CVE list with severity scores (CVSS) + - CVE detail page with description, remediation + - Alert rule for new critical CVEs +3. Timeline view: + - Visual timeline of all scans + - Filter by site or IP + - Click on timeline event to view scan + - Annotations for important events (cert renewals, config changes) +4. Advanced charts: + - Heatmap of port activity + - Service version tracking over time + - Certificate expiration forecast + - Top 10 services pie chart +5. Export/Import: + - Export scan data to CSV + - Import scan configs from CSV + - Bulk schedule creation +6. Additional integrations: + - Slack notifications (in addition to email) + - Webhook support (POST to custom URL on events) + - Prometheus metrics export + +**Deliverables:** +- Interactive sortable tables +- CVE integration +- PDF export +- Timeline view +- Additional integrations + +**Testing:** +- DataTables work with large datasets +- CVE data fetches correctly +- PDF exports render properly +- Timeline view performs well with many scans + +--- + +## Migration Strategy + +### From Current CLI to Web App + +**Current State:** +- CLI tool (`scanner.py`) runs standalone +- Outputs JSON, HTML, ZIP files +- No database, no web UI + +**Migration Path:** + +#### Step 1: Add Database Layer (Phase 1) +- Database runs alongside CLI +- CLI can optionally save to DB (flag: `--save-to-db`) +- No breaking changes + +#### Step 2: Launch Web App (Phase 2-3) +- Web app reads from DB +- Users can trigger scans via web UI +- CLI still works standalone + +#### Step 3: Transition Period (Phase 4-5) +- Users gradually adopt web UI +- CLI used for scripting/automation +- Both modes fully supported + +#### Step 4: API Client Mode (Phase 5) +- CLI becomes thin API client +- All scans stored in central DB +- Standalone mode remains for testing + +#### Step 5: Full Web App (Phase 6+) +- Primary interface is web UI +- CLI optional for power users + +### Backward Compatibility + +**Maintained:** +- Existing YAML config format +- JSON/HTML/ZIP output files +- Screenshot capture +- Docker deployment + +**Deprecated (eventually):** +- Standalone CLI mode (Phase 6+) +- Direct file output (replaced by DB + API) + +## Prioritized Feature List + +### Must-Have (Phases 1-3) +1. **Database foundation** (SQLite3 + SQLAlchemy) +2. **Flask web app core** (REST API, authentication) +3. **Dashboard with scan history** (list, detail, delete) +4. **Trend charts** (Chart.js - port counts, service distribution) +5. **Scheduled scans** (APScheduler + cron expressions) +6. **Manual scan trigger** (web UI button) + +### Should-Have (Phase 4) +7. **Email notifications** (SMTP integration) +8. **Alert rules** (cert expiry, unexpected ports, etc.) +9. **Scan comparison reports** (diff view) +10. **Settings UI** (SMTP, alerts, retention) + +### Nice-to-Have (Phases 5-6) +11. **CLI as API client** (token auth, backward compat) +12. **Sortable/filterable tables** (DataTables.js) +13. **PDF export** (WeasyPrint) +14. **Vulnerability detection** (CVE integration) +15. **Timeline view** (visual scan history) +16. **Embedded screenshot thumbnails** (lightbox) + +### Future/Deferred +17. **Multi-user support** (if requirements change) +18. **Slack/webhook integrations** +19. **Prometheus metrics** +20. **Mobile-responsive dashboard** (Bootstrap handles basics) + +## Development Workflow + +### Iteration Cycle +1. **Plan** - Define features for phase +2. **Implement** - Code backend + frontend +3. **Test** - Unit tests + manual testing +4. **Deploy** - Update Docker Compose +5. **Document** - Update README.md, ROADMAP.md +6. **Review** - Get user feedback +7. **Iterate** - Adjust priorities based on feedback + +### Git Workflow +- **main branch** - Stable releases +- **develop branch** - Active development +- **feature branches** - Individual features (`feature/dashboard`, `feature/scheduler`) +- **Pull requests** - Review before merge + +### Testing Strategy +- **Unit tests** - pytest for models, API endpoints +- **Integration tests** - Full scan → DB → API workflow +- **Manual testing** - UI/UX testing in browser +- **Performance tests** - Large scans, database queries + +### Documentation +- **README.md** - User-facing documentation (updated each phase) +- **ROADMAP.md** - This file (updated as priorities shift) +- **CLAUDE.md** - Developer documentation (architecture, code references) +- **API.md** - API documentation (OpenAPI/Swagger in Phase 4) + +## Success Metrics + +### Phase 1-3 Success +- [ ] Database stores scan results correctly +- [ ] Dashboard displays scans and trends +- [ ] Scheduled scans execute automatically +- [ ] Web UI is intuitive and responsive + +### Phase 4 Success +- [ ] Email notifications sent for critical alerts +- [ ] Comparison reports show meaningful diffs +- [ ] Settings UI allows configuration without editing files + +### Phase 5 Success +- [ ] CLI can trigger scans via API +- [ ] API tokens work for authentication +- [ ] Standalone CLI mode still functional + +### Phase 6+ Success +- [ ] CVE integration provides actionable vulnerability data +- [ ] Timeline view helps track infrastructure changes +- [ ] PDF exports are shareable and professional + +## Open Questions + +### Technical Decisions +- **Flask vs. FastAPI?** - Sticking with Flask for simplicity, but FastAPI offers async and auto-docs +- **APScheduler vs. Celery?** - APScheduler for simplicity (no Redis/RabbitMQ needed), but Celery scales better +- **Bootstrap vs. Tailwind?** - Bootstrap for speed (pre-built components), Tailwind for customization +- **Chart.js vs. Plotly?** - Chart.js for lightweight, Plotly for interactive (decide in Phase 3) + +### Product Questions +- **Should we support multiple configs per schedule?** - Start with 1:1, add later if needed +- **How many scans to keep in DB?** - Add retention setting (default: keep all) +- **Support for multi-tenancy?** - Not in scope (single-user), but database schema allows future expansion +- **Mobile app?** - Out of scope, but responsive web UI covers basics + +## Resources & References + +### Documentation +- [Flask Documentation](https://flask.palletsprojects.com/) +- [SQLAlchemy ORM](https://docs.sqlalchemy.org/) +- [APScheduler](https://apscheduler.readthedocs.io/) +- [Chart.js](https://www.chartjs.org/docs/) +- [Bootstrap 5](https://getbootstrap.com/docs/5.3/) + +### Tutorials +- [Flask Mega-Tutorial](https://blog.miguelgrinberg.com/post/the-flask-mega-tutorial-part-i-hello-world) +- [SQLAlchemy Tutorial](https://docs.sqlalchemy.org/en/20/tutorial/) +- [APScheduler with Flask](https://github.com/viniciuschiele/flask-apscheduler) + +### Similar Projects (Inspiration) +- [OpenVAS](https://www.openvas.org/) - Vulnerability scanner with web UI +- [Nessus](https://www.tenable.com/products/nessus) - Commercial scanner (inspiration for UI/UX) +- [OWASP ZAP](https://www.zaproxy.org/) - Web app scanner (comparison reports, alerts) + +## Changelog + +| Date | Version | Changes | +|------|---------|---------| +| 2025-11-14 | 1.0 | Initial roadmap created based on user requirements | + +--- + +**Last Updated:** 2025-11-14 +**Next Review:** After Phase 1 completion