sneakygeek/SneakyScan
2
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
Files
5301b07f3729ad91406a51b196aab9ebe92b4246
SneakyScan/docs/ai/ROADMAP.md
Phillip Tarrant 6fe24c3907 adding Phase4
2025-11-17 12:05:11 -06:00

36 KiB
Raw Blame History

SneakyScanner Roadmap

Status: Phase 2 Complete ✅ | Phase 3 Ready to Start

Progress Overview

  • ✅ Phase 1: Foundation - Complete (2025-11-13)
    • Database schema & SQLAlchemy models
    • Settings system with encryption
    • Flask app structure with API blueprints
    • Docker deployment support
  • ✅ Phase 2: Flask Web App Core - Complete (2025-11-14)
    • REST API for scan management (5 endpoints)
    • Background job queue with APScheduler
    • Session-based authentication system
    • Basic UI templates (dashboard, scans, login)
    • Comprehensive error handling and logging
    • 100 tests passing (1,825 lines of test code)
  • ✅ Phase 3: Dashboard & Scheduling - Complete (2025-11-14)
  • 📋 **Phase 4: Config Creator ** -Next up
  • 📋 Phase 5: Email & Comparisons - Planned (Weeks 7-8)
  • 📋 Phase 6: CLI as API Client - Planned (Week 9)
  • 📋 Phase 7: Advanced Features - Planned (Weeks 10+)

Vision & Goals

SneakyScanner is evolving from a CLI-based network scanning tool into a comprehensive Flask web application for infrastructure monitoring and security auditing. The web application will provide:

  • Centralized dashboard for viewing scan history and trends
  • Scheduled scanning for continuous infrastructure monitoring
  • Email notifications for critical changes and certificate expirations
  • Historical analysis with charts, graphs, and comparison reports
  • RESTful API for integration and automation
  • Simple deployment using SQLite3 (single-user, self-hosted)

The CLI scanner will evolve into an API client, maintaining backward compatibility while enabling web-based management and visualization.

Target Users

  • Infrastructure teams monitoring on-premises networks
  • Security professionals performing periodic security audits
  • DevOps engineers tracking infrastructure drift
  • Single users or small teams (not enterprise multi-tenant)

Technology Stack

Backend

  • Flask 3.x - Lightweight Python web framework
  • SQLAlchemy 2.x - ORM for database abstraction
  • SQLite3 - Embedded database (easy deployment, sufficient for single-user)
  • APScheduler 3.x - Background job scheduling for periodic scans
  • Flask-Login - Simple session-based authentication
  • Flask-CORS - API access control (optional for CLI API client)
  • Marshmallow - API serialization/deserialization

Frontend

  • Jinja2 - Server-side templating (already in use)
  • Bootstrap 5 - Responsive UI framework with dark theme support
  • Chart.js 4.x - Lightweight charting library for trends
  • DataTables.js - Interactive sortable/filterable tables (Phase 6)
  • Vanilla JavaScript - Keep dependencies minimal

Infrastructure

  • Docker Compose - Multi-container orchestration (Flask app + Scanner)
  • Gunicorn - WSGI server for production
  • Nginx - Reverse proxy (optional, for production deployments)

Existing Components (Keep)

  • Masscan - Fast port discovery
  • Nmap - Service detection
  • Playwright - Screenshot capture
  • sslyze - SSL/TLS analysis

Architecture Overview

┌─────────────────────────────────────────────────────────────┐
│                     Flask Web Application                   │
│  ┌──────────────┐  ┌──────────────┐  ┌──────────────────┐  │
│  │   Web UI     │  │   REST API   │  │   Scheduler      │  │
│  │  (Dashboard) │  │ (JSON/CRUD)  │  │  (APScheduler)   │  │
│  └──────┬───────┘  └──────┬───────┘  └────────┬─────────┘  │
│         │                 │                    │            │
│         └─────────────────┴────────────────────┘            │
│                           │                                 │
│                  ┌────────▼────────┐                        │
│                  │   SQLAlchemy    │                        │
│                  │   (ORM Layer)   │                        │
│                  └────────┬────────┘                        │
│                           │                                 │
│                  ┌────────▼────────┐                        │
│                  │   SQLite3 DB    │                        │
│                  │  (scan history) │                        │
│                  └─────────────────┘                        │
└───────────────────────────┬─────────────────────────────────┘
                            │
                 ┌──────────▼──────────┐
                 │  Scanner Engine     │
                 │  (scanner.py)       │
                 │  ┌────────────────┐ │
                 │  │ Masscan/Nmap   │ │
                 │  │ Playwright     │ │
                 │  │ sslyze         │ │
                 │  └────────────────┘ │
                 └─────────────────────┘
                            │
                 ┌──────────▼──────────┐
                 │  CLI API Client     │
                 │  (optional future)  │
                 └─────────────────────┘

Component Interaction

  1. Web UI - User interacts with dashboard, triggers scans, views history
  2. REST API - Handles requests from web UI and CLI client
  3. Scheduler (APScheduler) - Triggers scans based on cron-like schedules
  4. SQLAlchemy ORM - Abstracts database operations
  5. SQLite3 Database - Stores scan results, schedules, settings, alerts
  6. Scanner Engine - Performs actual network scanning (masscan, nmap, etc.)
  7. CLI API Client - Future: thin client that calls Flask API

Database Schema Design

Core Tables

scans

Stores metadata about each scan execution.

Column Type Description
id INTEGER PRIMARY KEY Unique scan ID
timestamp DATETIME Scan start time (UTC)
duration FLOAT Total scan duration (seconds)
status VARCHAR(20) running, completed, failed
config_file TEXT Path to YAML config used
title TEXT Scan title from config
json_path TEXT Path to JSON report
html_path TEXT Path to HTML report
zip_path TEXT Path to ZIP archive
screenshot_dir TEXT Path to screenshot directory
created_at DATETIME Record creation time
triggered_by VARCHAR(50) manual, scheduled, api
schedule_id INTEGER FK to schedules (if triggered by schedule)

scan_sites

Logical grouping of IPs by site.

Column Type Description
id INTEGER PRIMARY KEY Unique site record ID
scan_id INTEGER FK to scans
site_name VARCHAR(255) Site name from config

scan_ips

IP addresses scanned in each scan.

Column Type Description
id INTEGER PRIMARY KEY Unique IP record ID
scan_id INTEGER FK to scans
site_id INTEGER FK to scan_sites
ip_address VARCHAR(45) IPv4 or IPv6 address
ping_expected BOOLEAN Expected ping response
ping_actual BOOLEAN Actual ping response

scan_ports

Discovered TCP/UDP ports.

Column Type Description
id INTEGER PRIMARY KEY Unique port record ID
scan_id INTEGER FK to scans
ip_id INTEGER FK to scan_ips
port INTEGER Port number (1-65535)
protocol VARCHAR(10) tcp or udp
expected BOOLEAN Was this port expected?
state VARCHAR(20) open, closed, filtered

scan_services

Detected services on open ports.

Column Type Description
id INTEGER PRIMARY KEY Unique service record ID
scan_id INTEGER FK to scans
port_id INTEGER FK to scan_ports
service_name VARCHAR(100) Service name (e.g., ssh, http)
product VARCHAR(255) Product name (e.g., OpenSSH)
version VARCHAR(100) Version string
extrainfo TEXT Additional nmap info
ostype VARCHAR(100) OS type if detected
http_protocol VARCHAR(10) http or https (if web service)
screenshot_path TEXT Relative path to screenshot

scan_certificates

SSL/TLS certificates discovered on HTTPS services.

Column Type Description
id INTEGER PRIMARY KEY Unique certificate record ID
scan_id INTEGER FK to scans
service_id INTEGER FK to scan_services
subject TEXT Certificate subject (CN)
issuer TEXT Certificate issuer
serial_number TEXT Serial number
not_valid_before DATETIME Validity start date
not_valid_after DATETIME Validity end date
days_until_expiry INTEGER Days until expiration
sans TEXT JSON array of SANs
is_self_signed BOOLEAN Self-signed certificate flag

scan_tls_versions

TLS version support and cipher suites.

Column Type Description
id INTEGER PRIMARY KEY Unique TLS version record ID
scan_id INTEGER FK to scans
certificate_id INTEGER FK to scan_certificates
tls_version VARCHAR(20) TLS 1.0, TLS 1.1, TLS 1.2, TLS 1.3
supported BOOLEAN Is this version supported?
cipher_suites TEXT JSON array of cipher suites

Scheduling & Notifications Tables

schedules

Scheduled scan configurations.

Column Type Description
id INTEGER PRIMARY KEY Unique schedule ID
name VARCHAR(255) Schedule name (e.g., "Daily prod scan")
config_file TEXT Path to YAML config
cron_expression VARCHAR(100) Cron-like schedule (e.g., 0 2 * * *)
enabled BOOLEAN Is schedule active?
last_run DATETIME Last execution time
next_run DATETIME Next scheduled execution
created_at DATETIME Schedule creation time
updated_at DATETIME Last modification time

alerts

Alert history and notifications sent.

Column Type Description
id INTEGER PRIMARY KEY Unique alert ID
scan_id INTEGER FK to scans
alert_type VARCHAR(50) new_port, cert_expiry, service_change, ping_failed
severity VARCHAR(20) info, warning, critical
message TEXT Human-readable alert message
ip_address VARCHAR(45) Related IP (optional)
port INTEGER Related port (optional)
email_sent BOOLEAN Was email notification sent?
email_sent_at DATETIME Email send timestamp
created_at DATETIME Alert creation time

alert_rules

User-defined alert rules.

Column Type Description
id INTEGER PRIMARY KEY Unique rule ID
rule_type VARCHAR(50) unexpected_port, cert_expiry, service_down, etc.
enabled BOOLEAN Is rule active?
threshold INTEGER Threshold value (e.g., days for cert expiry)
email_enabled BOOLEAN Send email for this rule?
created_at DATETIME Rule creation time

Settings Table

settings

Application configuration key-value store.

Column Type Description
id INTEGER PRIMARY KEY Unique setting ID
key VARCHAR(255) UNIQUE Setting key (e.g., smtp_server)
value TEXT Setting value (JSON for complex values)
updated_at DATETIME Last modification time

Example Settings:

  • smtp_server - SMTP server hostname
  • smtp_port - SMTP port (587, 465, 25)
  • smtp_username - SMTP username
  • smtp_password - SMTP password (encrypted)
  • smtp_from_email - From email address
  • smtp_to_emails - JSON array of recipient emails
  • app_password - Single-user password hash (bcrypt)
  • retention_days - How long to keep old scans (0 = forever)

API Design

REST API Endpoints

All API endpoints return JSON and follow RESTful conventions.

Scans

Method Endpoint Description Request Body Response
GET /api/scans List all scans (paginated) - { "scans": [...], "total": N, "page": 1 }
GET /api/scans/{id} Get scan details - { "scan": {...} }
POST /api/scans Trigger new scan { "config_file": "path" } { "scan_id": N, "status": "running" }
DELETE /api/scans/{id} Delete scan and files - { "status": "deleted" }
GET /api/scans/{id}/status Get scan status - { "status": "running", "progress": "45%" }
GET /api/scans/{id1}/compare/{id2} Compare two scans - { "diff": {...} }

Schedules

Method Endpoint Description Request Body Response
GET /api/schedules List all schedules - { "schedules": [...] }
GET /api/schedules/{id} Get schedule details - { "schedule": {...} }
POST /api/schedules Create new schedule { "name": "...", "config_file": "...", "cron_expression": "..." } { "schedule_id": N }
PUT /api/schedules/{id} Update schedule { "enabled": true, "cron_expression": "..." } { "status": "updated" }
DELETE /api/schedules/{id} Delete schedule - { "status": "deleted" }
POST /api/schedules/{id}/trigger Manually trigger scheduled scan - { "scan_id": N }

Alerts

Method Endpoint Description Request Body Response
GET /api/alerts List recent alerts - { "alerts": [...] }
GET /api/alerts/rules List alert rules - { "rules": [...] }
POST /api/alerts/rules Create alert rule { "rule_type": "...", "threshold": N } { "rule_id": N }
PUT /api/alerts/rules/{id} Update alert rule { "enabled": false } { "status": "updated" }
DELETE /api/alerts/rules/{id} Delete alert rule - { "status": "deleted" }

Settings

Method Endpoint Description Request Body Response
GET /api/settings Get all settings (sanitized) - { "settings": {...} }
PUT /api/settings Update settings { "smtp_server": "...", ... } { "status": "updated" }
POST /api/settings/test-email Test email configuration - { "status": "sent" }

Statistics & Trends

Method Endpoint Description Request Body Response
GET /api/stats/summary Dashboard summary stats - { "total_scans": N, "last_scan": "...", ... }
GET /api/stats/trends Trend data for charts ?days=30&metric=port_count { "data": [...] }
GET /api/stats/certificates Certificate expiry overview - { "expiring_soon": [...], "expired": [...] }

Authentication

Phase 2-3: Simple session-based authentication (single-user)

  • Login endpoint: POST /api/auth/login (username/password)
  • Logout endpoint: POST /api/auth/logout
  • Session cookies with Flask-Login
  • Password stored as bcrypt hash in settings table

Phase 5: API token authentication for CLI client

  • Generate API token: POST /api/auth/token
  • Revoke token: DELETE /api/auth/token
  • CLI sends token in Authorization: Bearer <token> header

Phased Roadmap

Phase 1: Foundation ✅ COMPLETE

Completed: 2025-11-13 Priority: CRITICAL - Database and settings infrastructure

Goals:

  • ✅ Establish database schema
  • ✅ Create settings system
  • ✅ Set up Flask project structure

Tasks:

  1. ✅ Create SQLite database schema (use Alembic for migrations)
  2. ✅ Implement SQLAlchemy models for all tables (11 models)
  3. ✅ Create database initialization script (init_db.py)
  4. ✅ Implement settings system:
    • ✅ Settings model with get/set methods
    • ✅ Default settings initialization
    • ✅ Encrypted storage for passwords (cryptography library + bcrypt)
    • ✅ PasswordManager for bcrypt password hashing
  5. ✅ Set up Flask project structure: 
    SneakyScanner/
├── src/
│   ├── scanner.py (existing)
│   ├── screenshot_capture.py (existing)
│   └── report_generator.py (existing)
├── web/                              ✅ CREATED
│   ├── __init__.py                   ✅
│   ├── app.py (Flask app factory)    ✅
│   ├── models.py (SQLAlchemy models) ✅
│   ├── api/ (API blueprints)         ✅
│   │   ├── __init__.py               ✅
│   │   ├── scans.py                  ✅
│   │   ├── schedules.py              ✅
│   │   ├── alerts.py                 ✅
│   │   └── settings.py               ✅ (Fully functional!)
│   ├── templates/ (Jinja2 templates) ✅
│   ├── static/ (CSS, JS, images)     ✅
│   │   ├── css/                      ✅
│   │   ├── js/                       ✅
│   │   └── images/                   ✅
│   └── utils/ (helpers, decorators)  ✅
│       ├── __init__.py               ✅
│       └── settings.py               ✅
├── migrations/ (Alembic migrations)  ✅
│   ├── env.py                        ✅
│   ├── script.py.mako                ✅
│   └── versions/                     ✅
│       └── 001_initial_schema.py     ✅
├── alembic.ini                       ✅
├── configs/ (existing)
├── output/ (existing)
└── templates/ (existing - for reports)
  6. ✅ Create requirements-web.txt for Flask dependencies
  7. ✅ Update Dockerfile to support Flask app
  8. ✅ Create docker-compose-web.yml for web deployment
  9. ✅ Create validate_phase1.py for verification

Deliverables:

  • ✅ Working database with schema (SQLite3 + Alembic migrations)
  • ✅ Settings CRUD functionality (with encryption for sensitive values)
  • ✅ Flask app skeleton with functional Settings API
  • ✅ Database migration system (Alembic)
  • ✅ API blueprint stubs (scans, schedules, alerts, settings)
  • ✅ Docker support (Dockerfile updated, docker-compose-web.yml created)

Testing:

  • ✅ Database creates successfully (init_db.py works)
  • ✅ Settings can be stored/retrieved (encryption working)
  • ✅ Flask app starts without errors (python3 -m web.app works)
  • ✅ All validation checks pass (validate_phase1.py ✓)
  • ✅ All 11 database models defined correctly
  • ✅ Settings API endpoints functional and tested

Documentation:

  • ✅ PHASE1_COMPLETE.md - Complete Phase 1 summary with API reference and deployment guide
  • ✅ validate_phase1.py - Automated validation script

Phase 2: Flask Web App Core ✅ COMPLETE

Completed: 2025-11-14 Duration: 14 days (Weeks 3-4) Priority: HIGH

Goals:

  • ✅ Implement REST API for scans
  • ✅ Add background job queue
  • ✅ Create simple authentication
  • ✅ Integrate scanner with database

Deliverables Completed:

  • ✅ REST API - 5 scan endpoints (trigger, list, get, status, delete) + 3 settings endpoints
  • ✅ Background Jobs - APScheduler with ThreadPoolExecutor (up to 3 concurrent scans)
  • ✅ Authentication - Flask-Login session-based auth (login, logout, decorators)
  • ✅ Database Integration - Complete scan results saved to normalized schema
  • ✅ Web UI - Dashboard, scans list/detail, login, error templates
  • ✅ Error Handling - Content negotiation (JSON/HTML), custom error pages, request IDs
  • ✅ Logging - Rotating file handlers (10MB max), request timing, structured logs
  • ✅ Docker Deployment - Production-ready docker-compose with healthcheck
  • ✅ Testing - 100 test functions, 1,825 lines of test code, all passing
  • ✅ Documentation - API_REFERENCE.md, DEPLOYMENT.md, PHASE2_COMPLETE.md

Files Created: 34 files, ~7,500+ lines of code

Key Features:

  • Scans execute in background without blocking HTTP requests
  • Status tracking: running → completed/failed
  • Pagination and filtering for scan lists
  • Complete scan details with all relationships (sites, IPs, ports, services, certs, TLS)
  • Secure password hashing with bcrypt
  • SQLite WAL mode for better concurrency
  • Request IDs for debugging and correlation
  • Comprehensive error handling for all HTTP status codes

Testing Results:

  • ✅ All API endpoints tested (24 integration tests)
  • ✅ Service layer tested (15 unit tests)
  • ✅ Authentication tested (30+ tests)
  • ✅ Background jobs tested (13 tests)
  • ✅ Error handling tested (18+ tests)
  • ✅ All 100 tests passing

Documentation:

Phase 3: Dashboard & Scheduling (Weeks 5-6)

Priority: HIGH - User's top requested features

Goals:

  • Build web dashboard with scan history
  • Implement trend charts
  • Add scheduled scan management UI
  • Real-time scan progress

Tasks:

  1. Dashboard implementation:
    • Summary cards (total scans, last scan, total IPs, open ports)
    • Recent scans table (clickable to view details)
    • Security warnings section (expiring certs, weak TLS)
    • Drift alerts section (unexpected ports, new services)
  2. Scan detail page:
    • Display full scan results (sites, IPs, services)
    • Embedded screenshots or links
    • Download buttons (JSON, HTML, ZIP)
    • Delete scan button
  3. Trend charts with Chart.js:
    • Port count over time (line chart)
    • Service distribution (bar chart)
    • Certificate expiration timeline (timeline chart)
    • Charts update based on date range selector
  4. Scheduled scans UI:
    • List all schedules (table with enable/disable toggle)
    • Create schedule form (name, config file, cron expression)
    • Edit schedule form
    • Delete schedule button
    • Next run time display
  5. APScheduler integration:
    • Schedule manager class
    • Load schedules from DB on app start
    • Add/remove jobs dynamically
    • Cron expression validation
    • Update next_run and last_run in DB
  6. Manual scan trigger:
    • "Run Scan Now" button on dashboard
    • Config file selector
    • Show real-time progress (polling or WebSocket)
  7. Navigation menu:
    • Dashboard
    • Scans
    • Schedules
    • Alerts (placeholder)
    • Settings (placeholder)

Deliverables:

  • Functional dashboard with charts
  • Scan history browser
  • Scheduled scan management UI
  • Background scheduler running scans

Testing:

  • Dashboard displays accurate summary stats
  • Charts render correctly with real data
  • Scheduled scans execute at specified times
  • Manual scan trigger works
  • Real-time progress updates

Phase 4: Email & Comparisons (Weeks 7-8)

Priority: MEDIUM - Monitoring and analysis features

Goals:

  • Implement email notification system
  • Create scan comparison reports
  • Add alert rule configuration

Tasks:

  1. Email notification system:
    • SMTP integration (using smtplib or Flask-Mail)
    • Email template for alerts (Jinja2 HTML email)
    • Settings page for SMTP configuration
    • Test email button
  2. Alert rule engine:
    • Define alert rule types:
      • Unexpected TCP/UDP port opened
      • Expected port missing
      • Certificate expiring in < N days
      • Certificate expired
      • Service version changed
      • Ping failed (host down)
      • Weak TLS version detected (1.0/1.1)
    • Alert rule creation UI
    • Alert rule evaluation after each scan
    • Store alerts in alerts table
  3. Alert history page:
    • List all alerts (filterable by type, severity, date)
    • Mark alerts as "acknowledged"
    • Alert detail view
  4. Scan comparison:
    • Compare two scans API endpoint
    • Comparison algorithm:
      • New ports/services
      • Removed ports/services
      • Service version changes
      • Certificate changes
      • TLS configuration changes
    • Comparison report UI:
      • Side-by-side view
      • Diff highlighting (green = added, red = removed, yellow = changed)
    • "Compare" button on scan list (select 2 scans)
  5. Email notification triggers:
    • Send email when alert rule triggered
    • Daily digest email (summary of all alerts)
    • Weekly scan summary email
  6. Settings page:
    • SMTP configuration form
    • Alert rule management
    • Email recipient list
    • Test email button

Deliverables:

  • Working email notification system
  • Alert rules with email triggers
  • Scan comparison functionality
  • Settings UI for configuration

Testing:

  • Email sends successfully with SMTP config
  • Alert rules trigger correctly
  • Comparison shows accurate diffs
  • Settings persist correctly

Phase 5: CLI as API Client (Week 9)

Priority: MEDIUM - Backward compatibility and automation

Goals:

  • Refactor CLI to optionally call Flask API
  • Maintain standalone mode for testing
  • API token authentication

Tasks:

  1. API client mode for scanner.py:
    • Add --api-mode flag
    • Add --api-url and --api-token arguments
    • When --api-mode enabled:
      • Send scan request to POST /api/scans
      • Poll GET /api/scans/{id}/status for progress
      • Download results when complete
    • When --api-mode disabled (default):
      • Run standalone as currently works
  2. API token generation:
    • UI to generate API tokens (settings page)
    • Store tokens in api_tokens table (hashed)
    • API token authentication middleware
    • Token expiration and revocation
  3. CLI documentation:
    • Update README.md with API mode usage
    • Example commands for API mode
    • Token generation instructions
  4. Benefits of API mode:
    • Scans stored centrally in database
    • No need to mount volumes for output
    • Scheduled scans managed through web UI
    • Scan history accessible via web dashboard

Deliverables:

  • CLI with --api-mode flag
  • API token system
  • Updated documentation

Testing:

  • CLI can trigger scan via API
  • API token authentication works
  • Standalone mode still functional
  • Token revocation works

Phase 6: Advanced Features (Weeks 10+)

Priority: LOW - Nice-to-have enhancements

Goals:

  • Enhanced interactive reports
  • Vulnerability detection
  • PDF export
  • Timeline view

Tasks:

  1. Enhanced HTML reports:
    • Sortable/filterable tables (DataTables.js)
    • Inline screenshot thumbnails (lightbox on click)
    • Export to PDF button (WeasyPrint or pdfkit)
    • Print-friendly CSS
  2. Vulnerability detection:
    • Integrate with CVE databases (NVD API or Vulners API)
    • Match detected services/versions to known CVEs
    • Display CVE list with severity scores (CVSS)
    • CVE detail page with description, remediation
    • Alert rule for new critical CVEs
  3. Timeline view:
    • Visual timeline of all scans
    • Filter by site or IP
    • Click on timeline event to view scan
    • Annotations for important events (cert renewals, config changes)
  4. Advanced charts:
    • Heatmap of port activity
    • Service version tracking over time
    • Certificate expiration forecast
    • Top 10 services pie chart
  5. Export/Import:
    • Export scan data to CSV
    • Import scan configs from CSV
    • Bulk schedule creation
  6. Additional integrations:
    • Slack notifications (in addition to email)
    • Webhook support (POST to custom URL on events)
    • Prometheus metrics export

Deliverables:

  • Interactive sortable tables
  • CVE integration
  • PDF export
  • Timeline view
  • Additional integrations

Testing:

  • DataTables work with large datasets
  • CVE data fetches correctly
  • PDF exports render properly
  • Timeline view performs well with many scans

Migration Strategy

From Current CLI to Web App

Current State:

  • CLI tool (scanner.py) runs standalone
  • Outputs JSON, HTML, ZIP files
  • No database, no web UI

Migration Path:

Step 1: Add Database Layer (Phase 1)

  • Database runs alongside CLI
  • CLI can optionally save to DB (flag: --save-to-db)
  • No breaking changes

Step 2: Launch Web App (Phase 2-3)

  • Web app reads from DB
  • Users can trigger scans via web UI
  • CLI still works standalone

Step 3: Transition Period (Phase 4-5)

  • Users gradually adopt web UI
  • CLI used for scripting/automation
  • Both modes fully supported

Step 4: API Client Mode (Phase 5)

  • CLI becomes thin API client
  • All scans stored in central DB
  • Standalone mode remains for testing

Step 5: Full Web App (Phase 6+)

  • Primary interface is web UI
  • CLI optional for power users

Backward Compatibility

Maintained:

  • Existing YAML config format
  • JSON/HTML/ZIP output files
  • Screenshot capture
  • Docker deployment

Deprecated (eventually):

  • Standalone CLI mode (Phase 6+)
  • Direct file output (replaced by DB + API)

Prioritized Feature List

Must-Have (Phases 1-3)

  1. Database foundation (SQLite3 + SQLAlchemy)
  2. Flask web app core (REST API, authentication)
  3. Dashboard with scan history (list, detail, delete)
  4. Trend charts (Chart.js - port counts, service distribution)
  5. Scheduled scans (APScheduler + cron expressions)
  6. Manual scan trigger (web UI button)

Should-Have (Phase 4)

  1. Email notifications (SMTP integration)
  2. Alert rules (cert expiry, unexpected ports, etc.)
  3. Scan comparison reports (diff view)
  4. Settings UI (SMTP, alerts, retention)

Nice-to-Have (Phases 5-6)

  1. CLI as API client (token auth, backward compat)
  2. Sortable/filterable tables (DataTables.js)
  3. PDF export (WeasyPrint)
  4. Vulnerability detection (CVE integration)
  5. Timeline view (visual scan history)
  6. Embedded screenshot thumbnails (lightbox)

Future/Deferred

  1. Multi-user support (if requirements change)
  2. Slack/webhook integrations
  3. Prometheus metrics
  4. Mobile-responsive dashboard (Bootstrap handles basics)

Development Workflow

Iteration Cycle

  1. Plan - Define features for phase
  2. Implement - Code backend + frontend
  3. Test - Unit tests + manual testing
  4. Deploy - Update Docker Compose
  5. Document - Update README.md, ROADMAP.md
  6. Review - Get user feedback
  7. Iterate - Adjust priorities based on feedback

Git Workflow

  • main branch - Stable releases
  • develop branch - Active development
  • feature branches - Individual features (feature/dashboard, feature/scheduler)
  • Pull requests - Review before merge

Testing Strategy

  • Unit tests - pytest for models, API endpoints
  • Integration tests - Full scan → DB → API workflow
  • Manual testing - UI/UX testing in browser
  • Performance tests - Large scans, database queries

Documentation

  • README.md - User-facing documentation (updated each phase)
  • ROADMAP.md - This file (updated as priorities shift)
  • CLAUDE.md - Developer documentation (architecture, code references)
  • API.md - API documentation (OpenAPI/Swagger in Phase 4)

Success Metrics

Phase 1 Success ✅ ACHIEVED

  • Database creates successfully with all 11 tables
  • Settings can be stored/retrieved with encryption
  • Flask app starts without errors
  • API blueprints load correctly
  • All Python modules have valid syntax
  • Docker deployment configured

Phase 2 Success ✅ ACHIEVED

  • Database stores scan results correctly
  • REST API functional with all endpoints
  • Background scans execute asynchronously
  • Authentication protects all routes
  • Web UI is intuitive and responsive
  • 100 tests passing with comprehensive coverage
  • Docker deployment production-ready

Phase 3 Success (In Progress)

  • Dashboard displays scans and trends with charts
  • Scheduled scans execute automatically
  • Timeline view shows scan history
  • Real-time progress updates for running scans

Phase 4 Success

  • Email notifications sent for critical alerts
  • Comparison reports show meaningful diffs
  • Settings UI allows configuration without editing files

Phase 5 Success

  • CLI can trigger scans via API
  • API tokens work for authentication
  • Standalone CLI mode still functional

Phase 6+ Success

  • CVE integration provides actionable vulnerability data
  • Timeline view helps track infrastructure changes
  • PDF exports are shareable and professional

Open Questions

Technical Decisions

  • Flask vs. FastAPI? - Sticking with Flask for simplicity, but FastAPI offers async and auto-docs
  • APScheduler vs. Celery? - APScheduler for simplicity (no Redis/RabbitMQ needed), but Celery scales better
  • Bootstrap vs. Tailwind? - Bootstrap for speed (pre-built components), Tailwind for customization
  • Chart.js vs. Plotly? - Chart.js for lightweight, Plotly for interactive (decide in Phase 3)

Product Questions

  • Should we support multiple configs per schedule? - Start with 1:1, add later if needed
  • How many scans to keep in DB? - Add retention setting (default: keep all)
  • Support for multi-tenancy? - Not in scope (single-user), but database schema allows future expansion
  • Mobile app? - Out of scope, but responsive web UI covers basics

Resources & References

Documentation

Tutorials

Similar Projects (Inspiration)

  • OpenVAS - Vulnerability scanner with web UI
  • Nessus - Commercial scanner (inspiration for UI/UX)
  • OWASP ZAP - Web app scanner (comparison reports, alerts)

Changelog

Date Version Changes
2025-11-14 1.0 Initial roadmap created based on user requirements
2025-11-13 1.1 Phase 1 COMPLETE - Database schema, SQLAlchemy models, Flask app structure, settings system with encryption, Alembic migrations, API blueprints, Docker support, validation script
2025-11-14 1.2 Phase 2 COMPLETE - REST API (5 scan endpoints, 3 settings endpoints), background jobs (APScheduler), authentication (Flask-Login), web UI (dashboard, scans, login, errors), error handling (content negotiation, request IDs, logging), 100 tests passing, comprehensive documentation (API_REFERENCE.md, DEPLOYMENT.md, PHASE2_COMPLETE.md)

Last Updated: 2025-11-14 Next Review: Before Phase 3 kickoff (Dashboard enhancement, trend charts, scheduled scans)

Reference in New Issue View Git Blame Copy Permalink