Files

Phillip Tarrant cd840cb8ca restructure of dirs, huge docs update

2025-11-17 16:29:14 -06:00

35 KiB

Raw Blame History

SneakyScanner Roadmap

Vision & Goals

SneakyScanner is a comprehensive Flask web application for infrastructure monitoring and security auditing. The primary interface is the web GUI, with a CLI API client planned for scripting and automation needs.

Status: Phase 4 Complete ✅ | Phase 5 Next Up

Progress Overview

✅ Phase 1: Foundation - Complete (2025-11-13)
- Database schema, SQLAlchemy models, settings system, Flask app structure
✅ Phase 2: Flask Web App Core - Complete (2025-11-14)
- REST API, background jobs, authentication, web UI, testing (100 tests)
✅ Phase 3: Dashboard & Scheduling - Complete (2025-11-14)
- Dashboard, scan history, scheduled scans, trend charts
✅ Phase 4: Config Creator - Complete (2025-11-17)
- CIDR-based config creation, YAML editor, config management UI
📋 Phase 5: Email, Webhooks & Comparisons - Next Up
- Email notifications, alert rules, scan comparison
📋 Phase 6: CLI as API Client - Planned
- CLI for scripting and automation via API
📋 Phase 7: Advanced Features - Future
- CVE integration, timeline view, PDF export, enhanced reports

Core Features:

Centralized dashboard for viewing scan history and trends
Scheduled scanning for continuous infrastructure monitoring
Email notifications for critical changes and certificate expirations (coming soon)
Historical analysis with charts, graphs, and comparison reports
Config creator for easy CIDR-based scan configuration
RESTful API for integration and automation
Simple deployment using SQLite3 (single-user, self-hosted)

Planned:

CLI API client for scripting and automation workflows (Phase 6)

Target Users

Infrastructure teams monitoring on-premises networks
Security professionals performing periodic security audits
DevOps engineers tracking infrastructure drift
Single users or small teams (not enterprise multi-tenant)

Technology Stack

Backend

Flask 3.x - Lightweight Python web framework
SQLAlchemy 2.x - ORM for database abstraction
SQLite3 - Embedded database (easy deployment, sufficient for single-user)
APScheduler 3.x - Background job scheduling for periodic scans
Flask-Login - Simple session-based authentication
Flask-CORS - API access control (optional for CLI API client)
Marshmallow - API serialization/deserialization

Frontend

Jinja2 - Server-side templating (already in use)
Bootstrap 5 - Responsive UI framework with dark theme support
Chart.js 4.x - Lightweight charting library for trends
DataTables.js - Interactive sortable/filterable tables (Phase 6)
Vanilla JavaScript - Keep dependencies minimal

Infrastructure

Docker Compose - Multi-container orchestration (Flask app + Scanner)
Gunicorn - WSGI server for production
Nginx - Reverse proxy (optional, for production deployments)

Existing Components (Keep)

Masscan - Fast port discovery
Nmap - Service detection
Playwright - Screenshot capture
sslyze - SSL/TLS analysis

Architecture Overview

┌─────────────────────────────────────────────────────────────┐
│                     Flask Web Application                   │
│  ┌──────────────┐  ┌──────────────┐  ┌──────────────────┐  │
│  │   Web UI     │  │   REST API   │  │   Scheduler      │  │
│  │  (Dashboard) │  │ (JSON/CRUD)  │  │  (APScheduler)   │  │
│  └──────┬───────┘  └──────┬───────┘  └────────┬─────────┘  │
│         │                 │                    │            │
│         └─────────────────┴────────────────────┘            │
│                           │                                 │
│                  ┌────────▼────────┐                        │
│                  │   SQLAlchemy    │                        │
│                  │   (ORM Layer)   │                        │
│                  └────────┬────────┘                        │
│                           │                                 │
│                  ┌────────▼────────┐                        │
│                  │   SQLite3 DB    │                        │
│                  │  (scan history) │                        │
│                  └─────────────────┘                        │
└───────────────────────────┬─────────────────────────────────┘
                            │
                 ┌──────────▼──────────┐
                 │  Scanner Engine     │
                 │  (scanner.py)       │
                 │  ┌────────────────┐ │
                 │  │ Masscan/Nmap   │ │
                 │  │ Playwright     │ │
                 │  │ sslyze         │ │
                 │  └────────────────┘ │
                 └─────────────────────┘
                            │
                 ┌──────────▼──────────┐
                 │  CLI API Client     │
                 │  (optional future)  │
                 └─────────────────────┘

Component Interaction

Web UI - User interacts with dashboard, triggers scans, views history
REST API - Handles requests from web UI and CLI client
Scheduler (APScheduler) - Triggers scans based on cron-like schedules
SQLAlchemy ORM - Abstracts database operations
SQLite3 Database - Stores scan results, schedules, settings, alerts
Scanner Engine - Performs actual network scanning (masscan, nmap, etc.)
CLI API Client - Future: thin client that calls Flask API

Database Schema Design

Core Tables

`scans`

Stores metadata about each scan execution.

Column	Type	Description
`id`	INTEGER PRIMARY KEY	Unique scan ID
`timestamp`	DATETIME	Scan start time (UTC)
`duration`	FLOAT	Total scan duration (seconds)
`status`	VARCHAR(20)	`running`, `completed`, `failed`
`config_file`	TEXT	Path to YAML config used
`title`	TEXT	Scan title from config
`json_path`	TEXT	Path to JSON report
`html_path`	TEXT	Path to HTML report
`zip_path`	TEXT	Path to ZIP archive
`screenshot_dir`	TEXT	Path to screenshot directory
`created_at`	DATETIME	Record creation time
`triggered_by`	VARCHAR(50)	`manual`, `scheduled`, `api`
`schedule_id`	INTEGER	FK to schedules (if triggered by schedule)

`scan_sites`

Logical grouping of IPs by site.

Column	Type	Description
`id`	INTEGER PRIMARY KEY	Unique site record ID
`scan_id`	INTEGER	FK to scans
`site_name`	VARCHAR(255)	Site name from config

`scan_ips`

IP addresses scanned in each scan.

Column	Type	Description
`id`	INTEGER PRIMARY KEY	Unique IP record ID
`scan_id`	INTEGER	FK to scans
`site_id`	INTEGER	FK to scan_sites
`ip_address`	VARCHAR(45)	IPv4 or IPv6 address
`ping_expected`	BOOLEAN	Expected ping response
`ping_actual`	BOOLEAN	Actual ping response

`scan_ports`

Discovered TCP/UDP ports.

Column	Type	Description
`id`	INTEGER PRIMARY KEY	Unique port record ID
`scan_id`	INTEGER	FK to scans
`ip_id`	INTEGER	FK to scan_ips
`port`	INTEGER	Port number (1-65535)
`protocol`	VARCHAR(10)	`tcp` or `udp`
`expected`	BOOLEAN	Was this port expected?
`state`	VARCHAR(20)	`open`, `closed`, `filtered`

`scan_services`

Detected services on open ports.

Column	Type	Description
`id`	INTEGER PRIMARY KEY	Unique service record ID
`scan_id`	INTEGER	FK to scans
`port_id`	INTEGER	FK to scan_ports
`service_name`	VARCHAR(100)	Service name (e.g., `ssh`, `http`)
`product`	VARCHAR(255)	Product name (e.g., `OpenSSH`)
`version`	VARCHAR(100)	Version string
`extrainfo`	TEXT	Additional nmap info
`ostype`	VARCHAR(100)	OS type if detected
`http_protocol`	VARCHAR(10)	`http` or `https` (if web service)
`screenshot_path`	TEXT	Relative path to screenshot

`scan_certificates`

SSL/TLS certificates discovered on HTTPS services.

Column	Type	Description
`id`	INTEGER PRIMARY KEY	Unique certificate record ID
`scan_id`	INTEGER	FK to scans
`service_id`	INTEGER	FK to scan_services
`subject`	TEXT	Certificate subject (CN)
`issuer`	TEXT	Certificate issuer
`serial_number`	TEXT	Serial number
`not_valid_before`	DATETIME	Validity start date
`not_valid_after`	DATETIME	Validity end date
`days_until_expiry`	INTEGER	Days until expiration
`sans`	TEXT	JSON array of SANs
`is_self_signed`	BOOLEAN	Self-signed certificate flag

`scan_tls_versions`

TLS version support and cipher suites.

Column	Type	Description
`id`	INTEGER PRIMARY KEY	Unique TLS version record ID
`scan_id`	INTEGER	FK to scans
`certificate_id`	INTEGER	FK to scan_certificates
`tls_version`	VARCHAR(20)	`TLS 1.0`, `TLS 1.1`, `TLS 1.2`, `TLS 1.3`
`supported`	BOOLEAN	Is this version supported?
`cipher_suites`	TEXT	JSON array of cipher suites

Scheduling & Notifications Tables

`schedules`

Scheduled scan configurations.

Column	Type	Description
`id`	INTEGER PRIMARY KEY	Unique schedule ID
`name`	VARCHAR(255)	Schedule name (e.g., "Daily prod scan")
`config_file`	TEXT	Path to YAML config
`cron_expression`	VARCHAR(100)	Cron-like schedule (e.g., `0 2 * * *`)
`enabled`	BOOLEAN	Is schedule active?
`last_run`	DATETIME	Last execution time
`next_run`	DATETIME	Next scheduled execution
`created_at`	DATETIME	Schedule creation time
`updated_at`	DATETIME	Last modification time

`alerts`

Alert history and notifications sent.

Column	Type	Description
`id`	INTEGER PRIMARY KEY	Unique alert ID
`scan_id`	INTEGER	FK to scans
`alert_type`	VARCHAR(50)	`new_port`, `cert_expiry`, `service_change`, `ping_failed`
`severity`	VARCHAR(20)	`info`, `warning`, `critical`
`message`	TEXT	Human-readable alert message
`ip_address`	VARCHAR(45)	Related IP (optional)
`port`	INTEGER	Related port (optional)
`email_sent`	BOOLEAN	Was email notification sent?
`email_sent_at`	DATETIME	Email send timestamp
`created_at`	DATETIME	Alert creation time

`alert_rules`

User-defined alert rules.

Column	Type	Description
`id`	INTEGER PRIMARY KEY	Unique rule ID
`rule_type`	VARCHAR(50)	`unexpected_port`, `cert_expiry`, `service_down`, etc.
`enabled`	BOOLEAN	Is rule active?
`threshold`	INTEGER	Threshold value (e.g., days for cert expiry)
`email_enabled`	BOOLEAN	Send email for this rule?
`created_at`	DATETIME	Rule creation time

Settings Table

`settings`

Application configuration key-value store.

Column	Type	Description
`id`	INTEGER PRIMARY KEY	Unique setting ID
`key`	VARCHAR(255) UNIQUE	Setting key (e.g., `smtp_server`)
`value`	TEXT	Setting value (JSON for complex values)
`updated_at`	DATETIME	Last modification time

Example Settings:

smtp_server - SMTP server hostname
smtp_port - SMTP port (587, 465, 25)
smtp_username - SMTP username
smtp_password - SMTP password (encrypted)
smtp_from_email - From email address
smtp_to_emails - JSON array of recipient emails
app_password - Single-user password hash (bcrypt)
retention_days - How long to keep old scans (0 = forever)

API Design

REST API Endpoints

All API endpoints return JSON and follow RESTful conventions.

Scans

Method	Endpoint	Description	Request Body	Response
`GET`	`/api/scans`	List all scans (paginated)	-	`{ "scans": [...], "total": N, "page": 1 }`
`GET`	`/api/scans/{id}`	Get scan details	-	`{ "scan": {...} }`
`POST`	`/api/scans`	Trigger new scan	`{ "config_file": "path" }`	`{ "scan_id": N, "status": "running" }`
`DELETE`	`/api/scans/{id}`	Delete scan and files	-	`{ "status": "deleted" }`
`GET`	`/api/scans/{id}/status`	Get scan status	-	`{ "status": "running", "progress": "45%" }`
`GET`	`/api/scans/{id1}/compare/{id2}`	Compare two scans	-	`{ "diff": {...} }`

Schedules

Method	Endpoint	Description	Request Body	Response
`GET`	`/api/schedules`	List all schedules	-	`{ "schedules": [...] }`
`GET`	`/api/schedules/{id}`	Get schedule details	-	`{ "schedule": {...} }`
`POST`	`/api/schedules`	Create new schedule	`{ "name": "...", "config_file": "...", "cron_expression": "..." }`	`{ "schedule_id": N }`
`PUT`	`/api/schedules/{id}`	Update schedule	`{ "enabled": true, "cron_expression": "..." }`	`{ "status": "updated" }`
`DELETE`	`/api/schedules/{id}`	Delete schedule	-	`{ "status": "deleted" }`
`POST`	`/api/schedules/{id}/trigger`	Manually trigger scheduled scan	-	`{ "scan_id": N }`

Alerts

Method	Endpoint	Description	Request Body	Response
`GET`	`/api/alerts`	List recent alerts	-	`{ "alerts": [...] }`
`GET`	`/api/alerts/rules`	List alert rules	-	`{ "rules": [...] }`
`POST`	`/api/alerts/rules`	Create alert rule	`{ "rule_type": "...", "threshold": N }`	`{ "rule_id": N }`
`PUT`	`/api/alerts/rules/{id}`	Update alert rule	`{ "enabled": false }`	`{ "status": "updated" }`
`DELETE`	`/api/alerts/rules/{id}`	Delete alert rule	-	`{ "status": "deleted" }`

Settings

Method	Endpoint	Description	Request Body	Response
`GET`	`/api/settings`	Get all settings (sanitized)	-	`{ "settings": {...} }`
`PUT`	`/api/settings`	Update settings	`{ "smtp_server": "...", ... }`	`{ "status": "updated" }`
`POST`	`/api/settings/test-email`	Test email configuration	-	`{ "status": "sent" }`

Statistics & Trends

Method	Endpoint	Description	Request Body	Response
`GET`	`/api/stats/summary`	Dashboard summary stats	-	`{ "total_scans": N, "last_scan": "...", ... }`
`GET`	`/api/stats/trends`	Trend data for charts	`?days=30&metric=port_count`	`{ "data": [...] }`
`GET`	`/api/stats/certificates`	Certificate expiry overview	-	`{ "expiring_soon": [...], "expired": [...] }`

Authentication

Phase 2-3: Simple session-based authentication (single-user)

Login endpoint: POST /api/auth/login (username/password)
Logout endpoint: POST /api/auth/logout
Session cookies with Flask-Login
Password stored as bcrypt hash in settings table

Phase 5: API token authentication for CLI client

Generate API token: POST /api/auth/token
Revoke token: DELETE /api/auth/token
CLI sends token in Authorization: Bearer <token> header

Phased Roadmap

Phase 1: Foundation ✅ COMPLETE

Completed: 2025-11-13

Deliverables:

SQLite database with 11 tables (scans, sites, IPs, ports, services, certificates, TLS versions, schedules, alerts, alert_rules, settings)
SQLAlchemy ORM models with relationships
Alembic migration system
Settings system with encryption (bcrypt for passwords, Fernet for sensitive data)
Flask app structure with API blueprints
Docker Compose deployment configuration
Validation script for verification

Phase 2: Flask Web App Core ✅ COMPLETE

Completed: 2025-11-14

Deliverables:

REST API with 8 endpoints (scans: trigger, list, get, status, delete; settings: get, update, test-email)
Background job queue using APScheduler (up to 3 concurrent scans)
Session-based authentication with Flask-Login
Database integration for scan results (full normalized schema population)
Web UI templates (dashboard, scan list/detail, login, error pages)
Error handling with content negotiation (JSON/HTML) and request IDs
Logging system with rotating file handlers
Production Docker Compose deployment
Comprehensive test suite (100 tests, all passing)
Documentation (API_REFERENCE.md, DEPLOYMENT.md)

Phase 3: Dashboard & Scheduling ✅ COMPLETE

Completed: 2025-11-14

Deliverables:

Dashboard with summary stats (total scans, IPs, ports, services)
Recent scans table with clickable details
Scan detail page with full results display
Historical trend charts using Chart.js (port counts over time)
Scheduled scan management UI (create, edit, delete, enable/disable)
Schedule execution with APScheduler and cron expressions
Manual scan trigger from web UI
Navigation menu (Dashboard, Scans, Schedules, Configs, Settings)
Download buttons for scan reports (JSON, HTML, ZIP)

Phase 4: Config Creator ✅ COMPLETE

Completed: 2025-11-17

Deliverables:

CIDR-based config creation UI (simplified workflow for quick config generation)
YAML editor with CodeMirror (syntax highlighting, line numbers)
Config management UI (list, view, edit, download, delete)
Direct YAML upload for advanced users
REST API for config operations (7 endpoints: list, get, create, update, delete, upload, download)
Schedule dependency protection (prevents deleting configs used by schedules)
Comprehensive testing (25+ unit and integration tests)

Phase 5: Email, Webhooks & Comparisons

Status: Next Up Priority: MEDIUM

Goals:

Implement email notification system for infrastructure misconfigurations
Implement webhook notification system for real-time alerting
Create scan comparison reports to detect drift
Add alert rule configuration for unexpected exposure detection

Core Use Case: Monitor infrastructure for misconfigurations that expose unexpected ports/services to the world. When a scan detects an open port that wasn't defined in the YAML config's expected_ports list, trigger immediate notifications via email and/or webhooks.

Planned Features:

1. Alert Rule Engine

Purpose: Automatically detect and classify infrastructure anomalies after each scan.

Alert Types:

unexpected_port - Port open but not in config's expected_ports list
unexpected_service - Service detected that doesn't match expected service name
cert_expiry - SSL/TLS certificate expiring soon (configurable threshold)
ping_failed - Expected host not responding to ping
service_down - Previously detected service no longer responding
service_change - Service version/product changed between scans
weak_tls - TLS 1.0/1.1 detected or weak cipher suites
new_host - New IP address responding in CIDR range
host_disappeared - Previously seen IP no longer responding

Alert Severity Levels:

critical - Unexpected internet-facing service (ports 80/443/22/3389/etc.)
warning - Minor configuration drift or upcoming cert expiry
info - Informational alerts (new host discovered, service version change)

Alert Rule Configuration:

# Example alert rule configuration (stored in DB)
alert_rules:
  - id: 1
    rule_type: unexpected_port
    enabled: true
    severity: critical
    email_enabled: true
    webhook_enabled: true
    filter_conditions:
      ports: [22, 80, 443, 3389, 3306, 5432, 27017]  # High-risk ports

  - id: 2
    rule_type: cert_expiry
    enabled: true
    severity: warning
    threshold: 30  # Days before expiry
    email_enabled: true
    webhook_enabled: false

Implementation:

Evaluate alert rules after each scan completes
Compare current scan results to expected configuration
Generate alerts and store in alerts table
Trigger notifications based on rule configuration
Alert deduplication (don't spam for same issue)

2. Email Notifications

Purpose: Send detailed email alerts when infrastructure misconfigurations are detected.

SMTP Configuration (via Settings API):

{
  "smtp_server": "smtp.gmail.com",
  "smtp_port": 587,
  "smtp_use_tls": true,
  "smtp_username": "alerts@example.com",
  "smtp_password": "encrypted_password",
  "smtp_from_email": "SneakyScanner <alerts@example.com>",
  "smtp_to_emails": ["admin@example.com", "security@example.com"],
  "email_alerts_enabled": true
}

Email Template (Jinja2 HTML):

Subject: [SneakyScanner Alert] Unexpected Port Detected - {{ ip_address }}:{{ port }}

Body:
===============================================
SneakyScanner Security Alert
===============================================

Alert Type: {{ alert_type }}
Severity: {{ severity }}
Scan ID: {{ scan_id }}
Timestamp: {{ timestamp }}

Issue Detected:
{{ message }}

Details:
- IP Address: {{ ip_address }}
- Port: {{ port }}/{{ protocol }}
- Service: {{ service_name }} ({{ product }} {{ version }})
- Expected: No (not in expected_ports list)

Recommended Actions:
1. Verify if this service should be exposed
2. Check firewall rules for {{ ip_address }}
3. Review service configuration
4. Update scan config if this is intentional

View Full Scan Results:
{{ web_url }}/scans/{{ scan_id }}

===============================================

Email Features:

HTML email with styled alert box (Bootstrap-based)
Plain-text fallback for compatibility
Alert summary with actionable recommendations
Direct link to scan detail page
Configurable recipients (multiple emails)
Test email functionality in Settings UI
Email delivery tracking (email_sent, email_sent_at in alerts table)
Rate limiting to prevent email flood

Email API Endpoints:

POST /api/settings/email - Configure SMTP settings
POST /api/settings/email/test - Send test email
GET /api/alerts?email_sent=true - Get alerts with email status

3. Webhook Notifications

Purpose: Real-time HTTP POST notifications for integration with external systems (Slack, PagerDuty, custom dashboards, SIEM tools).

Webhook Configuration (via Settings API):

{
  "webhook_enabled": true,
  "webhook_urls": [
    {
      "id": 1,
      "name": "Slack Security Channel",
      "url": "https://hooks.slack.com/services/XXX/YYY/ZZZ",
      "enabled": true,
      "auth_type": "none",
      "custom_headers": {},
      "alert_types": ["unexpected_port", "unexpected_service", "weak_tls"],
      "severity_filter": ["critical", "warning"]
    },
    {
      "id": 2,
      "name": "PagerDuty",
      "url": "https://events.pagerduty.com/v2/enqueue",
      "enabled": true,
      "auth_type": "bearer",
      "auth_token": "encrypted_token",
      "custom_headers": {
        "Content-Type": "application/json"
      },
      "alert_types": ["unexpected_port"],
      "severity_filter": ["critical"]
    }
  ]
}

Webhook Payload Format (JSON):

{
  "event_type": "scan_alert",
  "alert_id": 42,
  "alert_type": "unexpected_port",
  "severity": "critical",
  "timestamp": "2025-11-17T14:23:45Z",
  "scan": {
    "scan_id": 123,
    "title": "Production Network Scan",
    "timestamp": "2025-11-17T14:15:00Z",
    "config_file": "prod_config.yaml",
    "triggered_by": "scheduled"
  },
  "alert_details": {
    "message": "Unexpected port 3306 (MySQL) exposed on 192.168.1.100",
    "ip_address": "192.168.1.100",
    "port": 3306,
    "protocol": "tcp",
    "state": "open",
    "service": {
      "name": "mysql",
      "product": "MySQL",
      "version": "8.0.32"
    },
    "expected": false,
    "site_name": "Production Servers"
  },
  "recommended_actions": [
    "Verify if MySQL should be exposed externally",
    "Check firewall rules for 192.168.1.100",
    "Review MySQL bind-address configuration"
  ],
  "web_url": "https://sneakyscanner.local/scans/123"
}

Webhook Features:

Multiple webhook URLs with independent configuration
Per-webhook filtering by alert type and severity
Custom headers support (for API keys, auth tokens)
Authentication methods:
- none - No authentication
- bearer - Bearer token in Authorization header
- basic - Basic authentication
- custom - Custom header-based auth
Retry logic with exponential backoff (3 attempts)
Webhook delivery tracking (webhook_sent, webhook_sent_at, webhook_response_code)
Test webhook functionality in Settings UI
Timeout configuration (default 10 seconds)
Webhook delivery history and logs

Webhook API Endpoints:

POST /api/webhooks - Create webhook configuration
GET /api/webhooks - List all webhooks
PUT /api/webhooks/{id} - Update webhook configuration
DELETE /api/webhooks/{id} - Delete webhook
POST /api/webhooks/{id}/test - Send test webhook
GET /api/webhooks/{id}/history - Get delivery history

Slack Integration Example: Transform webhook payload to Slack message format:

{
  "text": "SneakyScanner Alert: Unexpected Port Detected",
  "attachments": [
    {
      "color": "danger",
      "fields": [
        {"title": "IP Address", "value": "192.168.1.100", "short": true},
        {"title": "Port", "value": "3306/tcp", "short": true},
        {"title": "Service", "value": "MySQL 8.0.32", "short": true},
        {"title": "Severity", "value": "CRITICAL", "short": true}
      ],
      "footer": "SneakyScanner",
      "ts": 1700234625
    }
  ]
}

4. Alert Management UI

Purpose: Web interface for configuring alert rules, viewing alert history, and managing notifications.

Pages:

/alerts - Alert history with filtering and search
/alerts/rules - Alert rule configuration
/settings/email - Email notification settings
/settings/webhooks - Webhook configuration

Alert History Features:

Filter by alert type, severity, date range, IP address
Search by message content
Bulk acknowledge/dismiss alerts
Export alerts to CSV
Alert detail modal with full context

Alert Rule UI Features:

Enable/disable rules individually
Configure severity levels
Set thresholds (e.g., cert expiry days)
Toggle email/webhook per rule
Test rules against recent scans

5. Scan Comparison

Purpose: Detect infrastructure drift by comparing two scans and highlighting changes.

Comparison API:

GET /api/scans/{id1}/compare/{id2} - Compare two scans

Comparison Response:

{
  "scan1": {"id": 100, "timestamp": "2025-11-10T10:00:00Z"},
  "scan2": {"id": 123, "timestamp": "2025-11-17T14:15:00Z"},
  "summary": {
    "new_ports": 2,
    "removed_ports": 0,
    "service_changes": 1,
    "cert_changes": 0,
    "new_hosts": 1,
    "removed_hosts": 0
  },
  "differences": {
    "new_ports": [
      {"ip": "192.168.1.100", "port": 3306, "service": "mysql"}
    ],
    "removed_ports": [],
    "service_changes": [
      {
        "ip": "192.168.1.50",
        "port": 22,
        "old": "OpenSSH 8.2",
        "new": "OpenSSH 8.9"
      }
    ],
    "new_hosts": [
      {"ip": "192.168.1.200", "site": "Production Servers"}
    ]
  }
}

Comparison UI Features:

Side-by-side comparison view
Color-coded differences (green=new, red=removed, yellow=changed)
Filter by change type
Export comparison report to PDF/HTML
"Compare with previous scan" button on scan detail page

Phase 5 Implementation Plan:

Week 1: Alert Rule Engine
- Implement alert evaluation logic after scan completion
- Create alerts table population
- Add alert rule CRUD API
- Unit tests for alert detection
Week 2: Email Notifications
- SMTP integration with Flask-Mail
- Jinja2 email templates (HTML + plain text)
- Settings API for email configuration
- Test email functionality
- Email delivery tracking
Week 3: Webhook System
- Webhook configuration API
- HTTP POST delivery with retry logic
- Webhook template system for different platforms
- Test webhook functionality
- Delivery tracking and logging
Week 4: Alert UI & Scan Comparison
- Alert history page with filtering
- Alert rule management UI
- Email/webhook settings pages
- Scan comparison API and UI
- Integration testing

Success Criteria:

Alert triggered within 30 seconds of scan completion
Email delivered successfully to configured recipients
Webhook POST delivered with retry on failure
Scan comparison highlights all infrastructure changes
Zero false positives for expected ports/services
Alert deduplication prevents notification spam

Phase 6: CLI as API Client

Status: Planned Priority: MEDIUM

Goals:

Create CLI API client for scripting and automation
Maintain standalone mode for testing
API token authentication

Planned Features:

API Client Mode:
- --api-mode flag to enable API client mode
- --api-url and --api-token arguments
- Trigger scans via API, poll for status, download results
- Scans stored centrally in database
- Standalone mode still available for testing
API Token System:
- Token generation UI in settings page
- Secure token storage (hashed in database)
- Token authentication middleware
- Token expiration and revocation
Benefits:
- Centralized scan history accessible via web dashboard
- No need to mount volumes for output
- Scheduled scans managed through web UI
- Scriptable automation while leveraging web features

Phase 7: Advanced Features

Status: Future/Deferred Priority: LOW

Planned Features:

Enhanced Reports:
- Sortable/filterable tables (DataTables.js)
- Inline screenshot thumbnails with lightbox
- PDF export (WeasyPrint)
Vulnerability Detection:
- CVE database integration (NVD API)
- Service version matching to known CVEs
- CVSS severity scores
- Alert rules for critical CVEs
Timeline View:
- Visual scan history timeline
- Filter by site/IP
- Event annotations
Advanced Charts:
- Port activity heatmap
- Service version tracking
- Certificate expiration forecast
Integrations:
- Slack notifications
- Webhook support
- Prometheus metrics export
- CSV export/import

Current Architecture

Primary Interface: Web GUI (Phases 1-4 Complete)

Full-featured Flask web application
Dashboard, scan management, scheduling, config creator
REST API for all operations
Single-user deployment with SQLite

Coming Soon: CLI API Client (Phase 6 Planned)

Thin client for scripting and automation
Calls Flask API for scan operations
Results stored centrally in database
Access to all web features via command line

Core Scanning Engine:

Masscan for port discovery
Nmap for service detection
Playwright for screenshots
sslyze for SSL/TLS analysis

Deployment:

Docker Compose for easy deployment
SQLite database (single-user, embedded)
Gunicorn WSGI server
Optional Nginx reverse proxy

Development Workflow

Iteration Cycle

Plan - Define features for phase
Implement - Code backend + frontend
Test - Unit tests + manual testing
Deploy - Update Docker Compose
Document - Update README.md, ROADMAP.md
Review - Get user feedback
Iterate - Adjust priorities based on feedback

Git Workflow

main branch - Stable releases
develop branch - Active development
feature branches - Individual features (feature/dashboard, feature/scheduler)
Pull requests - Review before merge

Testing Strategy

Unit tests - pytest for models, API endpoints
Integration tests - Full scan → DB → API workflow
Manual testing - UI/UX testing in browser
Performance tests - Large scans, database queries

Documentation

README.md - User-facing documentation (updated each phase)
ROADMAP.md - This file (updated as priorities shift)
CLAUDE.md - Developer documentation (architecture, code references)
API.md - API documentation (OpenAPI/Swagger in Phase 4)

Resources & References

Documentation

Tutorials

Similar Projects (Inspiration)

OpenVAS - Vulnerability scanner with web UI
Nessus - Commercial scanner (inspiration for UI/UX)
OWASP ZAP - Web app scanner (comparison reports, alerts)

Changelog

Date	Version	Changes
2025-11-14	1.0	Initial roadmap created based on user requirements
2025-11-13	1.1	Phase 1 COMPLETE - Database schema, SQLAlchemy models, Flask app structure, settings system with encryption, Alembic migrations, API blueprints, Docker support, validation script
2025-11-14	1.2	Phase 2 COMPLETE - REST API (5 scan endpoints, 3 settings endpoints), background jobs (APScheduler), authentication (Flask-Login), web UI (dashboard, scans, login, errors), error handling (content negotiation, request IDs, logging), 100 tests passing, comprehensive documentation (API_REFERENCE.md, DEPLOYMENT.md, PHASE2_COMPLETE.md)
2025-11-17	1.3	Bug Fix - Fixed Chart.js infinite canvas growth issue in scan detail page (duplicate initialization, missing chart.destroy(), missing fixed-height container)
2025-11-17	1.4	Phase 4 COMPLETE - Config Creator with CIDR-based creation, YAML editor (CodeMirror), config management UI (list/edit/delete), REST API (7 endpoints), Docker volume permissions fix, comprehensive testing and documentation
2025-11-17	1.5	Roadmap Compression - Condensed completed phases (1-4) into concise summaries, updated project scope to emphasize web GUI frontend with CLI as API client coming soon (Phase 6), reorganized phases for clarity

Last Updated: 2025-11-17 Next Review: Before Phase 5 kickoff (Email & Comparisons)

35 KiB Raw Blame History

SneakyScanner Roadmap

Vision & Goals

Progress Overview

Target Users

Technology Stack

Backend

Frontend

Infrastructure

Existing Components (Keep)

Architecture Overview

Component Interaction

Database Schema Design

Core Tables

scans

scan_sites

scan_ips

scan_ports

scan_services

scan_certificates

scan_tls_versions

Scheduling & Notifications Tables

schedules

alerts

alert_rules

Settings Table

settings

API Design

REST API Endpoints

Scans

Schedules

Alerts

Settings

Statistics & Trends

Authentication

Phased Roadmap

Phase 1: Foundation ✅ COMPLETE

Phase 2: Flask Web App Core ✅ COMPLETE

Phase 3: Dashboard & Scheduling ✅ COMPLETE

Phase 4: Config Creator ✅ COMPLETE

Phase 5: Email, Webhooks & Comparisons

1. Alert Rule Engine

2. Email Notifications

3. Webhook Notifications

4. Alert Management UI

5. Scan Comparison

Phase 6: CLI as API Client

Phase 7: Advanced Features

Current Architecture

Development Workflow

Iteration Cycle

Git Workflow

Testing Strategy

Documentation

Resources & References

Documentation

Tutorials

Similar Projects (Inspiration)

Changelog

35 KiB

Raw Blame History

`scans`

`scan_sites`

`scan_ips`

`scan_ports`

`scan_services`

`scan_certificates`

`scan_tls_versions`

`schedules`

`alerts`

`alert_rules`

`settings`