SneakyScan/docs/ROADMAP.md

SneakyScanner Roadmap

Vision & Goals

SneakyScanner is a comprehensive Flask web application for infrastructure monitoring and security auditing. The primary interface is the web GUI, with a CLI API client planned for scripting and automation needs.

Current Phase: Phase 5 Complete ✅ | Phase 6 Next Up

Progress Overview

Note: For detailed architecture and technology stack information, see README.md

• ✅ Phase 1: Foundation - Complete (2025-11-13)
  • Database schema, SQLAlchemy models, settings system, Flask app structure
• ✅ Phase 2: Flask Web App Core - Complete (2025-11-14)
  • REST API, background jobs, authentication, web UI, testing (100 tests)
• ✅ Phase 3: Dashboard & Scheduling - Complete (2025-11-14)
  • Dashboard, scan history, scheduled scans, trend charts
• ✅ Phase 4: Config Creator - Complete (2025-11-17)
  • CIDR-based config creation, YAML editor, config management UI
• ✅ Phase 5: Webhooks & Alerting - Complete (2025-11-19)
  • Webhook notifications, alert rules, notification templates
• 📋 Phase 6: CLI as API Client - Planned
  • CLI for scripting and automation via API
• 📋 Phase 7: Advanced Features - Future
  • Email notifications, scan comparison, CVE integration, timeline view, PDF export

Target Users

• Infrastructure teams monitoring on-premises networks
• Security professionals performing periodic security audits
• DevOps engineers tracking infrastructure drift
• Single users or small teams (not enterprise multi-tenant)

Database Schema Design

Core Tables

scans

Stores metadata about each scan execution.

Column	Type	Description
id	INTEGER PRIMARY KEY	Unique scan ID
timestamp	DATETIME	Scan start time (UTC)
duration	FLOAT	Total scan duration (seconds)
status	VARCHAR(20)	running, completed, failed
config_file	TEXT	Path to YAML config used
title	TEXT	Scan title from config
json_path	TEXT	Path to JSON report
html_path	TEXT	Path to HTML report
zip_path	TEXT	Path to ZIP archive
screenshot_dir	TEXT	Path to screenshot directory
created_at	DATETIME	Record creation time
triggered_by	VARCHAR(50)	manual, scheduled, api
schedule_id	INTEGER	FK to schedules (if triggered by schedule)

scan_sites

Logical grouping of IPs by site.

Column	Type	Description
id	INTEGER PRIMARY KEY	Unique site record ID
scan_id	INTEGER	FK to scans
site_name	VARCHAR(255)	Site name from config

scan_ips

IP addresses scanned in each scan.

Column	Type	Description
id	INTEGER PRIMARY KEY	Unique IP record ID
scan_id	INTEGER	FK to scans
site_id	INTEGER	FK to scan_sites
ip_address	VARCHAR(45)	IPv4 or IPv6 address
ping_expected	BOOLEAN	Expected ping response
ping_actual	BOOLEAN	Actual ping response

scan_ports

Discovered TCP/UDP ports.

Column	Type	Description
id	INTEGER PRIMARY KEY	Unique port record ID
scan_id	INTEGER	FK to scans
ip_id	INTEGER	FK to scan_ips
port	INTEGER	Port number (1-65535)
protocol	VARCHAR(10)	tcp or udp
expected	BOOLEAN	Was this port expected?
state	VARCHAR(20)	open, closed, filtered

scan_services

Detected services on open ports.

Column	Type	Description
id	INTEGER PRIMARY KEY	Unique service record ID
scan_id	INTEGER	FK to scans
port_id	INTEGER	FK to scan_ports
service_name	VARCHAR(100)	Service name (e.g., ssh, http)
product	VARCHAR(255)	Product name (e.g., OpenSSH)
version	VARCHAR(100)	Version string
extrainfo	TEXT	Additional nmap info
ostype	VARCHAR(100)	OS type if detected
http_protocol	VARCHAR(10)	http or https (if web service)
screenshot_path	TEXT	Relative path to screenshot

scan_certificates

SSL/TLS certificates discovered on HTTPS services.

Column	Type	Description
id	INTEGER PRIMARY KEY	Unique certificate record ID
scan_id	INTEGER	FK to scans
service_id	INTEGER	FK to scan_services
subject	TEXT	Certificate subject (CN)
issuer	TEXT	Certificate issuer
serial_number	TEXT	Serial number
not_valid_before	DATETIME	Validity start date
not_valid_after	DATETIME	Validity end date
days_until_expiry	INTEGER	Days until expiration
sans	TEXT	JSON array of SANs
is_self_signed	BOOLEAN	Self-signed certificate flag

scan_tls_versions

TLS version support and cipher suites.

Column	Type	Description
id	INTEGER PRIMARY KEY	Unique TLS version record ID
scan_id	INTEGER	FK to scans
certificate_id	INTEGER	FK to scan_certificates
tls_version	VARCHAR(20)	TLS 1.0, TLS 1.1, TLS 1.2, TLS 1.3
supported	BOOLEAN	Is this version supported?
cipher_suites	TEXT	JSON array of cipher suites

Scheduling & Notifications Tables

schedules

Scheduled scan configurations.

Column	Type	Description
id	INTEGER PRIMARY KEY	Unique schedule ID
name	VARCHAR(255)	Schedule name (e.g., "Daily prod scan")
config_file	TEXT	Path to YAML config
cron_expression	VARCHAR(100)	Cron-like schedule (e.g., 0 2 * * *)
enabled	BOOLEAN	Is schedule active?
last_run	DATETIME	Last execution time
next_run	DATETIME	Next scheduled execution
created_at	DATETIME	Schedule creation time
updated_at	DATETIME	Last modification time

alerts

Alert history and notifications sent.

Column	Type	Description
id	INTEGER PRIMARY KEY	Unique alert ID
scan_id	INTEGER	FK to scans
alert_type	VARCHAR(50)	new_port, cert_expiry, service_change, ping_failed
severity	VARCHAR(20)	info, warning, critical
message	TEXT	Human-readable alert message
ip_address	VARCHAR(45)	Related IP (optional)
port	INTEGER	Related port (optional)
email_sent	BOOLEAN	Was email notification sent?
email_sent_at	DATETIME	Email send timestamp
created_at	DATETIME	Alert creation time

alert_rules

User-defined alert rules.

Column	Type	Description
id	INTEGER PRIMARY KEY	Unique rule ID
rule_type	VARCHAR(50)	unexpected_port, cert_expiry, service_down, etc.
enabled	BOOLEAN	Is rule active?
threshold	INTEGER	Threshold value (e.g., days for cert expiry)
email_enabled	BOOLEAN	Send email for this rule?
created_at	DATETIME	Rule creation time

Settings Table

settings

Application configuration key-value store.

Column	Type	Description
id	INTEGER PRIMARY KEY	Unique setting ID
key	VARCHAR(255) UNIQUE	Setting key (e.g., smtp_server)
value	TEXT	Setting value (JSON for complex values)
updated_at	DATETIME	Last modification time

Example Settings:

• smtp_server - SMTP server hostname
• smtp_port - SMTP port (587, 465, 25)
• smtp_username - SMTP username
• smtp_password - SMTP password (encrypted)
• smtp_from_email - From email address
• smtp_to_emails - JSON array of recipient emails
• app_password - Single-user password hash (bcrypt)
• retention_days - How long to keep old scans (0 = forever)

API Design

REST API Endpoints

All API endpoints return JSON and follow RESTful conventions.

Scans

Method	Endpoint	Description	Request Body	Response
GET	/api/scans	List all scans (paginated)	-	{ "scans": [...], "total": N, "page": 1 }
GET	/api/scans/{id}	Get scan details	-	{ "scan": {...} }
POST	/api/scans	Trigger new scan	{ "config_file": "path" }	{ "scan_id": N, "status": "running" }
DELETE	/api/scans/{id}	Delete scan and files	-	{ "status": "deleted" }
GET	/api/scans/{id}/status	Get scan status	-	{ "status": "running", "progress": "45%" }
GET	/api/scans/{id1}/compare/{id2}	Compare two scans	-	{ "diff": {...} }

Schedules

Method	Endpoint	Description	Request Body	Response
GET	/api/schedules	List all schedules	-	{ "schedules": [...] }
GET	/api/schedules/{id}	Get schedule details	-	{ "schedule": {...} }
POST	/api/schedules	Create new schedule	{ "name": "...", "config_file": "...", "cron_expression": "..." }	{ "schedule_id": N }
PUT	/api/schedules/{id}	Update schedule	{ "enabled": true, "cron_expression": "..." }	{ "status": "updated" }
DELETE	/api/schedules/{id}	Delete schedule	-	{ "status": "deleted" }
POST	/api/schedules/{id}/trigger	Manually trigger scheduled scan	-	{ "scan_id": N }

Alerts

Method	Endpoint	Description	Request Body	Response
GET	/api/alerts	List recent alerts	-	{ "alerts": [...] }
GET	/api/alerts/rules	List alert rules	-	{ "rules": [...] }
POST	/api/alerts/rules	Create alert rule	{ "rule_type": "...", "threshold": N }	{ "rule_id": N }
PUT	/api/alerts/rules/{id}	Update alert rule	{ "enabled": false }	{ "status": "updated" }
DELETE	/api/alerts/rules/{id}	Delete alert rule	-	{ "status": "deleted" }

Settings

Method	Endpoint	Description	Request Body	Response
GET	/api/settings	Get all settings (sanitized)	-	{ "settings": {...} }
PUT	/api/settings	Update settings	{ "smtp_server": "...", ... }	{ "status": "updated" }
POST	/api/settings/test-email	Test email configuration	-	{ "status": "sent" }

Statistics & Trends

Method	Endpoint	Description	Request Body	Response
GET	/api/stats/summary	Dashboard summary stats	-	{ "total_scans": N, "last_scan": "...", ... }
GET	/api/stats/trends	Trend data for charts	?days=30&metric=port_count	{ "data": [...] }
GET	/api/stats/certificates	Certificate expiry overview	-	{ "expiring_soon": [...], "expired": [...] }

Authentication

Phase 2-3: Simple session-based authentication (single-user)

• Login endpoint: POST /api/auth/login (username/password)
• Logout endpoint: POST /api/auth/logout
• Session cookies with Flask-Login
• Password stored as bcrypt hash in settings table

Phase 6: API token authentication for CLI client

• Generate API token: POST /api/auth/token
• Revoke token: DELETE /api/auth/token
• CLI sends token in Authorization: Bearer <token> header

Phased Roadmap

Phase 1: Foundation ✅ COMPLETE

Completed: 2025-11-13

Deliverables:

• SQLite database with 11 tables (scans, sites, IPs, ports, services, certificates, TLS versions, schedules, alerts, alert_rules, settings)
• SQLAlchemy ORM models with relationships
• Alembic migration system
• Settings system with encryption (bcrypt for passwords, Fernet for sensitive data)
• Flask app structure with API blueprints
• Docker Compose deployment configuration
• Validation script for verification

---

Phase 2: Flask Web App Core ✅ COMPLETE

Completed: 2025-11-14

Deliverables:

• REST API with 8 endpoints (scans: trigger, list, get, status, delete; settings: get, update, test-email)
• Background job queue using APScheduler (up to 3 concurrent scans)
• Session-based authentication with Flask-Login
• Database integration for scan results (full normalized schema population)
• Web UI templates (dashboard, scan list/detail, login, error pages)
• Error handling with content negotiation (JSON/HTML) and request IDs
• Logging system with rotating file handlers
• Production Docker Compose deployment
• Comprehensive test suite (100 tests, all passing)
• Documentation (API_REFERENCE.md, DEPLOYMENT.md)

---

Phase 3: Dashboard & Scheduling ✅ COMPLETE

Completed: 2025-11-14

Deliverables:

• Dashboard with summary stats (total scans, IPs, ports, services)
• Recent scans table with clickable details
• Scan detail page with full results display
• Historical trend charts using Chart.js (port counts over time)
• Scheduled scan management UI (create, edit, delete, enable/disable)
• Schedule execution with APScheduler and cron expressions
• Manual scan trigger from web UI
• Navigation menu (Dashboard, Scans, Schedules, Configs, Settings)
• Download buttons for scan reports (JSON, HTML, ZIP)

---

Phase 4: Config Creator ✅ COMPLETE

Completed: 2025-11-17

Deliverables:

• CIDR-based config creation UI (simplified workflow for quick config generation)
• YAML editor with CodeMirror (syntax highlighting, line numbers)
• Config management UI (list, view, edit, download, delete)
• Direct YAML upload for advanced users
• REST API for config operations (7 endpoints: list, get, create, update, delete, upload, download)
• Schedule dependency protection (prevents deleting configs used by schedules)
• Comprehensive testing (25+ unit and integration tests)

---

Phase 5: Webhooks & Alerting ✅ COMPLETE

Completed: 2025-11-19

Goals:

• ✅ Implement webhook notification system for real-time alerting
• ✅ Add alert rule configuration for unexpected exposure detection
• ✅ Create notification template system for flexible alerting

Core Use Case: Monitor infrastructure for misconfigurations that expose unexpected ports/services to the world. When a scan detects an open port that wasn't defined in the YAML config's expected_ports list, trigger immediate notifications via webhooks.

Implemented Features:

1. Alert Rule Engine ✅

Purpose: Automatically detect and classify infrastructure anomalies after each scan.

Alert Types:

• unexpected_port - Port open but not in config's expected_ports list
• unexpected_service - Service detected that doesn't match expected service name
• cert_expiry - SSL/TLS certificate expiring soon (configurable threshold)
• ping_failed - Expected host not responding to ping
• service_down - Previously detected service no longer responding
• service_change - Service version/product changed between scans
• weak_tls - TLS 1.0/1.1 detected or weak cipher suites
• new_host - New IP address responding in CIDR range
• host_disappeared - Previously seen IP no longer responding

Alert Severity Levels:

• critical - Unexpected internet-facing service (ports 80/443/22/3389/etc.)
• warning - Minor configuration drift or upcoming cert expiry
• info - Informational alerts (new host discovered, service version change)

Alert Rule Configuration:

# Example alert rule configuration (stored in DB)
alert_rules:
  - id: 1
    rule_type: unexpected_port
    enabled: true
    severity: critical
    webhook_enabled: true
    filter_conditions:
      ports: [22, 80, 443, 3389, 3306, 5432, 27017]  # High-risk ports

  - id: 2
    rule_type: cert_expiry
    enabled: true
    severity: warning
    threshold: 30  # Days before expiry
    webhook_enabled: true

Implementation:

• ✅ Evaluate alert rules after each scan completes
• ✅ Compare current scan results to expected configuration
• ✅ Generate alerts and store in alerts table
• ✅ Trigger notifications based on rule configuration
• ✅ Alert deduplication (don't spam for same issue)

2. Webhook Notifications ✅

Purpose: Real-time HTTP POST notifications for integration with external systems (Slack, PagerDuty, custom dashboards, SIEM tools).

Webhook Configuration (via Settings API):

{
  "webhook_enabled": true,
  "webhook_urls": [
    {
      "id": 1,
      "name": "Slack Security Channel",
      "url": "https://hooks.slack.com/services/XXX/YYY/ZZZ",
      "enabled": true,
      "auth_type": "none",
      "custom_headers": {},
      "alert_types": ["unexpected_port", "unexpected_service", "weak_tls"],
      "severity_filter": ["critical", "warning"]
    },
    {
      "id": 2,
      "name": "PagerDuty",
      "url": "https://events.pagerduty.com/v2/enqueue",
      "enabled": true,
      "auth_type": "bearer",
      "auth_token": "encrypted_token",
      "custom_headers": {
        "Content-Type": "application/json"
      },
      "alert_types": ["unexpected_port"],
      "severity_filter": ["critical"]
    }
  ]
}

Webhook Payload Format (JSON):

{
  "event_type": "scan_alert",
  "alert_id": 42,
  "alert_type": "unexpected_port",
  "severity": "critical",
  "timestamp": "2025-11-17T14:23:45Z",
  "scan": {
    "scan_id": 123,
    "title": "Production Network Scan",
    "timestamp": "2025-11-17T14:15:00Z",
    "config_file": "prod_config.yaml",
    "triggered_by": "scheduled"
  },
  "alert_details": {
    "message": "Unexpected port 3306 (MySQL) exposed on 192.168.1.100",
    "ip_address": "192.168.1.100",
    "port": 3306,
    "protocol": "tcp",
    "state": "open",
    "service": {
      "name": "mysql",
      "product": "MySQL",
      "version": "8.0.32"
    },
    "expected": false,
    "site_name": "Production Servers"
  },
  "recommended_actions": [
    "Verify if MySQL should be exposed externally",
    "Check firewall rules for 192.168.1.100",
    "Review MySQL bind-address configuration"
  ],
  "web_url": "https://sneakyscanner.local/scans/123"
}

Webhook Features:

• ✅ Multiple webhook URLs with independent configuration
• ✅ Per-webhook filtering by alert type and severity
• ✅ Custom headers support (for API keys, auth tokens)
• ✅ Authentication methods:
  • none - No authentication
  • bearer - Bearer token in Authorization header
  • basic - Basic authentication
  • custom - Custom header-based auth
• ✅ Retry logic with exponential backoff (3 attempts)
• ✅ Webhook delivery tracking (webhook_sent, webhook_sent_at, webhook_response_code)
• ✅ Test webhook functionality in Settings UI
• ✅ Timeout configuration (default 10 seconds)
• ✅ Webhook delivery history and logs

Webhook API Endpoints:

• ✅ POST /api/webhooks - Create webhook configuration
• ✅ GET /api/webhooks - List all webhooks
• ✅ PUT /api/webhooks/{id} - Update webhook configuration
• ✅ DELETE /api/webhooks/{id} - Delete webhook
• ✅ POST /api/webhooks/{id}/test - Send test webhook
• ✅ GET /api/webhooks/{id}/history - Get delivery history

Notification Templates: Flexible template system supporting multiple platforms (Slack, Discord, PagerDuty, etc.):

{
  "text": "SneakyScanner Alert: Unexpected Port Detected",
  "attachments": [
    {
      "color": "danger",
      "fields": [
        {"title": "IP Address", "value": "192.168.1.100", "short": true},
        {"title": "Port", "value": "3306/tcp", "short": true},
        {"title": "Service", "value": "MySQL 8.0.32", "short": true},
        {"title": "Severity", "value": "CRITICAL", "short": true}
      ],
      "footer": "SneakyScanner",
      "ts": 1700234625
    }
  ]
}

---

Deliverables:

• ✅ Alert Rule Engine with 9 alert types (unexpected_port, unexpected_service, cert_expiry, ping_failed, service_down, service_change, weak_tls, new_host, host_disappeared)
• ✅ Alert severity classification (critical, warning, info)
• ✅ Alert rule configuration API (CRUD operations)
• ✅ Webhook notification system with retry logic
• ✅ Multiple webhook URL support with independent configuration
• ✅ Notification template system for flexible platform integration (Slack, Discord, PagerDuty, custom)
• ✅ Webhook API endpoints (create, list, update, delete, test, history)
• ✅ Custom headers and authentication support (none, bearer, basic, custom)
• ✅ Webhook delivery tracking and logging
• ✅ Alert deduplication to prevent notification spam
• ✅ Integration with scan completion workflow

Success Criteria Met:

• ✅ Alerts triggered within 30 seconds of scan completion
• ✅ Webhook POST delivered with retry on failure
• ✅ Zero false positives for expected ports/services
• ✅ Alert deduplication prevents notification spam

---

Phase 6: CLI as API Client

Status: Planned Priority: MEDIUM

Goals:

• Create CLI API client for scripting and automation
• Maintain standalone mode for testing
• API token authentication

Planned Features:

1. API Client Mode:

   • --api-mode flag to enable API client mode
   • --api-url and --api-token arguments
   • Trigger scans via API, poll for status, download results
   • Scans stored centrally in database
   • Standalone mode still available for testing

2. API Token System:

   • Token generation UI in settings page
   • Secure token storage (hashed in database)
   • Token authentication middleware
   • Token expiration and revocation

3. Benefits:

   • Centralized scan history accessible via web dashboard
   • No need to mount volumes for output
   • Scheduled scans managed through web UI
   • Scriptable automation while leveraging web features

---

Phase 7: Advanced Features

Status: Future/Deferred Priority: LOW

Planned Features:

1. Email Notifications:

   • SMTP integration with Flask-Mail
   • Jinja2 email templates (HTML + plain text)
   • Settings API for email configuration
   • Test email functionality
   • Email delivery tracking
   • Rate limiting to prevent email flood
   • Configurable recipients (multiple emails)

2. Scan Comparison:

   • Compare two scans API endpoint
   • Side-by-side comparison view
   • Color-coded differences (green=new, red=removed, yellow=changed)
   • Filter by change type
   • Export comparison report to PDF/HTML
   • "Compare with previous scan" button on scan detail page

3. Enhanced Reports:

   • Sortable/filterable tables (DataTables.js)
   • Inline screenshot thumbnails with lightbox
   • PDF export (WeasyPrint)

4. Vulnerability Detection:

   • CVE database integration (NVD API)
   • Service version matching to known CVEs
   • CVSS severity scores
   • Alert rules for critical CVEs

5. Timeline View:

   • Visual scan history timeline
   • Filter by site/IP
   • Event annotations

6. Advanced Charts:

   • Port activity heatmap
   • Service version tracking
   • Certificate expiration forecast

7. Additional Integrations:

   • Prometheus metrics export
   • CSV export/import
   • Advanced reporting dashboards

---

Development Workflow

Iteration Cycle

1. Plan - Define features for phase
2. Implement - Code backend + frontend
3. Test - Unit tests + manual testing
4. Deploy - Update Docker Compose
5. Document - Update README.md, ROADMAP.md
6. Review - Get user feedback
7. Iterate - Adjust priorities based on feedback

Git Workflow

• main branch - Stable releases
• develop branch - Active development
• feature branches - Individual features (feature/dashboard, feature/scheduler)
• Pull requests - Review before merge

Testing Strategy

• Unit tests - pytest for models, API endpoints
• Integration tests - Full scan → DB → API workflow
• Manual testing - UI/UX testing in browser
• Performance tests - Large scans, database queries

Documentation

• README.md - User-facing documentation (updated each phase)
• ROADMAP.md - This file (updated as priorities shift)
• CLAUDE.md - Developer documentation (architecture, code references)
• API.md - API documentation (OpenAPI/Swagger in Phase 4)

Resources & References

Documentation

• Flask Documentation
• SQLAlchemy ORM
• APScheduler
• Chart.js
• Bootstrap 5

Tutorials

• Flask Mega-Tutorial
• SQLAlchemy Tutorial
• APScheduler with Flask

Similar Projects (Inspiration)

• OpenVAS - Vulnerability scanner with web UI
• Nessus - Commercial scanner (inspiration for UI/UX)
• OWASP ZAP - Web app scanner (comparison reports, alerts)

Changelog

Date	Version	Changes
2025-11-14	1.0	Initial roadmap created based on user requirements
2025-11-13	1.1	Phase 1 COMPLETE - Database schema, SQLAlchemy models, Flask app structure, settings system with encryption, Alembic migrations, API blueprints, Docker support, validation script
2025-11-14	1.2	Phase 2 COMPLETE - REST API (5 scan endpoints, 3 settings endpoints), background jobs (APScheduler), authentication (Flask-Login), web UI (dashboard, scans, login, errors), error handling (content negotiation, request IDs, logging), 100 tests passing, comprehensive documentation (API_REFERENCE.md, DEPLOYMENT.md, PHASE2_COMPLETE.md)
2025-11-17	1.3	Bug Fix - Fixed Chart.js infinite canvas growth issue in scan detail page (duplicate initialization, missing chart.destroy(), missing fixed-height container)
2025-11-17	1.4	Phase 4 COMPLETE - Config Creator with CIDR-based creation, YAML editor (CodeMirror), config management UI (list/edit/delete), REST API (7 endpoints), Docker volume permissions fix, comprehensive testing and documentation
2025-11-17	1.5	Roadmap Compression - Condensed completed phases (1-4) into concise summaries, updated project scope to emphasize web GUI frontend with CLI as API client coming soon (Phase 6), reorganized phases for clarity
2025-11-19	1.6	Phase 5 Progress - Completed webhooks, notification templates, and alerting rules. Alert Rule Engine and Webhook System implemented.
2025-11-19	1.7	Phase 5 COMPLETE - Webhooks & Alerting phase completed. Moved Email Notifications and Scan Comparison to Phase 7. Alert rules, webhook notifications, and notification templates fully implemented and tested.

---

Last Updated: 2025-11-19 Next Review: Before Phase 6 kickoff (CLI as API Client)