# SneakyScanner Roadmap

## Vision & Goals

SneakyScanner is a comprehensive **Flask web application** for infrastructure monitoring and security auditing. The primary interface is the web GUI, with a CLI API client planned for scripting and automation needs.

**Status:** Phase 4 Complete ✅ | Phase 5 Next Up

## Progress Overview
- ✅ **Phase 1: Foundation** - Complete (2025-11-13)
  - Database schema, SQLAlchemy models, settings system, Flask app structure
- ✅ **Phase 2: Flask Web App Core** - Complete (2025-11-14)
  - REST API, background jobs, authentication, web UI, testing (100 tests)
- ✅ **Phase 3: Dashboard & Scheduling** - Complete (2025-11-14)
  - Dashboard, scan history, scheduled scans, trend charts
- ✅ **Phase 4: Config Creator** - Complete (2025-11-17)
  - CIDR-based config creation, YAML editor, config management UI
- 📋 **Phase 5: Email, Webhooks & Comparisons** - Next Up
  - Email notifications, alert rules, scan comparison
- 📋 **Phase 6: CLI as API Client** - Planned
  - CLI for scripting and automation via API
- 📋 **Phase 7: Advanced Features** - Future
  - CVE integration, timeline view, PDF export, enhanced reports



**Core Features:**
- **Centralized dashboard** for viewing scan history and trends
- **Scheduled scanning** for continuous infrastructure monitoring
- **Email notifications** for critical changes and certificate expirations (coming soon)
- **Historical analysis** with charts, graphs, and comparison reports
- **Config creator** for easy CIDR-based scan configuration
- **RESTful API** for integration and automation
- **Simple deployment** using SQLite3 (single-user, self-hosted)

**Planned:**
- **CLI API client** for scripting and automation workflows (Phase 6)

## Target Users

- **Infrastructure teams** monitoring on-premises networks
- **Security professionals** performing periodic security audits
- **DevOps engineers** tracking infrastructure drift
- **Single users or small teams** (not enterprise multi-tenant)

## Technology Stack

### Backend
- **Flask** 3.x - Lightweight Python web framework
- **SQLAlchemy** 2.x - ORM for database abstraction
- **SQLite3** - Embedded database (easy deployment, sufficient for single-user)
- **APScheduler** 3.x - Background job scheduling for periodic scans
- **Flask-Login** - Simple session-based authentication
- **Flask-CORS** - API access control (optional for CLI API client)
- **Marshmallow** - API serialization/deserialization

### Frontend
- **Jinja2** - Server-side templating (already in use)
- **Bootstrap 5** - Responsive UI framework with dark theme support
- **Chart.js** 4.x - Lightweight charting library for trends
- **DataTables.js** - Interactive sortable/filterable tables (Phase 6)
- **Vanilla JavaScript** - Keep dependencies minimal

### Infrastructure
- **Docker Compose** - Multi-container orchestration (Flask app + Scanner)
- **Gunicorn** - WSGI server for production
- **Nginx** - Reverse proxy (optional, for production deployments)

### Existing Components (Keep)
- **Masscan** - Fast port discovery
- **Nmap** - Service detection
- **Playwright** - Screenshot capture
- **sslyze** - SSL/TLS analysis

## Architecture Overview

```
┌─────────────────────────────────────────────────────────────┐
│                     Flask Web Application                   │
│  ┌──────────────┐  ┌──────────────┐  ┌──────────────────┐  │
│  │   Web UI     │  │   REST API   │  │   Scheduler      │  │
│  │  (Dashboard) │  │ (JSON/CRUD)  │  │  (APScheduler)   │  │
│  └──────┬───────┘  └──────┬───────┘  └────────┬─────────┘  │
│         │                 │                    │            │
│         └─────────────────┴────────────────────┘            │
│                           │                                 │
│                  ┌────────▼────────┐                        │
│                  │   SQLAlchemy    │                        │
│                  │   (ORM Layer)   │                        │
│                  └────────┬────────┘                        │
│                           │                                 │
│                  ┌────────▼────────┐                        │
│                  │   SQLite3 DB    │                        │
│                  │  (scan history) │                        │
│                  └─────────────────┘                        │
└───────────────────────────┬─────────────────────────────────┘
                            │
                 ┌──────────▼──────────┐
                 │  Scanner Engine     │
                 │  (scanner.py)       │
                 │  ┌────────────────┐ │
                 │  │ Masscan/Nmap   │ │
                 │  │ Playwright     │ │
                 │  │ sslyze         │ │
                 │  └────────────────┘ │
                 └─────────────────────┘
                            │
                 ┌──────────▼──────────┐
                 │  CLI API Client     │
                 │  (optional future)  │
                 └─────────────────────┘
```

### Component Interaction

1. **Web UI** - User interacts with dashboard, triggers scans, views history
2. **REST API** - Handles requests from web UI and CLI client
3. **Scheduler (APScheduler)** - Triggers scans based on cron-like schedules
4. **SQLAlchemy ORM** - Abstracts database operations
5. **SQLite3 Database** - Stores scan results, schedules, settings, alerts
6. **Scanner Engine** - Performs actual network scanning (masscan, nmap, etc.)
7. **CLI API Client** - Future: thin client that calls Flask API

## Database Schema Design

### Core Tables

#### `scans`
Stores metadata about each scan execution.

| Column | Type | Description |
|--------|------|-------------|
| `id` | INTEGER PRIMARY KEY | Unique scan ID |
| `timestamp` | DATETIME | Scan start time (UTC) |
| `duration` | FLOAT | Total scan duration (seconds) |
| `status` | VARCHAR(20) | `running`, `completed`, `failed` |
| `config_file` | TEXT | Path to YAML config used |
| `title` | TEXT | Scan title from config |
| `json_path` | TEXT | Path to JSON report |
| `html_path` | TEXT | Path to HTML report |
| `zip_path` | TEXT | Path to ZIP archive |
| `screenshot_dir` | TEXT | Path to screenshot directory |
| `created_at` | DATETIME | Record creation time |
| `triggered_by` | VARCHAR(50) | `manual`, `scheduled`, `api` |
| `schedule_id` | INTEGER | FK to schedules (if triggered by schedule) |

#### `scan_sites`
Logical grouping of IPs by site.

| Column | Type | Description |
|--------|------|-------------|
| `id` | INTEGER PRIMARY KEY | Unique site record ID |
| `scan_id` | INTEGER | FK to scans |
| `site_name` | VARCHAR(255) | Site name from config |

#### `scan_ips`
IP addresses scanned in each scan.

| Column | Type | Description |
|--------|------|-------------|
| `id` | INTEGER PRIMARY KEY | Unique IP record ID |
| `scan_id` | INTEGER | FK to scans |
| `site_id` | INTEGER | FK to scan_sites |
| `ip_address` | VARCHAR(45) | IPv4 or IPv6 address |
| `ping_expected` | BOOLEAN | Expected ping response |
| `ping_actual` | BOOLEAN | Actual ping response |

#### `scan_ports`
Discovered TCP/UDP ports.

| Column | Type | Description |
|--------|------|-------------|
| `id` | INTEGER PRIMARY KEY | Unique port record ID |
| `scan_id` | INTEGER | FK to scans |
| `ip_id` | INTEGER | FK to scan_ips |
| `port` | INTEGER | Port number (1-65535) |
| `protocol` | VARCHAR(10) | `tcp` or `udp` |
| `expected` | BOOLEAN | Was this port expected? |
| `state` | VARCHAR(20) | `open`, `closed`, `filtered` |

#### `scan_services`
Detected services on open ports.

| Column | Type | Description |
|--------|------|-------------|
| `id` | INTEGER PRIMARY KEY | Unique service record ID |
| `scan_id` | INTEGER | FK to scans |
| `port_id` | INTEGER | FK to scan_ports |
| `service_name` | VARCHAR(100) | Service name (e.g., `ssh`, `http`) |
| `product` | VARCHAR(255) | Product name (e.g., `OpenSSH`) |
| `version` | VARCHAR(100) | Version string |
| `extrainfo` | TEXT | Additional nmap info |
| `ostype` | VARCHAR(100) | OS type if detected |
| `http_protocol` | VARCHAR(10) | `http` or `https` (if web service) |
| `screenshot_path` | TEXT | Relative path to screenshot |

#### `scan_certificates`
SSL/TLS certificates discovered on HTTPS services.

| Column | Type | Description |
|--------|------|-------------|
| `id` | INTEGER PRIMARY KEY | Unique certificate record ID |
| `scan_id` | INTEGER | FK to scans |
| `service_id` | INTEGER | FK to scan_services |
| `subject` | TEXT | Certificate subject (CN) |
| `issuer` | TEXT | Certificate issuer |
| `serial_number` | TEXT | Serial number |
| `not_valid_before` | DATETIME | Validity start date |
| `not_valid_after` | DATETIME | Validity end date |
| `days_until_expiry` | INTEGER | Days until expiration |
| `sans` | TEXT | JSON array of SANs |
| `is_self_signed` | BOOLEAN | Self-signed certificate flag |

#### `scan_tls_versions`
TLS version support and cipher suites.

| Column | Type | Description |
|--------|------|-------------|
| `id` | INTEGER PRIMARY KEY | Unique TLS version record ID |
| `scan_id` | INTEGER | FK to scans |
| `certificate_id` | INTEGER | FK to scan_certificates |
| `tls_version` | VARCHAR(20) | `TLS 1.0`, `TLS 1.1`, `TLS 1.2`, `TLS 1.3` |
| `supported` | BOOLEAN | Is this version supported? |
| `cipher_suites` | TEXT | JSON array of cipher suites |

### Scheduling & Notifications Tables

#### `schedules`
Scheduled scan configurations.

| Column | Type | Description |
|--------|------|-------------|
| `id` | INTEGER PRIMARY KEY | Unique schedule ID |
| `name` | VARCHAR(255) | Schedule name (e.g., "Daily prod scan") |
| `config_file` | TEXT | Path to YAML config |
| `cron_expression` | VARCHAR(100) | Cron-like schedule (e.g., `0 2 * * *`) |
| `enabled` | BOOLEAN | Is schedule active? |
| `last_run` | DATETIME | Last execution time |
| `next_run` | DATETIME | Next scheduled execution |
| `created_at` | DATETIME | Schedule creation time |
| `updated_at` | DATETIME | Last modification time |

#### `alerts`
Alert history and notifications sent.

| Column | Type | Description |
|--------|------|-------------|
| `id` | INTEGER PRIMARY KEY | Unique alert ID |
| `scan_id` | INTEGER | FK to scans |
| `alert_type` | VARCHAR(50) | `new_port`, `cert_expiry`, `service_change`, `ping_failed` |
| `severity` | VARCHAR(20) | `info`, `warning`, `critical` |
| `message` | TEXT | Human-readable alert message |
| `ip_address` | VARCHAR(45) | Related IP (optional) |
| `port` | INTEGER | Related port (optional) |
| `email_sent` | BOOLEAN | Was email notification sent? |
| `email_sent_at` | DATETIME | Email send timestamp |
| `created_at` | DATETIME | Alert creation time |

#### `alert_rules`
User-defined alert rules.

| Column | Type | Description |
|--------|------|-------------|
| `id` | INTEGER PRIMARY KEY | Unique rule ID |
| `rule_type` | VARCHAR(50) | `unexpected_port`, `cert_expiry`, `service_down`, etc. |
| `enabled` | BOOLEAN | Is rule active? |
| `threshold` | INTEGER | Threshold value (e.g., days for cert expiry) |
| `email_enabled` | BOOLEAN | Send email for this rule? |
| `created_at` | DATETIME | Rule creation time |

### Settings Table

#### `settings`
Application configuration key-value store.

| Column | Type | Description |
|--------|------|-------------|
| `id` | INTEGER PRIMARY KEY | Unique setting ID |
| `key` | VARCHAR(255) UNIQUE | Setting key (e.g., `smtp_server`) |
| `value` | TEXT | Setting value (JSON for complex values) |
| `updated_at` | DATETIME | Last modification time |

**Example Settings:**
- `smtp_server` - SMTP server hostname
- `smtp_port` - SMTP port (587, 465, 25)
- `smtp_username` - SMTP username
- `smtp_password` - SMTP password (encrypted)
- `smtp_from_email` - From email address
- `smtp_to_emails` - JSON array of recipient emails
- `app_password` - Single-user password hash (bcrypt)
- `retention_days` - How long to keep old scans (0 = forever)

## API Design

### REST API Endpoints

All API endpoints return JSON and follow RESTful conventions.

#### Scans

| Method | Endpoint | Description | Request Body | Response |
|--------|----------|-------------|--------------|----------|
| `GET` | `/api/scans` | List all scans (paginated) | - | `{ "scans": [...], "total": N, "page": 1 }` |
| `GET` | `/api/scans/{id}` | Get scan details | - | `{ "scan": {...} }` |
| `POST` | `/api/scans` | Trigger new scan | `{ "config_file": "path" }` | `{ "scan_id": N, "status": "running" }` |
| `DELETE` | `/api/scans/{id}` | Delete scan and files | - | `{ "status": "deleted" }` |
| `GET` | `/api/scans/{id}/status` | Get scan status | - | `{ "status": "running", "progress": "45%" }` |
| `GET` | `/api/scans/{id1}/compare/{id2}` | Compare two scans | - | `{ "diff": {...} }` |

#### Schedules

| Method | Endpoint | Description | Request Body | Response |
|--------|----------|-------------|--------------|----------|
| `GET` | `/api/schedules` | List all schedules | - | `{ "schedules": [...] }` |
| `GET` | `/api/schedules/{id}` | Get schedule details | - | `{ "schedule": {...} }` |
| `POST` | `/api/schedules` | Create new schedule | `{ "name": "...", "config_file": "...", "cron_expression": "..." }` | `{ "schedule_id": N }` |
| `PUT` | `/api/schedules/{id}` | Update schedule | `{ "enabled": true, "cron_expression": "..." }` | `{ "status": "updated" }` |
| `DELETE` | `/api/schedules/{id}` | Delete schedule | - | `{ "status": "deleted" }` |
| `POST` | `/api/schedules/{id}/trigger` | Manually trigger scheduled scan | - | `{ "scan_id": N }` |

#### Alerts

| Method | Endpoint | Description | Request Body | Response |
|--------|----------|-------------|--------------|----------|
| `GET` | `/api/alerts` | List recent alerts | - | `{ "alerts": [...] }` |
| `GET` | `/api/alerts/rules` | List alert rules | - | `{ "rules": [...] }` |
| `POST` | `/api/alerts/rules` | Create alert rule | `{ "rule_type": "...", "threshold": N }` | `{ "rule_id": N }` |
| `PUT` | `/api/alerts/rules/{id}` | Update alert rule | `{ "enabled": false }` | `{ "status": "updated" }` |
| `DELETE` | `/api/alerts/rules/{id}` | Delete alert rule | - | `{ "status": "deleted" }` |

#### Settings

| Method | Endpoint | Description | Request Body | Response |
|--------|----------|-------------|--------------|----------|
| `GET` | `/api/settings` | Get all settings (sanitized) | - | `{ "settings": {...} }` |
| `PUT` | `/api/settings` | Update settings | `{ "smtp_server": "...", ... }` | `{ "status": "updated" }` |
| `POST` | `/api/settings/test-email` | Test email configuration | - | `{ "status": "sent" }` |

#### Statistics & Trends

| Method | Endpoint | Description | Request Body | Response |
|--------|----------|-------------|--------------|----------|
| `GET` | `/api/stats/summary` | Dashboard summary stats | - | `{ "total_scans": N, "last_scan": "...", ... }` |
| `GET` | `/api/stats/trends` | Trend data for charts | `?days=30&metric=port_count` | `{ "data": [...] }` |
| `GET` | `/api/stats/certificates` | Certificate expiry overview | - | `{ "expiring_soon": [...], "expired": [...] }` |

### Authentication

**Phase 2-3:** Simple session-based authentication (single-user)
- Login endpoint: `POST /api/auth/login` (username/password)
- Logout endpoint: `POST /api/auth/logout`
- Session cookies with Flask-Login
- Password stored as bcrypt hash in settings table

**Phase 5:** API token authentication for CLI client
- Generate API token: `POST /api/auth/token`
- Revoke token: `DELETE /api/auth/token`
- CLI sends token in `Authorization: Bearer <token>` header

## Phased Roadmap

### Phase 1: Foundation ✅ COMPLETE
**Completed:** 2025-11-13

**Deliverables:**
- SQLite database with 11 tables (scans, sites, IPs, ports, services, certificates, TLS versions, schedules, alerts, alert_rules, settings)
- SQLAlchemy ORM models with relationships
- Alembic migration system
- Settings system with encryption (bcrypt for passwords, Fernet for sensitive data)
- Flask app structure with API blueprints
- Docker Compose deployment configuration
- Validation script for verification

---

### Phase 2: Flask Web App Core ✅ COMPLETE
**Completed:** 2025-11-14

**Deliverables:**
- REST API with 8 endpoints (scans: trigger, list, get, status, delete; settings: get, update, test-email)
- Background job queue using APScheduler (up to 3 concurrent scans)
- Session-based authentication with Flask-Login
- Database integration for scan results (full normalized schema population)
- Web UI templates (dashboard, scan list/detail, login, error pages)
- Error handling with content negotiation (JSON/HTML) and request IDs
- Logging system with rotating file handlers
- Production Docker Compose deployment
- Comprehensive test suite (100 tests, all passing)
- Documentation (API_REFERENCE.md, DEPLOYMENT.md)

---

### Phase 3: Dashboard & Scheduling ✅ COMPLETE
**Completed:** 2025-11-14

**Deliverables:**
- Dashboard with summary stats (total scans, IPs, ports, services)
- Recent scans table with clickable details
- Scan detail page with full results display
- Historical trend charts using Chart.js (port counts over time)
- Scheduled scan management UI (create, edit, delete, enable/disable)
- Schedule execution with APScheduler and cron expressions
- Manual scan trigger from web UI
- Navigation menu (Dashboard, Scans, Schedules, Configs, Settings)
- Download buttons for scan reports (JSON, HTML, ZIP)

---

### Phase 4: Config Creator ✅ COMPLETE
**Completed:** 2025-11-17

**Deliverables:**
- CIDR-based config creation UI (simplified workflow for quick config generation)
- YAML editor with CodeMirror (syntax highlighting, line numbers)
- Config management UI (list, view, edit, download, delete)
- Direct YAML upload for advanced users
- REST API for config operations (7 endpoints: list, get, create, update, delete, upload, download)
- Schedule dependency protection (prevents deleting configs used by schedules)
- Comprehensive testing (25+ unit and integration tests)

---

### Phase 5: Email, Webhooks & Comparisons
**Status:** Next Up
**Priority:** MEDIUM

**Goals:**
- Implement email notification system for infrastructure misconfigurations
- Implement webhook notification system for real-time alerting
- Create scan comparison reports to detect drift
- Add alert rule configuration for unexpected exposure detection

**Core Use Case:**
Monitor infrastructure for misconfigurations that expose unexpected ports/services to the world. When a scan detects an open port that wasn't defined in the YAML config's `expected_ports` list, trigger immediate notifications via email and/or webhooks.

**Planned Features:**

#### 1. Alert Rule Engine
**Purpose:** Automatically detect and classify infrastructure anomalies after each scan.

**Alert Types:**
- `unexpected_port` - Port open but not in config's `expected_ports` list
- `unexpected_service` - Service detected that doesn't match expected service name
- `cert_expiry` - SSL/TLS certificate expiring soon (configurable threshold)
- `ping_failed` - Expected host not responding to ping
- `service_down` - Previously detected service no longer responding
- `service_change` - Service version/product changed between scans
- `weak_tls` - TLS 1.0/1.1 detected or weak cipher suites
- `new_host` - New IP address responding in CIDR range
- `host_disappeared` - Previously seen IP no longer responding

**Alert Severity Levels:**
- `critical` - Unexpected internet-facing service (ports 80/443/22/3389/etc.)
- `warning` - Minor configuration drift or upcoming cert expiry
- `info` - Informational alerts (new host discovered, service version change)

**Alert Rule Configuration:**
```yaml
# Example alert rule configuration (stored in DB)
alert_rules:
  - id: 1
    rule_type: unexpected_port
    enabled: true
    severity: critical
    email_enabled: true
    webhook_enabled: true
    filter_conditions:
      ports: [22, 80, 443, 3389, 3306, 5432, 27017]  # High-risk ports

  - id: 2
    rule_type: cert_expiry
    enabled: true
    severity: warning
    threshold: 30  # Days before expiry
    email_enabled: true
    webhook_enabled: false
```

**Implementation:**
- Evaluate alert rules after each scan completes
- Compare current scan results to expected configuration
- Generate alerts and store in `alerts` table
- Trigger notifications based on rule configuration
- Alert deduplication (don't spam for same issue)

#### 2. Email Notifications
**Purpose:** Send detailed email alerts when infrastructure misconfigurations are detected.

**SMTP Configuration (via Settings API):**
```json
{
  "smtp_server": "smtp.gmail.com",
  "smtp_port": 587,
  "smtp_use_tls": true,
  "smtp_username": "alerts@example.com",
  "smtp_password": "encrypted_password",
  "smtp_from_email": "SneakyScanner <alerts@example.com>",
  "smtp_to_emails": ["admin@example.com", "security@example.com"],
  "email_alerts_enabled": true
}
```

**Email Template (Jinja2 HTML):**
```html
Subject: [SneakyScanner Alert] Unexpected Port Detected - {{ ip_address }}:{{ port }}

Body:
===============================================
SneakyScanner Security Alert
===============================================

Alert Type: {{ alert_type }}
Severity: {{ severity }}
Scan ID: {{ scan_id }}
Timestamp: {{ timestamp }}

Issue Detected:
{{ message }}

Details:
- IP Address: {{ ip_address }}
- Port: {{ port }}/{{ protocol }}
- Service: {{ service_name }} ({{ product }} {{ version }})
- Expected: No (not in expected_ports list)

Recommended Actions:
1. Verify if this service should be exposed
2. Check firewall rules for {{ ip_address }}
3. Review service configuration
4. Update scan config if this is intentional

View Full Scan Results:
{{ web_url }}/scans/{{ scan_id }}

===============================================
```

**Email Features:**
- HTML email with styled alert box (Bootstrap-based)
- Plain-text fallback for compatibility
- Alert summary with actionable recommendations
- Direct link to scan detail page
- Configurable recipients (multiple emails)
- Test email functionality in Settings UI
- Email delivery tracking (email_sent, email_sent_at in alerts table)
- Rate limiting to prevent email flood

**Email API Endpoints:**
- `POST /api/settings/email` - Configure SMTP settings
- `POST /api/settings/email/test` - Send test email
- `GET /api/alerts?email_sent=true` - Get alerts with email status

#### 3. Webhook Notifications
**Purpose:** Real-time HTTP POST notifications for integration with external systems (Slack, PagerDuty, custom dashboards, SIEM tools).

**Webhook Configuration (via Settings API):**
```json
{
  "webhook_enabled": true,
  "webhook_urls": [
    {
      "id": 1,
      "name": "Slack Security Channel",
      "url": "https://hooks.slack.com/services/XXX/YYY/ZZZ",
      "enabled": true,
      "auth_type": "none",
      "custom_headers": {},
      "alert_types": ["unexpected_port", "unexpected_service", "weak_tls"],
      "severity_filter": ["critical", "warning"]
    },
    {
      "id": 2,
      "name": "PagerDuty",
      "url": "https://events.pagerduty.com/v2/enqueue",
      "enabled": true,
      "auth_type": "bearer",
      "auth_token": "encrypted_token",
      "custom_headers": {
        "Content-Type": "application/json"
      },
      "alert_types": ["unexpected_port"],
      "severity_filter": ["critical"]
    }
  ]
}
```

**Webhook Payload Format (JSON):**
```json
{
  "event_type": "scan_alert",
  "alert_id": 42,
  "alert_type": "unexpected_port",
  "severity": "critical",
  "timestamp": "2025-11-17T14:23:45Z",
  "scan": {
    "scan_id": 123,
    "title": "Production Network Scan",
    "timestamp": "2025-11-17T14:15:00Z",
    "config_file": "prod_config.yaml",
    "triggered_by": "scheduled"
  },
  "alert_details": {
    "message": "Unexpected port 3306 (MySQL) exposed on 192.168.1.100",
    "ip_address": "192.168.1.100",
    "port": 3306,
    "protocol": "tcp",
    "state": "open",
    "service": {
      "name": "mysql",
      "product": "MySQL",
      "version": "8.0.32"
    },
    "expected": false,
    "site_name": "Production Servers"
  },
  "recommended_actions": [
    "Verify if MySQL should be exposed externally",
    "Check firewall rules for 192.168.1.100",
    "Review MySQL bind-address configuration"
  ],
  "web_url": "https://sneakyscanner.local/scans/123"
}
```

**Webhook Features:**
- Multiple webhook URLs with independent configuration
- Per-webhook filtering by alert type and severity
- Custom headers support (for API keys, auth tokens)
- Authentication methods:
  - `none` - No authentication
  - `bearer` - Bearer token in Authorization header
  - `basic` - Basic authentication
  - `custom` - Custom header-based auth
- Retry logic with exponential backoff (3 attempts)
- Webhook delivery tracking (webhook_sent, webhook_sent_at, webhook_response_code)
- Test webhook functionality in Settings UI
- Timeout configuration (default 10 seconds)
- Webhook delivery history and logs

**Webhook API Endpoints:**
- `POST /api/webhooks` - Create webhook configuration
- `GET /api/webhooks` - List all webhooks
- `PUT /api/webhooks/{id}` - Update webhook configuration
- `DELETE /api/webhooks/{id}` - Delete webhook
- `POST /api/webhooks/{id}/test` - Send test webhook
- `GET /api/webhooks/{id}/history` - Get delivery history

**Slack Integration Example:**
Transform webhook payload to Slack message format:
```json
{
  "text": "SneakyScanner Alert: Unexpected Port Detected",
  "attachments": [
    {
      "color": "danger",
      "fields": [
        {"title": "IP Address", "value": "192.168.1.100", "short": true},
        {"title": "Port", "value": "3306/tcp", "short": true},
        {"title": "Service", "value": "MySQL 8.0.32", "short": true},
        {"title": "Severity", "value": "CRITICAL", "short": true}
      ],
      "footer": "SneakyScanner",
      "ts": 1700234625
    }
  ]
}
```

#### 4. Alert Management UI
**Purpose:** Web interface for configuring alert rules, viewing alert history, and managing notifications.

**Pages:**
- `/alerts` - Alert history with filtering and search
- `/alerts/rules` - Alert rule configuration
- `/settings/email` - Email notification settings
- `/settings/webhooks` - Webhook configuration

**Alert History Features:**
- Filter by alert type, severity, date range, IP address
- Search by message content
- Bulk acknowledge/dismiss alerts
- Export alerts to CSV
- Alert detail modal with full context

**Alert Rule UI Features:**
- Enable/disable rules individually
- Configure severity levels
- Set thresholds (e.g., cert expiry days)
- Toggle email/webhook per rule
- Test rules against recent scans

#### 5. Scan Comparison
**Purpose:** Detect infrastructure drift by comparing two scans and highlighting changes.

**Comparison API:**
- `GET /api/scans/{id1}/compare/{id2}` - Compare two scans

**Comparison Response:**
```json
{
  "scan1": {"id": 100, "timestamp": "2025-11-10T10:00:00Z"},
  "scan2": {"id": 123, "timestamp": "2025-11-17T14:15:00Z"},
  "summary": {
    "new_ports": 2,
    "removed_ports": 0,
    "service_changes": 1,
    "cert_changes": 0,
    "new_hosts": 1,
    "removed_hosts": 0
  },
  "differences": {
    "new_ports": [
      {"ip": "192.168.1.100", "port": 3306, "service": "mysql"}
    ],
    "removed_ports": [],
    "service_changes": [
      {
        "ip": "192.168.1.50",
        "port": 22,
        "old": "OpenSSH 8.2",
        "new": "OpenSSH 8.9"
      }
    ],
    "new_hosts": [
      {"ip": "192.168.1.200", "site": "Production Servers"}
    ]
  }
}
```

**Comparison UI Features:**
- Side-by-side comparison view
- Color-coded differences (green=new, red=removed, yellow=changed)
- Filter by change type
- Export comparison report to PDF/HTML
- "Compare with previous scan" button on scan detail page

---

**Phase 5 Implementation Plan:**

1. **Week 1: Alert Rule Engine**
   - Implement alert evaluation logic after scan completion
   - Create `alerts` table population
   - Add alert rule CRUD API
   - Unit tests for alert detection

2. **Week 2: Email Notifications**
   - SMTP integration with Flask-Mail
   - Jinja2 email templates (HTML + plain text)
   - Settings API for email configuration
   - Test email functionality
   - Email delivery tracking

3. **Week 3: Webhook System**
   - Webhook configuration API
   - HTTP POST delivery with retry logic
   - Webhook template system for different platforms
   - Test webhook functionality
   - Delivery tracking and logging

4. **Week 4: Alert UI & Scan Comparison**
   - Alert history page with filtering
   - Alert rule management UI
   - Email/webhook settings pages
   - Scan comparison API and UI
   - Integration testing

**Success Criteria:**
- Alert triggered within 30 seconds of scan completion
- Email delivered successfully to configured recipients
- Webhook POST delivered with retry on failure
- Scan comparison highlights all infrastructure changes
- Zero false positives for expected ports/services
- Alert deduplication prevents notification spam

---

### Phase 6: CLI as API Client
**Status:** Planned
**Priority:** MEDIUM

**Goals:**
- Create CLI API client for scripting and automation
- Maintain standalone mode for testing
- API token authentication

**Planned Features:**
1. **API Client Mode:**
   - `--api-mode` flag to enable API client mode
   - `--api-url` and `--api-token` arguments
   - Trigger scans via API, poll for status, download results
   - Scans stored centrally in database
   - Standalone mode still available for testing

2. **API Token System:**
   - Token generation UI in settings page
   - Secure token storage (hashed in database)
   - Token authentication middleware
   - Token expiration and revocation

3. **Benefits:**
   - Centralized scan history accessible via web dashboard
   - No need to mount volumes for output
   - Scheduled scans managed through web UI
   - Scriptable automation while leveraging web features

---

### Phase 7: Advanced Features
**Status:** Future/Deferred
**Priority:** LOW

**Planned Features:**
1. **Enhanced Reports:**
   - Sortable/filterable tables (DataTables.js)
   - Inline screenshot thumbnails with lightbox
   - PDF export (WeasyPrint)

2. **Vulnerability Detection:**
   - CVE database integration (NVD API)
   - Service version matching to known CVEs
   - CVSS severity scores
   - Alert rules for critical CVEs

3. **Timeline View:**
   - Visual scan history timeline
   - Filter by site/IP
   - Event annotations

4. **Advanced Charts:**
   - Port activity heatmap
   - Service version tracking
   - Certificate expiration forecast

5. **Integrations:**
   - Slack notifications
   - Webhook support
   - Prometheus metrics export
   - CSV export/import

---

## Current Architecture

**Primary Interface:** Web GUI (Phases 1-4 Complete)
- Full-featured Flask web application
- Dashboard, scan management, scheduling, config creator
- REST API for all operations
- Single-user deployment with SQLite

**Coming Soon:** CLI API Client (Phase 6 Planned)
- Thin client for scripting and automation
- Calls Flask API for scan operations
- Results stored centrally in database
- Access to all web features via command line

**Core Scanning Engine:**
- Masscan for port discovery
- Nmap for service detection
- Playwright for screenshots
- sslyze for SSL/TLS analysis

**Deployment:**
- Docker Compose for easy deployment
- SQLite database (single-user, embedded)
- Gunicorn WSGI server
- Optional Nginx reverse proxy

## Development Workflow

### Iteration Cycle
1. **Plan** - Define features for phase
2. **Implement** - Code backend + frontend
3. **Test** - Unit tests + manual testing
4. **Deploy** - Update Docker Compose
5. **Document** - Update README.md, ROADMAP.md
6. **Review** - Get user feedback
7. **Iterate** - Adjust priorities based on feedback

### Git Workflow
- **main branch** - Stable releases
- **develop branch** - Active development
- **feature branches** - Individual features (`feature/dashboard`, `feature/scheduler`)
- **Pull requests** - Review before merge

### Testing Strategy
- **Unit tests** - pytest for models, API endpoints
- **Integration tests** - Full scan → DB → API workflow
- **Manual testing** - UI/UX testing in browser
- **Performance tests** - Large scans, database queries

### Documentation
- **README.md** - User-facing documentation (updated each phase)
- **ROADMAP.md** - This file (updated as priorities shift)
- **CLAUDE.md** - Developer documentation (architecture, code references)
- **API.md** - API documentation (OpenAPI/Swagger in Phase 4)

## Resources & References

### Documentation
- [Flask Documentation](https://flask.palletsprojects.com/)
- [SQLAlchemy ORM](https://docs.sqlalchemy.org/)
- [APScheduler](https://apscheduler.readthedocs.io/)
- [Chart.js](https://www.chartjs.org/docs/)
- [Bootstrap 5](https://getbootstrap.com/docs/5.3/)

### Tutorials
- [Flask Mega-Tutorial](https://blog.miguelgrinberg.com/post/the-flask-mega-tutorial-part-i-hello-world)
- [SQLAlchemy Tutorial](https://docs.sqlalchemy.org/en/20/tutorial/)
- [APScheduler with Flask](https://github.com/viniciuschiele/flask-apscheduler)

### Similar Projects (Inspiration)
- [OpenVAS](https://www.openvas.org/) - Vulnerability scanner with web UI
- [Nessus](https://www.tenable.com/products/nessus) - Commercial scanner (inspiration for UI/UX)
- [OWASP ZAP](https://www.zaproxy.org/) - Web app scanner (comparison reports, alerts)

## Changelog

| Date | Version | Changes |
|------|---------|---------|
| 2025-11-14 | 1.0 | Initial roadmap created based on user requirements |
| 2025-11-13 | 1.1 | **Phase 1 COMPLETE** - Database schema, SQLAlchemy models, Flask app structure, settings system with encryption, Alembic migrations, API blueprints, Docker support, validation script |
| 2025-11-14 | 1.2 | **Phase 2 COMPLETE** - REST API (5 scan endpoints, 3 settings endpoints), background jobs (APScheduler), authentication (Flask-Login), web UI (dashboard, scans, login, errors), error handling (content negotiation, request IDs, logging), 100 tests passing, comprehensive documentation (API_REFERENCE.md, DEPLOYMENT.md, PHASE2_COMPLETE.md) |
| 2025-11-17 | 1.3 | **Bug Fix** - Fixed Chart.js infinite canvas growth issue in scan detail page (duplicate initialization, missing chart.destroy(), missing fixed-height container) |
| 2025-11-17 | 1.4 | **Phase 4 COMPLETE** - Config Creator with CIDR-based creation, YAML editor (CodeMirror), config management UI (list/edit/delete), REST API (7 endpoints), Docker volume permissions fix, comprehensive testing and documentation |
| 2025-11-17 | 1.5 | **Roadmap Compression** - Condensed completed phases (1-4) into concise summaries, updated project scope to emphasize web GUI frontend with CLI as API client coming soon (Phase 6), reorganized phases for clarity |

---

**Last Updated:** 2025-11-17
**Next Review:** Before Phase 5 kickoff (Email & Comparisons)