doc changes

2025-11-19 10:42:49 -06:00
parent 21254c3522
commit 4a4c33a10b
2 changed files with 99 additions and 350 deletions
--- a/docs/ROADMAP.md
+++ b/docs/ROADMAP.md
@@ -4,9 +4,12 @@

 SneakyScanner is a comprehensive **Flask web application** for infrastructure monitoring and security auditing. The primary interface is the web GUI, with a CLI API client planned for scripting and automation needs.

-**Status:** Phase 4 Complete ✅ | Phase 5 Next Up
+**Current Phase:** Phase 5 Complete ✅ | Phase 6 Next Up

 ## Progress Overview
+
+**Note:** For detailed architecture and technology stack information, see [README.md](../README.md)
+
 - ✅ **Phase 1: Foundation** - Complete (2025-11-13)
  - Database schema, SQLAlchemy models, settings system, Flask app structure
 - ✅ **Phase 2: Flask Web App Core** - Complete (2025-11-14)
@@ -15,27 +18,14 @@ SneakyScanner is a comprehensive **Flask web application** for infrastructure mo
  - Dashboard, scan history, scheduled scans, trend charts
 - ✅ **Phase 4: Config Creator** - Complete (2025-11-17)
  - CIDR-based config creation, YAML editor, config management UI
- 📋 **Phase 5: Email, Webhooks & Comparisons** - Next Up
-  - Email notifications, alert rules, scan comparison
+- ✅ **Phase 5: Webhooks & Alerting** - Complete (2025-11-19)
+  - Webhook notifications, alert rules, notification templates
 - 📋 **Phase 6: CLI as API Client** - Planned
  - CLI for scripting and automation via API
 - 📋 **Phase 7: Advanced Features** - Future
-  - CVE integration, timeline view, PDF export, enhanced reports
+  - Email notifications, scan comparison, CVE integration, timeline view, PDF export


-
-**Core Features:**
- **Centralized dashboard** for viewing scan history and trends
- **Scheduled scanning** for continuous infrastructure monitoring
- **Email notifications** for critical changes and certificate expirations (coming soon)
- **Historical analysis** with charts, graphs, and comparison reports
- **Config creator** for easy CIDR-based scan configuration
- **RESTful API** for integration and automation
- **Simple deployment** using SQLite3 (single-user, self-hosted)
-
-**Planned:**
- **CLI API client** for scripting and automation workflows (Phase 6)
-
 ## Target Users

 - **Infrastructure teams** monitoring on-premises networks
@@ -43,84 +33,6 @@ SneakyScanner is a comprehensive **Flask web application** for infrastructure mo
 - **DevOps engineers** tracking infrastructure drift
 - **Single users or small teams** (not enterprise multi-tenant)

-## Technology Stack
-
-### Backend
- **Flask** 3.x - Lightweight Python web framework
- **SQLAlchemy** 2.x - ORM for database abstraction
- **SQLite3** - Embedded database (easy deployment, sufficient for single-user)
- **APScheduler** 3.x - Background job scheduling for periodic scans
- **Flask-Login** - Simple session-based authentication
- **Flask-CORS** - API access control (optional for CLI API client)
- **Marshmallow** - API serialization/deserialization
-
-### Frontend
- **Jinja2** - Server-side templating (already in use)
- **Bootstrap 5** - Responsive UI framework with dark theme support
- **Chart.js** 4.x - Lightweight charting library for trends
- **DataTables.js** - Interactive sortable/filterable tables (Phase 6)
- **Vanilla JavaScript** - Keep dependencies minimal
-
-### Infrastructure
- **Docker Compose** - Multi-container orchestration (Flask app + Scanner)
- **Gunicorn** - WSGI server for production
- **Nginx** - Reverse proxy (optional, for production deployments)
-
-### Existing Components (Keep)
- **Masscan** - Fast port discovery
- **Nmap** - Service detection
- **Playwright** - Screenshot capture
- **sslyze** - SSL/TLS analysis
-
-## Architecture Overview
-
-```
-┌─────────────────────────────────────────────────────────────┐
-│                     Flask Web Application                   │
-│  ┌──────────────┐  ┌──────────────┐  ┌──────────────────┐  │
-│  │   Web UI     │  │   REST API   │  │   Scheduler      │  │
-│  │  (Dashboard) │  │ (JSON/CRUD)  │  │  (APScheduler)   │  │
-│  └──────┬───────┘  └──────┬───────┘  └────────┬─────────┘  │
-│         │                 │                    │            │
-│         └─────────────────┴────────────────────┘            │
-│                           │                                 │
-│                  ┌────────▼────────┐                        │
-│                  │   SQLAlchemy    │                        │
-│                  │   (ORM Layer)   │                        │
-│                  └────────┬────────┘                        │
-│                           │                                 │
-│                  ┌────────▼────────┐                        │
-│                  │   SQLite3 DB    │                        │
-│                  │  (scan history) │                        │
-│                  └─────────────────┘                        │
-└───────────────────────────┬─────────────────────────────────┘
-                            │
-                 ┌──────────▼──────────┐
-                 │  Scanner Engine     │
-                 │  (scanner.py)       │
-                 │  ┌────────────────┐ │
-                 │  │ Masscan/Nmap   │ │
-                 │  │ Playwright     │ │
-                 │  │ sslyze         │ │
-                 │  └────────────────┘ │
-                 └─────────────────────┘
-                            │
-                 ┌──────────▼──────────┐
-                 │  CLI API Client     │
-                 │  (optional future)  │
-                 └─────────────────────┘
-```
-
-### Component Interaction
-
-1. **Web UI** - User interacts with dashboard, triggers scans, views history
-2. **REST API** - Handles requests from web UI and CLI client
-3. **Scheduler (APScheduler)** - Triggers scans based on cron-like schedules
-4. **SQLAlchemy ORM** - Abstracts database operations
-5. **SQLite3 Database** - Stores scan results, schedules, settings, alerts
-6. **Scanner Engine** - Performs actual network scanning (masscan, nmap, etc.)
-7. **CLI API Client** - Future: thin client that calls Flask API
-
 ## Database Schema Design

 ### Core Tables
@@ -352,7 +264,7 @@ All API endpoints return JSON and follow RESTful conventions.
 - Session cookies with Flask-Login
 - Password stored as bcrypt hash in settings table

-**Phase 5:** API token authentication for CLI client
+**Phase 6:** API token authentication for CLI client
 - Generate API token: `POST /api/auth/token`
 - Revoke token: `DELETE /api/auth/token`
 - CLI sends token in `Authorization: Bearer <token>` header
@@ -420,22 +332,20 @@ All API endpoints return JSON and follow RESTful conventions.

 ---

-### Phase 5: Email, Webhooks & Comparisons
-**Status:** Next Up
-**Priority:** MEDIUM
+### Phase 5: Webhooks & Alerting ✅ COMPLETE
+**Completed:** 2025-11-19

 **Goals:**
- Implement email notification system for infrastructure misconfigurations
- Implement webhook notification system for real-time alerting
- Create scan comparison reports to detect drift
- Add alert rule configuration for unexpected exposure detection
+- ✅ Implement webhook notification system for real-time alerting
+- ✅ Add alert rule configuration for unexpected exposure detection
+- ✅ Create notification template system for flexible alerting

 **Core Use Case:**
-Monitor infrastructure for misconfigurations that expose unexpected ports/services to the world. When a scan detects an open port that wasn't defined in the YAML config's `expected_ports` list, trigger immediate notifications via email and/or webhooks.
+Monitor infrastructure for misconfigurations that expose unexpected ports/services to the world. When a scan detects an open port that wasn't defined in the YAML config's `expected_ports` list, trigger immediate notifications via webhooks.

-**Planned Features:**
+**Implemented Features:**

-#### 1. Alert Rule Engine
+#### 1. Alert Rule Engine ✅
 **Purpose:** Automatically detect and classify infrastructure anomalies after each scan.

 **Alert Types:**
@@ -462,7 +372,6 @@ alert_rules:
    rule_type: unexpected_port
    enabled: true
    severity: critical
-    email_enabled: true
    webhook_enabled: true
    filter_conditions:
      ports: [22, 80, 443, 3389, 3306, 5432, 27017]  # High-risk ports
@@ -472,85 +381,17 @@ alert_rules:
    enabled: true
    severity: warning
    threshold: 30  # Days before expiry
-    email_enabled: true
-    webhook_enabled: false
+    webhook_enabled: true
 ```

 **Implementation:**
- Evaluate alert rules after each scan completes
- Compare current scan results to expected configuration
- Generate alerts and store in `alerts` table
- Trigger notifications based on rule configuration
- Alert deduplication (don't spam for same issue)
+- ✅ Evaluate alert rules after each scan completes
+- ✅ Compare current scan results to expected configuration
+- ✅ Generate alerts and store in `alerts` table
+- ✅ Trigger notifications based on rule configuration
+- ✅ Alert deduplication (don't spam for same issue)

-#### 2. Email Notifications
-**Purpose:** Send detailed email alerts when infrastructure misconfigurations are detected.
-
-**SMTP Configuration (via Settings API):**
-```json
-{
-  "smtp_server": "smtp.gmail.com",
-  "smtp_port": 587,
-  "smtp_use_tls": true,
-  "smtp_username": "alerts@example.com",
-  "smtp_password": "encrypted_password",
-  "smtp_from_email": "SneakyScanner <alerts@example.com>",
-  "smtp_to_emails": ["admin@example.com", "security@example.com"],
-  "email_alerts_enabled": true
-}
-```
-
-**Email Template (Jinja2 HTML):**
-```html
-Subject: [SneakyScanner Alert] Unexpected Port Detected - {{ ip_address }}:{{ port }}
-
-Body:
-===============================================
-SneakyScanner Security Alert
-===============================================
-
-Alert Type: {{ alert_type }}
-Severity: {{ severity }}
-Scan ID: {{ scan_id }}
-Timestamp: {{ timestamp }}
-
-Issue Detected:
-{{ message }}
-
-Details:
- IP Address: {{ ip_address }}
- Port: {{ port }}/{{ protocol }}
- Service: {{ service_name }} ({{ product }} {{ version }})
- Expected: No (not in expected_ports list)
-
-Recommended Actions:
-1. Verify if this service should be exposed
-2. Check firewall rules for {{ ip_address }}
-3. Review service configuration
-4. Update scan config if this is intentional
-
-View Full Scan Results:
-{{ web_url }}/scans/{{ scan_id }}
-
-===============================================
-```
-
-**Email Features:**
- HTML email with styled alert box (Bootstrap-based)
- Plain-text fallback for compatibility
- Alert summary with actionable recommendations
- Direct link to scan detail page
- Configurable recipients (multiple emails)
- Test email functionality in Settings UI
- Email delivery tracking (email_sent, email_sent_at in alerts table)
- Rate limiting to prevent email flood
-
-**Email API Endpoints:**
- `POST /api/settings/email` - Configure SMTP settings
- `POST /api/settings/email/test` - Send test email
- `GET /api/alerts?email_sent=true` - Get alerts with email status
-
-#### 3. Webhook Notifications
+#### 2. Webhook Notifications ✅
 **Purpose:** Real-time HTTP POST notifications for integration with external systems (Slack, PagerDuty, custom dashboards, SIEM tools).

 **Webhook Configuration (via Settings API):**
@@ -624,30 +465,30 @@ View Full Scan Results:
 ```

 **Webhook Features:**
- Multiple webhook URLs with independent configuration
- Per-webhook filtering by alert type and severity
- Custom headers support (for API keys, auth tokens)
- Authentication methods:
+- ✅ Multiple webhook URLs with independent configuration
+- ✅ Per-webhook filtering by alert type and severity
+- ✅ Custom headers support (for API keys, auth tokens)
+- ✅ Authentication methods:
  - `none` - No authentication
  - `bearer` - Bearer token in Authorization header
  - `basic` - Basic authentication
  - `custom` - Custom header-based auth
- Retry logic with exponential backoff (3 attempts)
- Webhook delivery tracking (webhook_sent, webhook_sent_at, webhook_response_code)
- Test webhook functionality in Settings UI
- Timeout configuration (default 10 seconds)
- Webhook delivery history and logs
+- ✅ Retry logic with exponential backoff (3 attempts)
+- ✅ Webhook delivery tracking (webhook_sent, webhook_sent_at, webhook_response_code)
+- ✅ Test webhook functionality in Settings UI
+- ✅ Timeout configuration (default 10 seconds)
+- ✅ Webhook delivery history and logs

 **Webhook API Endpoints:**
- `POST /api/webhooks` - Create webhook configuration
- `GET /api/webhooks` - List all webhooks
- `PUT /api/webhooks/{id}` - Update webhook configuration
- `DELETE /api/webhooks/{id}` - Delete webhook
- `POST /api/webhooks/{id}/test` - Send test webhook
- `GET /api/webhooks/{id}/history` - Get delivery history
+- ✅ `POST /api/webhooks` - Create webhook configuration
+- ✅ `GET /api/webhooks` - List all webhooks
+- ✅ `PUT /api/webhooks/{id}` - Update webhook configuration
+- ✅ `DELETE /api/webhooks/{id}` - Delete webhook
+- ✅ `POST /api/webhooks/{id}/test` - Send test webhook
+- ✅ `GET /api/webhooks/{id}/history` - Get delivery history

-**Slack Integration Example:**
-Transform webhook payload to Slack message format:
+**Notification Templates:**
+Flexible template system supporting multiple platforms (Slack, Discord, PagerDuty, etc.):
 ```json
 {
  "text": "SneakyScanner Alert: Unexpected Port Detected",
@@ -667,113 +508,26 @@ Transform webhook payload to Slack message format:
 }
 ```

-#### 4. Alert Management UI
-**Purpose:** Web interface for configuring alert rules, viewing alert history, and managing notifications.
-
-**Pages:**
- `/alerts` - Alert history with filtering and search
- `/alerts/rules` - Alert rule configuration
- `/settings/email` - Email notification settings
- `/settings/webhooks` - Webhook configuration
-
-**Alert History Features:**
- Filter by alert type, severity, date range, IP address
- Search by message content
- Bulk acknowledge/dismiss alerts
- Export alerts to CSV
- Alert detail modal with full context
-
-**Alert Rule UI Features:**
- Enable/disable rules individually
- Configure severity levels
- Set thresholds (e.g., cert expiry days)
- Toggle email/webhook per rule
- Test rules against recent scans
-
-#### 5. Scan Comparison
-**Purpose:** Detect infrastructure drift by comparing two scans and highlighting changes.
-
-**Comparison API:**
- `GET /api/scans/{id1}/compare/{id2}` - Compare two scans
-
-**Comparison Response:**
-```json
-{
-  "scan1": {"id": 100, "timestamp": "2025-11-10T10:00:00Z"},
-  "scan2": {"id": 123, "timestamp": "2025-11-17T14:15:00Z"},
-  "summary": {
-    "new_ports": 2,
-    "removed_ports": 0,
-    "service_changes": 1,
-    "cert_changes": 0,
-    "new_hosts": 1,
-    "removed_hosts": 0
-  },
-  "differences": {
-    "new_ports": [
-      {"ip": "192.168.1.100", "port": 3306, "service": "mysql"}
-    ],
-    "removed_ports": [],
-    "service_changes": [
-      {
-        "ip": "192.168.1.50",
-        "port": 22,
-        "old": "OpenSSH 8.2",
-        "new": "OpenSSH 8.9"
-      }
-    ],
-    "new_hosts": [
-      {"ip": "192.168.1.200", "site": "Production Servers"}
-    ]
-  }
-}
-```
-
-**Comparison UI Features:**
- Side-by-side comparison view
- Color-coded differences (green=new, red=removed, yellow=changed)
- Filter by change type
- Export comparison report to PDF/HTML
- "Compare with previous scan" button on scan detail page
-
 ---

-**Phase 5 Implementation Plan:**
+**Deliverables:**
+- ✅ Alert Rule Engine with 9 alert types (unexpected_port, unexpected_service, cert_expiry, ping_failed, service_down, service_change, weak_tls, new_host, host_disappeared)
+- ✅ Alert severity classification (critical, warning, info)
+- ✅ Alert rule configuration API (CRUD operations)
+- ✅ Webhook notification system with retry logic
+- ✅ Multiple webhook URL support with independent configuration
+- ✅ Notification template system for flexible platform integration (Slack, Discord, PagerDuty, custom)
+- ✅ Webhook API endpoints (create, list, update, delete, test, history)
+- ✅ Custom headers and authentication support (none, bearer, basic, custom)
+- ✅ Webhook delivery tracking and logging
+- ✅ Alert deduplication to prevent notification spam
+- ✅ Integration with scan completion workflow

-1. **Week 1: Alert Rule Engine**
-   - Implement alert evaluation logic after scan completion
-   - Create `alerts` table population
-   - Add alert rule CRUD API
-   - Unit tests for alert detection
-
-2. **Week 2: Email Notifications**
-   - SMTP integration with Flask-Mail
-   - Jinja2 email templates (HTML + plain text)
-   - Settings API for email configuration
-   - Test email functionality
-   - Email delivery tracking
-
-3. **Week 3: Webhook System**
-   - Webhook configuration API
-   - HTTP POST delivery with retry logic
-   - Webhook template system for different platforms
-   - Test webhook functionality
-   - Delivery tracking and logging
-
-4. **Week 4: Alert UI & Scan Comparison**
-   - Alert history page with filtering
-   - Alert rule management UI
-   - Email/webhook settings pages
-   - Scan comparison API and UI
-   - Integration testing
-
-**Success Criteria:**
- Alert triggered within 30 seconds of scan completion
- Email delivered successfully to configured recipients
- Webhook POST delivered with retry on failure
- Scan comparison highlights all infrastructure changes
- Zero false positives for expected ports/services
- Alert deduplication prevents notification spam
+**Success Criteria Met:**
+- ✅ Alerts triggered within 30 seconds of scan completion
+- ✅ Webhook POST delivered with retry on failure
+- ✅ Zero false positives for expected ports/services
+- ✅ Alert deduplication prevents notification spam

 ---

@@ -813,61 +567,52 @@ Transform webhook payload to Slack message format:
 **Priority:** LOW

 **Planned Features:**
-1. **Enhanced Reports:**
+
+1. **Email Notifications:**
+   - SMTP integration with Flask-Mail
+   - Jinja2 email templates (HTML + plain text)
+   - Settings API for email configuration
+   - Test email functionality
+   - Email delivery tracking
+   - Rate limiting to prevent email flood
+   - Configurable recipients (multiple emails)
+
+2. **Scan Comparison:**
+   - Compare two scans API endpoint
+   - Side-by-side comparison view
+   - Color-coded differences (green=new, red=removed, yellow=changed)
+   - Filter by change type
+   - Export comparison report to PDF/HTML
+   - "Compare with previous scan" button on scan detail page
+
+3. **Enhanced Reports:**
   - Sortable/filterable tables (DataTables.js)
   - Inline screenshot thumbnails with lightbox
   - PDF export (WeasyPrint)

-2. **Vulnerability Detection:**
+4. **Vulnerability Detection:**
   - CVE database integration (NVD API)
   - Service version matching to known CVEs
   - CVSS severity scores
   - Alert rules for critical CVEs

-3. **Timeline View:**
+5. **Timeline View:**
   - Visual scan history timeline
   - Filter by site/IP
   - Event annotations

-4. **Advanced Charts:**
+6. **Advanced Charts:**
   - Port activity heatmap
   - Service version tracking
   - Certificate expiration forecast

-5. **Integrations:**
-   - Slack notifications
-   - Webhook support
+7. **Additional Integrations:**
   - Prometheus metrics export
   - CSV export/import
+   - Advanced reporting dashboards

 ---

-## Current Architecture
-
-**Primary Interface:** Web GUI (Phases 1-4 Complete)
- Full-featured Flask web application
- Dashboard, scan management, scheduling, config creator
- REST API for all operations
- Single-user deployment with SQLite
-
-**Coming Soon:** CLI API Client (Phase 6 Planned)
- Thin client for scripting and automation
- Calls Flask API for scan operations
- Results stored centrally in database
- Access to all web features via command line
-
-**Core Scanning Engine:**
- Masscan for port discovery
- Nmap for service detection
- Playwright for screenshots
- sslyze for SSL/TLS analysis
-
-**Deployment:**
- Docker Compose for easy deployment
- SQLite database (single-user, embedded)
- Gunicorn WSGI server
- Optional Nginx reverse proxy
-
 ## Development Workflow

 ### Iteration Cycle
@@ -926,8 +671,10 @@ Transform webhook payload to Slack message format:
 | 2025-11-17 | 1.3 | **Bug Fix** - Fixed Chart.js infinite canvas growth issue in scan detail page (duplicate initialization, missing chart.destroy(), missing fixed-height container) |
 | 2025-11-17 | 1.4 | **Phase 4 COMPLETE** - Config Creator with CIDR-based creation, YAML editor (CodeMirror), config management UI (list/edit/delete), REST API (7 endpoints), Docker volume permissions fix, comprehensive testing and documentation |
 | 2025-11-17 | 1.5 | **Roadmap Compression** - Condensed completed phases (1-4) into concise summaries, updated project scope to emphasize web GUI frontend with CLI as API client coming soon (Phase 6), reorganized phases for clarity |
+| 2025-11-19 | 1.6 | **Phase 5 Progress** - Completed webhooks, notification templates, and alerting rules. Alert Rule Engine and Webhook System implemented. |
+| 2025-11-19 | 1.7 | **Phase 5 COMPLETE** - Webhooks & Alerting phase completed. Moved Email Notifications and Scan Comparison to Phase 7. Alert rules, webhook notifications, and notification templates fully implemented and tested. |

 ---

-**Last Updated:** 2025-11-17
-**Next Review:** Before Phase 5 kickoff (Email & Comparisons)
+**Last Updated:** 2025-11-19
+**Next Review:** Before Phase 6 kickoff (CLI as API Client)