diff --git a/README.md b/README.md index 7b32ce7..7c416b4 100644 --- a/README.md +++ b/README.md @@ -20,7 +20,8 @@ A comprehensive network scanning and infrastructure monitoring platform with web - 📈 **Drift Detection** - Expected vs. actual infrastructure comparison - 📋 **Multi-Format Reports** - JSON, HTML, and ZIP archives with visual reports - 🔐 **Authentication** - Session-based login for single-user deployments -- 🔔 **Alerts** *(Phase 5 - Coming Soon)* - Email and webhook notifications for misconfigurations +- 🔔 **Webhook Alerts** - Real-time notifications via Slack, Discord, PagerDuty, and custom integrations +- ⚠️ **Alert Rules** - Automated detection of infrastructure misconfigurations and anomalies --- @@ -98,8 +99,8 @@ docker compose -f docker-compose-standalone.yml up ## Current Status -**Latest Version**: Phase 4 Complete ✅ -**Last Updated**: 2025-11-17 +**Latest Version**: Phase 5 Complete ✅ +**Last Updated**: 2025-11-19 ### Completed Phases @@ -107,18 +108,19 @@ docker compose -f docker-compose-standalone.yml up - ✅ **Phase 2**: REST API, background jobs, authentication, web UI - ✅ **Phase 3**: Dashboard, scheduling, trend charts - ✅ **Phase 4**: Config creator, YAML editor, config management UI +- ✅ **Phase 5**: Webhooks & alerting, notification templates, alert rules -### Next Up: Phase 5 - Email, Webhooks & Comparisons +### Next Up: Phase 6 - CLI as API Client -**Core Use Case**: Monitor infrastructure for misconfigurations that expose unexpected ports/services. When a scan detects an open port not in the config's `expected_ports` list, trigger immediate notifications. +**Goal**: Create a thin CLI client that calls the Flask API for scan operations, enabling scripting and automation workflows while leveraging centralized database storage and web dashboard features. **Planned Features**: -- Email notifications for infrastructure changes -- Webhook integrations (Slack, PagerDuty, custom SIEM) -- Alert rule engine (unexpected ports, cert expiry, weak TLS) -- Scan comparison reports for drift detection +- API token authentication for CLI access +- Remote scan triggering and status polling +- Centralized scan history accessible via web dashboard +- Scriptable automation workflows -See [Roadmap](docs/ROADMAP.md) for complete feature timeline. +See [Roadmap](docs/ROADMAP.md) for complete feature timeline and future phases. --- @@ -212,5 +214,5 @@ MIT License - See LICENSE file for details --- -**Version**: Phase 4 Complete -**Last Updated**: 2025-11-17 +**Version**: Phase 5 Complete +**Last Updated**: 2025-11-19 diff --git a/docs/ROADMAP.md b/docs/ROADMAP.md index cf91176..f9ae96a 100644 --- a/docs/ROADMAP.md +++ b/docs/ROADMAP.md @@ -4,9 +4,12 @@ SneakyScanner is a comprehensive **Flask web application** for infrastructure monitoring and security auditing. The primary interface is the web GUI, with a CLI API client planned for scripting and automation needs. -**Status:** Phase 4 Complete ✅ | Phase 5 Next Up +**Current Phase:** Phase 5 Complete ✅ | Phase 6 Next Up ## Progress Overview + +**Note:** For detailed architecture and technology stack information, see [README.md](../README.md) + - ✅ **Phase 1: Foundation** - Complete (2025-11-13) - Database schema, SQLAlchemy models, settings system, Flask app structure - ✅ **Phase 2: Flask Web App Core** - Complete (2025-11-14) @@ -15,27 +18,14 @@ SneakyScanner is a comprehensive **Flask web application** for infrastructure mo - Dashboard, scan history, scheduled scans, trend charts - ✅ **Phase 4: Config Creator** - Complete (2025-11-17) - CIDR-based config creation, YAML editor, config management UI -- 📋 **Phase 5: Email, Webhooks & Comparisons** - Next Up - - Email notifications, alert rules, scan comparison +- ✅ **Phase 5: Webhooks & Alerting** - Complete (2025-11-19) + - Webhook notifications, alert rules, notification templates - 📋 **Phase 6: CLI as API Client** - Planned - CLI for scripting and automation via API - 📋 **Phase 7: Advanced Features** - Future - - CVE integration, timeline view, PDF export, enhanced reports + - Email notifications, scan comparison, CVE integration, timeline view, PDF export - -**Core Features:** -- **Centralized dashboard** for viewing scan history and trends -- **Scheduled scanning** for continuous infrastructure monitoring -- **Email notifications** for critical changes and certificate expirations (coming soon) -- **Historical analysis** with charts, graphs, and comparison reports -- **Config creator** for easy CIDR-based scan configuration -- **RESTful API** for integration and automation -- **Simple deployment** using SQLite3 (single-user, self-hosted) - -**Planned:** -- **CLI API client** for scripting and automation workflows (Phase 6) - ## Target Users - **Infrastructure teams** monitoring on-premises networks @@ -43,84 +33,6 @@ SneakyScanner is a comprehensive **Flask web application** for infrastructure mo - **DevOps engineers** tracking infrastructure drift - **Single users or small teams** (not enterprise multi-tenant) -## Technology Stack - -### Backend -- **Flask** 3.x - Lightweight Python web framework -- **SQLAlchemy** 2.x - ORM for database abstraction -- **SQLite3** - Embedded database (easy deployment, sufficient for single-user) -- **APScheduler** 3.x - Background job scheduling for periodic scans -- **Flask-Login** - Simple session-based authentication -- **Flask-CORS** - API access control (optional for CLI API client) -- **Marshmallow** - API serialization/deserialization - -### Frontend -- **Jinja2** - Server-side templating (already in use) -- **Bootstrap 5** - Responsive UI framework with dark theme support -- **Chart.js** 4.x - Lightweight charting library for trends -- **DataTables.js** - Interactive sortable/filterable tables (Phase 6) -- **Vanilla JavaScript** - Keep dependencies minimal - -### Infrastructure -- **Docker Compose** - Multi-container orchestration (Flask app + Scanner) -- **Gunicorn** - WSGI server for production -- **Nginx** - Reverse proxy (optional, for production deployments) - -### Existing Components (Keep) -- **Masscan** - Fast port discovery -- **Nmap** - Service detection -- **Playwright** - Screenshot capture -- **sslyze** - SSL/TLS analysis - -## Architecture Overview - -``` -┌─────────────────────────────────────────────────────────────┐ -│ Flask Web Application │ -│ ┌──────────────┐ ┌──────────────┐ ┌──────────────────┐ │ -│ │ Web UI │ │ REST API │ │ Scheduler │ │ -│ │ (Dashboard) │ │ (JSON/CRUD) │ │ (APScheduler) │ │ -│ └──────┬───────┘ └──────┬───────┘ └────────┬─────────┘ │ -│ │ │ │ │ -│ └─────────────────┴────────────────────┘ │ -│ │ │ -│ ┌────────▼────────┐ │ -│ │ SQLAlchemy │ │ -│ │ (ORM Layer) │ │ -│ └────────┬────────┘ │ -│ │ │ -│ ┌────────▼────────┐ │ -│ │ SQLite3 DB │ │ -│ │ (scan history) │ │ -│ └─────────────────┘ │ -└───────────────────────────┬─────────────────────────────────┘ - │ - ┌──────────▼──────────┐ - │ Scanner Engine │ - │ (scanner.py) │ - │ ┌────────────────┐ │ - │ │ Masscan/Nmap │ │ - │ │ Playwright │ │ - │ │ sslyze │ │ - │ └────────────────┘ │ - └─────────────────────┘ - │ - ┌──────────▼──────────┐ - │ CLI API Client │ - │ (optional future) │ - └─────────────────────┘ -``` - -### Component Interaction - -1. **Web UI** - User interacts with dashboard, triggers scans, views history -2. **REST API** - Handles requests from web UI and CLI client -3. **Scheduler (APScheduler)** - Triggers scans based on cron-like schedules -4. **SQLAlchemy ORM** - Abstracts database operations -5. **SQLite3 Database** - Stores scan results, schedules, settings, alerts -6. **Scanner Engine** - Performs actual network scanning (masscan, nmap, etc.) -7. **CLI API Client** - Future: thin client that calls Flask API - ## Database Schema Design ### Core Tables @@ -352,7 +264,7 @@ All API endpoints return JSON and follow RESTful conventions. - Session cookies with Flask-Login - Password stored as bcrypt hash in settings table -**Phase 5:** API token authentication for CLI client +**Phase 6:** API token authentication for CLI client - Generate API token: `POST /api/auth/token` - Revoke token: `DELETE /api/auth/token` - CLI sends token in `Authorization: Bearer ` header @@ -420,22 +332,20 @@ All API endpoints return JSON and follow RESTful conventions. --- -### Phase 5: Email, Webhooks & Comparisons -**Status:** Next Up -**Priority:** MEDIUM +### Phase 5: Webhooks & Alerting ✅ COMPLETE +**Completed:** 2025-11-19 **Goals:** -- Implement email notification system for infrastructure misconfigurations -- Implement webhook notification system for real-time alerting -- Create scan comparison reports to detect drift -- Add alert rule configuration for unexpected exposure detection +- ✅ Implement webhook notification system for real-time alerting +- ✅ Add alert rule configuration for unexpected exposure detection +- ✅ Create notification template system for flexible alerting **Core Use Case:** -Monitor infrastructure for misconfigurations that expose unexpected ports/services to the world. When a scan detects an open port that wasn't defined in the YAML config's `expected_ports` list, trigger immediate notifications via email and/or webhooks. +Monitor infrastructure for misconfigurations that expose unexpected ports/services to the world. When a scan detects an open port that wasn't defined in the YAML config's `expected_ports` list, trigger immediate notifications via webhooks. -**Planned Features:** +**Implemented Features:** -#### 1. Alert Rule Engine +#### 1. Alert Rule Engine ✅ **Purpose:** Automatically detect and classify infrastructure anomalies after each scan. **Alert Types:** @@ -462,7 +372,6 @@ alert_rules: rule_type: unexpected_port enabled: true severity: critical - email_enabled: true webhook_enabled: true filter_conditions: ports: [22, 80, 443, 3389, 3306, 5432, 27017] # High-risk ports @@ -472,85 +381,17 @@ alert_rules: enabled: true severity: warning threshold: 30 # Days before expiry - email_enabled: true - webhook_enabled: false + webhook_enabled: true ``` **Implementation:** -- Evaluate alert rules after each scan completes -- Compare current scan results to expected configuration -- Generate alerts and store in `alerts` table -- Trigger notifications based on rule configuration -- Alert deduplication (don't spam for same issue) +- ✅ Evaluate alert rules after each scan completes +- ✅ Compare current scan results to expected configuration +- ✅ Generate alerts and store in `alerts` table +- ✅ Trigger notifications based on rule configuration +- ✅ Alert deduplication (don't spam for same issue) -#### 2. Email Notifications -**Purpose:** Send detailed email alerts when infrastructure misconfigurations are detected. - -**SMTP Configuration (via Settings API):** -```json -{ - "smtp_server": "smtp.gmail.com", - "smtp_port": 587, - "smtp_use_tls": true, - "smtp_username": "alerts@example.com", - "smtp_password": "encrypted_password", - "smtp_from_email": "SneakyScanner ", - "smtp_to_emails": ["admin@example.com", "security@example.com"], - "email_alerts_enabled": true -} -``` - -**Email Template (Jinja2 HTML):** -```html -Subject: [SneakyScanner Alert] Unexpected Port Detected - {{ ip_address }}:{{ port }} - -Body: -=============================================== -SneakyScanner Security Alert -=============================================== - -Alert Type: {{ alert_type }} -Severity: {{ severity }} -Scan ID: {{ scan_id }} -Timestamp: {{ timestamp }} - -Issue Detected: -{{ message }} - -Details: -- IP Address: {{ ip_address }} -- Port: {{ port }}/{{ protocol }} -- Service: {{ service_name }} ({{ product }} {{ version }}) -- Expected: No (not in expected_ports list) - -Recommended Actions: -1. Verify if this service should be exposed -2. Check firewall rules for {{ ip_address }} -3. Review service configuration -4. Update scan config if this is intentional - -View Full Scan Results: -{{ web_url }}/scans/{{ scan_id }} - -=============================================== -``` - -**Email Features:** -- HTML email with styled alert box (Bootstrap-based) -- Plain-text fallback for compatibility -- Alert summary with actionable recommendations -- Direct link to scan detail page -- Configurable recipients (multiple emails) -- Test email functionality in Settings UI -- Email delivery tracking (email_sent, email_sent_at in alerts table) -- Rate limiting to prevent email flood - -**Email API Endpoints:** -- `POST /api/settings/email` - Configure SMTP settings -- `POST /api/settings/email/test` - Send test email -- `GET /api/alerts?email_sent=true` - Get alerts with email status - -#### 3. Webhook Notifications +#### 2. Webhook Notifications ✅ **Purpose:** Real-time HTTP POST notifications for integration with external systems (Slack, PagerDuty, custom dashboards, SIEM tools). **Webhook Configuration (via Settings API):** @@ -624,30 +465,30 @@ View Full Scan Results: ``` **Webhook Features:** -- Multiple webhook URLs with independent configuration -- Per-webhook filtering by alert type and severity -- Custom headers support (for API keys, auth tokens) -- Authentication methods: +- ✅ Multiple webhook URLs with independent configuration +- ✅ Per-webhook filtering by alert type and severity +- ✅ Custom headers support (for API keys, auth tokens) +- ✅ Authentication methods: - `none` - No authentication - `bearer` - Bearer token in Authorization header - `basic` - Basic authentication - `custom` - Custom header-based auth -- Retry logic with exponential backoff (3 attempts) -- Webhook delivery tracking (webhook_sent, webhook_sent_at, webhook_response_code) -- Test webhook functionality in Settings UI -- Timeout configuration (default 10 seconds) -- Webhook delivery history and logs +- ✅ Retry logic with exponential backoff (3 attempts) +- ✅ Webhook delivery tracking (webhook_sent, webhook_sent_at, webhook_response_code) +- ✅ Test webhook functionality in Settings UI +- ✅ Timeout configuration (default 10 seconds) +- ✅ Webhook delivery history and logs **Webhook API Endpoints:** -- `POST /api/webhooks` - Create webhook configuration -- `GET /api/webhooks` - List all webhooks -- `PUT /api/webhooks/{id}` - Update webhook configuration -- `DELETE /api/webhooks/{id}` - Delete webhook -- `POST /api/webhooks/{id}/test` - Send test webhook -- `GET /api/webhooks/{id}/history` - Get delivery history +- ✅ `POST /api/webhooks` - Create webhook configuration +- ✅ `GET /api/webhooks` - List all webhooks +- ✅ `PUT /api/webhooks/{id}` - Update webhook configuration +- ✅ `DELETE /api/webhooks/{id}` - Delete webhook +- ✅ `POST /api/webhooks/{id}/test` - Send test webhook +- ✅ `GET /api/webhooks/{id}/history` - Get delivery history -**Slack Integration Example:** -Transform webhook payload to Slack message format: +**Notification Templates:** +Flexible template system supporting multiple platforms (Slack, Discord, PagerDuty, etc.): ```json { "text": "SneakyScanner Alert: Unexpected Port Detected", @@ -667,113 +508,26 @@ Transform webhook payload to Slack message format: } ``` -#### 4. Alert Management UI -**Purpose:** Web interface for configuring alert rules, viewing alert history, and managing notifications. - -**Pages:** -- `/alerts` - Alert history with filtering and search -- `/alerts/rules` - Alert rule configuration -- `/settings/email` - Email notification settings -- `/settings/webhooks` - Webhook configuration - -**Alert History Features:** -- Filter by alert type, severity, date range, IP address -- Search by message content -- Bulk acknowledge/dismiss alerts -- Export alerts to CSV -- Alert detail modal with full context - -**Alert Rule UI Features:** -- Enable/disable rules individually -- Configure severity levels -- Set thresholds (e.g., cert expiry days) -- Toggle email/webhook per rule -- Test rules against recent scans - -#### 5. Scan Comparison -**Purpose:** Detect infrastructure drift by comparing two scans and highlighting changes. - -**Comparison API:** -- `GET /api/scans/{id1}/compare/{id2}` - Compare two scans - -**Comparison Response:** -```json -{ - "scan1": {"id": 100, "timestamp": "2025-11-10T10:00:00Z"}, - "scan2": {"id": 123, "timestamp": "2025-11-17T14:15:00Z"}, - "summary": { - "new_ports": 2, - "removed_ports": 0, - "service_changes": 1, - "cert_changes": 0, - "new_hosts": 1, - "removed_hosts": 0 - }, - "differences": { - "new_ports": [ - {"ip": "192.168.1.100", "port": 3306, "service": "mysql"} - ], - "removed_ports": [], - "service_changes": [ - { - "ip": "192.168.1.50", - "port": 22, - "old": "OpenSSH 8.2", - "new": "OpenSSH 8.9" - } - ], - "new_hosts": [ - {"ip": "192.168.1.200", "site": "Production Servers"} - ] - } -} -``` - -**Comparison UI Features:** -- Side-by-side comparison view -- Color-coded differences (green=new, red=removed, yellow=changed) -- Filter by change type -- Export comparison report to PDF/HTML -- "Compare with previous scan" button on scan detail page - --- -**Phase 5 Implementation Plan:** +**Deliverables:** +- ✅ Alert Rule Engine with 9 alert types (unexpected_port, unexpected_service, cert_expiry, ping_failed, service_down, service_change, weak_tls, new_host, host_disappeared) +- ✅ Alert severity classification (critical, warning, info) +- ✅ Alert rule configuration API (CRUD operations) +- ✅ Webhook notification system with retry logic +- ✅ Multiple webhook URL support with independent configuration +- ✅ Notification template system for flexible platform integration (Slack, Discord, PagerDuty, custom) +- ✅ Webhook API endpoints (create, list, update, delete, test, history) +- ✅ Custom headers and authentication support (none, bearer, basic, custom) +- ✅ Webhook delivery tracking and logging +- ✅ Alert deduplication to prevent notification spam +- ✅ Integration with scan completion workflow -1. **Week 1: Alert Rule Engine** - - Implement alert evaluation logic after scan completion - - Create `alerts` table population - - Add alert rule CRUD API - - Unit tests for alert detection - -2. **Week 2: Email Notifications** - - SMTP integration with Flask-Mail - - Jinja2 email templates (HTML + plain text) - - Settings API for email configuration - - Test email functionality - - Email delivery tracking - -3. **Week 3: Webhook System** - - Webhook configuration API - - HTTP POST delivery with retry logic - - Webhook template system for different platforms - - Test webhook functionality - - Delivery tracking and logging - -4. **Week 4: Alert UI & Scan Comparison** - - Alert history page with filtering - - Alert rule management UI - - Email/webhook settings pages - - Scan comparison API and UI - - Integration testing - -**Success Criteria:** -- Alert triggered within 30 seconds of scan completion -- Email delivered successfully to configured recipients -- Webhook POST delivered with retry on failure -- Scan comparison highlights all infrastructure changes -- Zero false positives for expected ports/services -- Alert deduplication prevents notification spam +**Success Criteria Met:** +- ✅ Alerts triggered within 30 seconds of scan completion +- ✅ Webhook POST delivered with retry on failure +- ✅ Zero false positives for expected ports/services +- ✅ Alert deduplication prevents notification spam --- @@ -813,61 +567,52 @@ Transform webhook payload to Slack message format: **Priority:** LOW **Planned Features:** -1. **Enhanced Reports:** + +1. **Email Notifications:** + - SMTP integration with Flask-Mail + - Jinja2 email templates (HTML + plain text) + - Settings API for email configuration + - Test email functionality + - Email delivery tracking + - Rate limiting to prevent email flood + - Configurable recipients (multiple emails) + +2. **Scan Comparison:** + - Compare two scans API endpoint + - Side-by-side comparison view + - Color-coded differences (green=new, red=removed, yellow=changed) + - Filter by change type + - Export comparison report to PDF/HTML + - "Compare with previous scan" button on scan detail page + +3. **Enhanced Reports:** - Sortable/filterable tables (DataTables.js) - Inline screenshot thumbnails with lightbox - PDF export (WeasyPrint) -2. **Vulnerability Detection:** +4. **Vulnerability Detection:** - CVE database integration (NVD API) - Service version matching to known CVEs - CVSS severity scores - Alert rules for critical CVEs -3. **Timeline View:** +5. **Timeline View:** - Visual scan history timeline - Filter by site/IP - Event annotations -4. **Advanced Charts:** +6. **Advanced Charts:** - Port activity heatmap - Service version tracking - Certificate expiration forecast -5. **Integrations:** - - Slack notifications - - Webhook support +7. **Additional Integrations:** - Prometheus metrics export - CSV export/import + - Advanced reporting dashboards --- -## Current Architecture - -**Primary Interface:** Web GUI (Phases 1-4 Complete) -- Full-featured Flask web application -- Dashboard, scan management, scheduling, config creator -- REST API for all operations -- Single-user deployment with SQLite - -**Coming Soon:** CLI API Client (Phase 6 Planned) -- Thin client for scripting and automation -- Calls Flask API for scan operations -- Results stored centrally in database -- Access to all web features via command line - -**Core Scanning Engine:** -- Masscan for port discovery -- Nmap for service detection -- Playwright for screenshots -- sslyze for SSL/TLS analysis - -**Deployment:** -- Docker Compose for easy deployment -- SQLite database (single-user, embedded) -- Gunicorn WSGI server -- Optional Nginx reverse proxy - ## Development Workflow ### Iteration Cycle @@ -926,8 +671,10 @@ Transform webhook payload to Slack message format: | 2025-11-17 | 1.3 | **Bug Fix** - Fixed Chart.js infinite canvas growth issue in scan detail page (duplicate initialization, missing chart.destroy(), missing fixed-height container) | | 2025-11-17 | 1.4 | **Phase 4 COMPLETE** - Config Creator with CIDR-based creation, YAML editor (CodeMirror), config management UI (list/edit/delete), REST API (7 endpoints), Docker volume permissions fix, comprehensive testing and documentation | | 2025-11-17 | 1.5 | **Roadmap Compression** - Condensed completed phases (1-4) into concise summaries, updated project scope to emphasize web GUI frontend with CLI as API client coming soon (Phase 6), reorganized phases for clarity | +| 2025-11-19 | 1.6 | **Phase 5 Progress** - Completed webhooks, notification templates, and alerting rules. Alert Rule Engine and Webhook System implemented. | +| 2025-11-19 | 1.7 | **Phase 5 COMPLETE** - Webhooks & Alerting phase completed. Moved Email Notifications and Scan Comparison to Phase 7. Alert rules, webhook notifications, and notification templates fully implemented and tested. | --- -**Last Updated:** 2025-11-17 -**Next Review:** Before Phase 5 kickoff (Email & Comparisons) +**Last Updated:** 2025-11-19 +**Next Review:** Before Phase 6 kickoff (CLI as API Client)