sneakygeek/SneakyScan
2
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

restructure of dirs, huge docs update

Browse Source
This commit is contained in:
Phillip Tarrant
2025-11-17 16:29:14 -06:00
parent 456e052389
commit cd840cb8ca
87 changed files with 2827 additions and 1094 deletions
Download Patch File Download Diff File Expand all files Collapse all files

483
docs/ROADMAP.md
View File

@@ -1,5 +1,9 @@
# SneakyScanner Roadmap
## Vision & Goals
SneakyScanner is a comprehensive **Flask web application** for infrastructure monitoring and security auditing. The primary interface is the web GUI, with a CLI API client planned for scripting and automation needs.
**Status:** Phase 4 Complete ✅ | Phase 5 Next Up
## Progress Overview
@@ -11,37 +15,14 @@
- Dashboard, scan history, scheduled scans, trend charts
- ✅ **Phase 4: Config Creator** - Complete (2025-11-17)
- CIDR-based config creation, YAML editor, config management UI
- 📋 **Phase 5: Email & Comparisons** - Next Up
- 📋 **Phase 5: Email, Webhooks & Comparisons** - Next Up
- Email notifications, alert rules, scan comparison
- 📋 **Phase 6: CLI as API Client** - Planned
- CLI for scripting and automation via API
- 📋 **Phase 7: Advanced Features** - Future
- CVE integration, timeline view, PDF export, enhanced reports
## Recent Bug Fixes
### 2025-11-17: Chart.js Infinite Canvas Growth Fix
**Issue:** Scan detail page (`scan_detail.html`) was experiencing infinite scrolling and page lock-up due to Chart.js canvas growing infinitely (height reaching 22302px+).
**Root Causes:**
1. Duplicate initialization - `loadScan()` was being called twice on page load
2. Multiple Chart.js instances created on the same canvas without destroying previous ones
3. Canvas element without fixed-height container caused infinite resize loop with `responsive: true` and `maintainAspectRatio: false`
**Fixes Applied:**
1. **Consolidated initialization** (`scan_detail.html:172-175`) - Moved `findPreviousScan()` and `loadHistoricalChart()` into `DOMContentLoaded` event listener, removed duplicate call
2. **Chart instance tracking** (`scan_detail.html:169`) - Added `let historyChart = null;` to store chart reference
3. **Destroy old charts** (`scan_detail.html:501-504`) - Added `historyChart.destroy()` before creating new chart instance
4. **Fixed-height container** (`scan_detail.html:136-138`) - Wrapped canvas in `<div style="position: relative; height: 300px;">` to prevent infinite resize loop
**Files Modified:**
- `web/templates/scan_detail.html`
**Status:** ✅ Fixed and tested
## Vision & Goals
SneakyScanner is a comprehensive **Flask web application** for infrastructure monitoring and security auditing. The primary interface is the web GUI, with a CLI API client planned for scripting and automation needs.
**Core Features:**
- **Centralized dashboard** for viewing scan history and trends
@@ -439,33 +420,360 @@ All API endpoints return JSON and follow RESTful conventions.
---
### Phase 5: Email & Comparisons
### Phase 5: Email, Webhooks & Comparisons
**Status:** Next Up
**Priority:** MEDIUM
**Goals:**
- Implement email notification system
- Create scan comparison reports
- Add alert rule configuration
- Implement email notification system for infrastructure misconfigurations
- Implement webhook notification system for real-time alerting
- Create scan comparison reports to detect drift
- Add alert rule configuration for unexpected exposure detection
**Core Use Case:**
Monitor infrastructure for misconfigurations that expose unexpected ports/services to the world. When a scan detects an open port that wasn't defined in the YAML config's `expected_ports` list, trigger immediate notifications via email and/or webhooks.
**Planned Features:**
1. **Email Notifications:**
- SMTP integration with configurable settings
- Alert email templates (Jinja2 HTML)
#### 1. Alert Rule Engine
**Purpose:** Automatically detect and classify infrastructure anomalies after each scan.
**Alert Types:**
- `unexpected_port` - Port open but not in config's `expected_ports` list
- `unexpected_service` - Service detected that doesn't match expected service name
- `cert_expiry` - SSL/TLS certificate expiring soon (configurable threshold)
- `ping_failed` - Expected host not responding to ping
- `service_down` - Previously detected service no longer responding
- `service_change` - Service version/product changed between scans
- `weak_tls` - TLS 1.0/1.1 detected or weak cipher suites
- `new_host` - New IP address responding in CIDR range
- `host_disappeared` - Previously seen IP no longer responding
**Alert Severity Levels:**
- `critical` - Unexpected internet-facing service (ports 80/443/22/3389/etc.)
- `warning` - Minor configuration drift or upcoming cert expiry
- `info` - Informational alerts (new host discovered, service version change)
**Alert Rule Configuration:**
```yaml
# Example alert rule configuration (stored in DB)
alert_rules:
- id: 1
rule_type: unexpected_port
enabled: true
severity: critical
email_enabled: true
webhook_enabled: true
filter_conditions:
ports: [22, 80, 443, 3389, 3306, 5432, 27017] # High-risk ports
- id: 2
rule_type: cert_expiry
enabled: true
severity: warning
threshold: 30 # Days before expiry
email_enabled: true
webhook_enabled: false
```
**Implementation:**
- Evaluate alert rules after each scan completes
- Compare current scan results to expected configuration
- Generate alerts and store in `alerts` table
- Trigger notifications based on rule configuration
- Alert deduplication (don't spam for same issue)
#### 2. Email Notifications
**Purpose:** Send detailed email alerts when infrastructure misconfigurations are detected.
**SMTP Configuration (via Settings API):**
```json
{
"smtp_server": "smtp.gmail.com",
"smtp_port": 587,
"smtp_use_tls": true,
"smtp_username": "alerts@example.com",
"smtp_password": "encrypted_password",
"smtp_from_email": "SneakyScanner <alerts@example.com>",
"smtp_to_emails": ["admin@example.com", "security@example.com"],
"email_alerts_enabled": true
}
```
**Email Template (Jinja2 HTML):**
```html
Subject: [SneakyScanner Alert] Unexpected Port Detected - {{ ip_address }}:{{ port }}
Body:
===============================================
SneakyScanner Security Alert
===============================================
Alert Type: {{ alert_type }}
Severity: {{ severity }}
Scan ID: {{ scan_id }}
Timestamp: {{ timestamp }}
Issue Detected:
{{ message }}
Details:
- IP Address: {{ ip_address }}
- Port: {{ port }}/{{ protocol }}
- Service: {{ service_name }} ({{ product }} {{ version }})
- Expected: No (not in expected_ports list)
Recommended Actions:
1. Verify if this service should be exposed
2. Check firewall rules for {{ ip_address }}
3. Review service configuration
4. Update scan config if this is intentional
View Full Scan Results:
{{ web_url }}/scans/{{ scan_id }}
===============================================
```
**Email Features:**
- HTML email with styled alert box (Bootstrap-based)
- Plain-text fallback for compatibility
- Alert summary with actionable recommendations
- Direct link to scan detail page
- Configurable recipients (multiple emails)
- Test email functionality in Settings UI
- Email delivery tracking (email_sent, email_sent_at in alerts table)
- Rate limiting to prevent email flood
**Email API Endpoints:**
- `POST /api/settings/email` - Configure SMTP settings
- `POST /api/settings/email/test` - Send test email
- `GET /api/alerts?email_sent=true` - Get alerts with email status
#### 3. Webhook Notifications
**Purpose:** Real-time HTTP POST notifications for integration with external systems (Slack, PagerDuty, custom dashboards, SIEM tools).
**Webhook Configuration (via Settings API):**
```json
{
"webhook_enabled": true,
"webhook_urls": [
{
"id": 1,
"name": "Slack Security Channel",
"url": "https://hooks.slack.com/services/XXX/YYY/ZZZ",
"enabled": true,
"auth_type": "none",
"custom_headers": {},
"alert_types": ["unexpected_port", "unexpected_service", "weak_tls"],
"severity_filter": ["critical", "warning"]
},
{
"id": 2,
"name": "PagerDuty",
"url": "https://events.pagerduty.com/v2/enqueue",
"enabled": true,
"auth_type": "bearer",
"auth_token": "encrypted_token",
"custom_headers": {
"Content-Type": "application/json"
},
"alert_types": ["unexpected_port"],
"severity_filter": ["critical"]
}
]
}
```
**Webhook Payload Format (JSON):**
```json
{
"event_type": "scan_alert",
"alert_id": 42,
"alert_type": "unexpected_port",
"severity": "critical",
"timestamp": "2025-11-17T14:23:45Z",
"scan": {
"scan_id": 123,
"title": "Production Network Scan",
"timestamp": "2025-11-17T14:15:00Z",
"config_file": "prod_config.yaml",
"triggered_by": "scheduled"
},
"alert_details": {
"message": "Unexpected port 3306 (MySQL) exposed on 192.168.1.100",
"ip_address": "192.168.1.100",
"port": 3306,
"protocol": "tcp",
"state": "open",
"service": {
"name": "mysql",
"product": "MySQL",
"version": "8.0.32"
},
"expected": false,
"site_name": "Production Servers"
},
"recommended_actions": [
"Verify if MySQL should be exposed externally",
"Check firewall rules for 192.168.1.100",
"Review MySQL bind-address configuration"
],
"web_url": "https://sneakyscanner.local/scans/123"
}
```
**Webhook Features:**
- Multiple webhook URLs with independent configuration
- Per-webhook filtering by alert type and severity
- Custom headers support (for API keys, auth tokens)
- Authentication methods:
- `none` - No authentication
- `bearer` - Bearer token in Authorization header
- `basic` - Basic authentication
- `custom` - Custom header-based auth
- Retry logic with exponential backoff (3 attempts)
- Webhook delivery tracking (webhook_sent, webhook_sent_at, webhook_response_code)
- Test webhook functionality in Settings UI
- Timeout configuration (default 10 seconds)
- Webhook delivery history and logs
**Webhook API Endpoints:**
- `POST /api/webhooks` - Create webhook configuration
- `GET /api/webhooks` - List all webhooks
- `PUT /api/webhooks/{id}` - Update webhook configuration
- `DELETE /api/webhooks/{id}` - Delete webhook
- `POST /api/webhooks/{id}/test` - Send test webhook
- `GET /api/webhooks/{id}/history` - Get delivery history
**Slack Integration Example:**
Transform webhook payload to Slack message format:
```json
{
"text": "SneakyScanner Alert: Unexpected Port Detected",
"attachments": [
{
"color": "danger",
"fields": [
{"title": "IP Address", "value": "192.168.1.100", "short": true},
{"title": "Port", "value": "3306/tcp", "short": true},
{"title": "Service", "value": "MySQL 8.0.32", "short": true},
{"title": "Severity", "value": "CRITICAL", "short": true}
],
"footer": "SneakyScanner",
"ts": 1700234625
}
]
}
```
#### 4. Alert Management UI
**Purpose:** Web interface for configuring alert rules, viewing alert history, and managing notifications.
**Pages:**
- `/alerts` - Alert history with filtering and search
- `/alerts/rules` - Alert rule configuration
- `/settings/email` - Email notification settings
- `/settings/webhooks` - Webhook configuration
**Alert History Features:**
- Filter by alert type, severity, date range, IP address
- Search by message content
- Bulk acknowledge/dismiss alerts
- Export alerts to CSV
- Alert detail modal with full context
**Alert Rule UI Features:**
- Enable/disable rules individually
- Configure severity levels
- Set thresholds (e.g., cert expiry days)
- Toggle email/webhook per rule
- Test rules against recent scans
#### 5. Scan Comparison
**Purpose:** Detect infrastructure drift by comparing two scans and highlighting changes.
**Comparison API:**
- `GET /api/scans/{id1}/compare/{id2}` - Compare two scans
**Comparison Response:**
```json
{
"scan1": {"id": 100, "timestamp": "2025-11-10T10:00:00Z"},
"scan2": {"id": 123, "timestamp": "2025-11-17T14:15:00Z"},
"summary": {
"new_ports": 2,
"removed_ports": 0,
"service_changes": 1,
"cert_changes": 0,
"new_hosts": 1,
"removed_hosts": 0
},
"differences": {
"new_ports": [
{"ip": "192.168.1.100", "port": 3306, "service": "mysql"}
],
"removed_ports": [],
"service_changes": [
{
"ip": "192.168.1.50",
"port": 22,
"old": "OpenSSH 8.2",
"new": "OpenSSH 8.9"
}
],
"new_hosts": [
{"ip": "192.168.1.200", "site": "Production Servers"}
]
}
}
```
**Comparison UI Features:**
- Side-by-side comparison view
- Color-coded differences (green=new, red=removed, yellow=changed)
- Filter by change type
- Export comparison report to PDF/HTML
- "Compare with previous scan" button on scan detail page
---
**Phase 5 Implementation Plan:**
1. **Week 1: Alert Rule Engine**
- Implement alert evaluation logic after scan completion
- Create `alerts` table population
- Add alert rule CRUD API
- Unit tests for alert detection
2. **Week 2: Email Notifications**
- SMTP integration with Flask-Mail
- Jinja2 email templates (HTML + plain text)
- Settings API for email configuration
- Test email functionality
- Email triggers for critical events
- Email delivery tracking
2. **Alert Rule Engine:**
- Alert types: unexpected ports, cert expiry, service changes, host down, weak TLS
- Alert rule creation and management UI
- Automatic evaluation after each scan
- Alert history with severity filtering
3. **Week 3: Webhook System**
- Webhook configuration API
- HTTP POST delivery with retry logic
- Webhook template system for different platforms
- Test webhook functionality
- Delivery tracking and logging
3. **Scan Comparison:**
- Compare two scans API endpoint
- Diff detection (new/removed ports, service changes, cert changes)
- Visual comparison UI with highlighting
- "Compare" button on scan list
4. **Week 4: Alert UI & Scan Comparison**
- Alert history page with filtering
- Alert rule management UI
- Email/webhook settings pages
- Scan comparison API and UI
- Integration testing
**Success Criteria:**
- Alert triggered within 30 seconds of scan completion
- Email delivered successfully to configured recipients
- Webhook POST delivered with retry on failure
- Scan comparison highlights all infrastructure changes
- Zero false positives for expected ports/services
- Alert deduplication prevents notification spam
---
@@ -560,34 +868,6 @@ All API endpoints return JSON and follow RESTful conventions.
- Gunicorn WSGI server
- Optional Nginx reverse proxy
## Prioritized Feature List
### Completed ✅ (Phases 1-4)
1. **Database foundation** (SQLite3 + SQLAlchemy)
2. **Flask web app core** (REST API, authentication)
3. **Dashboard with scan history** (list, detail, delete)
4. **Trend charts** (Chart.js - port counts over time)
5. **Scheduled scans** (APScheduler + cron expressions)
6. **Config creator** (CIDR-based, YAML editor)
### Next Up (Phase 5)
7. **Email notifications** (SMTP integration)
8. **Alert rules** (cert expiry, unexpected ports, etc.)
9. **Scan comparison reports** (diff view)
### Planned (Phase 6-7)
10. **CLI as API client** (token auth, scripting)
11. **Sortable/filterable tables** (DataTables.js)
12. **PDF export** (WeasyPrint)
13. **Vulnerability detection** (CVE integration)
14. **Timeline view** (visual scan history)
### Future/Deferred
15. **Multi-user support** (if requirements change)
16. **Slack/webhook integrations**
17. **Prometheus metrics**
18. **Advanced charts** (heatmaps, forecasts)
## Development Workflow
### Iteration Cycle
@@ -617,69 +897,6 @@ All API endpoints return JSON and follow RESTful conventions.
- **CLAUDE.md** - Developer documentation (architecture, code references)
- **API.md** - API documentation (OpenAPI/Swagger in Phase 4)
## Success Metrics
### Phase 1 Success ✅ ACHIEVED
- [x] Database creates successfully with all 11 tables
- [x] Settings can be stored/retrieved with encryption
- [x] Flask app starts without errors
- [x] API blueprints load correctly
- [x] All Python modules have valid syntax
- [x] Docker deployment configured
### Phase 2 Success ✅ ACHIEVED
- [x] Database stores scan results correctly
- [x] REST API functional with all endpoints
- [x] Background scans execute asynchronously
- [x] Authentication protects all routes
- [x] Web UI is intuitive and responsive
- [x] 100 tests passing with comprehensive coverage
- [x] Docker deployment production-ready
### Phase 3 Success ✅ ACHIEVED
- [x] Dashboard displays scans and trends with charts
- [x] Scheduled scans execute automatically
- [x] Historical trend charts show scan history
- [x] Real-time progress updates for running scans
### Phase 4 Success ✅ ACHIEVED
- [x] Users can create configs from CIDR ranges via web UI
- [x] YAML editor with syntax highlighting works correctly
- [x] Config management UI provides list/view/edit/download/delete operations
- [x] Direct YAML upload works for advanced users
- [x] Configs immediately usable in scan triggers and schedules
- [x] Delete protection prevents removal of configs used by schedules
- [x] All tests passing (25+ unit and integration tests)
### Phase 5 Success (Email & Comparisons)
- [ ] Email notifications sent for critical alerts
- [ ] Comparison reports show meaningful diffs
- [ ] Settings UI allows SMTP configuration without editing files
### Phase 6 Success (CLI as API Client)
- [ ] CLI can trigger scans via API
- [ ] API tokens work for authentication
- [ ] Standalone CLI mode still functional
### Phase 7 Success (Advanced Features)
- [ ] CVE integration provides actionable vulnerability data
- [ ] Timeline view helps track infrastructure changes
- [ ] PDF exports are shareable and professional
## Open Questions
### Technical Decisions
- **Flask vs. FastAPI?** - Sticking with Flask for simplicity, but FastAPI offers async and auto-docs
- **APScheduler vs. Celery?** - APScheduler for simplicity (no Redis/RabbitMQ needed), but Celery scales better
- **Bootstrap vs. Tailwind?** - Bootstrap for speed (pre-built components), Tailwind for customization
- **Chart.js vs. Plotly?** - Chart.js for lightweight, Plotly for interactive (decide in Phase 3)
### Product Questions
- **Should we support multiple configs per schedule?** - Start with 1:1, add later if needed
- **How many scans to keep in DB?** - Add retention setting (default: keep all)
- **Support for multi-tenancy?** - Not in scope (single-user), but database schema allows future expansion
- **Mobile app?** - Out of scope, but responsive web UI covers basics
## Resources & References
### Documentation