717 lines
30 KiB
Markdown
717 lines
30 KiB
Markdown
# SneakyScanner Roadmap
|
|
|
|
**Status:** Phase 4 Complete ✅ | Phase 5 Next Up
|
|
|
|
## Progress Overview
|
|
- ✅ **Phase 1: Foundation** - Complete (2025-11-13)
|
|
- Database schema, SQLAlchemy models, settings system, Flask app structure
|
|
- ✅ **Phase 2: Flask Web App Core** - Complete (2025-11-14)
|
|
- REST API, background jobs, authentication, web UI, testing (100 tests)
|
|
- ✅ **Phase 3: Dashboard & Scheduling** - Complete (2025-11-14)
|
|
- Dashboard, scan history, scheduled scans, trend charts
|
|
- ✅ **Phase 4: Config Creator** - Complete (2025-11-17)
|
|
- CIDR-based config creation, YAML editor, config management UI
|
|
- 📋 **Phase 5: Email & Comparisons** - Next Up
|
|
- Email notifications, alert rules, scan comparison
|
|
- 📋 **Phase 6: CLI as API Client** - Planned
|
|
- CLI for scripting and automation via API
|
|
- 📋 **Phase 7: Advanced Features** - Future
|
|
- CVE integration, timeline view, PDF export, enhanced reports
|
|
|
|
## Recent Bug Fixes
|
|
|
|
### 2025-11-17: Chart.js Infinite Canvas Growth Fix
|
|
**Issue:** Scan detail page (`scan_detail.html`) was experiencing infinite scrolling and page lock-up due to Chart.js canvas growing infinitely (height reaching 22302px+).
|
|
|
|
**Root Causes:**
|
|
1. Duplicate initialization - `loadScan()` was being called twice on page load
|
|
2. Multiple Chart.js instances created on the same canvas without destroying previous ones
|
|
3. Canvas element without fixed-height container caused infinite resize loop with `responsive: true` and `maintainAspectRatio: false`
|
|
|
|
**Fixes Applied:**
|
|
1. **Consolidated initialization** (`scan_detail.html:172-175`) - Moved `findPreviousScan()` and `loadHistoricalChart()` into `DOMContentLoaded` event listener, removed duplicate call
|
|
2. **Chart instance tracking** (`scan_detail.html:169`) - Added `let historyChart = null;` to store chart reference
|
|
3. **Destroy old charts** (`scan_detail.html:501-504`) - Added `historyChart.destroy()` before creating new chart instance
|
|
4. **Fixed-height container** (`scan_detail.html:136-138`) - Wrapped canvas in `<div style="position: relative; height: 300px;">` to prevent infinite resize loop
|
|
|
|
**Files Modified:**
|
|
- `web/templates/scan_detail.html`
|
|
|
|
**Status:** ✅ Fixed and tested
|
|
|
|
## Vision & Goals
|
|
|
|
SneakyScanner is a comprehensive **Flask web application** for infrastructure monitoring and security auditing. The primary interface is the web GUI, with a CLI API client planned for scripting and automation needs.
|
|
|
|
**Core Features:**
|
|
- **Centralized dashboard** for viewing scan history and trends
|
|
- **Scheduled scanning** for continuous infrastructure monitoring
|
|
- **Email notifications** for critical changes and certificate expirations (coming soon)
|
|
- **Historical analysis** with charts, graphs, and comparison reports
|
|
- **Config creator** for easy CIDR-based scan configuration
|
|
- **RESTful API** for integration and automation
|
|
- **Simple deployment** using SQLite3 (single-user, self-hosted)
|
|
|
|
**Planned:**
|
|
- **CLI API client** for scripting and automation workflows (Phase 6)
|
|
|
|
## Target Users
|
|
|
|
- **Infrastructure teams** monitoring on-premises networks
|
|
- **Security professionals** performing periodic security audits
|
|
- **DevOps engineers** tracking infrastructure drift
|
|
- **Single users or small teams** (not enterprise multi-tenant)
|
|
|
|
## Technology Stack
|
|
|
|
### Backend
|
|
- **Flask** 3.x - Lightweight Python web framework
|
|
- **SQLAlchemy** 2.x - ORM for database abstraction
|
|
- **SQLite3** - Embedded database (easy deployment, sufficient for single-user)
|
|
- **APScheduler** 3.x - Background job scheduling for periodic scans
|
|
- **Flask-Login** - Simple session-based authentication
|
|
- **Flask-CORS** - API access control (optional for CLI API client)
|
|
- **Marshmallow** - API serialization/deserialization
|
|
|
|
### Frontend
|
|
- **Jinja2** - Server-side templating (already in use)
|
|
- **Bootstrap 5** - Responsive UI framework with dark theme support
|
|
- **Chart.js** 4.x - Lightweight charting library for trends
|
|
- **DataTables.js** - Interactive sortable/filterable tables (Phase 6)
|
|
- **Vanilla JavaScript** - Keep dependencies minimal
|
|
|
|
### Infrastructure
|
|
- **Docker Compose** - Multi-container orchestration (Flask app + Scanner)
|
|
- **Gunicorn** - WSGI server for production
|
|
- **Nginx** - Reverse proxy (optional, for production deployments)
|
|
|
|
### Existing Components (Keep)
|
|
- **Masscan** - Fast port discovery
|
|
- **Nmap** - Service detection
|
|
- **Playwright** - Screenshot capture
|
|
- **sslyze** - SSL/TLS analysis
|
|
|
|
## Architecture Overview
|
|
|
|
```
|
|
┌─────────────────────────────────────────────────────────────┐
|
|
│ Flask Web Application │
|
|
│ ┌──────────────┐ ┌──────────────┐ ┌──────────────────┐ │
|
|
│ │ Web UI │ │ REST API │ │ Scheduler │ │
|
|
│ │ (Dashboard) │ │ (JSON/CRUD) │ │ (APScheduler) │ │
|
|
│ └──────┬───────┘ └──────┬───────┘ └────────┬─────────┘ │
|
|
│ │ │ │ │
|
|
│ └─────────────────┴────────────────────┘ │
|
|
│ │ │
|
|
│ ┌────────▼────────┐ │
|
|
│ │ SQLAlchemy │ │
|
|
│ │ (ORM Layer) │ │
|
|
│ └────────┬────────┘ │
|
|
│ │ │
|
|
│ ┌────────▼────────┐ │
|
|
│ │ SQLite3 DB │ │
|
|
│ │ (scan history) │ │
|
|
│ └─────────────────┘ │
|
|
└───────────────────────────┬─────────────────────────────────┘
|
|
│
|
|
┌──────────▼──────────┐
|
|
│ Scanner Engine │
|
|
│ (scanner.py) │
|
|
│ ┌────────────────┐ │
|
|
│ │ Masscan/Nmap │ │
|
|
│ │ Playwright │ │
|
|
│ │ sslyze │ │
|
|
│ └────────────────┘ │
|
|
└─────────────────────┘
|
|
│
|
|
┌──────────▼──────────┐
|
|
│ CLI API Client │
|
|
│ (optional future) │
|
|
└─────────────────────┘
|
|
```
|
|
|
|
### Component Interaction
|
|
|
|
1. **Web UI** - User interacts with dashboard, triggers scans, views history
|
|
2. **REST API** - Handles requests from web UI and CLI client
|
|
3. **Scheduler (APScheduler)** - Triggers scans based on cron-like schedules
|
|
4. **SQLAlchemy ORM** - Abstracts database operations
|
|
5. **SQLite3 Database** - Stores scan results, schedules, settings, alerts
|
|
6. **Scanner Engine** - Performs actual network scanning (masscan, nmap, etc.)
|
|
7. **CLI API Client** - Future: thin client that calls Flask API
|
|
|
|
## Database Schema Design
|
|
|
|
### Core Tables
|
|
|
|
#### `scans`
|
|
Stores metadata about each scan execution.
|
|
|
|
| Column | Type | Description |
|
|
|--------|------|-------------|
|
|
| `id` | INTEGER PRIMARY KEY | Unique scan ID |
|
|
| `timestamp` | DATETIME | Scan start time (UTC) |
|
|
| `duration` | FLOAT | Total scan duration (seconds) |
|
|
| `status` | VARCHAR(20) | `running`, `completed`, `failed` |
|
|
| `config_file` | TEXT | Path to YAML config used |
|
|
| `title` | TEXT | Scan title from config |
|
|
| `json_path` | TEXT | Path to JSON report |
|
|
| `html_path` | TEXT | Path to HTML report |
|
|
| `zip_path` | TEXT | Path to ZIP archive |
|
|
| `screenshot_dir` | TEXT | Path to screenshot directory |
|
|
| `created_at` | DATETIME | Record creation time |
|
|
| `triggered_by` | VARCHAR(50) | `manual`, `scheduled`, `api` |
|
|
| `schedule_id` | INTEGER | FK to schedules (if triggered by schedule) |
|
|
|
|
#### `scan_sites`
|
|
Logical grouping of IPs by site.
|
|
|
|
| Column | Type | Description |
|
|
|--------|------|-------------|
|
|
| `id` | INTEGER PRIMARY KEY | Unique site record ID |
|
|
| `scan_id` | INTEGER | FK to scans |
|
|
| `site_name` | VARCHAR(255) | Site name from config |
|
|
|
|
#### `scan_ips`
|
|
IP addresses scanned in each scan.
|
|
|
|
| Column | Type | Description |
|
|
|--------|------|-------------|
|
|
| `id` | INTEGER PRIMARY KEY | Unique IP record ID |
|
|
| `scan_id` | INTEGER | FK to scans |
|
|
| `site_id` | INTEGER | FK to scan_sites |
|
|
| `ip_address` | VARCHAR(45) | IPv4 or IPv6 address |
|
|
| `ping_expected` | BOOLEAN | Expected ping response |
|
|
| `ping_actual` | BOOLEAN | Actual ping response |
|
|
|
|
#### `scan_ports`
|
|
Discovered TCP/UDP ports.
|
|
|
|
| Column | Type | Description |
|
|
|--------|------|-------------|
|
|
| `id` | INTEGER PRIMARY KEY | Unique port record ID |
|
|
| `scan_id` | INTEGER | FK to scans |
|
|
| `ip_id` | INTEGER | FK to scan_ips |
|
|
| `port` | INTEGER | Port number (1-65535) |
|
|
| `protocol` | VARCHAR(10) | `tcp` or `udp` |
|
|
| `expected` | BOOLEAN | Was this port expected? |
|
|
| `state` | VARCHAR(20) | `open`, `closed`, `filtered` |
|
|
|
|
#### `scan_services`
|
|
Detected services on open ports.
|
|
|
|
| Column | Type | Description |
|
|
|--------|------|-------------|
|
|
| `id` | INTEGER PRIMARY KEY | Unique service record ID |
|
|
| `scan_id` | INTEGER | FK to scans |
|
|
| `port_id` | INTEGER | FK to scan_ports |
|
|
| `service_name` | VARCHAR(100) | Service name (e.g., `ssh`, `http`) |
|
|
| `product` | VARCHAR(255) | Product name (e.g., `OpenSSH`) |
|
|
| `version` | VARCHAR(100) | Version string |
|
|
| `extrainfo` | TEXT | Additional nmap info |
|
|
| `ostype` | VARCHAR(100) | OS type if detected |
|
|
| `http_protocol` | VARCHAR(10) | `http` or `https` (if web service) |
|
|
| `screenshot_path` | TEXT | Relative path to screenshot |
|
|
|
|
#### `scan_certificates`
|
|
SSL/TLS certificates discovered on HTTPS services.
|
|
|
|
| Column | Type | Description |
|
|
|--------|------|-------------|
|
|
| `id` | INTEGER PRIMARY KEY | Unique certificate record ID |
|
|
| `scan_id` | INTEGER | FK to scans |
|
|
| `service_id` | INTEGER | FK to scan_services |
|
|
| `subject` | TEXT | Certificate subject (CN) |
|
|
| `issuer` | TEXT | Certificate issuer |
|
|
| `serial_number` | TEXT | Serial number |
|
|
| `not_valid_before` | DATETIME | Validity start date |
|
|
| `not_valid_after` | DATETIME | Validity end date |
|
|
| `days_until_expiry` | INTEGER | Days until expiration |
|
|
| `sans` | TEXT | JSON array of SANs |
|
|
| `is_self_signed` | BOOLEAN | Self-signed certificate flag |
|
|
|
|
#### `scan_tls_versions`
|
|
TLS version support and cipher suites.
|
|
|
|
| Column | Type | Description |
|
|
|--------|------|-------------|
|
|
| `id` | INTEGER PRIMARY KEY | Unique TLS version record ID |
|
|
| `scan_id` | INTEGER | FK to scans |
|
|
| `certificate_id` | INTEGER | FK to scan_certificates |
|
|
| `tls_version` | VARCHAR(20) | `TLS 1.0`, `TLS 1.1`, `TLS 1.2`, `TLS 1.3` |
|
|
| `supported` | BOOLEAN | Is this version supported? |
|
|
| `cipher_suites` | TEXT | JSON array of cipher suites |
|
|
|
|
### Scheduling & Notifications Tables
|
|
|
|
#### `schedules`
|
|
Scheduled scan configurations.
|
|
|
|
| Column | Type | Description |
|
|
|--------|------|-------------|
|
|
| `id` | INTEGER PRIMARY KEY | Unique schedule ID |
|
|
| `name` | VARCHAR(255) | Schedule name (e.g., "Daily prod scan") |
|
|
| `config_file` | TEXT | Path to YAML config |
|
|
| `cron_expression` | VARCHAR(100) | Cron-like schedule (e.g., `0 2 * * *`) |
|
|
| `enabled` | BOOLEAN | Is schedule active? |
|
|
| `last_run` | DATETIME | Last execution time |
|
|
| `next_run` | DATETIME | Next scheduled execution |
|
|
| `created_at` | DATETIME | Schedule creation time |
|
|
| `updated_at` | DATETIME | Last modification time |
|
|
|
|
#### `alerts`
|
|
Alert history and notifications sent.
|
|
|
|
| Column | Type | Description |
|
|
|--------|------|-------------|
|
|
| `id` | INTEGER PRIMARY KEY | Unique alert ID |
|
|
| `scan_id` | INTEGER | FK to scans |
|
|
| `alert_type` | VARCHAR(50) | `new_port`, `cert_expiry`, `service_change`, `ping_failed` |
|
|
| `severity` | VARCHAR(20) | `info`, `warning`, `critical` |
|
|
| `message` | TEXT | Human-readable alert message |
|
|
| `ip_address` | VARCHAR(45) | Related IP (optional) |
|
|
| `port` | INTEGER | Related port (optional) |
|
|
| `email_sent` | BOOLEAN | Was email notification sent? |
|
|
| `email_sent_at` | DATETIME | Email send timestamp |
|
|
| `created_at` | DATETIME | Alert creation time |
|
|
|
|
#### `alert_rules`
|
|
User-defined alert rules.
|
|
|
|
| Column | Type | Description |
|
|
|--------|------|-------------|
|
|
| `id` | INTEGER PRIMARY KEY | Unique rule ID |
|
|
| `rule_type` | VARCHAR(50) | `unexpected_port`, `cert_expiry`, `service_down`, etc. |
|
|
| `enabled` | BOOLEAN | Is rule active? |
|
|
| `threshold` | INTEGER | Threshold value (e.g., days for cert expiry) |
|
|
| `email_enabled` | BOOLEAN | Send email for this rule? |
|
|
| `created_at` | DATETIME | Rule creation time |
|
|
|
|
### Settings Table
|
|
|
|
#### `settings`
|
|
Application configuration key-value store.
|
|
|
|
| Column | Type | Description |
|
|
|--------|------|-------------|
|
|
| `id` | INTEGER PRIMARY KEY | Unique setting ID |
|
|
| `key` | VARCHAR(255) UNIQUE | Setting key (e.g., `smtp_server`) |
|
|
| `value` | TEXT | Setting value (JSON for complex values) |
|
|
| `updated_at` | DATETIME | Last modification time |
|
|
|
|
**Example Settings:**
|
|
- `smtp_server` - SMTP server hostname
|
|
- `smtp_port` - SMTP port (587, 465, 25)
|
|
- `smtp_username` - SMTP username
|
|
- `smtp_password` - SMTP password (encrypted)
|
|
- `smtp_from_email` - From email address
|
|
- `smtp_to_emails` - JSON array of recipient emails
|
|
- `app_password` - Single-user password hash (bcrypt)
|
|
- `retention_days` - How long to keep old scans (0 = forever)
|
|
|
|
## API Design
|
|
|
|
### REST API Endpoints
|
|
|
|
All API endpoints return JSON and follow RESTful conventions.
|
|
|
|
#### Scans
|
|
|
|
| Method | Endpoint | Description | Request Body | Response |
|
|
|--------|----------|-------------|--------------|----------|
|
|
| `GET` | `/api/scans` | List all scans (paginated) | - | `{ "scans": [...], "total": N, "page": 1 }` |
|
|
| `GET` | `/api/scans/{id}` | Get scan details | - | `{ "scan": {...} }` |
|
|
| `POST` | `/api/scans` | Trigger new scan | `{ "config_file": "path" }` | `{ "scan_id": N, "status": "running" }` |
|
|
| `DELETE` | `/api/scans/{id}` | Delete scan and files | - | `{ "status": "deleted" }` |
|
|
| `GET` | `/api/scans/{id}/status` | Get scan status | - | `{ "status": "running", "progress": "45%" }` |
|
|
| `GET` | `/api/scans/{id1}/compare/{id2}` | Compare two scans | - | `{ "diff": {...} }` |
|
|
|
|
#### Schedules
|
|
|
|
| Method | Endpoint | Description | Request Body | Response |
|
|
|--------|----------|-------------|--------------|----------|
|
|
| `GET` | `/api/schedules` | List all schedules | - | `{ "schedules": [...] }` |
|
|
| `GET` | `/api/schedules/{id}` | Get schedule details | - | `{ "schedule": {...} }` |
|
|
| `POST` | `/api/schedules` | Create new schedule | `{ "name": "...", "config_file": "...", "cron_expression": "..." }` | `{ "schedule_id": N }` |
|
|
| `PUT` | `/api/schedules/{id}` | Update schedule | `{ "enabled": true, "cron_expression": "..." }` | `{ "status": "updated" }` |
|
|
| `DELETE` | `/api/schedules/{id}` | Delete schedule | - | `{ "status": "deleted" }` |
|
|
| `POST` | `/api/schedules/{id}/trigger` | Manually trigger scheduled scan | - | `{ "scan_id": N }` |
|
|
|
|
#### Alerts
|
|
|
|
| Method | Endpoint | Description | Request Body | Response |
|
|
|--------|----------|-------------|--------------|----------|
|
|
| `GET` | `/api/alerts` | List recent alerts | - | `{ "alerts": [...] }` |
|
|
| `GET` | `/api/alerts/rules` | List alert rules | - | `{ "rules": [...] }` |
|
|
| `POST` | `/api/alerts/rules` | Create alert rule | `{ "rule_type": "...", "threshold": N }` | `{ "rule_id": N }` |
|
|
| `PUT` | `/api/alerts/rules/{id}` | Update alert rule | `{ "enabled": false }` | `{ "status": "updated" }` |
|
|
| `DELETE` | `/api/alerts/rules/{id}` | Delete alert rule | - | `{ "status": "deleted" }` |
|
|
|
|
#### Settings
|
|
|
|
| Method | Endpoint | Description | Request Body | Response |
|
|
|--------|----------|-------------|--------------|----------|
|
|
| `GET` | `/api/settings` | Get all settings (sanitized) | - | `{ "settings": {...} }` |
|
|
| `PUT` | `/api/settings` | Update settings | `{ "smtp_server": "...", ... }` | `{ "status": "updated" }` |
|
|
| `POST` | `/api/settings/test-email` | Test email configuration | - | `{ "status": "sent" }` |
|
|
|
|
#### Statistics & Trends
|
|
|
|
| Method | Endpoint | Description | Request Body | Response |
|
|
|--------|----------|-------------|--------------|----------|
|
|
| `GET` | `/api/stats/summary` | Dashboard summary stats | - | `{ "total_scans": N, "last_scan": "...", ... }` |
|
|
| `GET` | `/api/stats/trends` | Trend data for charts | `?days=30&metric=port_count` | `{ "data": [...] }` |
|
|
| `GET` | `/api/stats/certificates` | Certificate expiry overview | - | `{ "expiring_soon": [...], "expired": [...] }` |
|
|
|
|
### Authentication
|
|
|
|
**Phase 2-3:** Simple session-based authentication (single-user)
|
|
- Login endpoint: `POST /api/auth/login` (username/password)
|
|
- Logout endpoint: `POST /api/auth/logout`
|
|
- Session cookies with Flask-Login
|
|
- Password stored as bcrypt hash in settings table
|
|
|
|
**Phase 5:** API token authentication for CLI client
|
|
- Generate API token: `POST /api/auth/token`
|
|
- Revoke token: `DELETE /api/auth/token`
|
|
- CLI sends token in `Authorization: Bearer <token>` header
|
|
|
|
## Phased Roadmap
|
|
|
|
### Phase 1: Foundation ✅ COMPLETE
|
|
**Completed:** 2025-11-13
|
|
|
|
**Deliverables:**
|
|
- SQLite database with 11 tables (scans, sites, IPs, ports, services, certificates, TLS versions, schedules, alerts, alert_rules, settings)
|
|
- SQLAlchemy ORM models with relationships
|
|
- Alembic migration system
|
|
- Settings system with encryption (bcrypt for passwords, Fernet for sensitive data)
|
|
- Flask app structure with API blueprints
|
|
- Docker Compose deployment configuration
|
|
- Validation script for verification
|
|
|
|
---
|
|
|
|
### Phase 2: Flask Web App Core ✅ COMPLETE
|
|
**Completed:** 2025-11-14
|
|
|
|
**Deliverables:**
|
|
- REST API with 8 endpoints (scans: trigger, list, get, status, delete; settings: get, update, test-email)
|
|
- Background job queue using APScheduler (up to 3 concurrent scans)
|
|
- Session-based authentication with Flask-Login
|
|
- Database integration for scan results (full normalized schema population)
|
|
- Web UI templates (dashboard, scan list/detail, login, error pages)
|
|
- Error handling with content negotiation (JSON/HTML) and request IDs
|
|
- Logging system with rotating file handlers
|
|
- Production Docker Compose deployment
|
|
- Comprehensive test suite (100 tests, all passing)
|
|
- Documentation (API_REFERENCE.md, DEPLOYMENT.md)
|
|
|
|
---
|
|
|
|
### Phase 3: Dashboard & Scheduling ✅ COMPLETE
|
|
**Completed:** 2025-11-14
|
|
|
|
**Deliverables:**
|
|
- Dashboard with summary stats (total scans, IPs, ports, services)
|
|
- Recent scans table with clickable details
|
|
- Scan detail page with full results display
|
|
- Historical trend charts using Chart.js (port counts over time)
|
|
- Scheduled scan management UI (create, edit, delete, enable/disable)
|
|
- Schedule execution with APScheduler and cron expressions
|
|
- Manual scan trigger from web UI
|
|
- Navigation menu (Dashboard, Scans, Schedules, Configs, Settings)
|
|
- Download buttons for scan reports (JSON, HTML, ZIP)
|
|
|
|
---
|
|
|
|
### Phase 4: Config Creator ✅ COMPLETE
|
|
**Completed:** 2025-11-17
|
|
|
|
**Deliverables:**
|
|
- CIDR-based config creation UI (simplified workflow for quick config generation)
|
|
- YAML editor with CodeMirror (syntax highlighting, line numbers)
|
|
- Config management UI (list, view, edit, download, delete)
|
|
- Direct YAML upload for advanced users
|
|
- REST API for config operations (7 endpoints: list, get, create, update, delete, upload, download)
|
|
- Schedule dependency protection (prevents deleting configs used by schedules)
|
|
- Comprehensive testing (25+ unit and integration tests)
|
|
|
|
---
|
|
|
|
### Phase 5: Email & Comparisons
|
|
**Status:** Next Up
|
|
**Priority:** MEDIUM
|
|
|
|
**Goals:**
|
|
- Implement email notification system
|
|
- Create scan comparison reports
|
|
- Add alert rule configuration
|
|
|
|
**Planned Features:**
|
|
1. **Email Notifications:**
|
|
- SMTP integration with configurable settings
|
|
- Alert email templates (Jinja2 HTML)
|
|
- Test email functionality
|
|
- Email triggers for critical events
|
|
|
|
2. **Alert Rule Engine:**
|
|
- Alert types: unexpected ports, cert expiry, service changes, host down, weak TLS
|
|
- Alert rule creation and management UI
|
|
- Automatic evaluation after each scan
|
|
- Alert history with severity filtering
|
|
|
|
3. **Scan Comparison:**
|
|
- Compare two scans API endpoint
|
|
- Diff detection (new/removed ports, service changes, cert changes)
|
|
- Visual comparison UI with highlighting
|
|
- "Compare" button on scan list
|
|
|
|
---
|
|
|
|
### Phase 6: CLI as API Client
|
|
**Status:** Planned
|
|
**Priority:** MEDIUM
|
|
|
|
**Goals:**
|
|
- Create CLI API client for scripting and automation
|
|
- Maintain standalone mode for testing
|
|
- API token authentication
|
|
|
|
**Planned Features:**
|
|
1. **API Client Mode:**
|
|
- `--api-mode` flag to enable API client mode
|
|
- `--api-url` and `--api-token` arguments
|
|
- Trigger scans via API, poll for status, download results
|
|
- Scans stored centrally in database
|
|
- Standalone mode still available for testing
|
|
|
|
2. **API Token System:**
|
|
- Token generation UI in settings page
|
|
- Secure token storage (hashed in database)
|
|
- Token authentication middleware
|
|
- Token expiration and revocation
|
|
|
|
3. **Benefits:**
|
|
- Centralized scan history accessible via web dashboard
|
|
- No need to mount volumes for output
|
|
- Scheduled scans managed through web UI
|
|
- Scriptable automation while leveraging web features
|
|
|
|
---
|
|
|
|
### Phase 7: Advanced Features
|
|
**Status:** Future/Deferred
|
|
**Priority:** LOW
|
|
|
|
**Planned Features:**
|
|
1. **Enhanced Reports:**
|
|
- Sortable/filterable tables (DataTables.js)
|
|
- Inline screenshot thumbnails with lightbox
|
|
- PDF export (WeasyPrint)
|
|
|
|
2. **Vulnerability Detection:**
|
|
- CVE database integration (NVD API)
|
|
- Service version matching to known CVEs
|
|
- CVSS severity scores
|
|
- Alert rules for critical CVEs
|
|
|
|
3. **Timeline View:**
|
|
- Visual scan history timeline
|
|
- Filter by site/IP
|
|
- Event annotations
|
|
|
|
4. **Advanced Charts:**
|
|
- Port activity heatmap
|
|
- Service version tracking
|
|
- Certificate expiration forecast
|
|
|
|
5. **Integrations:**
|
|
- Slack notifications
|
|
- Webhook support
|
|
- Prometheus metrics export
|
|
- CSV export/import
|
|
|
|
---
|
|
|
|
## Current Architecture
|
|
|
|
**Primary Interface:** Web GUI (Phases 1-4 Complete)
|
|
- Full-featured Flask web application
|
|
- Dashboard, scan management, scheduling, config creator
|
|
- REST API for all operations
|
|
- Single-user deployment with SQLite
|
|
|
|
**Coming Soon:** CLI API Client (Phase 6 Planned)
|
|
- Thin client for scripting and automation
|
|
- Calls Flask API for scan operations
|
|
- Results stored centrally in database
|
|
- Access to all web features via command line
|
|
|
|
**Core Scanning Engine:**
|
|
- Masscan for port discovery
|
|
- Nmap for service detection
|
|
- Playwright for screenshots
|
|
- sslyze for SSL/TLS analysis
|
|
|
|
**Deployment:**
|
|
- Docker Compose for easy deployment
|
|
- SQLite database (single-user, embedded)
|
|
- Gunicorn WSGI server
|
|
- Optional Nginx reverse proxy
|
|
|
|
## Prioritized Feature List
|
|
|
|
### Completed ✅ (Phases 1-4)
|
|
1. **Database foundation** (SQLite3 + SQLAlchemy)
|
|
2. **Flask web app core** (REST API, authentication)
|
|
3. **Dashboard with scan history** (list, detail, delete)
|
|
4. **Trend charts** (Chart.js - port counts over time)
|
|
5. **Scheduled scans** (APScheduler + cron expressions)
|
|
6. **Config creator** (CIDR-based, YAML editor)
|
|
|
|
### Next Up (Phase 5)
|
|
7. **Email notifications** (SMTP integration)
|
|
8. **Alert rules** (cert expiry, unexpected ports, etc.)
|
|
9. **Scan comparison reports** (diff view)
|
|
|
|
### Planned (Phase 6-7)
|
|
10. **CLI as API client** (token auth, scripting)
|
|
11. **Sortable/filterable tables** (DataTables.js)
|
|
12. **PDF export** (WeasyPrint)
|
|
13. **Vulnerability detection** (CVE integration)
|
|
14. **Timeline view** (visual scan history)
|
|
|
|
### Future/Deferred
|
|
15. **Multi-user support** (if requirements change)
|
|
16. **Slack/webhook integrations**
|
|
17. **Prometheus metrics**
|
|
18. **Advanced charts** (heatmaps, forecasts)
|
|
|
|
## Development Workflow
|
|
|
|
### Iteration Cycle
|
|
1. **Plan** - Define features for phase
|
|
2. **Implement** - Code backend + frontend
|
|
3. **Test** - Unit tests + manual testing
|
|
4. **Deploy** - Update Docker Compose
|
|
5. **Document** - Update README.md, ROADMAP.md
|
|
6. **Review** - Get user feedback
|
|
7. **Iterate** - Adjust priorities based on feedback
|
|
|
|
### Git Workflow
|
|
- **main branch** - Stable releases
|
|
- **develop branch** - Active development
|
|
- **feature branches** - Individual features (`feature/dashboard`, `feature/scheduler`)
|
|
- **Pull requests** - Review before merge
|
|
|
|
### Testing Strategy
|
|
- **Unit tests** - pytest for models, API endpoints
|
|
- **Integration tests** - Full scan → DB → API workflow
|
|
- **Manual testing** - UI/UX testing in browser
|
|
- **Performance tests** - Large scans, database queries
|
|
|
|
### Documentation
|
|
- **README.md** - User-facing documentation (updated each phase)
|
|
- **ROADMAP.md** - This file (updated as priorities shift)
|
|
- **CLAUDE.md** - Developer documentation (architecture, code references)
|
|
- **API.md** - API documentation (OpenAPI/Swagger in Phase 4)
|
|
|
|
## Success Metrics
|
|
|
|
### Phase 1 Success ✅ ACHIEVED
|
|
- [x] Database creates successfully with all 11 tables
|
|
- [x] Settings can be stored/retrieved with encryption
|
|
- [x] Flask app starts without errors
|
|
- [x] API blueprints load correctly
|
|
- [x] All Python modules have valid syntax
|
|
- [x] Docker deployment configured
|
|
|
|
### Phase 2 Success ✅ ACHIEVED
|
|
- [x] Database stores scan results correctly
|
|
- [x] REST API functional with all endpoints
|
|
- [x] Background scans execute asynchronously
|
|
- [x] Authentication protects all routes
|
|
- [x] Web UI is intuitive and responsive
|
|
- [x] 100 tests passing with comprehensive coverage
|
|
- [x] Docker deployment production-ready
|
|
|
|
### Phase 3 Success ✅ ACHIEVED
|
|
- [x] Dashboard displays scans and trends with charts
|
|
- [x] Scheduled scans execute automatically
|
|
- [x] Historical trend charts show scan history
|
|
- [x] Real-time progress updates for running scans
|
|
|
|
### Phase 4 Success ✅ ACHIEVED
|
|
- [x] Users can create configs from CIDR ranges via web UI
|
|
- [x] YAML editor with syntax highlighting works correctly
|
|
- [x] Config management UI provides list/view/edit/download/delete operations
|
|
- [x] Direct YAML upload works for advanced users
|
|
- [x] Configs immediately usable in scan triggers and schedules
|
|
- [x] Delete protection prevents removal of configs used by schedules
|
|
- [x] All tests passing (25+ unit and integration tests)
|
|
|
|
### Phase 5 Success (Email & Comparisons)
|
|
- [ ] Email notifications sent for critical alerts
|
|
- [ ] Comparison reports show meaningful diffs
|
|
- [ ] Settings UI allows SMTP configuration without editing files
|
|
|
|
### Phase 6 Success (CLI as API Client)
|
|
- [ ] CLI can trigger scans via API
|
|
- [ ] API tokens work for authentication
|
|
- [ ] Standalone CLI mode still functional
|
|
|
|
### Phase 7 Success (Advanced Features)
|
|
- [ ] CVE integration provides actionable vulnerability data
|
|
- [ ] Timeline view helps track infrastructure changes
|
|
- [ ] PDF exports are shareable and professional
|
|
|
|
## Open Questions
|
|
|
|
### Technical Decisions
|
|
- **Flask vs. FastAPI?** - Sticking with Flask for simplicity, but FastAPI offers async and auto-docs
|
|
- **APScheduler vs. Celery?** - APScheduler for simplicity (no Redis/RabbitMQ needed), but Celery scales better
|
|
- **Bootstrap vs. Tailwind?** - Bootstrap for speed (pre-built components), Tailwind for customization
|
|
- **Chart.js vs. Plotly?** - Chart.js for lightweight, Plotly for interactive (decide in Phase 3)
|
|
|
|
### Product Questions
|
|
- **Should we support multiple configs per schedule?** - Start with 1:1, add later if needed
|
|
- **How many scans to keep in DB?** - Add retention setting (default: keep all)
|
|
- **Support for multi-tenancy?** - Not in scope (single-user), but database schema allows future expansion
|
|
- **Mobile app?** - Out of scope, but responsive web UI covers basics
|
|
|
|
## Resources & References
|
|
|
|
### Documentation
|
|
- [Flask Documentation](https://flask.palletsprojects.com/)
|
|
- [SQLAlchemy ORM](https://docs.sqlalchemy.org/)
|
|
- [APScheduler](https://apscheduler.readthedocs.io/)
|
|
- [Chart.js](https://www.chartjs.org/docs/)
|
|
- [Bootstrap 5](https://getbootstrap.com/docs/5.3/)
|
|
|
|
### Tutorials
|
|
- [Flask Mega-Tutorial](https://blog.miguelgrinberg.com/post/the-flask-mega-tutorial-part-i-hello-world)
|
|
- [SQLAlchemy Tutorial](https://docs.sqlalchemy.org/en/20/tutorial/)
|
|
- [APScheduler with Flask](https://github.com/viniciuschiele/flask-apscheduler)
|
|
|
|
### Similar Projects (Inspiration)
|
|
- [OpenVAS](https://www.openvas.org/) - Vulnerability scanner with web UI
|
|
- [Nessus](https://www.tenable.com/products/nessus) - Commercial scanner (inspiration for UI/UX)
|
|
- [OWASP ZAP](https://www.zaproxy.org/) - Web app scanner (comparison reports, alerts)
|
|
|
|
## Changelog
|
|
|
|
| Date | Version | Changes |
|
|
|------|---------|---------|
|
|
| 2025-11-14 | 1.0 | Initial roadmap created based on user requirements |
|
|
| 2025-11-13 | 1.1 | **Phase 1 COMPLETE** - Database schema, SQLAlchemy models, Flask app structure, settings system with encryption, Alembic migrations, API blueprints, Docker support, validation script |
|
|
| 2025-11-14 | 1.2 | **Phase 2 COMPLETE** - REST API (5 scan endpoints, 3 settings endpoints), background jobs (APScheduler), authentication (Flask-Login), web UI (dashboard, scans, login, errors), error handling (content negotiation, request IDs, logging), 100 tests passing, comprehensive documentation (API_REFERENCE.md, DEPLOYMENT.md, PHASE2_COMPLETE.md) |
|
|
| 2025-11-17 | 1.3 | **Bug Fix** - Fixed Chart.js infinite canvas growth issue in scan detail page (duplicate initialization, missing chart.destroy(), missing fixed-height container) |
|
|
| 2025-11-17 | 1.4 | **Phase 4 COMPLETE** - Config Creator with CIDR-based creation, YAML editor (CodeMirror), config management UI (list/edit/delete), REST API (7 endpoints), Docker volume permissions fix, comprehensive testing and documentation |
|
|
| 2025-11-17 | 1.5 | **Roadmap Compression** - Condensed completed phases (1-4) into concise summaries, updated project scope to emphasize web GUI frontend with CLI as API client coming soon (Phase 6), reorganized phases for clarity |
|
|
|
|
---
|
|
|
|
**Last Updated:** 2025-11-17
|
|
**Next Review:** Before Phase 5 kickoff (Email & Comparisons)
|