sneakygeek/SneakyScan
2
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
Files
986c0d3d17d26f19aa55ae06ba69d5d7a35a089c
SneakyScan/ROADMAP.md
Phillip Tarrant 986c0d3d17 Complete Phase 1: Foundation - Flask web application infrastructure 
Implement complete database schema and Flask application structure for
SneakyScan web interface. This establishes the foundation for web-based
scan management, scheduling, and visualization.

Database & ORM:
- Add 11 SQLAlchemy models for comprehensive scan data storage
  (Scan, ScanSite, ScanIP, ScanPort, ScanService, ScanCertificate,
  ScanTLSVersion, Schedule, Alert, AlertRule, Setting)
- Configure Alembic migrations system with initial schema migration
- Add init_db.py script for database initialization and password setup
- Support both migration-based and direct table creation

Settings System:
- Implement SettingsManager with automatic encryption for sensitive values
- Add Fernet encryption for SMTP passwords and API tokens
- Implement PasswordManager with bcrypt password hashing (work factor 12)
- Initialize default settings for SMTP, authentication, and retention

Flask Application:
- Create Flask app factory pattern with scoped session management
- Add 4 API blueprints: scans, schedules, alerts, settings
- Implement functional Settings API (GET/PUT/DELETE endpoints)
- Add CORS support, error handlers, and request/response logging
- Configure development and production logging to file and console

Docker & Deployment:
- Update Dockerfile to install Flask dependencies
- Add docker-compose-web.yml for web application deployment
- Configure volume mounts for database, output, and logs persistence
- Expose port 5000 for Flask web server

Testing & Validation:
- Add validate_phase1.py script to verify all deliverables
- Validate directory structure, Python syntax, models, and endpoints
- All validation checks passing

Documentation:
- Add PHASE1_COMPLETE.md with comprehensive Phase 1 summary
- Update ROADMAP.md with Phase 1 completion status
- Update .gitignore to exclude database files and documentation

Files changed: 21 files
- New: web/ directory with complete Flask app structure
- New: migrations/ with Alembic configuration
- New: requirements-web.txt with Flask dependencies
- Modified: Dockerfile, ROADMAP.md, .gitignore
2025-11-13 23:59:23 -06:00

900 lines
34 KiB
Markdown
Raw Blame History

# SneakyScanner Roadmap
**Status:** Phase 1 Complete ✅ | Phase 2 Ready to Start
## Progress Overview
- ✅ **Phase 1: Foundation** - Complete (2025-11-13)
- Database schema & SQLAlchemy models
- Settings system with encryption
- Flask app structure with API blueprints
- Docker deployment support
- ⏳ **Phase 2: Flask Web App Core** - Next up (Weeks 3-4)
- 📋 **Phase 3: Dashboard & Scheduling** - Planned (Weeks 5-6)
- 📋 **Phase 4: Email & Comparisons** - Planned (Weeks 7-8)
- 📋 **Phase 5: CLI as API Client** - Planned (Week 9)
- 📋 **Phase 6: Advanced Features** - Planned (Weeks 10+)
## Vision & Goals
SneakyScanner is evolving from a CLI-based network scanning tool into a comprehensive **Flask web application** for infrastructure monitoring and security auditing. The web application will provide:
- **Centralized dashboard** for viewing scan history and trends
- **Scheduled scanning** for continuous infrastructure monitoring
- **Email notifications** for critical changes and certificate expirations
- **Historical analysis** with charts, graphs, and comparison reports
- **RESTful API** for integration and automation
- **Simple deployment** using SQLite3 (single-user, self-hosted)
The CLI scanner will evolve into an **API client**, maintaining backward compatibility while enabling web-based management and visualization.
## Target Users
- **Infrastructure teams** monitoring on-premises networks
- **Security professionals** performing periodic security audits
- **DevOps engineers** tracking infrastructure drift
- **Single users or small teams** (not enterprise multi-tenant)
## Technology Stack
### Backend
- **Flask** 3.x - Lightweight Python web framework
- **SQLAlchemy** 2.x - ORM for database abstraction
- **SQLite3** - Embedded database (easy deployment, sufficient for single-user)
- **APScheduler** 3.x - Background job scheduling for periodic scans
- **Flask-Login** - Simple session-based authentication
- **Flask-CORS** - API access control (optional for CLI API client)
- **Marshmallow** - API serialization/deserialization
### Frontend
- **Jinja2** - Server-side templating (already in use)
- **Bootstrap 5** - Responsive UI framework with dark theme support
- **Chart.js** 4.x - Lightweight charting library for trends
- **DataTables.js** - Interactive sortable/filterable tables (Phase 6)
- **Vanilla JavaScript** - Keep dependencies minimal
### Infrastructure
- **Docker Compose** - Multi-container orchestration (Flask app + Scanner)
- **Gunicorn** - WSGI server for production
- **Nginx** - Reverse proxy (optional, for production deployments)
### Existing Components (Keep)
- **Masscan** - Fast port discovery
- **Nmap** - Service detection
- **Playwright** - Screenshot capture
- **sslyze** - SSL/TLS analysis
## Architecture Overview
```
┌─────────────────────────────────────────────────────────────┐
│ Flask Web Application │
│ ┌──────────────┐ ┌──────────────┐ ┌──────────────────┐ │
│ │ Web UI │ │ REST API │ │ Scheduler │ │
│ │ (Dashboard) │ │ (JSON/CRUD) │ │ (APScheduler) │ │
│ └──────┬───────┘ └──────┬───────┘ └────────┬─────────┘ │
│ │ │ │ │
│ └─────────────────┴────────────────────┘ │
│ │ │
│ ┌────────▼────────┐ │
│ │ SQLAlchemy │ │
│ │ (ORM Layer) │ │
│ └────────┬────────┘ │
│ │ │
│ ┌────────▼────────┐ │
│ │ SQLite3 DB │ │
│ │ (scan history) │ │
│ └─────────────────┘ │
└───────────────────────────┬─────────────────────────────────┘
│
┌──────────▼──────────┐
│ Scanner Engine │
│ (scanner.py) │
│ ┌────────────────┐ │
│ │ Masscan/Nmap │ │
│ │ Playwright │ │
│ │ sslyze │ │
│ └────────────────┘ │
└─────────────────────┘
│
┌──────────▼──────────┐
│ CLI API Client │
│ (optional future) │
└─────────────────────┘
```
### Component Interaction
1. **Web UI** - User interacts with dashboard, triggers scans, views history
2. **REST API** - Handles requests from web UI and CLI client
3. **Scheduler (APScheduler)** - Triggers scans based on cron-like schedules
4. **SQLAlchemy ORM** - Abstracts database operations
5. **SQLite3 Database** - Stores scan results, schedules, settings, alerts
6. **Scanner Engine** - Performs actual network scanning (masscan, nmap, etc.)
7. **CLI API Client** - Future: thin client that calls Flask API
## Database Schema Design
### Core Tables
#### `scans`
Stores metadata about each scan execution.
| Column | Type | Description |
|--------|------|-------------|
| `id` | INTEGER PRIMARY KEY | Unique scan ID |
| `timestamp` | DATETIME | Scan start time (UTC) |
| `duration` | FLOAT | Total scan duration (seconds) |
| `status` | VARCHAR(20) | `running`, `completed`, `failed` |
| `config_file` | TEXT | Path to YAML config used |
| `title` | TEXT | Scan title from config |
| `json_path` | TEXT | Path to JSON report |
| `html_path` | TEXT | Path to HTML report |
| `zip_path` | TEXT | Path to ZIP archive |
| `screenshot_dir` | TEXT | Path to screenshot directory |
| `created_at` | DATETIME | Record creation time |
| `triggered_by` | VARCHAR(50) | `manual`, `scheduled`, `api` |
| `schedule_id` | INTEGER | FK to schedules (if triggered by schedule) |
#### `scan_sites`
Logical grouping of IPs by site.
| Column | Type | Description |
|--------|------|-------------|
| `id` | INTEGER PRIMARY KEY | Unique site record ID |
| `scan_id` | INTEGER | FK to scans |
| `site_name` | VARCHAR(255) | Site name from config |
#### `scan_ips`
IP addresses scanned in each scan.
| Column | Type | Description |
|--------|------|-------------|
| `id` | INTEGER PRIMARY KEY | Unique IP record ID |
| `scan_id` | INTEGER | FK to scans |
| `site_id` | INTEGER | FK to scan_sites |
| `ip_address` | VARCHAR(45) | IPv4 or IPv6 address |
| `ping_expected` | BOOLEAN | Expected ping response |
| `ping_actual` | BOOLEAN | Actual ping response |
#### `scan_ports`
Discovered TCP/UDP ports.
| Column | Type | Description |
|--------|------|-------------|
| `id` | INTEGER PRIMARY KEY | Unique port record ID |
| `scan_id` | INTEGER | FK to scans |
| `ip_id` | INTEGER | FK to scan_ips |
| `port` | INTEGER | Port number (1-65535) |
| `protocol` | VARCHAR(10) | `tcp` or `udp` |
| `expected` | BOOLEAN | Was this port expected? |
| `state` | VARCHAR(20) | `open`, `closed`, `filtered` |
#### `scan_services`
Detected services on open ports.
| Column | Type | Description |
|--------|------|-------------|
| `id` | INTEGER PRIMARY KEY | Unique service record ID |
| `scan_id` | INTEGER | FK to scans |
| `port_id` | INTEGER | FK to scan_ports |
| `service_name` | VARCHAR(100) | Service name (e.g., `ssh`, `http`) |
| `product` | VARCHAR(255) | Product name (e.g., `OpenSSH`) |
| `version` | VARCHAR(100) | Version string |
| `extrainfo` | TEXT | Additional nmap info |
| `ostype` | VARCHAR(100) | OS type if detected |
| `http_protocol` | VARCHAR(10) | `http` or `https` (if web service) |
| `screenshot_path` | TEXT | Relative path to screenshot |
#### `scan_certificates`
SSL/TLS certificates discovered on HTTPS services.
| Column | Type | Description |
|--------|------|-------------|
| `id` | INTEGER PRIMARY KEY | Unique certificate record ID |
| `scan_id` | INTEGER | FK to scans |
| `service_id` | INTEGER | FK to scan_services |
| `subject` | TEXT | Certificate subject (CN) |
| `issuer` | TEXT | Certificate issuer |
| `serial_number` | TEXT | Serial number |
| `not_valid_before` | DATETIME | Validity start date |
| `not_valid_after` | DATETIME | Validity end date |
| `days_until_expiry` | INTEGER | Days until expiration |
| `sans` | TEXT | JSON array of SANs |
| `is_self_signed` | BOOLEAN | Self-signed certificate flag |
#### `scan_tls_versions`
TLS version support and cipher suites.
| Column | Type | Description |
|--------|------|-------------|
| `id` | INTEGER PRIMARY KEY | Unique TLS version record ID |
| `scan_id` | INTEGER | FK to scans |
| `certificate_id` | INTEGER | FK to scan_certificates |
| `tls_version` | VARCHAR(20) | `TLS 1.0`, `TLS 1.1`, `TLS 1.2`, `TLS 1.3` |
| `supported` | BOOLEAN | Is this version supported? |
| `cipher_suites` | TEXT | JSON array of cipher suites |
### Scheduling & Notifications Tables
#### `schedules`
Scheduled scan configurations.
| Column | Type | Description |
|--------|------|-------------|
| `id` | INTEGER PRIMARY KEY | Unique schedule ID |
| `name` | VARCHAR(255) | Schedule name (e.g., "Daily prod scan") |
| `config_file` | TEXT | Path to YAML config |
| `cron_expression` | VARCHAR(100) | Cron-like schedule (e.g., `0 2 * * *`) |
| `enabled` | BOOLEAN | Is schedule active? |
| `last_run` | DATETIME | Last execution time |
| `next_run` | DATETIME | Next scheduled execution |
| `created_at` | DATETIME | Schedule creation time |
| `updated_at` | DATETIME | Last modification time |
#### `alerts`
Alert history and notifications sent.
| Column | Type | Description |
|--------|------|-------------|
| `id` | INTEGER PRIMARY KEY | Unique alert ID |
| `scan_id` | INTEGER | FK to scans |
| `alert_type` | VARCHAR(50) | `new_port`, `cert_expiry`, `service_change`, `ping_failed` |
| `severity` | VARCHAR(20) | `info`, `warning`, `critical` |
| `message` | TEXT | Human-readable alert message |
| `ip_address` | VARCHAR(45) | Related IP (optional) |
| `port` | INTEGER | Related port (optional) |
| `email_sent` | BOOLEAN | Was email notification sent? |
| `email_sent_at` | DATETIME | Email send timestamp |
| `created_at` | DATETIME | Alert creation time |
#### `alert_rules`
User-defined alert rules.
| Column | Type | Description |
|--------|------|-------------|
| `id` | INTEGER PRIMARY KEY | Unique rule ID |
| `rule_type` | VARCHAR(50) | `unexpected_port`, `cert_expiry`, `service_down`, etc. |
| `enabled` | BOOLEAN | Is rule active? |
| `threshold` | INTEGER | Threshold value (e.g., days for cert expiry) |
| `email_enabled` | BOOLEAN | Send email for this rule? |
| `created_at` | DATETIME | Rule creation time |
### Settings Table
#### `settings`
Application configuration key-value store.
| Column | Type | Description |
|--------|------|-------------|
| `id` | INTEGER PRIMARY KEY | Unique setting ID |
| `key` | VARCHAR(255) UNIQUE | Setting key (e.g., `smtp_server`) |
| `value` | TEXT | Setting value (JSON for complex values) |
| `updated_at` | DATETIME | Last modification time |
**Example Settings:**
- `smtp_server` - SMTP server hostname
- `smtp_port` - SMTP port (587, 465, 25)
- `smtp_username` - SMTP username
- `smtp_password` - SMTP password (encrypted)
- `smtp_from_email` - From email address
- `smtp_to_emails` - JSON array of recipient emails
- `app_password` - Single-user password hash (bcrypt)
- `retention_days` - How long to keep old scans (0 = forever)
## API Design
### REST API Endpoints
All API endpoints return JSON and follow RESTful conventions.
#### Scans
| Method | Endpoint | Description | Request Body | Response |
|--------|----------|-------------|--------------|----------|
| `GET` | `/api/scans` | List all scans (paginated) | - | `{ "scans": [...], "total": N, "page": 1 }` |
| `GET` | `/api/scans/{id}` | Get scan details | - | `{ "scan": {...} }` |
| `POST` | `/api/scans` | Trigger new scan | `{ "config_file": "path" }` | `{ "scan_id": N, "status": "running" }` |
| `DELETE` | `/api/scans/{id}` | Delete scan and files | - | `{ "status": "deleted" }` |
| `GET` | `/api/scans/{id}/status` | Get scan status | - | `{ "status": "running", "progress": "45%" }` |
| `GET` | `/api/scans/{id1}/compare/{id2}` | Compare two scans | - | `{ "diff": {...} }` |
#### Schedules
| Method | Endpoint | Description | Request Body | Response |
|--------|----------|-------------|--------------|----------|
| `GET` | `/api/schedules` | List all schedules | - | `{ "schedules": [...] }` |
| `GET` | `/api/schedules/{id}` | Get schedule details | - | `{ "schedule": {...} }` |
| `POST` | `/api/schedules` | Create new schedule | `{ "name": "...", "config_file": "...", "cron_expression": "..." }` | `{ "schedule_id": N }` |
| `PUT` | `/api/schedules/{id}` | Update schedule | `{ "enabled": true, "cron_expression": "..." }` | `{ "status": "updated" }` |
| `DELETE` | `/api/schedules/{id}` | Delete schedule | - | `{ "status": "deleted" }` |
| `POST` | `/api/schedules/{id}/trigger` | Manually trigger scheduled scan | - | `{ "scan_id": N }` |
#### Alerts
| Method | Endpoint | Description | Request Body | Response |
|--------|----------|-------------|--------------|----------|
| `GET` | `/api/alerts` | List recent alerts | - | `{ "alerts": [...] }` |
| `GET` | `/api/alerts/rules` | List alert rules | - | `{ "rules": [...] }` |
| `POST` | `/api/alerts/rules` | Create alert rule | `{ "rule_type": "...", "threshold": N }` | `{ "rule_id": N }` |
| `PUT` | `/api/alerts/rules/{id}` | Update alert rule | `{ "enabled": false }` | `{ "status": "updated" }` |
| `DELETE` | `/api/alerts/rules/{id}` | Delete alert rule | - | `{ "status": "deleted" }` |
#### Settings
| Method | Endpoint | Description | Request Body | Response |
|--------|----------|-------------|--------------|----------|
| `GET` | `/api/settings` | Get all settings (sanitized) | - | `{ "settings": {...} }` |
| `PUT` | `/api/settings` | Update settings | `{ "smtp_server": "...", ... }` | `{ "status": "updated" }` |
| `POST` | `/api/settings/test-email` | Test email configuration | - | `{ "status": "sent" }` |
#### Statistics & Trends
| Method | Endpoint | Description | Request Body | Response |
|--------|----------|-------------|--------------|----------|
| `GET` | `/api/stats/summary` | Dashboard summary stats | - | `{ "total_scans": N, "last_scan": "...", ... }` |
| `GET` | `/api/stats/trends` | Trend data for charts | `?days=30&metric=port_count` | `{ "data": [...] }` |
| `GET` | `/api/stats/certificates` | Certificate expiry overview | - | `{ "expiring_soon": [...], "expired": [...] }` |
### Authentication
**Phase 2-3:** Simple session-based authentication (single-user)
- Login endpoint: `POST /api/auth/login` (username/password)
- Logout endpoint: `POST /api/auth/logout`
- Session cookies with Flask-Login
- Password stored as bcrypt hash in settings table
**Phase 5:** API token authentication for CLI client
- Generate API token: `POST /api/auth/token`
- Revoke token: `DELETE /api/auth/token`
- CLI sends token in `Authorization: Bearer <token>` header
## Phased Roadmap
### Phase 1: Foundation ✅ COMPLETE
**Completed:** 2025-11-13
**Priority: CRITICAL** - Database and settings infrastructure
**Goals:**
- ✅ Establish database schema
- ✅ Create settings system
- ✅ Set up Flask project structure
**Tasks:**
1. ✅ Create SQLite database schema (use Alembic for migrations)
2. ✅ Implement SQLAlchemy models for all tables (11 models)
3. ✅ Create database initialization script (`init_db.py`)
4. ✅ Implement settings system:
- ✅ Settings model with get/set methods
- ✅ Default settings initialization
- ✅ Encrypted storage for passwords (cryptography library + bcrypt)
- ✅ PasswordManager for bcrypt password hashing
5. ✅ Set up Flask project structure:
```
SneakyScanner/
├── src/
│ ├── scanner.py (existing)
│ ├── screenshot_capture.py (existing)
│ └── report_generator.py (existing)
├── web/ ✅ CREATED
│ ├── __init__.py ✅
│ ├── app.py (Flask app factory) ✅
│ ├── models.py (SQLAlchemy models) ✅
│ ├── api/ (API blueprints) ✅
│ │ ├── __init__.py ✅
│ │ ├── scans.py ✅
│ │ ├── schedules.py ✅
│ │ ├── alerts.py ✅
│ │ └── settings.py ✅ (Fully functional!)
│ ├── templates/ (Jinja2 templates) ✅
│ ├── static/ (CSS, JS, images) ✅
│ │ ├── css/ ✅
│ │ ├── js/ ✅
│ │ └── images/ ✅
│ └── utils/ (helpers, decorators) ✅
│ ├── __init__.py ✅
│ └── settings.py ✅
├── migrations/ (Alembic migrations) ✅
│ ├── env.py ✅
│ ├── script.py.mako ✅
│ └── versions/ ✅
│ └── 001_initial_schema.py ✅
├── alembic.ini ✅
├── configs/ (existing)
├── output/ (existing)
└── templates/ (existing - for reports)
```
6. ✅ Create `requirements-web.txt` for Flask dependencies
7. ✅ Update Dockerfile to support Flask app
8. ✅ Create `docker-compose-web.yml` for web deployment
9. ✅ Create `validate_phase1.py` for verification
**Deliverables:**
- ✅ Working database with schema (SQLite3 + Alembic migrations)
- ✅ Settings CRUD functionality (with encryption for sensitive values)
- ✅ Flask app skeleton with functional Settings API
- ✅ Database migration system (Alembic)
- ✅ API blueprint stubs (scans, schedules, alerts, settings)
- ✅ Docker support (Dockerfile updated, docker-compose-web.yml created)
**Testing:**
- ✅ Database creates successfully (`init_db.py` works)
- ✅ Settings can be stored/retrieved (encryption working)
- ✅ Flask app starts without errors (`python3 -m web.app` works)
- ✅ All validation checks pass (`validate_phase1.py` ✓)
- ✅ All 11 database models defined correctly
- ✅ Settings API endpoints functional and tested
**Documentation:**
- ✅ `PHASE1_COMPLETE.md` - Complete Phase 1 summary with API reference and deployment guide
- ✅ `validate_phase1.py` - Automated validation script
---
### Phase 2: Flask Web App Core (Weeks 3-4)
**Priority: HIGH** - Basic web application with API
**Goals:**
- Implement REST API for scans
- Add background job queue
- Create simple authentication
- Integrate scanner with database
**Tasks:**
1. Implement scan API endpoints:
- `POST /api/scans` - trigger scan, save to DB
- `GET /api/scans` - list scans with pagination
- `GET /api/scans/{id}` - get scan details from DB
- `DELETE /api/scans/{id}` - delete scan
2. Integrate scanner with database:
- Modify `scanner.py` to save results to DB after scan
- Create `ScanService` class to handle scan → DB logic
- Maintain JSON/HTML/ZIP file generation
3. Set up background job queue:
- Install APScheduler
- Create job executor for scans
- Implement scan status tracking (`running`, `completed`, `failed`)
4. Implement authentication:
- Flask-Login for session management
- Login page (`/login`)
- Password verification against settings table
- Protect all routes with `@login_required` decorator
5. Create basic templates:
- `base.html` - Base layout with Bootstrap 5 dark theme
- `login.html` - Login page
- `dashboard.html` - Placeholder dashboard
6. Error handling and logging:
- API error responses (JSON format)
- Logging configuration (file + console)
7. Docker Compose setup:
- Flask container (Gunicorn)
- Volume mounts for DB, configs, output
- Port mapping (5000 for Flask)
**Deliverables:**
- Working REST API for scans
- Background scan execution
- Simple login system
- Scanner integrated with database
- Docker Compose deployment
**Testing:**
- API can trigger scan and return scan_id
- Scan results saved to database
- Pagination works for scan list
- Authentication protects routes
- Docker Compose brings up Flask app
---
### Phase 3: Dashboard & Scheduling (Weeks 5-6)
**Priority: HIGH** - User's top requested features
**Goals:**
- Build web dashboard with scan history
- Implement trend charts
- Add scheduled scan management UI
- Real-time scan progress
**Tasks:**
1. Dashboard implementation:
- Summary cards (total scans, last scan, total IPs, open ports)
- Recent scans table (clickable to view details)
- Security warnings section (expiring certs, weak TLS)
- Drift alerts section (unexpected ports, new services)
2. Scan detail page:
- Display full scan results (sites, IPs, services)
- Embedded screenshots or links
- Download buttons (JSON, HTML, ZIP)
- Delete scan button
3. Trend charts with Chart.js:
- Port count over time (line chart)
- Service distribution (bar chart)
- Certificate expiration timeline (timeline chart)
- Charts update based on date range selector
4. Scheduled scans UI:
- List all schedules (table with enable/disable toggle)
- Create schedule form (name, config file, cron expression)
- Edit schedule form
- Delete schedule button
- Next run time display
5. APScheduler integration:
- Schedule manager class
- Load schedules from DB on app start
- Add/remove jobs dynamically
- Cron expression validation
- Update `next_run` and `last_run` in DB
6. Manual scan trigger:
- "Run Scan Now" button on dashboard
- Config file selector
- Show real-time progress (polling or WebSocket)
7. Navigation menu:
- Dashboard
- Scans
- Schedules
- Alerts (placeholder)
- Settings (placeholder)
**Deliverables:**
- Functional dashboard with charts
- Scan history browser
- Scheduled scan management UI
- Background scheduler running scans
**Testing:**
- Dashboard displays accurate summary stats
- Charts render correctly with real data
- Scheduled scans execute at specified times
- Manual scan trigger works
- Real-time progress updates
---
### Phase 4: Email & Comparisons (Weeks 7-8)
**Priority: MEDIUM** - Monitoring and analysis features
**Goals:**
- Implement email notification system
- Create scan comparison reports
- Add alert rule configuration
**Tasks:**
1. Email notification system:
- SMTP integration (using `smtplib` or `Flask-Mail`)
- Email template for alerts (Jinja2 HTML email)
- Settings page for SMTP configuration
- Test email button
2. Alert rule engine:
- Define alert rule types:
- Unexpected TCP/UDP port opened
- Expected port missing
- Certificate expiring in < N days
- Certificate expired
- Service version changed
- Ping failed (host down)
- Weak TLS version detected (1.0/1.1)
- Alert rule creation UI
- Alert rule evaluation after each scan
- Store alerts in `alerts` table
3. Alert history page:
- List all alerts (filterable by type, severity, date)
- Mark alerts as "acknowledged"
- Alert detail view
4. Scan comparison:
- Compare two scans API endpoint
- Comparison algorithm:
- New ports/services
- Removed ports/services
- Service version changes
- Certificate changes
- TLS configuration changes
- Comparison report UI:
- Side-by-side view
- Diff highlighting (green = added, red = removed, yellow = changed)
- "Compare" button on scan list (select 2 scans)
5. Email notification triggers:
- Send email when alert rule triggered
- Daily digest email (summary of all alerts)
- Weekly scan summary email
6. Settings page:
- SMTP configuration form
- Alert rule management
- Email recipient list
- Test email button
**Deliverables:**
- Working email notification system
- Alert rules with email triggers
- Scan comparison functionality
- Settings UI for configuration
**Testing:**
- Email sends successfully with SMTP config
- Alert rules trigger correctly
- Comparison shows accurate diffs
- Settings persist correctly
---
### Phase 5: CLI as API Client (Week 9)
**Priority: MEDIUM** - Backward compatibility and automation
**Goals:**
- Refactor CLI to optionally call Flask API
- Maintain standalone mode for testing
- API token authentication
**Tasks:**
1. API client mode for `scanner.py`:
- Add `--api-mode` flag
- Add `--api-url` and `--api-token` arguments
- When `--api-mode` enabled:
- Send scan request to `POST /api/scans`
- Poll `GET /api/scans/{id}/status` for progress
- Download results when complete
- When `--api-mode` disabled (default):
- Run standalone as currently works
2. API token generation:
- UI to generate API tokens (settings page)
- Store tokens in `api_tokens` table (hashed)
- API token authentication middleware
- Token expiration and revocation
3. CLI documentation:
- Update README.md with API mode usage
- Example commands for API mode
- Token generation instructions
4. Benefits of API mode:
- Scans stored centrally in database
- No need to mount volumes for output
- Scheduled scans managed through web UI
- Scan history accessible via web dashboard
**Deliverables:**
- CLI with `--api-mode` flag
- API token system
- Updated documentation
**Testing:**
- CLI can trigger scan via API
- API token authentication works
- Standalone mode still functional
- Token revocation works
---
### Phase 6: Advanced Features (Weeks 10+)
**Priority: LOW** - Nice-to-have enhancements
**Goals:**
- Enhanced interactive reports
- Vulnerability detection
- PDF export
- Timeline view
**Tasks:**
1. Enhanced HTML reports:
- Sortable/filterable tables (DataTables.js)
- Inline screenshot thumbnails (lightbox on click)
- Export to PDF button (WeasyPrint or pdfkit)
- Print-friendly CSS
2. Vulnerability detection:
- Integrate with CVE databases (NVD API or Vulners API)
- Match detected services/versions to known CVEs
- Display CVE list with severity scores (CVSS)
- CVE detail page with description, remediation
- Alert rule for new critical CVEs
3. Timeline view:
- Visual timeline of all scans
- Filter by site or IP
- Click on timeline event to view scan
- Annotations for important events (cert renewals, config changes)
4. Advanced charts:
- Heatmap of port activity
- Service version tracking over time
- Certificate expiration forecast
- Top 10 services pie chart
5. Export/Import:
- Export scan data to CSV
- Import scan configs from CSV
- Bulk schedule creation
6. Additional integrations:
- Slack notifications (in addition to email)
- Webhook support (POST to custom URL on events)
- Prometheus metrics export
**Deliverables:**
- Interactive sortable tables
- CVE integration
- PDF export
- Timeline view
- Additional integrations
**Testing:**
- DataTables work with large datasets
- CVE data fetches correctly
- PDF exports render properly
- Timeline view performs well with many scans
---
## Migration Strategy
### From Current CLI to Web App
**Current State:**
- CLI tool (`scanner.py`) runs standalone
- Outputs JSON, HTML, ZIP files
- No database, no web UI
**Migration Path:**
#### Step 1: Add Database Layer (Phase 1)
- Database runs alongside CLI
- CLI can optionally save to DB (flag: `--save-to-db`)
- No breaking changes
#### Step 2: Launch Web App (Phase 2-3)
- Web app reads from DB
- Users can trigger scans via web UI
- CLI still works standalone
#### Step 3: Transition Period (Phase 4-5)
- Users gradually adopt web UI
- CLI used for scripting/automation
- Both modes fully supported
#### Step 4: API Client Mode (Phase 5)
- CLI becomes thin API client
- All scans stored in central DB
- Standalone mode remains for testing
#### Step 5: Full Web App (Phase 6+)
- Primary interface is web UI
- CLI optional for power users
### Backward Compatibility
**Maintained:**
- Existing YAML config format
- JSON/HTML/ZIP output files
- Screenshot capture
- Docker deployment
**Deprecated (eventually):**
- Standalone CLI mode (Phase 6+)
- Direct file output (replaced by DB + API)
## Prioritized Feature List
### Must-Have (Phases 1-3)
1. **Database foundation** (SQLite3 + SQLAlchemy)
2. **Flask web app core** (REST API, authentication)
3. **Dashboard with scan history** (list, detail, delete)
4. **Trend charts** (Chart.js - port counts, service distribution)
5. **Scheduled scans** (APScheduler + cron expressions)
6. **Manual scan trigger** (web UI button)
### Should-Have (Phase 4)
7. **Email notifications** (SMTP integration)
8. **Alert rules** (cert expiry, unexpected ports, etc.)
9. **Scan comparison reports** (diff view)
10. **Settings UI** (SMTP, alerts, retention)
### Nice-to-Have (Phases 5-6)
11. **CLI as API client** (token auth, backward compat)
12. **Sortable/filterable tables** (DataTables.js)
13. **PDF export** (WeasyPrint)
14. **Vulnerability detection** (CVE integration)
15. **Timeline view** (visual scan history)
16. **Embedded screenshot thumbnails** (lightbox)
### Future/Deferred
17. **Multi-user support** (if requirements change)
18. **Slack/webhook integrations**
19. **Prometheus metrics**
20. **Mobile-responsive dashboard** (Bootstrap handles basics)
## Development Workflow
### Iteration Cycle
1. **Plan** - Define features for phase
2. **Implement** - Code backend + frontend
3. **Test** - Unit tests + manual testing
4. **Deploy** - Update Docker Compose
5. **Document** - Update README.md, ROADMAP.md
6. **Review** - Get user feedback
7. **Iterate** - Adjust priorities based on feedback
### Git Workflow
- **main branch** - Stable releases
- **develop branch** - Active development
- **feature branches** - Individual features (`feature/dashboard`, `feature/scheduler`)
- **Pull requests** - Review before merge
### Testing Strategy
- **Unit tests** - pytest for models, API endpoints
- **Integration tests** - Full scan → DB → API workflow
- **Manual testing** - UI/UX testing in browser
- **Performance tests** - Large scans, database queries
### Documentation
- **README.md** - User-facing documentation (updated each phase)
- **ROADMAP.md** - This file (updated as priorities shift)
- **CLAUDE.md** - Developer documentation (architecture, code references)
- **API.md** - API documentation (OpenAPI/Swagger in Phase 4)
## Success Metrics
### Phase 1 Success ✅ ACHIEVED
- [x] Database creates successfully with all 11 tables
- [x] Settings can be stored/retrieved with encryption
- [x] Flask app starts without errors
- [x] API blueprints load correctly
- [x] All Python modules have valid syntax
- [x] Docker deployment configured
### Phase 2-3 Success (In Progress)
- [ ] Database stores scan results correctly
- [ ] Dashboard displays scans and trends
- [ ] Scheduled scans execute automatically
- [ ] Web UI is intuitive and responsive
### Phase 4 Success
- [ ] Email notifications sent for critical alerts
- [ ] Comparison reports show meaningful diffs
- [ ] Settings UI allows configuration without editing files
### Phase 5 Success
- [ ] CLI can trigger scans via API
- [ ] API tokens work for authentication
- [ ] Standalone CLI mode still functional
### Phase 6+ Success
- [ ] CVE integration provides actionable vulnerability data
- [ ] Timeline view helps track infrastructure changes
- [ ] PDF exports are shareable and professional
## Open Questions
### Technical Decisions
- **Flask vs. FastAPI?** - Sticking with Flask for simplicity, but FastAPI offers async and auto-docs
- **APScheduler vs. Celery?** - APScheduler for simplicity (no Redis/RabbitMQ needed), but Celery scales better
- **Bootstrap vs. Tailwind?** - Bootstrap for speed (pre-built components), Tailwind for customization
- **Chart.js vs. Plotly?** - Chart.js for lightweight, Plotly for interactive (decide in Phase 3)
### Product Questions
- **Should we support multiple configs per schedule?** - Start with 1:1, add later if needed
- **How many scans to keep in DB?** - Add retention setting (default: keep all)
- **Support for multi-tenancy?** - Not in scope (single-user), but database schema allows future expansion
- **Mobile app?** - Out of scope, but responsive web UI covers basics
## Resources & References
### Documentation
- [Flask Documentation](https://flask.palletsprojects.com/)
- [SQLAlchemy ORM](https://docs.sqlalchemy.org/)
- [APScheduler](https://apscheduler.readthedocs.io/)
- [Chart.js](https://www.chartjs.org/docs/)
- [Bootstrap 5](https://getbootstrap.com/docs/5.3/)
### Tutorials
- [Flask Mega-Tutorial](https://blog.miguelgrinberg.com/post/the-flask-mega-tutorial-part-i-hello-world)
- [SQLAlchemy Tutorial](https://docs.sqlalchemy.org/en/20/tutorial/)
- [APScheduler with Flask](https://github.com/viniciuschiele/flask-apscheduler)
### Similar Projects (Inspiration)
- [OpenVAS](https://www.openvas.org/) - Vulnerability scanner with web UI
- [Nessus](https://www.tenable.com/products/nessus) - Commercial scanner (inspiration for UI/UX)
- [OWASP ZAP](https://www.zaproxy.org/) - Web app scanner (comparison reports, alerts)
## Changelog
| Date | Version | Changes |
|------|---------|---------|
| 2025-11-14 | 1.0 | Initial roadmap created based on user requirements |
| 2025-11-13 | 1.1 | **Phase 1 COMPLETE** - Database schema, SQLAlchemy models, Flask app structure, settings system with encryption, Alembic migrations, API blueprints, Docker support, validation script |
---
**Last Updated:** 2025-11-13
**Next Review:** Before Phase 2 kickoff (REST API for scans implementation)
Reference in New Issue View Git Blame Copy Permalink