From 456e052389a823e5fe6016da3e876f8250d7ef9b Mon Sep 17 00:00:00 2001 From: Phillip Tarrant Date: Mon, 17 Nov 2025 15:50:15 -0600 Subject: [PATCH] updating docs --- docs/{ai => }/API_REFERENCE.md | 0 docs/{ai => }/DEPLOYMENT.md | 0 docs/{ai => }/ROADMAP.md | 570 +++------ docs/ai/MANUAL_TESTING.md | 876 ------------- docs/ai/PHASE1_COMPLETE.md | 404 ------ docs/ai/PHASE2.md | 1979 ---------------------------- docs/ai/PHASE2_COMPLETE.md | 872 ------------- docs/ai/PHASE3.md | 2204 -------------------------------- docs/ai/Phase4.md | 1811 -------------------------- 9 files changed, 166 insertions(+), 8550 deletions(-) rename docs/{ai => }/API_REFERENCE.md (100%) rename docs/{ai => }/DEPLOYMENT.md (100%) rename docs/{ai => }/ROADMAP.md (61%) delete mode 100644 docs/ai/MANUAL_TESTING.md delete mode 100644 docs/ai/PHASE1_COMPLETE.md delete mode 100644 docs/ai/PHASE2.md delete mode 100644 docs/ai/PHASE2_COMPLETE.md delete mode 100644 docs/ai/PHASE3.md delete mode 100644 docs/ai/Phase4.md diff --git a/docs/ai/API_REFERENCE.md b/docs/API_REFERENCE.md similarity index 100% rename from docs/ai/API_REFERENCE.md rename to docs/API_REFERENCE.md diff --git a/docs/ai/DEPLOYMENT.md b/docs/DEPLOYMENT.md similarity index 100% rename from docs/ai/DEPLOYMENT.md rename to docs/DEPLOYMENT.md diff --git a/docs/ai/ROADMAP.md b/docs/ROADMAP.md similarity index 61% rename from docs/ai/ROADMAP.md rename to docs/ROADMAP.md index 91638ca..17c0487 100644 --- a/docs/ai/ROADMAP.md +++ b/docs/ROADMAP.md @@ -4,33 +4,19 @@ ## Progress Overview - ✅ **Phase 1: Foundation** - Complete (2025-11-13) - - Database schema & SQLAlchemy models - - Settings system with encryption - - Flask app structure with API blueprints - - Docker deployment support + - Database schema, SQLAlchemy models, settings system, Flask app structure - ✅ **Phase 2: Flask Web App Core** - Complete (2025-11-14) - - REST API for scan management (5 endpoints) - - Background job queue with APScheduler - - Session-based authentication system - - Basic UI templates (dashboard, scans, login) - - Comprehensive error handling and logging - - 100 tests passing (1,825 lines of test code) + - REST API, background jobs, authentication, web UI, testing (100 tests) - ✅ **Phase 3: Dashboard & Scheduling** - Complete (2025-11-14) - - Dashboard with summary stats and recent scans - - Scan history browser with detail pages - - Scheduled scan management UI - - Background scheduler with APScheduler - - Trend charts with Chart.js + - Dashboard, scan history, scheduled scans, trend charts - ✅ **Phase 4: Config Creator** - Complete (2025-11-17) - - CIDR-based config creation (simplified workflow) - - YAML editor with CodeMirror (syntax highlighting) - - Config management UI (list, view, edit, download, delete) - - Direct YAML upload for advanced users - - Full REST API for config operations - - Schedule dependency protection (delete blocking) -- 📋 **Phase 5: Email & Comparisons** - Next up + - CIDR-based config creation, YAML editor, config management UI +- 📋 **Phase 5: Email & Comparisons** - Next Up + - Email notifications, alert rules, scan comparison - 📋 **Phase 6: CLI as API Client** - Planned -- 📋 **Phase 7: Advanced Features** - Planned + - CLI for scripting and automation via API +- 📋 **Phase 7: Advanced Features** - Future + - CVE integration, timeline view, PDF export, enhanced reports ## Recent Bug Fixes @@ -55,16 +41,19 @@ ## Vision & Goals -SneakyScanner is evolving from a CLI-based network scanning tool into a comprehensive **Flask web application** for infrastructure monitoring and security auditing. The web application will provide: +SneakyScanner is a comprehensive **Flask web application** for infrastructure monitoring and security auditing. The primary interface is the web GUI, with a CLI API client planned for scripting and automation needs. +**Core Features:** - **Centralized dashboard** for viewing scan history and trends - **Scheduled scanning** for continuous infrastructure monitoring -- **Email notifications** for critical changes and certificate expirations +- **Email notifications** for critical changes and certificate expirations (coming soon) - **Historical analysis** with charts, graphs, and comparison reports +- **Config creator** for easy CIDR-based scan configuration - **RESTful API** for integration and automation - **Simple deployment** using SQLite3 (single-user, self-hosted) -The CLI scanner will evolve into an **API client**, maintaining backward compatibility while enabling web-based management and visualization. +**Planned:** +- **CLI API client** for scripting and automation workflows (Phase 6) ## Target Users @@ -391,441 +380,213 @@ All API endpoints return JSON and follow RESTful conventions. ### Phase 1: Foundation ✅ COMPLETE **Completed:** 2025-11-13 -**Priority: CRITICAL** - Database and settings infrastructure - -**Goals:** -- ✅ Establish database schema -- ✅ Create settings system -- ✅ Set up Flask project structure - -**Tasks:** -1. ✅ Create SQLite database schema (use Alembic for migrations) -2. ✅ Implement SQLAlchemy models for all tables (11 models) -3. ✅ Create database initialization script (`init_db.py`) -4. ✅ Implement settings system: - - ✅ Settings model with get/set methods - - ✅ Default settings initialization - - ✅ Encrypted storage for passwords (cryptography library + bcrypt) - - ✅ PasswordManager for bcrypt password hashing -5. ✅ Set up Flask project structure: - ``` - SneakyScanner/ - ├── src/ - │ ├── scanner.py (existing) - │ ├── screenshot_capture.py (existing) - │ └── report_generator.py (existing) - ├── web/ ✅ CREATED - │ ├── __init__.py ✅ - │ ├── app.py (Flask app factory) ✅ - │ ├── models.py (SQLAlchemy models) ✅ - │ ├── api/ (API blueprints) ✅ - │ │ ├── __init__.py ✅ - │ │ ├── scans.py ✅ - │ │ ├── schedules.py ✅ - │ │ ├── alerts.py ✅ - │ │ └── settings.py ✅ (Fully functional!) - │ ├── templates/ (Jinja2 templates) ✅ - │ ├── static/ (CSS, JS, images) ✅ - │ │ ├── css/ ✅ - │ │ ├── js/ ✅ - │ │ └── images/ ✅ - │ └── utils/ (helpers, decorators) ✅ - │ ├── __init__.py ✅ - │ └── settings.py ✅ - ├── migrations/ (Alembic migrations) ✅ - │ ├── env.py ✅ - │ ├── script.py.mako ✅ - │ └── versions/ ✅ - │ └── 001_initial_schema.py ✅ - ├── alembic.ini ✅ - ├── configs/ (existing) - ├── output/ (existing) - └── templates/ (existing - for reports) - ``` -6. ✅ Create `requirements-web.txt` for Flask dependencies -7. ✅ Update Dockerfile to support Flask app -8. ✅ Create `docker-compose-web.yml` for web deployment -9. ✅ Create `validate_phase1.py` for verification **Deliverables:** -- ✅ Working database with schema (SQLite3 + Alembic migrations) -- ✅ Settings CRUD functionality (with encryption for sensitive values) -- ✅ Flask app skeleton with functional Settings API -- ✅ Database migration system (Alembic) -- ✅ API blueprint stubs (scans, schedules, alerts, settings) -- ✅ Docker support (Dockerfile updated, docker-compose-web.yml created) - -**Testing:** -- ✅ Database creates successfully (`init_db.py` works) -- ✅ Settings can be stored/retrieved (encryption working) -- ✅ Flask app starts without errors (`python3 -m web.app` works) -- ✅ All validation checks pass (`validate_phase1.py` ✓) -- ✅ All 11 database models defined correctly -- ✅ Settings API endpoints functional and tested - -**Documentation:** -- ✅ `PHASE1_COMPLETE.md` - Complete Phase 1 summary with API reference and deployment guide -- ✅ `validate_phase1.py` - Automated validation script +- SQLite database with 11 tables (scans, sites, IPs, ports, services, certificates, TLS versions, schedules, alerts, alert_rules, settings) +- SQLAlchemy ORM models with relationships +- Alembic migration system +- Settings system with encryption (bcrypt for passwords, Fernet for sensitive data) +- Flask app structure with API blueprints +- Docker Compose deployment configuration +- Validation script for verification --- ### Phase 2: Flask Web App Core ✅ COMPLETE **Completed:** 2025-11-14 -**Duration:** 14 days (Weeks 3-4) -**Priority:** HIGH - -**Goals:** -- ✅ Implement REST API for scans -- ✅ Add background job queue -- ✅ Create simple authentication -- ✅ Integrate scanner with database - -**Deliverables Completed:** -- ✅ **REST API** - 5 scan endpoints (trigger, list, get, status, delete) + 3 settings endpoints -- ✅ **Background Jobs** - APScheduler with ThreadPoolExecutor (up to 3 concurrent scans) -- ✅ **Authentication** - Flask-Login session-based auth (login, logout, decorators) -- ✅ **Database Integration** - Complete scan results saved to normalized schema -- ✅ **Web UI** - Dashboard, scans list/detail, login, error templates -- ✅ **Error Handling** - Content negotiation (JSON/HTML), custom error pages, request IDs -- ✅ **Logging** - Rotating file handlers (10MB max), request timing, structured logs -- ✅ **Docker Deployment** - Production-ready docker-compose with healthcheck -- ✅ **Testing** - 100 test functions, 1,825 lines of test code, all passing -- ✅ **Documentation** - API_REFERENCE.md, DEPLOYMENT.md, PHASE2_COMPLETE.md - -**Files Created:** 34 files, ~7,500+ lines of code - -**Key Features:** -- Scans execute in background without blocking HTTP requests -- Status tracking: `running` → `completed`/`failed` -- Pagination and filtering for scan lists -- Complete scan details with all relationships (sites, IPs, ports, services, certs, TLS) -- Secure password hashing with bcrypt -- SQLite WAL mode for better concurrency -- Request IDs for debugging and correlation -- Comprehensive error handling for all HTTP status codes - -**Testing Results:** -- ✅ All API endpoints tested (24 integration tests) -- ✅ Service layer tested (15 unit tests) -- ✅ Authentication tested (30+ tests) -- ✅ Background jobs tested (13 tests) -- ✅ Error handling tested (18+ tests) -- ✅ All 100 tests passing - -**Documentation:** -- [PHASE2_COMPLETE.md](PHASE2_COMPLETE.md) - Complete Phase 2 summary -- [API_REFERENCE.md](API_REFERENCE.md) - Comprehensive API documentation -- [DEPLOYMENT.md](DEPLOYMENT.md) - Production deployment guide -- README.md updated with Phase 2 features - ---- - -### Phase 3: Dashboard & Scheduling (Weeks 5-6) -**Priority: HIGH** - User's top requested features - -**Goals:** -- Build web dashboard with scan history -- Implement trend charts -- Add scheduled scan management UI -- Real-time scan progress - -**Tasks:** -1. Dashboard implementation: - - Summary cards (total scans, last scan, total IPs, open ports) - - Recent scans table (clickable to view details) - - Security warnings section (expiring certs, weak TLS) - - Drift alerts section (unexpected ports, new services) -2. Scan detail page: - - Display full scan results (sites, IPs, services) - - Embedded screenshots or links - - Download buttons (JSON, HTML, ZIP) - - Delete scan button -3. Trend charts with Chart.js: - - Port count over time (line chart) - - Service distribution (bar chart) - - Certificate expiration timeline (timeline chart) - - Charts update based on date range selector -4. Scheduled scans UI: - - List all schedules (table with enable/disable toggle) - - Create schedule form (name, config file, cron expression) - - Edit schedule form - - Delete schedule button - - Next run time display -5. APScheduler integration: - - Schedule manager class - - Load schedules from DB on app start - - Add/remove jobs dynamically - - Cron expression validation - - Update `next_run` and `last_run` in DB -6. Manual scan trigger: - - "Run Scan Now" button on dashboard - - Config file selector - - Show real-time progress (polling or WebSocket) -7. Navigation menu: - - Dashboard - - Scans - - Schedules - - Alerts (placeholder) - - Settings (placeholder) **Deliverables:** -- Functional dashboard with charts -- Scan history browser -- Scheduled scan management UI -- Background scheduler running scans - -**Testing:** -- Dashboard displays accurate summary stats -- Charts render correctly with real data -- Scheduled scans execute at specified times -- Manual scan trigger works -- Real-time progress updates +- REST API with 8 endpoints (scans: trigger, list, get, status, delete; settings: get, update, test-email) +- Background job queue using APScheduler (up to 3 concurrent scans) +- Session-based authentication with Flask-Login +- Database integration for scan results (full normalized schema population) +- Web UI templates (dashboard, scan list/detail, login, error pages) +- Error handling with content negotiation (JSON/HTML) and request IDs +- Logging system with rotating file handlers +- Production Docker Compose deployment +- Comprehensive test suite (100 tests, all passing) +- Documentation (API_REFERENCE.md, DEPLOYMENT.md) --- -### Phase 4: Email & Comparisons (Weeks 7-8) -**Priority: MEDIUM** - Monitoring and analysis features +### Phase 3: Dashboard & Scheduling ✅ COMPLETE +**Completed:** 2025-11-14 + +**Deliverables:** +- Dashboard with summary stats (total scans, IPs, ports, services) +- Recent scans table with clickable details +- Scan detail page with full results display +- Historical trend charts using Chart.js (port counts over time) +- Scheduled scan management UI (create, edit, delete, enable/disable) +- Schedule execution with APScheduler and cron expressions +- Manual scan trigger from web UI +- Navigation menu (Dashboard, Scans, Schedules, Configs, Settings) +- Download buttons for scan reports (JSON, HTML, ZIP) + +--- + +### Phase 4: Config Creator ✅ COMPLETE +**Completed:** 2025-11-17 + +**Deliverables:** +- CIDR-based config creation UI (simplified workflow for quick config generation) +- YAML editor with CodeMirror (syntax highlighting, line numbers) +- Config management UI (list, view, edit, download, delete) +- Direct YAML upload for advanced users +- REST API for config operations (7 endpoints: list, get, create, update, delete, upload, download) +- Schedule dependency protection (prevents deleting configs used by schedules) +- Comprehensive testing (25+ unit and integration tests) + +--- + +### Phase 5: Email & Comparisons +**Status:** Next Up +**Priority:** MEDIUM **Goals:** - Implement email notification system - Create scan comparison reports - Add alert rule configuration -**Tasks:** -1. Email notification system: - - SMTP integration (using `smtplib` or `Flask-Mail`) - - Email template for alerts (Jinja2 HTML email) - - Settings page for SMTP configuration - - Test email button -2. Alert rule engine: - - Define alert rule types: - - Unexpected TCP/UDP port opened - - Expected port missing - - Certificate expiring in < N days - - Certificate expired - - Service version changed - - Ping failed (host down) - - Weak TLS version detected (1.0/1.1) - - Alert rule creation UI - - Alert rule evaluation after each scan - - Store alerts in `alerts` table -3. Alert history page: - - List all alerts (filterable by type, severity, date) - - Mark alerts as "acknowledged" - - Alert detail view -4. Scan comparison: +**Planned Features:** +1. **Email Notifications:** + - SMTP integration with configurable settings + - Alert email templates (Jinja2 HTML) + - Test email functionality + - Email triggers for critical events + +2. **Alert Rule Engine:** + - Alert types: unexpected ports, cert expiry, service changes, host down, weak TLS + - Alert rule creation and management UI + - Automatic evaluation after each scan + - Alert history with severity filtering + +3. **Scan Comparison:** - Compare two scans API endpoint - - Comparison algorithm: - - New ports/services - - Removed ports/services - - Service version changes - - Certificate changes - - TLS configuration changes - - Comparison report UI: - - Side-by-side view - - Diff highlighting (green = added, red = removed, yellow = changed) - - "Compare" button on scan list (select 2 scans) -5. Email notification triggers: - - Send email when alert rule triggered - - Daily digest email (summary of all alerts) - - Weekly scan summary email -6. Settings page: - - SMTP configuration form - - Alert rule management - - Email recipient list - - Test email button - -**Deliverables:** -- Working email notification system -- Alert rules with email triggers -- Scan comparison functionality -- Settings UI for configuration - -**Testing:** -- Email sends successfully with SMTP config -- Alert rules trigger correctly -- Comparison shows accurate diffs -- Settings persist correctly + - Diff detection (new/removed ports, service changes, cert changes) + - Visual comparison UI with highlighting + - "Compare" button on scan list --- -### Phase 5: CLI as API Client (Week 9) -**Priority: MEDIUM** - Backward compatibility and automation +### Phase 6: CLI as API Client +**Status:** Planned +**Priority:** MEDIUM **Goals:** -- Refactor CLI to optionally call Flask API +- Create CLI API client for scripting and automation - Maintain standalone mode for testing - API token authentication -**Tasks:** -1. API client mode for `scanner.py`: - - Add `--api-mode` flag - - Add `--api-url` and `--api-token` arguments - - When `--api-mode` enabled: - - Send scan request to `POST /api/scans` - - Poll `GET /api/scans/{id}/status` for progress - - Download results when complete - - When `--api-mode` disabled (default): - - Run standalone as currently works -2. API token generation: - - UI to generate API tokens (settings page) - - Store tokens in `api_tokens` table (hashed) - - API token authentication middleware - - Token expiration and revocation -3. CLI documentation: - - Update README.md with API mode usage - - Example commands for API mode - - Token generation instructions -4. Benefits of API mode: +**Planned Features:** +1. **API Client Mode:** + - `--api-mode` flag to enable API client mode + - `--api-url` and `--api-token` arguments + - Trigger scans via API, poll for status, download results - Scans stored centrally in database + - Standalone mode still available for testing + +2. **API Token System:** + - Token generation UI in settings page + - Secure token storage (hashed in database) + - Token authentication middleware + - Token expiration and revocation + +3. **Benefits:** + - Centralized scan history accessible via web dashboard - No need to mount volumes for output - Scheduled scans managed through web UI - - Scan history accessible via web dashboard - -**Deliverables:** -- CLI with `--api-mode` flag -- API token system -- Updated documentation - -**Testing:** -- CLI can trigger scan via API -- API token authentication works -- Standalone mode still functional -- Token revocation works + - Scriptable automation while leveraging web features --- -### Phase 6: Advanced Features (Weeks 10+) -**Priority: LOW** - Nice-to-have enhancements +### Phase 7: Advanced Features +**Status:** Future/Deferred +**Priority:** LOW -**Goals:** -- Enhanced interactive reports -- Vulnerability detection -- PDF export -- Timeline view - -**Tasks:** -1. Enhanced HTML reports: +**Planned Features:** +1. **Enhanced Reports:** - Sortable/filterable tables (DataTables.js) - - Inline screenshot thumbnails (lightbox on click) - - Export to PDF button (WeasyPrint or pdfkit) - - Print-friendly CSS -2. Vulnerability detection: - - Integrate with CVE databases (NVD API or Vulners API) - - Match detected services/versions to known CVEs - - Display CVE list with severity scores (CVSS) - - CVE detail page with description, remediation - - Alert rule for new critical CVEs -3. Timeline view: - - Visual timeline of all scans - - Filter by site or IP - - Click on timeline event to view scan - - Annotations for important events (cert renewals, config changes) -4. Advanced charts: - - Heatmap of port activity - - Service version tracking over time + - Inline screenshot thumbnails with lightbox + - PDF export (WeasyPrint) + +2. **Vulnerability Detection:** + - CVE database integration (NVD API) + - Service version matching to known CVEs + - CVSS severity scores + - Alert rules for critical CVEs + +3. **Timeline View:** + - Visual scan history timeline + - Filter by site/IP + - Event annotations + +4. **Advanced Charts:** + - Port activity heatmap + - Service version tracking - Certificate expiration forecast - - Top 10 services pie chart -5. Export/Import: - - Export scan data to CSV - - Import scan configs from CSV - - Bulk schedule creation -6. Additional integrations: - - Slack notifications (in addition to email) - - Webhook support (POST to custom URL on events) + +5. **Integrations:** + - Slack notifications + - Webhook support - Prometheus metrics export - -**Deliverables:** -- Interactive sortable tables -- CVE integration -- PDF export -- Timeline view -- Additional integrations - -**Testing:** -- DataTables work with large datasets -- CVE data fetches correctly -- PDF exports render properly -- Timeline view performs well with many scans + - CSV export/import --- -## Migration Strategy +## Current Architecture -### From Current CLI to Web App +**Primary Interface:** Web GUI (Phases 1-4 Complete) +- Full-featured Flask web application +- Dashboard, scan management, scheduling, config creator +- REST API for all operations +- Single-user deployment with SQLite -**Current State:** -- CLI tool (`scanner.py`) runs standalone -- Outputs JSON, HTML, ZIP files -- No database, no web UI +**Coming Soon:** CLI API Client (Phase 6 Planned) +- Thin client for scripting and automation +- Calls Flask API for scan operations +- Results stored centrally in database +- Access to all web features via command line -**Migration Path:** +**Core Scanning Engine:** +- Masscan for port discovery +- Nmap for service detection +- Playwright for screenshots +- sslyze for SSL/TLS analysis -#### Step 1: Add Database Layer (Phase 1) -- Database runs alongside CLI -- CLI can optionally save to DB (flag: `--save-to-db`) -- No breaking changes - -#### Step 2: Launch Web App (Phase 2-3) -- Web app reads from DB -- Users can trigger scans via web UI -- CLI still works standalone - -#### Step 3: Transition Period (Phase 4-5) -- Users gradually adopt web UI -- CLI used for scripting/automation -- Both modes fully supported - -#### Step 4: API Client Mode (Phase 5) -- CLI becomes thin API client -- All scans stored in central DB -- Standalone mode remains for testing - -#### Step 5: Full Web App (Phase 6+) -- Primary interface is web UI -- CLI optional for power users - -### Backward Compatibility - -**Maintained:** -- Existing YAML config format -- JSON/HTML/ZIP output files -- Screenshot capture -- Docker deployment - -**Deprecated (eventually):** -- Standalone CLI mode (Phase 6+) -- Direct file output (replaced by DB + API) +**Deployment:** +- Docker Compose for easy deployment +- SQLite database (single-user, embedded) +- Gunicorn WSGI server +- Optional Nginx reverse proxy ## Prioritized Feature List -### Must-Have (Phases 1-3) +### Completed ✅ (Phases 1-4) 1. **Database foundation** (SQLite3 + SQLAlchemy) 2. **Flask web app core** (REST API, authentication) 3. **Dashboard with scan history** (list, detail, delete) -4. **Trend charts** (Chart.js - port counts, service distribution) +4. **Trend charts** (Chart.js - port counts over time) 5. **Scheduled scans** (APScheduler + cron expressions) -6. **Manual scan trigger** (web UI button) +6. **Config creator** (CIDR-based, YAML editor) -### Should-Have (Phase 4) +### Next Up (Phase 5) 7. **Email notifications** (SMTP integration) 8. **Alert rules** (cert expiry, unexpected ports, etc.) 9. **Scan comparison reports** (diff view) -10. **Settings UI** (SMTP, alerts, retention) -### Nice-to-Have (Phases 5-6) -11. **CLI as API client** (token auth, backward compat) -12. **Sortable/filterable tables** (DataTables.js) -13. **PDF export** (WeasyPrint) -14. **Vulnerability detection** (CVE integration) -15. **Timeline view** (visual scan history) -16. **Embedded screenshot thumbnails** (lightbox) +### Planned (Phase 6-7) +10. **CLI as API client** (token auth, scripting) +11. **Sortable/filterable tables** (DataTables.js) +12. **PDF export** (WeasyPrint) +13. **Vulnerability detection** (CVE integration) +14. **Timeline view** (visual scan history) ### Future/Deferred -17. **Multi-user support** (if requirements change) -18. **Slack/webhook integrations** -19. **Prometheus metrics** -20. **Mobile-responsive dashboard** (Bootstrap handles basics) +15. **Multi-user support** (if requirements change) +16. **Slack/webhook integrations** +17. **Prometheus metrics** +18. **Advanced charts** (heatmaps, forecasts) ## Development Workflow @@ -900,7 +661,7 @@ All API endpoints return JSON and follow RESTful conventions. - [ ] API tokens work for authentication - [ ] Standalone CLI mode still functional -### Phase 6+ Success +### Phase 7 Success (Advanced Features) - [ ] CVE integration provides actionable vulnerability data - [ ] Timeline view helps track infrastructure changes - [ ] PDF exports are shareable and professional @@ -947,6 +708,7 @@ All API endpoints return JSON and follow RESTful conventions. | 2025-11-14 | 1.2 | **Phase 2 COMPLETE** - REST API (5 scan endpoints, 3 settings endpoints), background jobs (APScheduler), authentication (Flask-Login), web UI (dashboard, scans, login, errors), error handling (content negotiation, request IDs, logging), 100 tests passing, comprehensive documentation (API_REFERENCE.md, DEPLOYMENT.md, PHASE2_COMPLETE.md) | | 2025-11-17 | 1.3 | **Bug Fix** - Fixed Chart.js infinite canvas growth issue in scan detail page (duplicate initialization, missing chart.destroy(), missing fixed-height container) | | 2025-11-17 | 1.4 | **Phase 4 COMPLETE** - Config Creator with CIDR-based creation, YAML editor (CodeMirror), config management UI (list/edit/delete), REST API (7 endpoints), Docker volume permissions fix, comprehensive testing and documentation | +| 2025-11-17 | 1.5 | **Roadmap Compression** - Condensed completed phases (1-4) into concise summaries, updated project scope to emphasize web GUI frontend with CLI as API client coming soon (Phase 6), reorganized phases for clarity | --- diff --git a/docs/ai/MANUAL_TESTING.md b/docs/ai/MANUAL_TESTING.md deleted file mode 100644 index 71b0345..0000000 --- a/docs/ai/MANUAL_TESTING.md +++ /dev/null @@ -1,876 +0,0 @@ -# SneakyScanner Phase 2 - Manual Testing Checklist - -**Version:** 2.0 (Phase 2) -**Last Updated:** 2025-11-14 - -This document provides a comprehensive manual testing checklist for validating the SneakyScanner web application. Use this checklist to verify all features work correctly before deployment or release. - ---- - -## Table of Contents - -1. [Prerequisites](#prerequisites) -2. [Deployment & Startup](#deployment--startup) -3. [Authentication](#authentication) -4. [Scan Management (Web UI)](#scan-management-web-ui) -5. [Scan Management (API)](#scan-management-api) -6. [Error Handling](#error-handling) -7. [Performance & Concurrency](#performance--concurrency) -8. [Data Persistence](#data-persistence) -9. [Security](#security) -10. [Cleanup](#cleanup) - ---- - -## Prerequisites - -Before starting manual testing: - -- [ ] Docker and Docker Compose installed -- [ ] `.env` file configured with proper keys -- [ ] Test scan configuration available (e.g., `configs/example-site.yaml`) -- [ ] Network access for scanning (if using real targets) -- [ ] Browser for web UI testing (Chrome, Firefox, Safari, Edge) -- [ ] `curl` and `jq` for API testing -- [ ] At least 2GB free disk space for scan results - -**Recommended Test Environment:** -- Clean database (no existing scans) -- Test config with 1-2 IPs, 2-3 expected ports -- Expected scan duration: 1-3 minutes - ---- - -## Deployment & Startup - -### Test 1: Environment Configuration - -**Objective:** Verify environment variables are properly configured. - -**Steps:** -1. Check `.env` file exists: - ```bash - ls -la .env - ``` -2. Verify required keys are set (not defaults): - ```bash - grep SECRET_KEY .env - grep SNEAKYSCANNER_ENCRYPTION_KEY .env - ``` -3. Verify keys are not default values: - ```bash - grep -v "your-secret-key-here" .env | grep SECRET_KEY - ``` - -**Expected Result:** -- [ ] `.env` file exists -- [ ] `SECRET_KEY` is set to unique value (not `your-secret-key-here`) -- [ ] `SNEAKYSCANNER_ENCRYPTION_KEY` is set to unique value -- [ ] All required environment variables present - -### Test 2: Docker Compose Startup - -**Objective:** Verify web application starts successfully. - -**Steps:** -1. Start services: - ```bash - docker-compose -f docker-compose-web.yml up -d - ``` -2. Check container status: - ```bash - docker-compose -f docker-compose-web.yml ps - ``` -3. Check logs for errors: - ```bash - docker-compose -f docker-compose-web.yml logs web | tail -50 - ``` -4. Wait 30 seconds for healthcheck to pass - -**Expected Result:** -- [ ] Container starts without errors -- [ ] Status shows "Up" or "healthy" -- [ ] No error messages in logs -- [ ] Port 5000 is listening - -### Test 3: Health Check - -**Objective:** Verify health check endpoint responds correctly. - -**Steps:** -1. Call health endpoint: - ```bash - curl -s http://localhost:5000/api/settings/health | jq '.' - ``` - -**Expected Result:** -- [ ] HTTP 200 status code -- [ ] Response: `{"status": "healthy", "database": "connected"}` -- [ ] No authentication required - -### Test 4: Database Initialization - -**Objective:** Verify database was created and initialized. - -**Steps:** -1. Check database file exists: - ```bash - docker exec sneakyscanner_web ls -lh /app/data/sneakyscanner.db - ``` -2. Verify database has tables: - ```bash - docker exec sneakyscanner_web sqlite3 /app/data/sneakyscanner.db ".tables" - ``` - -**Expected Result:** -- [ ] Database file exists (`sneakyscanner.db`) -- [ ] Database file size > 0 bytes -- [ ] All 11 tables present: `scans`, `scan_sites`, `scan_ips`, `scan_ports`, `scan_services`, `scan_certificates`, `scan_tls_versions`, `schedules`, `alerts`, `alert_rules`, `settings` - ---- - -## Authentication - -### Test 5: Login Page Access - -**Objective:** Verify unauthenticated users are redirected to login. - -**Steps:** -1. Open browser to http://localhost:5000/dashboard (without logging in) -2. Observe redirect - -**Expected Result:** -- [ ] Redirected to http://localhost:5000/login -- [ ] Login page displays correctly -- [ ] Dark theme applied (slate/grey colors) -- [ ] Username and password fields visible -- [ ] "Login" button visible - -### Test 6: Login with Correct Password - -**Objective:** Verify successful login flow. - -**Steps:** -1. Navigate to http://localhost:5000/login -2. Enter password (default: `admin`) -3. Click "Login" button - -**Expected Result:** -- [ ] Redirected to http://localhost:5000/dashboard -- [ ] No error messages -- [ ] Navigation bar shows "Dashboard", "Scans", "Settings", "Logout" -- [ ] Welcome message displayed - -### Test 7: Login with Incorrect Password - -**Objective:** Verify failed login handling. - -**Steps:** -1. Navigate to http://localhost:5000/login -2. Enter incorrect password (e.g., `wrongpassword`) -3. Click "Login" button - -**Expected Result:** -- [ ] Stays on login page (no redirect) -- [ ] Error message displayed: "Invalid password" -- [ ] Password field cleared -- [ ] Can retry login - -### Test 8: Logout - -**Objective:** Verify logout destroys session. - -**Steps:** -1. Login successfully -2. Navigate to http://localhost:5000/dashboard -3. Click "Logout" in navigation bar -4. Try to access http://localhost:5000/dashboard again - -**Expected Result:** -- [ ] Logout redirects to login page -- [ ] Flash message: "Logged out successfully" -- [ ] Session destroyed (redirected to login when accessing protected pages) -- [ ] Cannot access dashboard without re-logging in - -### Test 9: API Authentication (Session Cookie) - -**Objective:** Verify API endpoints require authentication. - -**Steps:** -1. Call API endpoint without authentication: - ```bash - curl -i http://localhost:5000/api/scans - ``` -2. Login and save session cookie: - ```bash - curl -X POST http://localhost:5000/auth/login \ - -H "Content-Type: application/json" \ - -d '{"password":"admin"}' \ - -c cookies.txt - ``` -3. Call API endpoint with session cookie: - ```bash - curl -b cookies.txt http://localhost:5000/api/scans - ``` - -**Expected Result:** -- [ ] Request without auth returns 401 Unauthorized -- [ ] Login returns 200 OK with session cookie -- [ ] Request with auth cookie returns 200 OK with scan data - ---- - -## Scan Management (Web UI) - -### Test 10: Dashboard Display - -**Objective:** Verify dashboard loads and displays correctly. - -**Steps:** -1. Login successfully -2. Navigate to http://localhost:5000/dashboard -3. Observe page content - -**Expected Result:** -- [ ] Dashboard loads without errors -- [ ] Welcome message displayed -- [ ] "Run Scan Now" button visible -- [ ] Recent scans section visible (may be empty) -- [ ] Navigation works - -### Test 11: Trigger Scan via Web UI - -**Objective:** Verify scan can be triggered from dashboard. - -**Steps:** -1. Login and go to dashboard -2. Click "Run Scan Now" button -3. Observe scan starts -4. Wait for scan to complete (1-3 minutes) - -**Expected Result:** -- [ ] Scan starts (status shows "Running") -- [ ] Scan appears in recent scans list -- [ ] Scan ID assigned and displayed -- [ ] Status updates to "Completed" after scan finishes -- [ ] No error messages - -**Note:** If "Run Scan Now" button not yet implemented, use API to trigger scan (Test 15). - -### Test 12: View Scan List - -**Objective:** Verify scan list page displays correctly. - -**Steps:** -1. Login successfully -2. Navigate to http://localhost:5000/scans -3. Trigger at least 3 scans (via API or UI) -4. Refresh scan list page - -**Expected Result:** -- [ ] Scan list page loads -- [ ] All scans displayed in table -- [ ] Columns: ID, Timestamp, Title, Status, Actions -- [ ] Pagination controls visible (if > 20 scans) -- [ ] Each scan has "View" and "Delete" buttons - -### Test 13: View Scan Details - -**Objective:** Verify scan detail page displays complete results. - -**Steps:** -1. From scan list, click "View" on a completed scan -2. Observe scan details page - -**Expected Result:** -- [ ] Scan details page loads (http://localhost:5000/scans/{id}) -- [ ] Scan metadata displayed (ID, timestamp, duration, status) -- [ ] Sites section visible -- [ ] IPs section visible with ping status -- [ ] Ports section visible (TCP/UDP) -- [ ] Services section visible with product/version -- [ ] HTTPS services show certificate details (if applicable) -- [ ] TLS versions displayed (if applicable) -- [ ] Screenshot links work (if screenshots captured) -- [ ] Download buttons for JSON/HTML/ZIP files - -### Test 14: Delete Scan via Web UI - -**Objective:** Verify scan deletion removes all data and files. - -**Steps:** -1. Login and navigate to scan list -2. Note a scan ID to delete -3. Click "Delete" button on scan -4. Confirm deletion -5. Check database and filesystem - -**Expected Result:** -- [ ] Confirmation prompt appears -- [ ] After confirmation, scan removed from list -- [ ] Scan no longer appears in database -- [ ] JSON/HTML/ZIP files deleted from filesystem -- [ ] Screenshot directory deleted -- [ ] Success message displayed - ---- - -## Scan Management (API) - -### Test 15: Trigger Scan via API - -**Objective:** Verify scan can be triggered via REST API. - -**Steps:** -1. Login and save session cookie (see Test 9) -2. Trigger scan: - ```bash - curl -X POST http://localhost:5000/api/scans \ - -H "Content-Type: application/json" \ - -d '{"config_file":"/app/configs/example-site.yaml"}' \ - -b cookies.txt | jq '.' - ``` -3. Note the `scan_id` from response - -**Expected Result:** -- [ ] HTTP 201 Created response -- [ ] Response includes `scan_id` (integer) -- [ ] Response includes `status: "running"` -- [ ] Response includes `message: "Scan queued successfully"` - -### Test 16: Poll Scan Status - -**Objective:** Verify scan status can be polled via API. - -**Steps:** -1. Trigger a scan (Test 15) and note `scan_id` -2. Poll status immediately: - ```bash - curl -b cookies.txt http://localhost:5000/api/scans/{scan_id}/status | jq '.' - ``` -3. Wait 30 seconds and poll again -4. Continue polling until status is `completed` or `failed` - -**Expected Result:** -- [ ] Initial status: `"running"` -- [ ] Response includes `started_at` timestamp -- [ ] Response includes `completed_at: null` while running -- [ ] After completion: status changes to `"completed"` or `"failed"` -- [ ] `completed_at` timestamp set when done -- [ ] If failed, `error_message` is present - -### Test 17: Get Scan Details via API - -**Objective:** Verify complete scan details can be retrieved via API. - -**Steps:** -1. Trigger a scan and wait for completion -2. Get scan details: - ```bash - curl -b cookies.txt http://localhost:5000/api/scans/{scan_id} | jq '.' - ``` - -**Expected Result:** -- [ ] HTTP 200 OK response -- [ ] Response includes all scan metadata (id, timestamp, duration, status, title) -- [ ] Response includes file paths (json_path, html_path, zip_path, screenshot_dir) -- [ ] Response includes `sites` array -- [ ] Each site includes `ips` array -- [ ] Each IP includes `ports` array -- [ ] Each port includes `services` array -- [ ] HTTPS services include `certificates` array (if applicable) -- [ ] Certificates include `tls_versions` array (if applicable) -- [ ] All relationships properly nested - -### Test 18: List Scans with Pagination - -**Objective:** Verify scan list API supports pagination. - -**Steps:** -1. Trigger at least 25 scans -2. List first page: - ```bash - curl -b cookies.txt "http://localhost:5000/api/scans?page=1&per_page=20" | jq '.' - ``` -3. List second page: - ```bash - curl -b cookies.txt "http://localhost:5000/api/scans?page=2&per_page=20" | jq '.' - ``` - -**Expected Result:** -- [ ] First page returns 20 scans -- [ ] Response includes `total` (total count) -- [ ] Response includes `page: 1` and `pages` (total pages) -- [ ] Response includes `per_page: 20` -- [ ] Second page returns remaining scans -- [ ] No duplicate scans between pages - -### Test 19: Filter Scans by Status - -**Objective:** Verify scan list can be filtered by status. - -**Steps:** -1. Trigger scans with different statuses (running, completed, failed) -2. Filter by running: - ```bash - curl -b cookies.txt "http://localhost:5000/api/scans?status=running" | jq '.' - ``` -3. Filter by completed: - ```bash - curl -b cookies.txt "http://localhost:5000/api/scans?status=completed" | jq '.' - ``` -4. Filter by failed: - ```bash - curl -b cookies.txt "http://localhost:5000/api/scans?status=failed" | jq '.' - ``` - -**Expected Result:** -- [ ] Each filter returns only scans with matching status -- [ ] Total count reflects filtered results -- [ ] Empty status filter returns all scans - -### Test 20: Delete Scan via API - -**Objective:** Verify scan deletion via REST API. - -**Steps:** -1. Trigger a scan and wait for completion -2. Note the `scan_id` -3. Delete scan: - ```bash - curl -X DELETE -b cookies.txt http://localhost:5000/api/scans/{scan_id} | jq '.' - ``` -4. Verify deletion: - ```bash - curl -b cookies.txt http://localhost:5000/api/scans/{scan_id} - ``` -5. Check filesystem for scan files - -**Expected Result:** -- [ ] Delete returns HTTP 200 OK -- [ ] Delete response: `{"message": "Scan {id} deleted successfully"}` -- [ ] Subsequent GET returns HTTP 404 Not Found -- [ ] JSON/HTML/ZIP files deleted from filesystem -- [ ] Screenshot directory deleted -- [ ] Database record removed - ---- - -## Error Handling - -### Test 21: Invalid Config File - -**Objective:** Verify proper error handling for invalid config files. - -**Steps:** -1. Trigger scan with non-existent config: - ```bash - curl -X POST http://localhost:5000/api/scans \ - -H "Content-Type: application/json" \ - -d '{"config_file":"/app/configs/nonexistent.yaml"}' \ - -b cookies.txt | jq '.' - ``` - -**Expected Result:** -- [ ] HTTP 400 Bad Request -- [ ] Response includes `error` and `message` fields -- [ ] Error message indicates config file invalid/not found -- [ ] No scan record created in database - -### Test 22: Missing Required Field - -**Objective:** Verify API validates required fields. - -**Steps:** -1. Trigger scan without config_file: - ```bash - curl -X POST http://localhost:5000/api/scans \ - -H "Content-Type: application/json" \ - -d '{}' \ - -b cookies.txt | jq '.' - ``` - -**Expected Result:** -- [ ] HTTP 400 Bad Request -- [ ] Error message indicates missing required field - -### Test 23: Non-Existent Scan ID - -**Objective:** Verify 404 handling for non-existent scans. - -**Steps:** -1. Get scan with invalid ID: - ```bash - curl -b cookies.txt http://localhost:5000/api/scans/99999 | jq '.' - ``` - -**Expected Result:** -- [ ] HTTP 404 Not Found -- [ ] Response: `{"error": "Scan not found", "message": "Scan with ID 99999 does not exist"}` - -### Test 24: Invalid Pagination Parameters - -**Objective:** Verify pagination parameter validation. - -**Steps:** -1. Request with invalid page number: - ```bash - curl -b cookies.txt "http://localhost:5000/api/scans?page=-1" | jq '.' - ``` -2. Request with invalid per_page: - ```bash - curl -b cookies.txt "http://localhost:5000/api/scans?per_page=1000" | jq '.' - ``` - -**Expected Result:** -- [ ] HTTP 400 Bad Request for negative page -- [ ] per_page capped at maximum (100) -- [ ] Error message indicates validation failure - -### Test 25: Content Negotiation - -**Objective:** Verify API returns JSON and web UI returns HTML for errors. - -**Steps:** -1. Access non-existent scan via API: - ```bash - curl -H "Accept: application/json" http://localhost:5000/api/scans/99999 - ``` -2. Access non-existent scan via browser: - - Open http://localhost:5000/scans/99999 in browser - -**Expected Result:** -- [ ] API request returns JSON error -- [ ] Browser request returns HTML error page -- [ ] HTML error page matches dark theme -- [ ] HTML error page has navigation back to dashboard - -### Test 26: Error Templates - -**Objective:** Verify custom error templates render correctly. - -**Steps:** -1. Trigger 400 error (bad request) -2. Trigger 401 error (unauthorized - access API without login) -3. Trigger 404 error (non-existent page - http://localhost:5000/nonexistent) -4. Trigger 405 error (method not allowed - POST to GET-only endpoint) - -**Expected Result:** -- [ ] Each error displays custom error page -- [ ] Error pages use dark theme -- [ ] Error pages include error code and message -- [ ] Error pages have "Back to Dashboard" link -- [ ] Navigation bar visible on error pages (if authenticated) - -### Test 27: Request ID Tracking - -**Objective:** Verify request IDs are generated and included in responses. - -**Steps:** -1. Make API request and check headers: - ```bash - curl -i -b cookies.txt http://localhost:5000/api/scans - ``` - -**Expected Result:** -- [ ] Response includes `X-Request-ID` header -- [ ] Request ID is 8-character hex string -- [ ] Response includes `X-Request-Duration-Ms` header -- [ ] Duration is positive integer (milliseconds) - -### Test 28: Logging - -**Objective:** Verify requests are logged with request IDs. - -**Steps:** -1. Make API request -2. Check logs: - ```bash - docker-compose -f docker-compose-web.yml logs web | tail -20 - ``` - -**Expected Result:** -- [ ] Logs include request ID in brackets `[a1b2c3d4]` -- [ ] Logs include HTTP method, path, status code -- [ ] Logs include request duration in milliseconds -- [ ] Error logs include stack traces (if applicable) - ---- - -## Performance & Concurrency - -### Test 29: Concurrent Scans - -**Objective:** Verify multiple scans can run concurrently. - -**Steps:** -1. Trigger 3 scans simultaneously: - ```bash - curl -X POST http://localhost:5000/api/scans \ - -H "Content-Type: application/json" \ - -d '{"config_file":"/app/configs/example-site.yaml"}' \ - -b cookies.txt & - - curl -X POST http://localhost:5000/api/scans \ - -H "Content-Type: application/json" \ - -d '{"config_file":"/app/configs/example-site.yaml"}' \ - -b cookies.txt & - - curl -X POST http://localhost:5000/api/scans \ - -H "Content-Type: application/json" \ - -d '{"config_file":"/app/configs/example-site.yaml"}' \ - -b cookies.txt & - ``` -2. Check all scans are running: - ```bash - curl -b cookies.txt "http://localhost:5000/api/scans?status=running" | jq '.total' - ``` - -**Expected Result:** -- [ ] All 3 scans start successfully -- [ ] All 3 scans have status "running" -- [ ] No database locking errors in logs -- [ ] All 3 scans eventually complete - -### Test 30: API Responsiveness During Scan - -**Objective:** Verify web UI and API remain responsive during long-running scans. - -**Steps:** -1. Trigger a long-running scan (5+ minutes) -2. While scan is running, perform these actions: - - Navigate to dashboard - - List scans via API - - Get scan status via API - - Login/logout - -**Expected Result:** -- [ ] Web UI loads quickly (< 2 seconds) -- [ ] API requests respond quickly (< 500ms) -- [ ] No timeouts or slow responses -- [ ] Background scan does not block HTTP requests - ---- - -## Data Persistence - -### Test 31: Database Persistence Across Restarts - -**Objective:** Verify database persists across container restarts. - -**Steps:** -1. Trigger a scan and wait for completion -2. Note the scan ID -3. Restart container: - ```bash - docker-compose -f docker-compose-web.yml restart web - ``` -4. Wait for container to restart (check health) -5. Query scan via API - -**Expected Result:** -- [ ] Container restarts successfully -- [ ] Database file persists -- [ ] Scan still accessible after restart -- [ ] All scan data intact - -### Test 32: File Persistence - -**Objective:** Verify scan files persist in volume. - -**Steps:** -1. Trigger a scan and wait for completion -2. Note the file paths (JSON, HTML, ZIP, screenshots) -3. Verify files exist: - ```bash - docker exec sneakyscanner_web ls -lh /app/output/scan_report_*.json - ``` -4. Restart container -5. Verify files still exist - -**Expected Result:** -- [ ] All scan files created (JSON, HTML, ZIP, screenshots) -- [ ] Files persist after container restart -- [ ] Files accessible from host (mounted volume) -- [ ] File sizes are non-zero - ---- - -## Security - -### Test 33: Password Hashing - -**Objective:** Verify passwords are hashed with bcrypt. - -**Steps:** -1. Check password in database: - ```bash - docker exec sneakyscanner_web sqlite3 /app/data/sneakyscanner.db \ - "SELECT value FROM settings WHERE key='app_password';" - ``` - -**Expected Result:** -- [ ] Password is not stored in plaintext -- [ ] Password starts with `$2b$` (bcrypt hash) -- [ ] Hash is ~60 characters long - -### Test 34: Session Cookie Security - -**Objective:** Verify session cookies have secure attributes (in production). - -**Steps:** -1. Login via browser (with developer tools open) -2. Inspect cookies (Application > Cookies) -3. Check session cookie attributes - -**Expected Result:** -- [ ] Session cookie has `HttpOnly` flag -- [ ] Session cookie has `Secure` flag (if HTTPS) -- [ ] Session cookie has `SameSite` attribute -- [ ] Session cookie expires on logout - -### Test 35: SQL Injection Protection - -**Objective:** Verify inputs are sanitized against SQL injection. - -**Steps:** -1. Attempt SQL injection in scan list filter: - ```bash - curl -b cookies.txt "http://localhost:5000/api/scans?status='; DROP TABLE scans; --" - ``` -2. Check database is intact: - ```bash - docker exec sneakyscanner_web sqlite3 /app/data/sneakyscanner.db ".tables" - ``` - -**Expected Result:** -- [ ] No SQL injection occurs -- [ ] Database tables intact -- [ ] API returns validation error or empty results -- [ ] No database errors in logs - -### Test 36: File Path Traversal Protection - -**Objective:** Verify config file paths are validated against path traversal. - -**Steps:** -1. Attempt path traversal in config_file: - ```bash - curl -X POST http://localhost:5000/api/scans \ - -H "Content-Type: application/json" \ - -d '{"config_file":"../../../etc/passwd"}' \ - -b cookies.txt - ``` - -**Expected Result:** -- [ ] Request rejected with 400 Bad Request -- [ ] Error message indicates invalid config file -- [ ] No file outside /app/configs accessed -- [ ] Security error logged - ---- - -## Cleanup - -### Test 37: Stop Services - -**Objective:** Gracefully stop all services. - -**Steps:** -1. Stop services: - ```bash - docker-compose -f docker-compose-web.yml down - ``` -2. Verify containers stopped: - ```bash - docker-compose -f docker-compose-web.yml ps - ``` - -**Expected Result:** -- [ ] Services stop gracefully (no kill signals) -- [ ] All containers stopped -- [ ] No error messages in logs -- [ ] Volumes preserved (data, output, logs, configs) - -### Test 38: Volume Cleanup (Optional) - -**Objective:** Remove all data volumes (only if needed). - -**Steps:** -1. Stop and remove volumes: - ```bash - docker-compose -f docker-compose-web.yml down -v - ``` -2. Verify volumes removed: - ```bash - docker volume ls | grep sneakyscanner - ``` - -**Expected Result:** -- [ ] All volumes removed -- [ ] Database deleted -- [ ] Scan results deleted -- [ ] Logs deleted - -**Warning:** This is destructive and removes all data! - ---- - -## Summary - -### Test Results Summary - -Total Tests: 38 - -| Category | Tests | Passed | Failed | -|----------|-------|--------|--------| -| Deployment & Startup | 4 | | | -| Authentication | 5 | | | -| Scan Management (Web UI) | 5 | | | -| Scan Management (API) | 6 | | | -| Error Handling | 8 | | | -| Performance & Concurrency | 2 | | | -| Data Persistence | 2 | | | -| Security | 4 | | | -| Cleanup | 2 | | | -| **Total** | **38** | | | - -### Critical Tests (Must Pass) - -These tests are critical and must pass for Phase 2 to be considered complete: - -- [ ] Test 2: Docker Compose Startup -- [ ] Test 3: Health Check -- [ ] Test 6: Login with Correct Password -- [ ] Test 15: Trigger Scan via API -- [ ] Test 16: Poll Scan Status -- [ ] Test 17: Get Scan Details via API -- [ ] Test 18: List Scans with Pagination -- [ ] Test 20: Delete Scan via API -- [ ] Test 29: Concurrent Scans -- [ ] Test 31: Database Persistence Across Restarts - -### Known Issues - -Document any known issues or test failures here: - -1. **Issue:** [Description] - - **Severity:** Critical | High | Medium | Low - - **Workaround:** [Workaround if available] - - **Fix:** [Planned fix] - ---- - -## Notes - -- Tests should be run in order, as later tests may depend on earlier setup -- Some tests require multiple scans - consider batch creating scans for efficiency -- Performance tests are environment-dependent (Docker resources, network speed) -- Security tests are basic - professional security audit recommended for production -- Manual testing complements automated tests - both are important - ---- - -**Manual Testing Checklist Version:** 1.0 -**Phase:** 2 - Flask Web App Core -**Last Updated:** 2025-11-14 diff --git a/docs/ai/PHASE1_COMPLETE.md b/docs/ai/PHASE1_COMPLETE.md deleted file mode 100644 index 799b67d..0000000 --- a/docs/ai/PHASE1_COMPLETE.md +++ /dev/null @@ -1,404 +0,0 @@ -# Phase 1: Foundation - COMPLETE ✓ - -**Date Completed:** 2025-11-13 - -Phase 1 of the SneakyScanner roadmap has been successfully implemented. This document summarizes what was delivered and how to use the new infrastructure. - ---- - -## ✓ Deliverables Completed - -### 1. Database Schema & Models -- **SQLAlchemy models** for all 11 database tables (`web/models.py`) - - Core tables: `Scan`, `ScanSite`, `ScanIP`, `ScanPort`, `ScanService`, `ScanCertificate`, `ScanTLSVersion` - - Scheduling tables: `Schedule`, `Alert`, `AlertRule` - - Configuration: `Setting` -- **Alembic migrations** system configured (`migrations/`) -- **Initial migration** created (`migrations/versions/001_initial_schema.py`) - -### 2. Settings System with Encryption -- **SettingsManager** class with CRUD operations (`web/utils/settings.py`) -- **Automatic encryption** for sensitive values (SMTP passwords, API tokens) -- **PasswordManager** for bcrypt password hashing -- **Default settings initialization** for SMTP, authentication, retention policies - -### 3. Flask Application Structure -- **Flask app factory** pattern implemented (`web/app.py`) -- **API blueprints** for all major endpoints: - - `/api/scans` - Scan management (stub for Phase 2) - - `/api/schedules` - Schedule management (stub for Phase 3) - - `/api/alerts` - Alert management (stub for Phase 4) - - `/api/settings` - Settings API (functional in Phase 1!) -- **Error handlers** for common HTTP status codes -- **CORS support** for API access -- **Logging** to file and console -- **Database session management** with scoped sessions - -### 4. Database Initialization -- **init_db.py** script for easy database setup -- Supports both Alembic migrations and direct table creation -- Password setting during initialization -- Database verification and settings display - -### 5. Docker Support -- **Updated Dockerfile** with Flask dependencies -- **docker-compose-web.yml** for running the web application -- Separate service definition for database initialization -- Volume mounts for persistence (database, output, logs) - -### 6. Validation & Testing -- **validate_phase1.py** script to verify all deliverables -- Validates directory structure, files, Python syntax, models, and API endpoints -- All checks passing ✓ - ---- - -## 📁 New Project Structure - -``` -SneakyScanner/ -├── web/ # Flask web application (NEW) -│ ├── __init__.py -│ ├── app.py # Flask app factory -│ ├── models.py # SQLAlchemy models (11 tables) -│ ├── api/ # API blueprints -│ │ ├── __init__.py -│ │ ├── scans.py # Scans API -│ │ ├── schedules.py # Schedules API -│ │ ├── alerts.py # Alerts API -│ │ └── settings.py # Settings API (functional!) -│ ├── templates/ # Jinja2 templates (for Phase 3) -│ ├── static/ # CSS, JS, images (for Phase 3) -│ │ ├── css/ -│ │ ├── js/ -│ │ └── images/ -│ └── utils/ # Utility modules -│ ├── __init__.py -│ └── settings.py # Settings manager with encryption -├── migrations/ # Alembic migrations (NEW) -│ ├── env.py # Alembic environment -│ ├── script.py.mako # Migration template -│ └── versions/ -│ └── 001_initial_schema.py # Initial database migration -├── alembic.ini # Alembic configuration (NEW) -├── init_db.py # Database initialization script (NEW) -├── validate_phase1.py # Phase 1 validation script (NEW) -├── requirements-web.txt # Flask dependencies (NEW) -├── docker-compose-web.yml # Docker Compose for web app (NEW) -├── Dockerfile # Updated with Flask support -├── src/ # Existing scanner code (unchanged) -├── templates/ # Existing report templates (unchanged) -├── configs/ # Existing YAML configs (unchanged) -└── output/ # Existing scan outputs (unchanged) -``` - ---- - -## 🚀 Getting Started - -### Option 1: Local Development (without Docker) - -#### 1. Install Dependencies - -```bash -# Install Flask and web dependencies -pip install -r requirements-web.txt -``` - -#### 2. Initialize Database - -```bash -# Create database and set password -python3 init_db.py --password YOUR_SECURE_PASSWORD - -# Verify database -python3 init_db.py --verify-only -``` - -#### 3. Run Flask Application - -```bash -# Run development server -python3 -m web.app - -# Application will be available at http://localhost:5000 -``` - -#### 4. Test API Endpoints - -```bash -# Health check -curl http://localhost:5000/api/settings/health - -# Get all settings (sanitized) -curl http://localhost:5000/api/settings - -# Get specific setting -curl http://localhost:5000/api/settings/smtp_server - -# Update a setting -curl -X PUT http://localhost:5000/api/settings/smtp_server \ - -H "Content-Type: application/json" \ - -d '{"value": "smtp.gmail.com"}' - -# Set application password -curl -X POST http://localhost:5000/api/settings/password \ - -H "Content-Type: application/json" \ - -d '{"password": "newsecurepassword"}' -``` - ---- - -### Option 2: Docker Deployment - -#### 1. Build Docker Image - -```bash -docker-compose -f docker-compose-web.yml build -``` - -#### 2. Initialize Database (one-time) - -```bash -# Create data directory -mkdir -p data - -# Initialize database -docker-compose -f docker-compose-web.yml run --rm init-db --password YOUR_SECURE_PASSWORD -``` - -#### 3. Run Web Application - -```bash -# Start Flask web server -docker-compose -f docker-compose-web.yml up -d web - -# View logs -docker-compose -f docker-compose-web.yml logs -f web -``` - -#### 4. Access Application - -- Web API: http://localhost:5000 -- Health checks: - - http://localhost:5000/api/scans/health - - http://localhost:5000/api/schedules/health - - http://localhost:5000/api/alerts/health - - http://localhost:5000/api/settings/health - ---- - -## 🔐 Security Features - -### Encryption -- **Fernet encryption** for sensitive settings (SMTP passwords, API tokens) -- Encryption key auto-generated and stored in settings table -- Can be overridden via `SNEAKYSCANNER_ENCRYPTION_KEY` environment variable - -### Password Hashing -- **Bcrypt** for application password hashing (work factor 12) -- Password stored as irreversible hash in settings table -- Minimum 8 characters enforced - -### Session Management -- Flask sessions with configurable `SECRET_KEY` -- Set via environment variable or config - ---- - -## 📊 Database Schema - -### Core Tables -- **scans** - Scan metadata and status -- **scan_sites** - Site groupings -- **scan_ips** - IP addresses scanned -- **scan_ports** - Discovered ports -- **scan_services** - Service detection results -- **scan_certificates** - SSL/TLS certificates -- **scan_tls_versions** - TLS version support - -### Scheduling & Alerts -- **schedules** - Cron-like scan schedules -- **alerts** - Alert history -- **alert_rules** - Alert rule definitions - -### Configuration -- **settings** - Application settings (key-value store) - -All tables include proper foreign keys, indexes, and cascade delete rules. - ---- - -## 🧪 Validation - -Run the Phase 1 validation script to verify everything is in place: - -```bash -python3 validate_phase1.py -``` - -Expected output: -``` -✓ All Phase 1 validation checks passed! -``` - ---- - -## 🔧 Environment Variables - -Configure the Flask app via environment variables: - -```bash -# Flask configuration -export FLASK_ENV=development -export FLASK_DEBUG=true -export FLASK_HOST=0.0.0.0 -export FLASK_PORT=5000 - -# Database -export DATABASE_URL=sqlite:///./sneakyscanner.db - -# Security -export SECRET_KEY=your-secret-key-here -export SNEAKYSCANNER_ENCRYPTION_KEY=your-encryption-key-here - -# CORS (comma-separated origins) -export CORS_ORIGINS=http://localhost:3000,https://your-domain.com - -# Logging -export LOG_LEVEL=INFO -``` - -Or use a `.env` file (supported via `python-dotenv`). - ---- - -## 📝 API Endpoints Summary - -### Settings API (Functional in Phase 1) -| Method | Endpoint | Description | Status | -|--------|----------|-------------|--------| -| GET | `/api/settings` | Get all settings (sanitized) | ✓ Working | -| PUT | `/api/settings` | Update multiple settings | ✓ Working | -| GET | `/api/settings/{key}` | Get specific setting | ✓ Working | -| PUT | `/api/settings/{key}` | Update specific setting | ✓ Working | -| DELETE | `/api/settings/{key}` | Delete setting | ✓ Working | -| POST | `/api/settings/password` | Set app password | ✓ Working | -| GET | `/api/settings/health` | Health check | ✓ Working | - -### Scans API (Stubs for Phase 2) -| Method | Endpoint | Description | Status | -|--------|----------|-------------|--------| -| GET | `/api/scans` | List scans | Phase 2 | -| GET | `/api/scans/{id}` | Get scan details | Phase 2 | -| POST | `/api/scans` | Trigger scan | Phase 2 | -| DELETE | `/api/scans/{id}` | Delete scan | Phase 2 | -| GET | `/api/scans/{id}/status` | Get scan status | Phase 2 | -| GET | `/api/scans/health` | Health check | ✓ Working | - -### Schedules API (Stubs for Phase 3) -| Method | Endpoint | Description | Status | -|--------|----------|-------------|--------| -| GET | `/api/schedules` | List schedules | Phase 3 | -| POST | `/api/schedules` | Create schedule | Phase 3 | -| PUT | `/api/schedules/{id}` | Update schedule | Phase 3 | -| DELETE | `/api/schedules/{id}` | Delete schedule | Phase 3 | -| POST | `/api/schedules/{id}/trigger` | Trigger schedule | Phase 3 | -| GET | `/api/schedules/health` | Health check | ✓ Working | - -### Alerts API (Stubs for Phase 4) -| Method | Endpoint | Description | Status | -|--------|----------|-------------|--------| -| GET | `/api/alerts` | List alerts | Phase 4 | -| GET | `/api/alerts/rules` | List alert rules | Phase 4 | -| POST | `/api/alerts/rules` | Create alert rule | Phase 4 | -| PUT | `/api/alerts/rules/{id}` | Update alert rule | Phase 4 | -| DELETE | `/api/alerts/rules/{id}` | Delete alert rule | Phase 4 | -| GET | `/api/alerts/health` | Health check | ✓ Working | - ---- - -## ✅ Testing Checklist - -- [x] Database creates successfully -- [x] Settings can be stored/retrieved -- [x] Encryption works for sensitive values -- [x] Password hashing works -- [x] Flask app starts without errors -- [x] API blueprints load correctly -- [x] Health check endpoints respond -- [x] All Python files have valid syntax -- [x] All models defined correctly -- [x] Database migrations work - ---- - -## 🎯 Next Steps: Phase 2 - -Phase 2 will implement: -1. **REST API for scans** - Trigger scans, list history, get results -2. **Background job queue** - APScheduler for async scan execution -3. **Authentication** - Flask-Login for session management -4. **Scanner integration** - Save scan results to database -5. **Docker Compose deployment** - Production-ready setup - -Estimated timeline: 2 weeks (as per roadmap) - ---- - -## 📚 References - -### Key Files -- `web/models.py` - Database models (lines 1-400+) -- `web/app.py` - Flask app factory (lines 1-250+) -- `web/utils/settings.py` - Settings manager (lines 1-300+) -- `init_db.py` - Database initialization (lines 1-200+) -- `migrations/versions/001_initial_schema.py` - Initial migration (lines 1-250+) - -### Documentation -- [Flask Documentation](https://flask.palletsprojects.com/) -- [SQLAlchemy ORM](https://docs.sqlalchemy.org/) -- [Alembic Migrations](https://alembic.sqlalchemy.org/) -- [Cryptography Library](https://cryptography.io/) -- [Bcrypt](https://github.com/pyca/bcrypt) - ---- - -## 🐛 Troubleshooting - -### Database Issues -```bash -# Reset database -rm sneakyscanner.db -python3 init_db.py --password newpassword - -# Check database -sqlite3 sneakyscanner.db ".schema" -``` - -### Flask Won't Start -```bash -# Check dependencies installed -pip list | grep -i flask - -# Check syntax errors -python3 validate_phase1.py - -# Run with debug output -FLASK_DEBUG=true python3 -m web.app -``` - -### Encryption Errors -```bash -# Generate new encryption key -python3 -c "from cryptography.fernet import Fernet; print(Fernet.generate_key().decode())" - -# Set in environment -export SNEAKYSCANNER_ENCRYPTION_KEY="your-key-here" -``` - ---- - -**Phase 1 Status:** ✅ COMPLETE - -All deliverables implemented, tested, and validated. Ready to proceed with Phase 2. diff --git a/docs/ai/PHASE2.md b/docs/ai/PHASE2.md deleted file mode 100644 index f6d6cde..0000000 --- a/docs/ai/PHASE2.md +++ /dev/null @@ -1,1979 +0,0 @@ -# Phase 2 Implementation Plan: Flask Web App Core - -**Status:** Step 7 Complete ✅ - Error Handling & Logging (Day 12) -**Progress:** 12/14 days complete (86%) -**Estimated Duration:** 14 days (2 weeks) -**Dependencies:** Phase 1 Complete ✅ - -## Progress Summary - -- ✅ **Step 1: Database & Service Layer** (Days 1-2) - COMPLETE - - ScanService with full CRUD operations - - Pagination and validation utilities - - Database migration for indexes - - 15 unit tests (100% passing) - - 1,668 lines of code added -- ✅ **Step 2: Scan API Endpoints** (Days 3-4) - COMPLETE - - All 5 scan endpoints implemented - - Comprehensive error handling and logging - - 24 integration tests written - - 300+ lines of code added -- ✅ **Step 3: Background Job Queue** (Days 5-6) - COMPLETE - - APScheduler integration with BackgroundScheduler - - Scan execution in background threads - - SchedulerService with job management - - Database migration for scan timing fields - - 13 unit tests (scheduler, timing, errors) - - 600+ lines of code added -- ✅ **Step 4: Authentication System** (Days 7-8) - COMPLETE - - Flask-Login integration with single-user support - - User model with bcrypt password hashing - - Login, logout, and password setup routes - - @login_required and @api_auth_required decorators - - All API endpoints protected with authentication - - Bootstrap 5 dark theme UI templates - - 30+ authentication tests - - 1,200+ lines of code added -- ✅ **Step 5: Basic UI Templates** (Days 9-10) - COMPLETE - - base.html template with navigation and slate dark theme - - dashboard.html with stats cards and recent scans - - scans.html with pagination and filtering - - scan_detail.html with comprehensive scan results display - - login.html updated to use dark theme - - All templates use matching color scheme from report_mockup.html - - AJAX-powered dynamic data loading - - Auto-refresh for running scans - - Responsive design with Bootstrap 5 -- ✅ **Step 6: Docker & Deployment** (Day 11) - COMPLETE - - Updated docker-compose-web.yml with scheduler configuration - - Added privileged mode and host networking for scanner support - - Configured health check endpoint monitoring - - Created .env.example with comprehensive configuration template - - Verified Dockerfile is production-ready - - Created comprehensive DEPLOYMENT.md documentation - - Deployment workflow validated -- ✅ **Step 7: Error Handling & Logging** (Day 12) - COMPLETE - - Enhanced logging with rotation (10MB per file, 10 backups) - - Structured logging with request IDs and timing - - Request/response logging middleware with duration tracking - - Database error handling with automatic rollback - - Custom error templates for 400, 401, 403, 404, 405, 500 - - Content negotiation (JSON for API, HTML for web) - - SQLite WAL mode for better concurrency - - Comprehensive error handling tests -- 📋 **Step 8: Testing & Documentation** (Days 13-14) - NEXT - ---- - -## Table of Contents - -1. [Overview](#overview) -2. [Current State Analysis](#current-state-analysis) -3. [Files to Create](#files-to-create) -4. [Files to Modify](#files-to-modify) -5. [Step-by-Step Implementation](#step-by-step-implementation) -6. [Dependencies & Prerequisites](#dependencies--prerequisites) -7. [Testing Approach](#testing-approach) -8. [Potential Challenges & Solutions](#potential-challenges--solutions) -9. [Success Criteria](#success-criteria) -10. [Migration Path](#migration-path) -11. [Estimated Timeline](#estimated-timeline) -12. [Key Design Decisions](#key-design-decisions) -13. [Documentation Deliverables](#documentation-deliverables) - ---- - -## Overview - -Phase 2 focuses on creating the core web application functionality by: - -1. **REST API for Scans** - Trigger scans and retrieve results via API -2. **Background Job Queue** - Execute scans asynchronously using APScheduler -3. **Authentication** - Simple Flask-Login session management -4. **Scanner Integration** - Save scan results to database automatically -5. **Basic UI** - Login page and dashboard placeholder - -### Goals - -- ✅ Working REST API for scan management (trigger, list, view, delete, status) -- ✅ Background scan execution with status tracking -- ✅ Basic authentication system with session management -- ✅ Scanner saves all results to database -- ✅ Simple login page and dashboard placeholder -- ✅ Production-ready Docker deployment - ---- - -## Current State Analysis - -### What's Already Done (Phase 1) - -- ✅ Database schema with 11 models (Scan, ScanSite, ScanIP, ScanPort, ScanService, ScanCertificate, ScanTLSVersion, Schedule, Alert, AlertRule, Setting) -- ✅ SQLAlchemy models with relationships -- ✅ Flask app factory pattern in `web/app.py` -- ✅ Settings management with encryption (SettingsManager, PasswordManager) -- ✅ API blueprint stubs for scans, schedules, alerts, settings -- ✅ Alembic migrations system -- ✅ Docker deployment infrastructure - -### Scanner Capabilities (src/scanner.py) - -The existing scanner has these key characteristics: - -- **scan() method** returns `(report_dict, timestamp)` tuple -- **generate_outputs()** creates JSON, HTML, ZIP files -- **Five-phase scanning:** ping, TCP ports, UDP ports, service detection, HTTP/HTTPS analysis -- **Screenshot capture** with Playwright -- **Results structured** by sites → IPs → ports → services - -**Key Methods:** -- `scan()` - Main workflow, returns report and timestamp -- `generate_outputs(report, timestamp)` - Creates JSON/HTML/ZIP files -- `save_report(report, timestamp)` - Saves JSON to disk -- `_run_masscan()` - Port scanning -- `_run_nmap_service_detection()` - Service detection -- `_run_http_analysis()` - HTTP/HTTPS and SSL/TLS analysis - ---- - -## Files to Create - -### Backend Services (Core Logic) - -#### 1. `web/services/__init__.py` -Services package initialization. - -#### 2. `web/services/scan_service.py` -Core service for scan orchestration and database integration. - -**Class: ScanService** - -Methods: -- `trigger_scan(config_file, triggered_by='manual', schedule_id=None)` → scan_id - - Validate config file exists - - Create Scan record with status='running' - - Queue background job - - Return scan_id - -- `get_scan(scan_id)` → scan dict with all related data - - Query Scan with all relationships - - Format for API response - - Include sites, IPs, ports, services, certificates, TLS versions - -- `list_scans(page=1, per_page=20, status_filter=None)` → paginated results - - Query with pagination - - Filter by status if provided - - Return total count and items - -- `delete_scan(scan_id)` → cleanup DB + files - - Delete database records (cascade handles relationships) - - Delete JSON, HTML, ZIP files - - Delete screenshot directory - - Handle missing files gracefully - -- `get_scan_status(scan_id)` → status dict - - Return current scan status - - Include progress percentage if available - - Return error message if failed - -- `_save_scan_to_db(report, scan_id, status='completed')` → persist results - - Update Scan record with duration, file paths - - Call _map_report_to_models() - - Commit transaction - -- `_map_report_to_models(report, scan_obj)` → create related records - - Map JSON structure to database models - - Create ScanSite, ScanIP, ScanPort, ScanService records - - Create ScanCertificate and ScanTLSVersion records - - Handle nested relationships properly - -#### 3. `web/services/scheduler_service.py` -APScheduler integration for scheduled scans. - -**Class: SchedulerService** - -Methods: -- `init_scheduler(app)` → setup APScheduler - - Initialize BackgroundScheduler - - Load existing schedules from DB - - Start scheduler - -- `add_job(schedule_id, config_file, cron_expression)` → create scheduled job - - Parse cron expression - - Add job to scheduler - - Store job_id in database - -- `remove_job(schedule_id)` → cancel job - - Remove from scheduler - - Update database - -- `trigger_scheduled_scan(schedule_id)` → manual trigger - - Load schedule from DB - - Trigger scan via ScanService - - Update last_run timestamp - -- `update_schedule_times(schedule_id, last_run, next_run)` → DB update - - Update Schedule record - - Commit transaction - -### Authentication System - -#### 4. `web/auth/__init__.py` -Authentication package initialization. - -#### 5. `web/auth/routes.py` -Login/logout routes blueprint. - -**Routes:** -- `GET /login` - Render login form -- `POST /login` - Authenticate and create session - - Verify password via PasswordManager - - Create Flask-Login session - - Redirect to dashboard -- `GET /logout` - Destroy session - - Logout user - - Redirect to login page - -#### 6. `web/auth/decorators.py` -Custom authentication decorators. - -**Decorators:** -- `@login_required` - Wrapper for Flask-Login's login_required -- `@api_auth_required` - For API endpoints (session-based for Phase 2) - -#### 7. `web/auth/models.py` -User model for Flask-Login. - -**Class: User** -- Simple class representing the single application user -- Load from settings table (app_password) -- Implement Flask-Login required methods (get_id, is_authenticated, etc.) - -### Frontend Templates - -#### 8. `web/templates/base.html` -Base layout with Bootstrap 5 dark theme. - -**Features:** -- Navigation bar (Dashboard, Scans, Settings, Logout) -- Flash message display -- Jinja2 blocks: title, content -- Footer with version info -- Bootstrap 5 dark theme CSS -- Mobile responsive - -#### 9. `web/templates/login.html` -Login page. - -**Features:** -- Username/password form -- Error message display -- Remember me checkbox (optional) -- Redirect to dashboard after login -- Clean, minimal design - -#### 10. `web/templates/dashboard.html` -Dashboard placeholder. - -**Features:** -- Welcome message -- "Run Scan Now" button (manual trigger) -- Recent scans table (5 most recent) -- Summary stats: - - Total scans - - Last scan time - - Scans running -- Link to full scan history - -### Background Jobs - -#### 11. `web/jobs/__init__.py` -Jobs package initialization. - -#### 12. `web/jobs/scan_job.py` -Background scan execution. - -**Function: execute_scan(config_file, scan_id, db_url)** -- Run scanner in subprocess -- Update scan status in DB (running → completed/failed) -- Handle exceptions and log errors -- Store scan results in database -- Generate JSON/HTML/ZIP files - -**Implementation:** -```python -import subprocess -from pathlib import Path -from sqlalchemy import create_engine -from sqlalchemy.orm import sessionmaker -from web.models import Scan -from src.scanner import SneakyScanner -from web.services.scan_service import ScanService - -def execute_scan(config_file, scan_id, db_url): - """Execute scan in background and save to database.""" - engine = create_engine(db_url) - Session = sessionmaker(bind=engine) - session = Session() - - try: - # Update status to running - scan = session.query(Scan).get(scan_id) - scan.status = 'running' - session.commit() - - # Run scanner - scanner = SneakyScanner(config_file) - report, timestamp = scanner.scan() - - # Generate outputs (JSON, HTML, ZIP) - scanner.generate_outputs(report, timestamp) - - # Save to database - scan_service = ScanService(session) - scan_service._save_scan_to_db(report, scan_id, status='completed') - - except Exception as e: - # Mark as failed - scan = session.query(Scan).get(scan_id) - scan.status = 'failed' - session.commit() - raise - finally: - session.close() -``` - -### Utilities - -#### 13. `web/utils/pagination.py` -Pagination helper. - -**Function: paginate(query, page, per_page)** -- Apply offset and limit to SQLAlchemy query -- Return paginated results with metadata -- Handle edge cases (invalid page, empty results) - -#### 14. `web/utils/validators.py` -Input validation utilities. - -**Functions:** -- `validate_config_file(path)` → check file exists and is valid YAML -- `validate_scan_status(status)` → enum validation (running, completed, failed) -- `validate_page_params(page, per_page)` → sanitize pagination params - -### Web Routes - -#### 15. `web/routes/__init__.py` -Web routes package initialization. - -#### 16. `web/routes/main.py` -Main web routes (dashboard, etc.). - -**Routes:** -- `GET /` - Redirect to dashboard -- `GET /dashboard` - Dashboard page (@login_required) -- `GET /scans` - Scan list page (@login_required) -- `GET /scans/` - Scan details page (@login_required) - -### Testing - -#### 17. `tests/__init__.py` -Test package initialization. - -#### 18. `tests/conftest.py` -Pytest fixtures and configuration. - -**Fixtures:** -- `app` - Flask app instance with test config -- `client` - Flask test client -- `db` - Test database session -- `sample_scan` - Sample scan data for testing - -#### 19. `tests/test_scan_api.py` -API endpoint tests. - -**Tests:** -- `test_list_scans` - GET /api/scans -- `test_get_scan` - GET /api/scans/ -- `test_trigger_scan` - POST /api/scans -- `test_delete_scan` - DELETE /api/scans/ -- `test_scan_status` - GET /api/scans//status -- `test_pagination` - List scans with page/per_page params -- `test_authentication` - Verify auth required - -#### 20. `tests/test_scan_service.py` -ScanService unit tests. - -**Tests:** -- `test_trigger_scan` - Scan creation and queuing -- `test_get_scan` - Retrieve scan with relationships -- `test_list_scans` - Pagination and filtering -- `test_delete_scan` - Cleanup files and DB records -- `test_save_scan_to_db` - Database persistence -- `test_map_report_to_models` - JSON to DB mapping - -#### 21. `tests/test_authentication.py` -Authentication tests. - -**Tests:** -- `test_login_success` - Valid credentials -- `test_login_failure` - Invalid credentials -- `test_logout` - Session destruction -- `test_protected_route` - Requires authentication - ---- - -## Files to Modify - -### Backend Updates - -#### 1. `web/app.py` -Flask application factory - add authentication and scheduler. - -**Changes:** -- Import Flask-Login and configure LoginManager -- Import APScheduler and initialize in app factory -- Register auth blueprint -- Register web routes blueprint -- Add user_loader callback for Flask-Login -- Add before_request handler for authentication -- Initialize scheduler service - -**New imports:** -```python -from flask_login import LoginManager -from web.auth.routes import bp as auth_bp -from web.routes.main import bp as main_bp -from web.services.scheduler_service import SchedulerService -``` - -**New code:** -```python -# Initialize Flask-Login -login_manager = LoginManager() -login_manager.login_view = 'auth.login' -login_manager.init_app(app) - -@login_manager.user_loader -def load_user(user_id): - from web.auth.models import User - return User.get(user_id) - -# Initialize APScheduler -scheduler_service = SchedulerService() -scheduler_service.init_scheduler(app) - -# Register blueprints -app.register_blueprint(auth_bp, url_prefix='/auth') -app.register_blueprint(main_bp, url_prefix='/') -``` - -#### 2. `web/api/scans.py` -Implement all scan endpoints (currently stubs). - -**Changes:** -- Import ScanService -- Implement all endpoint logic -- Add authentication decorators -- Add proper error handling -- Add input validation -- Add logging - -**Endpoint implementations:** - -```python -from web.services.scan_service import ScanService -from web.auth.decorators import api_auth_required -from web.utils.validators import validate_config_file - -@bp.route('', methods=['POST']) -@api_auth_required -def trigger_scan(): - """Trigger a new scan.""" - data = request.get_json() or {} - config_file = data.get('config_file') - - # Validate config file - if not validate_config_file(config_file): - return jsonify({'error': 'Invalid config file'}), 400 - - # Trigger scan - scan_service = ScanService(current_app.db_session) - scan_id = scan_service.trigger_scan(config_file, triggered_by='api') - - return jsonify({ - 'scan_id': scan_id, - 'status': 'running', - 'message': 'Scan queued successfully' - }), 201 - -# Similar implementations for GET, DELETE, status endpoints... -``` - -#### 3. `src/scanner.py` -Minor modifications for progress callbacks (optional). - -**Changes (optional):** -- Add optional `progress_callback` parameter to scan() method -- Call callback at each phase (ping, TCP, UDP, services, HTTP) -- No breaking changes to existing functionality - -**Example:** -```python -def scan(self, progress_callback=None): - """Run scan with optional progress reporting.""" - if progress_callback: - progress_callback('phase', 'ping', 0) - - # Run ping scan - ping_results = self._run_ping_scan(all_ips) - - if progress_callback: - progress_callback('phase', 'tcp_scan', 20) - - # Continue with other phases... -``` - -#### 4. `requirements-web.txt` -Add missing dependencies. - -**Add:** -``` -Flask-APScheduler==1.13.1 -``` - -All other dependencies already present from Phase 1. - -#### 5. `docker-compose-web.yml` -Updates for production deployment. - -**Changes:** -- Add environment variable for scheduler threads -- Ensure proper volume mounts for data persistence -- Add healthcheck for web service -- Configure restart policy - -**Example additions:** -```yaml -environment: - - SCHEDULER_EXECUTORS=2 # Number of concurrent scan jobs - - SCHEDULER_JOB_DEFAULTS_MAX_INSTANCES=3 -healthcheck: - test: ["CMD", "curl", "-f", "http://localhost:5000/api/settings/health"] - interval: 30s - timeout: 10s - retries: 3 -restart: unless-stopped -``` - -#### 6. Create New Alembic Migration - -**File:** `migrations/versions/002_add_scan_indexes.py` - -Add database indexes for better query performance: -- Index on scans.timestamp (for sorting recent scans) -- Index on scans.status (for filtering) -- Verify foreign key indexes exist (usually auto-created) - -**Migration code:** -```python -def upgrade(): - op.create_index('ix_scans_status', 'scans', ['status']) - # timestamp index already exists from 001 migration - -def downgrade(): - op.drop_index('ix_scans_status', 'scans') -``` - -#### 7. `docs/ai/ROADMAP.md` -Update with Phase 2 progress. - -**Changes:** -- Mark Phase 2 tasks as completed -- Update success metrics -- Add Phase 2 completion date - ---- - -## Step-by-Step Implementation - -### Step 1: Database & Service Layer ✅ COMPLETE (Days 1-2) -**Priority: CRITICAL** - Foundation for everything else - -**Status:** ✅ Complete - Committed: d7c68a2 - -**Tasks Completed:** -1. ✅ Created `web/services/` package -2. ✅ Implemented `ScanService` class (545 lines) - - ✅ `trigger_scan()` - Create scan records - - ✅ `get_scan()` - Retrieve with eager loading - - ✅ `list_scans()` - Paginated list with filtering - - ✅ `delete_scan()` - Remove DB records and files - - ✅ `get_scan_status()` - Poll scan status - - ✅ `_save_scan_to_db()` - Persist results - - ✅ `_map_report_to_models()` - Complex JSON-to-DB mapping - - ✅ Helper methods for dict conversion -3. ✅ Implemented pagination utility (`web/utils/pagination.py` - 153 lines) - - PaginatedResult class with metadata - - paginate() function for SQLAlchemy queries - - validate_page_params() for input sanitization -4. ✅ Implemented validators (`web/utils/validators.py` - 245 lines) - - validate_config_file() - YAML structure validation - - validate_scan_status() - Enum validation - - validate_scan_id(), validate_port(), validate_ip_address() - - sanitize_filename() - Security -5. ✅ Wrote comprehensive unit tests (374 lines) - - 15 tests covering all ScanService methods - - Test fixtures for DB, reports, config files - - Tests for trigger, get, list, delete, status - - Tests for complex database mapping - - **All tests passing ✓** -6. ✅ Created Alembic migration 002 for scan status index - -**Testing Results:** -- ✅ All 15 unit tests passing -- ✅ Database records created correctly with nested relationships -- ✅ Pagination logic validated -- ✅ Foreign key relationships working -- ✅ Complex JSON-to-DB mapping successful - -**Files Created:** -- web/services/__init__.py -- web/services/scan_service.py (545 lines) -- web/utils/pagination.py (153 lines) -- web/utils/validators.py (245 lines) -- migrations/versions/002_add_scan_indexes.py -- tests/__init__.py -- tests/conftest.py (142 lines) -- tests/test_scan_service.py (374 lines) - -**Total:** 8 files, 1,668 lines added - -**Key Challenge:** Mapping complex JSON structure to normalized database schema - -**Solution Implemented:** Process in order (sites → IPs → ports → services → certs → TLS), use SQLAlchemy relationships for FK handling, flush() after each level for ID generation - -### Step 2: Scan API Endpoints ✅ COMPLETE (Days 3-4) -**Priority: HIGH** - Core functionality - -**Status:** ✅ Complete - Committed: [pending] - -**Tasks Completed:** -1. ✅ Updated `web/api/scans.py`: - - ✅ Implemented `POST /api/scans` (trigger scan) - - ✅ Implemented `GET /api/scans` (list with pagination) - - ✅ Implemented `GET /api/scans/` (get details) - - ✅ Implemented `DELETE /api/scans/` (delete scan + files) - - ✅ Implemented `GET /api/scans//status` (status polling) -2. ✅ Added comprehensive error handling for all endpoints -3. ✅ Added structured logging with appropriate log levels -4. ✅ Wrote 24 integration tests covering: - - Empty and populated scan lists - - Pagination with multiple pages - - Status filtering - - Individual scan retrieval - - Scan triggering with validation - - Scan deletion - - Status polling - - Complete workflow integration test - - Error handling scenarios (404, 400, 500) - -**Testing Results:** -- ✅ All endpoints properly handle errors (400, 404, 500) -- ✅ Pagination logic implemented with metadata -- ✅ Input validation through validators -- ✅ Logging at appropriate levels (info, warning, error, debug) -- ✅ Integration tests written and ready to run in Docker - -**Files Modified:** -- web/api/scans.py (262 lines, all endpoints implemented) - -**Files Created:** -- tests/test_scan_api.py (301 lines, 24 tests) -- tests/conftest.py (updated with Flask fixtures) - -**Total:** 2 files modified, 563 lines added/modified - -**Key Implementation Details:** -- All endpoints use ScanService for business logic -- Proper HTTP status codes (200, 201, 400, 404, 500) -- Consistent JSON error format with 'error' and 'message' keys -- SQLAlchemy error handling with graceful degradation -- Logging includes request details and scan IDs for traceability - -**Key Challenge Addressed:** Long-running scans causing HTTP timeouts - -**Solution Implemented:** POST /api/scans immediately returns scan_id with status 'running', client polls GET /api/scans//status for updates - -### Step 3: Background Job Queue ✅ COMPLETE (Days 5-6) -**Priority: HIGH** - Async scan execution - -**Status:** ✅ Complete - Committed: [pending] - -**Tasks Completed:** -1. ✅ Created `web/jobs/` package structure -2. ✅ Implemented `web/jobs/scan_job.py` (130 lines): - - `execute_scan()` - Runs scanner in background thread - - Creates isolated database session per thread - - Updates scan status: running → completed/failed - - Handles exceptions with detailed error logging - - Stores error messages in database - - Tracks timing with started_at/completed_at -3. ✅ Created `SchedulerService` class (web/services/scheduler_service.py - 220 lines): - - Initialized APScheduler with BackgroundScheduler - - ThreadPoolExecutor for concurrent jobs (max 3 workers) - - `queue_scan()` - Queue immediate scan execution - - `add_scheduled_scan()` - Placeholder for future scheduled scans - - `remove_scheduled_scan()` - Remove scheduled jobs - - `list_jobs()` and `get_job_status()` - Job monitoring - - Graceful shutdown handling -4. ✅ Integrated APScheduler with Flask app (web/app.py): - - Created `init_scheduler()` function - - Initialized in app factory after extensions - - Stored scheduler in app context (`app.scheduler`) -5. ✅ Updated `ScanService.trigger_scan()` to queue background jobs: - - Added `scheduler` parameter - - Queues job immediately after creating scan record - - Handles job queuing failures gracefully -6. ✅ Added database fields for scan timing (migration 003): - - `started_at` - When scan execution began - - `completed_at` - When scan finished - - `error_message` - Error details for failed scans -7. ✅ Updated `ScanService.get_scan_status()` to include new fields -8. ✅ Updated API endpoint `POST /api/scans` to pass scheduler - -**Testing Results:** -- ✅ 13 unit tests for background jobs and scheduler -- ✅ Tests for scheduler initialization -- ✅ Tests for job queuing and status tracking -- ✅ Tests for scan timing fields -- ✅ Tests for error handling and storage -- ✅ Tests for job listing and monitoring -- ✅ Integration test for full workflow (skipped by default - requires scanner) - -**Files Created:** -- web/jobs/__init__.py (6 lines) -- web/jobs/scan_job.py (130 lines) -- web/services/scheduler_service.py (220 lines) -- migrations/versions/003_add_scan_timing_fields.py (38 lines) -- tests/test_background_jobs.py (232 lines) - -**Files Modified:** -- web/app.py (added init_scheduler function and call) -- web/models.py (added 3 fields to Scan model) -- web/services/scan_service.py (updated trigger_scan and get_scan_status) -- web/api/scans.py (pass scheduler to trigger_scan) - -**Total:** 5 files created, 4 files modified, 626 lines added - -**Key Implementation Details:** -- BackgroundScheduler runs in separate thread pool -- Each background job gets isolated database session -- Scan status tracked through lifecycle: created → running → completed/failed -- Error messages captured and stored in database -- Graceful shutdown waits for running jobs -- Job IDs follow pattern: `scan_{scan_id}` -- Support for concurrent scans (max 3 default, configurable) - -**Key Challenge Addressed:** Scanner requires privileged operations (masscan/nmap) - -**Solution Implemented:** -- Scanner runs in subprocess from background thread -- Docker container provides necessary privileges (--privileged, --network host) -- Background thread isolation prevents web app crashes -- Database session per thread avoids SQLite locking issues - -### Step 4: Authentication System ⏱️ Days 7-8 -**Priority: HIGH** - Security - -**Tasks:** -1. Create `web/auth/` package -2. Implement Flask-Login integration: - - Create User class (simple - single user) - - Configure LoginManager in app factory - - Implement user_loader callback -3. Create `auth/routes.py`: - - `GET /login` - render login form - - `POST /login` - authenticate and create session - - `GET /logout` - destroy session -4. Create `auth/decorators.py`: - - `@login_required` for web routes - - `@api_auth_required` for API endpoints -5. Apply decorators to all API endpoints -6. Test authentication flow - -**Testing:** -- Test login with correct/incorrect password -- Verify session persistence -- Test logout -- Verify protected routes require auth -- Test API authentication -- Test session timeout - -**Key Challenge:** Need both web UI and API authentication - -**Solution:** Use Flask-Login sessions for both (Phase 5 adds token auth) - -### Step 5: Basic UI Templates ⏱️ Days 9-10 -**Priority: MEDIUM** - User interface - -**Tasks:** -1. Create `base.html` template: - - Bootstrap 5 dark theme - - Navigation bar (Dashboard, Scans, Logout) - - Flash message display - - Footer with version info -2. Create `login.html`: - - Simple username/password form - - Error message display - - Redirect to dashboard after login -3. Create `dashboard.html`: - - Welcome message - - "Run Scan Now" button (manual trigger) - - Recent scans table (AJAX call to API) - - Summary stats (total scans, last scan time) -4. Create web routes blueprint (`web/routes/main.py`) -5. Style with Bootstrap 5 dark theme -6. Add minimal JavaScript for AJAX calls - -**Testing:** -- Verify templates render correctly -- Test responsive layout (desktop, tablet, mobile) -- Test login flow end-to-end -- Verify dashboard displays data from API -- Test manual scan trigger - -**Key Feature:** Dark theme matching existing HTML reports - -### Step 6: Docker & Deployment ✅ COMPLETE (Day 11) -**Priority: MEDIUM** - Production readiness - -**Status:** ✅ Complete - -**Tasks Completed:** -1. ✅ Reviewed Dockerfile (confirmed production-ready, no changes needed) -2. ✅ Updated `docker-compose-web.yml`: - - ✅ Verified volume mounts (configs, data, output, logs) - - ✅ Added environment variables for scheduler (SCHEDULER_EXECUTORS, SCHEDULER_JOB_DEFAULTS_MAX_INSTANCES) - - ✅ Added SNEAKYSCANNER_ENCRYPTION_KEY environment variable - - ✅ Set proper restart policy (`unless-stopped` already configured) - - ✅ Added comprehensive healthcheck with 30s interval, 10s timeout, 3 retries, 40s start period - - ✅ Added `privileged: true` for scanner raw socket access (masscan/nmap) - - ✅ Added `network_mode: host` for scanner network access - - ✅ Changed default FLASK_ENV to production -3. ✅ Created `.env.example` file with comprehensive configuration template: - - Flask configuration options - - Database configuration - - Security settings (SECRET_KEY, SNEAKYSCANNER_ENCRYPTION_KEY) - - CORS configuration - - Logging configuration - - Scheduler configuration - - Detailed comments and examples for key generation -4. ✅ Validated deployment workflow: - - Docker Compose configuration validated successfully - - All required directories exist - - Configuration syntax verified -5. ✅ Created comprehensive deployment documentation (`docs/ai/DEPLOYMENT.md`): - - Overview and architecture - - Prerequisites and system requirements - - Quick start guide - - Detailed configuration instructions - - First-time setup procedure - - Running and managing the application - - Volume management and backup procedures - - Health monitoring guide - - Extensive troubleshooting section - - Security considerations and best practices - - Upgrade and rollback procedures - - Backup and restore scripts - -**Testing Results:** -- ✅ Docker Compose configuration validated (minor version field warning only) -- ✅ All required directories present (configs, data, output, logs) -- ✅ Healthcheck endpoint configured correctly -- ✅ Volume mounts properly configured for data persistence - -**Files Created:** -- .env.example (57 lines with detailed comments) -- docs/ai/DEPLOYMENT.md (650+ lines comprehensive guide) - -**Files Modified:** -- docker-compose-web.yml (added scheduler config, healthcheck, privileged mode, host networking) - -**Total:** 2 files created, 1 file modified, ~710 lines added - -**Key Implementation Details:** -- Healthcheck uses Python urllib to check /api/settings/health endpoint -- Privileged mode enables raw socket access for masscan/nmap -- Host networking mode provides unrestricted network access for scanning -- Scheduler configuration allows 2 concurrent executors with max 3 job instances -- All secrets configurable via .env file (not hardcoded) -- Production defaults set (FLASK_ENV=production, FLASK_DEBUG=false) - -**Deliverable:** ✅ Production-ready Docker deployment with comprehensive documentation - -### Step 7: Error Handling & Logging ✅ COMPLETE (Day 12) -**Priority: MEDIUM** - Robustness - -**Status:** ✅ Complete - -**Tasks Completed:** -1. ✅ Enhanced logging configuration: - - Implemented RotatingFileHandler (10MB per file, 10 backups) - - Separate error log file for ERROR level messages - - Structured log format with request IDs and timestamps - - RequestIDLogFilter for request context injection - - Console logging in debug mode -2. ✅ Request/response logging middleware: - - Request ID generation (UUID-based, 8 chars) - - Request timing with millisecond precision - - User authentication context in logs - - Response duration tracking - - Security headers (X-Content-Type-Options, X-Frame-Options, X-XSS-Protection) - - X-Request-ID and X-Request-Duration-Ms headers for API responses -3. ✅ Enhanced database error handling: - - SQLite WAL mode for better concurrency - - Busy timeout configuration (15 seconds) - - Automatic rollback on request exceptions - - SQLAlchemyError handler with explicit rollback - - Connection pooling with pre-ping -4. ✅ Comprehensive error handlers: - - Content negotiation (JSON for API, HTML for web) - - Error handlers for 400, 401, 403, 404, 405, 500 - - Database rollback in error handlers - - Full exception logging with traceback -5. ✅ Custom error templates: - - Created web/templates/errors/ directory - - 400.html, 401.html, 403.html, 404.html, 405.html, 500.html - - Dark theme matching application design - - Helpful error messages and navigation -6. ✅ Comprehensive tests: - - Created tests/test_error_handling.py (200+ lines) - - Tests for JSON vs HTML error responses - - Tests for request ID and duration headers - - Tests for security headers - - Tests for log rotation configuration - - Tests for structured logging - - Tests for error template rendering - -**Testing Results:** -- ✅ Error handlers support both JSON (API) and HTML (web) responses -- ✅ Request IDs tracked throughout request lifecycle -- ✅ Log rotation configured to prevent unbounded growth -- ✅ Database rollback on errors verified -- ✅ Custom error templates created and styled -- ✅ Security headers added to all API responses -- ✅ Comprehensive test suite created - -**Files Created:** -- web/templates/errors/400.html (70 lines) -- web/templates/errors/401.html (70 lines) -- web/templates/errors/403.html (70 lines) -- web/templates/errors/404.html (70 lines) -- web/templates/errors/405.html (70 lines) -- web/templates/errors/500.html (90 lines) -- tests/test_error_handling.py (320 lines) - -**Files Modified:** -- web/app.py (enhanced logging, error handlers, request handlers) - - Added RequestIDLogFilter class - - Enhanced configure_logging() with rotation - - Enhanced init_database() with WAL mode - - Enhanced register_error_handlers() with content negotiation - - Enhanced register_request_handlers() with timing and IDs - -**Total:** 7 files created, 1 file modified, ~760 lines added - -**Key Implementation Details:** -- Log files: sneakyscanner.log (INFO+), sneakyscanner_errors.log (ERROR only) -- Request IDs: 8-character UUID prefix for correlation -- WAL mode: Better SQLite concurrency for background jobs -- Content negotiation: Automatic JSON/HTML response selection -- Error templates: Consistent dark theme matching main UI - -**Deliverable:** ✅ Production-ready error handling and logging system - -### Step 8: Testing & Documentation ⏱️ Days 13-14 -**Priority: HIGH** - Quality assurance - -**Tasks:** -1. Write comprehensive tests: - - Unit tests for services (ScanService, SchedulerService) - - Integration tests for API endpoints - - End-to-end tests for workflows (login → scan → view → delete) -2. Create API documentation: - - Endpoint descriptions with request/response examples - - Authentication instructions - - Error code reference -3. Update README.md: - - Phase 2 features - - Installation instructions - - Configuration guide - - API usage examples -4. Create `PHASE2_COMPLETE.md` in `docs/ai/` -5. Update `docs/ai/ROADMAP.md` with completion status - -**Testing:** -- Run full test suite -- Achieve >80% code coverage -- Manual testing of all features -- Performance testing (multiple concurrent scans) - -**Deliverables:** -- Comprehensive test suite -- API documentation -- Updated user documentation -- Phase 2 completion summary - ---- - -## Dependencies & Prerequisites - -### Python Packages - -**Add to `requirements-web.txt`:** -``` -Flask-APScheduler==1.13.1 -``` - -**Already Present (from Phase 1):** -- Flask==3.0.0 -- Werkzeug==3.0.1 -- SQLAlchemy==2.0.23 -- alembic==1.13.0 -- Flask-Login==0.6.3 -- bcrypt==4.1.2 -- cryptography==41.0.7 -- Flask-CORS==4.0.0 -- marshmallow==3.20.1 -- marshmallow-sqlalchemy==0.29.0 -- APScheduler==3.10.4 -- Flask-Mail==0.9.1 -- python-dotenv==1.0.0 -- pytest==7.4.3 -- pytest-flask==1.3.0 - -### System Requirements - -- Python 3.12+ (development) -- Docker and Docker Compose (deployment) -- SQLite3 (database) -- Masscan, Nmap, Playwright (scanner dependencies - in Dockerfile) - -### Configuration Files - -**New file: `.env.example`** -```bash -# Flask configuration -FLASK_ENV=production -FLASK_DEBUG=false -FLASK_HOST=0.0.0.0 -FLASK_PORT=5000 - -# Database -DATABASE_URL=sqlite:////app/data/sneakyscanner.db - -# Security -SECRET_KEY=your-secret-key-here-change-in-production -SNEAKYSCANNER_ENCRYPTION_KEY=your-encryption-key-here - -# CORS (comma-separated origins) -CORS_ORIGINS=* - -# Logging -LOG_LEVEL=INFO - -# Scheduler -SCHEDULER_EXECUTORS=2 -SCHEDULER_JOB_DEFAULTS_MAX_INSTANCES=3 -``` - ---- - -## Testing Approach - -### Unit Tests - -**Framework:** pytest - -**Coverage Target:** >80% - -**Test Files:** -- `tests/test_scan_service.py` - ScanService methods -- `tests/test_scheduler_service.py` - SchedulerService methods -- `tests/test_validators.py` - Validation functions -- `tests/test_pagination.py` - Pagination helper - -**Approach:** -- Mock database calls with pytest fixtures -- Test each method independently -- Test edge cases and error conditions -- Use `@pytest.mark.parametrize` for multiple scenarios - -### Integration Tests - -**Test Files:** -- `tests/test_scan_api.py` - API endpoints with real database -- `tests/test_authentication.py` - Auth flow - -**Approach:** -- Use test database (separate from production) -- Test API endpoints end-to-end -- Verify database state after operations -- Test authentication required for protected routes - -### End-to-End Tests - -**Test Files:** -- `tests/test_workflows.py` - Complete user workflows - -**Scenarios:** -- Login → Trigger scan → View results → Delete scan → Logout -- API: Trigger scan → Poll status → Get results → Delete -- Background job execution → Database update → File creation - -**Approach:** -- Use pytest-flask for Flask testing -- Test both UI and API flows -- Verify files created/deleted -- Test concurrent operations - -### Manual Testing Checklist - -Phase 2 complete when all items checked: - -- [ ] Login with correct password succeeds -- [ ] Login with incorrect password fails -- [ ] Trigger scan via UI button works -- [ ] Trigger scan via API `POST /api/scans` works -- [ ] Scan runs in background (doesn't block) -- [ ] Scan status updates correctly (running → completed) -- [ ] View scan list with pagination -- [ ] View scan details page -- [ ] Delete scan removes DB records and files -- [ ] Logout destroys session -- [ ] Access protected route while logged out redirects to login -- [ ] Background scan completes successfully -- [ ] Background scan handles errors gracefully (e.g., invalid config) -- [ ] Multiple concurrent scans work -- [ ] Docker Compose deployment works -- [ ] Database persists across container restarts -- [ ] Scan files persist in volume - ---- - -## Potential Challenges & Solutions - -### Challenge 1: Scanner Integration with Background Jobs - -**Problem:** Scanner runs privileged operations (masscan/nmap require CAP_NET_RAW). How to execute from web app? - -**Impact:** High - core functionality - -**Solution:** -- Run scanner in subprocess with proper privileges -- Use Docker's `--privileged` mode and `--network host` -- Pass database URL to subprocess for status updates -- Isolate scanner execution errors from web app -- Use APScheduler's BackgroundScheduler (runs in threads) - -**Implementation:** -```python -# web/jobs/scan_job.py -def execute_scan(config_file, scan_id, db_url): - # Create new DB session for this thread - engine = create_engine(db_url) - Session = sessionmaker(bind=engine) - session = Session() - - try: - # Update status - scan = session.query(Scan).get(scan_id) - scan.status = 'running' - session.commit() - - # Run scanner (has privileged access via Docker) - scanner = SneakyScanner(config_file) - report, timestamp = scanner.scan() - scanner.generate_outputs(report, timestamp) - - # Save to DB - scan_service = ScanService(session) - scan_service._save_scan_to_db(report, scan_id) - except Exception as e: - scan.status = 'failed' - session.commit() - raise - finally: - session.close() -``` - -### Challenge 2: Database Concurrency - -**Problem:** Background jobs and web requests accessing SQLite simultaneously can cause locking issues. - -**Impact:** Medium - affects reliability - -**Solution:** -- Enable SQLite WAL (Write-Ahead Logging) mode for better concurrency -- Use SQLAlchemy scoped sessions (already configured) -- Add proper transaction handling and rollback -- Use connection pool (already configured in app.py) -- Consider timeout for busy database - -**Implementation:** -```python -# web/app.py - when creating engine -engine = create_engine( - app.config['SQLALCHEMY_DATABASE_URI'], - echo=app.debug, - pool_pre_ping=True, - pool_recycle=3600, - connect_args={'timeout': 15} # 15 second timeout for locks -) - -# Enable WAL mode for SQLite -if 'sqlite' in app.config['SQLALCHEMY_DATABASE_URI']: - @event.listens_for(engine, "connect") - def set_sqlite_pragma(dbapi_conn, connection_record): - cursor = dbapi_conn.cursor() - cursor.execute("PRAGMA journal_mode=WAL") - cursor.close() -``` - -### Challenge 3: Scan Status Tracking - -**Problem:** Need to show real-time progress for long-running scans (can take 5-10 minutes). - -**Impact:** Medium - affects UX - -**Solution:** -- Store scan status in DB (running, completed, failed) -- Implement polling endpoint `GET /api/scans//status` -- Update status at each scan phase (ping, TCP, UDP, services, HTTP) -- Dashboard polls every 5 seconds for running scans -- Consider WebSocket for real-time updates (Phase 3 enhancement) - -**Implementation:** -```javascript -// Dashboard JavaScript - poll for status -function pollScanStatus(scanId) { - const interval = setInterval(async () => { - const response = await fetch(`/api/scans/${scanId}/status`); - const data = await response.json(); - - if (data.status === 'completed' || data.status === 'failed') { - clearInterval(interval); - refreshScanList(); - } - - updateProgressBar(data.progress); - }, 5000); // Poll every 5 seconds -} -``` - -### Challenge 4: File Cleanup on Scan Deletion - -**Problem:** Must delete JSON, HTML, ZIP, screenshots when scan deleted. Missing files shouldn't cause errors. - -**Impact:** Medium - affects storage and cleanup - -**Solution:** -- Store all file paths in DB (already in schema) -- Implement cleanup in `ScanService.delete_scan()` -- Use pathlib for safe file operations -- Handle missing files gracefully (log warning, continue) -- Use database cascade deletion for related records -- Delete screenshot directory recursively - -**Implementation:** -```python -# web/services/scan_service.py -def delete_scan(self, scan_id): - scan = self.db.query(Scan).get(scan_id) - if not scan: - raise ValueError(f"Scan {scan_id} not found") - - # Delete files (handle missing gracefully) - for file_path in [scan.json_path, scan.html_path, scan.zip_path]: - if file_path: - try: - Path(file_path).unlink() - except FileNotFoundError: - logger.warning(f"File not found: {file_path}") - - # Delete screenshot directory - if scan.screenshot_dir: - try: - shutil.rmtree(scan.screenshot_dir) - except FileNotFoundError: - logger.warning(f"Directory not found: {scan.screenshot_dir}") - - # Delete DB record (cascade handles relationships) - self.db.delete(scan) - self.db.commit() -``` - -### Challenge 5: Authentication for API - -**Problem:** Need to protect API endpoints but also allow programmatic access. Session cookies don't work well for API clients. - -**Impact:** Medium - affects API usability - -**Solution:** -- Use Flask-Login sessions for both web UI and API in Phase 2 -- Require session cookie for API calls (works with curl -c/-b) -- Add `@api_auth_required` decorator to all endpoints -- Phase 5 will add token authentication for CLI client -- For now, document API usage with session cookies - -**Implementation:** -```python -# web/auth/decorators.py -from functools import wraps -from flask_login import current_user -from flask import jsonify - -def api_auth_required(f): - @wraps(f) - def decorated_function(*args, **kwargs): - if not current_user.is_authenticated: - return jsonify({'error': 'Authentication required'}), 401 - return f(*args, **kwargs) - return decorated_function - -# Usage in API -@bp.route('', methods=['POST']) -@api_auth_required -def trigger_scan(): - # Protected endpoint - pass -``` - -**API Usage Example:** -```bash -# Login first to get session cookie -curl -X POST http://localhost:5000/auth/login \ - -H "Content-Type: application/json" \ - -d '{"password":"yourpassword"}' \ - -c cookies.txt - -# Use cookie for API calls -curl -X POST http://localhost:5000/api/scans \ - -H "Content-Type: application/json" \ - -d '{"config_file":"/app/configs/example.yaml"}' \ - -b cookies.txt -``` - -### Challenge 6: Scanner Output Mapping to Database - -**Problem:** Complex nested JSON structure needs to map to normalized relational database schema. Many relationships to handle. - -**Impact:** High - core functionality - -**Solution:** -- Create comprehensive `_map_report_to_models()` method -- Process in order: Scan → Sites → IPs → Ports → Services → Certificates → TLS Versions -- Use SQLAlchemy relationships for automatic FK handling -- Batch operations within single transaction -- Add detailed error logging for mapping issues -- Handle missing/optional fields gracefully - -**Implementation Strategy:** -```python -def _map_report_to_models(self, report, scan_obj): - """Map JSON report to database models.""" - - # 1. Process sites - for site_data in report['sites']: - site = ScanSite( - scan_id=scan_obj.id, - site_name=site_data['name'] - ) - self.db.add(site) - self.db.flush() # Get site.id - - # 2. Process IPs for this site - for ip_data in site_data['ips']: - ip = ScanIP( - scan_id=scan_obj.id, - site_id=site.id, - ip_address=ip_data['address'], - ping_expected=ip_data['expected']['ping'], - ping_actual=ip_data['actual']['ping'] - ) - self.db.add(ip) - self.db.flush() - - # 3. Process ports for this IP - for port_data in ip_data['actual']['tcp_ports']: - port = ScanPort( - scan_id=scan_obj.id, - ip_id=ip.id, - port=port_data['port'], - protocol='tcp', - expected=port_data.get('expected', False), - state='open' - ) - self.db.add(port) - self.db.flush() - - # 4. Process services for this port - service_data = port_data.get('service') - if service_data: - service = ScanService( - scan_id=scan_obj.id, - port_id=port.id, - service_name=service_data.get('name'), - product=service_data.get('product'), - version=service_data.get('version'), - # ... more fields - ) - self.db.add(service) - self.db.flush() - - # 5. Process certificate if HTTPS - cert_data = service_data.get('http_info', {}).get('certificate') - if cert_data: - cert = ScanCertificate( - scan_id=scan_obj.id, - service_id=service.id, - # ... cert fields - ) - self.db.add(cert) - self.db.flush() - - # 6. Process TLS versions - for tls_data in cert_data.get('tls_versions', []): - tls = ScanTLSVersion( - scan_id=scan_obj.id, - certificate_id=cert.id, - # ... tls fields - ) - self.db.add(tls) - - # Commit entire transaction - self.db.commit() -``` - -### Challenge 7: Long-Running Scans Timeout - -**Problem:** HTTP request might timeout during long scans (5-10 minutes). Browser/client gives up waiting. - -**Impact:** High - affects UX - -**Solution:** -- Queue scan job immediately -- Return scan_id right away (within seconds) -- Client polls `GET /api/scans//status` for progress -- Store scan status and progress in database -- Background job runs independently of HTTP connection -- Dashboard auto-refreshes scan list - -**Flow:** -1. User clicks "Run Scan" button -2. POST /api/scans → creates Scan record, queues job, returns scan_id -3. JavaScript starts polling status endpoint every 5 seconds -4. Background job runs scanner, updates status in DB -5. When status changes to 'completed', stop polling and refresh scan list - ---- - -## Success Criteria - -Phase 2 is **COMPLETE** when all criteria are met: - -### API Functionality -- [ ] `POST /api/scans` triggers background scan and returns scan_id -- [ ] `GET /api/scans` lists scans with pagination (page, per_page params) -- [ ] `GET /api/scans/` returns full scan details from database -- [ ] `DELETE /api/scans/` removes scan records and files -- [ ] `GET /api/scans//status` shows current scan progress - -### Database Integration -- [ ] Scan results automatically saved to database after completion -- [ ] All relationships populated correctly (sites, IPs, ports, services, certs, TLS) -- [ ] Database queries work efficiently (indexes in place) -- [ ] Cascade deletion works for related records - -### Background Jobs -- [ ] Scans execute in background (don't block HTTP requests) -- [ ] Multiple scans can run concurrently -- [ ] Scan status updates correctly (running → completed/failed) -- [ ] Failed scans marked appropriately with error message - -### Authentication -- [ ] Login page renders and accepts password -- [ ] Successful login creates session and redirects to dashboard -- [ ] Invalid password shows error message -- [ ] Logout destroys session -- [ ] Protected routes require authentication -- [ ] API endpoints require authentication - -### User Interface -- [ ] Dashboard displays welcome message and stats -- [ ] Dashboard shows recent scans in table -- [ ] "Run Scan Now" button triggers scan -- [ ] Login page has clean design -- [ ] Templates use Bootstrap 5 dark theme -- [ ] Navigation works between pages - -### File Management -- [ ] JSON, HTML, ZIP files still generated (backward compatible) -- [ ] Screenshot directory created with images -- [ ] Files referenced correctly in database -- [ ] Delete scan removes all associated files - -### Deployment -- [ ] Docker Compose starts web app successfully -- [ ] Database persists across container restarts -- [ ] Scan files persist in mounted volume -- [ ] Healthcheck endpoint responds correctly -- [ ] Logs written to volume - -### Testing -- [ ] All unit tests pass -- [ ] All integration tests pass -- [ ] Test coverage >80% -- [ ] Manual testing checklist complete - -### Documentation -- [ ] API endpoints documented with examples -- [ ] README.md updated with Phase 2 features -- [ ] PHASE2_COMPLETE.md created -- [ ] ROADMAP.md updated - ---- - -## Migration Path - -### From Phase 1 to Phase 2 - -**No Breaking Changes:** -- Database schema already complete (Phase 1) -- Existing `scanner.py` code unchanged (backward compatible) -- YAML config format unchanged -- JSON/HTML/ZIP output format unchanged -- Docker deployment configuration compatible - -**Additions:** -- New API endpoint implementations (replace stubs) -- New service layer (ScanService, SchedulerService) -- New authentication system -- New UI templates -- Background job system - -**Migration Steps:** -1. Pull latest code -2. Install new dependency: `pip install Flask-APScheduler` -3. Run new Alembic migration: `alembic upgrade head` -4. Set application password if not set: `python3 init_db.py --password YOUR_PASSWORD` -5. Rebuild Docker image: `docker-compose -f docker-compose-web.yml build` -6. Start services: `docker-compose -f docker-compose-web.yml up -d` - -### Backward Compatibility - -**CLI Scanner:** -- Continues to work standalone: `python3 src/scanner.py configs/example.yaml` -- Still generates JSON/HTML/ZIP files -- No changes to command-line interface - -**Existing Scans:** -- Old scan JSON files not automatically imported to database -- Can be imported manually if needed (not in Phase 2 scope) -- New scans saved to both files and database - -**Configuration:** -- Existing YAML configs work without modification -- Settings from Phase 1 preserved -- No config changes required - ---- - -## Estimated Timeline - -**Total Duration:** 14 working days (2 weeks) - -### Week 1: Backend Foundation -- **Days 1-2:** Database & Service Layer - - ScanService implementation - - Database mapping logic - - Unit tests - -- **Days 3-4:** Scan API Endpoints - - All 5 endpoints implemented - - Input validation - - Integration tests - -- **Days 5-6:** Background Job Queue - - APScheduler integration - - Job execution logic - - Concurrent scan testing - -- **Day 7:** Authentication System (Part 1) - - Flask-Login setup - - User model - - Login/logout routes - -### Week 2: Frontend & Polish -- **Day 8:** Authentication System (Part 2) - - Decorators - - Apply to all endpoints - - Authentication tests - -- **Days 9-10:** Basic UI Templates - - Base template - - Login page - - Dashboard - - Web routes - -- **Day 11:** Docker & Deployment - - Docker Compose updates - - Deployment testing - - Production configuration - -- **Day 12:** Error Handling & Logging - - Error pages - - Logging enhancements - - Error scenarios testing - -- **Days 13-14:** Testing & Documentation - - Complete test suite - - API documentation - - README updates - - PHASE2_COMPLETE.md - -### Critical Path -The critical path (tasks that must complete before others): -1. Service Layer (Days 1-2) → Everything depends on this -2. API Endpoints (Days 3-4) → Required for UI and background jobs -3. Background Jobs (Days 5-6) → Required for async scan execution -4. Authentication (Days 7-8) → Required for security - -UI templates and documentation can proceed in parallel after Day 8. - ---- - -## Key Design Decisions - -### Decision 1: Background Job Processing - -**Choice:** APScheduler BackgroundScheduler - -**Alternatives Considered:** -- Celery with Redis/RabbitMQ -- Python threading module -- Subprocess with cron - -**Rationale:** -- APScheduler is lightweight (no external dependencies) -- BackgroundScheduler runs in threads (simple, no message broker needed) -- Sufficient for single-user application -- Can handle concurrent scans -- Easy to integrate with Flask -- Meets all Phase 2 requirements - -**Trade-offs:** -- ✅ Simple deployment (no Redis needed) -- ✅ Low resource usage -- ✅ Built-in job scheduling -- ❌ Less scalable than Celery (but not needed for single-user) -- ❌ Jobs lost if app crashes (acceptable for this use case) - -### Decision 2: Authentication System - -**Choice:** Flask-Login with single-user password - -**Alternatives Considered:** -- Multi-user with SQLite user table -- JWT tokens -- Basic HTTP auth -- No authentication - -**Rationale:** -- Simple and meets requirements (single-user, self-hosted) -- Flask-Login is well-maintained and integrated with Flask -- Session-based auth works for both UI and API -- Sufficient security for local/internal deployment -- Easy to implement and test - -**Trade-offs:** -- ✅ Simple implementation -- ✅ Works for UI and API -- ✅ Secure (bcrypt password hashing) -- ❌ Not suitable for multi-user (not a requirement) -- ❌ Session cookies don't work well for CLI clients (Phase 5 adds tokens) - -### Decision 3: Database Storage Strategy - -**Choice:** Store complete scan results in normalized database schema - -**Alternatives Considered:** -- Store only metadata, keep JSON files for details -- Store JSON blob in database -- Hybrid approach (metadata in DB, details in files) - -**Rationale:** -- Enables powerful queries (find all scans with cert expiring in 30 days) -- Required for trending and comparison features (Phase 4) -- Normalized schema is more flexible for future features -- Small storage overhead acceptable (scans are small) -- Still generate JSON/HTML/ZIP for backward compatibility - -**Trade-offs:** -- ✅ Enables advanced queries -- ✅ Required for Phase 3-4 features -- ✅ Clean separation of concerns -- ❌ More complex mapping logic -- ❌ Slightly larger database size (minimal impact) - -### Decision 4: Scanner Execution Model - -**Choice:** Execute scanner in subprocess from web app - -**Alternatives Considered:** -- Refactor scanner into library (import directly) -- Separate scanner service (microservice) -- CLI wrapper - -**Rationale:** -- Maintains isolation (scanner errors don't crash web app) -- Reuses existing scanner code (no refactoring needed) -- Handles privileged operations via Docker -- Simple to implement -- Backward compatible with CLI usage - -**Trade-offs:** -- ✅ Isolation and stability -- ✅ Reuses existing code -- ✅ Backward compatible -- ❌ Slightly more overhead than library import (minimal) -- ❌ Inter-process communication needed (solved with DB) - -### Decision 5: API Authentication (Phase 2) - -**Choice:** Session-based authentication via Flask-Login - -**Alternatives Considered:** -- API tokens (Bearer authentication) -- OAuth2 -- No authentication for API, only UI - -**Rationale:** -- Consistent with web UI authentication -- Simple to implement (already using Flask-Login) -- Works for testing and initial API usage -- Phase 5 will add token authentication for CLI client -- Secure enough for single-user self-hosted deployment - -**Trade-offs:** -- ✅ Consistent with UI auth -- ✅ Simple implementation -- ✅ Secure for intended use case -- ❌ Not ideal for programmatic access (Phase 5 improvement) -- ❌ Requires cookie management in API clients - -**API Usage Pattern (Phase 2):** -```bash -# Login to get session cookie -curl -X POST http://localhost:5000/auth/login \ - -d '{"password":"yourpass"}' \ - -c cookies.txt - -# Use session cookie for API calls -curl -X GET http://localhost:5000/api/scans \ - -b cookies.txt -``` - ---- - -## Documentation Deliverables - -### 1. API Documentation (`docs/ai/API_REFERENCE.md`) - -**Contents:** -- Endpoint reference (all 5 scan endpoints) -- Request/response examples -- Authentication instructions -- Error codes and messages -- Pagination parameters -- Status codes - -**Example:** -```markdown -## POST /api/scans - -Trigger a new scan. - -**Authentication:** Required (session cookie) - -**Request Body:** -```json -{ - "config_file": "/app/configs/example.yaml" -} -``` - -**Response (201 Created):** -```json -{ - "scan_id": 42, - "status": "running", - "message": "Scan queued successfully" -} -``` -``` - -### 2. Deployment Guide (`docs/ai/DEPLOYMENT.md`) - -**Contents:** -- Docker Compose setup -- Environment variables -- Volume configuration -- Database initialization -- First-time setup -- Troubleshooting - -### 3. Developer Guide (`docs/ai/DEVELOPMENT.md`) - -**Contents:** -- Project structure -- Architecture overview -- Database schema -- Service layer design -- Adding new features -- Running tests -- Code style guide - -### 4. User Guide (`README.md` updates) - -**Contents:** -- Phase 2 features -- Web UI usage -- API usage -- Configuration -- Common tasks -- FAQ - -### 5. Phase 2 Completion Summary (`docs/ai/PHASE2_COMPLETE.md`) - -**Contents:** -- What was delivered -- Success criteria checklist -- Known limitations -- Next steps (Phase 3) -- Migration instructions -- Testing results - ---- - -## Appendix: Example API Calls - -### Authentication - -```bash -# Login -curl -X POST http://localhost:5000/auth/login \ - -H "Content-Type: application/json" \ - -d '{"password":"yourpassword"}' \ - -c cookies.txt - -# Logout -curl -X GET http://localhost:5000/auth/logout \ - -b cookies.txt -``` - -### Scan Management - -```bash -# Trigger scan -curl -X POST http://localhost:5000/api/scans \ - -H "Content-Type: application/json" \ - -d '{"config_file":"/app/configs/example.yaml"}' \ - -b cookies.txt - -# List scans (with pagination) -curl -X GET "http://localhost:5000/api/scans?page=1&per_page=20" \ - -b cookies.txt - -# Get scan details -curl -X GET http://localhost:5000/api/scans/42 \ - -b cookies.txt - -# Get scan status -curl -X GET http://localhost:5000/api/scans/42/status \ - -b cookies.txt - -# Delete scan -curl -X DELETE http://localhost:5000/api/scans/42 \ - -b cookies.txt -``` - -### Example Responses - -**GET /api/scans/42:** -```json -{ - "id": 42, - "timestamp": "2025-11-14T10:30:00Z", - "duration": 125.5, - "status": "completed", - "title": "Production Network Scan", - "config_file": "/app/configs/production.yaml", - "json_path": "/app/output/scan_report_20251114_103000.json", - "html_path": "/app/output/scan_report_20251114_103000.html", - "zip_path": "/app/output/scan_report_20251114_103000.zip", - "screenshot_dir": "/app/output/scan_report_20251114_103000_screenshots", - "triggered_by": "api", - "sites": [ - { - "id": 101, - "name": "Production DC", - "ips": [ - { - "id": 201, - "address": "192.168.1.10", - "ping_expected": true, - "ping_actual": true, - "ports": [ - { - "id": 301, - "port": 443, - "protocol": "tcp", - "state": "open", - "expected": true, - "services": [ - { - "id": 401, - "service_name": "https", - "product": "nginx", - "version": "1.24.0", - "http_protocol": "https", - "screenshot_path": "scan_report_20251114_103000_screenshots/192_168_1_10_443.png" - } - ] - } - ] - } - ] - } - ] -} -``` - -**GET /api/scans/42/status:** -```json -{ - "scan_id": 42, - "status": "running", - "progress": 60, - "current_phase": "service_detection", - "started_at": "2025-11-14T10:30:00Z" -} -``` - ---- - -**End of Phase 2 Plan** - -This plan will be followed during Phase 2 implementation. Upon completion, a new document `PHASE2_COMPLETE.md` will summarize actual implementation, challenges encountered, and lessons learned. diff --git a/docs/ai/PHASE2_COMPLETE.md b/docs/ai/PHASE2_COMPLETE.md deleted file mode 100644 index 15f897c..0000000 --- a/docs/ai/PHASE2_COMPLETE.md +++ /dev/null @@ -1,872 +0,0 @@ -# Phase 2: Flask Web App Core - COMPLETE ✓ - -**Date Completed:** 2025-11-14 -**Duration:** 14 days (2 weeks) -**Lines of Code Added:** ~4,500+ lines across backend, frontend, tests, and documentation - -Phase 2 of the SneakyScanner roadmap has been successfully implemented. This document summarizes what was delivered, how to use the new features, and lessons learned. - ---- - -## ✓ Success Criteria Met - -All success criteria from [PHASE2.md](PHASE2.md) have been achieved: - -### API Functionality ✅ -- ✅ `POST /api/scans` triggers background scan and returns scan_id -- ✅ `GET /api/scans` lists scans with pagination (page, per_page params) -- ✅ `GET /api/scans/` returns full scan details from database -- ✅ `DELETE /api/scans/` removes scan records and files -- ✅ `GET /api/scans//status` shows current scan progress - -### Database Integration ✅ -- ✅ Scan results automatically saved to database after completion -- ✅ All relationships populated correctly (sites, IPs, ports, services, certs, TLS) -- ✅ Database queries work efficiently (indexes in place) -- ✅ Cascade deletion works for related records - -### Background Jobs ✅ -- ✅ Scans execute in background (don't block HTTP requests) -- ✅ Multiple scans can run concurrently (configurable: 3 concurrent jobs) -- ✅ Scan status updates correctly (running → completed/failed) -- ✅ Failed scans marked appropriately with error message - -### Authentication ✅ -- ✅ Login page renders and accepts password -- ✅ Successful login creates session and redirects to dashboard -- ✅ Invalid password shows error message -- ✅ Logout destroys session -- ✅ Protected routes require authentication -- ✅ API endpoints require authentication - -### User Interface ✅ -- ✅ Dashboard displays welcome message and stats -- ✅ Dashboard shows recent scans in table -- ✅ Login page has clean design -- ✅ Templates use Bootstrap 5 dark theme (matching report style) -- ✅ Navigation works between pages -- ✅ Error pages for 400, 401, 403, 404, 405, 500 - -### File Management ✅ -- ✅ JSON, HTML, ZIP files still generated (backward compatible) -- ✅ Screenshot directory created with images -- ✅ Files referenced correctly in database -- ✅ Delete scan removes all associated files - -### Deployment ✅ -- ✅ Docker Compose starts web app successfully -- ✅ Database persists across container restarts -- ✅ Scan files persist in mounted volume -- ✅ Healthcheck endpoint responds correctly (`/api/settings/health`) -- ✅ Logs written to volume with rotation (10MB max, 10 backups) - -### Testing ✅ -- ✅ 100 test functions across 6 test files -- ✅ 1,825 lines of test code -- ✅ All tests passing (service layer, API, auth, error handling, background jobs) -- ✅ Comprehensive test coverage - -### Documentation ✅ -- ✅ API endpoints documented with examples (API_REFERENCE.md) -- ✅ README.md updated with Phase 2 features -- ✅ PHASE2_COMPLETE.md created (this document) -- ✅ ROADMAP.md updated -- ✅ DEPLOYMENT.md comprehensive deployment guide - ---- - -## 📦 Deliverables by Step - -### Step 1: Database & Service Layer ✅ -**Completed:** Day 2 - -**Files Created:** -- `web/services/__init__.py` -- `web/services/scan_service.py` (545 lines) - Core business logic for scan CRUD operations -- `web/utils/pagination.py` (153 lines) - Pagination utility with metadata -- `web/utils/validators.py` (245 lines) - Input validation functions -- `migrations/versions/002_add_scan_indexes.py` - Database indexes for performance -- `tests/conftest.py` (142 lines) - Pytest fixtures and configuration -- `tests/test_scan_service.py` (374 lines) - 15 unit tests - -**Key Features:** -- ScanService with full CRUD operations (`trigger_scan`, `get_scan`, `list_scans`, `delete_scan`, `get_scan_status`) -- Complex JSON-to-database mapping (`_map_report_to_models`) -- Validation for config files, scan IDs, ports, IP addresses -- Pagination helper with metadata (total, pages, current page) -- All 15 tests passing - -### Step 2: Scan API Endpoints ✅ -**Completed:** Day 4 - -**Files Modified:** -- `web/api/scans.py` (262 lines) - All 5 endpoints fully implemented - -**Files Created:** -- `tests/test_scan_api.py` (301 lines) - 24 integration tests - -**Key Features:** -- All endpoints with comprehensive error handling -- Input validation through validators -- Proper HTTP status codes (200, 201, 400, 404, 500) -- Structured logging with request details -- Pagination support with query parameters -- Status filtering (`?status=running|completed|failed`) -- All 24 tests passing - -### Step 3: Background Job Queue ✅ -**Completed:** Day 6 - -**Files Created:** -- `web/jobs/__init__.py` -- `web/jobs/scan_job.py` (130 lines) - Background scan execution -- `web/services/scheduler_service.py` (220 lines) - APScheduler integration -- `migrations/versions/003_add_scan_timing_fields.py` - Timing fields (started_at, completed_at, error_message) -- `tests/test_background_jobs.py` (232 lines) - 13 unit tests - -**Files Modified:** -- `web/app.py` - Scheduler initialization -- `web/models.py` - Added timing fields to Scan model -- `web/services/scan_service.py` - Updated for scheduler integration -- `web/api/scans.py` - Pass scheduler to trigger_scan - -**Key Features:** -- BackgroundScheduler with ThreadPoolExecutor (max 3 workers) -- Isolated database sessions per thread -- Status tracking through lifecycle (created → running → completed/failed) -- Error message capture and storage -- Graceful shutdown handling -- All 13 tests passing - -### Step 4: Authentication System ✅ -**Completed:** Day 8 - -**Files Created:** -- `web/auth/__init__.py` -- `web/auth/routes.py` (85 lines) - Login/logout routes -- `web/auth/decorators.py` (62 lines) - @login_required and @api_auth_required -- `web/auth/models.py` (48 lines) - User class for Flask-Login -- `web/templates/login.html` (95 lines) - Login page with dark theme -- `tests/test_authentication.py` (279 lines) - 30+ authentication tests - -**Files Modified:** -- `web/app.py` - Flask-Login integration, user_loader callback -- All API endpoints - Protected with @api_auth_required -- All web routes - Protected with @login_required - -**Key Features:** -- Flask-Login session management -- Single-user authentication with bcrypt password hashing -- Session-based auth for both UI and API -- Login/logout functionality -- Password setup on first run -- All 30+ tests passing - -### Step 5: Basic UI Templates ✅ -**Completed:** Day 10 - -**Files Created:** -- `web/templates/base.html` (120 lines) - Base layout with Bootstrap 5 dark theme -- `web/templates/dashboard.html` (180 lines) - Dashboard with stats and recent scans -- `web/templates/scans.html` (240 lines) - Scan list with pagination -- `web/templates/scan_detail.html` (320 lines) - Detailed scan results view -- `web/routes/__init__.py` -- `web/routes/main.py` (150 lines) - Web UI routes -- `web/static/css/custom.css` (85 lines) - Custom dark theme styles -- `web/static/js/dashboard.js` (120 lines) - AJAX and auto-refresh - -**Key Features:** -- Consistent dark theme matching HTML reports (slate/grey color scheme) -- Navigation bar (Dashboard, Scans, Settings, Logout) -- Flash message display -- AJAX-powered dynamic data loading -- Auto-refresh for running scans (5-second polling) -- Responsive design with Bootstrap 5 -- Pagination controls - -### Step 6: Docker & Deployment ✅ -**Completed:** Day 11 - -**Files Created:** -- `.env.example` (57 lines) - Comprehensive environment template -- `docs/ai/DEPLOYMENT.md` (650+ lines) - Complete deployment guide - -**Files Modified:** -- `docker-compose-web.yml` - Scheduler config, healthcheck, privileged mode, host networking - -**Key Features:** -- Healthcheck endpoint monitoring (30s interval, 10s timeout) -- Privileged mode for scanner raw socket access -- Host networking for unrestricted network scanning -- Environment variable configuration (SECRET_KEY, ENCRYPTION_KEY, scheduler settings) -- Volume mounts for data persistence (data, output, logs, configs) -- Production defaults (FLASK_ENV=production) -- Comprehensive deployment documentation - -### Step 7: Error Handling & Logging ✅ -**Completed:** Day 12 - -**Files Created:** -- `web/templates/errors/400.html` (70 lines) -- `web/templates/errors/401.html` (70 lines) -- `web/templates/errors/403.html` (70 lines) -- `web/templates/errors/404.html` (70 lines) -- `web/templates/errors/405.html` (70 lines) -- `web/templates/errors/500.html` (90 lines) -- `tests/test_error_handling.py` (320 lines) - Comprehensive error handling tests - -**Files Modified:** -- `web/app.py` - Enhanced logging, error handlers, request handlers - -**Key Features:** -- RotatingFileHandler (10MB per file, 10 backups) -- Separate error log file for ERROR level messages -- RequestIDLogFilter for request context injection -- Request timing with millisecond precision -- Content negotiation (JSON for API, HTML for web) -- SQLite WAL mode for better concurrency -- Security headers (X-Content-Type-Options, X-Frame-Options, X-XSS-Protection) -- Request IDs in logs and headers (X-Request-ID, X-Request-Duration-Ms) - -### Step 8: Testing & Documentation ✅ -**Completed:** Day 14 - -**Files Created:** -- `docs/ai/API_REFERENCE.md` (650+ lines) - Complete API documentation -- `docs/ai/PHASE2_COMPLETE.md` (this document) -- `docs/ai/MANUAL_TESTING.md` - Manual testing checklist - -**Files Modified:** -- `README.md` - Comprehensive update with Phase 2 features -- `docs/ai/ROADMAP.md` - Updated with Phase 2 completion - -**Documentation Deliverables:** -- API reference with request/response examples -- Updated README with web application features -- Phase 2 completion summary -- Manual testing checklist -- Updated roadmap - ---- - -## 📊 Statistics - -### Code Metrics - -| Category | Files | Lines of Code | -|----------|-------|---------------| -| Backend Services | 3 | 965 | -| API Endpoints | 1 (modified) | 262 | -| Background Jobs | 2 | 350 | -| Authentication | 3 | 195 | -| Web UI Templates | 11 | 1,440 | -| Utilities | 2 | 398 | -| Database Migrations | 2 | 76 | -| Tests | 6 | 1,825 | -| Documentation | 4 | 2,000+ | -| **Total** | **34** | **~7,500+** | - -### Test Coverage - -- **Test Files:** 6 -- **Test Functions:** 100 -- **Lines of Test Code:** 1,825 -- **Coverage Areas:** - - Service layer (ScanService, SchedulerService) - - API endpoints (all 5 scan endpoints) - - Authentication (login, logout, decorators) - - Background jobs (scheduler, job execution, timing) - - Error handling (all HTTP status codes, content negotiation) - - Pagination and validation - -### Database Schema - -- **Tables:** 11 (no changes from Phase 1) -- **Migrations:** 3 total - - `001_initial_schema.py` (Phase 1) - - `002_add_scan_indexes.py` (Step 1) - - `003_add_scan_timing_fields.py` (Step 3) -- **Indexes:** Status index for efficient filtering -- **Mode:** SQLite WAL for better concurrency - ---- - -## 🎯 Key Accomplishments - -### 1. Complete REST API for Scan Management - -All CRUD operations implemented with comprehensive error handling: - -```bash -# Trigger scan -POST /api/scans -{"config_file": "/app/configs/example.yaml"} -→ {"scan_id": 42, "status": "running"} - -# List scans (paginated) -GET /api/scans?page=1&per_page=20&status=completed -→ {"scans": [...], "total": 42, "page": 1, "pages": 3} - -# Get scan details -GET /api/scans/42 -→ {full scan with all relationships} - -# Poll status -GET /api/scans/42/status -→ {"status": "running", "started_at": "...", "completed_at": null} - -# Delete scan -DELETE /api/scans/42 -→ {"message": "Scan 42 deleted successfully"} -``` - -### 2. Asynchronous Scan Execution - -Scans run in background threads without blocking HTTP requests: - -- APScheduler BackgroundScheduler with ThreadPoolExecutor -- Up to 3 concurrent scans (configurable) -- Isolated database sessions per thread -- Status tracking: `running` → `completed`/`failed` -- Error capture and storage - -**Result:** Web UI remains responsive during long-running scans (2-10 minutes) - -### 3. Complete Database Integration - -Complex JSON scan reports mapped to normalized relational schema: - -- **Hierarchy:** Scan → Sites → IPs → Ports → Services → Certificates → TLS Versions -- **Relationships:** Proper foreign keys and cascade deletion -- **Efficient Queries:** Indexes on status, timestamp -- **Concurrency:** SQLite WAL mode for multiple readers/writers - -**Result:** All scan data queryable in database for future trend analysis - -### 4. Secure Authentication System - -Single-user authentication with Flask-Login: - -- Session-based auth for both UI and API -- Bcrypt password hashing (cost factor 12) -- Protected routes with decorators -- Login/logout functionality -- Password setup on first run - -**Result:** Secure access control for all features - -### 5. Production-Ready Deployment - -Complete Docker deployment with persistent data: - -- Docker Compose configuration with healthcheck -- Privileged mode for scanner operations -- Environment-based configuration -- Volume mounts for data persistence -- Comprehensive deployment documentation - -**Result:** Easy deployment with `docker-compose up` - -### 6. Comprehensive Error Handling - -Robust error handling and logging: - -- Content negotiation (JSON for API, HTML for web) -- Custom error templates (400, 401, 403, 404, 405, 500) -- Structured logging with request IDs -- Log rotation (10MB files, 10 backups) -- Request timing and duration tracking - -**Result:** Production-ready error handling and debugging - -### 7. Extensive Test Coverage - -Comprehensive test suite: - -- 100 test functions across 6 test files -- 1,825 lines of test code -- All major components tested -- Integration tests for complete workflows -- All tests passing - -**Result:** High confidence in code quality and reliability - ---- - -## 🔧 Technical Implementation Details - -### Service Layer Architecture - -**ScanService** (`web/services/scan_service.py`) - 545 lines: -- `trigger_scan(config_file, triggered_by, schedule_id)` - Create scan record and queue job -- `get_scan(scan_id)` - Retrieve complete scan with all relationships (eager loading) -- `list_scans(page, per_page, status_filter)` - Paginated list with filtering -- `delete_scan(scan_id)` - Remove DB records and files (JSON, HTML, ZIP, screenshots) -- `get_scan_status(scan_id)` - Poll scan status for real-time updates -- `_save_scan_to_db(report, scan_id, status)` - Persist scan results -- `_map_report_to_models(report, scan_obj)` - Complex JSON→DB mapping - -**SchedulerService** (`web/services/scheduler_service.py`) - 220 lines: -- `init_scheduler(app)` - Initialize APScheduler -- `queue_scan(config_file, scan_id, db_url)` - Queue immediate scan execution -- `add_scheduled_scan(schedule)` - Placeholder for Phase 3 scheduled scans -- `remove_scheduled_scan(schedule_id)` - Remove scheduled jobs -- `list_jobs()` - List all scheduler jobs -- `shutdown()` - Graceful shutdown - -### Background Job Execution - -**Scan Job** (`web/jobs/scan_job.py`) - 130 lines: - -```python -def execute_scan(config_file, scan_id, db_url): - """Execute scan in background thread.""" - # 1. Create isolated DB session - engine = create_engine(db_url) - Session = sessionmaker(bind=engine) - session = Session() - - try: - # 2. Update status to running - scan = session.query(Scan).get(scan_id) - scan.status = 'running' - scan.started_at = datetime.utcnow() - session.commit() - - # 3. Run scanner - scanner = SneakyScanner(config_file) - report, timestamp = scanner.scan() - scanner.generate_outputs(report, timestamp) - - # 4. Save to database - scan_service = ScanService(session) - scan_service._save_scan_to_db(report, scan_id, status='completed') - - # 5. Update timing - scan.completed_at = datetime.utcnow() - session.commit() - - except Exception as e: - # 6. Mark as failed - scan.status = 'failed' - scan.error_message = str(e) - scan.completed_at = datetime.utcnow() - session.commit() - logger.error(f"Scan {scan_id} failed: {e}") - - finally: - session.close() -``` - -### Database Mapping Strategy - -Complex JSON structure mapped to normalized schema in specific order: - -1. **Scan** - Top-level metadata -2. **Sites** - Logical grouping from config -3. **IPs** - IP addresses per site -4. **Ports** - Open ports per IP -5. **Services** - Service detection per port -6. **Certificates** - SSL/TLS certs per HTTPS service -7. **TLS Versions** - TLS version support per certificate - -**Key Technique:** Use `session.flush()` after each level to generate IDs for foreign keys - -### Authentication Flow - -``` -┌──────────────────────────────────────┐ -│ 1. User visits /dashboard │ -│ (not authenticated) │ -└───────────┬──────────────────────────┘ - │ - ▼ -┌──────────────────────────────────────┐ -│ 2. @login_required redirects to │ -│ /login │ -└───────────┬──────────────────────────┘ - │ - ▼ -┌──────────────────────────────────────┐ -│ 3. User enters password │ -│ POST /auth/login │ -└───────────┬──────────────────────────┘ - │ - ▼ -┌──────────────────────────────────────┐ -│ 4. Verify password (bcrypt) │ -│ - Load password from settings │ -│ - Check with bcrypt.checkpw() │ -└───────────┬──────────────────────────┘ - │ - ▼ -┌──────────────────────────────────────┐ -│ 5. Create Flask-Login session │ -│ login_user(user) │ -└───────────┬──────────────────────────┘ - │ - ▼ -┌──────────────────────────────────────┐ -│ 6. Redirect to /dashboard │ -│ (authenticated, can access) │ -└──────────────────────────────────────┘ -``` - -### Error Handling Architecture - -**Content Negotiation:** -```python -def render_error(status_code, error_type, message): - """Render error as JSON or HTML based on request.""" - # Check if JSON response expected - if request.path.startswith('/api/') or \ - request.accept_mimetypes.best == 'application/json': - return jsonify({ - 'error': error_type, - 'message': message - }), status_code - - # Otherwise return HTML error page - return render_template(f'errors/{status_code}.html', - error=error_type, - message=message), status_code -``` - -**Request ID Tracking:** -```python -@app.before_request -def before_request(): - """Add request ID and start timing.""" - request.id = uuid.uuid4().hex[:8] - request.start_time = time.time() - -@app.after_request -def after_request(response): - """Add timing and request ID headers.""" - duration_ms = int((time.time() - request.start_time) * 1000) - response.headers['X-Request-ID'] = request.id - response.headers['X-Request-Duration-Ms'] = str(duration_ms) - return response -``` - ---- - -## 📚 API Endpoints Reference - -See [API_REFERENCE.md](API_REFERENCE.md) for complete documentation. - -### Scans - -| Method | Endpoint | Description | -|--------|----------|-------------| -| POST | `/api/scans` | Trigger new scan | -| GET | `/api/scans` | List scans (paginated, filterable) | -| GET | `/api/scans/{id}` | Get scan details | -| GET | `/api/scans/{id}/status` | Get scan status | -| DELETE | `/api/scans/{id}` | Delete scan and files | - -### Authentication - -| Method | Endpoint | Description | -|--------|----------|-------------| -| POST | `/auth/login` | Login and create session | -| GET | `/auth/logout` | Logout and destroy session | - -### Settings - -| Method | Endpoint | Description | -|--------|----------|-------------| -| GET | `/api/settings` | Get all settings | -| PUT | `/api/settings/{key}` | Update setting | -| GET | `/api/settings/health` | Health check | - -### Web UI - -| Method | Route | Description | -|--------|-------|-------------| -| GET | `/` | Redirect to dashboard | -| GET | `/login` | Login page | -| GET | `/dashboard` | Dashboard with stats | -| GET | `/scans` | Browse scan history | -| GET | `/scans/` | View scan details | - ---- - -## 🚀 Getting Started - -### Quick Start (Docker) - -1. **Clone repository:** -```bash -git clone https://github.com/yourusername/sneakyscanner.git -cd sneakyscanner -``` - -2. **Configure environment:** -```bash -cp .env.example .env -# Edit .env and set SECRET_KEY and SNEAKYSCANNER_ENCRYPTION_KEY -``` - -3. **Start web application:** -```bash -docker-compose -f docker-compose-web.yml up -d -``` - -4. **Access web interface:** -- Open http://localhost:5000 -- Default password: `admin` (change immediately!) - -5. **Trigger first scan:** -- Click "Run Scan Now" on dashboard -- Or use API: -```bash -curl -X POST http://localhost:5000/api/scans \ - -H "Content-Type: application/json" \ - -d '{"config_file":"/app/configs/example-site.yaml"}' \ - -b cookies.txt -``` - -See [DEPLOYMENT.md](DEPLOYMENT.md) for detailed setup instructions. - -### API Usage Example - -```bash -#!/bin/bash - -# 1. Login -curl -X POST http://localhost:5000/auth/login \ - -H "Content-Type: application/json" \ - -d '{"password":"yourpassword"}' \ - -c cookies.txt - -# 2. Trigger scan -SCAN_ID=$(curl -s -X POST http://localhost:5000/api/scans \ - -H "Content-Type: application/json" \ - -d '{"config_file":"/app/configs/production.yaml"}' \ - -b cookies.txt | jq -r '.scan_id') - -echo "Scan ID: $SCAN_ID" - -# 3. Poll status -while true; do - STATUS=$(curl -s -X GET http://localhost:5000/api/scans/$SCAN_ID/status \ - -b cookies.txt | jq -r '.status') - echo "Status: $STATUS" - - if [ "$STATUS" == "completed" ] || [ "$STATUS" == "failed" ]; then - break - fi - - sleep 5 -done - -# 4. Get results -curl -X GET http://localhost:5000/api/scans/$SCAN_ID \ - -b cookies.txt | jq '.' -``` - ---- - -## 🧪 Testing - -### Run All Tests - -**In Docker:** -```bash -docker-compose -f docker-compose-web.yml run --rm web pytest tests/ -v -``` - -**Locally:** -```bash -pip install -r requirements-web.txt -pytest tests/ -v -``` - -### Test Breakdown - -| Test File | Tests | Description | -|-----------|-------|-------------| -| `test_scan_service.py` | 15 | Service layer CRUD operations | -| `test_scan_api.py` | 24 | API endpoints integration tests | -| `test_authentication.py` | 30+ | Login, logout, decorators | -| `test_background_jobs.py` | 13 | Scheduler and job execution | -| `test_error_handling.py` | 18+ | Error handlers, logging, headers | -| **Total** | **100** | **All passing ✓** | - -### Manual Testing - -See [MANUAL_TESTING.md](MANUAL_TESTING.md) for comprehensive manual testing checklist. - -**Quick Manual Tests:** -1. Login with correct password → succeeds -2. Login with incorrect password → fails -3. Trigger scan via UI → runs in background -4. View scan list → shows pagination -5. View scan details → displays all data -6. Delete scan → removes files and DB records -7. Logout → destroys session - ---- - -## 🎓 Lessons Learned - -### What Went Well - -1. **Service Layer Architecture** - Clean separation between API endpoints and business logic made testing much easier - -2. **Background Job Integration** - APScheduler worked perfectly for async scan execution without needing Redis/Celery - -3. **Database Mapping Strategy** - Processing in order (sites → IPs → ports → services → certs → TLS) with `flush()` after each level handled foreign keys elegantly - -4. **Test-First Approach** - Writing tests for Steps 1-3 before implementation caught many edge cases early - -5. **Comprehensive Documentation** - Detailed PHASE2.md plan made implementation straightforward and prevented scope creep - -### Challenges Overcome - -1. **SQLite Concurrency** - Initial database locking issues with concurrent scans - - **Solution:** Enabled WAL mode, added connection pooling, increased busy timeout to 15s - -2. **Complex JSON→DB Mapping** - Nested JSON structure with many relationships - - **Solution:** Created `_map_report_to_models()` with ordered processing and `flush()` for ID generation - -3. **Background Thread Sessions** - SQLAlchemy session management in threads - - **Solution:** Create isolated session per thread, pass `db_url` to background job - -4. **Content Negotiation** - API and web requests need different error formats - - **Solution:** Check `request.path.startswith('/api/')` and `Accept` header - -5. **Request ID Correlation** - Difficult to correlate logs across request lifecycle - - **Solution:** Add RequestIDLogFilter with UUID-based request IDs in logs and headers - -### Technical Decisions - -1. **APScheduler over Celery** - Simpler deployment, sufficient for single-user use case -2. **Session Auth over JWT** - Simpler for Phase 2, token auth deferred to Phase 5 -3. **SQLite WAL Mode** - Better concurrency without switching databases -4. **Bootstrap 5 Dark Theme** - Matches existing HTML report aesthetics -5. **Pytest over unittest** - More powerful fixtures, better parametrization - ---- - -## 🔮 What's Next: Phase 3 - -**Target Duration:** Weeks 5-6 (2 weeks) - -**Goals:** -- Enhanced dashboard with trend charts (Chart.js) -- Scheduled scan management UI -- Real-time scan progress -- Timeline view of scan history - -**Key Features:** -- **Dashboard Enhancement:** - - Summary cards (total scans, last scan, IPs, ports) - - Recent scans table - - Security warnings section - - Drift alerts section - -- **Trend Charts:** - - Port count over time (line chart) - - Service distribution (bar chart) - - Certificate expiration timeline - -- **Scheduled Scans:** - - List/create/edit/delete schedules - - Cron expression configuration - - Next run time display - - APScheduler job management - -See [ROADMAP.md](ROADMAP.md) for complete Phase 3 plan. - ---- - -## 📝 Migration from Phase 1 - -Phase 2 is fully backward compatible with Phase 1: - -**No Breaking Changes:** -- ✅ Database schema unchanged (11 tables from Phase 1) -- ✅ CLI scanner still works standalone -- ✅ YAML config format unchanged -- ✅ JSON/HTML/ZIP output format unchanged -- ✅ Settings system compatible - -**New Additions:** -- ✅ REST API endpoints (were stubs in Phase 1) -- ✅ Background job system -- ✅ Authentication system -- ✅ Web UI templates -- ✅ 3 new database migrations - -**Migration Steps:** -1. Pull latest code -2. Run database migrations: `alembic upgrade head` -3. Set application password (if not set): `python3 init_db.py --password YOUR_PASSWORD` -4. Rebuild Docker image: `docker-compose -f docker-compose-web.yml build` -5. Start services: `docker-compose -f docker-compose-web.yml up -d` - ---- - -## 📊 Final Metrics - -### Code Coverage - -- **Total Lines Added:** ~7,500+ -- **Files Created:** 34 -- **Files Modified:** 10 -- **Test Coverage:** 100 test functions, 1,825 lines -- **Documentation:** 2,000+ lines - -### Features Delivered - -- ✅ 5 REST API endpoints (scans CRUD + status) -- ✅ 3 settings endpoints (get, update, health) -- ✅ Background job queue with APScheduler -- ✅ Session-based authentication -- ✅ 5 web UI pages (login, dashboard, scans list/detail, errors) -- ✅ 6 error templates (400, 401, 403, 404, 405, 500) -- ✅ Comprehensive error handling and logging -- ✅ Docker deployment with healthcheck -- ✅ Complete API documentation -- ✅ Deployment guide - -### Success Rate - -- ✅ All 100 tests passing -- ✅ All success criteria met -- ✅ All deliverables completed on time -- ✅ Zero critical bugs -- ✅ Production-ready deployment - ---- - -## 🙏 Acknowledgments - -**Technologies Used:** -- Flask 3.0 - Web framework -- SQLAlchemy 2.0 - ORM -- APScheduler 3.10 - Background jobs -- Flask-Login 0.6 - Authentication -- Bootstrap 5 - UI framework -- pytest 7.4 - Testing -- Alembic 1.13 - Database migrations - ---- - -## 📞 Support - -**Documentation:** -- [API Reference](API_REFERENCE.md) -- [Deployment Guide](DEPLOYMENT.md) -- [Developer Guide](../../CLAUDE.md) -- [Roadmap](ROADMAP.md) - -**Issues:** https://github.com/anthropics/sneakyscanner/issues - ---- - -**Phase 2 Status:** COMPLETE ✓ -**Next Phase:** Phase 3 - Dashboard & Scheduling -**Last Updated:** 2025-11-14 diff --git a/docs/ai/PHASE3.md b/docs/ai/PHASE3.md deleted file mode 100644 index f5199e5..0000000 --- a/docs/ai/PHASE3.md +++ /dev/null @@ -1,2204 +0,0 @@ -# Phase 3 Implementation Plan: Dashboard Enhancement & Scheduled Scans - -**Status:** In Progress -**Progress:** 9/14 days complete (64%) -**Estimated Duration:** 14 days (2 weeks) -**Dependencies:** Phase 2 Complete ✅ - -## Progress Summary - -- ✅ **Step 1: Fix Styling Issues & CSS Refactor** (Day 1) - COMPLETE -- ✅ **Step 2: ScheduleService Implementation** (Days 2-3) - COMPLETE -- ✅ **Step 3: Schedules API Endpoints** (Days 4-5) - COMPLETE -- ✅ **Step 4: Schedule Management UI** (Days 6-7) - COMPLETE -- ✅ **Step 5: Enhanced Dashboard with Charts** (Days 8-9) - COMPLETE -- ✅ **Step 6: Scheduler Integration** (Day 10) - COMPLETE -- ✅ **Step 7: Scan Comparison Features** (Days 11-12) - COMPLETE -- ✅ **Step 8: Testing & Documentation** (Days 13-14) - COMPLETE - ---- - -## Table of Contents - -1. [Overview](#overview) -2. [Current State Analysis](#current-state-analysis) -3. [Critical Bug Fix](#critical-bug-fix) -4. [Files to Create](#files-to-create) -5. [Files to Modify](#files-to-modify) -6. [Step-by-Step Implementation](#step-by-step-implementation) -7. [Dependencies & Prerequisites](#dependencies--prerequisites) -8. [Testing Approach](#testing-approach) -9. [Potential Challenges & Solutions](#potential-challenges--solutions) -10. [Success Criteria](#success-criteria) -11. [Migration Path](#migration-path) -12. [Estimated Timeline](#estimated-timeline) -13. [Key Design Decisions](#key-design-decisions) -14. [Documentation Deliverables](#documentation-deliverables) - ---- - -## Overview - -Phase 3 focuses on enhancing the web application with scheduling capabilities and improved dashboard visualizations: - -1. **Critical Bug Fix** - Fix white row coloring in scan tables -2. **Scheduled Scans** - Complete schedule management system with cron expressions -3. **Enhanced Dashboard** - Trending charts, schedule widgets, alert summaries -4. **Scan Comparison** - Historical analysis and infrastructure drift detection - -### Goals - -- 🐛 Fix white row bug affecting scan tables (critical UX issue) -- ✅ Complete schedule management (CRUD operations) -- ✅ Scheduled scan execution with cron expressions -- ✅ Enhanced dashboard with Chart.js visualizations -- ✅ Scan comparison and historical analysis -- ✅ CSS extraction and better maintainability - ---- - -## Current State Analysis - -### What's Already Done (Phase 2) - -**✅ Phase 2 Complete** - Full web application core: - -- **Database Schema** - All 11 models including Schedule table -- **ScanService** - Complete CRUD operations with 545 lines -- **Scan API** - 5 endpoints fully implemented -- **SchedulerService** - Partial implementation (258 lines) - - ✅ `queue_scan()` - Immediate scan execution works - - ⚠️ `add_scheduled_scan()` - Placeholder (marked for Phase 3) - - ⚠️ `_trigger_scheduled_scan()` - Skeleton with TODO comments - - ✅ `remove_scheduled_scan()` - Basic implementation -- **Authentication** - Flask-Login with @login_required decorators -- **UI Templates** - Dashboard, scans list/detail, login pages -- **Background Jobs** - APScheduler with concurrent execution (max 3) -- **Error Handling** - Content negotiation, custom error pages -- **Testing** - 100 test functions, 1,825 lines of test code - -### Schedule-Related Components Analysis - -#### Schedule Model (`web/models.py` - lines 142-158) -```python -class Schedule(Base): - __tablename__ = 'schedules' - - id = Column(Integer, primary_key=True) - name = Column(String(255), nullable=False) - config_file = Column(Text, nullable=False) - cron_expression = Column(String(100), nullable=False) - enabled = Column(Boolean, default=True) - last_run = Column(DateTime) - next_run = Column(DateTime) - created_at = Column(DateTime, default=datetime.utcnow) - updated_at = Column(DateTime, default=datetime.utcnow, onupdate=datetime.utcnow) -``` - -**Status:** ✅ Complete - No migration needed - -#### Schedules API (`web/api/schedules.py` - 159 lines) -```python -@bp.route('', methods=['GET']) -@api_auth_required -def list_schedules(): - """List all schedules. TODO: Implement in Phase 3.""" - return jsonify({'message': 'Schedules list - to be implemented in Phase 3'}), 200 -``` - -**Status:** ⚠️ All 6 endpoints are stubs returning placeholders - -**Endpoints to Implement:** -- `GET /api/schedules` - List all schedules -- `GET /api/schedules/` - Get schedule details -- `POST /api/schedules` - Create new schedule -- `PUT /api/schedules/` - Update schedule -- `DELETE /api/schedules/` - Delete schedule -- `POST /api/schedules//trigger` - Manually trigger scheduled scan - -#### SchedulerService Gaps - -**Missing Implementations:** -1. `add_scheduled_scan(schedule)` - Currently placeholder -2. `_trigger_scheduled_scan(schedule_id)` - TODO comments only -3. Loading schedules on app startup - Not implemented -4. Cron expression parsing - No validation -5. Next run time calculation - Missing - -### UI Components Status - -**Existing Templates:** -- `web/templates/base.html` (346 lines) - Has inline CSS (need to extract) -- `web/templates/dashboard.html` (356 lines) - Basic stats, needs charts -- `web/templates/scans.html` (469 lines) - Has white row bug -- `web/templates/scan_detail.html` (399 lines) - Has white row bug in port tables - -**Missing Templates:** -- Schedule list page -- Schedule create form -- Schedule edit form - -**Web Routes (`web/routes/main.py` - 69 lines):** -- Only has scan routes, no schedule routes - ---- - -## Critical Bug Fix - -### White Row Coloring Issue - -**Problem:** Dynamically created table rows in scans tables display with white background instead of dark theme colors. - -**Affected Files:** -1. `web/templates/scans.html` (lines 208-241) - renderScansTable() -2. `web/templates/dashboard.html` (lines 221-260) - renderScansTable() -3. `web/templates/scan_detail.html` (lines 305-327) - Port tables - -**Root Cause:** - -JavaScript dynamically creates `` elements that don't inherit CSS styles properly: - -```javascript -const row = document.createElement('tr'); -row.innerHTML = ` - ${scan.id} - ${scan.title || 'Untitled Scan'} - ... -`; -tbody.appendChild(row); -``` - -**Current CSS (base.html lines 157-165):** -```css -.table tbody tr { - background-color: #1e293b; /* Dark slate */ - border-color: #334155; -} - -.table tbody tr:hover { - background-color: #334155; - cursor: pointer; -} -``` - -**Solution:** - -Add explicit class to dynamically created rows: - -```javascript -const row = document.createElement('tr'); -row.classList.add('scan-row'); // Add explicit class -row.innerHTML = `...`; -``` - -And enhance CSS with higher specificity: -```css -.table tbody tr.scan-row, -.table tbody tr { - background-color: #1e293b !important; - border-color: #334155 !important; -} -``` - ---- - -## Files to Create - -### Backend Services - -#### 1. `web/services/schedule_service.py` -Schedule CRUD operations and business logic. - -**Class: ScheduleService** - -**Estimated Size:** ~400 lines - -**Methods:** -- `__init__(self, db_session)` - Initialize with database session -- `create_schedule(name, config_file, cron_expression, enabled=True)` → schedule_id - - Validate cron expression - - Validate config file exists - - Calculate next_run time - - Create Schedule record - - Return schedule_id - -- `get_schedule(schedule_id)` → schedule dict - - Query Schedule by ID - - Format for API response - - Include execution history (recent scans with this schedule_id) - -- `list_schedules(page=1, per_page=20, enabled_filter=None)` → paginated results - - Query with pagination - - Filter by enabled status if provided - - Calculate relative next_run time ("in 2 hours", "tomorrow at 3:00 PM") - - Return total count and items - -- `update_schedule(schedule_id, **updates)` → schedule dict - - Validate cron_expression if changed - - Recalculate next_run if cron changed - - Update Schedule record - - Reload schedule in APScheduler - - Return updated schedule - -- `delete_schedule(schedule_id)` → success - - Remove job from APScheduler - - Delete Schedule record (do NOT delete associated scans) - - Return success status - -- `toggle_enabled(schedule_id, enabled)` → schedule dict - - Enable or disable schedule - - Add/remove from APScheduler - - Return updated schedule - -- `update_run_times(schedule_id, last_run, next_run)` → success - - Update last_run and next_run timestamps - - Called after each execution - - Return success status - -- `validate_cron_expression(cron_expr)` → (valid, error_message) - - Use croniter library - - Return True if valid, False + error message if invalid - -- `calculate_next_run(cron_expr, from_time=None)` → datetime - - Calculate next run time from cron expression - - Use croniter - - Return datetime (UTC) - -- `get_schedule_history(schedule_id, limit=10)` → list of scans - - Query recent scans triggered by this schedule - - Return list of scan dicts - -**Helper Methods:** -- `_schedule_to_dict(schedule_obj)` - Convert Schedule model to dict -- `_get_relative_time(dt)` - Format datetime as "in 2 hours", "yesterday" - -#### 2. `web/static/css/styles.css` -Extracted CSS for better maintainability. - -**Estimated Size:** ~350 lines (extracted from base.html) - -**Sections:** -- Variables (CSS custom properties for colors) -- Global styles -- Navigation -- Cards and containers -- Tables (including fix for white rows) -- Forms -- Buttons -- Badges and labels -- Charts (dark theme) -- Utilities - -**Example Structure:** -```css -/* CSS Variables */ -:root { - --bg-primary: #0f172a; - --bg-secondary: #1e293b; - --bg-tertiary: #334155; - --text-primary: #e2e8f0; - --text-secondary: #94a3b8; - --border-color: #334155; - --accent-blue: #60a5fa; - --success-bg: #065f46; - --success-text: #6ee7b7; - --warning-bg: #78350f; - --warning-text: #fcd34d; - --danger-bg: #7f1d1d; - --danger-text: #fca5a5; -} - -/* Fix for dynamically created table rows */ -.table tbody tr, -.table tbody tr.scan-row { - background-color: var(--bg-secondary) !important; - border-color: var(--border-color) !important; -} - -.table tbody tr:hover { - background-color: var(--bg-tertiary) !important; - cursor: pointer; -} -``` - -### Frontend Templates - -#### 3. `web/templates/schedules.html` -List all schedules with enable/disable toggles. - -**Estimated Size:** ~400 lines - -**Layout:** -```html -{% extends "base.html" %} - -{% block content %} -
-
-

Scheduled Scans

- - New Schedule - -
- - -
-
-
-
-
Total Schedules
-

-

-
-
-
-
-
-
-
Enabled
-

-

-
-
-
-
-
-
-
Next Run
-
-
-
-
-
-
-
-
-
Executions (24h)
-

-

-
-
-
-
- - -
-
- - - - - - - - - - - - - - -
NameSchedule (Cron)Next RunLast RunStatusActions
-
-
-
- - -{% endblock %} -``` - -#### 4. `web/templates/schedule_create.html` -Form to create new scheduled scan. - -**Estimated Size:** ~300 lines - -**Features:** -- Schedule name input -- Config file selector (dropdown of available configs) -- Cron expression builder OR manual entry -- Cron expression validator (client-side and server-side) -- Human-readable cron description -- Enable/disable toggle -- Submit button - -**Cron Expression Builder:** -```html -
-
-
Schedule Configuration
- - -
- -
- - - - -
-
- - -
- - - - Format: minute hour day month weekday - -
- - -
- Enter a cron expression above -
- - -
- Next 5 runs: -
    -
    -
    -
    -``` - -#### 5. `web/templates/schedule_edit.html` -Form to edit existing schedule. - -**Estimated Size:** ~250 lines - -**Similar to create, but:** -- Pre-populate fields from existing schedule -- Show execution history (last 10 scans) -- "Delete Schedule" button -- "Test Run Now" button - -### Testing Files - -#### 6. `tests/test_schedule_service.py` -Unit tests for ScheduleService. - -**Estimated Size:** ~450 lines, 18+ tests - -**Test Coverage:** -- `test_create_schedule` - Valid schedule creation -- `test_create_schedule_invalid_cron` - Cron validation -- `test_create_schedule_invalid_config` - Config file validation -- `test_get_schedule` - Retrieve schedule -- `test_get_schedule_not_found` - 404 handling -- `test_list_schedules` - Pagination -- `test_list_schedules_filter_enabled` - Filter by enabled status -- `test_update_schedule` - Update fields -- `test_update_schedule_cron` - Recalculate next_run when cron changes -- `test_delete_schedule` - Delete schedule -- `test_toggle_enabled` - Enable/disable -- `test_update_run_times` - Update after execution -- `test_validate_cron_expression_valid` - Valid expressions -- `test_validate_cron_expression_invalid` - Invalid expressions -- `test_calculate_next_run` - Next run calculation -- `test_get_schedule_history` - Execution history -- `test_schedule_to_dict` - Serialization -- `test_concurrent_schedule_operations` - Thread safety - -#### 7. `tests/test_schedule_api.py` -Integration tests for schedules API. - -**Estimated Size:** ~500 lines, 22+ tests - -**Test Coverage:** -- `test_list_schedules_empty` - Empty list -- `test_list_schedules_populated` - Multiple schedules -- `test_list_schedules_pagination` - Pagination -- `test_list_schedules_filter_enabled` - Filter -- `test_get_schedule` - Get details -- `test_get_schedule_not_found` - 404 -- `test_create_schedule` - Create new -- `test_create_schedule_invalid_cron` - Validation -- `test_create_schedule_invalid_config` - File validation -- `test_update_schedule` - Update fields -- `test_update_schedule_not_found` - 404 -- `test_delete_schedule` - Delete -- `test_delete_schedule_not_found` - 404 -- `test_trigger_schedule` - Manual trigger -- `test_trigger_schedule_not_found` - 404 -- `test_toggle_enabled_via_update` - Enable/disable -- `test_schedules_require_authentication` - Auth required -- `test_schedule_execution_history` - Show related scans -- `test_schedule_next_run_calculation` - Correct calculation -- `test_schedule_workflow_integration` - Complete workflow -- `test_concurrent_schedule_updates` - Concurrency - -#### 8. `tests/test_charts.py` -Tests for chart data generation. - -**Estimated Size:** ~200 lines, 8+ tests - -**Test Coverage:** -- `test_scan_trend_data` - Scans per day calculation -- `test_port_count_trend` - Port count over time -- `test_service_distribution` - Service type breakdown -- `test_certificate_expiry_timeline` - Cert expiry dates -- `test_empty_data_handling` - No scans case -- `test_date_range_filtering` - Filter by date range -- `test_data_format_for_chartjs` - Correct JSON format -- `test_trend_data_caching` - Performance optimization - ---- - -## Files to Modify - -### Backend Updates - -#### 1. `web/api/schedules.py` -Replace all stub implementations with working code. - -**Current State:** 159 lines, all placeholders - -**Changes:** -- Import ScheduleService -- Implement all 6 endpoints: - - `GET /api/schedules` - Call ScheduleService.list_schedules() - - `GET /api/schedules/` - Call ScheduleService.get_schedule() - - `POST /api/schedules` - Call ScheduleService.create_schedule() - - `PUT /api/schedules/` - Call ScheduleService.update_schedule() - - `DELETE /api/schedules/` - Call ScheduleService.delete_schedule() - - `POST /api/schedules//trigger` - Trigger immediate scan with schedule_id - -**New Code (~300 lines total):** -```python -from web.services.schedule_service import ScheduleService -from web.auth.decorators import api_auth_required -from flask import current_app, jsonify, request - -@bp.route('', methods=['GET']) -@api_auth_required -def list_schedules(): - """List all schedules with pagination.""" - page = request.args.get('page', 1, type=int) - per_page = request.args.get('per_page', 20, type=int) - enabled_filter = request.args.get('enabled', type=lambda x: x.lower() == 'true') - - schedule_service = ScheduleService(current_app.db_session) - result = schedule_service.list_schedules(page, per_page, enabled_filter) - - return jsonify(result), 200 - -@bp.route('', methods=['POST']) -@api_auth_required -def create_schedule(): - """Create a new schedule.""" - data = request.get_json() or {} - - # Validate required fields - required = ['name', 'config_file', 'cron_expression'] - for field in required: - if field not in data: - return jsonify({'error': f'Missing required field: {field}'}), 400 - - schedule_service = ScheduleService(current_app.db_session) - - try: - schedule_id = schedule_service.create_schedule( - name=data['name'], - config_file=data['config_file'], - cron_expression=data['cron_expression'], - enabled=data.get('enabled', True) - ) - - # Add to APScheduler - schedule = schedule_service.get_schedule(schedule_id) - current_app.scheduler.add_scheduled_scan(schedule) - - return jsonify({ - 'schedule_id': schedule_id, - 'message': 'Schedule created successfully' - }), 201 - except ValueError as e: - return jsonify({'error': str(e)}), 400 - -# ... more endpoints -``` - -#### 2. `web/services/scheduler_service.py` -Complete placeholder implementations. - -**Current State:** 258 lines, partial implementation - -**Changes:** -- Complete `add_scheduled_scan(schedule)` - Add cron job to APScheduler -- Complete `_trigger_scheduled_scan(schedule_id)` - Execute scheduled scan -- Add `load_schedules_on_startup()` - Load all enabled schedules -- Add cron expression parsing with croniter - -**New/Updated Methods:** -```python -from croniter import croniter -from datetime import datetime - -def add_scheduled_scan(self, schedule): - """Add a cron job for scheduled scan execution.""" - if not schedule.get('enabled'): - return - - job_id = f"schedule_{schedule['id']}" - - # Parse cron expression - trigger = CronTrigger.from_crontab(schedule['cron_expression']) - - # Add job to scheduler - self.scheduler.add_job( - func=self._trigger_scheduled_scan, - trigger=trigger, - id=job_id, - args=[schedule['id']], - replace_existing=True, - max_instances=1 - ) - - logger.info(f"Added scheduled scan: {schedule['name']} ({job_id})") - -def _trigger_scheduled_scan(self, schedule_id): - """Execute a scheduled scan.""" - from web.services.schedule_service import ScheduleService - - logger.info(f"Triggering scheduled scan: schedule_id={schedule_id}") - - # Get schedule details - schedule_service = ScheduleService(self.db_session) - schedule = schedule_service.get_schedule(schedule_id) - - if not schedule: - logger.error(f"Schedule {schedule_id} not found") - return - - # Trigger scan with schedule_id - from web.services.scan_service import ScanService - scan_service = ScanService(self.db_session) - - scan_id = scan_service.trigger_scan( - config_file=schedule['config_file'], - triggered_by='scheduled', - schedule_id=schedule_id - ) - - # Queue the scan - self.queue_scan(schedule['config_file'], scan_id, self.db_url) - - # Update last_run timestamp - schedule_service.update_run_times( - schedule_id=schedule_id, - last_run=datetime.utcnow(), - next_run=self._calculate_next_run(schedule['cron_expression']) - ) - - logger.info(f"Scheduled scan queued: scan_id={scan_id}") - -def load_schedules_on_startup(self): - """Load all enabled schedules from database on app startup.""" - from web.services.schedule_service import ScheduleService - - schedule_service = ScheduleService(self.db_session) - schedules = schedule_service.list_schedules(page=1, per_page=1000, enabled_filter=True) - - for schedule in schedules['schedules']: - self.add_scheduled_scan(schedule) - - logger.info(f"Loaded {len(schedules['schedules'])} schedules on startup") - -def _calculate_next_run(self, cron_expression): - """Calculate next run time from cron expression.""" - cron = croniter(cron_expression, datetime.utcnow()) - return cron.get_next(datetime) -``` - -#### 3. `web/routes/main.py` -Add schedule management routes. - -**Current State:** 69 lines, only scan routes - -**Changes:** -- Add schedule routes: - - `GET /schedules` - List schedules - - `GET /schedules/create` - Create form - - `GET /schedules//edit` - Edit form - -**New Code:** -```python -@bp.route('/schedules') -@login_required -def schedules(): - """List all schedules.""" - return render_template('schedules.html') - -@bp.route('/schedules/create') -@login_required -def create_schedule(): - """Create new schedule form.""" - # Get list of available config files - import os - configs_dir = '/app/configs' - config_files = [f for f in os.listdir(configs_dir) if f.endswith('.yaml')] - return render_template('schedule_create.html', config_files=config_files) - -@bp.route('/schedules//edit') -@login_required -def edit_schedule(schedule_id): - """Edit existing schedule form.""" - from web.services.schedule_service import ScheduleService - schedule_service = ScheduleService(current_app.db_session) - - try: - schedule = schedule_service.get_schedule(schedule_id) - return render_template('schedule_edit.html', schedule=schedule) - except Exception as e: - flash(f'Schedule not found: {e}', 'danger') - return redirect(url_for('main.schedules')) -``` - -#### 4. `web/templates/base.html` -Extract CSS, add Chart.js, fix white rows. - -**Current State:** 346 lines with 280 lines of inline CSS (lines 8-288) - -**Changes:** -- Replace inline `