Phase 2 Step 6: Docker & Deployment Configuration

Implement production-ready Docker deployment with comprehensive configuration and documentation for SneakyScanner web application. Changes: - Update docker-compose-web.yml with production configuration - Add scheduler environment variables (SCHEDULER_EXECUTORS, SCHEDULER_JOB_DEFAULTS_MAX_INSTANCES) - Enable privileged mode and host networking for scanner operations - Configure health check endpoint monitoring (30s interval, 40s start period) - Set production defaults (FLASK_ENV=production, FLASK_DEBUG=false) - Add SNEAKYSCANNER_ENCRYPTION_KEY support - Create .env.example configuration template - Flask, database, and security settings - Scheduler configuration options - Detailed comments with key generation examples - Production deployment guidance - Create comprehensive deployment documentation (docs/ai/DEPLOYMENT.md) - Quick start guide and prerequisites - Detailed configuration instructions - Volume management and backup procedures - Health monitoring and troubleshooting - Security considerations and best practices - Upgrade/rollback and backup/restore procedures - Update PHASE2.md progress tracker - Mark Step 6 as complete - Update progress to 11/14 days (79%) - Document deliverables and implementation details Deployment is now production-ready with proper security defaults, health monitoring, and comprehensive documentation for system administrators.
2025-11-14 12:01:21 -06:00
parent 19a64b0cbe
commit ebfefa9df3
4 changed files with 821 additions and 30 deletions
--- a/docs/ai/DEPLOYMENT.md
+++ b/docs/ai/DEPLOYMENT.md
@@ -0,0 +1,666 @@
+# SneakyScanner Deployment Guide
+
+## Table of Contents
+
+1. [Overview](#overview)
+2. [Prerequisites](#prerequisites)
+3. [Quick Start](#quick-start)
+4. [Configuration](#configuration)
+5. [First-Time Setup](#first-time-setup)
+6. [Running the Application](#running-the-application)
+7. [Volume Management](#volume-management)
+8. [Health Monitoring](#health-monitoring)
+9. [Troubleshooting](#troubleshooting)
+10. [Security Considerations](#security-considerations)
+11. [Upgrading](#upgrading)
+12. [Backup and Restore](#backup-and-restore)
+
+---
+
+## Overview
+
+SneakyScanner is deployed as a Docker container running a Flask web application with an integrated network scanner. The application requires privileged mode and host networking to perform network scans using masscan and nmap.
+
+**Architecture:**
+- **Web Application**: Flask app on port 5000
+- **Database**: SQLite (persisted to volume)
+- **Background Jobs**: APScheduler for async scan execution
+- **Scanner**: masscan, nmap, sslyze, Playwright
+
+---
+
+## Prerequisites
+
+### System Requirements
+
+- **Operating System**: Linux (Ubuntu 20.04+, Debian 11+, or similar)
+- **Docker**: Version 20.10+ or Docker Engine 24.0+
+- **Docker Compose**: Version 2.0+ (or docker-compose 1.29+)
+- **Memory**: Minimum 2GB RAM (4GB+ recommended)
+- **Disk Space**: Minimum 5GB free space
+- **Permissions**: Root/sudo access for Docker privileged mode
+
+### Network Requirements
+
+- Outbound internet access for Docker image downloads
+- Access to target networks for scanning
+- Port 5000 available on host (or configure alternative)
+
+### Install Docker and Docker Compose
+
+**Ubuntu/Debian:**
+```bash
+# Install Docker
+curl -fsSL https://get.docker.com -o get-docker.sh
+sudo sh get-docker.sh
+
+# Add your user to docker group
+sudo usermod -aG docker $USER
+newgrp docker
+
+# Verify installation
+docker --version
+docker compose version
+```
+
+**Other Linux distributions:** See [Docker installation guide](https://docs.docker.com/engine/install/)
+
+---
+
+## Quick Start
+
+For users who want to get started immediately:
+
+```bash
+# 1. Clone the repository
+git clone <repository-url>
+cd SneakyScan
+
+# 2. Create environment file
+cp .env.example .env
+# Edit .env and set SECRET_KEY and SNEAKYSCANNER_ENCRYPTION_KEY
+nano .env
+
+# 3. Build the Docker image
+docker compose -f docker-compose-web.yml build
+
+# 4. Initialize the database and set password
+docker compose -f docker-compose-web.yml run --rm init-db --password "YourSecurePassword"
+
+# 5. Start the application
+docker compose -f docker-compose-web.yml up -d
+
+# 6. Access the web interface
+# Open browser to: http://localhost:5000
+```
+
+---
+
+## Configuration
+
+### Environment Variables
+
+SneakyScanner is configured via environment variables. The recommended approach is to use a `.env` file.
+
+#### Creating Your .env File
+
+```bash
+# Copy the example file
+cp .env.example .env
+
+# Generate secure keys
+python3 -c "import secrets; print('SECRET_KEY=' + secrets.token_hex(32))" >> .env
+python3 -c "from cryptography.fernet import Fernet; print('SNEAKYSCANNER_ENCRYPTION_KEY=' + Fernet.generate_key().decode())" >> .env
+
+# Edit other settings as needed
+nano .env
+```
+
+#### Key Configuration Options
+
+| Variable | Description | Default | Required |
+|----------|-------------|---------|----------|
+| `FLASK_ENV` | Environment mode (`production` or `development`) | `production` | Yes |
+| `FLASK_DEBUG` | Enable debug mode (`true` or `false`) | `false` | Yes |
+| `SECRET_KEY` | Flask session secret (change in production!) | `dev-secret-key-change-in-production` | **Yes** |
+| `SNEAKYSCANNER_ENCRYPTION_KEY` | Encryption key for sensitive settings | (empty) | **Yes** |
+| `DATABASE_URL` | SQLite database path | `sqlite:////app/data/sneakyscanner.db` | Yes |
+| `LOG_LEVEL` | Logging level (DEBUG, INFO, WARNING, ERROR) | `INFO` | No |
+| `SCHEDULER_EXECUTORS` | Number of concurrent scan threads | `2` | No |
+| `SCHEDULER_JOB_DEFAULTS_MAX_INSTANCES` | Max instances of same job | `3` | No |
+| `CORS_ORIGINS` | CORS allowed origins (comma-separated) | `*` | No |
+
+**Important Security Note:**
+- **ALWAYS** change `SECRET_KEY` and `SNEAKYSCANNER_ENCRYPTION_KEY` in production
+- Never commit `.env` file to version control
+- Use strong, randomly-generated keys
+
+---
+
+## First-Time Setup
+
+### Step 1: Prepare Directories
+
+The application needs these directories (created automatically by Docker):
+
+```bash
+# Verify directories exist
+ls -la configs/ data/ output/ logs/
+
+# If missing, create them
+mkdir -p configs data output logs
+```
+
+### Step 2: Configure Scan Targets
+
+Create YAML configuration files for your scan targets:
+
+```bash
+# Example configuration
+cat > configs/my-network.yaml <<EOF
+title: "My Network Infrastructure"
+sites:
+  - name: "Web Servers"
+    ips:
+      - address: "192.168.1.10"
+        expected:
+          ping: true
+          tcp_ports: [80, 443]
+          udp_ports: []
+          services: ["http", "https"]
+EOF
+```
+
+### Step 3: Build Docker Image
+
+```bash
+# Build the image (takes 5-10 minutes on first run)
+docker compose -f docker-compose-web.yml build
+
+# Verify image was created
+docker images | grep sneakyscanner
+```
+
+### Step 4: Initialize Database
+
+The database must be initialized before first use:
+
+```bash
+# Initialize database and set application password
+docker compose -f docker-compose-web.yml run --rm init-db --password "YourSecurePassword"
+
+# The init-db command will:
+# - Create database schema
+# - Run all Alembic migrations
+# - Set the application password
+# - Create default settings
+```
+
+**Password Requirements:**
+- Minimum 8 characters recommended
+- Use a strong, unique password
+- Store securely (password manager)
+
+### Step 5: Verify Configuration
+
+```bash
+# Check database file was created
+ls -lh data/sneakyscanner.db
+
+# Verify Docker Compose configuration
+docker compose -f docker-compose-web.yml config
+```
+
+---
+
+## Running the Application
+
+### Starting the Application
+
+```bash
+# Start in detached mode (background)
+docker compose -f docker-compose-web.yml up -d
+
+# View logs during startup
+docker compose -f docker-compose-web.yml logs -f web
+
+# Expected output:
+# web_1  | INFO:werkzeug: * Running on http://0.0.0.0:5000/ (Press CTRL+C to quit)
+```
+
+### Accessing the Web Interface
+
+1. Open browser to: **http://localhost:5000**
+2. Login with the password you set during database initialization
+3. Dashboard will display recent scans and statistics
+
+### Stopping the Application
+
+```bash
+# Stop containers (preserves data)
+docker compose -f docker-compose-web.yml down
+
+# Stop and remove volumes (WARNING: deletes all data!)
+docker compose -f docker-compose-web.yml down -v
+```
+
+### Restarting the Application
+
+```bash
+# Restart all services
+docker compose -f docker-compose-web.yml restart
+
+# Restart only the web service
+docker compose -f docker-compose-web.yml restart web
+```
+
+### Viewing Logs
+
+```bash
+# View all logs
+docker compose -f docker-compose-web.yml logs
+
+# Follow logs in real-time
+docker compose -f docker-compose-web.yml logs -f
+
+# View last 100 lines
+docker compose -f docker-compose-web.yml logs --tail=100
+
+# View logs for specific service
+docker compose -f docker-compose-web.yml logs web
+```
+
+---
+
+## Volume Management
+
+### Understanding Volumes
+
+SneakyScanner uses several mounted volumes for data persistence:
+
+| Volume | Container Path | Purpose | Important? |
+|--------|----------------|---------|------------|
+| `./configs` | `/app/configs` | Scan configuration files (read-only) | Yes |
+| `./data` | `/app/data` | SQLite database | **Critical** |
+| `./output` | `/app/output` | Scan results (JSON, HTML, ZIP) | Yes |
+| `./logs` | `/app/logs` | Application logs | No |
+
+### Backing Up Data
+
+```bash
+# Create backup directory
+mkdir -p backups/$(date +%Y%m%d)
+
+# Backup database
+cp data/sneakyscanner.db backups/$(date +%Y%m%d)/
+
+# Backup scan outputs
+tar -czf backups/$(date +%Y%m%d)/output.tar.gz output/
+
+# Backup configurations
+tar -czf backups/$(date +%Y%m%d)/configs.tar.gz configs/
+```
+
+### Restoring Data
+
+```bash
+# Stop application
+docker compose -f docker-compose-web.yml down
+
+# Restore database
+cp backups/YYYYMMDD/sneakyscanner.db data/
+
+# Restore outputs
+tar -xzf backups/YYYYMMDD/output.tar.gz
+
+# Restart application
+docker compose -f docker-compose-web.yml up -d
+```
+
+### Cleaning Up Old Scan Results
+
+```bash
+# Find old scan results (older than 30 days)
+find output/ -type f -name "scan_report_*.json" -mtime +30
+
+# Delete old scan results
+find output/ -type f -name "scan_report_*" -mtime +30 -delete
+
+# Or use the API to delete scans from UI/API
+```
+
+---
+
+## Health Monitoring
+
+### Health Check Endpoint
+
+SneakyScanner includes a built-in health check endpoint:
+
+```bash
+# Check application health
+curl http://localhost:5000/api/settings/health
+
+# Expected response:
+# {"status": "healthy"}
+```
+
+### Docker Health Status
+
+```bash
+# Check container health status
+docker ps | grep sneakyscanner-web
+
+# View health check logs
+docker inspect sneakyscanner-web | grep -A 10 Health
+```
+
+### Monitoring Logs
+
+```bash
+# Watch for errors in logs
+docker compose -f docker-compose-web.yml logs -f | grep ERROR
+
+# Check application log file
+tail -f logs/sneakyscanner.log
+```
+
+---
+
+## Troubleshooting
+
+### Container Won't Start
+
+**Problem**: Container exits immediately after starting
+
+```bash
+# Check logs for errors
+docker compose -f docker-compose-web.yml logs web
+
+# Common issues:
+# 1. Database not initialized - run init-db first
+# 2. Permission issues with volumes - check directory ownership
+# 3. Port 5000 already in use - change FLASK_PORT or stop conflicting service
+```
+
+### Database Initialization Fails
+
+**Problem**: `init_db.py` fails with errors
+
+```bash
+# Check database directory permissions
+ls -la data/
+
+# Fix permissions if needed
+sudo chown -R $USER:$USER data/
+
+# Verify SQLite is accessible
+sqlite3 data/sneakyscanner.db "SELECT 1;" 2>&1
+
+# Remove corrupted database and reinitialize
+rm data/sneakyscanner.db
+docker compose -f docker-compose-web.yml run --rm init-db --password "YourPassword"
+```
+
+### Scans Fail with "Permission Denied"
+
+**Problem**: Scanner cannot run masscan/nmap
+
+```bash
+# Verify container is running in privileged mode
+docker inspect sneakyscanner-web | grep Privileged
+# Should show: "Privileged": true
+
+# Verify network mode is host
+docker inspect sneakyscanner-web | grep NetworkMode
+# Should show: "NetworkMode": "host"
+
+# If not, verify docker-compose-web.yml has:
+# privileged: true
+# network_mode: host
+```
+
+### Can't Access Web Interface
+
+**Problem**: Browser can't connect to http://localhost:5000
+
+```bash
+# Verify container is running
+docker ps | grep sneakyscanner-web
+
+# Check if Flask is listening
+docker compose -f docker-compose-web.yml exec web netstat -tlnp | grep 5000
+
+# Check firewall rules
+sudo ufw status | grep 5000
+
+# Try from container host
+curl http://localhost:5000/api/settings/health
+
+# Check logs for binding errors
+docker compose -f docker-compose-web.yml logs web | grep -i bind
+```
+
+### Background Scans Not Running
+
+**Problem**: Scans stay in "running" status forever
+
+```bash
+# Check scheduler is initialized
+docker compose -f docker-compose-web.yml logs web | grep -i scheduler
+
+# Check for job execution errors
+docker compose -f docker-compose-web.yml logs web | grep -i "execute_scan"
+
+# Verify APScheduler environment variables
+docker compose -f docker-compose-web.yml exec web env | grep SCHEDULER
+```
+
+### Health Check Failing
+
+**Problem**: Docker health check shows "unhealthy"
+
+```bash
+# Run health check manually
+docker compose -f docker-compose-web.yml exec web \
+  python3 -c "import urllib.request; print(urllib.request.urlopen('http://localhost:5000/api/settings/health').read())"
+
+# Check if health endpoint exists
+curl -v http://localhost:5000/api/settings/health
+
+# Common causes:
+# 1. Application crashed - check logs
+# 2. Database locked - check for long-running scans
+# 3. Flask not fully started - wait 40s (start_period)
+```
+
+---
+
+## Security Considerations
+
+### Production Deployment Checklist
+
+- [ ] Changed `SECRET_KEY` to random value
+- [ ] Changed `SNEAKYSCANNER_ENCRYPTION_KEY` to random value
+- [ ] Set strong application password
+- [ ] Set `FLASK_ENV=production`
+- [ ] Set `FLASK_DEBUG=false`
+- [ ] Configured proper `CORS_ORIGINS` (not `*`)
+- [ ] Using HTTPS/TLS (reverse proxy recommended)
+- [ ] Restricted network access (firewall rules)
+- [ ] Regular backups configured
+- [ ] Log monitoring enabled
+
+### Network Security
+
+**Privileged Mode Considerations:**
+- Container runs with `--privileged` flag for raw socket access (masscan/nmap)
+- This grants extensive host capabilities - only run on trusted networks
+- Restrict Docker host access with firewall rules
+- Consider running on dedicated scan server
+
+**Recommendations:**
+```bash
+# Restrict access to port 5000 with firewall
+sudo ufw allow from 192.168.1.0/24 to any port 5000
+sudo ufw enable
+
+# Or use reverse proxy (nginx, Apache) with authentication
+```
+
+### HTTPS/TLS Setup
+
+SneakyScanner does not include built-in TLS. For production, use a reverse proxy:
+
+**Example nginx configuration:**
+```nginx
+server {
+    listen 443 ssl http2;
+    server_name scanner.example.com;
+
+    ssl_certificate /path/to/cert.pem;
+    ssl_certificate_key /path/to/key.pem;
+
+    location / {
+        proxy_pass http://localhost:5000;
+        proxy_set_header Host $host;
+        proxy_set_header X-Real-IP $remote_addr;
+        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
+        proxy_set_header X-Forwarded-Proto $scheme;
+    }
+}
+```
+
+### File Permissions
+
+```bash
+# Ensure proper ownership of data directories
+sudo chown -R $USER:$USER data/ output/ logs/
+
+# Restrict database file permissions
+chmod 600 data/sneakyscanner.db
+
+# Configs should be read-only
+chmod 444 configs/*.yaml
+```
+
+---
+
+## Upgrading
+
+### Upgrading to New Version
+
+```bash
+# 1. Stop the application
+docker compose -f docker-compose-web.yml down
+
+# 2. Backup database
+cp data/sneakyscanner.db data/sneakyscanner.db.backup
+
+# 3. Pull latest code
+git pull origin master
+
+# 4. Rebuild Docker image
+docker compose -f docker-compose-web.yml build
+
+# 5. Run database migrations
+docker compose -f docker-compose-web.yml run --rm web alembic upgrade head
+
+# 6. Start application
+docker compose -f docker-compose-web.yml up -d
+
+# 7. Verify upgrade
+docker compose -f docker-compose-web.yml logs -f
+curl http://localhost:5000/api/settings/health
+```
+
+### Rolling Back
+
+If upgrade fails:
+
+```bash
+# Stop new version
+docker compose -f docker-compose-web.yml down
+
+# Restore database backup
+cp data/sneakyscanner.db.backup data/sneakyscanner.db
+
+# Checkout previous version
+git checkout <previous-version-tag>
+
+# Rebuild and start
+docker compose -f docker-compose-web.yml build
+docker compose -f docker-compose-web.yml up -d
+```
+
+---
+
+## Backup and Restore
+
+### Automated Backup Script
+
+Create `backup.sh`:
+
+```bash
+#!/bin/bash
+BACKUP_DIR="backups/$(date +%Y%m%d_%H%M%S)"
+mkdir -p "$BACKUP_DIR"
+
+# Stop application for consistent backup
+docker compose -f docker-compose-web.yml stop web
+
+# Backup database
+cp data/sneakyscanner.db "$BACKUP_DIR/"
+
+# Backup outputs (last 30 days only)
+find output/ -type f -mtime -30 -exec cp --parents {} "$BACKUP_DIR/" \;
+
+# Backup configs
+cp -r configs/ "$BACKUP_DIR/"
+
+# Restart application
+docker compose -f docker-compose-web.yml start web
+
+echo "Backup complete: $BACKUP_DIR"
+```
+
+Make executable and schedule with cron:
+
+```bash
+chmod +x backup.sh
+
+# Add to crontab (daily at 2 AM)
+crontab -e
+# Add line:
+0 2 * * * /path/to/SneakyScan/backup.sh
+```
+
+### Restore from Backup
+
+```bash
+# Stop application
+docker compose -f docker-compose-web.yml down
+
+# Restore files
+cp backups/YYYYMMDD_HHMMSS/sneakyscanner.db data/
+cp -r backups/YYYYMMDD_HHMMSS/configs/* configs/
+cp -r backups/YYYYMMDD_HHMMSS/output/* output/
+
+# Start application
+docker compose -f docker-compose-web.yml up -d
+```
+
+---
+
+## Support and Further Reading
+
+- **Project README**: `README.md` - General project information
+- **API Documentation**: `docs/ai/API_REFERENCE.md` - REST API reference
+- **Developer Guide**: `docs/ai/DEVELOPMENT.md` - Development setup and architecture
+- **Phase 2 Documentation**: `docs/ai/PHASE2.md` - Implementation details
+- **Issue Tracker**: File bugs and feature requests on GitHub
+
+---
+
+**Last Updated**: 2025-11-14
+**Version**: Phase 2 - Web Application Complete