Files

Phillip Tarrant 6fe24c3907 adding Phase4

2025-11-17 12:05:11 -06:00

45 KiB

Raw Blame History

Phase 4: Config Creator - CSV Upload & Management

Status: Ready to Start Priority: HIGH - Core usability feature Estimated Duration: 4-5 days Dependencies: Phase 2 Complete (REST API, Authentication)

Overview

Phase 4 introduces a Config Creator feature that allows users to manage scan configurations through the web UI instead of manually creating YAML files. This dramatically improves usability by providing:

CSV Template Download - Pre-formatted CSV template for config creation
CSV Upload & Conversion - Upload filled CSV, automatically convert to YAML
YAML Upload - Direct YAML upload for advanced users
Config Management UI - List, view, download, and delete configs
Integration - Seamlessly works with existing scan triggers and schedules

User Workflow

Current State (Manual):

1. User creates YAML file locally (requires YAML knowledge)
2. User uploads to server via Docker volume or container shell
3. User references config in scan trigger/schedule

New Workflow (Phase 4):

1. User clicks "Download CSV Template" in web UI
2. User fills CSV with sites, IPs, expected ports in Excel/Google Sheets
3. User uploads CSV via drag-drop or file picker
4. System validates CSV and converts to YAML
5. User previews generated YAML and confirms
6. Config saved and immediately available for scans/schedules

Design Decisions

Based on project requirements and complexity analysis:

1. CSV Scope: One CSV = One Config ✓

Each CSV file creates a single YAML config file
All rows share the same scan_title (first column)
Simpler to implement and understand
Users create multiple CSVs for multiple configs

2. Creation Methods: CSV/YAML Upload Only ✓

CSV upload with conversion (primary method)
Direct YAML upload (for advanced users)
Form-based editor deferred to future phase
Focused scope for faster delivery

3. Versioning: No Version History ✓

Configs overwrite when updated (no .bak files)
Simpler implementation, less storage
Users can download existing configs before editing

4. Deletion Safety: Block if Used by Schedules ✓

Prevent deletion of configs referenced by active schedules
Show error message listing which schedules use the config
Safest approach, prevents schedule failures

5. Additional Scope Decisions

File naming: Auto-generated from scan title (sanitized)
File extensions: Accept .yaml, .yml for uploads
CSV export: Not in Phase 4 (future enhancement)
Config editing: Download → Edit locally → Re-upload (no inline editor yet)

CSV Template Specification

CSV Format

Columns (in order):

scan_title,site_name,ip_address,ping_expected,tcp_ports,udp_ports,services

Example CSV:

scan_title,site_name,ip_address,ping_expected,tcp_ports,udp_ports,services
Sneaky Infra Scan,Production Web Servers,10.10.20.4,true,"22,53,80",53,"ssh,domain,http"
Sneaky Infra Scan,Production Web Servers,10.10.20.11,true,"22,111,3128,8006","","ssh,rpcbind,squid"
Sneaky Infra Scan,Database Servers,10.10.30.5,true,"22,3306","",""
Sneaky Infra Scan,Database Servers,10.10.30.6,false,"22,5432","",""

Field Specifications

Column	Type	Required	Format	Example	Description
`scan_title`	string	Yes	Any text	`"Sneaky Infra Scan"`	Becomes YAML `title` field. Must be same for all rows.
`site_name`	string	Yes	Any text	`"Production Web Servers"`	Logical grouping. Rows with same site_name are grouped together.
`ip_address`	string	Yes	IPv4 or IPv6	`"10.10.20.4"`	Target IP address to scan.
`ping_expected`	boolean	No	`true`/`false` (case-insensitive)	`true`	Whether host should respond to ping. Default: `false`
`tcp_ports`	list[int]	No	Comma-separated, quoted if multiple	`"22,80,443"` or `22`	Expected TCP ports. Empty = no expected TCP ports.
`udp_ports`	list[int]	No	Comma-separated, quoted if multiple	`"53,123"` or `53`	Expected UDP ports. Empty = no expected UDP ports.
`services`	list[str]	No	Comma-separated, quoted if multiple	`"ssh,http,https"`	Expected service names (optional).

Validation Rules

Row-level validation:

scan_title must be non-empty and same across all rows
site_name must be non-empty
ip_address must be valid IPv4 or IPv6 format
ping_expected must be true, false, TRUE, FALSE, or empty (default false)
Port numbers must be integers 1-65535
Port lists must be comma-separated (spaces optional)
Duplicate IPs within same config are allowed (different expected values)

File-level validation:

CSV must have exactly 7 columns with correct headers
Must have at least 1 data row (besides header)
Must have at least 1 site defined
Must have at least 1 IP per site

Filename validation:

Generated filename from scan_title (lowercase, spaces→hyphens, special chars removed)
Must not conflict with existing config files
Max filename length: 200 characters

CSV-to-YAML Conversion Logic

Python pseudocode:

def csv_to_yaml(csv_content: str) -> str:
    rows = parse_csv(csv_content)

    # Extract scan title (same for all rows)
    scan_title = rows[0]['scan_title']

    # Group rows by site_name
    sites = {}
    for row in rows:
        site_name = row['site_name']
        if site_name not in sites:
            sites[site_name] = []

        # Parse ports and services
        tcp_ports = parse_port_list(row['tcp_ports'])
        udp_ports = parse_port_list(row['udp_ports'])
        services = parse_string_list(row['services'])
        ping = parse_bool(row['ping_expected'], default=False)

        sites[site_name].append({
            'address': row['ip_address'],
            'expected': {
                'ping': ping,
                'tcp_ports': tcp_ports,
                'udp_ports': udp_ports,
                'services': services  # Optional, omit if empty
            }
        })

    # Build YAML structure
    yaml_data = {
        'title': scan_title,
        'sites': [
            {
                'name': site_name,
                'ips': ips
            }
            for site_name, ips in sites.items()
        ]
    }

    return yaml.dump(yaml_data, sort_keys=False)

Example Conversion:

Input CSV:

scan_title,site_name,ip_address,ping_expected,tcp_ports,udp_ports,services
Prod Scan,Web Servers,10.10.20.4,true,"22,80,443",53,ssh
Prod Scan,Web Servers,10.10.20.5,true,"22,80",,"ssh,http"

Output YAML:

title: Prod Scan
sites:
  - name: Web Servers
    ips:
      - address: 10.10.20.4
        expected:
          ping: true
          tcp_ports: [22, 80, 443]
          udp_ports: [53]
          services: [ssh]
      - address: 10.10.20.5
        expected:
          ping: true
          tcp_ports: [22, 80]
          udp_ports: []
          services: [ssh, http]

Architecture

Backend Components

1. API Blueprint: `web/api/configs.py`

New endpoints:

Method	Endpoint	Description	Auth	Request Body	Response
`GET`	`/api/configs`	List all config files	Required	-	`{ "configs": [{filename, title, path, created_at, used_by_schedules}] }`
`GET`	`/api/configs/<filename>`	Get config content (YAML)	Required	-	`{ "filename": "...", "content": "...", "parsed": {...} }`
`POST`	`/api/configs/upload-csv`	Upload CSV and convert to YAML	Required	`multipart/form-data` with file	`{ "filename": "...", "preview": "...", "success": true }`
`POST`	`/api/configs/upload-yaml`	Upload YAML directly	Required	`multipart/form-data` with file	`{ "filename": "...", "success": true }`
`GET`	`/api/configs/template`	Download CSV template	Required	-	CSV file download
`DELETE`	`/api/configs/<filename>`	Delete config file	Required	-	`{ "success": true }` or error if used by schedules

Error responses:

400 - Invalid CSV format, validation errors
404 - Config file not found
409 - Config filename conflict
422 - Cannot delete (used by schedules)
500 - Server error

Blueprint structure:

# web/api/configs.py
from flask import Blueprint, jsonify, request, send_file
from werkzeug.utils import secure_filename
from web.auth.decorators import api_auth_required
from web.services.config_service import ConfigService
import logging

bp = Blueprint('configs', __name__)
logger = logging.getLogger(__name__)

@bp.route('', methods=['GET'])
@api_auth_required
def list_configs():
    """List all config files with metadata"""
    pass

@bp.route('/<filename>', methods=['GET'])
@api_auth_required
def get_config(filename: str):
    """Get config file content"""
    pass

@bp.route('/upload-csv', methods=['POST'])
@api_auth_required
def upload_csv():
    """Upload CSV and convert to YAML"""
    pass

@bp.route('/upload-yaml', methods=['POST'])
@api_auth_required
def upload_yaml():
    """Upload YAML file directly"""
    pass

@bp.route('/template', methods=['GET'])
@api_auth_required
def download_template():
    """Download CSV template"""
    pass

@bp.route('/<filename>', methods=['DELETE'])
@api_auth_required
def delete_config(filename: str):
    """Delete config file"""
    pass

2. Service Layer: `web/services/config_service.py`

Class definition:

class ConfigService:
    """Business logic for config management"""

    def __init__(self, configs_dir: str = '/app/configs'):
        self.configs_dir = configs_dir

    def list_configs(self) -> List[Dict[str, Any]]:
        """
        List all config files with metadata.

        Returns:
            [
                {
                    "filename": "prod-scan.yaml",
                    "title": "Prod Scan",
                    "path": "/app/configs/prod-scan.yaml",
                    "created_at": "2025-11-15T10:30:00Z",
                    "size_bytes": 1234,
                    "used_by_schedules": ["Daily Scan", "Weekly Audit"]
                }
            ]
        """
        pass

    def get_config(self, filename: str) -> Dict[str, Any]:
        """
        Get config file content and parsed data.

        Returns:
            {
                "filename": "prod-scan.yaml",
                "content": "title: Prod Scan\n...",
                "parsed": {"title": "Prod Scan", "sites": [...]}
            }
        """
        pass

    def create_from_yaml(self, filename: str, content: str) -> str:
        """
        Create config from YAML content.

        Args:
            filename: Desired filename (will be sanitized)
            content: YAML content string

        Returns:
            Final filename

        Raises:
            ValueError: If content invalid or filename conflict
        """
        pass

    def create_from_csv(self, csv_file, suggested_filename: str = None) -> Tuple[str, str]:
        """
        Create config from CSV file.

        Args:
            csv_file: File object from request.files
            suggested_filename: Optional filename (else auto-generate from title)

        Returns:
            (final_filename, yaml_preview)

        Raises:
            ValueError: If CSV invalid
        """
        pass

    def delete_config(self, filename: str) -> None:
        """
        Delete config file if not used by schedules.

        Raises:
            FileNotFoundError: If config doesn't exist
            ValueError: If config used by active schedules
        """
        pass

    def validate_config_content(self, content: Dict) -> Tuple[bool, str]:
        """
        Validate parsed YAML config structure.

        Returns:
            (is_valid, error_message)
        """
        pass

    def get_schedules_using_config(self, filename: str) -> List[str]:
        """
        Get list of schedule names using this config.

        Returns:
            ["Daily Scan", "Weekly Audit"]
        """
        pass

    def generate_filename_from_title(self, title: str) -> str:
        """
        Generate safe filename from scan title.

        Example: "Prod Scan 2025" -> "prod-scan-2025.yaml"
        """
        pass

3. CSV Parser: `web/utils/csv_parser.py`

Class definition:

class CSVConfigParser:
    """Parse and validate CSV config files"""

    REQUIRED_COLUMNS = [
        'scan_title', 'site_name', 'ip_address',
        'ping_expected', 'tcp_ports', 'udp_ports', 'services'
    ]

    def __init__(self):
        pass

    def parse_csv_to_yaml(self, csv_file) -> str:
        """
        Convert CSV file to YAML string.

        Args:
            csv_file: File object or file path

        Returns:
            YAML string

        Raises:
            ValueError: If CSV invalid
        """
        pass

    def validate_csv_structure(self, csv_file) -> Tuple[bool, List[str]]:
        """
        Validate CSV structure and content.

        Returns:
            (is_valid, error_messages)
        """
        pass

    def _parse_port_list(self, value: str) -> List[int]:
        """Parse comma-separated port list"""
        pass

    def _parse_string_list(self, value: str) -> List[str]:
        """Parse comma-separated string list"""
        pass

    def _parse_bool(self, value: str, default: bool = False) -> bool:
        """Parse boolean value (true/false/1/0)"""
        pass

    def _validate_ip_address(self, ip: str) -> bool:
        """Validate IPv4/IPv6 address format"""
        pass

    def _validate_port(self, port: int) -> bool:
        """Validate port number (1-65535)"""
        pass

4. Template Generator: `web/utils/template_generator.py`

Function:

def generate_csv_template() -> str:
    """
    Generate CSV template with headers and example rows.

    Returns:
        CSV string with headers and 2 example rows
    """
    template = [
        ['scan_title', 'site_name', 'ip_address', 'ping_expected', 'tcp_ports', 'udp_ports', 'services'],
        ['Example Infrastructure Scan', 'Production Web Servers', '10.10.20.4', 'true', '22,80,443', '53', 'ssh,http,https'],
        ['Example Infrastructure Scan', 'Production Web Servers', '10.10.20.5', 'true', '22,3306', '', 'ssh,mysql'],
    ]

    output = io.StringIO()
    writer = csv.writer(output)
    writer.writerows(template)
    return output.getvalue()

Frontend Components

1. New Route: Configs Management Page

File: web/routes/main.py

@bp.route('/configs')
@login_required
def configs():
    """Config management page"""
    return render_template('configs.html')

@bp.route('/configs/upload')
@login_required
def upload_config():
    """Config upload page"""
    return render_template('config_upload.html')

2. Template: Config List Page

File: web/templates/configs.html

Features:

Table listing all configs with columns:
- Filename
- Title (from YAML)
- Created date
- Size
- Used by schedules (badge count)
- Actions (view, download, delete)
"Create New Config" button → redirects to upload page
"Download CSV Template" button
Delete confirmation modal
Search/filter box (client-side)

Layout:

{% extends "base.html" %}

{% block content %}
<div class="container mt-4">
    <div class="d-flex justify-content-between align-items-center mb-4">
        <h2>Configuration Files</h2>
        <div>
            <a href="{{ url_for('main.upload_config') }}" class="btn btn-primary">
                <i class="bi bi-plus-circle"></i> Create New Config
            </a>
            <a href="{{ url_for('api_configs.download_template') }}"
               class="btn btn-outline-secondary" download>
                <i class="bi bi-download"></i> Download CSV Template
            </a>
        </div>
    </div>

    <div class="card">
        <div class="card-body">
            <div class="mb-3">
                <input type="text" id="search" class="form-control"
                       placeholder="Search configs...">
            </div>

            <table class="table table-hover" id="configs-table">
                <thead>
                    <tr>
                        <th>Filename</th>
                        <th>Title</th>
                        <th>Created</th>
                        <th>Size</th>
                        <th>Used By</th>
                        <th>Actions</th>
                    </tr>
                </thead>
                <tbody>
                    <!-- Populated via JavaScript -->
                </tbody>
            </table>
        </div>
    </div>
</div>

<!-- Delete Confirmation Modal -->
<div class="modal fade" id="deleteModal" tabindex="-1">
    <!-- Modal content -->
</div>
{% endblock %}

3. Template: Config Upload Page

File: web/templates/config_upload.html

Features:

Two upload methods (tabs):
- Tab 1: CSV Upload (default)
  - Drag-drop zone or file picker (.csv only)
  - Instructions: "Download template, fill with your data, upload here"
  - Preview pane showing generated YAML after upload
  - "Save Config" button (disabled until valid upload)
- Tab 2: YAML Upload (for advanced users)
  - Drag-drop zone or file picker (.yaml, .yml only)
  - Direct upload without conversion
Real-time validation feedback
Error messages with specific issues
Success message with link to configs list

Layout:

{% extends "base.html" %}

{% block content %}
<div class="container mt-4">
    <h2>Create New Configuration</h2>

    <ul class="nav nav-tabs mb-4" id="uploadTabs" role="tablist">
        <li class="nav-item">
            <a class="nav-link active" id="csv-tab" data-bs-toggle="tab"
               href="#csv" role="tab">Upload CSV</a>
        </li>
        <li class="nav-item">
            <a class="nav-link" id="yaml-tab" data-bs-toggle="tab"
               href="#yaml" role="tab">Upload YAML</a>
        </li>
    </ul>

    <div class="tab-content" id="uploadTabsContent">
        <!-- CSV Upload Tab -->
        <div class="tab-pane fade show active" id="csv" role="tabpanel">
            <div class="row">
                <div class="col-md-6">
                    <div class="card">
                        <div class="card-body">
                            <h5>Step 1: Upload CSV</h5>
                            <p class="text-muted">
                                <a href="{{ url_for('api_configs.download_template') }}" download>
                                    Download template
                                </a> first if you haven't already.
                            </p>

                            <div id="csv-dropzone" class="dropzone">
                                <i class="bi bi-cloud-upload"></i>
                                <p>Drag & drop CSV file here or click to browse</p>
                                <input type="file" id="csv-file-input" accept=".csv" hidden>
                            </div>

                            <div id="csv-errors" class="alert alert-danger mt-3" style="display:none;">
                            </div>
                        </div>
                    </div>
                </div>

                <div class="col-md-6">
                    <div class="card">
                        <div class="card-body">
                            <h5>Step 2: Preview & Save</h5>
                            <div id="yaml-preview" style="display:none;">
                                <pre><code id="yaml-content" class="language-yaml"></code></pre>
                                <button id="save-config-btn" class="btn btn-success mt-3">
                                    Save Configuration
                                </button>
                            </div>
                            <div id="preview-placeholder" class="text-muted text-center py-5">
                                Upload a CSV file to see preview
                            </div>
                        </div>
                    </div>
                </div>
            </div>
        </div>

        <!-- YAML Upload Tab -->
        <div class="tab-pane fade" id="yaml" role="tabpanel">
            <div class="card">
                <div class="card-body">
                    <h5>Upload YAML Configuration</h5>
                    <p class="text-muted">
                        For advanced users: upload a YAML config file directly.
                    </p>

                    <div id="yaml-dropzone" class="dropzone">
                        <i class="bi bi-cloud-upload"></i>
                        <p>Drag & drop YAML file here or click to browse</p>
                        <input type="file" id="yaml-file-input" accept=".yaml,.yml" hidden>
                    </div>

                    <button id="upload-yaml-btn" class="btn btn-primary mt-3" disabled>
                        Upload YAML
                    </button>

                    <div id="yaml-errors" class="alert alert-danger mt-3" style="display:none;">
                    </div>
                </div>
            </div>
        </div>
    </div>
</div>
{% endblock %}

4. JavaScript: Config Manager

File: web/static/js/config-manager.js

Functions:

class ConfigManager {
    constructor() {
        this.apiBase = '/api/configs';
    }

    // List configs and populate table
    async loadConfigs() {
        const response = await fetch(this.apiBase);
        const data = await response.json();
        this.renderConfigsTable(data.configs);
    }

    // Upload CSV file
    async uploadCSV(file) {
        const formData = new FormData();
        formData.append('file', file);

        const response = await fetch(`${this.apiBase}/upload-csv`, {
            method: 'POST',
            body: formData
        });

        if (!response.ok) {
            const error = await response.json();
            throw new Error(error.message);
        }

        return await response.json();
    }

    // Upload YAML file
    async uploadYAML(file, filename) {
        // Similar to uploadCSV
    }

    // Delete config
    async deleteConfig(filename) {
        const response = await fetch(`${this.apiBase}/${filename}`, {
            method: 'DELETE'
        });

        if (!response.ok) {
            const error = await response.json();
            throw new Error(error.message);
        }

        return await response.json();
    }

    // Show preview of YAML
    showPreview(yamlContent) {
        document.getElementById('yaml-content').textContent = yamlContent;
        document.getElementById('yaml-preview').style.display = 'block';
        document.getElementById('preview-placeholder').style.display = 'none';
    }

    // Render configs table
    renderConfigsTable(configs) {
        // Populate table with config data
    }

    // Client-side search filter
    filterConfigs(searchTerm) {
        // Filter table rows by search term
    }
}

// Drag-drop handlers
function setupDropzone(elementId, fileInputId, fileType) {
    const dropzone = document.getElementById(elementId);
    const fileInput = document.getElementById(fileInputId);

    dropzone.addEventListener('click', () => fileInput.click());

    dropzone.addEventListener('dragover', (e) => {
        e.preventDefault();
        dropzone.classList.add('dragover');
    });

    dropzone.addEventListener('drop', (e) => {
        e.preventDefault();
        dropzone.classList.remove('dragover');
        const file = e.dataTransfer.files[0];
        handleFileUpload(file, fileType);
    });

    fileInput.addEventListener('change', (e) => {
        const file = e.target.files[0];
        handleFileUpload(file, fileType);
    });
}

5. CSS Styling

File: web/static/css/config-manager.css

.dropzone {
    border: 2px dashed #6c757d;
    border-radius: 8px;
    padding: 40px;
    text-align: center;
    cursor: pointer;
    transition: all 0.3s ease;
    background-color: #f8f9fa;
}

.dropzone:hover,
.dropzone.dragover {
    border-color: #0d6efd;
    background-color: #e7f1ff;
}

.dropzone i {
    font-size: 48px;
    color: #6c757d;
    margin-bottom: 16px;
}

#yaml-preview pre {
    background-color: #f8f9fa;
    border: 1px solid #dee2e6;
    border-radius: 4px;
    padding: 16px;
    max-height: 400px;
    overflow-y: auto;
}

.config-actions {
    white-space: nowrap;
}

.config-actions .btn {
    margin-right: 4px;
}

.schedule-badge {
    background-color: #0d6efd;
    color: white;
    padding: 4px 8px;
    border-radius: 4px;
    font-size: 0.85em;
}

Update: web/templates/base.html

Add "Configs" link to navigation menu:

<nav class="navbar navbar-expand-lg navbar-dark bg-dark">
    <div class="container-fluid">
        <a class="navbar-brand" href="/">SneakyScanner</a>
        <ul class="navbar-nav">
            <li class="nav-item">
                <a class="nav-link" href="{{ url_for('main.dashboard') }}">Dashboard</a>
            </li>
            <li class="nav-item">
                <a class="nav-link" href="{{ url_for('main.scans') }}">Scans</a>
            </li>
            <li class="nav-item">
                <a class="nav-link" href="{{ url_for('main.schedules') }}">Schedules</a>
            </li>
            <li class="nav-item">
                <a class="nav-link" href="{{ url_for('main.configs') }}">Configs</a> <!-- NEW -->
            </li>
            <li class="nav-item">
                <a class="nav-link" href="{{ url_for('main.settings') }}">Settings</a>
            </li>
        </ul>
    </div>
</nav>

Implementation Tasks

Task Breakdown (15 tasks)

Backend (8 tasks)

Create CSV parser utility (web/utils/csv_parser.py)
- Implement CSVConfigParser class
- Parse CSV rows into dict structure
- Validate CSV structure (columns, data types)
- Handle edge cases (empty cells, quotes, commas in values)
- Write unit tests (10+ test cases)
Create template generator (web/utils/template_generator.py)
- Generate CSV template with headers + example rows
- Write unit tests
Create config service (web/services/config_service.py)
- Implement ConfigService class with all methods
- CSV-to-YAML conversion logic
- Filename sanitization and conflict detection
- Schedule dependency checking
- File operations (read, write, delete)
- Write unit tests (15+ test cases)
Create configs API blueprint (web/api/configs.py)
- Implement 6 API endpoints
- Error handling with proper HTTP status codes
- Request validation
- File upload handling (multipart/form-data)
- File download headers
- Integrate with ConfigService
Register configs blueprint (web/app.py)
- Import and register blueprint
- Add to API routes
Add file upload limits (web/app.py)
- Set MAX_CONTENT_LENGTH for uploads (2MB for CSV/YAML)
- Add file size validation
Update existing services (if needed)
- schedule_service.py: Add config dependency check method
- Ensure config validation called before scan trigger
Write integration tests (tests/test_config_api.py)
- Test all API endpoints (20+ test cases)
- Test CSV upload → conversion → scan trigger workflow
- Test error cases (invalid CSV, filename conflict, delete protection)
- Test file download

Frontend (5 tasks)

Create configs list template (web/templates/configs.html)
- Table layout with columns
- Action buttons (view, download, delete)
- Delete confirmation modal
- Search box
- Links to upload page and template download
Create config upload template (web/templates/config_upload.html)
- Two-tab interface (CSV upload, YAML upload)
- Drag-drop zones for both tabs
- YAML preview pane
- Error message displays
- Success/failure feedback
Add routes to main blueprint (web/routes/main.py)
- /configs - List configs
- /configs/upload - Upload page
Create JavaScript module (web/static/js/config-manager.js)
- ConfigManager class
- API fetch wrappers for all endpoints
- Drag-drop handlers
- File upload with FormData
- Table rendering and search filtering
- Preview display
- Error display helpers
Create CSS styling (web/static/css/config-manager.css)
- Dropzone styling with hover effects
- Preview pane styling
- Table action buttons
- Responsive layout adjustments

Integration & Documentation (2 tasks)

Update navigation (web/templates/base.html)
- Add "Configs" link to navbar
- Set active state based on current route
Update documentation
- Update README.md with Config Creator section
- Update API_REFERENCE.md with new endpoints
- Create this document (Phase4.md) ✓

Testing Strategy

Unit Tests

File: tests/test_csv_parser.py

Test cases (10+):

Valid CSV with multiple sites and IPs
Single site, single IP
Empty optional fields (udp_ports, services)
Boolean parsing (true/false/TRUE/FALSE)
Port list parsing (single port, multiple ports, quoted/unquoted)
Invalid IP addresses
Invalid port numbers (0, 65536, negative)
Missing required columns
Inconsistent scan_title across rows
Empty CSV (no data rows)
Malformed CSV (wrong number of columns)

File: tests/test_config_service.py

Test cases (15+):

List configs (empty directory, multiple configs)
Get config (valid, non-existent)
Create from YAML (valid, invalid syntax, duplicate filename)
Create from CSV (valid, invalid CSV, duplicate filename)
Delete config (valid, non-existent, used by schedule)
Validate config content (valid, missing title, missing sites, invalid structure)
Get schedules using config (none, multiple)
Generate filename from title (various titles, special characters, long titles)

Integration Tests

File: tests/test_config_api.py

Test cases (20+):

GET /api/configs
- List configs successfully
- Empty list when no configs exist
- Includes schedule usage counts
GET /api/configs/
- Get existing config
- 404 for non-existent config
- Returns parsed YAML data
POST /api/configs/upload-csv
- Upload valid CSV → creates YAML
- Upload invalid CSV → 400 error with details
- Upload CSV with duplicate filename → 409 error
- Upload non-CSV file → 400 error
- Upload file too large → 413 error
POST /api/configs/upload-yaml
- Upload valid YAML → creates config
- Upload invalid YAML syntax → 400 error
- Upload YAML missing required fields → 400 error
GET /api/configs/template
- Download CSV template
- Returns text/csv mimetype
- Template has correct headers and examples
DELETE /api/configs/
- Delete unused config → success
- Delete used-by-schedule config → 422 error with schedule list
- Delete non-existent config → 404 error
Authentication
- All endpoints require auth
- Unauthenticated requests → 401 error

End-to-End Tests

File: tests/test_config_workflow.py

Test cases (5):

Complete CSV workflow:
- Download template
- Upload modified CSV
- Verify YAML created correctly
- Trigger scan with new config
- Verify scan completes successfully
Config deletion protection:
- Create config
- Create schedule using config
- Attempt to delete config → fails with error
- Disable schedule
- Delete config → succeeds
Filename conflict handling:
- Create config "test-scan.yaml"
- Upload CSV with same title
- Verify error returned
- User changes filename → succeeds
YAML direct upload:
- Upload valid YAML
- Config immediately usable
- Trigger scan → works
CSV validation errors:
- Upload CSV with invalid IP
- Verify clear error message returned
- Fix CSV and re-upload → succeeds

Manual Testing Checklist

UI/UX:

Drag-drop file upload works
File picker works
Preview shows correct YAML
Error messages are clear and actionable
Success messages appear after save
Table search/filter works
Delete confirmation modal works
Navigation links work
Responsive layout on mobile

File Handling:

CSV template downloads correctly
CSV with special characters (commas, quotes) parses correctly
Large CSV files upload successfully
YAML files with UTF-8 characters work
Generated YAML is valid and scanner accepts it

Error Cases:

Invalid CSV format shows helpful error
Duplicate filename shows conflict error
Delete protected config shows which schedules use it
Network errors handled gracefully
File too large shows size limit error

Security Considerations

Input Validation

Filename sanitization:
- Use werkzeug.utils.secure_filename()
- Remove path traversal attempts (../, ./)
- Limit filename length (200 chars)
- Only allow alphanumeric, hyphens, underscores
File type validation:
- Check file extension (.csv, .yaml, .yml)
- Verify MIME type matches extension
- Reject executable or script file extensions
CSV content validation:
- Validate all IPs with ipaddress module
- Validate port ranges (1-65535)
- Limit CSV row count (max 1000 rows)
- Limit CSV file size (max 2MB)
YAML parsing security:
- Always use yaml.safe_load() (never yaml.load())
- Prevents arbitrary code execution
- Only load basic data types (dict, list, str, int, bool)

Access Control

Authentication required:
- All API endpoints require @api_auth_required
- All web routes require @login_required
- Single-user model (no multi-tenant concerns)
File system access:
- Restrict all operations to /app/configs/ directory
- Validate no path traversal in any file operations
- Use os.path.join() and os.path.abspath() safely

File Upload Security

Size limits:
- CSV: 2MB max
- YAML: 2MB max
- Configurable in Flask config
Rate limiting (future consideration):
- Limit upload frequency per session
- Prevent DoS via repeated large uploads
Virus scanning (future consideration):
- For production deployments
- Scan uploaded files with ClamAV

Database Changes

No database schema changes required for Phase 4.

Configs are stored as files, not in database. However, for future enhancement, consider adding:

# Optional future enhancement (not Phase 4)
class Config(Base):
    __tablename__ = 'configs'

    id = Column(Integer, primary_key=True)
    filename = Column(String(255), unique=True, nullable=False, index=True)
    title = Column(String(255), nullable=False)
    created_at = Column(DateTime, default=datetime.utcnow, nullable=False)
    updated_at = Column(DateTime, default=datetime.utcnow, onupdate=datetime.utcnow)
    created_by = Column(String(50))  # 'csv', 'yaml', 'manual'
    file_hash = Column(String(64))  # SHA256 hash for change detection

Benefits of database tracking:

Faster metadata queries (no need to parse YAML for title)
Change detection and versioning
Usage statistics (how often config used)
Search and filtering

Decision: Keep Phase 4 simple (file-based only). Add database tracking in Phase 5+ if needed.

Integration with Existing Features

Dashboard Integration

Current: Dashboard has "Run Scan Now" button with config dropdown

Change: Config dropdown populated via /api/configs instead of filesystem scan

File: web/templates/dashboard.html

// Before (Phase 3)
fetch('/api/configs-list')  // Doesn't exist yet

// After (Phase 4)
fetch('/api/configs')
    .then(res => res.json())
    .then(data => {
        const select = document.getElementById('config-select');
        data.configs.forEach(config => {
            const option = new Option(config.title, config.filename);
            select.add(option);
        });
    });

Schedule Management Integration

Current: Schedule form has config file input (text field or dropdown)

Change: Config selector uses /api/configs to show available configs

File: web/templates/schedule_form.html

<!-- Before -->
<input type="text" name="config_file" placeholder="/app/configs/example.yaml">

<!-- After -->
<select name="config_file" id="config-select">
    <option value="">Select a configuration...</option>
    <!-- Populated via JavaScript from /api/configs -->
</select>
<p class="text-muted">
    Don't see your config? <a href="{{ url_for('main.upload_config') }}">Create a new one</a>
</p>

Scan Trigger Integration

Current: Scan trigger validates config file exists

Change: No changes needed, validation already in place via validators.validate_config_file()

File: web/services/scan_service.py

def trigger_scan(self, config_file: str, triggered_by: str = 'manual'):
    # Existing validation
    is_valid, error = validate_config_file(config_file)
    if not is_valid:
        raise ValueError(f"Invalid config file: {error}")

    # Continue with scan...

Success Criteria

Phase 4 Complete When:

CSV Template Download:
- ✓ User can download CSV template from UI
- ✓ Template includes headers and example rows
- ✓ Template format matches specification
CSV Upload & Conversion:
- ✓ User can upload CSV via drag-drop or file picker
- ✓ CSV validates correctly (structure, data types, IPs, ports)
- ✓ CSV converts to valid YAML
- ✓ Generated YAML matches expected structure
- ✓ Config saved to /app/configs/ directory
YAML Upload:
- ✓ User can upload YAML directly
- ✓ YAML validates before saving
- ✓ Invalid YAML shows clear error message
Config Management:
- ✓ User can list all configs with metadata
- ✓ User can view config content (YAML)
- ✓ User can download existing configs
- ✓ User can delete unused configs
- ✓ Deletion blocked if config used by schedules
Integration:
- ✓ Generated configs work with scan trigger
- ✓ Configs appear in dashboard scan selector
- ✓ Configs appear in schedule form selector
- ✓ No breaking changes to existing workflows
Testing:
- ✓ All unit tests pass (25+ tests)
- ✓ All integration tests pass (20+ tests)
- ✓ Manual testing checklist complete
Documentation:
- ✓ README.md updated with Config Creator section
- ✓ API_REFERENCE.md updated with new endpoints
- ✓ Phase4.md created (this document)

Timeline Estimate

Total Duration: 4-5 days

Day 1: Backend Foundation (CSV Parser & Service)

Create csv_parser.py with CSVConfigParser class (3 hours)
Write unit tests for CSV parser (2 hours)
Create template_generator.py (1 hour)
Create config_service.py skeleton (2 hours)

Day 2: Backend Implementation (Config Service & API)

Implement all ConfigService methods (4 hours)
Write unit tests for ConfigService (3 hours)
Create configs.py API blueprint (2 hours)

Day 3: Backend Testing & Frontend Start

Write integration tests for API endpoints (3 hours)
Create configs.html template (2 hours)
Create config_upload.html template (2 hours)
Add routes to main.py (1 hour)

Day 4: Frontend JavaScript & Styling

Create config-manager.js with all functionality (4 hours)
Implement drag-drop handlers (2 hours)
Create config-manager.css styling (1 hour)
Update navigation in base.html (30 min)
Manual testing and bug fixes (2 hours)

Day 5: Integration & Documentation

End-to-end testing (2 hours)
Update dashboard/schedule integration (2 hours)
Update README.md and API_REFERENCE.md (2 hours)
Final manual testing checklist (2 hours)
Bug fixes and polish (2 hours)

Buffer: +1 day for unexpected issues or scope additions

Future Enhancements (Post-Phase 4)

Not in scope for Phase 4, but consider for future phases:

Config Editor UI (Phase 5+):
- Web form to create configs without CSV
- Add/remove sites and IPs dynamically
- Port picker with common presets
- Inline YAML editor with syntax highlighting
Config Versioning (Phase 5+):
- Track config changes over time
- Compare versions (diff view)
- Rollback to previous version
- Store versions in database
CSV Export (Phase 5+):
- Export existing YAML configs to CSV
- Edit in Excel and re-upload
- Useful for bulk updates
Config Templates (Phase 6+):
- Pre-built templates for common scenarios:
  - Web server infrastructure
  - Database cluster
  - Network devices
- User selects template, fills IPs, done
Bulk Import (Phase 6+):
- Upload multiple CSV files at once
- ZIP archive with multiple configs
- Import from external sources (CMDB, spreadsheet)
Config Validation on Schedule (Phase 6+):
- Periodic validation of all configs
- Alert if config becomes invalid (file deleted, corrupted)
- Auto-disable schedules with invalid configs
Config Metadata & Tags (Phase 6+):
- Add tags/labels to configs (environment, team, etc.)
- Filter/search by tags
- Group related configs
Config Diff Tool (Phase 6+):
- Compare two configs side-by-side
- Highlight differences (IPs, ports, sites)
- Useful for environment parity checks

Open Questions

Resolved:

✓ One CSV = one config or multiple? One config
✓ Form editor or upload only? Upload only (form later)
✓ Config versioning? No (Phase 4), maybe later
✓ Delete protection? Block if used by schedules

Remaining:

Filename handling: Auto-generate from title or let user specify?
- Recommendation: Auto-generate with option to customize (add filename input field)
Duplicate IP handling: Allow or reject duplicate IPs in same config?
- Recommendation: Allow (user might scan same IP with different expected ports)
Config validation frequency: Validate on upload only, or re-validate periodically?
- Recommendation: Upload only (Phase 4), periodic validation in Phase 6+
CSV encoding: Support only UTF-8 or other encodings?
- Recommendation: UTF-8 only (standard, avoids complexity)

Risk Assessment

Low Risk:

CSV parsing (standard library, well-tested)
File upload handling (Flask/werkzeug built-in)
YAML generation (PyYAML library)

Medium Risk:

Complex CSV validation edge cases (commas in values, quotes)
- Mitigation: Comprehensive unit tests, use csv.reader()
Filename conflicts and race conditions
- Mitigation: Check existence before write, use atomic operations

High Risk:

Breaking existing scan/schedule workflows
- Mitigation: Extensive integration tests, no changes to existing validation
Security vulnerabilities (path traversal, code injection)
- Mitigation: Use secure_filename(), yaml.safe_load(), input validation

Contingency:

If CSV parsing too complex: Start with YAML upload only, add CSV in Phase 4.5
If schedule deletion check too slow: Cache schedule-config mappings
If file-based config management becomes bottleneck: Migrate to database in Phase 5

Deployment Notes

Requirements:

No new Python dependencies (csv, yaml, os, io all in stdlib)
No new system dependencies
No database migrations

Configuration:

Add to .env (optional):

# Config upload limits
MAX_CONFIG_SIZE_MB=2
MAX_CSV_ROWS=1000
CONFIGS_DIR=/app/configs

Deployment Steps:

Pull latest code
Restart Flask app: docker-compose -f docker-compose-web.yml restart web
Verify /api/configs/template endpoint works
Test CSV upload with template

Rollback Plan:

No database changes, so rollback is safe
Revert code changes and restart
Configs created during Phase 4 remain valid (files are backward compatible)

References

Phase 2 Complete - REST API patterns, authentication
API Reference - Existing API structure
Deployment Guide - Production deployment

External Resources:

Code References:

src/scanner.py:42-54 - Config loading and validation
web/utils/validators.py:14-85 - Existing validation patterns
web/services/scan_service.py - Scan trigger with config validation
web/api/scans.py - API endpoint patterns

Changelog

Date	Version	Changes
2025-11-17	1.0	Initial Phase 4 plan created based on user requirements and design decisions

Last Updated: 2025-11-17 Next Review: After Phase 4 implementation complete Approval Status: Ready for Implementation

Phase 4 Goal: Enable non-technical users to create scan configurations via CSV upload, eliminating the need for manual YAML editing and server file access.

45 KiB Raw Blame History