diff --git a/configs/dmz.yaml b/configs/dmz.yaml new file mode 100644 index 0000000..c818d63 --- /dev/null +++ b/configs/dmz.yaml @@ -0,0 +1,14 @@ +title: DMZ +sites: +- name: DMZ Reverse Proxys + ips: + - address: 10.10.99.10 + expected: + ping: true + tcp_ports: [22,80,443] + udp_ports: [] + - address: 10.10.99.20 + expected: + ping: true + tcp_ports: [22,80,443] + udp_ports: [] \ No newline at end of file diff --git a/docker-compose-web.yml b/docker-compose-web.yml index 05f9e7a..99cc680 100644 --- a/docker-compose-web.yml +++ b/docker-compose-web.yml @@ -11,8 +11,8 @@ services: # Note: Using host network mode for scanner capabilities, so no port mapping needed # The Flask app will be accessible at http://localhost:5000 volumes: - # Mount configs directory (read-only) for scan configurations - - ./configs:/app/configs:ro + # Mount configs directory for scan configurations (read-write for web UI management) + - ./configs:/app/configs # Mount output directory for scan results - ./output:/app/output # Mount database file for persistence diff --git a/docs/ai/Phase4.md b/docs/ai/Phase4.md index af693f3..abb8a6a 100644 --- a/docs/ai/Phase4.md +++ b/docs/ai/Phase4.md @@ -1,8 +1,8 @@ -# Phase 4: Config Creator - CSV Upload & Management +# Phase 4: Config Creator - CIDR-Based Creation & YAML Editing (REVISED) -**Status:** Ready to Start +**Status:** ✅ COMPLETE - Implementation Finished **Priority:** HIGH - Core usability feature -**Estimated Duration:** 4-5 days +**Actual Duration:** 5 days **Dependencies:** Phase 2 Complete (REST API, Authentication) --- @@ -11,12 +11,23 @@ Phase 4 introduces a **Config Creator** feature that allows users to manage scan configurations through the web UI instead of manually creating YAML files. This dramatically improves usability by providing: -1. **CSV Template Download** - Pre-formatted CSV template for config creation -2. **CSV Upload & Conversion** - Upload filled CSV, automatically convert to YAML -3. **YAML Upload** - Direct YAML upload for advanced users -4. **Config Management UI** - List, view, download, and delete configs +1. **CIDR-Based Config Creation** - Generate configs from CIDR ranges (e.g., 10.0.0.0/24) +2. **YAML Editor** - Edit existing configs with syntax-highlighted editor +3. **YAML Upload** - Direct YAML upload for advanced users (API) +4. **Config Management UI** - List, view, edit, download, and delete configs 5. **Integration** - Seamlessly works with existing scan triggers and schedules +### Design Pivot (Mid-Implementation) + +**Original Plan:** CSV upload with template download and conversion to YAML +**Revised Plan:** CIDR-based form for initial creation + YAML editor for modifications + +**Rationale for Change:** +- CSV parsing with comma-separated ports/services was complex and error-prone +- CIDR input is simpler for bulk IP ranges (the primary use case) +- Direct YAML editing provides more flexibility than CSV +- Eliminates need for users to download templates and understand CSV format + ### User Workflow **Current State (Manual):** @@ -26,14 +37,21 @@ Phase 4 introduces a **Config Creator** feature that allows users to manage scan 3. User references config in scan trigger/schedule ``` -**New Workflow (Phase 4):** +**New Workflow (Phase 4 - CIDR-Based):** ``` -1. User clicks "Download CSV Template" in web UI -2. User fills CSV with sites, IPs, expected ports in Excel/Google Sheets -3. User uploads CSV via drag-drop or file picker -4. System validates CSV and converts to YAML -5. User previews generated YAML and confirms -6. Config saved and immediately available for scans/schedules +1. User navigates to "Configs" → "Create New Config" +2. User enters CIDR range (e.g., 10.0.0.0/24) with title and optional site name +3. System expands CIDR to individual IPs and creates YAML config +4. User clicks "Edit" on newly created config +5. User modifies YAML in code editor to add expected ports/services +6. User saves changes +7. Config immediately available for scans/schedules +``` + +**Alternative Workflow (Advanced Users):** +``` +1. User uploads complete YAML config via API +2. Config immediately available for scans/schedules ``` --- @@ -42,163 +60,192 @@ Phase 4 introduces a **Config Creator** feature that allows users to manage scan Based on project requirements and complexity analysis: -### 1. CSV Scope: One CSV = One Config ✓ -- Each CSV file creates a single YAML config file -- All rows share the same `scan_title` (first column) -- Simpler to implement and understand -- Users create multiple CSVs for multiple configs +### 1. Creation Method: CIDR Form + YAML Editor ✓ +- Primary method: CIDR-based form for bulk IP generation +- Secondary method: Direct YAML upload via API (advanced users) +- Edit method: In-browser YAML text editor with syntax highlighting +- Simpler than CSV, more flexible than form-only approach -### 2. Creation Methods: CSV/YAML Upload Only ✓ -- CSV upload with conversion (primary method) -- Direct YAML upload (for advanced users) -- Form-based editor deferred to future phase -- Focused scope for faster delivery +### 2. CIDR Expansion: Single CIDR per Submission ✓ +- Each submission accepts one CIDR range (e.g., 10.0.0.0/24) +- System expands to individual IP addresses automatically +- Limit: 10,000 addresses per CIDR to prevent abuse +- Users can create multiple configs for multiple ranges -### 3. Versioning: No Version History ✓ +### 3. Edit Interface: YAML Text Editor ✓ +- CodeMirror editor with YAML syntax highlighting +- Direct YAML editing provides maximum flexibility +- Real-time validation before save +- Supports all YAML features (complex structures, comments, etc.) + +### 4. Versioning: No Version History ✓ - Configs overwrite when updated (no `.bak` files) - Simpler implementation, less storage - Users can download existing configs before editing -### 4. Deletion Safety: Block if Used by Schedules ✓ +### 5. Deletion Safety: Block if Used by Schedules ✓ - Prevent deletion of configs referenced by active schedules - Show error message listing which schedules use the config - Safest approach, prevents schedule failures -### 5. Additional Scope Decisions -- **File naming:** Auto-generated from scan title (sanitized) -- **File extensions:** Accept `.yaml`, `.yml` for uploads -- **CSV export:** Not in Phase 4 (future enhancement) -- **Config editing:** Download → Edit locally → Re-upload (no inline editor yet) +### 6. Additional Scope Decisions +- **File naming:** Auto-generated from config title (sanitized) +- **File extensions:** `.yaml` only (consistent naming) +- **CIDR validation:** ipaddress module for validation +- **File permissions:** Read-write mount for configs directory --- -## CSV Template Specification +## CIDR Form Specification -### CSV Format +### Form Fields -**Columns (in order):** -``` -scan_title,site_name,ip_address,ping_expected,tcp_ports,udp_ports,services -``` - -**Example CSV:** -```csv -scan_title,site_name,ip_address,ping_expected,tcp_ports,udp_ports,services -Sneaky Infra Scan,Production Web Servers,10.10.20.4,true,"22,53,80",53,"ssh,domain,http" -Sneaky Infra Scan,Production Web Servers,10.10.20.11,true,"22,111,3128,8006","","ssh,rpcbind,squid" -Sneaky Infra Scan,Database Servers,10.10.30.5,true,"22,3306","","" -Sneaky Infra Scan,Database Servers,10.10.30.6,false,"22,5432","","" -``` - -### Field Specifications - -| Column | Type | Required | Format | Example | Description | -|--------|------|----------|--------|---------|-------------| -| `scan_title` | string | Yes | Any text | `"Sneaky Infra Scan"` | Becomes YAML `title` field. Must be same for all rows. | -| `site_name` | string | Yes | Any text | `"Production Web Servers"` | Logical grouping. Rows with same site_name are grouped together. | -| `ip_address` | string | Yes | IPv4 or IPv6 | `"10.10.20.4"` | Target IP address to scan. | -| `ping_expected` | boolean | No | `true`/`false` (case-insensitive) | `true` | Whether host should respond to ping. Default: `false` | -| `tcp_ports` | list[int] | No | Comma-separated, quoted if multiple | `"22,80,443"` or `22` | Expected TCP ports. Empty = no expected TCP ports. | -| `udp_ports` | list[int] | No | Comma-separated, quoted if multiple | `"53,123"` or `53` | Expected UDP ports. Empty = no expected UDP ports. | -| `services` | list[str] | No | Comma-separated, quoted if multiple | `"ssh,http,https"` | Expected service names (optional). | +| Field | Type | Required | Format | Example | Description | +|-------|------|----------|--------|---------|-------------| +| `title` | string | Yes | Any text | `"Production Network Scan"` | Becomes YAML `title` field and basis for filename | +| `cidr` | string | Yes | CIDR notation | `"10.0.0.0/24"` | IP range to scan. Must be valid IPv4 or IPv6 CIDR | +| `site_name` | string | No | Any text | `"Production Servers"` | Optional site name for grouping. Default: "Default Site" | +| `ping_default` | boolean | No | checkbox | `true`/`false` | Whether all IPs should expect ping by default. Default: `false` | ### Validation Rules -**Row-level validation:** -1. `scan_title` must be non-empty and same across all rows -2. `site_name` must be non-empty -3. `ip_address` must be valid IPv4 or IPv6 format -4. `ping_expected` must be `true`, `false`, `TRUE`, `FALSE`, or empty (default false) -5. Port numbers must be integers 1-65535 -6. Port lists must be comma-separated (spaces optional) -7. Duplicate IPs within same config are allowed (different expected values) +**Field validation:** +1. `title` must be non-empty (min 3 characters) +2. `cidr` must be valid CIDR notation (validated via Python `ipaddress` module) +3. CIDR range must not exceed 10,000 addresses +4. `site_name` defaults to "Default Site" if empty +5. `ping_default` defaults to `false` if unchecked -**File-level validation:** -1. CSV must have exactly 7 columns with correct headers -2. Must have at least 1 data row (besides header) -3. Must have at least 1 site defined -4. Must have at least 1 IP per site - -**Filename validation:** -1. Generated filename from scan_title (lowercase, spaces→hyphens, special chars removed) +**Filename generation:** +1. Generated from `title` (lowercase, spaces→hyphens, special chars removed) 2. Must not conflict with existing config files 3. Max filename length: 200 characters +4. Always appends `.yaml` extension -### CSV-to-YAML Conversion Logic +### CIDR-to-YAML Conversion Logic -**Python pseudocode:** +**Python implementation:** ```python -def csv_to_yaml(csv_content: str) -> str: - rows = parse_csv(csv_content) +def create_from_cidr( + self, + title: str, + cidr: str, + site_name: Optional[str] = None, + ping_default: bool = False +) -> Tuple[str, str]: + """ + Create config from CIDR range. - # Extract scan title (same for all rows) - scan_title = rows[0]['scan_title'] + Args: + title: Config title + cidr: CIDR notation (e.g., "10.0.0.0/24") + site_name: Optional site name (default: "Default Site") + ping_default: Whether to expect ping by default - # Group rows by site_name - sites = {} - for row in rows: - site_name = row['site_name'] - if site_name not in sites: - sites[site_name] = [] + Returns: + (filename, yaml_content) + """ + # Validate and parse CIDR + try: + network = ipaddress.ip_network(cidr, strict=False) + except ValueError as e: + raise ValueError(f"Invalid CIDR range: {str(e)}") - # Parse ports and services - tcp_ports = parse_port_list(row['tcp_ports']) - udp_ports = parse_port_list(row['udp_ports']) - services = parse_string_list(row['services']) - ping = parse_bool(row['ping_expected'], default=False) + # Check if network is too large + if network.num_addresses > 10000: + raise ValueError(f"CIDR range too large: {network.num_addresses} addresses. Maximum is 10,000.") - sites[site_name].append({ - 'address': row['ip_address'], - 'expected': { - 'ping': ping, - 'tcp_ports': tcp_ports, - 'udp_ports': udp_ports, - 'services': services # Optional, omit if empty - } - }) + # Expand CIDR to list of IP addresses + ip_list = [str(ip) for ip in network.hosts()] + + # If network has only 1 address (like /32), hosts() returns empty + if not ip_list: + ip_list = [str(network.network_address)] # Build YAML structure + site_name = site_name or "Default Site" + yaml_data = { - 'title': scan_title, + 'title': title, 'sites': [ { 'name': site_name, - 'ips': ips + 'ips': [ + { + 'address': ip, + 'expected': { + 'ping': ping_default, + 'tcp_ports': [], + 'udp_ports': [] + } + } + for ip in ip_list + ] } - for site_name, ips in sites.items() ] } - return yaml.dump(yaml_data, sort_keys=False) + # Generate filename + filename = self.generate_filename_from_title(title) + + # Save config + yaml_content = yaml.dump(yaml_data, default_flow_style=False, sort_keys=False) + filepath = os.path.join(self.configs_dir, filename) + + with open(filepath, 'w') as f: + f.write(yaml_content) + + return filename, yaml_content ``` -**Example Conversion:** +### Example CIDR Conversion -**Input CSV:** -```csv -scan_title,site_name,ip_address,ping_expected,tcp_ports,udp_ports,services -Prod Scan,Web Servers,10.10.20.4,true,"22,80,443",53,ssh -Prod Scan,Web Servers,10.10.20.5,true,"22,80",,"ssh,http" +**Input Form Data:** +```json +{ + "title": "Production Network", + "cidr": "10.0.0.0/30", + "site_name": "Web Servers", + "ping_default": true +} ``` -**Output YAML:** +**Output YAML (`production-network.yaml`):** ```yaml -title: Prod Scan +title: Production Network sites: - name: Web Servers ips: - - address: 10.10.20.4 + - address: 10.0.0.1 + expected: + ping: true + tcp_ports: [] + udp_ports: [] + - address: 10.0.0.2 + expected: + ping: true + tcp_ports: [] + udp_ports: [] +``` + +**Then user edits to add expected ports:** +```yaml +title: Production Network +sites: + - name: Web Servers + ips: + - address: 10.0.0.1 expected: ping: true tcp_ports: [22, 80, 443] udp_ports: [53] - services: [ssh] - - address: 10.10.20.5 + services: [ssh, http, https, domain] + - address: 10.0.0.2 expected: ping: true - tcp_ports: [22, 80] + tcp_ports: [22, 3306] udp_ports: [] - services: [ssh, http] + services: [ssh, mysql] ``` --- @@ -209,19 +256,20 @@ sites: #### 1. API Blueprint: `web/api/configs.py` -**New endpoints:** +**Implemented endpoints:** | Method | Endpoint | Description | Auth | Request Body | Response | |--------|----------|-------------|------|--------------|----------| | `GET` | `/api/configs` | List all config files | Required | - | `{ "configs": [{filename, title, path, created_at, used_by_schedules}] }` | | `GET` | `/api/configs/` | Get config content (YAML) | Required | - | `{ "filename": "...", "content": "...", "parsed": {...} }` | -| `POST` | `/api/configs/upload-csv` | Upload CSV and convert to YAML | Required | `multipart/form-data` with file | `{ "filename": "...", "preview": "...", "success": true }` | +| `GET` | `/api/configs//download` | Download config file | Required | - | YAML file download | +| `POST` | `/api/configs/create-from-cidr` | Create config from CIDR range | Required | `{"title": "...", "cidr": "...", "site_name": "...", "ping_default": false}` | `{ "success": true, "filename": "...", "preview": "..." }` | | `POST` | `/api/configs/upload-yaml` | Upload YAML directly | Required | `multipart/form-data` with file | `{ "filename": "...", "success": true }` | -| `GET` | `/api/configs/template` | Download CSV template | Required | - | CSV file download | +| `PUT` | `/api/configs/` | Update existing config | Required | `{"content": "YAML content..."}` | `{ "success": true, "message": "..." }` | | `DELETE` | `/api/configs/` | Delete config file | Required | - | `{ "success": true }` or error if used by schedules | **Error responses:** -- `400` - Invalid CSV format, validation errors +- `400` - Invalid CIDR, invalid YAML, validation errors - `404` - Config file not found - `409` - Config filename conflict - `422` - Cannot delete (used by schedules) @@ -251,23 +299,40 @@ def get_config(filename: str): """Get config file content""" pass -@bp.route('/upload-csv', methods=['POST']) +@bp.route('//download', methods=['GET']) @api_auth_required -def upload_csv(): - """Upload CSV and convert to YAML""" +def download_config(filename: str): + """Download config file""" pass +@bp.route('/create-from-cidr', methods=['POST']) +@api_auth_required +def create_from_cidr(): + """Create config from CIDR range""" + data = request.get_json() + config_service = ConfigService() + filename, yaml_preview = config_service.create_from_cidr( + title=data['title'], + cidr=data['cidr'], + site_name=data.get('site_name'), + ping_default=data.get('ping_default', False) + ) + return jsonify({'success': True, 'filename': filename, 'preview': yaml_preview}), 201 + @bp.route('/upload-yaml', methods=['POST']) @api_auth_required def upload_yaml(): """Upload YAML file directly""" pass -@bp.route('/template', methods=['GET']) +@bp.route('/', methods=['PUT']) @api_auth_required -def download_template(): - """Download CSV template""" - pass +def update_config(filename: str): + """Update existing config""" + data = request.get_json() + config_service = ConfigService() + config_service.update_config(filename, data['content']) + return jsonify({'success': True, 'message': 'Config updated successfully'}) @bp.route('/', methods=['DELETE']) @api_auth_required @@ -317,6 +382,35 @@ class ConfigService: """ pass + def create_from_cidr( + self, + title: str, + cidr: str, + site_name: Optional[str] = None, + ping_default: bool = False + ) -> Tuple[str, str]: + """ + Create config from CIDR range. + + Args: + title: Config title + cidr: CIDR notation (e.g., "10.0.0.0/24") + site_name: Optional site name (default: "Default Site") + ping_default: Whether to expect ping by default + + Returns: + (final_filename, yaml_content) + + Raises: + ValueError: If CIDR invalid or filename conflict + """ + # Validate and parse CIDR using ipaddress module + # Expand to individual IPs + # Generate YAML structure + # Save to file + # Return filename and preview + pass + def create_from_yaml(self, filename: str, content: str) -> str: """ Create config from YAML content. @@ -333,20 +427,22 @@ class ConfigService: """ pass - def create_from_csv(self, csv_file, suggested_filename: str = None) -> Tuple[str, str]: + def update_config(self, filename: str, yaml_content: str) -> None: """ - Create config from CSV file. + Update existing config with new YAML content. Args: - csv_file: File object from request.files - suggested_filename: Optional filename (else auto-generate from title) - - Returns: - (final_filename, yaml_preview) + filename: Config filename to update + yaml_content: New YAML content Raises: - ValueError: If CSV invalid + FileNotFoundError: If config doesn't exist + ValueError: If YAML invalid or structure invalid """ + # Validate file exists + # Parse and validate YAML + # Validate config structure + # Write updated content pass def delete_config(self, filename: str) -> None: @@ -386,92 +482,19 @@ class ConfigService: pass ``` -#### 3. CSV Parser: `web/utils/csv_parser.py` +#### 3. REMOVED: CSV Parser and Template Generator -**Class definition:** -```python -class CSVConfigParser: - """Parse and validate CSV config files""" +**Note:** The original plan included CSV parsing components (`web/utils/csv_parser.py` and `web/utils/template_generator.py`), but these were removed during the mid-implementation pivot to CIDR-based creation. The following files were deleted: - REQUIRED_COLUMNS = [ - 'scan_title', 'site_name', 'ip_address', - 'ping_expected', 'tcp_ports', 'udp_ports', 'services' - ] +- ❌ `web/utils/csv_parser.py` - CSV parsing logic (DELETED) +- ❌ `web/utils/template_generator.py` - CSV template generation (DELETED) +- ❌ `tests/test_csv_parser.py` - CSV parser tests (DELETED) - def __init__(self): - pass - - def parse_csv_to_yaml(self, csv_file) -> str: - """ - Convert CSV file to YAML string. - - Args: - csv_file: File object or file path - - Returns: - YAML string - - Raises: - ValueError: If CSV invalid - """ - pass - - def validate_csv_structure(self, csv_file) -> Tuple[bool, List[str]]: - """ - Validate CSV structure and content. - - Returns: - (is_valid, error_messages) - """ - pass - - def _parse_port_list(self, value: str) -> List[int]: - """Parse comma-separated port list""" - pass - - def _parse_string_list(self, value: str) -> List[str]: - """Parse comma-separated string list""" - pass - - def _parse_bool(self, value: str, default: bool = False) -> bool: - """Parse boolean value (true/false/1/0)""" - pass - - def _validate_ip_address(self, ip: str) -> bool: - """Validate IPv4/IPv6 address format""" - pass - - def _validate_port(self, port: int) -> bool: - """Validate port number (1-65535)""" - pass -``` - -#### 4. Template Generator: `web/utils/template_generator.py` - -**Function:** -```python -def generate_csv_template() -> str: - """ - Generate CSV template with headers and example rows. - - Returns: - CSV string with headers and 2 example rows - """ - template = [ - ['scan_title', 'site_name', 'ip_address', 'ping_expected', 'tcp_ports', 'udp_ports', 'services'], - ['Example Infrastructure Scan', 'Production Web Servers', '10.10.20.4', 'true', '22,80,443', '53', 'ssh,http,https'], - ['Example Infrastructure Scan', 'Production Web Servers', '10.10.20.5', 'true', '22,3306', '', 'ssh,mysql'], - ] - - output = io.StringIO() - writer = csv.writer(output) - writer.writerows(template) - return output.getvalue() -``` +**Rationale:** CIDR-based creation is simpler and more appropriate for the primary use case (bulk IP ranges). Direct YAML editing provides more flexibility than CSV for complex configurations. ### Frontend Components -#### 1. New Route: Configs Management Page +#### 1. New Routes: Configs Management Pages **File:** `web/routes/main.py` @@ -479,14 +502,34 @@ def generate_csv_template() -> str: @bp.route('/configs') @login_required def configs(): - """Config management page""" + """Config management page - list all configs""" return render_template('configs.html') @bp.route('/configs/upload') @login_required def upload_config(): - """Config upload page""" + """Config upload page - CIDR form and YAML upload""" return render_template('config_upload.html') + +@bp.route('/configs/edit/') +@login_required +def edit_config(filename): + """Config edit page - YAML editor""" + from web.services.config_service import ConfigService + from flask import flash, redirect + + try: + config_service = ConfigService() + config_data = config_service.get_config(filename) + + return render_template( + 'config_edit.html', + filename=config_data['filename'], + content=config_data['content'] + ) + except FileNotFoundError: + flash(f"Config file '{filename}' not found", 'error') + return redirect(url_for('main.configs')) ``` #### 2. Template: Config List Page @@ -563,18 +606,18 @@ def upload_config(): **File:** `web/templates/config_upload.html` **Features:** -- Two upload methods (tabs): - - **Tab 1: CSV Upload** (default) - - Drag-drop zone or file picker (`.csv` only) - - Instructions: "Download template, fill with your data, upload here" - - Preview pane showing generated YAML after upload - - "Save Config" button (disabled until valid upload) +- Two creation methods (tabs): + - **Tab 1: CIDR Form** (default) + - Input fields: Config Title (required), CIDR Range (required), Site Name (optional), Ping Default (checkbox) + - CIDR validation (e.g., 10.0.0.0/24) + - Real-time YAML preview after submission + - Success modal with links to edit or view configs - **Tab 2: YAML Upload** (for advanced users) - Drag-drop zone or file picker (`.yaml`, `.yml` only) - Direct upload without conversion - Real-time validation feedback -- Error messages with specific issues -- Success message with link to configs list +- Error messages with specific issues (invalid CIDR, file too large, etc.) +- Success messages with action buttons **Layout:** ```html @@ -586,8 +629,8 @@ def upload_config():