diff --git a/docs/ai/PHASE3.md b/docs/ai/PHASE3.md new file mode 100644 index 0000000..0f6fb49 --- /dev/null +++ b/docs/ai/PHASE3.md @@ -0,0 +1,2204 @@ +# Phase 3 Implementation Plan: Dashboard Enhancement & Scheduled Scans + +**Status:** Ready to Start +**Progress:** 0/14 days complete (0%) +**Estimated Duration:** 14 days (2 weeks) +**Dependencies:** Phase 2 Complete ✅ + +## Progress Summary + +- 📋 **Step 1: Fix Styling Issues & CSS Refactor** (Day 1) - NEXT +- 📋 **Step 2: ScheduleService Implementation** (Days 2-3) +- 📋 **Step 3: Schedules API Endpoints** (Days 4-5) +- 📋 **Step 4: Schedule Management UI** (Days 6-7) +- 📋 **Step 5: Enhanced Dashboard with Charts** (Days 8-9) +- 📋 **Step 6: Scheduler Integration** (Day 10) +- 📋 **Step 7: Scan Comparison Features** (Days 11-12) +- 📋 **Step 8: Testing & Documentation** (Days 13-14) + +--- + +## Table of Contents + +1. [Overview](#overview) +2. [Current State Analysis](#current-state-analysis) +3. [Critical Bug Fix](#critical-bug-fix) +4. [Files to Create](#files-to-create) +5. [Files to Modify](#files-to-modify) +6. [Step-by-Step Implementation](#step-by-step-implementation) +7. [Dependencies & Prerequisites](#dependencies--prerequisites) +8. [Testing Approach](#testing-approach) +9. [Potential Challenges & Solutions](#potential-challenges--solutions) +10. [Success Criteria](#success-criteria) +11. [Migration Path](#migration-path) +12. [Estimated Timeline](#estimated-timeline) +13. [Key Design Decisions](#key-design-decisions) +14. [Documentation Deliverables](#documentation-deliverables) + +--- + +## Overview + +Phase 3 focuses on enhancing the web application with scheduling capabilities and improved dashboard visualizations: + +1. **Critical Bug Fix** - Fix white row coloring in scan tables +2. **Scheduled Scans** - Complete schedule management system with cron expressions +3. **Enhanced Dashboard** - Trending charts, schedule widgets, alert summaries +4. **Scan Comparison** - Historical analysis and infrastructure drift detection + +### Goals + +- 🐛 Fix white row bug affecting scan tables (critical UX issue) +- ✅ Complete schedule management (CRUD operations) +- ✅ Scheduled scan execution with cron expressions +- ✅ Enhanced dashboard with Chart.js visualizations +- ✅ Scan comparison and historical analysis +- ✅ CSS extraction and better maintainability + +--- + +## Current State Analysis + +### What's Already Done (Phase 2) + +**✅ Phase 2 Complete** - Full web application core: + +- **Database Schema** - All 11 models including Schedule table +- **ScanService** - Complete CRUD operations with 545 lines +- **Scan API** - 5 endpoints fully implemented +- **SchedulerService** - Partial implementation (258 lines) + - ✅ `queue_scan()` - Immediate scan execution works + - ⚠️ `add_scheduled_scan()` - Placeholder (marked for Phase 3) + - ⚠️ `_trigger_scheduled_scan()` - Skeleton with TODO comments + - ✅ `remove_scheduled_scan()` - Basic implementation +- **Authentication** - Flask-Login with @login_required decorators +- **UI Templates** - Dashboard, scans list/detail, login pages +- **Background Jobs** - APScheduler with concurrent execution (max 3) +- **Error Handling** - Content negotiation, custom error pages +- **Testing** - 100 test functions, 1,825 lines of test code + +### Schedule-Related Components Analysis + +#### Schedule Model (`web/models.py` - lines 142-158) +```python +class Schedule(Base): + __tablename__ = 'schedules' + + id = Column(Integer, primary_key=True) + name = Column(String(255), nullable=False) + config_file = Column(Text, nullable=False) + cron_expression = Column(String(100), nullable=False) + enabled = Column(Boolean, default=True) + last_run = Column(DateTime) + next_run = Column(DateTime) + created_at = Column(DateTime, default=datetime.utcnow) + updated_at = Column(DateTime, default=datetime.utcnow, onupdate=datetime.utcnow) +``` + +**Status:** ✅ Complete - No migration needed + +#### Schedules API (`web/api/schedules.py` - 159 lines) +```python +@bp.route('', methods=['GET']) +@api_auth_required +def list_schedules(): + """List all schedules. TODO: Implement in Phase 3.""" + return jsonify({'message': 'Schedules list - to be implemented in Phase 3'}), 200 +``` + +**Status:** ⚠️ All 6 endpoints are stubs returning placeholders + +**Endpoints to Implement:** +- `GET /api/schedules` - List all schedules +- `GET /api/schedules/` - Get schedule details +- `POST /api/schedules` - Create new schedule +- `PUT /api/schedules/` - Update schedule +- `DELETE /api/schedules/` - Delete schedule +- `POST /api/schedules//trigger` - Manually trigger scheduled scan + +#### SchedulerService Gaps + +**Missing Implementations:** +1. `add_scheduled_scan(schedule)` - Currently placeholder +2. `_trigger_scheduled_scan(schedule_id)` - TODO comments only +3. Loading schedules on app startup - Not implemented +4. Cron expression parsing - No validation +5. Next run time calculation - Missing + +### UI Components Status + +**Existing Templates:** +- `web/templates/base.html` (346 lines) - Has inline CSS (need to extract) +- `web/templates/dashboard.html` (356 lines) - Basic stats, needs charts +- `web/templates/scans.html` (469 lines) - Has white row bug +- `web/templates/scan_detail.html` (399 lines) - Has white row bug in port tables + +**Missing Templates:** +- Schedule list page +- Schedule create form +- Schedule edit form + +**Web Routes (`web/routes/main.py` - 69 lines):** +- Only has scan routes, no schedule routes + +--- + +## Critical Bug Fix + +### White Row Coloring Issue + +**Problem:** Dynamically created table rows in scans tables display with white background instead of dark theme colors. + +**Affected Files:** +1. `web/templates/scans.html` (lines 208-241) - renderScansTable() +2. `web/templates/dashboard.html` (lines 221-260) - renderScansTable() +3. `web/templates/scan_detail.html` (lines 305-327) - Port tables + +**Root Cause:** + +JavaScript dynamically creates `` elements that don't inherit CSS styles properly: + +```javascript +const row = document.createElement('tr'); +row.innerHTML = ` + ${scan.id} + ${scan.title || 'Untitled Scan'} + ... +`; +tbody.appendChild(row); +``` + +**Current CSS (base.html lines 157-165):** +```css +.table tbody tr { + background-color: #1e293b; /* Dark slate */ + border-color: #334155; +} + +.table tbody tr:hover { + background-color: #334155; + cursor: pointer; +} +``` + +**Solution:** + +Add explicit class to dynamically created rows: + +```javascript +const row = document.createElement('tr'); +row.classList.add('scan-row'); // Add explicit class +row.innerHTML = `...`; +``` + +And enhance CSS with higher specificity: +```css +.table tbody tr.scan-row, +.table tbody tr { + background-color: #1e293b !important; + border-color: #334155 !important; +} +``` + +--- + +## Files to Create + +### Backend Services + +#### 1. `web/services/schedule_service.py` +Schedule CRUD operations and business logic. + +**Class: ScheduleService** + +**Estimated Size:** ~400 lines + +**Methods:** +- `__init__(self, db_session)` - Initialize with database session +- `create_schedule(name, config_file, cron_expression, enabled=True)` → schedule_id + - Validate cron expression + - Validate config file exists + - Calculate next_run time + - Create Schedule record + - Return schedule_id + +- `get_schedule(schedule_id)` → schedule dict + - Query Schedule by ID + - Format for API response + - Include execution history (recent scans with this schedule_id) + +- `list_schedules(page=1, per_page=20, enabled_filter=None)` → paginated results + - Query with pagination + - Filter by enabled status if provided + - Calculate relative next_run time ("in 2 hours", "tomorrow at 3:00 PM") + - Return total count and items + +- `update_schedule(schedule_id, **updates)` → schedule dict + - Validate cron_expression if changed + - Recalculate next_run if cron changed + - Update Schedule record + - Reload schedule in APScheduler + - Return updated schedule + +- `delete_schedule(schedule_id)` → success + - Remove job from APScheduler + - Delete Schedule record (do NOT delete associated scans) + - Return success status + +- `toggle_enabled(schedule_id, enabled)` → schedule dict + - Enable or disable schedule + - Add/remove from APScheduler + - Return updated schedule + +- `update_run_times(schedule_id, last_run, next_run)` → success + - Update last_run and next_run timestamps + - Called after each execution + - Return success status + +- `validate_cron_expression(cron_expr)` → (valid, error_message) + - Use croniter library + - Return True if valid, False + error message if invalid + +- `calculate_next_run(cron_expr, from_time=None)` → datetime + - Calculate next run time from cron expression + - Use croniter + - Return datetime (UTC) + +- `get_schedule_history(schedule_id, limit=10)` → list of scans + - Query recent scans triggered by this schedule + - Return list of scan dicts + +**Helper Methods:** +- `_schedule_to_dict(schedule_obj)` - Convert Schedule model to dict +- `_get_relative_time(dt)` - Format datetime as "in 2 hours", "yesterday" + +#### 2. `web/static/css/styles.css` +Extracted CSS for better maintainability. + +**Estimated Size:** ~350 lines (extracted from base.html) + +**Sections:** +- Variables (CSS custom properties for colors) +- Global styles +- Navigation +- Cards and containers +- Tables (including fix for white rows) +- Forms +- Buttons +- Badges and labels +- Charts (dark theme) +- Utilities + +**Example Structure:** +```css +/* CSS Variables */ +:root { + --bg-primary: #0f172a; + --bg-secondary: #1e293b; + --bg-tertiary: #334155; + --text-primary: #e2e8f0; + --text-secondary: #94a3b8; + --border-color: #334155; + --accent-blue: #60a5fa; + --success-bg: #065f46; + --success-text: #6ee7b7; + --warning-bg: #78350f; + --warning-text: #fcd34d; + --danger-bg: #7f1d1d; + --danger-text: #fca5a5; +} + +/* Fix for dynamically created table rows */ +.table tbody tr, +.table tbody tr.scan-row { + background-color: var(--bg-secondary) !important; + border-color: var(--border-color) !important; +} + +.table tbody tr:hover { + background-color: var(--bg-tertiary) !important; + cursor: pointer; +} +``` + +### Frontend Templates + +#### 3. `web/templates/schedules.html` +List all schedules with enable/disable toggles. + +**Estimated Size:** ~400 lines + +**Layout:** +```html +{% extends "base.html" %} + +{% block content %} +
+
+

Scheduled Scans

+ + New Schedule + +
+ + +
+
+
+
+
Total Schedules
+

-

+
+
+
+
+
+
+
Enabled
+

-

+
+
+
+
+
+
+
Next Run
+
-
+
+
+
+
+
+
+
Executions (24h)
+

-

+
+
+
+
+ + +
+
+ + + + + + + + + + + + + + +
NameSchedule (Cron)Next RunLast RunStatusActions
+
+
+
+ + +{% endblock %} +``` + +#### 4. `web/templates/schedule_create.html` +Form to create new scheduled scan. + +**Estimated Size:** ~300 lines + +**Features:** +- Schedule name input +- Config file selector (dropdown of available configs) +- Cron expression builder OR manual entry +- Cron expression validator (client-side and server-side) +- Human-readable cron description +- Enable/disable toggle +- Submit button + +**Cron Expression Builder:** +```html +
+
+
Schedule Configuration
+ + +
+ +
+ + + + +
+
+ + +
+ + + + Format: minute hour day month weekday + +
+ + +
+ Enter a cron expression above +
+ + +
+ Next 5 runs: +
    +
    +
    +
    +``` + +#### 5. `web/templates/schedule_edit.html` +Form to edit existing schedule. + +**Estimated Size:** ~250 lines + +**Similar to create, but:** +- Pre-populate fields from existing schedule +- Show execution history (last 10 scans) +- "Delete Schedule" button +- "Test Run Now" button + +### Testing Files + +#### 6. `tests/test_schedule_service.py` +Unit tests for ScheduleService. + +**Estimated Size:** ~450 lines, 18+ tests + +**Test Coverage:** +- `test_create_schedule` - Valid schedule creation +- `test_create_schedule_invalid_cron` - Cron validation +- `test_create_schedule_invalid_config` - Config file validation +- `test_get_schedule` - Retrieve schedule +- `test_get_schedule_not_found` - 404 handling +- `test_list_schedules` - Pagination +- `test_list_schedules_filter_enabled` - Filter by enabled status +- `test_update_schedule` - Update fields +- `test_update_schedule_cron` - Recalculate next_run when cron changes +- `test_delete_schedule` - Delete schedule +- `test_toggle_enabled` - Enable/disable +- `test_update_run_times` - Update after execution +- `test_validate_cron_expression_valid` - Valid expressions +- `test_validate_cron_expression_invalid` - Invalid expressions +- `test_calculate_next_run` - Next run calculation +- `test_get_schedule_history` - Execution history +- `test_schedule_to_dict` - Serialization +- `test_concurrent_schedule_operations` - Thread safety + +#### 7. `tests/test_schedule_api.py` +Integration tests for schedules API. + +**Estimated Size:** ~500 lines, 22+ tests + +**Test Coverage:** +- `test_list_schedules_empty` - Empty list +- `test_list_schedules_populated` - Multiple schedules +- `test_list_schedules_pagination` - Pagination +- `test_list_schedules_filter_enabled` - Filter +- `test_get_schedule` - Get details +- `test_get_schedule_not_found` - 404 +- `test_create_schedule` - Create new +- `test_create_schedule_invalid_cron` - Validation +- `test_create_schedule_invalid_config` - File validation +- `test_update_schedule` - Update fields +- `test_update_schedule_not_found` - 404 +- `test_delete_schedule` - Delete +- `test_delete_schedule_not_found` - 404 +- `test_trigger_schedule` - Manual trigger +- `test_trigger_schedule_not_found` - 404 +- `test_toggle_enabled_via_update` - Enable/disable +- `test_schedules_require_authentication` - Auth required +- `test_schedule_execution_history` - Show related scans +- `test_schedule_next_run_calculation` - Correct calculation +- `test_schedule_workflow_integration` - Complete workflow +- `test_concurrent_schedule_updates` - Concurrency + +#### 8. `tests/test_charts.py` +Tests for chart data generation. + +**Estimated Size:** ~200 lines, 8+ tests + +**Test Coverage:** +- `test_scan_trend_data` - Scans per day calculation +- `test_port_count_trend` - Port count over time +- `test_service_distribution` - Service type breakdown +- `test_certificate_expiry_timeline` - Cert expiry dates +- `test_empty_data_handling` - No scans case +- `test_date_range_filtering` - Filter by date range +- `test_data_format_for_chartjs` - Correct JSON format +- `test_trend_data_caching` - Performance optimization + +--- + +## Files to Modify + +### Backend Updates + +#### 1. `web/api/schedules.py` +Replace all stub implementations with working code. + +**Current State:** 159 lines, all placeholders + +**Changes:** +- Import ScheduleService +- Implement all 6 endpoints: + - `GET /api/schedules` - Call ScheduleService.list_schedules() + - `GET /api/schedules/` - Call ScheduleService.get_schedule() + - `POST /api/schedules` - Call ScheduleService.create_schedule() + - `PUT /api/schedules/` - Call ScheduleService.update_schedule() + - `DELETE /api/schedules/` - Call ScheduleService.delete_schedule() + - `POST /api/schedules//trigger` - Trigger immediate scan with schedule_id + +**New Code (~300 lines total):** +```python +from web.services.schedule_service import ScheduleService +from web.auth.decorators import api_auth_required +from flask import current_app, jsonify, request + +@bp.route('', methods=['GET']) +@api_auth_required +def list_schedules(): + """List all schedules with pagination.""" + page = request.args.get('page', 1, type=int) + per_page = request.args.get('per_page', 20, type=int) + enabled_filter = request.args.get('enabled', type=lambda x: x.lower() == 'true') + + schedule_service = ScheduleService(current_app.db_session) + result = schedule_service.list_schedules(page, per_page, enabled_filter) + + return jsonify(result), 200 + +@bp.route('', methods=['POST']) +@api_auth_required +def create_schedule(): + """Create a new schedule.""" + data = request.get_json() or {} + + # Validate required fields + required = ['name', 'config_file', 'cron_expression'] + for field in required: + if field not in data: + return jsonify({'error': f'Missing required field: {field}'}), 400 + + schedule_service = ScheduleService(current_app.db_session) + + try: + schedule_id = schedule_service.create_schedule( + name=data['name'], + config_file=data['config_file'], + cron_expression=data['cron_expression'], + enabled=data.get('enabled', True) + ) + + # Add to APScheduler + schedule = schedule_service.get_schedule(schedule_id) + current_app.scheduler.add_scheduled_scan(schedule) + + return jsonify({ + 'schedule_id': schedule_id, + 'message': 'Schedule created successfully' + }), 201 + except ValueError as e: + return jsonify({'error': str(e)}), 400 + +# ... more endpoints +``` + +#### 2. `web/services/scheduler_service.py` +Complete placeholder implementations. + +**Current State:** 258 lines, partial implementation + +**Changes:** +- Complete `add_scheduled_scan(schedule)` - Add cron job to APScheduler +- Complete `_trigger_scheduled_scan(schedule_id)` - Execute scheduled scan +- Add `load_schedules_on_startup()` - Load all enabled schedules +- Add cron expression parsing with croniter + +**New/Updated Methods:** +```python +from croniter import croniter +from datetime import datetime + +def add_scheduled_scan(self, schedule): + """Add a cron job for scheduled scan execution.""" + if not schedule.get('enabled'): + return + + job_id = f"schedule_{schedule['id']}" + + # Parse cron expression + trigger = CronTrigger.from_crontab(schedule['cron_expression']) + + # Add job to scheduler + self.scheduler.add_job( + func=self._trigger_scheduled_scan, + trigger=trigger, + id=job_id, + args=[schedule['id']], + replace_existing=True, + max_instances=1 + ) + + logger.info(f"Added scheduled scan: {schedule['name']} ({job_id})") + +def _trigger_scheduled_scan(self, schedule_id): + """Execute a scheduled scan.""" + from web.services.schedule_service import ScheduleService + + logger.info(f"Triggering scheduled scan: schedule_id={schedule_id}") + + # Get schedule details + schedule_service = ScheduleService(self.db_session) + schedule = schedule_service.get_schedule(schedule_id) + + if not schedule: + logger.error(f"Schedule {schedule_id} not found") + return + + # Trigger scan with schedule_id + from web.services.scan_service import ScanService + scan_service = ScanService(self.db_session) + + scan_id = scan_service.trigger_scan( + config_file=schedule['config_file'], + triggered_by='scheduled', + schedule_id=schedule_id + ) + + # Queue the scan + self.queue_scan(schedule['config_file'], scan_id, self.db_url) + + # Update last_run timestamp + schedule_service.update_run_times( + schedule_id=schedule_id, + last_run=datetime.utcnow(), + next_run=self._calculate_next_run(schedule['cron_expression']) + ) + + logger.info(f"Scheduled scan queued: scan_id={scan_id}") + +def load_schedules_on_startup(self): + """Load all enabled schedules from database on app startup.""" + from web.services.schedule_service import ScheduleService + + schedule_service = ScheduleService(self.db_session) + schedules = schedule_service.list_schedules(page=1, per_page=1000, enabled_filter=True) + + for schedule in schedules['schedules']: + self.add_scheduled_scan(schedule) + + logger.info(f"Loaded {len(schedules['schedules'])} schedules on startup") + +def _calculate_next_run(self, cron_expression): + """Calculate next run time from cron expression.""" + cron = croniter(cron_expression, datetime.utcnow()) + return cron.get_next(datetime) +``` + +#### 3. `web/routes/main.py` +Add schedule management routes. + +**Current State:** 69 lines, only scan routes + +**Changes:** +- Add schedule routes: + - `GET /schedules` - List schedules + - `GET /schedules/create` - Create form + - `GET /schedules//edit` - Edit form + +**New Code:** +```python +@bp.route('/schedules') +@login_required +def schedules(): + """List all schedules.""" + return render_template('schedules.html') + +@bp.route('/schedules/create') +@login_required +def create_schedule(): + """Create new schedule form.""" + # Get list of available config files + import os + configs_dir = '/app/configs' + config_files = [f for f in os.listdir(configs_dir) if f.endswith('.yaml')] + return render_template('schedule_create.html', config_files=config_files) + +@bp.route('/schedules//edit') +@login_required +def edit_schedule(schedule_id): + """Edit existing schedule form.""" + from web.services.schedule_service import ScheduleService + schedule_service = ScheduleService(current_app.db_session) + + try: + schedule = schedule_service.get_schedule(schedule_id) + return render_template('schedule_edit.html', schedule=schedule) + except Exception as e: + flash(f'Schedule not found: {e}', 'danger') + return redirect(url_for('main.schedules')) +``` + +#### 4. `web/templates/base.html` +Extract CSS, add Chart.js, fix white rows. + +**Current State:** 346 lines with 280 lines of inline CSS (lines 8-288) + +**Changes:** +- Replace inline ` + {% block extra_styles %}{% endblock %} {% if not hide_nav %} @@ -335,7 +77,7 @@ diff --git a/web/templates/dashboard.html b/web/templates/dashboard.html index 31390a7..7d04fc5 100644 --- a/web/templates/dashboard.html +++ b/web/templates/dashboard.html @@ -224,6 +224,7 @@ scans.forEach(scan => { const row = document.createElement('tr'); + row.classList.add('scan-row'); // Fix white row bug // Format timestamp const timestamp = new Date(scan.timestamp).toLocaleString(); diff --git a/web/templates/scan_detail.html b/web/templates/scan_detail.html index c53543c..c1a7482 100644 --- a/web/templates/scan_detail.html +++ b/web/templates/scan_detail.html @@ -306,12 +306,13 @@ const ports = ip.ports || []; if (ports.length === 0) { - portsContainer.innerHTML = 'No ports found'; + portsContainer.innerHTML = 'No ports found'; } else { ports.forEach(port => { const service = port.services && port.services.length > 0 ? port.services[0] : null; const row = document.createElement('tr'); + row.classList.add('scan-row'); // Fix white row bug row.innerHTML = ` ${port.port} ${port.protocol.toUpperCase()} diff --git a/web/templates/scans.html b/web/templates/scans.html index b26a718..27640eb 100644 --- a/web/templates/scans.html +++ b/web/templates/scans.html @@ -206,6 +206,7 @@ scans.forEach(scan => { const row = document.createElement('tr'); + row.classList.add('scan-row'); // Fix white row bug // Format timestamp const timestamp = new Date(scan.timestamp).toLocaleString();