sneakygeek/SneakyScan
2
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
Files
72c4f3d29b38a1c0321b205ac4cea14e0c1ca164
SneakyScan/docs/ai/PHASE3.md
Phillip Tarrant 489284bde1 updating Phase3.md
2025-11-14 16:31:35 -06:00

64 KiB
Raw Blame History

Phase 3 Implementation Plan: Dashboard Enhancement & Scheduled Scans

Status: In Progress Progress: 9/14 days complete (64%) Estimated Duration: 14 days (2 weeks) Dependencies: Phase 2 Complete ✅

Progress Summary

  • ✅ Step 1: Fix Styling Issues & CSS Refactor (Day 1) - COMPLETE
  • ✅ Step 2: ScheduleService Implementation (Days 2-3) - COMPLETE
  • ✅ Step 3: Schedules API Endpoints (Days 4-5) - COMPLETE
  • ✅ Step 4: Schedule Management UI (Days 6-7) - COMPLETE
  • ✅ Step 5: Enhanced Dashboard with Charts (Days 8-9) - COMPLETE
  • ✅ Step 6: Scheduler Integration (Day 10) - COMPLETE
  • ✅ Step 7: Scan Comparison Features (Days 11-12) - COMPLETE
  • ✅ Step 8: Testing & Documentation (Days 13-14) - COMPLETE

Table of Contents

  1. Overview
  2. Current State Analysis
  3. Critical Bug Fix
  4. Files to Create
  5. Files to Modify
  6. Step-by-Step Implementation
  7. Dependencies & Prerequisites
  8. Testing Approach
  9. Potential Challenges & Solutions
  10. Success Criteria
  11. Migration Path
  12. Estimated Timeline
  13. Key Design Decisions
  14. Documentation Deliverables

Overview

Phase 3 focuses on enhancing the web application with scheduling capabilities and improved dashboard visualizations:

  1. Critical Bug Fix - Fix white row coloring in scan tables
  2. Scheduled Scans - Complete schedule management system with cron expressions
  3. Enhanced Dashboard - Trending charts, schedule widgets, alert summaries
  4. Scan Comparison - Historical analysis and infrastructure drift detection

Goals

  • 🐛 Fix white row bug affecting scan tables (critical UX issue)
  • ✅ Complete schedule management (CRUD operations)
  • ✅ Scheduled scan execution with cron expressions
  • ✅ Enhanced dashboard with Chart.js visualizations
  • ✅ Scan comparison and historical analysis
  • ✅ CSS extraction and better maintainability

Current State Analysis

What's Already Done (Phase 2)

✅ Phase 2 Complete - Full web application core:

  • Database Schema - All 11 models including Schedule table
  • ScanService - Complete CRUD operations with 545 lines
  • Scan API - 5 endpoints fully implemented
  • SchedulerService - Partial implementation (258 lines)
    • ✅ queue_scan() - Immediate scan execution works
    • ⚠️ add_scheduled_scan() - Placeholder (marked for Phase 3)
    • ⚠️ _trigger_scheduled_scan() - Skeleton with TODO comments
    • ✅ remove_scheduled_scan() - Basic implementation
  • Authentication - Flask-Login with @login_required decorators
  • UI Templates - Dashboard, scans list/detail, login pages
  • Background Jobs - APScheduler with concurrent execution (max 3)
  • Error Handling - Content negotiation, custom error pages
  • Testing - 100 test functions, 1,825 lines of test code

Schedule-Related Components Analysis

Schedule Model (web/models.py - lines 142-158)

class Schedule(Base):
    __tablename__ = 'schedules'

    id = Column(Integer, primary_key=True)
    name = Column(String(255), nullable=False)
    config_file = Column(Text, nullable=False)
    cron_expression = Column(String(100), nullable=False)
    enabled = Column(Boolean, default=True)
    last_run = Column(DateTime)
    next_run = Column(DateTime)
    created_at = Column(DateTime, default=datetime.utcnow)
    updated_at = Column(DateTime, default=datetime.utcnow, onupdate=datetime.utcnow)

Status: ✅ Complete - No migration needed

Schedules API (web/api/schedules.py - 159 lines)

@bp.route('', methods=['GET'])
@api_auth_required
def list_schedules():
    """List all schedules. TODO: Implement in Phase 3."""
    return jsonify({'message': 'Schedules list - to be implemented in Phase 3'}), 200

Status: ⚠️ All 6 endpoints are stubs returning placeholders

Endpoints to Implement:

  • GET /api/schedules - List all schedules
  • GET /api/schedules/<id> - Get schedule details
  • POST /api/schedules - Create new schedule
  • PUT /api/schedules/<id> - Update schedule
  • DELETE /api/schedules/<id> - Delete schedule
  • POST /api/schedules/<id>/trigger - Manually trigger scheduled scan

SchedulerService Gaps

Missing Implementations:

  1. add_scheduled_scan(schedule) - Currently placeholder
  2. _trigger_scheduled_scan(schedule_id) - TODO comments only
  3. Loading schedules on app startup - Not implemented
  4. Cron expression parsing - No validation
  5. Next run time calculation - Missing

UI Components Status

Existing Templates:

  • web/templates/base.html (346 lines) - Has inline CSS (need to extract)
  • web/templates/dashboard.html (356 lines) - Basic stats, needs charts
  • web/templates/scans.html (469 lines) - Has white row bug
  • web/templates/scan_detail.html (399 lines) - Has white row bug in port tables

Missing Templates:

  • Schedule list page
  • Schedule create form
  • Schedule edit form

Web Routes (web/routes/main.py - 69 lines):

  • Only has scan routes, no schedule routes

Critical Bug Fix

White Row Coloring Issue

Problem: Dynamically created table rows in scans tables display with white background instead of dark theme colors.

Affected Files:

  1. web/templates/scans.html (lines 208-241) - renderScansTable()
  2. web/templates/dashboard.html (lines 221-260) - renderScansTable()
  3. web/templates/scan_detail.html (lines 305-327) - Port tables

Root Cause:

JavaScript dynamically creates <tr> elements that don't inherit CSS styles properly:

const row = document.createElement('tr');
row.innerHTML = `
    <td class="mono">${scan.id}</td>
    <td>${scan.title || 'Untitled Scan'}</td>
    ...
`;
tbody.appendChild(row);

Current CSS (base.html lines 157-165):

.table tbody tr {
    background-color: #1e293b;  /* Dark slate */
    border-color: #334155;
}

.table tbody tr:hover {
    background-color: #334155;
    cursor: pointer;
}

Solution:

Add explicit class to dynamically created rows:

const row = document.createElement('tr');
row.classList.add('scan-row');  // Add explicit class
row.innerHTML = `...`;

And enhance CSS with higher specificity:

.table tbody tr.scan-row,
.table tbody tr {
    background-color: #1e293b !important;
    border-color: #334155 !important;
}

Files to Create

Backend Services

1. web/services/schedule_service.py

Schedule CRUD operations and business logic.

Class: ScheduleService

Estimated Size: ~400 lines

Methods:

  • __init__(self, db_session) - Initialize with database session

  • create_schedule(name, config_file, cron_expression, enabled=True) → schedule_id

    • Validate cron expression
    • Validate config file exists
    • Calculate next_run time
    • Create Schedule record
    • Return schedule_id

  • get_schedule(schedule_id) → schedule dict

    • Query Schedule by ID
    • Format for API response
    • Include execution history (recent scans with this schedule_id)

  • list_schedules(page=1, per_page=20, enabled_filter=None) → paginated results

    • Query with pagination
    • Filter by enabled status if provided
    • Calculate relative next_run time ("in 2 hours", "tomorrow at 3:00 PM")
    • Return total count and items

  • update_schedule(schedule_id, **updates) → schedule dict

    • Validate cron_expression if changed
    • Recalculate next_run if cron changed
    • Update Schedule record
    • Reload schedule in APScheduler
    • Return updated schedule

  • delete_schedule(schedule_id) → success

    • Remove job from APScheduler
    • Delete Schedule record (do NOT delete associated scans)
    • Return success status

  • toggle_enabled(schedule_id, enabled) → schedule dict

    • Enable or disable schedule
    • Add/remove from APScheduler
    • Return updated schedule

  • update_run_times(schedule_id, last_run, next_run) → success

    • Update last_run and next_run timestamps
    • Called after each execution
    • Return success status

  • validate_cron_expression(cron_expr) → (valid, error_message)

    • Use croniter library
    • Return True if valid, False + error message if invalid

  • calculate_next_run(cron_expr, from_time=None) → datetime

    • Calculate next run time from cron expression
    • Use croniter
    • Return datetime (UTC)

  • get_schedule_history(schedule_id, limit=10) → list of scans

    • Query recent scans triggered by this schedule
    • Return list of scan dicts

Helper Methods:

  • _schedule_to_dict(schedule_obj) - Convert Schedule model to dict
  • _get_relative_time(dt) - Format datetime as "in 2 hours", "yesterday"

2. web/static/css/styles.css

Extracted CSS for better maintainability.

Estimated Size: ~350 lines (extracted from base.html)

Sections:

  • Variables (CSS custom properties for colors)
  • Global styles
  • Navigation
  • Cards and containers
  • Tables (including fix for white rows)
  • Forms
  • Buttons
  • Badges and labels
  • Charts (dark theme)
  • Utilities

Example Structure:

/* CSS Variables */
:root {
    --bg-primary: #0f172a;
    --bg-secondary: #1e293b;
    --bg-tertiary: #334155;
    --text-primary: #e2e8f0;
    --text-secondary: #94a3b8;
    --border-color: #334155;
    --accent-blue: #60a5fa;
    --success-bg: #065f46;
    --success-text: #6ee7b7;
    --warning-bg: #78350f;
    --warning-text: #fcd34d;
    --danger-bg: #7f1d1d;
    --danger-text: #fca5a5;
}

/* Fix for dynamically created table rows */
.table tbody tr,
.table tbody tr.scan-row {
    background-color: var(--bg-secondary) !important;
    border-color: var(--border-color) !important;
}

.table tbody tr:hover {
    background-color: var(--bg-tertiary) !important;
    cursor: pointer;
}

Frontend Templates

3. web/templates/schedules.html

List all schedules with enable/disable toggles.

Estimated Size: ~400 lines

Layout:

{% extends "base.html" %}

{% block content %}
<div class="container mt-4">
    <div class="d-flex justify-content-between align-items-center mb-4">
        <h2>Scheduled Scans</h2>
        <a href="{{ url_for('main.create_schedule') }}" class="btn btn-primary">
            <i class="bi bi-plus-circle"></i> New Schedule
        </a>
    </div>

    <!-- Stats Cards -->
    <div class="row mb-4">
        <div class="col-md-3">
            <div class="card">
                <div class="card-body">
                    <h6>Total Schedules</h6>
                    <h2 id="total-schedules">-</h2>
                </div>
            </div>
        </div>
        <div class="col-md-3">
            <div class="card">
                <div class="card-body">
                    <h6>Enabled</h6>
                    <h2 id="enabled-schedules">-</h2>
                </div>
            </div>
        </div>
        <div class="col-md-3">
            <div class="card">
                <div class="card-body">
                    <h6>Next Run</h6>
                    <h5 id="next-run-time">-</h5>
                </div>
            </div>
        </div>
        <div class="col-md-3">
            <div class="card">
                <div class="card-body">
                    <h6>Executions (24h)</h6>
                    <h2 id="recent-executions">-</h2>
                </div>
            </div>
        </div>
    </div>

    <!-- Schedules Table -->
    <div class="card">
        <div class="card-body">
            <table class="table">
                <thead>
                    <tr>
                        <th>Name</th>
                        <th>Schedule (Cron)</th>
                        <th>Next Run</th>
                        <th>Last Run</th>
                        <th>Status</th>
                        <th>Actions</th>
                    </tr>
                </thead>
                <tbody id="schedules-tbody">
                    <!-- Populated by JavaScript -->
                </tbody>
            </table>
        </div>
    </div>
</div>

<script>
// Fetch and render schedules
async function loadSchedules() {
    const response = await fetch('/api/schedules');
    const data = await response.json();
    renderSchedulesTable(data.schedules);
    updateStats(data);
}

function renderSchedulesTable(schedules) {
    const tbody = document.getElementById('schedules-tbody');
    tbody.innerHTML = '';

    schedules.forEach(schedule => {
        const row = document.createElement('tr');
        row.classList.add('schedule-row');  // Fix white rows
        row.innerHTML = `
            <td>${schedule.name}</td>
            <td><code>${schedule.cron_expression}</code></td>
            <td>${formatRelativeTime(schedule.next_run)}</td>
            <td>${formatRelativeTime(schedule.last_run) || 'Never'}</td>
            <td>
                <div class="form-check form-switch">
                    <input class="form-check-input" type="checkbox"
                           id="enable-${schedule.id}"
                           ${schedule.enabled ? 'checked' : ''}
                           onchange="toggleSchedule(${schedule.id}, this.checked)">
                </div>
            </td>
            <td>
                <button class="btn btn-sm btn-secondary" onclick="triggerSchedule(${schedule.id})">
                    Run Now
                </button>
                <a href="/schedules/${schedule.id}/edit" class="btn btn-sm btn-secondary">
                    Edit
                </a>
                <button class="btn btn-sm btn-danger" onclick="deleteSchedule(${schedule.id})">
                    Delete
                </button>
            </td>
        `;
        tbody.appendChild(row);
    });
}

// Toggle schedule enabled/disabled
async function toggleSchedule(scheduleId, enabled) {
    await fetch(`/api/schedules/${scheduleId}`, {
        method: 'PUT',
        headers: {'Content-Type': 'application/json'},
        body: JSON.stringify({enabled: enabled})
    });
    loadSchedules();
}

// Manually trigger schedule
async function triggerSchedule(scheduleId) {
    const response = await fetch(`/api/schedules/${scheduleId}/trigger`, {
        method: 'POST'
    });
    const data = await response.json();
    alert(`Scan triggered! Scan ID: ${data.scan_id}`);
    window.location.href = `/scans/${data.scan_id}`;
}

// Delete schedule
async function deleteSchedule(scheduleId) {
    if (!confirm('Delete this schedule?')) return;
    await fetch(`/api/schedules/${scheduleId}`, {method: 'DELETE'});
    loadSchedules();
}

// Load on page load
loadSchedules();
setInterval(loadSchedules, 30000);  // Refresh every 30 seconds
</script>
{% endblock %}

4. web/templates/schedule_create.html

Form to create new scheduled scan.

Estimated Size: ~300 lines

Features:

  • Schedule name input
  • Config file selector (dropdown of available configs)
  • Cron expression builder OR manual entry
  • Cron expression validator (client-side and server-side)
  • Human-readable cron description
  • Enable/disable toggle
  • Submit button

Cron Expression Builder:

<div class="card mb-3">
    <div class="card-body">
        <h5>Schedule Configuration</h5>

        <!-- Quick Templates -->
        <div class="mb-3">
            <label>Quick Templates:</label>
            <div class="btn-group" role="group">
                <button type="button" class="btn btn-secondary btn-sm" onclick="setCron('0 0 * * *')">Daily at Midnight</button>
                <button type="button" class="btn btn-secondary btn-sm" onclick="setCron('0 2 * * *')">Daily at 2 AM</button>
                <button type="button" class="btn btn-secondary btn-sm" onclick="setCron('0 0 * * 0')">Weekly (Sunday)</button>
                <button type="button" class="btn btn-secondary btn-sm" onclick="setCron('0 0 1 * *')">Monthly (1st)</button>
            </div>
        </div>

        <!-- Manual Entry -->
        <div class="mb-3">
            <label for="cron-expression">Cron Expression:</label>
            <input type="text" class="form-control font-monospace"
                   id="cron-expression"
                   name="cron_expression"
                   placeholder="0 2 * * *"
                   onchange="validateCron()">
            <small class="form-text text-muted">
                Format: minute hour day month weekday
            </small>
        </div>

        <!-- Human-Readable Description -->
        <div class="alert alert-info" id="cron-description">
            Enter a cron expression above
        </div>

        <!-- Next Run Times -->
        <div id="next-runs">
            <strong>Next 5 runs:</strong>
            <ul id="next-runs-list"></ul>
        </div>
    </div>
</div>

5. web/templates/schedule_edit.html

Form to edit existing schedule.

Estimated Size: ~250 lines

Similar to create, but:

  • Pre-populate fields from existing schedule
  • Show execution history (last 10 scans)
  • "Delete Schedule" button
  • "Test Run Now" button

Testing Files

6. tests/test_schedule_service.py

Unit tests for ScheduleService.

Estimated Size: ~450 lines, 18+ tests

Test Coverage:

  • test_create_schedule - Valid schedule creation
  • test_create_schedule_invalid_cron - Cron validation
  • test_create_schedule_invalid_config - Config file validation
  • test_get_schedule - Retrieve schedule
  • test_get_schedule_not_found - 404 handling
  • test_list_schedules - Pagination
  • test_list_schedules_filter_enabled - Filter by enabled status
  • test_update_schedule - Update fields
  • test_update_schedule_cron - Recalculate next_run when cron changes
  • test_delete_schedule - Delete schedule
  • test_toggle_enabled - Enable/disable
  • test_update_run_times - Update after execution
  • test_validate_cron_expression_valid - Valid expressions
  • test_validate_cron_expression_invalid - Invalid expressions
  • test_calculate_next_run - Next run calculation
  • test_get_schedule_history - Execution history
  • test_schedule_to_dict - Serialization
  • test_concurrent_schedule_operations - Thread safety

7. tests/test_schedule_api.py

Integration tests for schedules API.

Estimated Size: ~500 lines, 22+ tests

Test Coverage:

  • test_list_schedules_empty - Empty list
  • test_list_schedules_populated - Multiple schedules
  • test_list_schedules_pagination - Pagination
  • test_list_schedules_filter_enabled - Filter
  • test_get_schedule - Get details
  • test_get_schedule_not_found - 404
  • test_create_schedule - Create new
  • test_create_schedule_invalid_cron - Validation
  • test_create_schedule_invalid_config - File validation
  • test_update_schedule - Update fields
  • test_update_schedule_not_found - 404
  • test_delete_schedule - Delete
  • test_delete_schedule_not_found - 404
  • test_trigger_schedule - Manual trigger
  • test_trigger_schedule_not_found - 404
  • test_toggle_enabled_via_update - Enable/disable
  • test_schedules_require_authentication - Auth required
  • test_schedule_execution_history - Show related scans
  • test_schedule_next_run_calculation - Correct calculation
  • test_schedule_workflow_integration - Complete workflow
  • test_concurrent_schedule_updates - Concurrency

8. tests/test_charts.py

Tests for chart data generation.

Estimated Size: ~200 lines, 8+ tests

Test Coverage:

  • test_scan_trend_data - Scans per day calculation
  • test_port_count_trend - Port count over time
  • test_service_distribution - Service type breakdown
  • test_certificate_expiry_timeline - Cert expiry dates
  • test_empty_data_handling - No scans case
  • test_date_range_filtering - Filter by date range
  • test_data_format_for_chartjs - Correct JSON format
  • test_trend_data_caching - Performance optimization

Files to Modify

Backend Updates

1. web/api/schedules.py

Replace all stub implementations with working code.

Current State: 159 lines, all placeholders

Changes:

  • Import ScheduleService
  • Implement all 6 endpoints:
    • GET /api/schedules - Call ScheduleService.list_schedules()
    • GET /api/schedules/<id> - Call ScheduleService.get_schedule()
    • POST /api/schedules - Call ScheduleService.create_schedule()
    • PUT /api/schedules/<id> - Call ScheduleService.update_schedule()
    • DELETE /api/schedules/<id> - Call ScheduleService.delete_schedule()
    • POST /api/schedules/<id>/trigger - Trigger immediate scan with schedule_id

New Code (~300 lines total):

from web.services.schedule_service import ScheduleService
from web.auth.decorators import api_auth_required
from flask import current_app, jsonify, request

@bp.route('', methods=['GET'])
@api_auth_required
def list_schedules():
    """List all schedules with pagination."""
    page = request.args.get('page', 1, type=int)
    per_page = request.args.get('per_page', 20, type=int)
    enabled_filter = request.args.get('enabled', type=lambda x: x.lower() == 'true')

    schedule_service = ScheduleService(current_app.db_session)
    result = schedule_service.list_schedules(page, per_page, enabled_filter)

    return jsonify(result), 200

@bp.route('', methods=['POST'])
@api_auth_required
def create_schedule():
    """Create a new schedule."""
    data = request.get_json() or {}

    # Validate required fields
    required = ['name', 'config_file', 'cron_expression']
    for field in required:
        if field not in data:
            return jsonify({'error': f'Missing required field: {field}'}), 400

    schedule_service = ScheduleService(current_app.db_session)

    try:
        schedule_id = schedule_service.create_schedule(
            name=data['name'],
            config_file=data['config_file'],
            cron_expression=data['cron_expression'],
            enabled=data.get('enabled', True)
        )

        # Add to APScheduler
        schedule = schedule_service.get_schedule(schedule_id)
        current_app.scheduler.add_scheduled_scan(schedule)

        return jsonify({
            'schedule_id': schedule_id,
            'message': 'Schedule created successfully'
        }), 201
    except ValueError as e:
        return jsonify({'error': str(e)}), 400

# ... more endpoints

2. web/services/scheduler_service.py

Complete placeholder implementations.

Current State: 258 lines, partial implementation

Changes:

  • Complete add_scheduled_scan(schedule) - Add cron job to APScheduler
  • Complete _trigger_scheduled_scan(schedule_id) - Execute scheduled scan
  • Add load_schedules_on_startup() - Load all enabled schedules
  • Add cron expression parsing with croniter

New/Updated Methods:

from croniter import croniter
from datetime import datetime

def add_scheduled_scan(self, schedule):
    """Add a cron job for scheduled scan execution."""
    if not schedule.get('enabled'):
        return

    job_id = f"schedule_{schedule['id']}"

    # Parse cron expression
    trigger = CronTrigger.from_crontab(schedule['cron_expression'])

    # Add job to scheduler
    self.scheduler.add_job(
        func=self._trigger_scheduled_scan,
        trigger=trigger,
        id=job_id,
        args=[schedule['id']],
        replace_existing=True,
        max_instances=1
    )

    logger.info(f"Added scheduled scan: {schedule['name']} ({job_id})")

def _trigger_scheduled_scan(self, schedule_id):
    """Execute a scheduled scan."""
    from web.services.schedule_service import ScheduleService

    logger.info(f"Triggering scheduled scan: schedule_id={schedule_id}")

    # Get schedule details
    schedule_service = ScheduleService(self.db_session)
    schedule = schedule_service.get_schedule(schedule_id)

    if not schedule:
        logger.error(f"Schedule {schedule_id} not found")
        return

    # Trigger scan with schedule_id
    from web.services.scan_service import ScanService
    scan_service = ScanService(self.db_session)

    scan_id = scan_service.trigger_scan(
        config_file=schedule['config_file'],
        triggered_by='scheduled',
        schedule_id=schedule_id
    )

    # Queue the scan
    self.queue_scan(schedule['config_file'], scan_id, self.db_url)

    # Update last_run timestamp
    schedule_service.update_run_times(
        schedule_id=schedule_id,
        last_run=datetime.utcnow(),
        next_run=self._calculate_next_run(schedule['cron_expression'])
    )

    logger.info(f"Scheduled scan queued: scan_id={scan_id}")

def load_schedules_on_startup(self):
    """Load all enabled schedules from database on app startup."""
    from web.services.schedule_service import ScheduleService

    schedule_service = ScheduleService(self.db_session)
    schedules = schedule_service.list_schedules(page=1, per_page=1000, enabled_filter=True)

    for schedule in schedules['schedules']:
        self.add_scheduled_scan(schedule)

    logger.info(f"Loaded {len(schedules['schedules'])} schedules on startup")

def _calculate_next_run(self, cron_expression):
    """Calculate next run time from cron expression."""
    cron = croniter(cron_expression, datetime.utcnow())
    return cron.get_next(datetime)

3. web/routes/main.py

Add schedule management routes.

Current State: 69 lines, only scan routes

Changes:

  • Add schedule routes:
    • GET /schedules - List schedules
    • GET /schedules/create - Create form
    • GET /schedules/<id>/edit - Edit form

New Code:

@bp.route('/schedules')
@login_required
def schedules():
    """List all schedules."""
    return render_template('schedules.html')

@bp.route('/schedules/create')
@login_required
def create_schedule():
    """Create new schedule form."""
    # Get list of available config files
    import os
    configs_dir = '/app/configs'
    config_files = [f for f in os.listdir(configs_dir) if f.endswith('.yaml')]
    return render_template('schedule_create.html', config_files=config_files)

@bp.route('/schedules/<int:schedule_id>/edit')
@login_required
def edit_schedule(schedule_id):
    """Edit existing schedule form."""
    from web.services.schedule_service import ScheduleService
    schedule_service = ScheduleService(current_app.db_session)

    try:
        schedule = schedule_service.get_schedule(schedule_id)
        return render_template('schedule_edit.html', schedule=schedule)
    except Exception as e:
        flash(f'Schedule not found: {e}', 'danger')
        return redirect(url_for('main.schedules'))

4. web/templates/base.html

Extract CSS, add Chart.js, fix white rows.

Current State: 346 lines with 280 lines of inline CSS (lines 8-288)

Changes:

  • Replace inline <style> block with <link> to external CSS
  • Add Chart.js CDN
  • Add dark theme configuration for Chart.js

Modifications:

<head>
    <meta charset="UTF-8">
    <meta name="viewport" content="width=device-width, initial-scale=1.0">
    <title>{% block title %}SneakyScanner{% endblock %}</title>

    <!-- Bootstrap 5 CSS -->
    <link href="https://cdn.jsdelivr.net/npm/bootstrap@5.3.0/dist/css/bootstrap.min.css" rel="stylesheet">

    <!-- Bootstrap Icons -->
    <link rel="stylesheet" href="https://cdn.jsdelivr.net/npm/bootstrap-icons@1.11.0/font/bootstrap-icons.css">

    <!-- Custom CSS (extracted from inline) -->
    <link rel="stylesheet" href="{{ url_for('static', filename='css/styles.css') }}">

    <!-- Chart.js for visualizations -->
    <script src="https://cdn.jsdelivr.net/npm/chart.js@4.4.0/dist/chart.umd.min.js"></script>

    <!-- Chart.js Dark Theme Configuration -->
    <script>
        Chart.defaults.color = '#e2e8f0';
        Chart.defaults.borderColor = '#334155';
        Chart.defaults.backgroundColor = '#1e293b';
    </script>
</head>

5. web/templates/dashboard.html

Fix white rows, add trending charts.

Current State: 356 lines, basic stats

Changes:

  • Fix white rows in renderScansTable() (add .scan-row class)
  • Add "Schedules" widget showing next scheduled scan
  • Add Chart.js trending chart (scans per day, last 30 days)
  • Add "Quick Actions" section

New Section - Schedules Widget:

<div class="col-md-6">
    <div class="card h-100">
        <div class="card-header d-flex justify-content-between align-items-center">
            <h5 class="mb-0">Upcoming Scheduled Scans</h5>
            <a href="{{ url_for('main.schedules') }}" class="btn btn-sm btn-secondary">Manage</a>
        </div>
        <div class="card-body">
            <div id="next-schedules">
                <p class="text-muted">Loading...</p>
            </div>
        </div>
    </div>
</div>

New Section - Trending Chart:

<div class="row mb-4">
    <div class="col-12">
        <div class="card">
            <div class="card-header">
                <h5 class="mb-0">Scan Activity (Last 30 Days)</h5>
            </div>
            <div class="card-body">
                <canvas id="scanTrendChart" height="80"></canvas>
            </div>
        </div>
    </div>
</div>

<script>
async function loadScanTrend() {
    const response = await fetch('/api/stats/scan-trend?days=30');
    const data = await response.json();

    const ctx = document.getElementById('scanTrendChart').getContext('2d');
    new Chart(ctx, {
        type: 'line',
        data: {
            labels: data.labels,  // ['2025-01-01', '2025-01-02', ...]
            datasets: [{
                label: 'Scans',
                data: data.values,  // [5, 3, 7, 2, ...]
                borderColor: '#60a5fa',
                backgroundColor: 'rgba(96, 165, 250, 0.1)',
                tension: 0.3
            }]
        },
        options: {
            responsive: true,
            maintainAspectRatio: false,
            plugins: {
                legend: {display: false}
            },
            scales: {
                y: {
                    beginAtZero: true,
                    ticks: {stepSize: 1}
                }
            }
        }
    });
}

loadScanTrend();
</script>

Fix for White Rows (line ~221):

function renderScansTable(scans) {
    const tbody = document.getElementById('recent-scans');
    tbody.innerHTML = '';

    if (scans.length === 0) {
        tbody.innerHTML = '<tr class="scan-row"><td colspan="5" class="text-center text-muted">No scans yet</td></tr>';
        return;
    }

    scans.forEach(scan => {
        const row = document.createElement('tr');
        row.classList.add('scan-row');  // ← FIX: Add explicit class
        row.onclick = () => window.location.href = `/scans/${scan.id}`;
        row.innerHTML = `
            <td class="mono">${scan.id}</td>
            <td>${scan.title || 'Untitled Scan'}</td>
            <td>${new Date(scan.timestamp).toLocaleString()}</td>
            <td>${getStatusBadge(scan.status)}</td>
        `;
        tbody.appendChild(row);
    });
}

6. web/templates/scans.html

Fix white rows bug.

Current State: 469 lines, white row bug at lines 208-241

Changes:

  • Add .scan-row class to dynamically created rows (same fix as dashboard)

Fix (line ~208):

function renderScansTable(scans) {
    const tbody = document.getElementById('scans-tbody');
    tbody.innerHTML = '';

    if (scans.length === 0) {
        tbody.innerHTML = '<tr class="scan-row"><td colspan="6" class="text-center text-muted">No scans found</td></tr>';
        return;
    }

    scans.forEach(scan => {
        const row = document.createElement('tr');
        row.classList.add('scan-row');  // ← FIX: Add explicit class
        row.onclick = () => window.location.href = `/scans/${scan.id}`;
        // ... rest of row creation
    });
}

7. web/templates/scan_detail.html

Fix white rows in port tables.

Current State: 399 lines, white rows in port tables (lines 305-327)

Changes:

  • Add .scan-row class to port table rows

Fix (in renderPortsTable function):

function renderPortsTable(ports, containerId) {
    const tbody = document.getElementById(containerId);
    tbody.innerHTML = '';

    ports.forEach(port => {
        const row = document.createElement('tr');
        row.classList.add('scan-row');  // ← FIX: Add explicit class
        row.innerHTML = `
            <td>${port.port}</td>
            <td>${port.protocol.toUpperCase()}</td>
            <td>${getExpectedBadge(port.expected)}</td>
            <td>${port.state}</td>
        `;
        tbody.appendChild(row);
    });
}

8. web/app.py

Load schedules on startup.

Current State: Initializes scheduler but doesn't load schedules

Changes:

  • After scheduler initialization, load all enabled schedules

New Code (after scheduler init):

def init_scheduler(app):
    """Initialize APScheduler and load schedules."""
    from web.services.scheduler_service import SchedulerService

    scheduler_service = SchedulerService()
    scheduler_service.init_scheduler(app)

    # Store in app context
    app.scheduler = scheduler_service

    # Load all enabled schedules from database
    with app.app_context():
        scheduler_service.load_schedules_on_startup()

    logger.info("Scheduler initialized and schedules loaded")

9. requirements-web.txt

Add croniter dependency.

Current State: All Phase 2 dependencies

Changes:

  • Add croniter for cron expression parsing

New Line:

croniter==2.0.1

Step-by-Step Implementation

Step 1: Fix Styling Issues & CSS Refactor ✅ (Day 1)

Priority: CRITICAL - Fixes user-facing bug

Tasks:

  1. Create web/static/css/styles.css

    • Extract all CSS from base.html <style> block
    • Add CSS variables for colors
    • Add fix for white row bug with .scan-row class
    • Add Chart.js dark theme styles

  2. Update web/templates/base.html

    • Remove inline <style> block
    • Add <link> to external styles.css
    • Add Chart.js CDN script
    • Add Chart.js default configuration for dark theme

  3. Fix white rows in all templates:

    • web/templates/dashboard.html - Add .scan-row class
    • web/templates/scans.html - Add .scan-row class
    • web/templates/scan_detail.html - Add .scan-row class to port tables

  4. Test CSS changes:

    • Verify all pages render correctly
    • Verify dark theme maintained
    • Verify table rows are dark slate, not white
    • Verify hover effects work

Deliverables:

  • web/static/css/styles.css (~350 lines)
  • Updated base.html (reduced by ~280 lines)
  • Fixed white rows in 3 templates
  • Chart.js ready for use in next steps

Testing:

  • Manual: View dashboard, scans list, scan detail - verify no white rows
  • Manual: Test hover effects on tables
  • Manual: Verify no CSS regressions

Step 2: ScheduleService Implementation ✅ (Days 2-3)

Priority: HIGH - Core business logic for schedules

Tasks:

  1. Create web/services/schedule_service.py (~400 lines)

    • Implement all CRUD methods
    • Add cron expression validation with croniter
    • Add next run time calculation
    • Add relative time formatting
    • Add schedule-to-dict serialization
    • Add execution history retrieval

  2. Add croniter to requirements

    • Update requirements-web.txt

  3. Write comprehensive unit tests

    • Create tests/test_schedule_service.py (~450 lines)
    • 18+ test functions covering all methods
    • Test cron validation edge cases
    • Test next run calculation accuracy
    • Test error handling

Key Methods to Implement:

  • create_schedule() - Validate and create
  • get_schedule() - Retrieve with history
  • list_schedules() - Paginated list
  • update_schedule() - Update with validation
  • delete_schedule() - Remove schedule
  • toggle_enabled() - Enable/disable
  • validate_cron_expression() - Validation
  • calculate_next_run() - Next run time

Deliverables:

  • ScheduleService class fully implemented
  • croniter dependency added
  • 18+ unit tests written and passing

Testing:

  • Unit: All ScheduleService methods
  • Edge cases: Invalid cron, missing config files
  • Concurrency: Multiple simultaneous schedule operations

Step 3: Schedules API Endpoints ✅ (Days 4-5)

Priority: HIGH - API for schedule management

Tasks:

  1. Update web/api/schedules.py (~300 lines total)

    • Replace all 6 stub endpoints with implementations
    • Add comprehensive error handling
    • Add input validation
    • Add logging
    • Integrate with ScheduleService and SchedulerService

  2. Endpoints to implement:

    • GET /api/schedules - List with pagination and filtering
    • GET /api/schedules/<id> - Get schedule details + history
    • POST /api/schedules - Create and add to scheduler
    • PUT /api/schedules/<id> - Update and reload in scheduler
    • DELETE /api/schedules/<id> - Delete and remove from scheduler
    • POST /api/schedules/<id>/trigger - Manually trigger scan

  3. Write integration tests

    • Create tests/test_schedule_api.py (~500 lines)
    • 22+ integration tests
    • Test complete workflows
    • Test authentication
    • Test error scenarios

Implementation Example:

@bp.route('', methods=['POST'])
@api_auth_required
def create_schedule():
    data = request.get_json() or {}

    # Validate
    if not all(k in data for k in ['name', 'config_file', 'cron_expression']):
        return jsonify({'error': 'Missing required fields'}), 400

    schedule_service = ScheduleService(current_app.db_session)

    try:
        schedule_id = schedule_service.create_schedule(
            name=data['name'],
            config_file=data['config_file'],
            cron_expression=data['cron_expression'],
            enabled=data.get('enabled', True)
        )

        # Add to APScheduler
        schedule = schedule_service.get_schedule(schedule_id)
        current_app.scheduler.add_scheduled_scan(schedule)

        return jsonify({
            'schedule_id': schedule_id,
            'message': 'Schedule created successfully'
        }), 201
    except ValueError as e:
        return jsonify({'error': str(e)}), 400

Deliverables:

  • All 6 API endpoints implemented
  • 22+ integration tests written and passing
  • Comprehensive error handling
  • API fully functional

Testing:

  • Integration: All endpoints with real database
  • Workflow: Create → update → trigger → delete
  • Error: Invalid inputs, missing schedules, auth failures

Step 4: Schedule Management UI ✅ (Days 6-7)

Priority: HIGH - User interface for schedules

Tasks:

  1. Create web/templates/schedules.html (~400 lines)

    • Stats cards (total, enabled, next run, executions)
    • Schedules table with enable/disable toggles
    • Next run time display (human-readable)
    • Action buttons (Run Now, Edit, Delete)
    • AJAX data loading
    • Auto-refresh every 30 seconds

  2. Create web/templates/schedule_create.html (~300 lines)

    • Schedule name input
    • Config file dropdown (list available configs)
    • Cron expression input with validation
    • Quick cron templates (daily, weekly, monthly)
    • Human-readable cron description
    • Next 5 run times preview
    • Enable/disable toggle
    • Submit button

  3. Create web/templates/schedule_edit.html (~250 lines)

    • Pre-populated form fields
    • Execution history (last 10 scans)
    • Delete schedule button
    • Test run button

  4. Update web/routes/main.py

    • Add /schedules route
    • Add /schedules/create route
    • Add /schedules/<id>/edit route

  5. Add cron helper JavaScript

    • Cron expression validator (client-side)
    • Human-readable formatter
    • Next runs calculator

Cron Expression Builder:

<!-- Quick Templates -->
<button onclick="setCron('0 0 * * *')">Daily at Midnight</button>
<button onclick="setCron('0 2 * * *')">Daily at 2 AM</button>
<button onclick="setCron('0 0 * * 0')">Weekly (Sunday)</button>
<button onclick="setCron('0 0 1 * *')">Monthly</button>

<!-- Manual Entry with Validation -->
<input type="text" id="cron-expression" onchange="validateCron()">
<div id="cron-description"><!-- Human-readable description --></div>
<div id="next-runs"><!-- Next 5 run times --></div>

Deliverables:

  • 3 new templates created
  • 3 new web routes added
  • Fully functional schedule management UI
  • Client-side cron validation

Testing:

  • Manual: Create schedule via UI
  • Manual: Edit existing schedule
  • Manual: Enable/disable toggle
  • Manual: Manual trigger works
  • Manual: Delete schedule
  • Verify: Cron validation prevents invalid expressions

Step 5: Enhanced Dashboard with Charts ✅ (Days 8-9)

Priority: MEDIUM - Improved dashboard visualization

Tasks:

  1. Add API endpoint for trend data

    • Create GET /api/stats/scan-trend?days=30
    • Return scan counts per day
    • Return labels and values for Chart.js

  2. Update web/templates/dashboard.html

    • Add "Schedules" widget showing next scheduled scans
    • Add Chart.js line chart for scan activity (last 30 days)
    • Add "Quick Actions" section (Run Scan, Create Schedule)
    • Improve layout with Bootstrap grid

  3. Add additional chart (optional)

    • Port count trend over time
    • Service distribution pie chart
    • Certificate expiry timeline

  4. Write tests for chart data

    • Create tests/test_charts.py (~200 lines)
    • Test data aggregation
    • Test date range filtering
    • Test empty data handling

Chart Implementation:

async function loadScanTrend() {
    const response = await fetch('/api/stats/scan-trend?days=30');
    const data = await response.json();

    const ctx = document.getElementById('scanTrendChart').getContext('2d');
    new Chart(ctx, {
        type: 'line',
        data: {
            labels: data.labels,
            datasets: [{
                label: 'Scans',
                data: data.values,
                borderColor: '#60a5fa',
                backgroundColor: 'rgba(96, 165, 250, 0.1)',
                tension: 0.3
            }]
        },
        options: {
            responsive: true,
            maintainAspectRatio: false,
            scales: {
                y: {beginAtZero: true}
            }
        }
    });
}

Deliverables:

  • Enhanced dashboard with charts
  • Schedule widget showing next runs
  • Stats API endpoint for trends
  • Chart tests written

Testing:

  • Manual: Verify charts render correctly
  • Manual: Verify dark theme applied to charts
  • Unit: Chart data generation
  • Edge: Empty data, single data point

Step 6: Scheduler Integration ✅ COMPLETE (Day 10)

Priority: CRITICAL - Complete scheduled execution

Tasks:

  1. Update web/services/scheduler_service.py

    • Complete add_scheduled_scan() implementation
    • Complete _trigger_scheduled_scan() implementation
    • Add load_schedules_on_startup() method
    • Add cron job to APScheduler with CronTrigger

  2. Update web/app.py

    • Call load_schedules_on_startup() after scheduler init
    • Ensure app context available for database queries

  3. Test scheduled execution

    • Create test schedule with cron "* * * * *" (every minute)
    • Verify scan is triggered automatically
    • Verify last_run and next_run are updated
    • Verify scans have triggered_by='scheduled'

  4. Test enable/disable

    • Disable schedule
    • Verify job removed from APScheduler
    • Enable schedule
    • Verify job added back to APScheduler

Key Implementation:

from apscheduler.triggers.cron import CronTrigger

def add_scheduled_scan(self, schedule):
    """Add cron job for scheduled scan."""
    job_id = f"schedule_{schedule['id']}"

    trigger = CronTrigger.from_crontab(schedule['cron_expression'])

    self.scheduler.add_job(
        func=self._trigger_scheduled_scan,
        trigger=trigger,
        id=job_id,
        args=[schedule['id']],
        replace_existing=True
    )

def _trigger_scheduled_scan(self, schedule_id):
    """Execute scheduled scan."""
    # Get schedule
    schedule = ScheduleService(self.db_session).get_schedule(schedule_id)

    # Trigger scan
    scan_id = ScanService(self.db_session).trigger_scan(
        config_file=schedule['config_file'],
        triggered_by='scheduled',
        schedule_id=schedule_id
    )

    # Queue scan
    self.queue_scan(schedule['config_file'], scan_id, self.db_url)

    # Update timestamps
    ScheduleService(self.db_session).update_run_times(
        schedule_id, datetime.utcnow(), self._calculate_next_run(...)
    )

Deliverables:

  • Scheduled scans execute automatically
  • Schedules loaded on app startup
  • Enable/disable functionality works
  • Timestamps updated after execution

Testing:

  • Integration: Create schedule with "* * * * *", wait 1 minute, verify scan created
  • Manual: Enable/disable schedule, verify APScheduler jobs
  • Edge: Schedule during scan, multiple schedules at same time

Step 7: Scan Comparison Features ✅ (Days 11-12)

Priority: MEDIUM - Historical analysis

Tasks:

  1. Add comparison API endpoint

    • GET /api/scans/<id1>/compare/<id2>
    • Return diff of ports, services, certificates
    • Highlight added, removed, changed items

  2. Add comparison UI

    • "Compare with Previous" button on scan detail page
    • Side-by-side comparison view
    • Color coding: green (added), red (removed), yellow (changed)

  3. Add historical charts to scan detail

    • Port count trend for this site
    • Service version changes over time
    • Certificate expiry timeline

  4. Add drift detection

    • Calculate "drift score" (how much changed)
    • Highlight unexpected changes
    • Link to alert creation

Comparison Algorithm:

def compare_scans(scan1_id, scan2_id):
    scan1 = ScanService.get_scan(scan1_id)
    scan2 = ScanService.get_scan(scan2_id)

    # Extract port lists
    ports1 = set(p['port'] for site in scan1['sites']
                 for ip in site['ips'] for p in ip['ports'])
    ports2 = set(p['port'] for site in scan2['sites']
                 for ip in site['ips'] for p in ip['ports'])

    return {
        'added_ports': list(ports2 - ports1),
        'removed_ports': list(ports1 - ports2),
        'unchanged_ports': list(ports1 & ports2),
        'services_changed': _compare_services(scan1, scan2),
        'certificates_changed': _compare_certs(scan1, scan2)
    }

Deliverables:

  • Scan comparison API endpoint
  • Comparison UI with diff view
  • Historical charts on scan detail
  • Drift score calculation

Testing:

  • Unit: Comparison algorithm with various scenarios
  • Manual: Compare two scans with differences
  • Edge: Compare same scan, compare non-existent scans

Step 8: Testing & Documentation ✅ (Days 13-14)

Priority: HIGH - Quality assurance

Tasks:

  1. Complete test coverage

    • Verify all 80+ tests passing
    • Add missing edge cases
    • Test concurrent schedule operations
    • Test cron execution accuracy

  2. Create manual testing checklist

    • Create docs/ai/MANUAL_TESTING_PHASE3.md
    • 30+ manual tests for all Phase 3 features
    • Include schedule creation, editing, execution
    • Include chart rendering verification

  3. Update documentation

    • Update README.md with Phase 3 features
    • Create docs/ai/PHASE3_COMPLETE.md
    • Update docs/ai/ROADMAP.md with Phase 3 completion

  4. Run full regression tests

    • Verify all Phase 2 tests still pass
    • Verify no breaking changes
    • Test backward compatibility

  5. Performance testing

    • Test with 50+ schedules
    • Test with 1000+ scans (chart performance)
    • Verify APScheduler handles many cron jobs

Documentation Updates:

README.md additions:

  • Scheduled scans section
  • Chart.js visualizations
  • Schedule management UI

PHASE3_COMPLETE.md sections:

  • What was delivered
  • Success criteria checklist
  • Known limitations
  • Testing results
  • Performance metrics
  • Lessons learned

Deliverables:

  • All 80+ tests passing
  • Manual testing checklist created
  • Documentation updated
  • Phase 3 completion summary
  • Performance validated

Testing:

  • Full test suite: pytest tests/ -v
  • Manual: Complete manual testing checklist
  • Performance: Load test with many schedules
  • Regression: Verify Phase 2 features unchanged

Dependencies & Prerequisites

Python Packages

Add to requirements-web.txt:

croniter==2.0.1  # Cron expression parsing

Already Present (from Phase 2):

  • Flask==3.0.0
  • SQLAlchemy==2.0.23
  • APScheduler==3.10.4
  • Flask-Login==0.6.3
  • Bootstrap 5 (CDN)

Frontend Libraries

Add to base.html:

  • Chart.js 4.4.0 (CDN): https://cdn.jsdelivr.net/npm/chart.js@4.4.0/dist/chart.umd.min.js

Already Present:

  • Bootstrap 5.3.0 (CDN)
  • Bootstrap Icons 1.11.0 (CDN)

System Requirements

  • Python 3.12+ (same as Phase 2)
  • Docker and Docker Compose
  • SQLite3 with WAL mode (already configured)

Configuration Files

No new configuration needed - Uses existing:

  • .env file (from Phase 2)
  • docker-compose-web.yml (no changes)
  • Database schema (Schedule table already exists)

Testing Approach

Unit Tests

New Test Files:

  1. tests/test_schedule_service.py (450 lines, 18+ tests)
  2. tests/test_charts.py (200 lines, 8+ tests)

Coverage Target: >80% for new code

Test Strategy:

  • Mock database calls for service tests
  • Test cron validation with valid/invalid expressions
  • Test next run calculations with various timezones
  • Test edge cases (empty data, null values)

Integration Tests

New Test Files:

  1. tests/test_schedule_api.py (500 lines, 22+ tests)

Test Strategy:

  • Use test database (separate from production)
  • Test complete workflows (create → update → execute → delete)
  • Test API authentication
  • Verify APScheduler integration

Manual Testing

Create: docs/ai/MANUAL_TESTING_PHASE3.md

Tests to Include:

  1. CSS/Styling (5 tests)

    • Verify white rows fixed
    • Verify dark theme consistent
    • Verify Chart.js dark theme

  2. Schedules (15 tests)

    • Create schedule via UI
    • Edit schedule
    • Enable/disable toggle
    • Manual trigger
    • Delete schedule
    • Cron validation
    • Next run display

  3. Dashboard (5 tests)

    • Verify charts render
    • Verify schedule widget
    • Verify stats accurate

  4. Scheduled Execution (5 tests)

    • Create schedule with "* * * * *"
    • Wait 1 minute, verify scan created
    • Verify timestamps updated
    • Disable schedule, verify stops
    • Enable schedule, verify resumes

Performance Tests

Scenarios:

  1. Load 100 schedules, verify startup time < 5 seconds
  2. Generate chart with 1000 scans, verify response < 2 seconds
  3. List 500 schedules with pagination, verify response < 500ms
  4. Concurrent schedule creation (10 simultaneous), verify no conflicts

Potential Challenges & Solutions

Challenge 1: Cron Expression Validation

Problem: Users may enter invalid cron expressions causing APScheduler errors.

Impact: High - Could crash scheduler

Solution:

  • Validate cron expressions with croniter before saving
  • Show human-readable description to user
  • Preview next 5 run times
  • Provide quick templates (daily, weekly, monthly)
  • Catch APScheduler exceptions and log errors

Implementation:

from croniter import croniter

def validate_cron_expression(cron_expr):
    try:
        # Test if cron is valid
        cron = croniter(cron_expr, datetime.utcnow())
        cron.get_next()  # Try to get next run
        return True, None
    except Exception as e:
        return False, f"Invalid cron expression: {str(e)}"

Challenge 2: Timezone Handling

Problem: Cron expressions are timezone-dependent. User expects "2 AM" in their local timezone, but server runs in UTC.

Impact: Medium - Schedules run at unexpected times

Solution:

  • Store all timestamps in UTC (already done in Phase 2)
  • Display times in user's browser timezone (JavaScript)
  • Document that cron expressions are in UTC
  • Phase 4 enhancement: Add timezone selector per schedule

Implementation:

// Display in user's local timezone
const utcTime = new Date(schedule.next_run + 'Z');  // Parse as UTC
document.getElementById('next-run').textContent = utcTime.toLocaleString();

Challenge 3: Concurrent Schedule Execution

Problem: If two schedules have same cron expression, both may trigger at exact same time, potentially overwhelming system.

Impact: Medium - Resource contention

Solution:

  • APScheduler already handles concurrency (ThreadPoolExecutor)
  • Max 3 concurrent scans (configured in Phase 2)
  • If more than 3 scans queued, they wait in queue
  • Add warning in UI if > 5 schedules have same cron

Implementation:

# In SchedulerService
self.scheduler = BackgroundScheduler(
    executors={
        'default': ThreadPoolExecutor(3)  # Max 3 concurrent
    },
    job_defaults={
        'coalesce': False,  # Don't combine missed runs
        'max_instances': 1   # Only 1 instance per job
    }
)

Challenge 4: Missed Schedule Execution

Problem: If app is down during scheduled time, scan is missed. When should it run after restart?

Impact: Medium - Data gaps

Solution:

  • APScheduler has misfire_grace_time setting
  • Set to 3600 seconds (1 hour)
  • If app down < 1 hour, scan will run when app starts
  • If app down > 1 hour, skip and wait for next scheduled time
  • Log missed schedules for admin review

Implementation:

self.scheduler = BackgroundScheduler(
    job_defaults={
        'misfire_grace_time': 3600  # Run if missed within 1 hour
    }
)

Challenge 5: Chart.js Dark Theme

Problem: Default Chart.js colors don't match dark theme (white background, black text).

Impact: Low - Aesthetic only

Solution:

  • Set Chart.js global defaults in base.html
  • Override colors, fonts, borders
  • Test all chart types (line, bar, pie)

Implementation:

Chart.defaults.color = '#e2e8f0';  // Light text
Chart.defaults.borderColor = '#334155';  // Slate borders
Chart.defaults.backgroundColor = '#1e293b';  // Dark background
Chart.defaults.font.family = "'Inter', sans-serif";

Challenge 6: Schedule Deletion with Active Scans

Problem: User deletes schedule while scan from that schedule is running. Should scan be cancelled?

Impact: Low - Edge case

Solution:

  • Do NOT cancel running scans
  • Only prevent future executions
  • Running scan completes normally
  • Database foreign key allows null schedule_id (schedule deleted but scan remains)
  • Document behavior in UI ("Active scans will complete")

Implementation:

def delete_schedule(self, schedule_id):
    # Remove from APScheduler (prevents future runs)
    self.scheduler.remove_scheduled_scan(schedule_id)

    # Delete from database (scans remain, schedule_id becomes null)
    schedule = self.db.query(Schedule).get(schedule_id)
    self.db.delete(schedule)
    self.db.commit()

    logger.info(f"Schedule {schedule_id} deleted. Active scans will complete.")

Challenge 7: CSS Extraction Breaking Existing Styles

Problem: Moving inline CSS to external file might cause selector specificity issues.

Impact: Medium - Visual regressions

Solution:

  • Extract CSS exactly as-is first
  • Test all pages after extraction
  • Only then refactor/improve
  • Use CSS variables for consistency
  • Keep !important flags where needed for dynamic rows

Testing:

  • Manual: View every page (dashboard, scans, scan detail, login, schedules)
  • Compare: Screenshots before and after extraction
  • Verify: Hover effects, active states, responsive behavior

Success Criteria

Phase 3 is COMPLETE when all criteria are met:

Bug Fixes

  • White row bug fixed in dashboard.html
  • White row bug fixed in scans.html
  • White row bug fixed in scan_detail.html
  • All tables use dark theme consistently
  • Hover effects work correctly

CSS & Styling

  • CSS extracted to web/static/css/styles.css
  • base.html links to external CSS
  • Chart.js added and configured for dark theme
  • CSS variables used for colors
  • No visual regressions on any page

Schedule Service

  • ScheduleService fully implemented (~400 lines)
  • All CRUD methods working
  • Cron validation with croniter
  • Next run time calculation accurate
  • 18+ unit tests passing

Schedules API

  • All 6 API endpoints implemented
  • GET /api/schedules lists with pagination
  • POST /api/schedules creates and validates
  • PUT /api/schedules/ updates and reloads
  • DELETE /api/schedules/ removes schedule
  • POST /api/schedules//trigger manually triggers
  • 22+ integration tests passing

Schedule UI

  • schedules.html lists all schedules
  • schedule_create.html creates new schedules
  • schedule_edit.html edits existing schedules
  • Enable/disable toggle works
  • Manual trigger button works
  • Cron validation prevents invalid expressions
  • Human-readable cron description shown
  • Next 5 run times previewed

Scheduled Execution

  • Schedules load on app startup
  • Cron jobs added to APScheduler
  • Scans execute automatically at scheduled time
  • last_run and next_run timestamps updated
  • Scans have triggered_by='scheduled'
  • Enable/disable adds/removes jobs

Dashboard Enhancements

  • Schedule widget shows next scheduled scans
  • Trending chart displays scan activity (last 30 days)
  • Chart.js renders with dark theme
  • Stats accurate (total, enabled, next run)

Scan Comparison

  • Comparison API endpoint returns diffs
  • "Compare with Previous" button on scan detail
  • Side-by-side comparison view
  • Color coding for added/removed/changed

Testing

  • All 80+ new tests passing
  • All Phase 2 tests still passing
  • Manual testing checklist completed
  • Performance acceptable (50+ schedules, 1000+ scans)

Documentation

  • README.md updated with Phase 3 features
  • PHASE3_COMPLETE.md created
  • ROADMAP.md updated with Phase 3 completion
  • Manual testing checklist created

Migration Path

From Phase 2 to Phase 3

No Breaking Changes:

  • All Phase 2 APIs remain functional
  • Database schema unchanged (Schedule table already exists)
  • Existing scans, users, settings unchanged
  • Docker deployment compatible

Additions:

  • New schedules API endpoints (stubs → implementations)
  • New schedule management UI
  • New dashboard charts
  • External CSS file (improves maintainability)
  • croniter dependency

Migration Steps:

  1. Pull latest code from phase3 branch
  2. Install new dependency: pip install croniter==2.0.1
  3. No database migration needed (Schedule table already exists from Phase 1)
  4. Rebuild Docker image: docker-compose -f docker-compose-web.yml build
  5. Restart services: docker-compose -f docker-compose-web.yml restart
  6. Verify schedules load on startup (check logs)

Backward Compatibility:

  • CLI scanner: Continues to work standalone (unchanged)
  • Existing scans: All viewable in web UI
  • Phase 2 features: All functional (no regressions)
  • API clients: All Phase 2 endpoints unchanged

Estimated Timeline

Total Duration: 14 working days (2 weeks)

Week 1: Backend Foundation

  • Day 1: Fix styling issues & CSS refactor

    • Extract CSS to external file
    • Fix white row bug
    • Add Chart.js

  • Days 2-3: ScheduleService implementation

    • Create ScheduleService class
    • Implement CRUD methods
    • Add cron validation
    • Write unit tests

  • Days 4-5: Schedules API endpoints

    • Implement all 6 endpoints
    • Add error handling
    • Write integration tests

  • Days 6-7: Schedule Management UI

    • Create schedules list page
    • Create schedule create form
    • Create schedule edit form
    • Add web routes

Week 2: Enhancements & Polish

  • Days 8-9: Enhanced dashboard with charts

    • Add trending charts
    • Add schedule widget
    • Create stats API endpoints

  • Day 10: Scheduler integration

    • Complete scheduled execution
    • Load schedules on startup
    • Test cron execution

  • Days 11-12: Scan comparison features

    • Implement comparison algorithm
    • Create comparison UI
    • Add historical charts

  • Days 13-14: Testing & documentation

    • Complete test suite
    • Manual testing
    • Update documentation
    • Phase 3 completion summary

Critical Path

Must Complete in Order:

  1. Day 1: CSS extraction and white row fix (prerequisite for all UI work)
  2. Days 2-3: ScheduleService (prerequisite for API)
  3. Days 4-5: Schedules API (prerequisite for UI)
  4. Days 6-7: Schedule UI (can proceed once API done)

Can Proceed in Parallel:

  • Dashboard charts (Days 8-9) can start any time after Day 1
  • Scan comparison (Days 11-12) independent of schedules
  • Testing (Days 13-14) can start earlier for completed components

Key Design Decisions

Decision 1: Cron Expression Library

Choice: croniter

Alternatives Considered:

  • APScheduler's built-in cron parsing
  • python-crontab
  • Manual parsing with regex

Rationale:

  • croniter is lightweight and focused
  • Accurate next run calculation
  • Handles edge cases (leap years, DST)
  • Easy validation and error messages
  • Well-maintained (active development)

Trade-offs:

  • ✅ Accurate and reliable
  • ✅ Simple API
  • ✅ Good error messages
  • ❌ Another dependency (acceptable - small library)

Decision 2: External CSS vs Inline

Choice: Extract to external CSS file

Alternatives Considered:

  • Keep inline CSS in base.html
  • Use CSS-in-JS
  • Use CSS framework (Tailwind)

Rationale:

  • Better maintainability (single source of truth)
  • Easier to customize theme
  • Browser caching (performance)
  • Separation of concerns
  • Easier to test (CSS linting)

Trade-offs:

  • ✅ Maintainable and organized
  • ✅ Better performance (caching)
  • ✅ Easier theme customization
  • ❌ One more HTTP request (minimal impact with HTTP/2)

Decision 3: Chart.js for Visualizations

Choice: Chart.js 4.4.0

Alternatives Considered:

  • Plotly.js (more powerful, heavier)
  • D3.js (maximum flexibility, steeper learning curve)
  • ApexCharts (modern, but less documented)
  • Server-side rendering (matplotlib → image)

Rationale:

  • Lightweight (< 200KB)
  • Simple API
  • Good dark theme support
  • Responsive by default
  • Well-documented
  • Active community

Trade-offs:

  • ✅ Easy to implement
  • ✅ Good performance
  • ✅ Responsive
  • ❌ Less powerful than Plotly (sufficient for Phase 3)
  • ❌ Not as customizable as D3 (acceptable trade-off)

Decision 4: Schedule Deletion Behavior

Choice: Delete schedule but keep associated scans

Alternatives Considered:

  • Cascade delete (delete scans too)
  • Soft delete (mark schedule as deleted)
  • Prevent deletion if scans exist

Rationale:

  • Scans are historical data (valuable)
  • Schedule is just a template (can be recreated)
  • User might want to delete schedule but keep history
  • Deleting scans would cause data loss

Trade-offs:

  • ✅ Preserves historical data
  • ✅ Flexible (can recreate schedule)
  • ❌ Schedule reference in scans becomes null (acceptable)

Decision 5: Timezone Handling

Choice: Store all times in UTC, display in user's browser timezone

Alternatives Considered:

  • Store in user's timezone
  • Add timezone field to Schedule model
  • Use server's local timezone

Rationale:

  • UTC is standard for databases
  • Avoids DST issues
  • Browser automatically converts to local time
  • Consistent with Phase 2 approach
  • Simple and reliable

Trade-offs:

  • ✅ No DST issues
  • ✅ Consistent across users
  • ✅ Simple implementation
  • ❌ Cron expressions in UTC (document clearly)
  • ❌ Phase 4 enhancement: per-schedule timezone

Decision 6: Scan Comparison Scope

Choice: Start with port and service comparison, defer certificate comparison to Phase 4

Alternatives Considered:

  • Full comparison (ports, services, certs, TLS)
  • Only port comparison
  • Advanced diff with visualization

Rationale:

  • Ports and services are most important for drift detection
  • Certificates are complex (multiple per scan, expiry dates)
  • Keep Phase 3 scope manageable
  • Certificate comparison better suited for Phase 4 (alerts)

Trade-offs:

  • ✅ Achievable in 2 days
  • ✅ Most valuable comparison (ports/services)
  • ❌ Not comprehensive (acceptable for Phase 3)
  • ✅ Can enhance in Phase 4

Documentation Deliverables

1. Updated README.md

New Sections:

  • Scheduled Scans
    • How to create schedules
    • Cron expression syntax
    • Enable/disable schedules
  • Dashboard Enhancements
    • Trending charts
    • Schedule widget
  • Scan Comparison
    • Compare feature
    • Historical analysis

2. PHASE3_COMPLETE.md

Sections:

  • What was delivered (8 steps)
  • Success criteria checklist
  • Files created/modified summary
  • Testing results
  • Performance metrics
  • Known limitations
  • Lessons learned
  • What's next (Phase 4)

3. MANUAL_TESTING_PHASE3.md

Sections:

  • Prerequisites
  • CSS/Styling tests (5 tests)
  • Schedule management tests (15 tests)
  • Dashboard tests (5 tests)
  • Scheduled execution tests (5 tests)
  • Scan comparison tests (3 tests)
  • Performance tests (3 tests)
  • Test results summary

4. Updated ROADMAP.md

Changes:

  • Mark Phase 3 as COMPLETE ✅
  • Update progress overview
  • Add Phase 3 deliverables section
  • Update success criteria
  • Update changelog
  • Set next review for Phase 4

End of Phase 3 Plan

This plan will be followed during Phase 3 implementation. Upon completion, PHASE3_COMPLETE.md will summarize actual implementation, challenges encountered, and lessons learned.

Reference in New Issue View Git Blame Copy Permalink