Files
SneakyScan/docs/ai/PHASE3.md
Phillip Tarrant 7969068c36 Phase 3 Step 2: Implement ScheduleService with cron support
Implement comprehensive schedule management service for automated scans:

New Files:
- web/services/schedule_service.py (470 lines)
  * Complete CRUD operations for schedules
  * Cron expression validation using croniter
  * Next run time calculation
  * Execution history tracking
  * Human-readable relative time formatting

- tests/test_schedule_service.py (671 lines, 40+ tests)
  * Create/get/list/update/delete schedule tests
  * Cron validation and next run calculation tests
  * Pagination and filtering tests
  * Schedule history and serialization tests

Changes:
- requirements-web.txt: Add croniter==2.0.1 dependency
- docs/ai/PHASE3.md: Mark Step 1 complete, Step 2 next

Key Features:
- Validates cron expressions before saving
- Automatically calculates next execution time
- Preserves historical scans when schedules deleted
- Supports pagination and filtering by enabled status
- Provides relative time display (e.g., "in 2 hours")
2025-11-14 13:41:49 -06:00

2205 lines
64 KiB
Markdown

# Phase 3 Implementation Plan: Dashboard Enhancement & Scheduled Scans
**Status:** Ready to Start
**Progress:** 0/14 days complete (0%)
**Estimated Duration:** 14 days (2 weeks)
**Dependencies:** Phase 2 Complete ✅
## Progress Summary
-**Step 1: Fix Styling Issues & CSS Refactor** (Day 1) - COMPLETE
- 📋 **Step 2: ScheduleService Implementation** (Days 2-3) - NEXT
- 📋 **Step 3: Schedules API Endpoints** (Days 4-5)
- 📋 **Step 4: Schedule Management UI** (Days 6-7)
- 📋 **Step 5: Enhanced Dashboard with Charts** (Days 8-9)
- 📋 **Step 6: Scheduler Integration** (Day 10)
- 📋 **Step 7: Scan Comparison Features** (Days 11-12)
- 📋 **Step 8: Testing & Documentation** (Days 13-14)
---
## Table of Contents
1. [Overview](#overview)
2. [Current State Analysis](#current-state-analysis)
3. [Critical Bug Fix](#critical-bug-fix)
4. [Files to Create](#files-to-create)
5. [Files to Modify](#files-to-modify)
6. [Step-by-Step Implementation](#step-by-step-implementation)
7. [Dependencies & Prerequisites](#dependencies--prerequisites)
8. [Testing Approach](#testing-approach)
9. [Potential Challenges & Solutions](#potential-challenges--solutions)
10. [Success Criteria](#success-criteria)
11. [Migration Path](#migration-path)
12. [Estimated Timeline](#estimated-timeline)
13. [Key Design Decisions](#key-design-decisions)
14. [Documentation Deliverables](#documentation-deliverables)
---
## Overview
Phase 3 focuses on enhancing the web application with scheduling capabilities and improved dashboard visualizations:
1. **Critical Bug Fix** - Fix white row coloring in scan tables
2. **Scheduled Scans** - Complete schedule management system with cron expressions
3. **Enhanced Dashboard** - Trending charts, schedule widgets, alert summaries
4. **Scan Comparison** - Historical analysis and infrastructure drift detection
### Goals
- 🐛 Fix white row bug affecting scan tables (critical UX issue)
- ✅ Complete schedule management (CRUD operations)
- ✅ Scheduled scan execution with cron expressions
- ✅ Enhanced dashboard with Chart.js visualizations
- ✅ Scan comparison and historical analysis
- ✅ CSS extraction and better maintainability
---
## Current State Analysis
### What's Already Done (Phase 2)
**✅ Phase 2 Complete** - Full web application core:
- **Database Schema** - All 11 models including Schedule table
- **ScanService** - Complete CRUD operations with 545 lines
- **Scan API** - 5 endpoints fully implemented
- **SchedulerService** - Partial implementation (258 lines)
-`queue_scan()` - Immediate scan execution works
- ⚠️ `add_scheduled_scan()` - Placeholder (marked for Phase 3)
- ⚠️ `_trigger_scheduled_scan()` - Skeleton with TODO comments
-`remove_scheduled_scan()` - Basic implementation
- **Authentication** - Flask-Login with @login_required decorators
- **UI Templates** - Dashboard, scans list/detail, login pages
- **Background Jobs** - APScheduler with concurrent execution (max 3)
- **Error Handling** - Content negotiation, custom error pages
- **Testing** - 100 test functions, 1,825 lines of test code
### Schedule-Related Components Analysis
#### Schedule Model (`web/models.py` - lines 142-158)
```python
class Schedule(Base):
__tablename__ = 'schedules'
id = Column(Integer, primary_key=True)
name = Column(String(255), nullable=False)
config_file = Column(Text, nullable=False)
cron_expression = Column(String(100), nullable=False)
enabled = Column(Boolean, default=True)
last_run = Column(DateTime)
next_run = Column(DateTime)
created_at = Column(DateTime, default=datetime.utcnow)
updated_at = Column(DateTime, default=datetime.utcnow, onupdate=datetime.utcnow)
```
**Status:** ✅ Complete - No migration needed
#### Schedules API (`web/api/schedules.py` - 159 lines)
```python
@bp.route('', methods=['GET'])
@api_auth_required
def list_schedules():
"""List all schedules. TODO: Implement in Phase 3."""
return jsonify({'message': 'Schedules list - to be implemented in Phase 3'}), 200
```
**Status:** ⚠️ All 6 endpoints are stubs returning placeholders
**Endpoints to Implement:**
- `GET /api/schedules` - List all schedules
- `GET /api/schedules/<id>` - Get schedule details
- `POST /api/schedules` - Create new schedule
- `PUT /api/schedules/<id>` - Update schedule
- `DELETE /api/schedules/<id>` - Delete schedule
- `POST /api/schedules/<id>/trigger` - Manually trigger scheduled scan
#### SchedulerService Gaps
**Missing Implementations:**
1. `add_scheduled_scan(schedule)` - Currently placeholder
2. `_trigger_scheduled_scan(schedule_id)` - TODO comments only
3. Loading schedules on app startup - Not implemented
4. Cron expression parsing - No validation
5. Next run time calculation - Missing
### UI Components Status
**Existing Templates:**
- `web/templates/base.html` (346 lines) - Has inline CSS (need to extract)
- `web/templates/dashboard.html` (356 lines) - Basic stats, needs charts
- `web/templates/scans.html` (469 lines) - Has white row bug
- `web/templates/scan_detail.html` (399 lines) - Has white row bug in port tables
**Missing Templates:**
- Schedule list page
- Schedule create form
- Schedule edit form
**Web Routes (`web/routes/main.py` - 69 lines):**
- Only has scan routes, no schedule routes
---
## Critical Bug Fix
### White Row Coloring Issue
**Problem:** Dynamically created table rows in scans tables display with white background instead of dark theme colors.
**Affected Files:**
1. `web/templates/scans.html` (lines 208-241) - renderScansTable()
2. `web/templates/dashboard.html` (lines 221-260) - renderScansTable()
3. `web/templates/scan_detail.html` (lines 305-327) - Port tables
**Root Cause:**
JavaScript dynamically creates `<tr>` elements that don't inherit CSS styles properly:
```javascript
const row = document.createElement('tr');
row.innerHTML = `
<td class="mono">${scan.id}</td>
<td>${scan.title || 'Untitled Scan'}</td>
...
`;
tbody.appendChild(row);
```
**Current CSS (base.html lines 157-165):**
```css
.table tbody tr {
background-color: #1e293b; /* Dark slate */
border-color: #334155;
}
.table tbody tr:hover {
background-color: #334155;
cursor: pointer;
}
```
**Solution:**
Add explicit class to dynamically created rows:
```javascript
const row = document.createElement('tr');
row.classList.add('scan-row'); // Add explicit class
row.innerHTML = `...`;
```
And enhance CSS with higher specificity:
```css
.table tbody tr.scan-row,
.table tbody tr {
background-color: #1e293b !important;
border-color: #334155 !important;
}
```
---
## Files to Create
### Backend Services
#### 1. `web/services/schedule_service.py`
Schedule CRUD operations and business logic.
**Class: ScheduleService**
**Estimated Size:** ~400 lines
**Methods:**
- `__init__(self, db_session)` - Initialize with database session
- `create_schedule(name, config_file, cron_expression, enabled=True)` → schedule_id
- Validate cron expression
- Validate config file exists
- Calculate next_run time
- Create Schedule record
- Return schedule_id
- `get_schedule(schedule_id)` → schedule dict
- Query Schedule by ID
- Format for API response
- Include execution history (recent scans with this schedule_id)
- `list_schedules(page=1, per_page=20, enabled_filter=None)` → paginated results
- Query with pagination
- Filter by enabled status if provided
- Calculate relative next_run time ("in 2 hours", "tomorrow at 3:00 PM")
- Return total count and items
- `update_schedule(schedule_id, **updates)` → schedule dict
- Validate cron_expression if changed
- Recalculate next_run if cron changed
- Update Schedule record
- Reload schedule in APScheduler
- Return updated schedule
- `delete_schedule(schedule_id)` → success
- Remove job from APScheduler
- Delete Schedule record (do NOT delete associated scans)
- Return success status
- `toggle_enabled(schedule_id, enabled)` → schedule dict
- Enable or disable schedule
- Add/remove from APScheduler
- Return updated schedule
- `update_run_times(schedule_id, last_run, next_run)` → success
- Update last_run and next_run timestamps
- Called after each execution
- Return success status
- `validate_cron_expression(cron_expr)` → (valid, error_message)
- Use croniter library
- Return True if valid, False + error message if invalid
- `calculate_next_run(cron_expr, from_time=None)` → datetime
- Calculate next run time from cron expression
- Use croniter
- Return datetime (UTC)
- `get_schedule_history(schedule_id, limit=10)` → list of scans
- Query recent scans triggered by this schedule
- Return list of scan dicts
**Helper Methods:**
- `_schedule_to_dict(schedule_obj)` - Convert Schedule model to dict
- `_get_relative_time(dt)` - Format datetime as "in 2 hours", "yesterday"
#### 2. `web/static/css/styles.css`
Extracted CSS for better maintainability.
**Estimated Size:** ~350 lines (extracted from base.html)
**Sections:**
- Variables (CSS custom properties for colors)
- Global styles
- Navigation
- Cards and containers
- Tables (including fix for white rows)
- Forms
- Buttons
- Badges and labels
- Charts (dark theme)
- Utilities
**Example Structure:**
```css
/* CSS Variables */
:root {
--bg-primary: #0f172a;
--bg-secondary: #1e293b;
--bg-tertiary: #334155;
--text-primary: #e2e8f0;
--text-secondary: #94a3b8;
--border-color: #334155;
--accent-blue: #60a5fa;
--success-bg: #065f46;
--success-text: #6ee7b7;
--warning-bg: #78350f;
--warning-text: #fcd34d;
--danger-bg: #7f1d1d;
--danger-text: #fca5a5;
}
/* Fix for dynamically created table rows */
.table tbody tr,
.table tbody tr.scan-row {
background-color: var(--bg-secondary) !important;
border-color: var(--border-color) !important;
}
.table tbody tr:hover {
background-color: var(--bg-tertiary) !important;
cursor: pointer;
}
```
### Frontend Templates
#### 3. `web/templates/schedules.html`
List all schedules with enable/disable toggles.
**Estimated Size:** ~400 lines
**Layout:**
```html
{% extends "base.html" %}
{% block content %}
<div class="container mt-4">
<div class="d-flex justify-content-between align-items-center mb-4">
<h2>Scheduled Scans</h2>
<a href="{{ url_for('main.create_schedule') }}" class="btn btn-primary">
<i class="bi bi-plus-circle"></i> New Schedule
</a>
</div>
<!-- Stats Cards -->
<div class="row mb-4">
<div class="col-md-3">
<div class="card">
<div class="card-body">
<h6>Total Schedules</h6>
<h2 id="total-schedules">-</h2>
</div>
</div>
</div>
<div class="col-md-3">
<div class="card">
<div class="card-body">
<h6>Enabled</h6>
<h2 id="enabled-schedules">-</h2>
</div>
</div>
</div>
<div class="col-md-3">
<div class="card">
<div class="card-body">
<h6>Next Run</h6>
<h5 id="next-run-time">-</h5>
</div>
</div>
</div>
<div class="col-md-3">
<div class="card">
<div class="card-body">
<h6>Executions (24h)</h6>
<h2 id="recent-executions">-</h2>
</div>
</div>
</div>
</div>
<!-- Schedules Table -->
<div class="card">
<div class="card-body">
<table class="table">
<thead>
<tr>
<th>Name</th>
<th>Schedule (Cron)</th>
<th>Next Run</th>
<th>Last Run</th>
<th>Status</th>
<th>Actions</th>
</tr>
</thead>
<tbody id="schedules-tbody">
<!-- Populated by JavaScript -->
</tbody>
</table>
</div>
</div>
</div>
<script>
// Fetch and render schedules
async function loadSchedules() {
const response = await fetch('/api/schedules');
const data = await response.json();
renderSchedulesTable(data.schedules);
updateStats(data);
}
function renderSchedulesTable(schedules) {
const tbody = document.getElementById('schedules-tbody');
tbody.innerHTML = '';
schedules.forEach(schedule => {
const row = document.createElement('tr');
row.classList.add('schedule-row'); // Fix white rows
row.innerHTML = `
<td>${schedule.name}</td>
<td><code>${schedule.cron_expression}</code></td>
<td>${formatRelativeTime(schedule.next_run)}</td>
<td>${formatRelativeTime(schedule.last_run) || 'Never'}</td>
<td>
<div class="form-check form-switch">
<input class="form-check-input" type="checkbox"
id="enable-${schedule.id}"
${schedule.enabled ? 'checked' : ''}
onchange="toggleSchedule(${schedule.id}, this.checked)">
</div>
</td>
<td>
<button class="btn btn-sm btn-secondary" onclick="triggerSchedule(${schedule.id})">
Run Now
</button>
<a href="/schedules/${schedule.id}/edit" class="btn btn-sm btn-secondary">
Edit
</a>
<button class="btn btn-sm btn-danger" onclick="deleteSchedule(${schedule.id})">
Delete
</button>
</td>
`;
tbody.appendChild(row);
});
}
// Toggle schedule enabled/disabled
async function toggleSchedule(scheduleId, enabled) {
await fetch(`/api/schedules/${scheduleId}`, {
method: 'PUT',
headers: {'Content-Type': 'application/json'},
body: JSON.stringify({enabled: enabled})
});
loadSchedules();
}
// Manually trigger schedule
async function triggerSchedule(scheduleId) {
const response = await fetch(`/api/schedules/${scheduleId}/trigger`, {
method: 'POST'
});
const data = await response.json();
alert(`Scan triggered! Scan ID: ${data.scan_id}`);
window.location.href = `/scans/${data.scan_id}`;
}
// Delete schedule
async function deleteSchedule(scheduleId) {
if (!confirm('Delete this schedule?')) return;
await fetch(`/api/schedules/${scheduleId}`, {method: 'DELETE'});
loadSchedules();
}
// Load on page load
loadSchedules();
setInterval(loadSchedules, 30000); // Refresh every 30 seconds
</script>
{% endblock %}
```
#### 4. `web/templates/schedule_create.html`
Form to create new scheduled scan.
**Estimated Size:** ~300 lines
**Features:**
- Schedule name input
- Config file selector (dropdown of available configs)
- Cron expression builder OR manual entry
- Cron expression validator (client-side and server-side)
- Human-readable cron description
- Enable/disable toggle
- Submit button
**Cron Expression Builder:**
```html
<div class="card mb-3">
<div class="card-body">
<h5>Schedule Configuration</h5>
<!-- Quick Templates -->
<div class="mb-3">
<label>Quick Templates:</label>
<div class="btn-group" role="group">
<button type="button" class="btn btn-secondary btn-sm" onclick="setCron('0 0 * * *')">Daily at Midnight</button>
<button type="button" class="btn btn-secondary btn-sm" onclick="setCron('0 2 * * *')">Daily at 2 AM</button>
<button type="button" class="btn btn-secondary btn-sm" onclick="setCron('0 0 * * 0')">Weekly (Sunday)</button>
<button type="button" class="btn btn-secondary btn-sm" onclick="setCron('0 0 1 * *')">Monthly (1st)</button>
</div>
</div>
<!-- Manual Entry -->
<div class="mb-3">
<label for="cron-expression">Cron Expression:</label>
<input type="text" class="form-control font-monospace"
id="cron-expression"
name="cron_expression"
placeholder="0 2 * * *"
onchange="validateCron()">
<small class="form-text text-muted">
Format: minute hour day month weekday
</small>
</div>
<!-- Human-Readable Description -->
<div class="alert alert-info" id="cron-description">
Enter a cron expression above
</div>
<!-- Next Run Times -->
<div id="next-runs">
<strong>Next 5 runs:</strong>
<ul id="next-runs-list"></ul>
</div>
</div>
</div>
```
#### 5. `web/templates/schedule_edit.html`
Form to edit existing schedule.
**Estimated Size:** ~250 lines
**Similar to create, but:**
- Pre-populate fields from existing schedule
- Show execution history (last 10 scans)
- "Delete Schedule" button
- "Test Run Now" button
### Testing Files
#### 6. `tests/test_schedule_service.py`
Unit tests for ScheduleService.
**Estimated Size:** ~450 lines, 18+ tests
**Test Coverage:**
- `test_create_schedule` - Valid schedule creation
- `test_create_schedule_invalid_cron` - Cron validation
- `test_create_schedule_invalid_config` - Config file validation
- `test_get_schedule` - Retrieve schedule
- `test_get_schedule_not_found` - 404 handling
- `test_list_schedules` - Pagination
- `test_list_schedules_filter_enabled` - Filter by enabled status
- `test_update_schedule` - Update fields
- `test_update_schedule_cron` - Recalculate next_run when cron changes
- `test_delete_schedule` - Delete schedule
- `test_toggle_enabled` - Enable/disable
- `test_update_run_times` - Update after execution
- `test_validate_cron_expression_valid` - Valid expressions
- `test_validate_cron_expression_invalid` - Invalid expressions
- `test_calculate_next_run` - Next run calculation
- `test_get_schedule_history` - Execution history
- `test_schedule_to_dict` - Serialization
- `test_concurrent_schedule_operations` - Thread safety
#### 7. `tests/test_schedule_api.py`
Integration tests for schedules API.
**Estimated Size:** ~500 lines, 22+ tests
**Test Coverage:**
- `test_list_schedules_empty` - Empty list
- `test_list_schedules_populated` - Multiple schedules
- `test_list_schedules_pagination` - Pagination
- `test_list_schedules_filter_enabled` - Filter
- `test_get_schedule` - Get details
- `test_get_schedule_not_found` - 404
- `test_create_schedule` - Create new
- `test_create_schedule_invalid_cron` - Validation
- `test_create_schedule_invalid_config` - File validation
- `test_update_schedule` - Update fields
- `test_update_schedule_not_found` - 404
- `test_delete_schedule` - Delete
- `test_delete_schedule_not_found` - 404
- `test_trigger_schedule` - Manual trigger
- `test_trigger_schedule_not_found` - 404
- `test_toggle_enabled_via_update` - Enable/disable
- `test_schedules_require_authentication` - Auth required
- `test_schedule_execution_history` - Show related scans
- `test_schedule_next_run_calculation` - Correct calculation
- `test_schedule_workflow_integration` - Complete workflow
- `test_concurrent_schedule_updates` - Concurrency
#### 8. `tests/test_charts.py`
Tests for chart data generation.
**Estimated Size:** ~200 lines, 8+ tests
**Test Coverage:**
- `test_scan_trend_data` - Scans per day calculation
- `test_port_count_trend` - Port count over time
- `test_service_distribution` - Service type breakdown
- `test_certificate_expiry_timeline` - Cert expiry dates
- `test_empty_data_handling` - No scans case
- `test_date_range_filtering` - Filter by date range
- `test_data_format_for_chartjs` - Correct JSON format
- `test_trend_data_caching` - Performance optimization
---
## Files to Modify
### Backend Updates
#### 1. `web/api/schedules.py`
Replace all stub implementations with working code.
**Current State:** 159 lines, all placeholders
**Changes:**
- Import ScheduleService
- Implement all 6 endpoints:
- `GET /api/schedules` - Call ScheduleService.list_schedules()
- `GET /api/schedules/<id>` - Call ScheduleService.get_schedule()
- `POST /api/schedules` - Call ScheduleService.create_schedule()
- `PUT /api/schedules/<id>` - Call ScheduleService.update_schedule()
- `DELETE /api/schedules/<id>` - Call ScheduleService.delete_schedule()
- `POST /api/schedules/<id>/trigger` - Trigger immediate scan with schedule_id
**New Code (~300 lines total):**
```python
from web.services.schedule_service import ScheduleService
from web.auth.decorators import api_auth_required
from flask import current_app, jsonify, request
@bp.route('', methods=['GET'])
@api_auth_required
def list_schedules():
"""List all schedules with pagination."""
page = request.args.get('page', 1, type=int)
per_page = request.args.get('per_page', 20, type=int)
enabled_filter = request.args.get('enabled', type=lambda x: x.lower() == 'true')
schedule_service = ScheduleService(current_app.db_session)
result = schedule_service.list_schedules(page, per_page, enabled_filter)
return jsonify(result), 200
@bp.route('', methods=['POST'])
@api_auth_required
def create_schedule():
"""Create a new schedule."""
data = request.get_json() or {}
# Validate required fields
required = ['name', 'config_file', 'cron_expression']
for field in required:
if field not in data:
return jsonify({'error': f'Missing required field: {field}'}), 400
schedule_service = ScheduleService(current_app.db_session)
try:
schedule_id = schedule_service.create_schedule(
name=data['name'],
config_file=data['config_file'],
cron_expression=data['cron_expression'],
enabled=data.get('enabled', True)
)
# Add to APScheduler
schedule = schedule_service.get_schedule(schedule_id)
current_app.scheduler.add_scheduled_scan(schedule)
return jsonify({
'schedule_id': schedule_id,
'message': 'Schedule created successfully'
}), 201
except ValueError as e:
return jsonify({'error': str(e)}), 400
# ... more endpoints
```
#### 2. `web/services/scheduler_service.py`
Complete placeholder implementations.
**Current State:** 258 lines, partial implementation
**Changes:**
- Complete `add_scheduled_scan(schedule)` - Add cron job to APScheduler
- Complete `_trigger_scheduled_scan(schedule_id)` - Execute scheduled scan
- Add `load_schedules_on_startup()` - Load all enabled schedules
- Add cron expression parsing with croniter
**New/Updated Methods:**
```python
from croniter import croniter
from datetime import datetime
def add_scheduled_scan(self, schedule):
"""Add a cron job for scheduled scan execution."""
if not schedule.get('enabled'):
return
job_id = f"schedule_{schedule['id']}"
# Parse cron expression
trigger = CronTrigger.from_crontab(schedule['cron_expression'])
# Add job to scheduler
self.scheduler.add_job(
func=self._trigger_scheduled_scan,
trigger=trigger,
id=job_id,
args=[schedule['id']],
replace_existing=True,
max_instances=1
)
logger.info(f"Added scheduled scan: {schedule['name']} ({job_id})")
def _trigger_scheduled_scan(self, schedule_id):
"""Execute a scheduled scan."""
from web.services.schedule_service import ScheduleService
logger.info(f"Triggering scheduled scan: schedule_id={schedule_id}")
# Get schedule details
schedule_service = ScheduleService(self.db_session)
schedule = schedule_service.get_schedule(schedule_id)
if not schedule:
logger.error(f"Schedule {schedule_id} not found")
return
# Trigger scan with schedule_id
from web.services.scan_service import ScanService
scan_service = ScanService(self.db_session)
scan_id = scan_service.trigger_scan(
config_file=schedule['config_file'],
triggered_by='scheduled',
schedule_id=schedule_id
)
# Queue the scan
self.queue_scan(schedule['config_file'], scan_id, self.db_url)
# Update last_run timestamp
schedule_service.update_run_times(
schedule_id=schedule_id,
last_run=datetime.utcnow(),
next_run=self._calculate_next_run(schedule['cron_expression'])
)
logger.info(f"Scheduled scan queued: scan_id={scan_id}")
def load_schedules_on_startup(self):
"""Load all enabled schedules from database on app startup."""
from web.services.schedule_service import ScheduleService
schedule_service = ScheduleService(self.db_session)
schedules = schedule_service.list_schedules(page=1, per_page=1000, enabled_filter=True)
for schedule in schedules['schedules']:
self.add_scheduled_scan(schedule)
logger.info(f"Loaded {len(schedules['schedules'])} schedules on startup")
def _calculate_next_run(self, cron_expression):
"""Calculate next run time from cron expression."""
cron = croniter(cron_expression, datetime.utcnow())
return cron.get_next(datetime)
```
#### 3. `web/routes/main.py`
Add schedule management routes.
**Current State:** 69 lines, only scan routes
**Changes:**
- Add schedule routes:
- `GET /schedules` - List schedules
- `GET /schedules/create` - Create form
- `GET /schedules/<id>/edit` - Edit form
**New Code:**
```python
@bp.route('/schedules')
@login_required
def schedules():
"""List all schedules."""
return render_template('schedules.html')
@bp.route('/schedules/create')
@login_required
def create_schedule():
"""Create new schedule form."""
# Get list of available config files
import os
configs_dir = '/app/configs'
config_files = [f for f in os.listdir(configs_dir) if f.endswith('.yaml')]
return render_template('schedule_create.html', config_files=config_files)
@bp.route('/schedules/<int:schedule_id>/edit')
@login_required
def edit_schedule(schedule_id):
"""Edit existing schedule form."""
from web.services.schedule_service import ScheduleService
schedule_service = ScheduleService(current_app.db_session)
try:
schedule = schedule_service.get_schedule(schedule_id)
return render_template('schedule_edit.html', schedule=schedule)
except Exception as e:
flash(f'Schedule not found: {e}', 'danger')
return redirect(url_for('main.schedules'))
```
#### 4. `web/templates/base.html`
Extract CSS, add Chart.js, fix white rows.
**Current State:** 346 lines with 280 lines of inline CSS (lines 8-288)
**Changes:**
- Replace inline `<style>` block with `<link>` to external CSS
- Add Chart.js CDN
- Add dark theme configuration for Chart.js
**Modifications:**
```html
<head>
<meta charset="UTF-8">
<meta name="viewport" content="width=device-width, initial-scale=1.0">
<title>{% block title %}SneakyScanner{% endblock %}</title>
<!-- Bootstrap 5 CSS -->
<link href="https://cdn.jsdelivr.net/npm/bootstrap@5.3.0/dist/css/bootstrap.min.css" rel="stylesheet">
<!-- Bootstrap Icons -->
<link rel="stylesheet" href="https://cdn.jsdelivr.net/npm/bootstrap-icons@1.11.0/font/bootstrap-icons.css">
<!-- Custom CSS (extracted from inline) -->
<link rel="stylesheet" href="{{ url_for('static', filename='css/styles.css') }}">
<!-- Chart.js for visualizations -->
<script src="https://cdn.jsdelivr.net/npm/chart.js@4.4.0/dist/chart.umd.min.js"></script>
<!-- Chart.js Dark Theme Configuration -->
<script>
Chart.defaults.color = '#e2e8f0';
Chart.defaults.borderColor = '#334155';
Chart.defaults.backgroundColor = '#1e293b';
</script>
</head>
```
#### 5. `web/templates/dashboard.html`
Fix white rows, add trending charts.
**Current State:** 356 lines, basic stats
**Changes:**
- Fix white rows in renderScansTable() (add `.scan-row` class)
- Add "Schedules" widget showing next scheduled scan
- Add Chart.js trending chart (scans per day, last 30 days)
- Add "Quick Actions" section
**New Section - Schedules Widget:**
```html
<div class="col-md-6">
<div class="card h-100">
<div class="card-header d-flex justify-content-between align-items-center">
<h5 class="mb-0">Upcoming Scheduled Scans</h5>
<a href="{{ url_for('main.schedules') }}" class="btn btn-sm btn-secondary">Manage</a>
</div>
<div class="card-body">
<div id="next-schedules">
<p class="text-muted">Loading...</p>
</div>
</div>
</div>
</div>
```
**New Section - Trending Chart:**
```html
<div class="row mb-4">
<div class="col-12">
<div class="card">
<div class="card-header">
<h5 class="mb-0">Scan Activity (Last 30 Days)</h5>
</div>
<div class="card-body">
<canvas id="scanTrendChart" height="80"></canvas>
</div>
</div>
</div>
</div>
<script>
async function loadScanTrend() {
const response = await fetch('/api/stats/scan-trend?days=30');
const data = await response.json();
const ctx = document.getElementById('scanTrendChart').getContext('2d');
new Chart(ctx, {
type: 'line',
data: {
labels: data.labels, // ['2025-01-01', '2025-01-02', ...]
datasets: [{
label: 'Scans',
data: data.values, // [5, 3, 7, 2, ...]
borderColor: '#60a5fa',
backgroundColor: 'rgba(96, 165, 250, 0.1)',
tension: 0.3
}]
},
options: {
responsive: true,
maintainAspectRatio: false,
plugins: {
legend: {display: false}
},
scales: {
y: {
beginAtZero: true,
ticks: {stepSize: 1}
}
}
}
});
}
loadScanTrend();
</script>
```
**Fix for White Rows (line ~221):**
```javascript
function renderScansTable(scans) {
const tbody = document.getElementById('recent-scans');
tbody.innerHTML = '';
if (scans.length === 0) {
tbody.innerHTML = '<tr class="scan-row"><td colspan="5" class="text-center text-muted">No scans yet</td></tr>';
return;
}
scans.forEach(scan => {
const row = document.createElement('tr');
row.classList.add('scan-row'); // ← FIX: Add explicit class
row.onclick = () => window.location.href = `/scans/${scan.id}`;
row.innerHTML = `
<td class="mono">${scan.id}</td>
<td>${scan.title || 'Untitled Scan'}</td>
<td>${new Date(scan.timestamp).toLocaleString()}</td>
<td>${getStatusBadge(scan.status)}</td>
`;
tbody.appendChild(row);
});
}
```
#### 6. `web/templates/scans.html`
Fix white rows bug.
**Current State:** 469 lines, white row bug at lines 208-241
**Changes:**
- Add `.scan-row` class to dynamically created rows (same fix as dashboard)
**Fix (line ~208):**
```javascript
function renderScansTable(scans) {
const tbody = document.getElementById('scans-tbody');
tbody.innerHTML = '';
if (scans.length === 0) {
tbody.innerHTML = '<tr class="scan-row"><td colspan="6" class="text-center text-muted">No scans found</td></tr>';
return;
}
scans.forEach(scan => {
const row = document.createElement('tr');
row.classList.add('scan-row'); // ← FIX: Add explicit class
row.onclick = () => window.location.href = `/scans/${scan.id}`;
// ... rest of row creation
});
}
```
#### 7. `web/templates/scan_detail.html`
Fix white rows in port tables.
**Current State:** 399 lines, white rows in port tables (lines 305-327)
**Changes:**
- Add `.scan-row` class to port table rows
**Fix (in renderPortsTable function):**
```javascript
function renderPortsTable(ports, containerId) {
const tbody = document.getElementById(containerId);
tbody.innerHTML = '';
ports.forEach(port => {
const row = document.createElement('tr');
row.classList.add('scan-row'); // ← FIX: Add explicit class
row.innerHTML = `
<td>${port.port}</td>
<td>${port.protocol.toUpperCase()}</td>
<td>${getExpectedBadge(port.expected)}</td>
<td>${port.state}</td>
`;
tbody.appendChild(row);
});
}
```
#### 8. `web/app.py`
Load schedules on startup.
**Current State:** Initializes scheduler but doesn't load schedules
**Changes:**
- After scheduler initialization, load all enabled schedules
**New Code (after scheduler init):**
```python
def init_scheduler(app):
"""Initialize APScheduler and load schedules."""
from web.services.scheduler_service import SchedulerService
scheduler_service = SchedulerService()
scheduler_service.init_scheduler(app)
# Store in app context
app.scheduler = scheduler_service
# Load all enabled schedules from database
with app.app_context():
scheduler_service.load_schedules_on_startup()
logger.info("Scheduler initialized and schedules loaded")
```
#### 9. `requirements-web.txt`
Add croniter dependency.
**Current State:** All Phase 2 dependencies
**Changes:**
- Add croniter for cron expression parsing
**New Line:**
```
croniter==2.0.1
```
---
## Step-by-Step Implementation
### Step 1: Fix Styling Issues & CSS Refactor ✅ (Day 1)
**Priority: CRITICAL** - Fixes user-facing bug
**Tasks:**
1. Create `web/static/css/styles.css`
- Extract all CSS from base.html `<style>` block
- Add CSS variables for colors
- Add fix for white row bug with `.scan-row` class
- Add Chart.js dark theme styles
2. Update `web/templates/base.html`
- Remove inline `<style>` block
- Add `<link>` to external styles.css
- Add Chart.js CDN script
- Add Chart.js default configuration for dark theme
3. Fix white rows in all templates:
- `web/templates/dashboard.html` - Add `.scan-row` class
- `web/templates/scans.html` - Add `.scan-row` class
- `web/templates/scan_detail.html` - Add `.scan-row` class to port tables
4. Test CSS changes:
- Verify all pages render correctly
- Verify dark theme maintained
- Verify table rows are dark slate, not white
- Verify hover effects work
**Deliverables:**
- `web/static/css/styles.css` (~350 lines)
- Updated base.html (reduced by ~280 lines)
- Fixed white rows in 3 templates
- Chart.js ready for use in next steps
**Testing:**
- Manual: View dashboard, scans list, scan detail - verify no white rows
- Manual: Test hover effects on tables
- Manual: Verify no CSS regressions
---
### Step 2: ScheduleService Implementation ✅ (Days 2-3)
**Priority: HIGH** - Core business logic for schedules
**Tasks:**
1. Create `web/services/schedule_service.py` (~400 lines)
- Implement all CRUD methods
- Add cron expression validation with croniter
- Add next run time calculation
- Add relative time formatting
- Add schedule-to-dict serialization
- Add execution history retrieval
2. Add croniter to requirements
- Update `requirements-web.txt`
3. Write comprehensive unit tests
- Create `tests/test_schedule_service.py` (~450 lines)
- 18+ test functions covering all methods
- Test cron validation edge cases
- Test next run calculation accuracy
- Test error handling
**Key Methods to Implement:**
- `create_schedule()` - Validate and create
- `get_schedule()` - Retrieve with history
- `list_schedules()` - Paginated list
- `update_schedule()` - Update with validation
- `delete_schedule()` - Remove schedule
- `toggle_enabled()` - Enable/disable
- `validate_cron_expression()` - Validation
- `calculate_next_run()` - Next run time
**Deliverables:**
- ScheduleService class fully implemented
- croniter dependency added
- 18+ unit tests written and passing
**Testing:**
- Unit: All ScheduleService methods
- Edge cases: Invalid cron, missing config files
- Concurrency: Multiple simultaneous schedule operations
---
### Step 3: Schedules API Endpoints ✅ (Days 4-5)
**Priority: HIGH** - API for schedule management
**Tasks:**
1. Update `web/api/schedules.py` (~300 lines total)
- Replace all 6 stub endpoints with implementations
- Add comprehensive error handling
- Add input validation
- Add logging
- Integrate with ScheduleService and SchedulerService
2. Endpoints to implement:
- `GET /api/schedules` - List with pagination and filtering
- `GET /api/schedules/<id>` - Get schedule details + history
- `POST /api/schedules` - Create and add to scheduler
- `PUT /api/schedules/<id>` - Update and reload in scheduler
- `DELETE /api/schedules/<id>` - Delete and remove from scheduler
- `POST /api/schedules/<id>/trigger` - Manually trigger scan
3. Write integration tests
- Create `tests/test_schedule_api.py` (~500 lines)
- 22+ integration tests
- Test complete workflows
- Test authentication
- Test error scenarios
**Implementation Example:**
```python
@bp.route('', methods=['POST'])
@api_auth_required
def create_schedule():
data = request.get_json() or {}
# Validate
if not all(k in data for k in ['name', 'config_file', 'cron_expression']):
return jsonify({'error': 'Missing required fields'}), 400
schedule_service = ScheduleService(current_app.db_session)
try:
schedule_id = schedule_service.create_schedule(
name=data['name'],
config_file=data['config_file'],
cron_expression=data['cron_expression'],
enabled=data.get('enabled', True)
)
# Add to APScheduler
schedule = schedule_service.get_schedule(schedule_id)
current_app.scheduler.add_scheduled_scan(schedule)
return jsonify({
'schedule_id': schedule_id,
'message': 'Schedule created successfully'
}), 201
except ValueError as e:
return jsonify({'error': str(e)}), 400
```
**Deliverables:**
- All 6 API endpoints implemented
- 22+ integration tests written and passing
- Comprehensive error handling
- API fully functional
**Testing:**
- Integration: All endpoints with real database
- Workflow: Create → update → trigger → delete
- Error: Invalid inputs, missing schedules, auth failures
---
### Step 4: Schedule Management UI ✅ (Days 6-7)
**Priority: HIGH** - User interface for schedules
**Tasks:**
1. Create `web/templates/schedules.html` (~400 lines)
- Stats cards (total, enabled, next run, executions)
- Schedules table with enable/disable toggles
- Next run time display (human-readable)
- Action buttons (Run Now, Edit, Delete)
- AJAX data loading
- Auto-refresh every 30 seconds
2. Create `web/templates/schedule_create.html` (~300 lines)
- Schedule name input
- Config file dropdown (list available configs)
- Cron expression input with validation
- Quick cron templates (daily, weekly, monthly)
- Human-readable cron description
- Next 5 run times preview
- Enable/disable toggle
- Submit button
3. Create `web/templates/schedule_edit.html` (~250 lines)
- Pre-populated form fields
- Execution history (last 10 scans)
- Delete schedule button
- Test run button
4. Update `web/routes/main.py`
- Add `/schedules` route
- Add `/schedules/create` route
- Add `/schedules/<id>/edit` route
5. Add cron helper JavaScript
- Cron expression validator (client-side)
- Human-readable formatter
- Next runs calculator
**Cron Expression Builder:**
```html
<!-- Quick Templates -->
<button onclick="setCron('0 0 * * *')">Daily at Midnight</button>
<button onclick="setCron('0 2 * * *')">Daily at 2 AM</button>
<button onclick="setCron('0 0 * * 0')">Weekly (Sunday)</button>
<button onclick="setCron('0 0 1 * *')">Monthly</button>
<!-- Manual Entry with Validation -->
<input type="text" id="cron-expression" onchange="validateCron()">
<div id="cron-description"><!-- Human-readable description --></div>
<div id="next-runs"><!-- Next 5 run times --></div>
```
**Deliverables:**
- 3 new templates created
- 3 new web routes added
- Fully functional schedule management UI
- Client-side cron validation
**Testing:**
- Manual: Create schedule via UI
- Manual: Edit existing schedule
- Manual: Enable/disable toggle
- Manual: Manual trigger works
- Manual: Delete schedule
- Verify: Cron validation prevents invalid expressions
---
### Step 5: Enhanced Dashboard with Charts ✅ (Days 8-9)
**Priority: MEDIUM** - Improved dashboard visualization
**Tasks:**
1. Add API endpoint for trend data
- Create `GET /api/stats/scan-trend?days=30`
- Return scan counts per day
- Return labels and values for Chart.js
2. Update `web/templates/dashboard.html`
- Add "Schedules" widget showing next scheduled scans
- Add Chart.js line chart for scan activity (last 30 days)
- Add "Quick Actions" section (Run Scan, Create Schedule)
- Improve layout with Bootstrap grid
3. Add additional chart (optional)
- Port count trend over time
- Service distribution pie chart
- Certificate expiry timeline
4. Write tests for chart data
- Create `tests/test_charts.py` (~200 lines)
- Test data aggregation
- Test date range filtering
- Test empty data handling
**Chart Implementation:**
```javascript
async function loadScanTrend() {
const response = await fetch('/api/stats/scan-trend?days=30');
const data = await response.json();
const ctx = document.getElementById('scanTrendChart').getContext('2d');
new Chart(ctx, {
type: 'line',
data: {
labels: data.labels,
datasets: [{
label: 'Scans',
data: data.values,
borderColor: '#60a5fa',
backgroundColor: 'rgba(96, 165, 250, 0.1)',
tension: 0.3
}]
},
options: {
responsive: true,
maintainAspectRatio: false,
scales: {
y: {beginAtZero: true}
}
}
});
}
```
**Deliverables:**
- Enhanced dashboard with charts
- Schedule widget showing next runs
- Stats API endpoint for trends
- Chart tests written
**Testing:**
- Manual: Verify charts render correctly
- Manual: Verify dark theme applied to charts
- Unit: Chart data generation
- Edge: Empty data, single data point
---
### Step 6: Scheduler Integration ✅ (Day 10)
**Priority: CRITICAL** - Complete scheduled execution
**Tasks:**
1. Update `web/services/scheduler_service.py`
- Complete `add_scheduled_scan()` implementation
- Complete `_trigger_scheduled_scan()` implementation
- Add `load_schedules_on_startup()` method
- Add cron job to APScheduler with CronTrigger
2. Update `web/app.py`
- Call `load_schedules_on_startup()` after scheduler init
- Ensure app context available for database queries
3. Test scheduled execution
- Create test schedule with cron "* * * * *" (every minute)
- Verify scan is triggered automatically
- Verify last_run and next_run are updated
- Verify scans have `triggered_by='scheduled'`
4. Test enable/disable
- Disable schedule
- Verify job removed from APScheduler
- Enable schedule
- Verify job added back to APScheduler
**Key Implementation:**
```python
from apscheduler.triggers.cron import CronTrigger
def add_scheduled_scan(self, schedule):
"""Add cron job for scheduled scan."""
job_id = f"schedule_{schedule['id']}"
trigger = CronTrigger.from_crontab(schedule['cron_expression'])
self.scheduler.add_job(
func=self._trigger_scheduled_scan,
trigger=trigger,
id=job_id,
args=[schedule['id']],
replace_existing=True
)
def _trigger_scheduled_scan(self, schedule_id):
"""Execute scheduled scan."""
# Get schedule
schedule = ScheduleService(self.db_session).get_schedule(schedule_id)
# Trigger scan
scan_id = ScanService(self.db_session).trigger_scan(
config_file=schedule['config_file'],
triggered_by='scheduled',
schedule_id=schedule_id
)
# Queue scan
self.queue_scan(schedule['config_file'], scan_id, self.db_url)
# Update timestamps
ScheduleService(self.db_session).update_run_times(
schedule_id, datetime.utcnow(), self._calculate_next_run(...)
)
```
**Deliverables:**
- Scheduled scans execute automatically
- Schedules loaded on app startup
- Enable/disable functionality works
- Timestamps updated after execution
**Testing:**
- Integration: Create schedule with "* * * * *", wait 1 minute, verify scan created
- Manual: Enable/disable schedule, verify APScheduler jobs
- Edge: Schedule during scan, multiple schedules at same time
---
### Step 7: Scan Comparison Features ✅ (Days 11-12)
**Priority: MEDIUM** - Historical analysis
**Tasks:**
1. Add comparison API endpoint
- `GET /api/scans/<id1>/compare/<id2>`
- Return diff of ports, services, certificates
- Highlight added, removed, changed items
2. Add comparison UI
- "Compare with Previous" button on scan detail page
- Side-by-side comparison view
- Color coding: green (added), red (removed), yellow (changed)
3. Add historical charts to scan detail
- Port count trend for this site
- Service version changes over time
- Certificate expiry timeline
4. Add drift detection
- Calculate "drift score" (how much changed)
- Highlight unexpected changes
- Link to alert creation
**Comparison Algorithm:**
```python
def compare_scans(scan1_id, scan2_id):
scan1 = ScanService.get_scan(scan1_id)
scan2 = ScanService.get_scan(scan2_id)
# Extract port lists
ports1 = set(p['port'] for site in scan1['sites']
for ip in site['ips'] for p in ip['ports'])
ports2 = set(p['port'] for site in scan2['sites']
for ip in site['ips'] for p in ip['ports'])
return {
'added_ports': list(ports2 - ports1),
'removed_ports': list(ports1 - ports2),
'unchanged_ports': list(ports1 & ports2),
'services_changed': _compare_services(scan1, scan2),
'certificates_changed': _compare_certs(scan1, scan2)
}
```
**Deliverables:**
- Scan comparison API endpoint
- Comparison UI with diff view
- Historical charts on scan detail
- Drift score calculation
**Testing:**
- Unit: Comparison algorithm with various scenarios
- Manual: Compare two scans with differences
- Edge: Compare same scan, compare non-existent scans
---
### Step 8: Testing & Documentation ✅ (Days 13-14)
**Priority: HIGH** - Quality assurance
**Tasks:**
1. Complete test coverage
- Verify all 80+ tests passing
- Add missing edge cases
- Test concurrent schedule operations
- Test cron execution accuracy
2. Create manual testing checklist
- Create `docs/ai/MANUAL_TESTING_PHASE3.md`
- 30+ manual tests for all Phase 3 features
- Include schedule creation, editing, execution
- Include chart rendering verification
3. Update documentation
- Update `README.md` with Phase 3 features
- Create `docs/ai/PHASE3_COMPLETE.md`
- Update `docs/ai/ROADMAP.md` with Phase 3 completion
4. Run full regression tests
- Verify all Phase 2 tests still pass
- Verify no breaking changes
- Test backward compatibility
5. Performance testing
- Test with 50+ schedules
- Test with 1000+ scans (chart performance)
- Verify APScheduler handles many cron jobs
**Documentation Updates:**
**README.md additions:**
- Scheduled scans section
- Chart.js visualizations
- Schedule management UI
**PHASE3_COMPLETE.md sections:**
- What was delivered
- Success criteria checklist
- Known limitations
- Testing results
- Performance metrics
- Lessons learned
**Deliverables:**
- All 80+ tests passing
- Manual testing checklist created
- Documentation updated
- Phase 3 completion summary
- Performance validated
**Testing:**
- Full test suite: `pytest tests/ -v`
- Manual: Complete manual testing checklist
- Performance: Load test with many schedules
- Regression: Verify Phase 2 features unchanged
---
## Dependencies & Prerequisites
### Python Packages
**Add to `requirements-web.txt`:**
```
croniter==2.0.1 # Cron expression parsing
```
**Already Present (from Phase 2):**
- Flask==3.0.0
- SQLAlchemy==2.0.23
- APScheduler==3.10.4
- Flask-Login==0.6.3
- Bootstrap 5 (CDN)
### Frontend Libraries
**Add to base.html:**
- Chart.js 4.4.0 (CDN): `https://cdn.jsdelivr.net/npm/chart.js@4.4.0/dist/chart.umd.min.js`
**Already Present:**
- Bootstrap 5.3.0 (CDN)
- Bootstrap Icons 1.11.0 (CDN)
### System Requirements
- Python 3.12+ (same as Phase 2)
- Docker and Docker Compose
- SQLite3 with WAL mode (already configured)
### Configuration Files
**No new configuration needed** - Uses existing:
- `.env` file (from Phase 2)
- `docker-compose-web.yml` (no changes)
- Database schema (Schedule table already exists)
---
## Testing Approach
### Unit Tests
**New Test Files:**
1. `tests/test_schedule_service.py` (450 lines, 18+ tests)
2. `tests/test_charts.py` (200 lines, 8+ tests)
**Coverage Target:** >80% for new code
**Test Strategy:**
- Mock database calls for service tests
- Test cron validation with valid/invalid expressions
- Test next run calculations with various timezones
- Test edge cases (empty data, null values)
### Integration Tests
**New Test Files:**
1. `tests/test_schedule_api.py` (500 lines, 22+ tests)
**Test Strategy:**
- Use test database (separate from production)
- Test complete workflows (create → update → execute → delete)
- Test API authentication
- Verify APScheduler integration
### Manual Testing
**Create:** `docs/ai/MANUAL_TESTING_PHASE3.md`
**Tests to Include:**
1. CSS/Styling (5 tests)
- Verify white rows fixed
- Verify dark theme consistent
- Verify Chart.js dark theme
2. Schedules (15 tests)
- Create schedule via UI
- Edit schedule
- Enable/disable toggle
- Manual trigger
- Delete schedule
- Cron validation
- Next run display
3. Dashboard (5 tests)
- Verify charts render
- Verify schedule widget
- Verify stats accurate
4. Scheduled Execution (5 tests)
- Create schedule with "* * * * *"
- Wait 1 minute, verify scan created
- Verify timestamps updated
- Disable schedule, verify stops
- Enable schedule, verify resumes
### Performance Tests
**Scenarios:**
1. Load 100 schedules, verify startup time < 5 seconds
2. Generate chart with 1000 scans, verify response < 2 seconds
3. List 500 schedules with pagination, verify response < 500ms
4. Concurrent schedule creation (10 simultaneous), verify no conflicts
---
## Potential Challenges & Solutions
### Challenge 1: Cron Expression Validation
**Problem:** Users may enter invalid cron expressions causing APScheduler errors.
**Impact:** High - Could crash scheduler
**Solution:**
- Validate cron expressions with croniter before saving
- Show human-readable description to user
- Preview next 5 run times
- Provide quick templates (daily, weekly, monthly)
- Catch APScheduler exceptions and log errors
**Implementation:**
```python
from croniter import croniter
def validate_cron_expression(cron_expr):
try:
# Test if cron is valid
cron = croniter(cron_expr, datetime.utcnow())
cron.get_next() # Try to get next run
return True, None
except Exception as e:
return False, f"Invalid cron expression: {str(e)}"
```
### Challenge 2: Timezone Handling
**Problem:** Cron expressions are timezone-dependent. User expects "2 AM" in their local timezone, but server runs in UTC.
**Impact:** Medium - Schedules run at unexpected times
**Solution:**
- Store all timestamps in UTC (already done in Phase 2)
- Display times in user's browser timezone (JavaScript)
- Document that cron expressions are in UTC
- Phase 4 enhancement: Add timezone selector per schedule
**Implementation:**
```javascript
// Display in user's local timezone
const utcTime = new Date(schedule.next_run + 'Z'); // Parse as UTC
document.getElementById('next-run').textContent = utcTime.toLocaleString();
```
### Challenge 3: Concurrent Schedule Execution
**Problem:** If two schedules have same cron expression, both may trigger at exact same time, potentially overwhelming system.
**Impact:** Medium - Resource contention
**Solution:**
- APScheduler already handles concurrency (ThreadPoolExecutor)
- Max 3 concurrent scans (configured in Phase 2)
- If more than 3 scans queued, they wait in queue
- Add warning in UI if > 5 schedules have same cron
**Implementation:**
```python
# In SchedulerService
self.scheduler = BackgroundScheduler(
executors={
'default': ThreadPoolExecutor(3) # Max 3 concurrent
},
job_defaults={
'coalesce': False, # Don't combine missed runs
'max_instances': 1 # Only 1 instance per job
}
)
```
### Challenge 4: Missed Schedule Execution
**Problem:** If app is down during scheduled time, scan is missed. When should it run after restart?
**Impact:** Medium - Data gaps
**Solution:**
- APScheduler has `misfire_grace_time` setting
- Set to 3600 seconds (1 hour)
- If app down < 1 hour, scan will run when app starts
- If app down > 1 hour, skip and wait for next scheduled time
- Log missed schedules for admin review
**Implementation:**
```python
self.scheduler = BackgroundScheduler(
job_defaults={
'misfire_grace_time': 3600 # Run if missed within 1 hour
}
)
```
### Challenge 5: Chart.js Dark Theme
**Problem:** Default Chart.js colors don't match dark theme (white background, black text).
**Impact:** Low - Aesthetic only
**Solution:**
- Set Chart.js global defaults in base.html
- Override colors, fonts, borders
- Test all chart types (line, bar, pie)
**Implementation:**
```javascript
Chart.defaults.color = '#e2e8f0'; // Light text
Chart.defaults.borderColor = '#334155'; // Slate borders
Chart.defaults.backgroundColor = '#1e293b'; // Dark background
Chart.defaults.font.family = "'Inter', sans-serif";
```
### Challenge 6: Schedule Deletion with Active Scans
**Problem:** User deletes schedule while scan from that schedule is running. Should scan be cancelled?
**Impact:** Low - Edge case
**Solution:**
- Do NOT cancel running scans
- Only prevent future executions
- Running scan completes normally
- Database foreign key allows null schedule_id (schedule deleted but scan remains)
- Document behavior in UI ("Active scans will complete")
**Implementation:**
```python
def delete_schedule(self, schedule_id):
# Remove from APScheduler (prevents future runs)
self.scheduler.remove_scheduled_scan(schedule_id)
# Delete from database (scans remain, schedule_id becomes null)
schedule = self.db.query(Schedule).get(schedule_id)
self.db.delete(schedule)
self.db.commit()
logger.info(f"Schedule {schedule_id} deleted. Active scans will complete.")
```
### Challenge 7: CSS Extraction Breaking Existing Styles
**Problem:** Moving inline CSS to external file might cause selector specificity issues.
**Impact:** Medium - Visual regressions
**Solution:**
- Extract CSS exactly as-is first
- Test all pages after extraction
- Only then refactor/improve
- Use CSS variables for consistency
- Keep !important flags where needed for dynamic rows
**Testing:**
- Manual: View every page (dashboard, scans, scan detail, login, schedules)
- Compare: Screenshots before and after extraction
- Verify: Hover effects, active states, responsive behavior
---
## Success Criteria
Phase 3 is **COMPLETE** when all criteria are met:
### Bug Fixes
- [ ] White row bug fixed in dashboard.html
- [ ] White row bug fixed in scans.html
- [ ] White row bug fixed in scan_detail.html
- [ ] All tables use dark theme consistently
- [ ] Hover effects work correctly
### CSS & Styling
- [ ] CSS extracted to web/static/css/styles.css
- [ ] base.html links to external CSS
- [ ] Chart.js added and configured for dark theme
- [ ] CSS variables used for colors
- [ ] No visual regressions on any page
### Schedule Service
- [ ] ScheduleService fully implemented (~400 lines)
- [ ] All CRUD methods working
- [ ] Cron validation with croniter
- [ ] Next run time calculation accurate
- [ ] 18+ unit tests passing
### Schedules API
- [ ] All 6 API endpoints implemented
- [ ] GET /api/schedules lists with pagination
- [ ] POST /api/schedules creates and validates
- [ ] PUT /api/schedules/<id> updates and reloads
- [ ] DELETE /api/schedules/<id> removes schedule
- [ ] POST /api/schedules/<id>/trigger manually triggers
- [ ] 22+ integration tests passing
### Schedule UI
- [ ] schedules.html lists all schedules
- [ ] schedule_create.html creates new schedules
- [ ] schedule_edit.html edits existing schedules
- [ ] Enable/disable toggle works
- [ ] Manual trigger button works
- [ ] Cron validation prevents invalid expressions
- [ ] Human-readable cron description shown
- [ ] Next 5 run times previewed
### Scheduled Execution
- [ ] Schedules load on app startup
- [ ] Cron jobs added to APScheduler
- [ ] Scans execute automatically at scheduled time
- [ ] last_run and next_run timestamps updated
- [ ] Scans have triggered_by='scheduled'
- [ ] Enable/disable adds/removes jobs
### Dashboard Enhancements
- [ ] Schedule widget shows next scheduled scans
- [ ] Trending chart displays scan activity (last 30 days)
- [ ] Chart.js renders with dark theme
- [ ] Stats accurate (total, enabled, next run)
### Scan Comparison
- [ ] Comparison API endpoint returns diffs
- [ ] "Compare with Previous" button on scan detail
- [ ] Side-by-side comparison view
- [ ] Color coding for added/removed/changed
### Testing
- [ ] All 80+ new tests passing
- [ ] All Phase 2 tests still passing
- [ ] Manual testing checklist completed
- [ ] Performance acceptable (50+ schedules, 1000+ scans)
### Documentation
- [ ] README.md updated with Phase 3 features
- [ ] PHASE3_COMPLETE.md created
- [ ] ROADMAP.md updated with Phase 3 completion
- [ ] Manual testing checklist created
---
## Migration Path
### From Phase 2 to Phase 3
**No Breaking Changes:**
- All Phase 2 APIs remain functional
- Database schema unchanged (Schedule table already exists)
- Existing scans, users, settings unchanged
- Docker deployment compatible
**Additions:**
- New schedules API endpoints (stubs → implementations)
- New schedule management UI
- New dashboard charts
- External CSS file (improves maintainability)
- croniter dependency
**Migration Steps:**
1. Pull latest code from phase3 branch
2. Install new dependency: `pip install croniter==2.0.1`
3. No database migration needed (Schedule table already exists from Phase 1)
4. Rebuild Docker image: `docker-compose -f docker-compose-web.yml build`
5. Restart services: `docker-compose -f docker-compose-web.yml restart`
6. Verify schedules load on startup (check logs)
**Backward Compatibility:**
- CLI scanner: Continues to work standalone (unchanged)
- Existing scans: All viewable in web UI
- Phase 2 features: All functional (no regressions)
- API clients: All Phase 2 endpoints unchanged
---
## Estimated Timeline
**Total Duration:** 14 working days (2 weeks)
### Week 1: Backend Foundation
- **Day 1:** Fix styling issues & CSS refactor
- Extract CSS to external file
- Fix white row bug
- Add Chart.js
- **Days 2-3:** ScheduleService implementation
- Create ScheduleService class
- Implement CRUD methods
- Add cron validation
- Write unit tests
- **Days 4-5:** Schedules API endpoints
- Implement all 6 endpoints
- Add error handling
- Write integration tests
- **Days 6-7:** Schedule Management UI
- Create schedules list page
- Create schedule create form
- Create schedule edit form
- Add web routes
### Week 2: Enhancements & Polish
- **Days 8-9:** Enhanced dashboard with charts
- Add trending charts
- Add schedule widget
- Create stats API endpoints
- **Day 10:** Scheduler integration
- Complete scheduled execution
- Load schedules on startup
- Test cron execution
- **Days 11-12:** Scan comparison features
- Implement comparison algorithm
- Create comparison UI
- Add historical charts
- **Days 13-14:** Testing & documentation
- Complete test suite
- Manual testing
- Update documentation
- Phase 3 completion summary
### Critical Path
**Must Complete in Order:**
1. Day 1: CSS extraction and white row fix (prerequisite for all UI work)
2. Days 2-3: ScheduleService (prerequisite for API)
3. Days 4-5: Schedules API (prerequisite for UI)
4. Days 6-7: Schedule UI (can proceed once API done)
**Can Proceed in Parallel:**
- Dashboard charts (Days 8-9) can start any time after Day 1
- Scan comparison (Days 11-12) independent of schedules
- Testing (Days 13-14) can start earlier for completed components
---
## Key Design Decisions
### Decision 1: Cron Expression Library
**Choice:** croniter
**Alternatives Considered:**
- APScheduler's built-in cron parsing
- python-crontab
- Manual parsing with regex
**Rationale:**
- croniter is lightweight and focused
- Accurate next run calculation
- Handles edge cases (leap years, DST)
- Easy validation and error messages
- Well-maintained (active development)
**Trade-offs:**
- ✅ Accurate and reliable
- ✅ Simple API
- ✅ Good error messages
- ❌ Another dependency (acceptable - small library)
### Decision 2: External CSS vs Inline
**Choice:** Extract to external CSS file
**Alternatives Considered:**
- Keep inline CSS in base.html
- Use CSS-in-JS
- Use CSS framework (Tailwind)
**Rationale:**
- Better maintainability (single source of truth)
- Easier to customize theme
- Browser caching (performance)
- Separation of concerns
- Easier to test (CSS linting)
**Trade-offs:**
- ✅ Maintainable and organized
- ✅ Better performance (caching)
- ✅ Easier theme customization
- ❌ One more HTTP request (minimal impact with HTTP/2)
### Decision 3: Chart.js for Visualizations
**Choice:** Chart.js 4.4.0
**Alternatives Considered:**
- Plotly.js (more powerful, heavier)
- D3.js (maximum flexibility, steeper learning curve)
- ApexCharts (modern, but less documented)
- Server-side rendering (matplotlib → image)
**Rationale:**
- Lightweight (< 200KB)
- Simple API
- Good dark theme support
- Responsive by default
- Well-documented
- Active community
**Trade-offs:**
- ✅ Easy to implement
- ✅ Good performance
- ✅ Responsive
- ❌ Less powerful than Plotly (sufficient for Phase 3)
- ❌ Not as customizable as D3 (acceptable trade-off)
### Decision 4: Schedule Deletion Behavior
**Choice:** Delete schedule but keep associated scans
**Alternatives Considered:**
- Cascade delete (delete scans too)
- Soft delete (mark schedule as deleted)
- Prevent deletion if scans exist
**Rationale:**
- Scans are historical data (valuable)
- Schedule is just a template (can be recreated)
- User might want to delete schedule but keep history
- Deleting scans would cause data loss
**Trade-offs:**
- ✅ Preserves historical data
- ✅ Flexible (can recreate schedule)
- ❌ Schedule reference in scans becomes null (acceptable)
### Decision 5: Timezone Handling
**Choice:** Store all times in UTC, display in user's browser timezone
**Alternatives Considered:**
- Store in user's timezone
- Add timezone field to Schedule model
- Use server's local timezone
**Rationale:**
- UTC is standard for databases
- Avoids DST issues
- Browser automatically converts to local time
- Consistent with Phase 2 approach
- Simple and reliable
**Trade-offs:**
- ✅ No DST issues
- ✅ Consistent across users
- ✅ Simple implementation
- ❌ Cron expressions in UTC (document clearly)
- ❌ Phase 4 enhancement: per-schedule timezone
### Decision 6: Scan Comparison Scope
**Choice:** Start with port and service comparison, defer certificate comparison to Phase 4
**Alternatives Considered:**
- Full comparison (ports, services, certs, TLS)
- Only port comparison
- Advanced diff with visualization
**Rationale:**
- Ports and services are most important for drift detection
- Certificates are complex (multiple per scan, expiry dates)
- Keep Phase 3 scope manageable
- Certificate comparison better suited for Phase 4 (alerts)
**Trade-offs:**
- ✅ Achievable in 2 days
- ✅ Most valuable comparison (ports/services)
- ❌ Not comprehensive (acceptable for Phase 3)
- ✅ Can enhance in Phase 4
---
## Documentation Deliverables
### 1. Updated README.md
**New Sections:**
- Scheduled Scans
- How to create schedules
- Cron expression syntax
- Enable/disable schedules
- Dashboard Enhancements
- Trending charts
- Schedule widget
- Scan Comparison
- Compare feature
- Historical analysis
### 2. PHASE3_COMPLETE.md
**Sections:**
- What was delivered (8 steps)
- Success criteria checklist
- Files created/modified summary
- Testing results
- Performance metrics
- Known limitations
- Lessons learned
- What's next (Phase 4)
### 3. MANUAL_TESTING_PHASE3.md
**Sections:**
- Prerequisites
- CSS/Styling tests (5 tests)
- Schedule management tests (15 tests)
- Dashboard tests (5 tests)
- Scheduled execution tests (5 tests)
- Scan comparison tests (3 tests)
- Performance tests (3 tests)
- Test results summary
### 4. Updated ROADMAP.md
**Changes:**
- Mark Phase 3 as COMPLETE ✅
- Update progress overview
- Add Phase 3 deliverables section
- Update success criteria
- Update changelog
- Set next review for Phase 4
---
**End of Phase 3 Plan**
This plan will be followed during Phase 3 implementation. Upon completion, `PHASE3_COMPLETE.md` will summarize actual implementation, challenges encountered, and lessons learned.