Implemented complete scheduler integration with automatic schedule loading, orphaned scan cleanup, and conversion to local timezone for better UX. Backend Changes: - Added load_schedules_on_startup() to load enabled schedules on app start - Implemented cleanup_orphaned_scans() to handle crashed/interrupted scans - Converted scheduler from UTC to local system timezone throughout - Enhanced scheduler service with robust error handling and logging Frontend Changes: - Updated all schedule UI templates to display local time instead of UTC - Improved timezone indicators and user messaging - Removed confusing timezone converter (no longer needed) - Updated quick templates and help text for local time Bug Fixes: - Fixed critical timezone bug causing cron expressions to run at wrong times - Fixed orphaned scans stuck in 'running' status after system crashes - Improved time display clarity across all schedule pages All schedules now use local system time for intuitive scheduling.
2205 lines
64 KiB
Markdown
2205 lines
64 KiB
Markdown
# Phase 3 Implementation Plan: Dashboard Enhancement & Scheduled Scans
|
|
|
|
**Status:** In Progress
|
|
**Progress:** 9/14 days complete (64%)
|
|
**Estimated Duration:** 14 days (2 weeks)
|
|
**Dependencies:** Phase 2 Complete ✅
|
|
|
|
## Progress Summary
|
|
|
|
- ✅ **Step 1: Fix Styling Issues & CSS Refactor** (Day 1) - COMPLETE
|
|
- ✅ **Step 2: ScheduleService Implementation** (Days 2-3) - COMPLETE
|
|
- ✅ **Step 3: Schedules API Endpoints** (Days 4-5) - COMPLETE
|
|
- ✅ **Step 4: Schedule Management UI** (Days 6-7) - COMPLETE
|
|
- ✅ **Step 5: Enhanced Dashboard with Charts** (Days 8-9) - COMPLETE
|
|
- 📋 **Step 6: Scheduler Integration** (Day 10) - NEXT
|
|
- 📋 **Step 7: Scan Comparison Features** (Days 11-12)
|
|
- 📋 **Step 8: Testing & Documentation** (Days 13-14)
|
|
|
|
---
|
|
|
|
## Table of Contents
|
|
|
|
1. [Overview](#overview)
|
|
2. [Current State Analysis](#current-state-analysis)
|
|
3. [Critical Bug Fix](#critical-bug-fix)
|
|
4. [Files to Create](#files-to-create)
|
|
5. [Files to Modify](#files-to-modify)
|
|
6. [Step-by-Step Implementation](#step-by-step-implementation)
|
|
7. [Dependencies & Prerequisites](#dependencies--prerequisites)
|
|
8. [Testing Approach](#testing-approach)
|
|
9. [Potential Challenges & Solutions](#potential-challenges--solutions)
|
|
10. [Success Criteria](#success-criteria)
|
|
11. [Migration Path](#migration-path)
|
|
12. [Estimated Timeline](#estimated-timeline)
|
|
13. [Key Design Decisions](#key-design-decisions)
|
|
14. [Documentation Deliverables](#documentation-deliverables)
|
|
|
|
---
|
|
|
|
## Overview
|
|
|
|
Phase 3 focuses on enhancing the web application with scheduling capabilities and improved dashboard visualizations:
|
|
|
|
1. **Critical Bug Fix** - Fix white row coloring in scan tables
|
|
2. **Scheduled Scans** - Complete schedule management system with cron expressions
|
|
3. **Enhanced Dashboard** - Trending charts, schedule widgets, alert summaries
|
|
4. **Scan Comparison** - Historical analysis and infrastructure drift detection
|
|
|
|
### Goals
|
|
|
|
- 🐛 Fix white row bug affecting scan tables (critical UX issue)
|
|
- ✅ Complete schedule management (CRUD operations)
|
|
- ✅ Scheduled scan execution with cron expressions
|
|
- ✅ Enhanced dashboard with Chart.js visualizations
|
|
- ✅ Scan comparison and historical analysis
|
|
- ✅ CSS extraction and better maintainability
|
|
|
|
---
|
|
|
|
## Current State Analysis
|
|
|
|
### What's Already Done (Phase 2)
|
|
|
|
**✅ Phase 2 Complete** - Full web application core:
|
|
|
|
- **Database Schema** - All 11 models including Schedule table
|
|
- **ScanService** - Complete CRUD operations with 545 lines
|
|
- **Scan API** - 5 endpoints fully implemented
|
|
- **SchedulerService** - Partial implementation (258 lines)
|
|
- ✅ `queue_scan()` - Immediate scan execution works
|
|
- ⚠️ `add_scheduled_scan()` - Placeholder (marked for Phase 3)
|
|
- ⚠️ `_trigger_scheduled_scan()` - Skeleton with TODO comments
|
|
- ✅ `remove_scheduled_scan()` - Basic implementation
|
|
- **Authentication** - Flask-Login with @login_required decorators
|
|
- **UI Templates** - Dashboard, scans list/detail, login pages
|
|
- **Background Jobs** - APScheduler with concurrent execution (max 3)
|
|
- **Error Handling** - Content negotiation, custom error pages
|
|
- **Testing** - 100 test functions, 1,825 lines of test code
|
|
|
|
### Schedule-Related Components Analysis
|
|
|
|
#### Schedule Model (`web/models.py` - lines 142-158)
|
|
```python
|
|
class Schedule(Base):
|
|
__tablename__ = 'schedules'
|
|
|
|
id = Column(Integer, primary_key=True)
|
|
name = Column(String(255), nullable=False)
|
|
config_file = Column(Text, nullable=False)
|
|
cron_expression = Column(String(100), nullable=False)
|
|
enabled = Column(Boolean, default=True)
|
|
last_run = Column(DateTime)
|
|
next_run = Column(DateTime)
|
|
created_at = Column(DateTime, default=datetime.utcnow)
|
|
updated_at = Column(DateTime, default=datetime.utcnow, onupdate=datetime.utcnow)
|
|
```
|
|
|
|
**Status:** ✅ Complete - No migration needed
|
|
|
|
#### Schedules API (`web/api/schedules.py` - 159 lines)
|
|
```python
|
|
@bp.route('', methods=['GET'])
|
|
@api_auth_required
|
|
def list_schedules():
|
|
"""List all schedules. TODO: Implement in Phase 3."""
|
|
return jsonify({'message': 'Schedules list - to be implemented in Phase 3'}), 200
|
|
```
|
|
|
|
**Status:** ⚠️ All 6 endpoints are stubs returning placeholders
|
|
|
|
**Endpoints to Implement:**
|
|
- `GET /api/schedules` - List all schedules
|
|
- `GET /api/schedules/<id>` - Get schedule details
|
|
- `POST /api/schedules` - Create new schedule
|
|
- `PUT /api/schedules/<id>` - Update schedule
|
|
- `DELETE /api/schedules/<id>` - Delete schedule
|
|
- `POST /api/schedules/<id>/trigger` - Manually trigger scheduled scan
|
|
|
|
#### SchedulerService Gaps
|
|
|
|
**Missing Implementations:**
|
|
1. `add_scheduled_scan(schedule)` - Currently placeholder
|
|
2. `_trigger_scheduled_scan(schedule_id)` - TODO comments only
|
|
3. Loading schedules on app startup - Not implemented
|
|
4. Cron expression parsing - No validation
|
|
5. Next run time calculation - Missing
|
|
|
|
### UI Components Status
|
|
|
|
**Existing Templates:**
|
|
- `web/templates/base.html` (346 lines) - Has inline CSS (need to extract)
|
|
- `web/templates/dashboard.html` (356 lines) - Basic stats, needs charts
|
|
- `web/templates/scans.html` (469 lines) - Has white row bug
|
|
- `web/templates/scan_detail.html` (399 lines) - Has white row bug in port tables
|
|
|
|
**Missing Templates:**
|
|
- Schedule list page
|
|
- Schedule create form
|
|
- Schedule edit form
|
|
|
|
**Web Routes (`web/routes/main.py` - 69 lines):**
|
|
- Only has scan routes, no schedule routes
|
|
|
|
---
|
|
|
|
## Critical Bug Fix
|
|
|
|
### White Row Coloring Issue
|
|
|
|
**Problem:** Dynamically created table rows in scans tables display with white background instead of dark theme colors.
|
|
|
|
**Affected Files:**
|
|
1. `web/templates/scans.html` (lines 208-241) - renderScansTable()
|
|
2. `web/templates/dashboard.html` (lines 221-260) - renderScansTable()
|
|
3. `web/templates/scan_detail.html` (lines 305-327) - Port tables
|
|
|
|
**Root Cause:**
|
|
|
|
JavaScript dynamically creates `<tr>` elements that don't inherit CSS styles properly:
|
|
|
|
```javascript
|
|
const row = document.createElement('tr');
|
|
row.innerHTML = `
|
|
<td class="mono">${scan.id}</td>
|
|
<td>${scan.title || 'Untitled Scan'}</td>
|
|
...
|
|
`;
|
|
tbody.appendChild(row);
|
|
```
|
|
|
|
**Current CSS (base.html lines 157-165):**
|
|
```css
|
|
.table tbody tr {
|
|
background-color: #1e293b; /* Dark slate */
|
|
border-color: #334155;
|
|
}
|
|
|
|
.table tbody tr:hover {
|
|
background-color: #334155;
|
|
cursor: pointer;
|
|
}
|
|
```
|
|
|
|
**Solution:**
|
|
|
|
Add explicit class to dynamically created rows:
|
|
|
|
```javascript
|
|
const row = document.createElement('tr');
|
|
row.classList.add('scan-row'); // Add explicit class
|
|
row.innerHTML = `...`;
|
|
```
|
|
|
|
And enhance CSS with higher specificity:
|
|
```css
|
|
.table tbody tr.scan-row,
|
|
.table tbody tr {
|
|
background-color: #1e293b !important;
|
|
border-color: #334155 !important;
|
|
}
|
|
```
|
|
|
|
---
|
|
|
|
## Files to Create
|
|
|
|
### Backend Services
|
|
|
|
#### 1. `web/services/schedule_service.py`
|
|
Schedule CRUD operations and business logic.
|
|
|
|
**Class: ScheduleService**
|
|
|
|
**Estimated Size:** ~400 lines
|
|
|
|
**Methods:**
|
|
- `__init__(self, db_session)` - Initialize with database session
|
|
- `create_schedule(name, config_file, cron_expression, enabled=True)` → schedule_id
|
|
- Validate cron expression
|
|
- Validate config file exists
|
|
- Calculate next_run time
|
|
- Create Schedule record
|
|
- Return schedule_id
|
|
|
|
- `get_schedule(schedule_id)` → schedule dict
|
|
- Query Schedule by ID
|
|
- Format for API response
|
|
- Include execution history (recent scans with this schedule_id)
|
|
|
|
- `list_schedules(page=1, per_page=20, enabled_filter=None)` → paginated results
|
|
- Query with pagination
|
|
- Filter by enabled status if provided
|
|
- Calculate relative next_run time ("in 2 hours", "tomorrow at 3:00 PM")
|
|
- Return total count and items
|
|
|
|
- `update_schedule(schedule_id, **updates)` → schedule dict
|
|
- Validate cron_expression if changed
|
|
- Recalculate next_run if cron changed
|
|
- Update Schedule record
|
|
- Reload schedule in APScheduler
|
|
- Return updated schedule
|
|
|
|
- `delete_schedule(schedule_id)` → success
|
|
- Remove job from APScheduler
|
|
- Delete Schedule record (do NOT delete associated scans)
|
|
- Return success status
|
|
|
|
- `toggle_enabled(schedule_id, enabled)` → schedule dict
|
|
- Enable or disable schedule
|
|
- Add/remove from APScheduler
|
|
- Return updated schedule
|
|
|
|
- `update_run_times(schedule_id, last_run, next_run)` → success
|
|
- Update last_run and next_run timestamps
|
|
- Called after each execution
|
|
- Return success status
|
|
|
|
- `validate_cron_expression(cron_expr)` → (valid, error_message)
|
|
- Use croniter library
|
|
- Return True if valid, False + error message if invalid
|
|
|
|
- `calculate_next_run(cron_expr, from_time=None)` → datetime
|
|
- Calculate next run time from cron expression
|
|
- Use croniter
|
|
- Return datetime (UTC)
|
|
|
|
- `get_schedule_history(schedule_id, limit=10)` → list of scans
|
|
- Query recent scans triggered by this schedule
|
|
- Return list of scan dicts
|
|
|
|
**Helper Methods:**
|
|
- `_schedule_to_dict(schedule_obj)` - Convert Schedule model to dict
|
|
- `_get_relative_time(dt)` - Format datetime as "in 2 hours", "yesterday"
|
|
|
|
#### 2. `web/static/css/styles.css`
|
|
Extracted CSS for better maintainability.
|
|
|
|
**Estimated Size:** ~350 lines (extracted from base.html)
|
|
|
|
**Sections:**
|
|
- Variables (CSS custom properties for colors)
|
|
- Global styles
|
|
- Navigation
|
|
- Cards and containers
|
|
- Tables (including fix for white rows)
|
|
- Forms
|
|
- Buttons
|
|
- Badges and labels
|
|
- Charts (dark theme)
|
|
- Utilities
|
|
|
|
**Example Structure:**
|
|
```css
|
|
/* CSS Variables */
|
|
:root {
|
|
--bg-primary: #0f172a;
|
|
--bg-secondary: #1e293b;
|
|
--bg-tertiary: #334155;
|
|
--text-primary: #e2e8f0;
|
|
--text-secondary: #94a3b8;
|
|
--border-color: #334155;
|
|
--accent-blue: #60a5fa;
|
|
--success-bg: #065f46;
|
|
--success-text: #6ee7b7;
|
|
--warning-bg: #78350f;
|
|
--warning-text: #fcd34d;
|
|
--danger-bg: #7f1d1d;
|
|
--danger-text: #fca5a5;
|
|
}
|
|
|
|
/* Fix for dynamically created table rows */
|
|
.table tbody tr,
|
|
.table tbody tr.scan-row {
|
|
background-color: var(--bg-secondary) !important;
|
|
border-color: var(--border-color) !important;
|
|
}
|
|
|
|
.table tbody tr:hover {
|
|
background-color: var(--bg-tertiary) !important;
|
|
cursor: pointer;
|
|
}
|
|
```
|
|
|
|
### Frontend Templates
|
|
|
|
#### 3. `web/templates/schedules.html`
|
|
List all schedules with enable/disable toggles.
|
|
|
|
**Estimated Size:** ~400 lines
|
|
|
|
**Layout:**
|
|
```html
|
|
{% extends "base.html" %}
|
|
|
|
{% block content %}
|
|
<div class="container mt-4">
|
|
<div class="d-flex justify-content-between align-items-center mb-4">
|
|
<h2>Scheduled Scans</h2>
|
|
<a href="{{ url_for('main.create_schedule') }}" class="btn btn-primary">
|
|
<i class="bi bi-plus-circle"></i> New Schedule
|
|
</a>
|
|
</div>
|
|
|
|
<!-- Stats Cards -->
|
|
<div class="row mb-4">
|
|
<div class="col-md-3">
|
|
<div class="card">
|
|
<div class="card-body">
|
|
<h6>Total Schedules</h6>
|
|
<h2 id="total-schedules">-</h2>
|
|
</div>
|
|
</div>
|
|
</div>
|
|
<div class="col-md-3">
|
|
<div class="card">
|
|
<div class="card-body">
|
|
<h6>Enabled</h6>
|
|
<h2 id="enabled-schedules">-</h2>
|
|
</div>
|
|
</div>
|
|
</div>
|
|
<div class="col-md-3">
|
|
<div class="card">
|
|
<div class="card-body">
|
|
<h6>Next Run</h6>
|
|
<h5 id="next-run-time">-</h5>
|
|
</div>
|
|
</div>
|
|
</div>
|
|
<div class="col-md-3">
|
|
<div class="card">
|
|
<div class="card-body">
|
|
<h6>Executions (24h)</h6>
|
|
<h2 id="recent-executions">-</h2>
|
|
</div>
|
|
</div>
|
|
</div>
|
|
</div>
|
|
|
|
<!-- Schedules Table -->
|
|
<div class="card">
|
|
<div class="card-body">
|
|
<table class="table">
|
|
<thead>
|
|
<tr>
|
|
<th>Name</th>
|
|
<th>Schedule (Cron)</th>
|
|
<th>Next Run</th>
|
|
<th>Last Run</th>
|
|
<th>Status</th>
|
|
<th>Actions</th>
|
|
</tr>
|
|
</thead>
|
|
<tbody id="schedules-tbody">
|
|
<!-- Populated by JavaScript -->
|
|
</tbody>
|
|
</table>
|
|
</div>
|
|
</div>
|
|
</div>
|
|
|
|
<script>
|
|
// Fetch and render schedules
|
|
async function loadSchedules() {
|
|
const response = await fetch('/api/schedules');
|
|
const data = await response.json();
|
|
renderSchedulesTable(data.schedules);
|
|
updateStats(data);
|
|
}
|
|
|
|
function renderSchedulesTable(schedules) {
|
|
const tbody = document.getElementById('schedules-tbody');
|
|
tbody.innerHTML = '';
|
|
|
|
schedules.forEach(schedule => {
|
|
const row = document.createElement('tr');
|
|
row.classList.add('schedule-row'); // Fix white rows
|
|
row.innerHTML = `
|
|
<td>${schedule.name}</td>
|
|
<td><code>${schedule.cron_expression}</code></td>
|
|
<td>${formatRelativeTime(schedule.next_run)}</td>
|
|
<td>${formatRelativeTime(schedule.last_run) || 'Never'}</td>
|
|
<td>
|
|
<div class="form-check form-switch">
|
|
<input class="form-check-input" type="checkbox"
|
|
id="enable-${schedule.id}"
|
|
${schedule.enabled ? 'checked' : ''}
|
|
onchange="toggleSchedule(${schedule.id}, this.checked)">
|
|
</div>
|
|
</td>
|
|
<td>
|
|
<button class="btn btn-sm btn-secondary" onclick="triggerSchedule(${schedule.id})">
|
|
Run Now
|
|
</button>
|
|
<a href="/schedules/${schedule.id}/edit" class="btn btn-sm btn-secondary">
|
|
Edit
|
|
</a>
|
|
<button class="btn btn-sm btn-danger" onclick="deleteSchedule(${schedule.id})">
|
|
Delete
|
|
</button>
|
|
</td>
|
|
`;
|
|
tbody.appendChild(row);
|
|
});
|
|
}
|
|
|
|
// Toggle schedule enabled/disabled
|
|
async function toggleSchedule(scheduleId, enabled) {
|
|
await fetch(`/api/schedules/${scheduleId}`, {
|
|
method: 'PUT',
|
|
headers: {'Content-Type': 'application/json'},
|
|
body: JSON.stringify({enabled: enabled})
|
|
});
|
|
loadSchedules();
|
|
}
|
|
|
|
// Manually trigger schedule
|
|
async function triggerSchedule(scheduleId) {
|
|
const response = await fetch(`/api/schedules/${scheduleId}/trigger`, {
|
|
method: 'POST'
|
|
});
|
|
const data = await response.json();
|
|
alert(`Scan triggered! Scan ID: ${data.scan_id}`);
|
|
window.location.href = `/scans/${data.scan_id}`;
|
|
}
|
|
|
|
// Delete schedule
|
|
async function deleteSchedule(scheduleId) {
|
|
if (!confirm('Delete this schedule?')) return;
|
|
await fetch(`/api/schedules/${scheduleId}`, {method: 'DELETE'});
|
|
loadSchedules();
|
|
}
|
|
|
|
// Load on page load
|
|
loadSchedules();
|
|
setInterval(loadSchedules, 30000); // Refresh every 30 seconds
|
|
</script>
|
|
{% endblock %}
|
|
```
|
|
|
|
#### 4. `web/templates/schedule_create.html`
|
|
Form to create new scheduled scan.
|
|
|
|
**Estimated Size:** ~300 lines
|
|
|
|
**Features:**
|
|
- Schedule name input
|
|
- Config file selector (dropdown of available configs)
|
|
- Cron expression builder OR manual entry
|
|
- Cron expression validator (client-side and server-side)
|
|
- Human-readable cron description
|
|
- Enable/disable toggle
|
|
- Submit button
|
|
|
|
**Cron Expression Builder:**
|
|
```html
|
|
<div class="card mb-3">
|
|
<div class="card-body">
|
|
<h5>Schedule Configuration</h5>
|
|
|
|
<!-- Quick Templates -->
|
|
<div class="mb-3">
|
|
<label>Quick Templates:</label>
|
|
<div class="btn-group" role="group">
|
|
<button type="button" class="btn btn-secondary btn-sm" onclick="setCron('0 0 * * *')">Daily at Midnight</button>
|
|
<button type="button" class="btn btn-secondary btn-sm" onclick="setCron('0 2 * * *')">Daily at 2 AM</button>
|
|
<button type="button" class="btn btn-secondary btn-sm" onclick="setCron('0 0 * * 0')">Weekly (Sunday)</button>
|
|
<button type="button" class="btn btn-secondary btn-sm" onclick="setCron('0 0 1 * *')">Monthly (1st)</button>
|
|
</div>
|
|
</div>
|
|
|
|
<!-- Manual Entry -->
|
|
<div class="mb-3">
|
|
<label for="cron-expression">Cron Expression:</label>
|
|
<input type="text" class="form-control font-monospace"
|
|
id="cron-expression"
|
|
name="cron_expression"
|
|
placeholder="0 2 * * *"
|
|
onchange="validateCron()">
|
|
<small class="form-text text-muted">
|
|
Format: minute hour day month weekday
|
|
</small>
|
|
</div>
|
|
|
|
<!-- Human-Readable Description -->
|
|
<div class="alert alert-info" id="cron-description">
|
|
Enter a cron expression above
|
|
</div>
|
|
|
|
<!-- Next Run Times -->
|
|
<div id="next-runs">
|
|
<strong>Next 5 runs:</strong>
|
|
<ul id="next-runs-list"></ul>
|
|
</div>
|
|
</div>
|
|
</div>
|
|
```
|
|
|
|
#### 5. `web/templates/schedule_edit.html`
|
|
Form to edit existing schedule.
|
|
|
|
**Estimated Size:** ~250 lines
|
|
|
|
**Similar to create, but:**
|
|
- Pre-populate fields from existing schedule
|
|
- Show execution history (last 10 scans)
|
|
- "Delete Schedule" button
|
|
- "Test Run Now" button
|
|
|
|
### Testing Files
|
|
|
|
#### 6. `tests/test_schedule_service.py`
|
|
Unit tests for ScheduleService.
|
|
|
|
**Estimated Size:** ~450 lines, 18+ tests
|
|
|
|
**Test Coverage:**
|
|
- `test_create_schedule` - Valid schedule creation
|
|
- `test_create_schedule_invalid_cron` - Cron validation
|
|
- `test_create_schedule_invalid_config` - Config file validation
|
|
- `test_get_schedule` - Retrieve schedule
|
|
- `test_get_schedule_not_found` - 404 handling
|
|
- `test_list_schedules` - Pagination
|
|
- `test_list_schedules_filter_enabled` - Filter by enabled status
|
|
- `test_update_schedule` - Update fields
|
|
- `test_update_schedule_cron` - Recalculate next_run when cron changes
|
|
- `test_delete_schedule` - Delete schedule
|
|
- `test_toggle_enabled` - Enable/disable
|
|
- `test_update_run_times` - Update after execution
|
|
- `test_validate_cron_expression_valid` - Valid expressions
|
|
- `test_validate_cron_expression_invalid` - Invalid expressions
|
|
- `test_calculate_next_run` - Next run calculation
|
|
- `test_get_schedule_history` - Execution history
|
|
- `test_schedule_to_dict` - Serialization
|
|
- `test_concurrent_schedule_operations` - Thread safety
|
|
|
|
#### 7. `tests/test_schedule_api.py`
|
|
Integration tests for schedules API.
|
|
|
|
**Estimated Size:** ~500 lines, 22+ tests
|
|
|
|
**Test Coverage:**
|
|
- `test_list_schedules_empty` - Empty list
|
|
- `test_list_schedules_populated` - Multiple schedules
|
|
- `test_list_schedules_pagination` - Pagination
|
|
- `test_list_schedules_filter_enabled` - Filter
|
|
- `test_get_schedule` - Get details
|
|
- `test_get_schedule_not_found` - 404
|
|
- `test_create_schedule` - Create new
|
|
- `test_create_schedule_invalid_cron` - Validation
|
|
- `test_create_schedule_invalid_config` - File validation
|
|
- `test_update_schedule` - Update fields
|
|
- `test_update_schedule_not_found` - 404
|
|
- `test_delete_schedule` - Delete
|
|
- `test_delete_schedule_not_found` - 404
|
|
- `test_trigger_schedule` - Manual trigger
|
|
- `test_trigger_schedule_not_found` - 404
|
|
- `test_toggle_enabled_via_update` - Enable/disable
|
|
- `test_schedules_require_authentication` - Auth required
|
|
- `test_schedule_execution_history` - Show related scans
|
|
- `test_schedule_next_run_calculation` - Correct calculation
|
|
- `test_schedule_workflow_integration` - Complete workflow
|
|
- `test_concurrent_schedule_updates` - Concurrency
|
|
|
|
#### 8. `tests/test_charts.py`
|
|
Tests for chart data generation.
|
|
|
|
**Estimated Size:** ~200 lines, 8+ tests
|
|
|
|
**Test Coverage:**
|
|
- `test_scan_trend_data` - Scans per day calculation
|
|
- `test_port_count_trend` - Port count over time
|
|
- `test_service_distribution` - Service type breakdown
|
|
- `test_certificate_expiry_timeline` - Cert expiry dates
|
|
- `test_empty_data_handling` - No scans case
|
|
- `test_date_range_filtering` - Filter by date range
|
|
- `test_data_format_for_chartjs` - Correct JSON format
|
|
- `test_trend_data_caching` - Performance optimization
|
|
|
|
---
|
|
|
|
## Files to Modify
|
|
|
|
### Backend Updates
|
|
|
|
#### 1. `web/api/schedules.py`
|
|
Replace all stub implementations with working code.
|
|
|
|
**Current State:** 159 lines, all placeholders
|
|
|
|
**Changes:**
|
|
- Import ScheduleService
|
|
- Implement all 6 endpoints:
|
|
- `GET /api/schedules` - Call ScheduleService.list_schedules()
|
|
- `GET /api/schedules/<id>` - Call ScheduleService.get_schedule()
|
|
- `POST /api/schedules` - Call ScheduleService.create_schedule()
|
|
- `PUT /api/schedules/<id>` - Call ScheduleService.update_schedule()
|
|
- `DELETE /api/schedules/<id>` - Call ScheduleService.delete_schedule()
|
|
- `POST /api/schedules/<id>/trigger` - Trigger immediate scan with schedule_id
|
|
|
|
**New Code (~300 lines total):**
|
|
```python
|
|
from web.services.schedule_service import ScheduleService
|
|
from web.auth.decorators import api_auth_required
|
|
from flask import current_app, jsonify, request
|
|
|
|
@bp.route('', methods=['GET'])
|
|
@api_auth_required
|
|
def list_schedules():
|
|
"""List all schedules with pagination."""
|
|
page = request.args.get('page', 1, type=int)
|
|
per_page = request.args.get('per_page', 20, type=int)
|
|
enabled_filter = request.args.get('enabled', type=lambda x: x.lower() == 'true')
|
|
|
|
schedule_service = ScheduleService(current_app.db_session)
|
|
result = schedule_service.list_schedules(page, per_page, enabled_filter)
|
|
|
|
return jsonify(result), 200
|
|
|
|
@bp.route('', methods=['POST'])
|
|
@api_auth_required
|
|
def create_schedule():
|
|
"""Create a new schedule."""
|
|
data = request.get_json() or {}
|
|
|
|
# Validate required fields
|
|
required = ['name', 'config_file', 'cron_expression']
|
|
for field in required:
|
|
if field not in data:
|
|
return jsonify({'error': f'Missing required field: {field}'}), 400
|
|
|
|
schedule_service = ScheduleService(current_app.db_session)
|
|
|
|
try:
|
|
schedule_id = schedule_service.create_schedule(
|
|
name=data['name'],
|
|
config_file=data['config_file'],
|
|
cron_expression=data['cron_expression'],
|
|
enabled=data.get('enabled', True)
|
|
)
|
|
|
|
# Add to APScheduler
|
|
schedule = schedule_service.get_schedule(schedule_id)
|
|
current_app.scheduler.add_scheduled_scan(schedule)
|
|
|
|
return jsonify({
|
|
'schedule_id': schedule_id,
|
|
'message': 'Schedule created successfully'
|
|
}), 201
|
|
except ValueError as e:
|
|
return jsonify({'error': str(e)}), 400
|
|
|
|
# ... more endpoints
|
|
```
|
|
|
|
#### 2. `web/services/scheduler_service.py`
|
|
Complete placeholder implementations.
|
|
|
|
**Current State:** 258 lines, partial implementation
|
|
|
|
**Changes:**
|
|
- Complete `add_scheduled_scan(schedule)` - Add cron job to APScheduler
|
|
- Complete `_trigger_scheduled_scan(schedule_id)` - Execute scheduled scan
|
|
- Add `load_schedules_on_startup()` - Load all enabled schedules
|
|
- Add cron expression parsing with croniter
|
|
|
|
**New/Updated Methods:**
|
|
```python
|
|
from croniter import croniter
|
|
from datetime import datetime
|
|
|
|
def add_scheduled_scan(self, schedule):
|
|
"""Add a cron job for scheduled scan execution."""
|
|
if not schedule.get('enabled'):
|
|
return
|
|
|
|
job_id = f"schedule_{schedule['id']}"
|
|
|
|
# Parse cron expression
|
|
trigger = CronTrigger.from_crontab(schedule['cron_expression'])
|
|
|
|
# Add job to scheduler
|
|
self.scheduler.add_job(
|
|
func=self._trigger_scheduled_scan,
|
|
trigger=trigger,
|
|
id=job_id,
|
|
args=[schedule['id']],
|
|
replace_existing=True,
|
|
max_instances=1
|
|
)
|
|
|
|
logger.info(f"Added scheduled scan: {schedule['name']} ({job_id})")
|
|
|
|
def _trigger_scheduled_scan(self, schedule_id):
|
|
"""Execute a scheduled scan."""
|
|
from web.services.schedule_service import ScheduleService
|
|
|
|
logger.info(f"Triggering scheduled scan: schedule_id={schedule_id}")
|
|
|
|
# Get schedule details
|
|
schedule_service = ScheduleService(self.db_session)
|
|
schedule = schedule_service.get_schedule(schedule_id)
|
|
|
|
if not schedule:
|
|
logger.error(f"Schedule {schedule_id} not found")
|
|
return
|
|
|
|
# Trigger scan with schedule_id
|
|
from web.services.scan_service import ScanService
|
|
scan_service = ScanService(self.db_session)
|
|
|
|
scan_id = scan_service.trigger_scan(
|
|
config_file=schedule['config_file'],
|
|
triggered_by='scheduled',
|
|
schedule_id=schedule_id
|
|
)
|
|
|
|
# Queue the scan
|
|
self.queue_scan(schedule['config_file'], scan_id, self.db_url)
|
|
|
|
# Update last_run timestamp
|
|
schedule_service.update_run_times(
|
|
schedule_id=schedule_id,
|
|
last_run=datetime.utcnow(),
|
|
next_run=self._calculate_next_run(schedule['cron_expression'])
|
|
)
|
|
|
|
logger.info(f"Scheduled scan queued: scan_id={scan_id}")
|
|
|
|
def load_schedules_on_startup(self):
|
|
"""Load all enabled schedules from database on app startup."""
|
|
from web.services.schedule_service import ScheduleService
|
|
|
|
schedule_service = ScheduleService(self.db_session)
|
|
schedules = schedule_service.list_schedules(page=1, per_page=1000, enabled_filter=True)
|
|
|
|
for schedule in schedules['schedules']:
|
|
self.add_scheduled_scan(schedule)
|
|
|
|
logger.info(f"Loaded {len(schedules['schedules'])} schedules on startup")
|
|
|
|
def _calculate_next_run(self, cron_expression):
|
|
"""Calculate next run time from cron expression."""
|
|
cron = croniter(cron_expression, datetime.utcnow())
|
|
return cron.get_next(datetime)
|
|
```
|
|
|
|
#### 3. `web/routes/main.py`
|
|
Add schedule management routes.
|
|
|
|
**Current State:** 69 lines, only scan routes
|
|
|
|
**Changes:**
|
|
- Add schedule routes:
|
|
- `GET /schedules` - List schedules
|
|
- `GET /schedules/create` - Create form
|
|
- `GET /schedules/<id>/edit` - Edit form
|
|
|
|
**New Code:**
|
|
```python
|
|
@bp.route('/schedules')
|
|
@login_required
|
|
def schedules():
|
|
"""List all schedules."""
|
|
return render_template('schedules.html')
|
|
|
|
@bp.route('/schedules/create')
|
|
@login_required
|
|
def create_schedule():
|
|
"""Create new schedule form."""
|
|
# Get list of available config files
|
|
import os
|
|
configs_dir = '/app/configs'
|
|
config_files = [f for f in os.listdir(configs_dir) if f.endswith('.yaml')]
|
|
return render_template('schedule_create.html', config_files=config_files)
|
|
|
|
@bp.route('/schedules/<int:schedule_id>/edit')
|
|
@login_required
|
|
def edit_schedule(schedule_id):
|
|
"""Edit existing schedule form."""
|
|
from web.services.schedule_service import ScheduleService
|
|
schedule_service = ScheduleService(current_app.db_session)
|
|
|
|
try:
|
|
schedule = schedule_service.get_schedule(schedule_id)
|
|
return render_template('schedule_edit.html', schedule=schedule)
|
|
except Exception as e:
|
|
flash(f'Schedule not found: {e}', 'danger')
|
|
return redirect(url_for('main.schedules'))
|
|
```
|
|
|
|
#### 4. `web/templates/base.html`
|
|
Extract CSS, add Chart.js, fix white rows.
|
|
|
|
**Current State:** 346 lines with 280 lines of inline CSS (lines 8-288)
|
|
|
|
**Changes:**
|
|
- Replace inline `<style>` block with `<link>` to external CSS
|
|
- Add Chart.js CDN
|
|
- Add dark theme configuration for Chart.js
|
|
|
|
**Modifications:**
|
|
```html
|
|
<head>
|
|
<meta charset="UTF-8">
|
|
<meta name="viewport" content="width=device-width, initial-scale=1.0">
|
|
<title>{% block title %}SneakyScanner{% endblock %}</title>
|
|
|
|
<!-- Bootstrap 5 CSS -->
|
|
<link href="https://cdn.jsdelivr.net/npm/bootstrap@5.3.0/dist/css/bootstrap.min.css" rel="stylesheet">
|
|
|
|
<!-- Bootstrap Icons -->
|
|
<link rel="stylesheet" href="https://cdn.jsdelivr.net/npm/bootstrap-icons@1.11.0/font/bootstrap-icons.css">
|
|
|
|
<!-- Custom CSS (extracted from inline) -->
|
|
<link rel="stylesheet" href="{{ url_for('static', filename='css/styles.css') }}">
|
|
|
|
<!-- Chart.js for visualizations -->
|
|
<script src="https://cdn.jsdelivr.net/npm/chart.js@4.4.0/dist/chart.umd.min.js"></script>
|
|
|
|
<!-- Chart.js Dark Theme Configuration -->
|
|
<script>
|
|
Chart.defaults.color = '#e2e8f0';
|
|
Chart.defaults.borderColor = '#334155';
|
|
Chart.defaults.backgroundColor = '#1e293b';
|
|
</script>
|
|
</head>
|
|
```
|
|
|
|
#### 5. `web/templates/dashboard.html`
|
|
Fix white rows, add trending charts.
|
|
|
|
**Current State:** 356 lines, basic stats
|
|
|
|
**Changes:**
|
|
- Fix white rows in renderScansTable() (add `.scan-row` class)
|
|
- Add "Schedules" widget showing next scheduled scan
|
|
- Add Chart.js trending chart (scans per day, last 30 days)
|
|
- Add "Quick Actions" section
|
|
|
|
**New Section - Schedules Widget:**
|
|
```html
|
|
<div class="col-md-6">
|
|
<div class="card h-100">
|
|
<div class="card-header d-flex justify-content-between align-items-center">
|
|
<h5 class="mb-0">Upcoming Scheduled Scans</h5>
|
|
<a href="{{ url_for('main.schedules') }}" class="btn btn-sm btn-secondary">Manage</a>
|
|
</div>
|
|
<div class="card-body">
|
|
<div id="next-schedules">
|
|
<p class="text-muted">Loading...</p>
|
|
</div>
|
|
</div>
|
|
</div>
|
|
</div>
|
|
```
|
|
|
|
**New Section - Trending Chart:**
|
|
```html
|
|
<div class="row mb-4">
|
|
<div class="col-12">
|
|
<div class="card">
|
|
<div class="card-header">
|
|
<h5 class="mb-0">Scan Activity (Last 30 Days)</h5>
|
|
</div>
|
|
<div class="card-body">
|
|
<canvas id="scanTrendChart" height="80"></canvas>
|
|
</div>
|
|
</div>
|
|
</div>
|
|
</div>
|
|
|
|
<script>
|
|
async function loadScanTrend() {
|
|
const response = await fetch('/api/stats/scan-trend?days=30');
|
|
const data = await response.json();
|
|
|
|
const ctx = document.getElementById('scanTrendChart').getContext('2d');
|
|
new Chart(ctx, {
|
|
type: 'line',
|
|
data: {
|
|
labels: data.labels, // ['2025-01-01', '2025-01-02', ...]
|
|
datasets: [{
|
|
label: 'Scans',
|
|
data: data.values, // [5, 3, 7, 2, ...]
|
|
borderColor: '#60a5fa',
|
|
backgroundColor: 'rgba(96, 165, 250, 0.1)',
|
|
tension: 0.3
|
|
}]
|
|
},
|
|
options: {
|
|
responsive: true,
|
|
maintainAspectRatio: false,
|
|
plugins: {
|
|
legend: {display: false}
|
|
},
|
|
scales: {
|
|
y: {
|
|
beginAtZero: true,
|
|
ticks: {stepSize: 1}
|
|
}
|
|
}
|
|
}
|
|
});
|
|
}
|
|
|
|
loadScanTrend();
|
|
</script>
|
|
```
|
|
|
|
**Fix for White Rows (line ~221):**
|
|
```javascript
|
|
function renderScansTable(scans) {
|
|
const tbody = document.getElementById('recent-scans');
|
|
tbody.innerHTML = '';
|
|
|
|
if (scans.length === 0) {
|
|
tbody.innerHTML = '<tr class="scan-row"><td colspan="5" class="text-center text-muted">No scans yet</td></tr>';
|
|
return;
|
|
}
|
|
|
|
scans.forEach(scan => {
|
|
const row = document.createElement('tr');
|
|
row.classList.add('scan-row'); // ← FIX: Add explicit class
|
|
row.onclick = () => window.location.href = `/scans/${scan.id}`;
|
|
row.innerHTML = `
|
|
<td class="mono">${scan.id}</td>
|
|
<td>${scan.title || 'Untitled Scan'}</td>
|
|
<td>${new Date(scan.timestamp).toLocaleString()}</td>
|
|
<td>${getStatusBadge(scan.status)}</td>
|
|
`;
|
|
tbody.appendChild(row);
|
|
});
|
|
}
|
|
```
|
|
|
|
#### 6. `web/templates/scans.html`
|
|
Fix white rows bug.
|
|
|
|
**Current State:** 469 lines, white row bug at lines 208-241
|
|
|
|
**Changes:**
|
|
- Add `.scan-row` class to dynamically created rows (same fix as dashboard)
|
|
|
|
**Fix (line ~208):**
|
|
```javascript
|
|
function renderScansTable(scans) {
|
|
const tbody = document.getElementById('scans-tbody');
|
|
tbody.innerHTML = '';
|
|
|
|
if (scans.length === 0) {
|
|
tbody.innerHTML = '<tr class="scan-row"><td colspan="6" class="text-center text-muted">No scans found</td></tr>';
|
|
return;
|
|
}
|
|
|
|
scans.forEach(scan => {
|
|
const row = document.createElement('tr');
|
|
row.classList.add('scan-row'); // ← FIX: Add explicit class
|
|
row.onclick = () => window.location.href = `/scans/${scan.id}`;
|
|
// ... rest of row creation
|
|
});
|
|
}
|
|
```
|
|
|
|
#### 7. `web/templates/scan_detail.html`
|
|
Fix white rows in port tables.
|
|
|
|
**Current State:** 399 lines, white rows in port tables (lines 305-327)
|
|
|
|
**Changes:**
|
|
- Add `.scan-row` class to port table rows
|
|
|
|
**Fix (in renderPortsTable function):**
|
|
```javascript
|
|
function renderPortsTable(ports, containerId) {
|
|
const tbody = document.getElementById(containerId);
|
|
tbody.innerHTML = '';
|
|
|
|
ports.forEach(port => {
|
|
const row = document.createElement('tr');
|
|
row.classList.add('scan-row'); // ← FIX: Add explicit class
|
|
row.innerHTML = `
|
|
<td>${port.port}</td>
|
|
<td>${port.protocol.toUpperCase()}</td>
|
|
<td>${getExpectedBadge(port.expected)}</td>
|
|
<td>${port.state}</td>
|
|
`;
|
|
tbody.appendChild(row);
|
|
});
|
|
}
|
|
```
|
|
|
|
#### 8. `web/app.py`
|
|
Load schedules on startup.
|
|
|
|
**Current State:** Initializes scheduler but doesn't load schedules
|
|
|
|
**Changes:**
|
|
- After scheduler initialization, load all enabled schedules
|
|
|
|
**New Code (after scheduler init):**
|
|
```python
|
|
def init_scheduler(app):
|
|
"""Initialize APScheduler and load schedules."""
|
|
from web.services.scheduler_service import SchedulerService
|
|
|
|
scheduler_service = SchedulerService()
|
|
scheduler_service.init_scheduler(app)
|
|
|
|
# Store in app context
|
|
app.scheduler = scheduler_service
|
|
|
|
# Load all enabled schedules from database
|
|
with app.app_context():
|
|
scheduler_service.load_schedules_on_startup()
|
|
|
|
logger.info("Scheduler initialized and schedules loaded")
|
|
```
|
|
|
|
#### 9. `requirements-web.txt`
|
|
Add croniter dependency.
|
|
|
|
**Current State:** All Phase 2 dependencies
|
|
|
|
**Changes:**
|
|
- Add croniter for cron expression parsing
|
|
|
|
**New Line:**
|
|
```
|
|
croniter==2.0.1
|
|
```
|
|
|
|
---
|
|
|
|
## Step-by-Step Implementation
|
|
|
|
### Step 1: Fix Styling Issues & CSS Refactor ✅ (Day 1)
|
|
**Priority: CRITICAL** - Fixes user-facing bug
|
|
|
|
**Tasks:**
|
|
1. Create `web/static/css/styles.css`
|
|
- Extract all CSS from base.html `<style>` block
|
|
- Add CSS variables for colors
|
|
- Add fix for white row bug with `.scan-row` class
|
|
- Add Chart.js dark theme styles
|
|
|
|
2. Update `web/templates/base.html`
|
|
- Remove inline `<style>` block
|
|
- Add `<link>` to external styles.css
|
|
- Add Chart.js CDN script
|
|
- Add Chart.js default configuration for dark theme
|
|
|
|
3. Fix white rows in all templates:
|
|
- `web/templates/dashboard.html` - Add `.scan-row` class
|
|
- `web/templates/scans.html` - Add `.scan-row` class
|
|
- `web/templates/scan_detail.html` - Add `.scan-row` class to port tables
|
|
|
|
4. Test CSS changes:
|
|
- Verify all pages render correctly
|
|
- Verify dark theme maintained
|
|
- Verify table rows are dark slate, not white
|
|
- Verify hover effects work
|
|
|
|
**Deliverables:**
|
|
- `web/static/css/styles.css` (~350 lines)
|
|
- Updated base.html (reduced by ~280 lines)
|
|
- Fixed white rows in 3 templates
|
|
- Chart.js ready for use in next steps
|
|
|
|
**Testing:**
|
|
- Manual: View dashboard, scans list, scan detail - verify no white rows
|
|
- Manual: Test hover effects on tables
|
|
- Manual: Verify no CSS regressions
|
|
|
|
---
|
|
|
|
### Step 2: ScheduleService Implementation ✅ (Days 2-3)
|
|
**Priority: HIGH** - Core business logic for schedules
|
|
|
|
**Tasks:**
|
|
1. Create `web/services/schedule_service.py` (~400 lines)
|
|
- Implement all CRUD methods
|
|
- Add cron expression validation with croniter
|
|
- Add next run time calculation
|
|
- Add relative time formatting
|
|
- Add schedule-to-dict serialization
|
|
- Add execution history retrieval
|
|
|
|
2. Add croniter to requirements
|
|
- Update `requirements-web.txt`
|
|
|
|
3. Write comprehensive unit tests
|
|
- Create `tests/test_schedule_service.py` (~450 lines)
|
|
- 18+ test functions covering all methods
|
|
- Test cron validation edge cases
|
|
- Test next run calculation accuracy
|
|
- Test error handling
|
|
|
|
**Key Methods to Implement:**
|
|
- `create_schedule()` - Validate and create
|
|
- `get_schedule()` - Retrieve with history
|
|
- `list_schedules()` - Paginated list
|
|
- `update_schedule()` - Update with validation
|
|
- `delete_schedule()` - Remove schedule
|
|
- `toggle_enabled()` - Enable/disable
|
|
- `validate_cron_expression()` - Validation
|
|
- `calculate_next_run()` - Next run time
|
|
|
|
**Deliverables:**
|
|
- ScheduleService class fully implemented
|
|
- croniter dependency added
|
|
- 18+ unit tests written and passing
|
|
|
|
**Testing:**
|
|
- Unit: All ScheduleService methods
|
|
- Edge cases: Invalid cron, missing config files
|
|
- Concurrency: Multiple simultaneous schedule operations
|
|
|
|
---
|
|
|
|
### Step 3: Schedules API Endpoints ✅ (Days 4-5)
|
|
**Priority: HIGH** - API for schedule management
|
|
|
|
**Tasks:**
|
|
1. Update `web/api/schedules.py` (~300 lines total)
|
|
- Replace all 6 stub endpoints with implementations
|
|
- Add comprehensive error handling
|
|
- Add input validation
|
|
- Add logging
|
|
- Integrate with ScheduleService and SchedulerService
|
|
|
|
2. Endpoints to implement:
|
|
- `GET /api/schedules` - List with pagination and filtering
|
|
- `GET /api/schedules/<id>` - Get schedule details + history
|
|
- `POST /api/schedules` - Create and add to scheduler
|
|
- `PUT /api/schedules/<id>` - Update and reload in scheduler
|
|
- `DELETE /api/schedules/<id>` - Delete and remove from scheduler
|
|
- `POST /api/schedules/<id>/trigger` - Manually trigger scan
|
|
|
|
3. Write integration tests
|
|
- Create `tests/test_schedule_api.py` (~500 lines)
|
|
- 22+ integration tests
|
|
- Test complete workflows
|
|
- Test authentication
|
|
- Test error scenarios
|
|
|
|
**Implementation Example:**
|
|
```python
|
|
@bp.route('', methods=['POST'])
|
|
@api_auth_required
|
|
def create_schedule():
|
|
data = request.get_json() or {}
|
|
|
|
# Validate
|
|
if not all(k in data for k in ['name', 'config_file', 'cron_expression']):
|
|
return jsonify({'error': 'Missing required fields'}), 400
|
|
|
|
schedule_service = ScheduleService(current_app.db_session)
|
|
|
|
try:
|
|
schedule_id = schedule_service.create_schedule(
|
|
name=data['name'],
|
|
config_file=data['config_file'],
|
|
cron_expression=data['cron_expression'],
|
|
enabled=data.get('enabled', True)
|
|
)
|
|
|
|
# Add to APScheduler
|
|
schedule = schedule_service.get_schedule(schedule_id)
|
|
current_app.scheduler.add_scheduled_scan(schedule)
|
|
|
|
return jsonify({
|
|
'schedule_id': schedule_id,
|
|
'message': 'Schedule created successfully'
|
|
}), 201
|
|
except ValueError as e:
|
|
return jsonify({'error': str(e)}), 400
|
|
```
|
|
|
|
**Deliverables:**
|
|
- All 6 API endpoints implemented
|
|
- 22+ integration tests written and passing
|
|
- Comprehensive error handling
|
|
- API fully functional
|
|
|
|
**Testing:**
|
|
- Integration: All endpoints with real database
|
|
- Workflow: Create → update → trigger → delete
|
|
- Error: Invalid inputs, missing schedules, auth failures
|
|
|
|
---
|
|
|
|
### Step 4: Schedule Management UI ✅ (Days 6-7)
|
|
**Priority: HIGH** - User interface for schedules
|
|
|
|
**Tasks:**
|
|
1. Create `web/templates/schedules.html` (~400 lines)
|
|
- Stats cards (total, enabled, next run, executions)
|
|
- Schedules table with enable/disable toggles
|
|
- Next run time display (human-readable)
|
|
- Action buttons (Run Now, Edit, Delete)
|
|
- AJAX data loading
|
|
- Auto-refresh every 30 seconds
|
|
|
|
2. Create `web/templates/schedule_create.html` (~300 lines)
|
|
- Schedule name input
|
|
- Config file dropdown (list available configs)
|
|
- Cron expression input with validation
|
|
- Quick cron templates (daily, weekly, monthly)
|
|
- Human-readable cron description
|
|
- Next 5 run times preview
|
|
- Enable/disable toggle
|
|
- Submit button
|
|
|
|
3. Create `web/templates/schedule_edit.html` (~250 lines)
|
|
- Pre-populated form fields
|
|
- Execution history (last 10 scans)
|
|
- Delete schedule button
|
|
- Test run button
|
|
|
|
4. Update `web/routes/main.py`
|
|
- Add `/schedules` route
|
|
- Add `/schedules/create` route
|
|
- Add `/schedules/<id>/edit` route
|
|
|
|
5. Add cron helper JavaScript
|
|
- Cron expression validator (client-side)
|
|
- Human-readable formatter
|
|
- Next runs calculator
|
|
|
|
**Cron Expression Builder:**
|
|
```html
|
|
<!-- Quick Templates -->
|
|
<button onclick="setCron('0 0 * * *')">Daily at Midnight</button>
|
|
<button onclick="setCron('0 2 * * *')">Daily at 2 AM</button>
|
|
<button onclick="setCron('0 0 * * 0')">Weekly (Sunday)</button>
|
|
<button onclick="setCron('0 0 1 * *')">Monthly</button>
|
|
|
|
<!-- Manual Entry with Validation -->
|
|
<input type="text" id="cron-expression" onchange="validateCron()">
|
|
<div id="cron-description"><!-- Human-readable description --></div>
|
|
<div id="next-runs"><!-- Next 5 run times --></div>
|
|
```
|
|
|
|
**Deliverables:**
|
|
- 3 new templates created
|
|
- 3 new web routes added
|
|
- Fully functional schedule management UI
|
|
- Client-side cron validation
|
|
|
|
**Testing:**
|
|
- Manual: Create schedule via UI
|
|
- Manual: Edit existing schedule
|
|
- Manual: Enable/disable toggle
|
|
- Manual: Manual trigger works
|
|
- Manual: Delete schedule
|
|
- Verify: Cron validation prevents invalid expressions
|
|
|
|
---
|
|
|
|
### Step 5: Enhanced Dashboard with Charts ✅ (Days 8-9)
|
|
**Priority: MEDIUM** - Improved dashboard visualization
|
|
|
|
**Tasks:**
|
|
1. Add API endpoint for trend data
|
|
- Create `GET /api/stats/scan-trend?days=30`
|
|
- Return scan counts per day
|
|
- Return labels and values for Chart.js
|
|
|
|
2. Update `web/templates/dashboard.html`
|
|
- Add "Schedules" widget showing next scheduled scans
|
|
- Add Chart.js line chart for scan activity (last 30 days)
|
|
- Add "Quick Actions" section (Run Scan, Create Schedule)
|
|
- Improve layout with Bootstrap grid
|
|
|
|
3. Add additional chart (optional)
|
|
- Port count trend over time
|
|
- Service distribution pie chart
|
|
- Certificate expiry timeline
|
|
|
|
4. Write tests for chart data
|
|
- Create `tests/test_charts.py` (~200 lines)
|
|
- Test data aggregation
|
|
- Test date range filtering
|
|
- Test empty data handling
|
|
|
|
**Chart Implementation:**
|
|
```javascript
|
|
async function loadScanTrend() {
|
|
const response = await fetch('/api/stats/scan-trend?days=30');
|
|
const data = await response.json();
|
|
|
|
const ctx = document.getElementById('scanTrendChart').getContext('2d');
|
|
new Chart(ctx, {
|
|
type: 'line',
|
|
data: {
|
|
labels: data.labels,
|
|
datasets: [{
|
|
label: 'Scans',
|
|
data: data.values,
|
|
borderColor: '#60a5fa',
|
|
backgroundColor: 'rgba(96, 165, 250, 0.1)',
|
|
tension: 0.3
|
|
}]
|
|
},
|
|
options: {
|
|
responsive: true,
|
|
maintainAspectRatio: false,
|
|
scales: {
|
|
y: {beginAtZero: true}
|
|
}
|
|
}
|
|
});
|
|
}
|
|
```
|
|
|
|
**Deliverables:**
|
|
- Enhanced dashboard with charts
|
|
- Schedule widget showing next runs
|
|
- Stats API endpoint for trends
|
|
- Chart tests written
|
|
|
|
**Testing:**
|
|
- Manual: Verify charts render correctly
|
|
- Manual: Verify dark theme applied to charts
|
|
- Unit: Chart data generation
|
|
- Edge: Empty data, single data point
|
|
|
|
---
|
|
|
|
### Step 6: Scheduler Integration ✅ COMPLETE (Day 10)
|
|
**Priority: CRITICAL** - Complete scheduled execution
|
|
|
|
**Tasks:**
|
|
1. Update `web/services/scheduler_service.py`
|
|
- Complete `add_scheduled_scan()` implementation
|
|
- Complete `_trigger_scheduled_scan()` implementation
|
|
- Add `load_schedules_on_startup()` method
|
|
- Add cron job to APScheduler with CronTrigger
|
|
|
|
2. Update `web/app.py`
|
|
- Call `load_schedules_on_startup()` after scheduler init
|
|
- Ensure app context available for database queries
|
|
|
|
3. Test scheduled execution
|
|
- Create test schedule with cron "* * * * *" (every minute)
|
|
- Verify scan is triggered automatically
|
|
- Verify last_run and next_run are updated
|
|
- Verify scans have `triggered_by='scheduled'`
|
|
|
|
4. Test enable/disable
|
|
- Disable schedule
|
|
- Verify job removed from APScheduler
|
|
- Enable schedule
|
|
- Verify job added back to APScheduler
|
|
|
|
**Key Implementation:**
|
|
```python
|
|
from apscheduler.triggers.cron import CronTrigger
|
|
|
|
def add_scheduled_scan(self, schedule):
|
|
"""Add cron job for scheduled scan."""
|
|
job_id = f"schedule_{schedule['id']}"
|
|
|
|
trigger = CronTrigger.from_crontab(schedule['cron_expression'])
|
|
|
|
self.scheduler.add_job(
|
|
func=self._trigger_scheduled_scan,
|
|
trigger=trigger,
|
|
id=job_id,
|
|
args=[schedule['id']],
|
|
replace_existing=True
|
|
)
|
|
|
|
def _trigger_scheduled_scan(self, schedule_id):
|
|
"""Execute scheduled scan."""
|
|
# Get schedule
|
|
schedule = ScheduleService(self.db_session).get_schedule(schedule_id)
|
|
|
|
# Trigger scan
|
|
scan_id = ScanService(self.db_session).trigger_scan(
|
|
config_file=schedule['config_file'],
|
|
triggered_by='scheduled',
|
|
schedule_id=schedule_id
|
|
)
|
|
|
|
# Queue scan
|
|
self.queue_scan(schedule['config_file'], scan_id, self.db_url)
|
|
|
|
# Update timestamps
|
|
ScheduleService(self.db_session).update_run_times(
|
|
schedule_id, datetime.utcnow(), self._calculate_next_run(...)
|
|
)
|
|
```
|
|
|
|
**Deliverables:**
|
|
- Scheduled scans execute automatically
|
|
- Schedules loaded on app startup
|
|
- Enable/disable functionality works
|
|
- Timestamps updated after execution
|
|
|
|
**Testing:**
|
|
- Integration: Create schedule with "* * * * *", wait 1 minute, verify scan created
|
|
- Manual: Enable/disable schedule, verify APScheduler jobs
|
|
- Edge: Schedule during scan, multiple schedules at same time
|
|
|
|
---
|
|
|
|
### Step 7: Scan Comparison Features ✅ (Days 11-12)
|
|
**Priority: MEDIUM** - Historical analysis
|
|
|
|
**Tasks:**
|
|
1. Add comparison API endpoint
|
|
- `GET /api/scans/<id1>/compare/<id2>`
|
|
- Return diff of ports, services, certificates
|
|
- Highlight added, removed, changed items
|
|
|
|
2. Add comparison UI
|
|
- "Compare with Previous" button on scan detail page
|
|
- Side-by-side comparison view
|
|
- Color coding: green (added), red (removed), yellow (changed)
|
|
|
|
3. Add historical charts to scan detail
|
|
- Port count trend for this site
|
|
- Service version changes over time
|
|
- Certificate expiry timeline
|
|
|
|
4. Add drift detection
|
|
- Calculate "drift score" (how much changed)
|
|
- Highlight unexpected changes
|
|
- Link to alert creation
|
|
|
|
**Comparison Algorithm:**
|
|
```python
|
|
def compare_scans(scan1_id, scan2_id):
|
|
scan1 = ScanService.get_scan(scan1_id)
|
|
scan2 = ScanService.get_scan(scan2_id)
|
|
|
|
# Extract port lists
|
|
ports1 = set(p['port'] for site in scan1['sites']
|
|
for ip in site['ips'] for p in ip['ports'])
|
|
ports2 = set(p['port'] for site in scan2['sites']
|
|
for ip in site['ips'] for p in ip['ports'])
|
|
|
|
return {
|
|
'added_ports': list(ports2 - ports1),
|
|
'removed_ports': list(ports1 - ports2),
|
|
'unchanged_ports': list(ports1 & ports2),
|
|
'services_changed': _compare_services(scan1, scan2),
|
|
'certificates_changed': _compare_certs(scan1, scan2)
|
|
}
|
|
```
|
|
|
|
**Deliverables:**
|
|
- Scan comparison API endpoint
|
|
- Comparison UI with diff view
|
|
- Historical charts on scan detail
|
|
- Drift score calculation
|
|
|
|
**Testing:**
|
|
- Unit: Comparison algorithm with various scenarios
|
|
- Manual: Compare two scans with differences
|
|
- Edge: Compare same scan, compare non-existent scans
|
|
|
|
---
|
|
|
|
### Step 8: Testing & Documentation ✅ (Days 13-14)
|
|
**Priority: HIGH** - Quality assurance
|
|
|
|
**Tasks:**
|
|
1. Complete test coverage
|
|
- Verify all 80+ tests passing
|
|
- Add missing edge cases
|
|
- Test concurrent schedule operations
|
|
- Test cron execution accuracy
|
|
|
|
2. Create manual testing checklist
|
|
- Create `docs/ai/MANUAL_TESTING_PHASE3.md`
|
|
- 30+ manual tests for all Phase 3 features
|
|
- Include schedule creation, editing, execution
|
|
- Include chart rendering verification
|
|
|
|
3. Update documentation
|
|
- Update `README.md` with Phase 3 features
|
|
- Create `docs/ai/PHASE3_COMPLETE.md`
|
|
- Update `docs/ai/ROADMAP.md` with Phase 3 completion
|
|
|
|
4. Run full regression tests
|
|
- Verify all Phase 2 tests still pass
|
|
- Verify no breaking changes
|
|
- Test backward compatibility
|
|
|
|
5. Performance testing
|
|
- Test with 50+ schedules
|
|
- Test with 1000+ scans (chart performance)
|
|
- Verify APScheduler handles many cron jobs
|
|
|
|
**Documentation Updates:**
|
|
|
|
**README.md additions:**
|
|
- Scheduled scans section
|
|
- Chart.js visualizations
|
|
- Schedule management UI
|
|
|
|
**PHASE3_COMPLETE.md sections:**
|
|
- What was delivered
|
|
- Success criteria checklist
|
|
- Known limitations
|
|
- Testing results
|
|
- Performance metrics
|
|
- Lessons learned
|
|
|
|
**Deliverables:**
|
|
- All 80+ tests passing
|
|
- Manual testing checklist created
|
|
- Documentation updated
|
|
- Phase 3 completion summary
|
|
- Performance validated
|
|
|
|
**Testing:**
|
|
- Full test suite: `pytest tests/ -v`
|
|
- Manual: Complete manual testing checklist
|
|
- Performance: Load test with many schedules
|
|
- Regression: Verify Phase 2 features unchanged
|
|
|
|
---
|
|
|
|
## Dependencies & Prerequisites
|
|
|
|
### Python Packages
|
|
|
|
**Add to `requirements-web.txt`:**
|
|
```
|
|
croniter==2.0.1 # Cron expression parsing
|
|
```
|
|
|
|
**Already Present (from Phase 2):**
|
|
- Flask==3.0.0
|
|
- SQLAlchemy==2.0.23
|
|
- APScheduler==3.10.4
|
|
- Flask-Login==0.6.3
|
|
- Bootstrap 5 (CDN)
|
|
|
|
### Frontend Libraries
|
|
|
|
**Add to base.html:**
|
|
- Chart.js 4.4.0 (CDN): `https://cdn.jsdelivr.net/npm/chart.js@4.4.0/dist/chart.umd.min.js`
|
|
|
|
**Already Present:**
|
|
- Bootstrap 5.3.0 (CDN)
|
|
- Bootstrap Icons 1.11.0 (CDN)
|
|
|
|
### System Requirements
|
|
|
|
- Python 3.12+ (same as Phase 2)
|
|
- Docker and Docker Compose
|
|
- SQLite3 with WAL mode (already configured)
|
|
|
|
### Configuration Files
|
|
|
|
**No new configuration needed** - Uses existing:
|
|
- `.env` file (from Phase 2)
|
|
- `docker-compose-web.yml` (no changes)
|
|
- Database schema (Schedule table already exists)
|
|
|
|
---
|
|
|
|
## Testing Approach
|
|
|
|
### Unit Tests
|
|
|
|
**New Test Files:**
|
|
1. `tests/test_schedule_service.py` (450 lines, 18+ tests)
|
|
2. `tests/test_charts.py` (200 lines, 8+ tests)
|
|
|
|
**Coverage Target:** >80% for new code
|
|
|
|
**Test Strategy:**
|
|
- Mock database calls for service tests
|
|
- Test cron validation with valid/invalid expressions
|
|
- Test next run calculations with various timezones
|
|
- Test edge cases (empty data, null values)
|
|
|
|
### Integration Tests
|
|
|
|
**New Test Files:**
|
|
1. `tests/test_schedule_api.py` (500 lines, 22+ tests)
|
|
|
|
**Test Strategy:**
|
|
- Use test database (separate from production)
|
|
- Test complete workflows (create → update → execute → delete)
|
|
- Test API authentication
|
|
- Verify APScheduler integration
|
|
|
|
### Manual Testing
|
|
|
|
**Create:** `docs/ai/MANUAL_TESTING_PHASE3.md`
|
|
|
|
**Tests to Include:**
|
|
1. CSS/Styling (5 tests)
|
|
- Verify white rows fixed
|
|
- Verify dark theme consistent
|
|
- Verify Chart.js dark theme
|
|
|
|
2. Schedules (15 tests)
|
|
- Create schedule via UI
|
|
- Edit schedule
|
|
- Enable/disable toggle
|
|
- Manual trigger
|
|
- Delete schedule
|
|
- Cron validation
|
|
- Next run display
|
|
|
|
3. Dashboard (5 tests)
|
|
- Verify charts render
|
|
- Verify schedule widget
|
|
- Verify stats accurate
|
|
|
|
4. Scheduled Execution (5 tests)
|
|
- Create schedule with "* * * * *"
|
|
- Wait 1 minute, verify scan created
|
|
- Verify timestamps updated
|
|
- Disable schedule, verify stops
|
|
- Enable schedule, verify resumes
|
|
|
|
### Performance Tests
|
|
|
|
**Scenarios:**
|
|
1. Load 100 schedules, verify startup time < 5 seconds
|
|
2. Generate chart with 1000 scans, verify response < 2 seconds
|
|
3. List 500 schedules with pagination, verify response < 500ms
|
|
4. Concurrent schedule creation (10 simultaneous), verify no conflicts
|
|
|
|
---
|
|
|
|
## Potential Challenges & Solutions
|
|
|
|
### Challenge 1: Cron Expression Validation
|
|
|
|
**Problem:** Users may enter invalid cron expressions causing APScheduler errors.
|
|
|
|
**Impact:** High - Could crash scheduler
|
|
|
|
**Solution:**
|
|
- Validate cron expressions with croniter before saving
|
|
- Show human-readable description to user
|
|
- Preview next 5 run times
|
|
- Provide quick templates (daily, weekly, monthly)
|
|
- Catch APScheduler exceptions and log errors
|
|
|
|
**Implementation:**
|
|
```python
|
|
from croniter import croniter
|
|
|
|
def validate_cron_expression(cron_expr):
|
|
try:
|
|
# Test if cron is valid
|
|
cron = croniter(cron_expr, datetime.utcnow())
|
|
cron.get_next() # Try to get next run
|
|
return True, None
|
|
except Exception as e:
|
|
return False, f"Invalid cron expression: {str(e)}"
|
|
```
|
|
|
|
### Challenge 2: Timezone Handling
|
|
|
|
**Problem:** Cron expressions are timezone-dependent. User expects "2 AM" in their local timezone, but server runs in UTC.
|
|
|
|
**Impact:** Medium - Schedules run at unexpected times
|
|
|
|
**Solution:**
|
|
- Store all timestamps in UTC (already done in Phase 2)
|
|
- Display times in user's browser timezone (JavaScript)
|
|
- Document that cron expressions are in UTC
|
|
- Phase 4 enhancement: Add timezone selector per schedule
|
|
|
|
**Implementation:**
|
|
```javascript
|
|
// Display in user's local timezone
|
|
const utcTime = new Date(schedule.next_run + 'Z'); // Parse as UTC
|
|
document.getElementById('next-run').textContent = utcTime.toLocaleString();
|
|
```
|
|
|
|
### Challenge 3: Concurrent Schedule Execution
|
|
|
|
**Problem:** If two schedules have same cron expression, both may trigger at exact same time, potentially overwhelming system.
|
|
|
|
**Impact:** Medium - Resource contention
|
|
|
|
**Solution:**
|
|
- APScheduler already handles concurrency (ThreadPoolExecutor)
|
|
- Max 3 concurrent scans (configured in Phase 2)
|
|
- If more than 3 scans queued, they wait in queue
|
|
- Add warning in UI if > 5 schedules have same cron
|
|
|
|
**Implementation:**
|
|
```python
|
|
# In SchedulerService
|
|
self.scheduler = BackgroundScheduler(
|
|
executors={
|
|
'default': ThreadPoolExecutor(3) # Max 3 concurrent
|
|
},
|
|
job_defaults={
|
|
'coalesce': False, # Don't combine missed runs
|
|
'max_instances': 1 # Only 1 instance per job
|
|
}
|
|
)
|
|
```
|
|
|
|
### Challenge 4: Missed Schedule Execution
|
|
|
|
**Problem:** If app is down during scheduled time, scan is missed. When should it run after restart?
|
|
|
|
**Impact:** Medium - Data gaps
|
|
|
|
**Solution:**
|
|
- APScheduler has `misfire_grace_time` setting
|
|
- Set to 3600 seconds (1 hour)
|
|
- If app down < 1 hour, scan will run when app starts
|
|
- If app down > 1 hour, skip and wait for next scheduled time
|
|
- Log missed schedules for admin review
|
|
|
|
**Implementation:**
|
|
```python
|
|
self.scheduler = BackgroundScheduler(
|
|
job_defaults={
|
|
'misfire_grace_time': 3600 # Run if missed within 1 hour
|
|
}
|
|
)
|
|
```
|
|
|
|
### Challenge 5: Chart.js Dark Theme
|
|
|
|
**Problem:** Default Chart.js colors don't match dark theme (white background, black text).
|
|
|
|
**Impact:** Low - Aesthetic only
|
|
|
|
**Solution:**
|
|
- Set Chart.js global defaults in base.html
|
|
- Override colors, fonts, borders
|
|
- Test all chart types (line, bar, pie)
|
|
|
|
**Implementation:**
|
|
```javascript
|
|
Chart.defaults.color = '#e2e8f0'; // Light text
|
|
Chart.defaults.borderColor = '#334155'; // Slate borders
|
|
Chart.defaults.backgroundColor = '#1e293b'; // Dark background
|
|
Chart.defaults.font.family = "'Inter', sans-serif";
|
|
```
|
|
|
|
### Challenge 6: Schedule Deletion with Active Scans
|
|
|
|
**Problem:** User deletes schedule while scan from that schedule is running. Should scan be cancelled?
|
|
|
|
**Impact:** Low - Edge case
|
|
|
|
**Solution:**
|
|
- Do NOT cancel running scans
|
|
- Only prevent future executions
|
|
- Running scan completes normally
|
|
- Database foreign key allows null schedule_id (schedule deleted but scan remains)
|
|
- Document behavior in UI ("Active scans will complete")
|
|
|
|
**Implementation:**
|
|
```python
|
|
def delete_schedule(self, schedule_id):
|
|
# Remove from APScheduler (prevents future runs)
|
|
self.scheduler.remove_scheduled_scan(schedule_id)
|
|
|
|
# Delete from database (scans remain, schedule_id becomes null)
|
|
schedule = self.db.query(Schedule).get(schedule_id)
|
|
self.db.delete(schedule)
|
|
self.db.commit()
|
|
|
|
logger.info(f"Schedule {schedule_id} deleted. Active scans will complete.")
|
|
```
|
|
|
|
### Challenge 7: CSS Extraction Breaking Existing Styles
|
|
|
|
**Problem:** Moving inline CSS to external file might cause selector specificity issues.
|
|
|
|
**Impact:** Medium - Visual regressions
|
|
|
|
**Solution:**
|
|
- Extract CSS exactly as-is first
|
|
- Test all pages after extraction
|
|
- Only then refactor/improve
|
|
- Use CSS variables for consistency
|
|
- Keep !important flags where needed for dynamic rows
|
|
|
|
**Testing:**
|
|
- Manual: View every page (dashboard, scans, scan detail, login, schedules)
|
|
- Compare: Screenshots before and after extraction
|
|
- Verify: Hover effects, active states, responsive behavior
|
|
|
|
---
|
|
|
|
## Success Criteria
|
|
|
|
Phase 3 is **COMPLETE** when all criteria are met:
|
|
|
|
### Bug Fixes
|
|
- [ ] White row bug fixed in dashboard.html
|
|
- [ ] White row bug fixed in scans.html
|
|
- [ ] White row bug fixed in scan_detail.html
|
|
- [ ] All tables use dark theme consistently
|
|
- [ ] Hover effects work correctly
|
|
|
|
### CSS & Styling
|
|
- [ ] CSS extracted to web/static/css/styles.css
|
|
- [ ] base.html links to external CSS
|
|
- [ ] Chart.js added and configured for dark theme
|
|
- [ ] CSS variables used for colors
|
|
- [ ] No visual regressions on any page
|
|
|
|
### Schedule Service
|
|
- [ ] ScheduleService fully implemented (~400 lines)
|
|
- [ ] All CRUD methods working
|
|
- [ ] Cron validation with croniter
|
|
- [ ] Next run time calculation accurate
|
|
- [ ] 18+ unit tests passing
|
|
|
|
### Schedules API
|
|
- [ ] All 6 API endpoints implemented
|
|
- [ ] GET /api/schedules lists with pagination
|
|
- [ ] POST /api/schedules creates and validates
|
|
- [ ] PUT /api/schedules/<id> updates and reloads
|
|
- [ ] DELETE /api/schedules/<id> removes schedule
|
|
- [ ] POST /api/schedules/<id>/trigger manually triggers
|
|
- [ ] 22+ integration tests passing
|
|
|
|
### Schedule UI
|
|
- [ ] schedules.html lists all schedules
|
|
- [ ] schedule_create.html creates new schedules
|
|
- [ ] schedule_edit.html edits existing schedules
|
|
- [ ] Enable/disable toggle works
|
|
- [ ] Manual trigger button works
|
|
- [ ] Cron validation prevents invalid expressions
|
|
- [ ] Human-readable cron description shown
|
|
- [ ] Next 5 run times previewed
|
|
|
|
### Scheduled Execution
|
|
- [ ] Schedules load on app startup
|
|
- [ ] Cron jobs added to APScheduler
|
|
- [ ] Scans execute automatically at scheduled time
|
|
- [ ] last_run and next_run timestamps updated
|
|
- [ ] Scans have triggered_by='scheduled'
|
|
- [ ] Enable/disable adds/removes jobs
|
|
|
|
### Dashboard Enhancements
|
|
- [ ] Schedule widget shows next scheduled scans
|
|
- [ ] Trending chart displays scan activity (last 30 days)
|
|
- [ ] Chart.js renders with dark theme
|
|
- [ ] Stats accurate (total, enabled, next run)
|
|
|
|
### Scan Comparison
|
|
- [ ] Comparison API endpoint returns diffs
|
|
- [ ] "Compare with Previous" button on scan detail
|
|
- [ ] Side-by-side comparison view
|
|
- [ ] Color coding for added/removed/changed
|
|
|
|
### Testing
|
|
- [ ] All 80+ new tests passing
|
|
- [ ] All Phase 2 tests still passing
|
|
- [ ] Manual testing checklist completed
|
|
- [ ] Performance acceptable (50+ schedules, 1000+ scans)
|
|
|
|
### Documentation
|
|
- [ ] README.md updated with Phase 3 features
|
|
- [ ] PHASE3_COMPLETE.md created
|
|
- [ ] ROADMAP.md updated with Phase 3 completion
|
|
- [ ] Manual testing checklist created
|
|
|
|
---
|
|
|
|
## Migration Path
|
|
|
|
### From Phase 2 to Phase 3
|
|
|
|
**No Breaking Changes:**
|
|
- All Phase 2 APIs remain functional
|
|
- Database schema unchanged (Schedule table already exists)
|
|
- Existing scans, users, settings unchanged
|
|
- Docker deployment compatible
|
|
|
|
**Additions:**
|
|
- New schedules API endpoints (stubs → implementations)
|
|
- New schedule management UI
|
|
- New dashboard charts
|
|
- External CSS file (improves maintainability)
|
|
- croniter dependency
|
|
|
|
**Migration Steps:**
|
|
1. Pull latest code from phase3 branch
|
|
2. Install new dependency: `pip install croniter==2.0.1`
|
|
3. No database migration needed (Schedule table already exists from Phase 1)
|
|
4. Rebuild Docker image: `docker-compose -f docker-compose-web.yml build`
|
|
5. Restart services: `docker-compose -f docker-compose-web.yml restart`
|
|
6. Verify schedules load on startup (check logs)
|
|
|
|
**Backward Compatibility:**
|
|
- CLI scanner: Continues to work standalone (unchanged)
|
|
- Existing scans: All viewable in web UI
|
|
- Phase 2 features: All functional (no regressions)
|
|
- API clients: All Phase 2 endpoints unchanged
|
|
|
|
---
|
|
|
|
## Estimated Timeline
|
|
|
|
**Total Duration:** 14 working days (2 weeks)
|
|
|
|
### Week 1: Backend Foundation
|
|
- **Day 1:** Fix styling issues & CSS refactor
|
|
- Extract CSS to external file
|
|
- Fix white row bug
|
|
- Add Chart.js
|
|
|
|
- **Days 2-3:** ScheduleService implementation
|
|
- Create ScheduleService class
|
|
- Implement CRUD methods
|
|
- Add cron validation
|
|
- Write unit tests
|
|
|
|
- **Days 4-5:** Schedules API endpoints
|
|
- Implement all 6 endpoints
|
|
- Add error handling
|
|
- Write integration tests
|
|
|
|
- **Days 6-7:** Schedule Management UI
|
|
- Create schedules list page
|
|
- Create schedule create form
|
|
- Create schedule edit form
|
|
- Add web routes
|
|
|
|
### Week 2: Enhancements & Polish
|
|
- **Days 8-9:** Enhanced dashboard with charts
|
|
- Add trending charts
|
|
- Add schedule widget
|
|
- Create stats API endpoints
|
|
|
|
- **Day 10:** Scheduler integration
|
|
- Complete scheduled execution
|
|
- Load schedules on startup
|
|
- Test cron execution
|
|
|
|
- **Days 11-12:** Scan comparison features
|
|
- Implement comparison algorithm
|
|
- Create comparison UI
|
|
- Add historical charts
|
|
|
|
- **Days 13-14:** Testing & documentation
|
|
- Complete test suite
|
|
- Manual testing
|
|
- Update documentation
|
|
- Phase 3 completion summary
|
|
|
|
### Critical Path
|
|
|
|
**Must Complete in Order:**
|
|
1. Day 1: CSS extraction and white row fix (prerequisite for all UI work)
|
|
2. Days 2-3: ScheduleService (prerequisite for API)
|
|
3. Days 4-5: Schedules API (prerequisite for UI)
|
|
4. Days 6-7: Schedule UI (can proceed once API done)
|
|
|
|
**Can Proceed in Parallel:**
|
|
- Dashboard charts (Days 8-9) can start any time after Day 1
|
|
- Scan comparison (Days 11-12) independent of schedules
|
|
- Testing (Days 13-14) can start earlier for completed components
|
|
|
|
---
|
|
|
|
## Key Design Decisions
|
|
|
|
### Decision 1: Cron Expression Library
|
|
|
|
**Choice:** croniter
|
|
|
|
**Alternatives Considered:**
|
|
- APScheduler's built-in cron parsing
|
|
- python-crontab
|
|
- Manual parsing with regex
|
|
|
|
**Rationale:**
|
|
- croniter is lightweight and focused
|
|
- Accurate next run calculation
|
|
- Handles edge cases (leap years, DST)
|
|
- Easy validation and error messages
|
|
- Well-maintained (active development)
|
|
|
|
**Trade-offs:**
|
|
- ✅ Accurate and reliable
|
|
- ✅ Simple API
|
|
- ✅ Good error messages
|
|
- ❌ Another dependency (acceptable - small library)
|
|
|
|
### Decision 2: External CSS vs Inline
|
|
|
|
**Choice:** Extract to external CSS file
|
|
|
|
**Alternatives Considered:**
|
|
- Keep inline CSS in base.html
|
|
- Use CSS-in-JS
|
|
- Use CSS framework (Tailwind)
|
|
|
|
**Rationale:**
|
|
- Better maintainability (single source of truth)
|
|
- Easier to customize theme
|
|
- Browser caching (performance)
|
|
- Separation of concerns
|
|
- Easier to test (CSS linting)
|
|
|
|
**Trade-offs:**
|
|
- ✅ Maintainable and organized
|
|
- ✅ Better performance (caching)
|
|
- ✅ Easier theme customization
|
|
- ❌ One more HTTP request (minimal impact with HTTP/2)
|
|
|
|
### Decision 3: Chart.js for Visualizations
|
|
|
|
**Choice:** Chart.js 4.4.0
|
|
|
|
**Alternatives Considered:**
|
|
- Plotly.js (more powerful, heavier)
|
|
- D3.js (maximum flexibility, steeper learning curve)
|
|
- ApexCharts (modern, but less documented)
|
|
- Server-side rendering (matplotlib → image)
|
|
|
|
**Rationale:**
|
|
- Lightweight (< 200KB)
|
|
- Simple API
|
|
- Good dark theme support
|
|
- Responsive by default
|
|
- Well-documented
|
|
- Active community
|
|
|
|
**Trade-offs:**
|
|
- ✅ Easy to implement
|
|
- ✅ Good performance
|
|
- ✅ Responsive
|
|
- ❌ Less powerful than Plotly (sufficient for Phase 3)
|
|
- ❌ Not as customizable as D3 (acceptable trade-off)
|
|
|
|
### Decision 4: Schedule Deletion Behavior
|
|
|
|
**Choice:** Delete schedule but keep associated scans
|
|
|
|
**Alternatives Considered:**
|
|
- Cascade delete (delete scans too)
|
|
- Soft delete (mark schedule as deleted)
|
|
- Prevent deletion if scans exist
|
|
|
|
**Rationale:**
|
|
- Scans are historical data (valuable)
|
|
- Schedule is just a template (can be recreated)
|
|
- User might want to delete schedule but keep history
|
|
- Deleting scans would cause data loss
|
|
|
|
**Trade-offs:**
|
|
- ✅ Preserves historical data
|
|
- ✅ Flexible (can recreate schedule)
|
|
- ❌ Schedule reference in scans becomes null (acceptable)
|
|
|
|
### Decision 5: Timezone Handling
|
|
|
|
**Choice:** Store all times in UTC, display in user's browser timezone
|
|
|
|
**Alternatives Considered:**
|
|
- Store in user's timezone
|
|
- Add timezone field to Schedule model
|
|
- Use server's local timezone
|
|
|
|
**Rationale:**
|
|
- UTC is standard for databases
|
|
- Avoids DST issues
|
|
- Browser automatically converts to local time
|
|
- Consistent with Phase 2 approach
|
|
- Simple and reliable
|
|
|
|
**Trade-offs:**
|
|
- ✅ No DST issues
|
|
- ✅ Consistent across users
|
|
- ✅ Simple implementation
|
|
- ❌ Cron expressions in UTC (document clearly)
|
|
- ❌ Phase 4 enhancement: per-schedule timezone
|
|
|
|
### Decision 6: Scan Comparison Scope
|
|
|
|
**Choice:** Start with port and service comparison, defer certificate comparison to Phase 4
|
|
|
|
**Alternatives Considered:**
|
|
- Full comparison (ports, services, certs, TLS)
|
|
- Only port comparison
|
|
- Advanced diff with visualization
|
|
|
|
**Rationale:**
|
|
- Ports and services are most important for drift detection
|
|
- Certificates are complex (multiple per scan, expiry dates)
|
|
- Keep Phase 3 scope manageable
|
|
- Certificate comparison better suited for Phase 4 (alerts)
|
|
|
|
**Trade-offs:**
|
|
- ✅ Achievable in 2 days
|
|
- ✅ Most valuable comparison (ports/services)
|
|
- ❌ Not comprehensive (acceptable for Phase 3)
|
|
- ✅ Can enhance in Phase 4
|
|
|
|
---
|
|
|
|
## Documentation Deliverables
|
|
|
|
### 1. Updated README.md
|
|
|
|
**New Sections:**
|
|
- Scheduled Scans
|
|
- How to create schedules
|
|
- Cron expression syntax
|
|
- Enable/disable schedules
|
|
- Dashboard Enhancements
|
|
- Trending charts
|
|
- Schedule widget
|
|
- Scan Comparison
|
|
- Compare feature
|
|
- Historical analysis
|
|
|
|
### 2. PHASE3_COMPLETE.md
|
|
|
|
**Sections:**
|
|
- What was delivered (8 steps)
|
|
- Success criteria checklist
|
|
- Files created/modified summary
|
|
- Testing results
|
|
- Performance metrics
|
|
- Known limitations
|
|
- Lessons learned
|
|
- What's next (Phase 4)
|
|
|
|
### 3. MANUAL_TESTING_PHASE3.md
|
|
|
|
**Sections:**
|
|
- Prerequisites
|
|
- CSS/Styling tests (5 tests)
|
|
- Schedule management tests (15 tests)
|
|
- Dashboard tests (5 tests)
|
|
- Scheduled execution tests (5 tests)
|
|
- Scan comparison tests (3 tests)
|
|
- Performance tests (3 tests)
|
|
- Test results summary
|
|
|
|
### 4. Updated ROADMAP.md
|
|
|
|
**Changes:**
|
|
- Mark Phase 3 as COMPLETE ✅
|
|
- Update progress overview
|
|
- Add Phase 3 deliverables section
|
|
- Update success criteria
|
|
- Update changelog
|
|
- Set next review for Phase 4
|
|
|
|
---
|
|
|
|
**End of Phase 3 Plan**
|
|
|
|
This plan will be followed during Phase 3 implementation. Upon completion, `PHASE3_COMPLETE.md` will summarize actual implementation, challenges encountered, and lessons learned.
|