sneakygeek/SneakyScan
2
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Fix schedule management and update documentation for database-backed configs

Browse Source 
This commit addresses multiple issues with schedule management and updates
  documentation to reflect the transition from YAML-based to database-backed
  configuration system.

  **Documentation Updates:**
  - Update DEPLOYMENT.md to remove all references to YAML config files
  - Document that all configurations are now stored in SQLite database
  - Update API examples to use config IDs instead of YAML filenames
  - Remove configs directory from backup/restore procedures
  - Update volume management section to reflect database-only storage

  **Cron Expression Handling:**
  - Add comprehensive documentation for APScheduler cron format conversion
  - Document that from_crontab() accepts standard format (Sunday=0) and converts automatically
  - Add validate_cron_expression() helper method with detailed error messages
  - Include helpful hints for day-of-week field errors in validation
  - Fix all deprecated datetime.utcnow() calls, replace with datetime.now(timezone.utc)

  **Timezone-Aware DateTime Fixes:**
  - Fix "can't subtract offset-naive and offset-aware datetimes" error
  - Add timezone awareness to croniter.get_next() return values
  - Make _get_relative_time() defensive to handle both naive and aware datetimes
  - Ensure all datetime comparisons use timezone-aware objects

  **Schedule Edit UI Fixes:**
  - Fix JavaScript error "Cannot set properties of null (setting 'value')"
  - Change reference from non-existent 'config-id' to correct 'config-file' element
  - Add config_name field to schedule API responses for better UX
  - Eagerly load Schedule.config relationship using joinedload()
  - Fix AttributeError: use schedule.config.title instead of .name
  - Display config title and ID in schedule edit form

  **Technical Details:**
  - app/web/services/schedule_service.py: 6 datetime.utcnow() fixes, validation enhancements
  - app/web/services/scheduler_service.py: Documentation, validation, timezone fixes
  - app/web/templates/schedule_edit.html: JavaScript element reference fix
  - docs/DEPLOYMENT.md: Complete rewrite of config management sections

  Fixes scheduling for Sunday at midnight (cron: 0 0 * * 0)
  Fixes schedule edit page JavaScript errors
  Improves user experience with config title display
This commit is contained in:
Phillip Tarrant
2025-11-24 12:53:06 -06:00
parent f24bd11dfd
commit 5e3a70f837
7 changed files with 276 additions and 139 deletions
Download Patch File Download Diff File Expand all files Collapse all files

203
docs/DEPLOYMENT.md
View File

@@ -24,10 +24,10 @@ SneakyScanner is deployed as a Docker container running a Flask web application
**Architecture:**
- **Web Application**: Flask app on port 5000 with modern web UI
- **Database**: SQLite (persisted to volume)
- **Database**: SQLite (persisted to volume) - stores all configurations, scan results, and settings
- **Background Jobs**: APScheduler for async scan execution
- **Scanner**: masscan, nmap, sslyze, Playwright
- **Config Creator**: Web-based CIDR-to-YAML configuration builder
- **Config Management**: Database-backed configuration system managed entirely via web UI
- **Scheduling**: Cron-based scheduled scans with dashboard management
---
@@ -143,6 +143,13 @@ docker compose -f docker-compose-standalone.yml up
SneakyScanner is configured via environment variables. The recommended approach is to use a `.env` file.
**UDP Port Scanning**
- UDP Port scanning is disabled by default.
- You can turn it on via the .env variable.
- By Default, UDP port scanning only scans the top 20 ports, for convenience I have included the NMAP top 100 UDP ports as well.
#### Creating Your .env File
```bash
@@ -160,6 +167,7 @@ python3 -c "from cryptography.fernet import Fernet; print('SNEAKYSCANNER_ENCRYPT
nano .env
```
#### Key Configuration Options
| Variable | Description | Default | Required |
@@ -190,54 +198,30 @@ The application needs these directories (created automatically by Docker):
```bash
# Verify directories exist
ls -la configs/ data/ output/ logs/
ls -la data/ output/ logs/
# If missing, create them
mkdir -p configs data output logs
mkdir -p data output logs
```
### Step 2: Configure Scan Targets
You can create scan configurations in two ways:
After starting the application, create scan configurations using the web UI:
**Option A: Using the Web UI (Recommended - Phase 4 Feature)**
**Creating Configurations via Web UI**
1. Navigate to **Configs** in the web interface
2. Click **"Create New Config"**
3. Use the CIDR-based config creator for quick setup:
3. Use the form-based config creator:
- Enter site name
- Enter CIDR range (e.g., `192.168.1.0/24`)
- Select expected ports from dropdowns
- Click **"Generate Config"**
4. Or use the **YAML Editor** for advanced configurations
5. Save and use immediately in scans or schedules
- Select expected TCP/UDP ports from dropdowns
- Optionally enable ping checks
4. Click **"Save Configuration"**
5. Configuration is saved to database and immediately available for scans and schedules
**Option B: Manual YAML Files**
**Note**: All configurations are stored in the database, not as files. This provides better reliability, easier backup, and seamless management through the web interface.
Create YAML configuration files manually in the `configs/` directory:
```bash
# Example configuration
cat > configs/my-network.yaml <<EOF
title: "My Network Infrastructure"
sites:
- name: "Web Servers"
cidr: "192.168.1.0/24" # Scan entire subnet
expected_ports:
- port: 80
protocol: tcp
service: "http"
- port: 443
protocol: tcp
service: "https"
- port: 22
protocol: tcp
service: "ssh"
ping_expected: true
EOF
```
**Note**: Phase 4 introduced a powerful config creator in the web UI that makes it easy to generate configs from CIDR ranges without manually editing YAML.
### Step 3: Build Docker Image
@@ -389,38 +373,37 @@ The dashboard provides a central view of your scanning activity:
- **Trend Charts**: Port count trends over time using Chart.js
- **Quick Actions**: Buttons to run scans, create configs, manage schedules
### Managing Scan Configurations (Phase 4)
### Managing Scan Configurations
All scan configurations are stored in the database and managed entirely through the web interface.
**Creating Configs:**
1. Navigate to **Configs** → **Create New Config**
2. **CIDR Creator Mode**:
2. Fill in the configuration form:
- Enter site name (e.g., "Production Servers")
- Enter CIDR range (e.g., `192.168.1.0/24`)
- Select expected TCP/UDP ports from dropdowns
- Click **"Generate Config"** to create YAML
3. **YAML Editor Mode**:
- Switch to editor tab for advanced configurations
- Syntax highlighting with line numbers
- Validate YAML before saving
- Enable/disable ping checks
3. Click **"Save Configuration"**
4. Configuration is immediately stored in database and available for use
**Editing Configs:**
1. Navigate to **Configs** → Select config
1. Navigate to **Configs** → Select config from list
2. Click **"Edit"** button
3. Make changes in YAML editor
4. Save changes (validates YAML automatically)
3. Modify any fields in the configuration form
4. Click **"Save Changes"** to update database
**Uploading Configs:**
1. Navigate to **Configs** → **Upload**
2. Select YAML file from your computer
3. File is validated and saved to `configs/` directory
**Downloading Configs:**
- Click **"Download"** button next to any config
- Saves YAML file to your local machine
**Viewing Configs:**
- Navigate to **Configs** page to see all saved configurations
- Each config shows site name, CIDR range, and expected ports
- Click on any config to view full details
**Deleting Configs:**
- Click **"Delete"** button
- Click **"Delete"** button next to any config
- **Warning**: Cannot delete configs used by active schedules
- Deletion removes the configuration from the database permanently
**Note**: All configurations are database-backed, providing automatic backups when you backup the database file.
### Running Scans
@@ -477,12 +460,11 @@ SneakyScanner uses several mounted volumes for data persistence:
| Volume | Container Path | Purpose | Important? |
|--------|----------------|---------|------------|
| `./configs` | `/app/configs` | Scan configuration files (managed via web UI) | Yes |
| `./data` | `/app/data` | SQLite database (contains all scan history) | **Critical** |
| `./data` | `/app/data` | SQLite database (contains configurations, scan history, settings) | **Critical** |
| `./output` | `/app/output` | Scan results (JSON, HTML, ZIP, screenshots) | Yes |
| `./logs` | `/app/logs` | Application logs (rotating file handler) | No |
**Note**: As of Phase 4, the `./configs` volume is read-write to support the web-based config creator and editor. The web UI can now create, edit, and delete configuration files directly.
**Note**: All scan configurations are stored in the SQLite database (`./data/sneakyscanner.db`). There is no separate configs directory or YAML files. Backing up the database file ensures all your configurations are preserved.
### Backing Up Data
@@ -490,23 +472,22 @@ SneakyScanner uses several mounted volumes for data persistence:
# Create backup directory
mkdir -p backups/$(date +%Y%m%d)
# Backup database
# Backup database (includes all configurations)
cp data/sneakyscanner.db backups/$(date +%Y%m%d)/
# Backup scan outputs
tar -czf backups/$(date +%Y%m%d)/output.tar.gz output/
# Backup configurations
tar -czf backups/$(date +%Y%m%d)/configs.tar.gz configs/
```
**Important**: The database backup includes all scan configurations, settings, schedules, and scan history. No separate configuration file backup is needed.
### Restoring Data
```bash
# Stop application
docker compose -f docker-compose.yml down
# Restore database
# Restore database (includes all configurations)
cp backups/YYYYMMDD/sneakyscanner.db data/
# Restore outputs
@@ -516,6 +497,8 @@ tar -xzf backups/YYYYMMDD/output.tar.gz
docker compose -f docker-compose.yml up -d
```
**Note**: Restoring the database file restores all configurations, settings, schedules, and scan history.
### Cleaning Up Old Scan Results
**Option A: Using the Web UI (Recommended)**
@@ -564,50 +547,52 @@ curl -X POST http://localhost:5000/api/auth/logout \
-b cookies.txt
```
### Config Management (Phase 4)
### Config Management
```bash
# List all configs
curl http://localhost:5000/api/configs \
-b cookies.txt
# Get specific config
curl http://localhost:5000/api/configs/prod-network.yaml \
# Get specific config by ID
curl http://localhost:5000/api/configs/1 \
-b cookies.txt
# Create new config
curl -X POST http://localhost:5000/api/configs \
-H "Content-Type: application/json" \
-d '{
"filename": "test-network.yaml",
"content": "title: Test Network\nsites:\n - name: Test\n cidr: 10.0.0.0/24"
"name": "Test Network",
"cidr": "10.0.0.0/24",
"expected_ports": [
{"port": 80, "protocol": "tcp", "service": "http"},
{"port": 443, "protocol": "tcp", "service": "https"}
],
"ping_expected": true
}' \
-b cookies.txt
# Update config
curl -X PUT http://localhost:5000/api/configs/test-network.yaml \
curl -X PUT http://localhost:5000/api/configs/1 \
-H "Content-Type: application/json" \
-d '{
"content": "title: Updated Test Network\nsites:\n - name: Test Site\n cidr: 10.0.0.0/24"
"name": "Updated Test Network",
"cidr": "10.0.1.0/24"
}' \
-b cookies.txt
# Download config
curl http://localhost:5000/api/configs/test-network.yaml/download \
-b cookies.txt -o test-network.yaml
# Delete config
curl -X DELETE http://localhost:5000/api/configs/test-network.yaml \
curl -X DELETE http://localhost:5000/api/configs/1 \
-b cookies.txt
```
### Scan Management
```bash
# Trigger a scan
# Trigger a scan (using config ID from database)
curl -X POST http://localhost:5000/api/scans \
-H "Content-Type: application/json" \
-d '{"config_id": "/app/configs/prod-network.yaml"}' \
-d '{"config_id": 1}' \
-b cookies.txt
# List all scans
@@ -634,12 +619,12 @@ curl -X DELETE http://localhost:5000/api/scans/123 \
curl http://localhost:5000/api/schedules \
-b cookies.txt
# Create schedule
# Create schedule (using config ID from database)
curl -X POST http://localhost:5000/api/schedules \
-H "Content-Type: application/json" \
-d '{
"name": "Daily Production Scan",
"config_id": "/app/configs/prod-network.yaml",
"config_id": 1,
"cron_expression": "0 2 * * *",
"enabled": true
}' \
@@ -875,24 +860,25 @@ docker compose -f docker-compose.yml logs web | grep -E "(ERROR|Exception|Traceb
docker compose -f docker-compose.yml exec web which masscan nmap
```
### Config Files Not Appearing in Web UI
### Configs Not Appearing in Web UI
**Problem**: Manually created configs don't show up in web interface
**Problem**: Created configs don't show up in web interface
```bash
# Check file permissions (must be readable by web container)
ls -la configs/
# Check database connectivity
docker compose -f docker-compose.yml logs web | grep -i "database"
# Fix permissions if needed
sudo chown -R 1000:1000 configs/
chmod 644 configs/*.yaml
# Verify database file exists and is readable
ls -lh data/sneakyscanner.db
# Verify YAML syntax is valid
docker compose -f docker-compose.yml exec web python3 -c \
"import yaml; yaml.safe_load(open('/app/configs/your-config.yaml'))"
# Check web logs for parsing errors
# Check for errors when creating configs
docker compose -f docker-compose.yml logs web | grep -i "config"
# Try accessing configs via API
curl http://localhost:5000/api/configs -b cookies.txt
# If database is corrupted, check integrity
docker compose -f docker-compose.yml exec web sqlite3 /app/data/sneakyscanner.db "PRAGMA integrity_check;"
```
### Health Check Failing
@@ -979,11 +965,11 @@ server {
# Ensure proper ownership of data directories
sudo chown -R $USER:$USER data/ output/ logs/
# Restrict database file permissions
# Restrict database file permissions (contains configurations and sensitive data)
chmod 600 data/sneakyscanner.db
# Configs should be read-only
chmod 444 configs/*.yaml
# Ensure database directory is writable
chmod 700 data/
```
---
@@ -1051,19 +1037,17 @@ mkdir -p "$BACKUP_DIR"
# Stop application for consistent backup
docker compose -f docker-compose.yml stop web
# Backup database
# Backup database (includes all configurations)
cp data/sneakyscanner.db "$BACKUP_DIR/"
# Backup outputs (last 30 days only)
find output/ -type f -mtime -30 -exec cp --parents {} "$BACKUP_DIR/" \;
# Backup configs
cp -r configs/ "$BACKUP_DIR/"
# Restart application
docker compose -f docker-compose.yml start web
echo "Backup complete: $BACKUP_DIR"
echo "Database backup includes all configurations, settings, and scan history"
```
Make executable and schedule with cron:
@@ -1083,15 +1067,18 @@ crontab -e
# Stop application
docker compose -f docker-compose.yml down
# Restore files
# Restore database (includes all configurations)
cp backups/YYYYMMDD_HHMMSS/sneakyscanner.db data/
cp -r backups/YYYYMMDD_HHMMSS/configs/* configs/
# Restore output files
cp -r backups/YYYYMMDD_HHMMSS/output/* output/
# Start application
docker compose -f docker-compose.yml up -d
```
**Note**: Restoring the database file will restore all configurations, settings, schedules, and scan history from the backup.
---
## Support and Further Reading
@@ -1105,13 +1092,13 @@ docker compose -f docker-compose.yml up -d
## What's New
### Phase 4 (2025-11-17) - Config Creator ✅
- **CIDR-based Config Creator**: Web UI for generating scan configs from CIDR ranges
- **YAML Editor**: Built-in editor with syntax highlighting (CodeMirror)
- **Config Management UI**: List, view, edit, download, and delete configs via web interface
- **Config Upload**: Direct YAML file upload for advanced users
- **REST API**: 7 new config management endpoints
### Phase 4+ (2025-11-17) - Database-Backed Configuration System ✅
- **Database-Backed Configs**: All configurations stored in SQLite database (no YAML files)
- **Web-Based Config Creator**: Form-based UI for creating scan configs from CIDR ranges
- **Config Management UI**: List, view, edit, and delete configs via web interface
- **REST API**: Full config management via RESTful API with database storage
- **Schedule Protection**: Prevents deleting configs used by active schedules
- **Automatic Backups**: Configurations included in database backups
### Phase 3 (2025-11-14) - Dashboard & Scheduling ✅
- **Dashboard**: Summary stats, recent scans, trend charts
@@ -1133,5 +1120,5 @@ docker compose -f docker-compose.yml up -d
---
**Last Updated**: 2025-11-17
**Version**: Phase 4 - Config Creator Complete
**Last Updated**: 2025-11-24
**Version**: Phase 4+ - Database-Backed Configuration System

0
docs/KNOWN_ISSUES.md Normal file
View File