adding punch list for improvement plan

punchlist.md

@@ -0,0 +1,297 @@
+# Phase 0 – Baseline + Decisions (Quick)
+
+**Goals**
+
+* Confirm we’ll keep one Jinja template with palette includes.
+* Choose PDF engine flags and file layout.
+
+**Decisions**
+
+* PDF engine: `wkhtmltopdf` via `subprocess` (wrapper optional).
+* Page size: **Letter** (or A4 if you prefer).
+* Output layout: `/data/output/<timestamp>/<config_slug>/report.pdf`.
+* ZIP per run: `/data/output/reports_<timestamp>.zip`.
+
+**Deliverables**
+
+* Short `DECISIONS.md` capturing above.
+* “Current state” run that still only outputs HTML (for baseline timings).
+
+**Acceptance**
+
+* We can state in one sentence where PDFs will land and how they’re named.
+
+---
+
+# Phase 1 – HTML → PDF Renderer (Additive)
+
+**Goals**
+
+* Convert your already-rendered HTML into a PDF file per scan config.
+
+**Tasks**
+
+* Create `report_renderer.py` with:
+
+  * `render_html(context) -> str` (calls existing Jinja template)
+  * `html_to_pdf(html, out_path, *, page="Letter", margin_mm=12, dpi=96) -> Path`
+* Ensure local assets (e.g., logo) resolve without network.
+* Add `run_repo_scan()` call: after HTML render, write PDF.
+
+**Deliverables**
+
+* One PDF per config alongside existing HTML.
+
+**Acceptance**
+
+* PDFs open cleanly (no missing fonts/boxes).
+* Render time acceptable (<1–2s per average report on your host).
+
+---
+
+# Phase 2 – Bundle + Manifest
+
+**Goals**
+
+* Zip all PDFs for a run and write an audit manifest.
+
+**Tasks**
+
+* Create `report_bundler.py` with:
+
+  * `collect_pdfs(run_dir) -> List[Path]`
+  * `write_manifest(run_info, per_config_stats, out_path) -> Path`
+  * `make_zip(files, out_zip) -> Path`
+* Manifest fields (per file and rollup):
+
+  * `run_id, generated_at, timezone`
+  * Per config: `name, config_slug, total_hosts, ok_hosts, hosts_with_issues, pdf_path, pdf_sha256, duration_ms`
+  * Bundle: `zip_path, zip_size_bytes, pdf_count`
+* Add SHA-256 hashing utility.
+
+**Deliverables**
+
+* `reports_<timestamp>.zip`
+* `manifest_<timestamp>.json`
+
+**Acceptance**
+
+* ZIP opens; manifest accurately lists all PDFs with correct hashes.
+
+---
+
+# Phase 3 – Email Output
+
+**Goals**
+
+* Email either the ZIP (preferred) or top-N PDFs when size overs threshold.
+
+**Tasks**
+
+* Create `report_emailer.py`:
+
+  * `send_reports(summary, files, to, cc, subject, size_limit_mb)`
+  * Compose a concise plain-text body (global counts + path to ZIP).
+  * Size check: if ZIP > limit → attach top N “issue” PDFs, mention storage path for ZIP.
+* Configure SMTP from environment (no secrets in code).
+
+**Deliverables**
+
+* Successful email with attachments on a sample run.
+
+**Acceptance**
+
+* Emails received by `email_to` with intended attachments.
+* Logs include message id / delivery status (if available).
+
+---
+
+# Phase 4 – Unify Templates (Light/Dark via palette)
+
+**Goals**
+
+* One structural template; palette includes for dark/light.
+
+**Tasks**
+
+* Introduce `_palette_dark.j2` / `_palette_light.j2`.
+* Replace hardcoded colors with palette tokens.
+* Keep **inline styles** for email/PDF reliability.
+* Ensure `reporting.dark_mode` drives palette include.
+
+**Deliverables**
+
+* Single `report.html.j2` + two small palette partials.
+
+**Acceptance**
+
+* Dark/Light flips correctly; visual parity with previous separate templates.
+
+---
+
+# Phase 5 – Config Flags & CLI ergonomics
+
+**Goals**
+
+* Make format and emailing behavior configurable per run.
+
+**Tasks**
+
+* New config/env flags:
+
+  * `OUTPUT_HTML=true|false`
+  * `OUTPUT_PDF=true|false`
+  * `BUNDLE_ZIP=true|false`
+  * `EMAIL_ENABLED=true|false`
+  * `EMAIL_SIZE_LIMIT_MB=15`
+* Optional CLI args override envs for ad-hoc runs.
+
+**Deliverables**
+
+* Documented flags in `README` or `DECISIONS.md`.
+
+**Acceptance**
+
+* Toggling flags changes outputs without code edits.
+
+---
+
+# Phase 6 – Logging Overhaul (Structured, Useful)
+
+**Goals**
+
+* Replace scattered prints with consistent, structured logs you can search.
+
+**Tasks**
+
+* Adopt one logger (e.g., `structlog` or `logging` with JSON formatter).
+* Standard fields on every log line:
+
+  * `run_id`, `config_name`, `config_slug`, `phase`, `duration_ms`
+* Log events:
+
+  * Start/end per config, counts (ok/issues), output paths, sizes, hashes.
+  * PDF render timings and failures (with `exc_info=True`).
+  * Bundling: zip path/size; Email: recipients, attachment count, result.
+* Set sane defaults:
+
+  * Level INFO in prod, DEBUG when `DEBUG=1`.
+  * Rotate or daily-split logs; permissions via umask `0027`.
+
+**Deliverables**
+
+* Consistent logs; sample snippet documented in `LOGGING.md`.
+
+**Acceptance**
+
+* You can answer “what happened to corp-wan at 03:05?” from logs alone.
+
+---
+
+# Phase 7 – Docker & Dependencies
+
+**Goals**
+
+* Bake the toolchain into the image cleanly.
+
+**Tasks**
+
+* `apt-get install -y wkhtmltopdf fonts-dejavu-core`
+* Keep pip deps minimal (Jinja2 already present).
+* Health check: small mock HTML → PDF during build or startup test (optional).
+
+**Deliverables**
+
+* Rebuilt image; size acceptable.
+
+**Acceptance**
+
+* Container can render PDFs in a clean runtime (no missing libraries).
+
+---
+
+# Phase 8 – Validation & Load Testing
+
+**Goals**
+
+* Confidence under multiple configs and larger reports.
+
+**Tasks**
+
+* Run with N configs (e.g., 10–20) and capture:
+
+  * Total runtime, average PDF time, largest PDF size.
+* Edge tests:
+
+  * No targets → still produce a PDF with “No hosts”.
+  * All OK vs many Issues.
+  * Long hostnames and port lists (layout wrapping).
+* Verify email size behavior.
+
+**Deliverables**
+
+* Short `QA_NOTES.md` with timings and observations.
+
+**Acceptance**
+
+* No crashes; PDFs legible; email strategy holds at size thresholds.
+
+---
+
+# Phase 9 – Docs & Ops
+
+**Goals**
+
+* Make future you (or anyone) productive in 5 minutes.
+
+**Tasks**
+
+* `README` updates: flow diagram, flags, outputs.
+* `DECISIONS.md` finalized.
+* Sample manifest + how to verify checksums.
+* Runbook snippet: “How to re-send last run’s ZIP.”
+
+**Deliverables**
+
+* Docs committed and discoverable.
+
+**Acceptance**
+
+* You can hand this to a teammate and they’ll ship a report same day.
+
+---
+
+# Phase 10 – Nice-to-Haves / Backlog
+
+* “Issues-only” PDF variant (short brief for exec/IT).
+* Per-config S3 upload (instead of email attachments).
+* HTML preview endpoint for a single report (if you add a tiny web UI).
+* Retention policy job: purge artifacts >30 days.
+* Watermark/status banner for partial/incomplete scans.
+* Per-run metrics export (Prometheus counters/gauges).
+
+---
+
+## Global Punch List (copy/paste into your tracker)
+
+* [ ] Phase 0: Lock decisions (page size, paths, file naming)
+* [ ] Phase 1: Add renderer module and wire PDF creation in `run_repo_scan()`
+* [ ] Phase 2: Add bundler + manifest (with SHA-256)
+* [ ] Phase 3: Add email sender with size-aware attachment strategy
+* [ ] Phase 4: Merge templates; add palette partials; test light/dark
+* [ ] Phase 5: Add env/CLI flags for outputs & email
+* [ ] Phase 6: Overhaul logging (structured, consistent fields, timings)
+* [ ] Phase 7: Update Dockerfile (wkhtmltopdf, fonts), rebuild, smoke test
+* [ ] Phase 8: Load test with multiple configs; capture timings; fix edge cases
+* [ ] Phase 9: Update README/DECISIONS/LOGGING; add runbook snippet
+* [ ] Phase 10: (Optional) Implement backlog items
+
+---
+
+## Quick win(s) to start next session
+
+* Pick **Letter vs A4** and the exact output path format.
+* Decide whether to **keep writing HTML to disk** (handy for debugging) or only in memory.
+* Choose **structlog vs stdlib logging + JSON** (I can tailor either).
+
+When you’re ready, we can start with Phase 1, and I’ll draft the exact renderer function signatures and logging fields so it slots right into your style.