# Phase 6: Story Progression & Lore System

**Goal:** Vector database-powered NPC knowledge and world lore
**Priority:** High
**Status:** Not Started
**Last Updated:** November 29, 2025

---

## Overview

Implement a layered knowledge system using vector databases (Weaviate) to provide NPCs and the Dungeon Master with contextual lore, regional history, and world knowledge. This enhances NPC conversations with authentic, contextual responses.

**Key Principles:**
- **Three-tier knowledge hierarchy** - World Lore → Regional Lore → NPC Persona
- **RAG (Retrieval-Augmented Generation)** - Semantic search for relevant context
- **Knowledge boundaries** - NPCs only know what they should know
- **Index-once strategy** - Embeddings generated at build time, not runtime

**Reference:** See `/docs/VECTOR_DATABASE_STRATEGY.md` for detailed architecture.

---

## Knowledge Hierarchy

### Three-Tier Structure

1. **World Lore DB** (Global)
   - Broad historical events, mythology, major kingdoms, legendary figures
   - Accessible to all NPCs and DM for player questions
   - Examples: "The Great War 200 years ago", "The origin of magic"

2. **Regional/Town Lore DB** (Location-specific)
   - Local history, notable events, landmarks, politics, rumors
   - Current town leadership, recent events, local legends
   - Trade routes, neighboring settlements, regional conflicts

3. **NPC Persona** (Individual, YAML-defined)
   - Personal background, personality, motivations
   - Specific knowledge based on profession/role
   - Already implemented in `/api/app/data/npcs/*.yaml`

---

## Task Groups

### Vector Database Setup (4 tasks)

| Task ID | Task | Status | Notes |
|---------|------|--------|-------|
| 6.1 | Add Weaviate to Docker Compose | ⬜ | Self-hosted for development |
| 6.2 | Create Weaviate service configuration | ⬜ | `/api/config/*.yaml` - dev/prod endpoints |
| 6.3 | Create WeaviateService class | ⬜ | `/api/app/services/weaviate_service.py` - connection, health check |
| 6.4 | **Checkpoint:** Verify Weaviate connectivity | ⬜ | Test connection and basic operations |

**Docker Compose Addition:**
```yaml
services:
  weaviate:
    image: semitechnologies/weaviate:latest
    ports:
      - "8080:8080"
    environment:
      AUTHENTICATION_ANONYMOUS_ACCESS_ENABLED: 'true'
      PERSISTENCE_DATA_PATH: '/var/lib/weaviate'
    volumes:
      - weaviate_data:/var/lib/weaviate
```

**Deliverable:** Working Weaviate instance in development environment

---

### Weaviate Schema & Collections (4 tasks)

| Task ID | Task | Status | Notes |
|---------|------|--------|-------|
| 6.5 | Define WorldLore collection schema | ⬜ | Metadata: knowledge_type, time_period, required_profession |
| 6.6 | Define RegionalLore collection schema | ⬜ | Metadata: region_id, knowledge_type, social_class |
| 6.7 | Create schema initialization script | ⬜ | `/api/scripts/init_weaviate_schema.py` |
| 6.8 | **Checkpoint:** Verify collections created | ⬜ | Test schema via Weaviate console |

**Collection Schema:**
```
WorldLore:
  - content: text (vectorized)
  - title: string
  - knowledge_type: string (academic, common, secret)
  - time_period: string (ancient, historical, recent, current)
  - required_profession: string[] (optional filter)
  - tags: string[]

RegionalLore:
  - content: text (vectorized)
  - title: string
  - region_id: string
  - knowledge_type: string
  - social_class: string[] (noble, merchant, commoner)
  - tags: string[]
```

**Deliverable:** Weaviate schema ready for data ingestion

---

### World Lore Content (4 tasks)

| Task ID | Task | Status | Notes |
|---------|------|--------|-------|
| 6.9 | Create lore content directory structure | ⬜ | `/api/app/data/lore/world/` |
| 6.10 | Write world history YAML | ⬜ | Major events, wars, kingdoms, mythology |
| 6.11 | Write mythology YAML | ⬜ | Gods, creation stories, magic origins |
| 6.12 | Write kingdoms YAML | ⬜ | Major factions, political landscape |

**Directory Structure:**
```
/api/app/data/lore/
  world/
    history.yaml        # Major historical events
    mythology.yaml      # Gods, legends, magic
    kingdoms.yaml       # Factions, politics
  regions/
    crossville/
      history.yaml      # Local history
      locations.yaml    # Notable places
      rumors.yaml       # Current gossip
```

**Example Lore Entry:**
```yaml
- id: "great_war"
  title: "The Great War of the Five Kingdoms"
  content: |
    Two hundred years ago, the Five Kingdoms united against
    the Shadow Empire in a conflict that reshaped the world...
  metadata:
    knowledge_type: "common"
    time_period: "historical"
    required_profession: null
  tags:
    - "war"
    - "five-kingdoms"
    - "shadow-empire"
```

**Deliverable:** 20+ world lore entries ready for embedding

---

### Regional Lore Content (3 tasks)

| Task ID | Task | Status | Notes |
|---------|------|--------|-------|
| 6.13 | Write Crossville regional history | ⬜ | Founding, notable events, local legends |
| 6.14 | Write Crossville locations lore | ⬜ | Descriptions for tavern, forest, dungeon |
| 6.15 | Write Crossville rumors | ⬜ | Current gossip, quest hooks |

**Deliverable:** 15+ regional lore entries for starter region

---

### Embedding Generation (4 tasks)

| Task ID | Task | Status | Notes |
|---------|------|--------|-------|
| 6.16 | Create embedding generation script | ⬜ | `/api/scripts/generate_lore_embeddings.py` |
| 6.17 | Implement chunking strategy | ⬜ | Split by logical units (events, locations, figures) |
| 6.18 | Generate and upload world lore embeddings | ⬜ | One-time generation, upload to Weaviate |
| 6.19 | **Checkpoint:** Verify embedding quality | ⬜ | Test semantic search returns relevant results |

**Embedding Model Options:**
- **Development:** sentence-transformers (free, fast iteration)
- **Production:** OpenAI text-embedding-3-large or Replicate multilingual-e5-large

**Deliverable:** All lore content embedded and searchable in Weaviate

---

### NPC Knowledge Integration (5 tasks)

> **Note:** The LoreService interface and MockLoreService stub were created in Phase 5 (`/api/app/services/lore_service.py`). This phase replaces the mock with the real Weaviate implementation.

| Task ID | Task | Status | Notes |
|---------|------|--------|-------|
| 6.20 | Implement WeaviateLoreService | ⬜ | Replace MockLoreService, query World + Regional DBs |
| 6.21 | Add knowledge filtering by NPC role | ⬜ | Profession, social class, relationship level |
| 6.22 | Integrate lore into NPC conversation prompts | ⬜ | RAG pattern: retrieve → inject → generate |
| 6.23 | Implement "I don't know" responses | ⬜ | Authentic ignorance when NPC lacks knowledge |
| 6.24 | **Checkpoint:** Test NPC conversations with lore | ⬜ | Verify NPCs use contextual knowledge |

**RAG Pattern for NPC Dialogue:**
```
[NPC Persona from YAML]
+
[Top 3-5 relevant chunks from Regional DB]
+
[Top 2-3 relevant chunks from World Lore if historical topic]
+
[Conversation history]
→ Feed to Claude with instruction to stay in character
```

**Knowledge Filtering Example:**
```python
# Blacksmith asking about metallurgy → Return results
# Blacksmith asking about court politics → "I don't know about that"

def filter_knowledge_for_npc(results, npc):
    return [r for r in results if npc_can_know(r.metadata, npc)]
```

**Deliverable:** NPCs respond with contextually appropriate lore

---

### DM Lore Access (2 tasks)

| Task ID | Task | Status | Notes |
|---------|------|--------|-------|
| 6.25 | Implement DM lore query endpoint | ⬜ | `POST /api/v1/lore/query` - Full access, no filtering |
| 6.26 | Integrate lore into story narration | ⬜ | DM uses lore for atmospheric descriptions |

**DM vs NPC Knowledge:**
- **DM Mode:** Access to ALL databases without restrictions
- **NPC Mode:** Knowledge filtered by persona/role/location

**Deliverable:** DM can access all lore for narrative purposes

---

## API Endpoints

| Method | Endpoint | Description |
|--------|----------|-------------|
| `POST` | `/api/v1/lore/query` | Query lore by topic (DM access) |
| `GET` | `/api/v1/lore/regions/{region_id}` | Get all lore for a region |
| `GET` | `/api/v1/lore/world` | Get world lore summary |

---

## Files to Create/Modify

**New Files:**
- `/api/app/services/weaviate_service.py` - Weaviate connection and queries
- `/api/scripts/init_weaviate_schema.py` - Schema initialization
- `/api/scripts/generate_lore_embeddings.py` - Embedding generation
- `/api/app/data/lore/world/history.yaml`
- `/api/app/data/lore/world/mythology.yaml`
- `/api/app/data/lore/world/kingdoms.yaml`
- `/api/app/data/lore/regions/crossville/history.yaml`
- `/api/app/data/lore/regions/crossville/locations.yaml`
- `/api/app/data/lore/regions/crossville/rumors.yaml`
- `/api/app/api/lore.py` - Lore API blueprint

**Modified Files:**
- `/api/app/services/lore_service.py` - Already exists from Phase 5 (stub). Add WeaviateLoreService implementation
- `/docker-compose.yml` - Add Weaviate service
- `/api/config/development.yaml` - Weaviate dev config
- `/api/config/production.yaml` - Weaviate Cloud Services config
- `/api/app/services/npc_loader.py` - Integrate lore queries
- `/api/app/ai/prompts.py` - Add lore context to prompts

---

## Testing Criteria

### Unit Tests
- [ ] Weaviate connection and health check
- [ ] Schema creation
- [ ] Lore content loading from YAML
- [ ] Embedding generation
- [ ] Semantic search returns relevant results
- [ ] Knowledge filtering by NPC role

### Integration Tests
- [ ] Full RAG pipeline: query → retrieve → inject → generate
- [ ] NPC conversation includes relevant lore
- [ ] NPC correctly says "I don't know" for unknown topics
- [ ] DM has full lore access

### Manual Testing
- [ ] Ask NPC about local history → Returns regional lore
- [ ] Ask NPC about world history → Returns filtered world lore
- [ ] Ask farmer about court politics → "I wouldn't know about that"
- [ ] Ask scholar about ancient history → Detailed response

---

## Success Criteria

- [ ] Weaviate running in development environment
- [ ] 20+ world lore entries embedded
- [ ] 15+ regional lore entries for Crossville
- [ ] NPC conversations enhanced with contextual lore
- [ ] Knowledge filtering working by profession/class
- [ ] DM has full lore access for narration
- [ ] Semantic search returning relevant results

---

## Dependencies

**Requires (already implemented):**
- NPC system (`/api/app/services/npc_loader.py`)
- Location system (`/api/app/services/location_loader.py`)
- AI service (`/api/app/ai/`)

**Requires (from Phase 5):**
- Quest system (quests can reference lore) ✅
- LoreService interface stub (`/api/app/services/lore_service.py`) ✅ - Ready to replace MockLoreService with WeaviateLoreService

**Enables (future phases):**
- Phase 7: Multiplayer (shared world lore)
- Lore codex/discovery system (optional future feature)

---

## Production Migration Path

### Zero-Code Migration to Weaviate Cloud Services

1. Export data from self-hosted Weaviate
2. Create Weaviate Cloud Services cluster
3. Import data to WCS
4. Change environment variable: `WEAVIATE_URL`
5. Deploy (no code changes required)

**Environment Configuration:**
```yaml
# development.yaml
weaviate:
  url: "http://localhost:8080"
  api_key: null

# production.yaml
weaviate:
  url: "https://your-cluster.weaviate.network"
  api_key: "${WEAVIATE_API_KEY}"
```

---

## Task Summary

| Group | Tasks | Checkpoints |
|-------|-------|-------------|
| Vector Database Setup | 3 | 1 |
| Weaviate Schema & Collections | 3 | 1 |
| World Lore Content | 4 | 0 |
| Regional Lore Content | 3 | 0 |
| Embedding Generation | 3 | 1 |
| NPC Knowledge Integration | 4 | 1 |
| DM Lore Access | 2 | 0 |
| **Total** | **22** | **4** |