Code_of_Conquest/docs/phases/Phase6-StoryProgression.md

Phase 6: Story Progression & Lore System

Goal: Vector database-powered NPC knowledge and world lore Priority: High Status: Not Started Last Updated: November 29, 2025

---

Overview

Implement a layered knowledge system using vector databases (Weaviate) to provide NPCs and the Dungeon Master with contextual lore, regional history, and world knowledge. This enhances NPC conversations with authentic, contextual responses.

Key Principles:

• Three-tier knowledge hierarchy - World Lore → Regional Lore → NPC Persona
• RAG (Retrieval-Augmented Generation) - Semantic search for relevant context
• Knowledge boundaries - NPCs only know what they should know
• Index-once strategy - Embeddings generated at build time, not runtime

Reference: See /docs/VECTOR_DATABASE_STRATEGY.md for detailed architecture.

---

Knowledge Hierarchy

Three-Tier Structure

1. World Lore DB (Global)

   • Broad historical events, mythology, major kingdoms, legendary figures
   • Accessible to all NPCs and DM for player questions
   • Examples: "The Great War 200 years ago", "The origin of magic"

2. Regional/Town Lore DB (Location-specific)

   • Local history, notable events, landmarks, politics, rumors
   • Current town leadership, recent events, local legends
   • Trade routes, neighboring settlements, regional conflicts

3. NPC Persona (Individual, YAML-defined)

   • Personal background, personality, motivations
   • Specific knowledge based on profession/role
   • Already implemented in /api/app/data/npcs/*.yaml

---

Task Groups

Vector Database Setup (4 tasks)

Task ID	Task	Status	Notes
6.1	Add Weaviate to Docker Compose	⬜	Self-hosted for development
6.2	Create Weaviate service configuration	⬜	/api/config/*.yaml - dev/prod endpoints
6.3	Create WeaviateService class	⬜	/api/app/services/weaviate_service.py - connection, health check
6.4	Checkpoint: Verify Weaviate connectivity	⬜	Test connection and basic operations

Docker Compose Addition:

services:
  weaviate:
    image: semitechnologies/weaviate:latest
    ports:
      - "8080:8080"
    environment:
      AUTHENTICATION_ANONYMOUS_ACCESS_ENABLED: 'true'
      PERSISTENCE_DATA_PATH: '/var/lib/weaviate'
    volumes:
      - weaviate_data:/var/lib/weaviate

Deliverable: Working Weaviate instance in development environment

---

Weaviate Schema & Collections (4 tasks)

Task ID	Task	Status	Notes
6.5	Define WorldLore collection schema	⬜	Metadata: knowledge_type, time_period, required_profession
6.6	Define RegionalLore collection schema	⬜	Metadata: region_id, knowledge_type, social_class
6.7	Create schema initialization script	⬜	/api/scripts/init_weaviate_schema.py
6.8	Checkpoint: Verify collections created	⬜	Test schema via Weaviate console

Collection Schema:

WorldLore:
  - content: text (vectorized)
  - title: string
  - knowledge_type: string (academic, common, secret)
  - time_period: string (ancient, historical, recent, current)
  - required_profession: string[] (optional filter)
  - tags: string[]

RegionalLore:
  - content: text (vectorized)
  - title: string
  - region_id: string
  - knowledge_type: string
  - social_class: string[] (noble, merchant, commoner)
  - tags: string[]

Deliverable: Weaviate schema ready for data ingestion

---

World Lore Content (4 tasks)

Task ID	Task	Status	Notes
6.9	Create lore content directory structure	⬜	/api/app/data/lore/world/
6.10	Write world history YAML	⬜	Major events, wars, kingdoms, mythology
6.11	Write mythology YAML	⬜	Gods, creation stories, magic origins
6.12	Write kingdoms YAML	⬜	Major factions, political landscape

Directory Structure:

/api/app/data/lore/
  world/
    history.yaml        # Major historical events
    mythology.yaml      # Gods, legends, magic
    kingdoms.yaml       # Factions, politics
  regions/
    crossville/
      history.yaml      # Local history
      locations.yaml    # Notable places
      rumors.yaml       # Current gossip

Example Lore Entry:

- id: "great_war"
  title: "The Great War of the Five Kingdoms"
  content: |
    Two hundred years ago, the Five Kingdoms united against
    the Shadow Empire in a conflict that reshaped the world...
  metadata:
    knowledge_type: "common"
    time_period: "historical"
    required_profession: null
  tags:
    - "war"
    - "five-kingdoms"
    - "shadow-empire"

Deliverable: 20+ world lore entries ready for embedding

---

Regional Lore Content (3 tasks)

Task ID	Task	Status	Notes
6.13	Write Crossville regional history	⬜	Founding, notable events, local legends
6.14	Write Crossville locations lore	⬜	Descriptions for tavern, forest, dungeon
6.15	Write Crossville rumors	⬜	Current gossip, quest hooks

Deliverable: 15+ regional lore entries for starter region

---

Embedding Generation (4 tasks)

Task ID	Task	Status	Notes
6.16	Create embedding generation script	⬜	/api/scripts/generate_lore_embeddings.py
6.17	Implement chunking strategy	⬜	Split by logical units (events, locations, figures)
6.18	Generate and upload world lore embeddings	⬜	One-time generation, upload to Weaviate
6.19	Checkpoint: Verify embedding quality	⬜	Test semantic search returns relevant results

Embedding Model Options:

• Development: sentence-transformers (free, fast iteration)
• Production: OpenAI text-embedding-3-large or Replicate multilingual-e5-large

Deliverable: All lore content embedded and searchable in Weaviate

---

NPC Knowledge Integration (5 tasks)

Note: The LoreService interface and MockLoreService stub were created in Phase 5 (/api/app/services/lore_service.py). This phase replaces the mock with the real Weaviate implementation.

Task ID	Task	Status	Notes
6.20	Implement WeaviateLoreService	⬜	Replace MockLoreService, query World + Regional DBs
6.21	Add knowledge filtering by NPC role	⬜	Profession, social class, relationship level
6.22	Integrate lore into NPC conversation prompts	⬜	RAG pattern: retrieve → inject → generate
6.23	Implement "I don't know" responses	⬜	Authentic ignorance when NPC lacks knowledge
6.24	Checkpoint: Test NPC conversations with lore	⬜	Verify NPCs use contextual knowledge

RAG Pattern for NPC Dialogue:

[NPC Persona from YAML]
+
[Top 3-5 relevant chunks from Regional DB]
+
[Top 2-3 relevant chunks from World Lore if historical topic]
+
[Conversation history]
→ Feed to Claude with instruction to stay in character

Knowledge Filtering Example:

# Blacksmith asking about metallurgy → Return results
# Blacksmith asking about court politics → "I don't know about that"

def filter_knowledge_for_npc(results, npc):
    return [r for r in results if npc_can_know(r.metadata, npc)]

Deliverable: NPCs respond with contextually appropriate lore

---

DM Lore Access (2 tasks)

Task ID	Task	Status	Notes
6.25	Implement DM lore query endpoint	⬜	POST /api/v1/lore/query - Full access, no filtering
6.26	Integrate lore into story narration	⬜	DM uses lore for atmospheric descriptions

DM vs NPC Knowledge:

• DM Mode: Access to ALL databases without restrictions
• NPC Mode: Knowledge filtered by persona/role/location

Deliverable: DM can access all lore for narrative purposes

---

API Endpoints

Method	Endpoint	Description
POST	/api/v1/lore/query	Query lore by topic (DM access)
GET	/api/v1/lore/regions/{region_id}	Get all lore for a region
GET	/api/v1/lore/world	Get world lore summary

---

Files to Create/Modify

New Files:

• /api/app/services/weaviate_service.py - Weaviate connection and queries
• /api/scripts/init_weaviate_schema.py - Schema initialization
• /api/scripts/generate_lore_embeddings.py - Embedding generation
• /api/app/data/lore/world/history.yaml
• /api/app/data/lore/world/mythology.yaml
• /api/app/data/lore/world/kingdoms.yaml
• /api/app/data/lore/regions/crossville/history.yaml
• /api/app/data/lore/regions/crossville/locations.yaml
• /api/app/data/lore/regions/crossville/rumors.yaml
• /api/app/api/lore.py - Lore API blueprint

Modified Files:

• /api/app/services/lore_service.py - Already exists from Phase 5 (stub). Add WeaviateLoreService implementation
• /docker-compose.yml - Add Weaviate service
• /api/config/development.yaml - Weaviate dev config
• /api/config/production.yaml - Weaviate Cloud Services config
• /api/app/services/npc_loader.py - Integrate lore queries
• /api/app/ai/prompts.py - Add lore context to prompts

---

Testing Criteria

Unit Tests

• Weaviate connection and health check
• Schema creation
• Lore content loading from YAML
• Embedding generation
• Semantic search returns relevant results
• Knowledge filtering by NPC role

Integration Tests

• Full RAG pipeline: query → retrieve → inject → generate
• NPC conversation includes relevant lore
• NPC correctly says "I don't know" for unknown topics
• DM has full lore access

Manual Testing

• Ask NPC about local history → Returns regional lore
• Ask NPC about world history → Returns filtered world lore
• Ask farmer about court politics → "I wouldn't know about that"
• Ask scholar about ancient history → Detailed response

---

Success Criteria

• Weaviate running in development environment
• 20+ world lore entries embedded
• 15+ regional lore entries for Crossville
• NPC conversations enhanced with contextual lore
• Knowledge filtering working by profession/class
• DM has full lore access for narration
• Semantic search returning relevant results

---

Dependencies

Requires (already implemented):

• NPC system (/api/app/services/npc_loader.py)
• Location system (/api/app/services/location_loader.py)
• AI service (/api/app/ai/)

Requires (from Phase 5):

• Quest system (quests can reference lore) ✅
• LoreService interface stub (/api/app/services/lore_service.py) ✅ - Ready to replace MockLoreService with WeaviateLoreService

Enables (future phases):

• Phase 7: Multiplayer (shared world lore)
• Lore codex/discovery system (optional future feature)

---

Production Migration Path

Zero-Code Migration to Weaviate Cloud Services

1. Export data from self-hosted Weaviate
2. Create Weaviate Cloud Services cluster
3. Import data to WCS
4. Change environment variable: WEAVIATE_URL
5. Deploy (no code changes required)

Environment Configuration:

# development.yaml
weaviate:
  url: "http://localhost:8080"
  api_key: null

# production.yaml
weaviate:
  url: "https://your-cluster.weaviate.network"
  api_key: "${WEAVIATE_API_KEY}"

---

Task Summary

Group	Tasks	Checkpoints
Vector Database Setup	3	1
Weaviate Schema & Collections	3	1
World Lore Content	4	0
Regional Lore Content	3	0
Embedding Generation	3	1
NPC Knowledge Integration	4	1
DM Lore Access	2	0
Total	22	4