Code_of_Conquest/api/docs/CHAT_SYSTEM.md

# Chat / Conversation History System

## Overview

The Chat System provides complete player-NPC conversation history tracking with unlimited storage, fast AI context retrieval, and powerful search capabilities. It replaces the previous inline dialogue_history with a scalable, performant architecture.

**Key Features:**
- Unlimited conversation history (no 10-message cap)
- Hybrid storage for performance (cache + full history)
- Quest and faction tracking ready
- Full-text search with filters
- Soft delete for privacy/moderation
- Future-ready for player-to-player chat

---

## Architecture

### Hybrid Storage Design

The system uses a **two-tier storage approach** for optimal performance:

```
┌─────────────────────────────────────────────────┐
│ Character Document                              │
│                                                 │
│ npc_interactions[npc_id]:                      │
│   └─ recent_messages: [last 3 messages]       │  ← AI Context (fast)
│   └─ total_messages: 15                        │
│   └─ relationship_level, flags, etc.           │
└─────────────────────────────────────────────────┘
                    │
                    │ Updates on each new message
                    ▼
┌─────────────────────────────────────────────────┐
│ chat_messages Collection                        │
│                                                 │
│ ┌─────────────────────────────────────────┐   │
│ │ Message 1 (oldest)                       │   │
│ │ Message 2                                │   │
│ │ ...                                      │   │  ← Full History (queries)
│ │ Message 15 (newest)                      │   │
│ └─────────────────────────────────────────┘   │
│                                                 │
│ Indexes: character+npc+time, session, context  │
└─────────────────────────────────────────────────┘
```

**Benefits:**
- **90% of queries** (AI context) read from character document cache → No database query
- **10% of queries** (user browsing history, search) use indexed collection → Fast retrieval
- Character documents stay small (3 messages vs unlimited)
- No performance degradation as conversations grow

### Data Flow

**Saving a New Message:**
```
User talks to NPC
    │
    ▼
POST /api/v1/npcs/{npc_id}/talk
    │
    ▼
AI generates dialogue (background task)
    │
    ▼
ChatMessageService.save_dialogue_exchange()
    │
    ├─→ 1. Save to chat_messages collection (UUID, full data)
    │
    └─→ 2. Update character.npc_interactions[npc_id]:
            - Append to recent_messages (keep last 3)
            - Increment total_messages counter
            - Update last_interaction timestamp
```

**Reading for AI Context:**
```
AI needs conversation history
    │
    ▼
CharacterService.get_npc_dialogue_history()
    │
    └─→ Returns character.npc_interactions[npc_id].recent_messages
        (Already loaded in memory, instant access)
```

**Reading for User UI:**
```
User views conversation history
    │
    ▼
GET /api/v1/characters/{char_id}/chats/{npc_id}?limit=50&offset=0
    │
    ▼
ChatMessageService.get_conversation_history()
    │
    └─→ Query chat_messages collection
        WHERE character_id = X AND npc_id = Y
        ORDER BY timestamp DESC
        LIMIT 50 OFFSET 0
```

---

## Database Schema

### Collection: `chat_messages`

| Column | Type | Size | Indexed | Description |
|--------|------|------|---------|-------------|
| `message_id` | string | 36 | Primary | UUID |
| `character_id` | string | 100 | Yes | Player character |
| `npc_id` | string | 100 | Yes | NPC identifier |
| `player_message` | string | 2000 | No | Player input |
| `npc_response` | string | 5000 | No | AI-generated reply |
| `timestamp` | datetime | - | Yes | ISO 8601 format |
| `session_id` | string | 100 | Yes | Game session |
| `location_id` | string | 100 | No | Where conversation happened |
| `context` | string | 50 | Yes | MessageContext enum |
| `metadata` | string (JSON) | 1000 | No | Extensible (quest_id, etc.) |
| `is_deleted` | boolean | - | No | Soft delete flag |

### Indexes

1. **idx_character_npc_time** (character_id + npc_id + timestamp DESC)
   - **Purpose**: Get conversation between character and specific NPC
   - **Query**: "Show me my chat with Grom"
   - **Used by**: `get_conversation_history()`

2. **idx_character_time** (character_id + timestamp DESC)
   - **Purpose**: Get all messages for a character across all NPCs
   - **Query**: "Show me all my conversations"
   - **Used by**: `get_all_conversations_summary()`

3. **idx_session_time** (session_id + timestamp DESC)
   - **Purpose**: Get all chat messages from a specific game session
   - **Query**: "What did I discuss during this session?"
   - **Used by**: Session replay, analytics

4. **idx_context** (context)
   - **Purpose**: Filter messages by interaction type
   - **Query**: "Show me all quest offers"
   - **Used by**: `search_messages()` with context filter

5. **idx_timestamp** (timestamp DESC)
   - **Purpose**: Date range queries
   - **Query**: "Show messages from last week"
   - **Used by**: `search_messages()` with date filters

---

## Integration Points

### 1. NPC Dialogue Generation (`/api/app/tasks/ai_tasks.py`)

**Old Flow (Deprecated):**
```python
# Saved to character.npc_interactions[npc_id].dialogue_history
character_service.add_npc_dialogue_exchange(
    character_id, npc_id, player_line, npc_response
)
```

**New Flow:**
```python
# Saves to chat_messages + updates recent_messages cache
chat_service = get_chat_message_service()
chat_service.save_dialogue_exchange(
    character_id=character_id,
    user_id=user_id,
    npc_id=npc_id,
    player_message=player_line,
    npc_response=npc_response,
    context=MessageContext.DIALOGUE,
    metadata={},  # Can include quest_id, item_id when available
    session_id=session_id,
    location_id=current_location
)
```

### 2. Character Service (`/api/app/services/character_service.py`)

**Updated Method:**
```python
def get_npc_dialogue_history(character_id, user_id, npc_id, limit=5):
    """
    Get recent dialogue history from recent_messages cache.
    Falls back to dialogue_history for backward compatibility.
    """
    interaction = character.npc_interactions.get(npc_id, {})

    # NEW: Read from recent_messages (last 3 messages)
    recent_messages = interaction.get("recent_messages")
    if recent_messages is not None:
        return recent_messages[-limit:]

    # DEPRECATED: Fall back to old dialogue_history field
    dialogue_history = interaction.get("dialogue_history", [])
    return dialogue_history[-limit:]
```

### 3. Character Document Structure

**Updated `npc_interactions` Field:**
```python
{
    "npc_grom_ironbeard": {
        "npc_id": "npc_grom_ironbeard",
        "first_met": "2025-11-25T10:00:00Z",
        "last_interaction": "2025-11-25T14:30:00Z",
        "interaction_count": 5,
        "revealed_secrets": [0, 2],
        "relationship_level": 65,
        "custom_flags": {"helped_with_rats": true},

        # NEW FIELDS
        "recent_messages": [  # Last 3 messages for AI context
            {
                "player_message": "What rumors have you heard?",
                "npc_response": "*leans in* Strange folk...",
                "timestamp": "2025-11-25T14:30:00Z"
            }
        ],
        "total_messages": 15,  # Total count for UI display

        # DEPRECATED (kept for backward compatibility)
        "dialogue_history": []  # Will be removed after full migration
    }
}
```

---

## API Endpoints

See `API_REFERENCE.md` for complete endpoint documentation.

**Summary:**
- `GET /api/v1/characters/{char_id}/chats` - All conversations summary
- `GET /api/v1/characters/{char_id}/chats/{npc_id}` - Full conversation with pagination
- `GET /api/v1/characters/{char_id}/chats/search` - Search with filters
- `DELETE /api/v1/characters/{char_id}/chats/{msg_id}` - Soft delete

---

## Performance Characteristics

### Storage Estimates

| Scenario | Messages | Storage per Character |
|----------|----------|----------------------|
| Casual player | 50 messages across 5 NPCs | ~25 KB |
| Active player | 200 messages across 10 NPCs | ~100 KB |
| Heavy player | 1000 messages across 20 NPCs | ~500 KB |

**Character Document Impact:**
- Recent messages (3 per NPC): ~1.5 KB per NPC
- 10 NPCs: ~15 KB total (vs ~50 KB for old 10-message history)
- 67% reduction in character document size

### Query Performance

**AI Context Retrieval:**
- **Old**: ~50ms database query + deserialization
- **New**: ~1ms (read from already-loaded character object)
- **Improvement**: 50x faster

**User History Browsing:**
- **Old**: Limited to last 10 messages (no pagination)
- **New**: Unlimited with pagination (50ms per page)
- **Improvement**: Unlimited history with same performance

**Search:**
- **Old**: Not available (would require scanning all character docs)
- **New**: ~100ms for filtered search across thousands of messages
- **Improvement**: Previously impossible

### Scalability

**Growth Characteristics:**
- Character document size: O(1) - Fixed at 3 messages per NPC
- Chat collection size: O(n) - Linear growth with message count
- Query performance: O(log n) - Indexed queries remain fast

**Capacity Estimate:**
- 1000 active players
- Average 500 messages per player
- Total: 500,000 messages = ~250 MB
- Appwrite easily handles this scale

---

## Future Extensions

### Quest Tracking
```python
# Save quest offer with metadata
chat_service.save_dialogue_exchange(
    ...,
    context=MessageContext.QUEST_OFFERED,
    metadata={"quest_id": "quest_cellar_rats"}
)

# Query all quest offers
results = chat_service.search_messages(
    ...,
    context=MessageContext.QUEST_OFFERED
)
```

### Faction Tracking
```python
# Save reputation change with metadata
chat_service.save_dialogue_exchange(
    ...,
    context=MessageContext.DIALOGUE,
    metadata={
        "faction_id": "crossville_guard",
        "reputation_change": +10
    }
)
```

### Player-to-Player Chat
```python
# Add message_type enum to distinguish chat types
class MessageType(Enum):
    PLAYER_TO_NPC = "player_to_npc"
    NPC_TO_PLAYER = "npc_to_player"
    PLAYER_TO_PLAYER = "player_to_player"
    SYSTEM = "system"

# Extend ChatMessage with message_type field
# Extend indexes with sender_id + recipient_id
```

### Read Receipts
```python
# Add read_at field to ChatMessage
# Update when player views conversation
# Show "unread" badge in UI
```

### Chat Channels
```python
# Add channel_id field (party, guild, global)
# Index by channel_id for group chats
# Support broadcasting to multiple recipients
```

---

## Migration Strategy

### Phase 1: Dual Write (Current)
- ✅ New messages saved to both locations
- ✅ Old dialogue_history field still maintained
- ✅ Reads prefer recent_messages, fallback to dialogue_history
- ✅ Zero downtime migration

### Phase 2: Data Backfill (Optional)
- Migrate existing dialogue_history to chat_messages
- Script: `/api/scripts/migrate_dialogue_history.py`
- Can be run anytime without disrupting service

### Phase 3: Deprecation (Future)
- Stop writing to dialogue_history field
- Remove field from Character model
- Remove add_npc_dialogue_exchange() method

**Timeline:** Phase 2-3 can wait until full testing complete. No rush.

---

## Security & Privacy

**Access Control:**
- Users can only access their own character's messages
- Ownership validated on every request
- Character service validates user_id matches

**Soft Delete:**
- Messages marked is_deleted=true, not removed
- Filtered from all queries automatically
- Preserved for audit/moderation if needed

**Data Retention:**
- No automatic cleanup implemented
- Messages persist indefinitely
- Future: Could add retention policy per user preference

---

## Monitoring & Analytics

**Usage Tracking:**
- Total messages per character (total_messages field)
- Interaction counts per NPC (interaction_count field)
- Message context distribution (via context field)

**Performance Metrics:**
- Query latency (via logging)
- Cache hit rate (recent_messages vs full query)
- Storage growth (collection size monitoring)

**Business Metrics:**
- Most-contacted NPCs
- Average conversation length
- Quest offer acceptance rate (via context tracking)
- Player engagement (messages per session)

---

## Development Notes

**File Locations:**
- Model: `/app/models/chat_message.py`
- Service: `/app/services/chat_message_service.py`
- API: `/app/api/chat.py`
- Database Init: `/app/services/database_init.py` (init_chat_messages_table)

**Testing:**
- Unit tests: `/tests/test_chat_message_service.py`
- Integration tests: `/tests/test_chat_api.py`
- Manual testing: See API_REFERENCE.md for curl examples

**Dependencies:**
- Appwrite SDK (database operations)
- Character Service (ownership validation)
- NPC Loader (NPC name resolution)

---

## Troubleshooting

**Issue: Messages not appearing in history**
- Check chat_messages collection exists (run init_database.py)
- Verify ChatMessageService is being called in ai_tasks.py
- Check logs for errors during save_dialogue_exchange()

**Issue: Slow queries**
- Verify indexes exist (check Appwrite console)
- Monitor query patterns (check logs)
- Consider adjusting pagination limits

**Issue: Character document too large**
- Should not happen (recent_messages capped at 3)
- If occurring, check for old dialogue_history accumulation
- Run migration script to clean up old data

---

## References

- API_REFERENCE.md - Chat API endpoints
- DATA_MODELS.md - ChatMessage model details
- APPWRITE_SETUP.md - Database configuration