Gen AI — Context Windows

What is context window?

The maximum amount of text a model can process in one request.

Context sizes

Model	Context Window
GPT-3.5	4K / 16K
GPT-4	8K / 32K / 128K
Claude 3	200K
Gemini	1M

Managing context

def manage_context(messages, max_tokens=4000):
    """Keep messages within token limit."""
    system = messages[0]
    conversation = messages[1:]
    
    # Count tokens
    total = count_tokens(system["content"])
    kept = []
    
    for msg in reversed(conversation):
        msg_tokens = count_tokens(msg["content"])
        if total + msg_tokens > max_tokens:
            break
        kept.insert(0, msg)
        total += msg_tokens
    
    return [system] + kept

Sliding window

class ContextWindow:
    def __init__(self, max_tokens=4000):
        self.max_tokens = max_tokens
        self.messages = []
    
    def add(self, message):
        self.messages.append(message)
        
        while self.total_tokens() > self.max_tokens:
            # Remove oldest non-system message
            if len(self.messages) > 1:
                self.messages.pop(1)
    
    def total_tokens(self):
        return sum(count_tokens(m["content"]) for m in self.messages)

Summarization strategy

def summarize_context(messages):
    """Summarize old messages to save context."""
    old_messages = messages[1:-4]  # Keep recent messages
    
    summary = client.chat.completions.create(
        model="gpt-3.5-turbo",
        messages=[{
            "role": "user",
            "content": f"Summarize this conversation:\n{format_messages(old_messages)}"
        }]
    )
    
    return [{"role": "system", "content": summary.choices[0].message.content}]

Mini Practice

1. Implement context management
2. Build a sliding window
3. Test summarization strategy
4. Monitor context usage

Up Next

Continue with API Design - Building AI APIs.