Gen AI — Tokens

What are tokens?

Pieces of text that models use to process language.

Tokenization

import tiktoken

encoder = tiktoken.encoding_for_model("gpt-4")
tokens = encoder.encode("Hello, world!")
print(tokens)  # [9906, 11, 1917, 0]
print(len(tokens))  # 4 tokens

Token counting

def count_tokens(text, model="gpt-4"):
    encoder = tiktoken.encoding_for_model(model)
    return len(encoder.encode(text))

# Usage
text = "This is a test message"
print(f"Tokens: {count_tokens(text)}")

Token costs

Model	Input	Output
GPT-4	$0.03/1K	$0.06/1K
GPT-3.5	$0.001/1K	$0.002/1K
Claude 3	$0.015/1K	$0.075/1K

Cost calculation

def calculate_cost(input_tokens, output_tokens, model="gpt-4"):
    rates = {
        "gpt-4": {"input": 0.03, "output": 0.06},
        "gpt-3.5": {"input": 0.001, "output": 0.002}
    }
    
    input_cost = (input_tokens / 1000) * rates[model]["input"]
    output_cost = (output_tokens / 1000) * rates[model]["output"]
    
    return input_cost + output_cost

Token optimization

1. Concise prompts: Remove unnecessary words
2. Caching: Reuse common prefixes
3. Compression: Summarize long texts
4. Model selection: Use smaller models for simple tasks

Mini Practice

1. Count tokens in a text
2. Calculate API costs
3. Optimize a prompt for fewer tokens
4. Compare tokenizers

Up Next

Continue with Context Window - Managing context.