Gen AI — Embeddings

What are embeddings?

Numerical representations of text that capture semantic meaning.

OpenAI embeddings

import openai

client = openai.OpenAI()

response = client.embeddings.create(
    model="text-embedding-3-small",
    input="The quick brown fox"
)

embedding = response.data[0].embedding
print(f"Dimensions: {len(embedding)}")  # 1536

Similarity search

import numpy as np

def cosine_similarity(a, b):
    return np.dot(a, b) / (np.linalg.norm(a) * np.linalg.norm(b))

texts = [
    "The cat sat on the mat",
    "A feline rested on the rug",
    "Python is a programming language"
]

embeddings = get_embeddings(texts)

# Compare similarities
sim_01 = cosine_similarity(embeddings[0], embeddings[1])  # High
sim_02 = cosine_similarity(embeddings[0], embeddings[2])  # Low

Batch embedding

response = client.embeddings.create(
    model="text-embedding-3-small",
    input=["Text 1", "Text 2", "Text 3"]
)

embeddings = [item.embedding for item in response.data]

Using for search

def search(query, documents, top_k=3):
    query_embedding = get_embedding(query)
    doc_embeddings = get_embeddings(documents)
    
    similarities = [
        cosine_similarity(query_embedding, doc_emb)
        for doc_emb in doc_embeddings
    ]
    
    ranked = sorted(
        zip(documents, similarities),
        key=lambda x: x[1],
        reverse=True
    )
    
    return ranked[:top_k]

Mini Practice

1. Generate embeddings for text
2. Calculate cosine similarity
3. Build a simple search engine
4. Compare embedding models

Up Next

Continue with Vector Databases - Storing embeddings.