Retrieval Pipeline

The retrieval pipeline searches and ranks relevant documents through a 6-step process. Each step can be customized with pluggable strategies to optimize for specific use cases.

Pipeline Overview

Pipeline Function

export function aktorVektorRetrievalPipeline(input: {
    client: Aktor<Client>;
    query: Aktor<string>;
    metadataQuery?: Aktor<string>;
    limit?: Aktor<number>;
    
    // Optional strategy overrides
    queryPreprocessingStrategy?: AktorQueryPreprocessingStrategy;
    queryEmbeddingStrategy?: AktorQueryEmbeddingStrategy;
    vectorSearchStrategy?: AktorVectorSearchStrategy;
    metadataFilterStrategy?: AktorMetadataFilterStrategy;
    rerankingStrategy?: AktorRerankingStrategy;
    postProcessingStrategy?: AktorRetrievalPostProcessingStrategy;
})

Strategies

1. Query Preprocessing Strategy

Purpose: Clean and optimize user queries for better search results

Type Definition:

type QueryPreprocessingStrategy = (input: { query: string }) => Promise<PreprocessedQuery>;

interface PreprocessedQuery {
    originalQuery: string;
    processedQuery: string;
    queryType?: 'semantic' | 'keyword' | 'hybrid';
    expandedTerms?: string[];
}

Default Implementation:

export const defaultQueryPreprocessingStrategy: QueryPreprocessingStrategy = async ({ query }) => {
    const processedQuery = query.trim().toLowerCase();
    
    return {
        originalQuery: query,
        processedQuery,
        queryType: 'semantic',
    };
};

What it does:

• Normalizes query text (trimming, casing)
• Can expand queries with synonyms
• Detects query intent (semantic vs keyword)
• Supports query rewriting and optimization

---

2. Query Embedding Strategy

Purpose: Generate vector embeddings for search queries

Type Definition:

type QueryEmbeddingStrategy = (input: { query: PreprocessedQuery }) => Promise<QueryEmbedding>;

interface QueryEmbedding {
    query: string;
    embedding: number[];
    model: string;
    embeddedAt: string;
}

Default Implementation:

export const defaultQueryEmbeddingStrategy: QueryEmbeddingStrategy = async ({ query }) => {
    const embedding = await generateQueryEmbedding(query.processedQuery);
    
    return {
        query: query.processedQuery,
        embedding,
        model: 'text-embedding-3-small',
        embeddedAt: new Date().toISOString(),
    };
};

What it does:

• Generates embeddings using the same model as documents
• Supports multiple embedding providers
• Handles batch query processing
• Ensures embedding compatibility with stored vectors

---

3. Vector Search Strategy

Purpose: Find similar vectors using LibSQL vector operations

Type Definition:

type VectorSearchStrategy = (input: { 
    client: Client; 
    queryEmbedding: QueryEmbedding; 
    limit: number; 
}) => Promise<VectorSearchResult[]>;

interface VectorSearchResult {
    chunkId: string;
    documentId: string;
    content: string;
    similarity: number;
    metadata: DocumentMetadata;
    distance?: number;
    sourceFile?: string;
    documentTitle?: string;
}

Available Search Aktors:

• aktorSearchSimilarVectors: Basic vector similarity search
• aktorSearchSimilarVectorsWithChunks: Search with chunk context
• aktorGetVectorsByDocumentId: Get all vectors for a document
• aktorGetStoredVectorData: Get vector with full metadata

Default Implementation:

export const defaultVectorSearchStrategy: VectorSearchStrategy = async ({ 
    client, 
    queryEmbedding, 
    limit 
}) => {
    const searchQuery = `
        SELECT 
            c.chunk_id,
            c.document_id,
            c.text as content,
            c.metadata,
            d.title as document_title,
            f.filename as source_file,
            vector_distance_cos(v.vector, vector32(?)) as distance,
            (1 - vector_distance_cos(v.vector, vector32(?))) as similarity
        FROM vector_top_k('vectors_idx', vector32(?), ?) AS top_k
        JOIN vectors v ON v.chunk_id = top_k.id
        JOIN chunks c ON c.chunk_id = v.chunk_id
        JOIN documents d ON d.document_id = c.document_id
        JOIN files f ON f.file_id = d.file_id
        ORDER BY distance ASC
    `;

    const embeddingArray = new Float32Array(queryEmbedding.embedding);
    const result = await client.execute({
        sql: searchQuery,
        args: [embeddingArray, embeddingArray, embeddingArray, limit]
    });

    return result.rows.map((row) => ({
        chunkId: row.chunk_id as string,
        documentId: row.document_id as string,
        content: row.content as string,
        similarity: row.similarity as number,
        distance: row.distance as number,
        metadata: JSON.parse(row.metadata as string),
        documentTitle: row.document_title as string,
        sourceFile: row.source_file as string,
    }));
};

Alternative: Using Search Aktors:

// Use dedicated search aktors for better type safety
const searchResults = await aktorSearchSimilarVectorsWithChunks({
    client,
    queryEmbedding: queryEmbedding.embedding,
    limit,
}).get();

What it does:

• Uses LibSQL's vector_top_k for efficient similarity search
• Calculates cosine similarity scores
• Joins vector, chunk, and document data
• Supports configurable result limits

---

4. Metadata Filter Strategy

Purpose: Apply metadata-based filtering to search results

Type Definition:

type MetadataFilterStrategy = (input: { 
    client: Client; 
    results: VectorSearchResult[]; 
    metadataQuery?: string; 
}) => Promise<FilteredResult[]>;

interface FilteredResult {
    chunkId: string;
    documentId: string;
    content: string;
    similarity: number;
    metadata: DocumentMetadata;
    matchedFilters?: string[];
    filterScore?: number;
}

Default Implementation:

export const defaultMetadataFilterStrategy: MetadataFilterStrategy = async ({ 
    results, 
    metadataQuery 
}) => {
    if (!metadataQuery) {
        return results.map(result => ({
            ...result,
            matchedFilters: [],
            filterScore: 1.0,
        }));
    }

    const filteredResults = results.filter(result => {
        const metadata = result.metadata;
        const query = metadataQuery.toLowerCase();
        const metadataString = JSON.stringify(metadata).toLowerCase();
        return metadataString.includes(query);
    });

    return filteredResults.map(result => ({
        ...result,
        matchedFilters: [metadataQuery],
        filterScore: 1.0,
    }));
};

What it does:

• Filters results based on metadata criteria
• Supports complex metadata queries
• Can filter by document type, tags, dates, etc.
• Maintains filter provenance for transparency

---

5. Reranking Strategy

Purpose: Improve result relevance using advanced ranking algorithms

Type Definition:

type RerankingStrategy = (input: { 
    query: PreprocessedQuery; 
    results: FilteredResult[]; 
}) => Promise<RankedResult[]>;

interface RankedResult {
    chunkId: string;
    documentId: string;
    content: string;
    similarity: number;
    relevanceScore: number;
    metadata: DocumentMetadata;
    rankingReason?: string;
    rerankedAt?: string;
}

Default Implementation:

export const defaultRerankingStrategy: RerankingStrategy = async ({ query, results }) => {
    const rerankedResults = results.map(result => {
        const relevanceScore = calculateRelevanceScore(query.processedQuery, result.content);
        
        return {
            ...result,
            relevanceScore,
            rankingReason: 'keyword_overlap',
            rerankedAt: new Date().toISOString(),
        };
    });

    // Sort by combined similarity and relevance score
    rerankedResults.sort((a, b) => {
        const scoreA = (a.similarity * 0.7) + (a.relevanceScore * 0.3);
        const scoreB = (b.similarity * 0.7) + (b.relevanceScore * 0.3);
        return scoreB - scoreA;
    });

    return rerankedResults;
};

function calculateRelevanceScore(query: string, content: string): number {
    const queryTerms = query.toLowerCase().split(/\s+/);
    const contentLower = content.toLowerCase();
    
    const matches = queryTerms.filter(term => contentLower.includes(term)).length;
    const termRatio = matches / queryTerms.length;
    
    let score = termRatio;
    
    // Bonus for exact phrase matches
    if (contentLower.includes(query.toLowerCase())) {
        score += 0.2;
    }
    
    // Bonus for terms appearing early in content
    const firstTermIndex = contentLower.indexOf(queryTerms[0]);
    if (firstTermIndex >= 0 && firstTermIndex < 100) {
        score += 0.1;
    }
    
    return Math.min(score, 1.0);
}

What it does:

• Calculates keyword overlap and phrase matching
• Combines semantic similarity with lexical relevance
• Supports position-based scoring (terms early in text)
• Can be extended with cross-encoder models

---

6. Post-Processing Strategy

Purpose: Format results for optimal LLM consumption

Type Definition:

type PostProcessingStrategy = (input: { 
    query: PreprocessedQuery; 
    results: RankedResult[]; 
}) => Promise<FormattedResult[]>;

interface FormattedResult {
    chunkId: string;
    documentId: string;
    content: string;
    similarity: number;
    relevanceScore: number;
    metadata: DocumentMetadata;
    citation?: string;
    highlights?: string[];
    contextBlock?: string;
}

Default Implementation:

export const defaultPostProcessingStrategy: PostProcessingStrategy = async ({ query, results }) => {
    return results.map(result => ({
        ...result,
        citation: generateCitation(result),
        highlights: highlightQueryTerms(result.content, query.processedQuery),
        contextBlock: createContextBlock(result),
    }));
};

function generateCitation(result: RankedResult): string {
    const { metadata } = result;
    return `${metadata.source || 'Unknown Source'} (Document: ${result.documentId})`;
}

function highlightQueryTerms(content: string, query: string): string[] {
    const queryTerms = query.toLowerCase().split(/\s+/);
    const highlights: string[] = [];
    const contentLower = content.toLowerCase();
    
    for (const term of queryTerms) {
        const index = contentLower.indexOf(term);
        if (index >= 0) {
            const start = Math.max(0, index - 50);
            const end = Math.min(content.length, index + term.length + 50);
            const highlight = content.substring(start, end);
            highlights.push(`...${highlight}...`);
        }
    }
    
    return highlights;
}

function createContextBlock(result: RankedResult): string {
    const maxLength = 500;
    if (result.content.length <= maxLength) {
        return result.content;
    }
    
    return result.content.substring(0, maxLength) + '...';
}

What it does:

• Generates proper citations for source attribution
• Creates highlighted snippets showing query matches
• Formats content blocks for LLM context
• Ensures results are ready for RAG applications

Usage Examples

Basic Search (All Defaults)

const results = aktorVektorRetrievalPipeline({
    client: dbClient,
    query: queryString,
    limit: limitNumber,
});

Search with Metadata Filtering

const results = aktorVektorRetrievalPipeline({
    client: dbClient,
    query: queryString,
    metadataQuery: "type:documentation",
    limit: limitNumber,
});

Custom Reranking Strategy

const results = aktorVektorRetrievalPipeline({
    client: dbClient,
    query: queryString,
    limit: limitNumber,
    rerankingStrategy: aktorCrossEncoderRerankingStrategy,
});

Multiple Custom Strategies

const results = aktorVektorRetrievalPipeline({
    client: dbClient,
    query: queryString,
    limit: limitNumber,
    queryPreprocessingStrategy: aktorAdvancedQueryPreprocessor,
    vectorSearchStrategy: aktorHybridSearchStrategy,
    rerankingStrategy: aktorCrossEncoderRerankingStrategy,
});

Pipeline Return Type

interface RetrievalPipelineResult {
    results: FormattedResult[];
    queryInfo: {
        originalQuery: string;
        processedQuery: string;
        embedding: QueryEmbedding;
        processingTime: number;
    };
    searchStats: {
        totalResults: number;
        filteredResults: number;
        rerankedResults: number;
    };
}

Working with Results

const result = await aktorVektorRetrievalPipeline({
    client: dbClient,
    query: queryString,
    limit: 10,
}).get();

// Access formatted results
result.results.forEach(item => {
    console.log(`Similarity: ${item.similarity.toFixed(3)}`);
    console.log(`Source: ${item.citation}`);
    console.log(`Content: ${item.contextBlock}`);
    console.log(`Highlights: ${item.highlights?.join(', ')}`);
    console.log('---');
});

// Check search statistics
console.log(`Found ${result.searchStats.totalResults} results`);
console.log(`Query processed in ${result.queryInfo.processingTime}ms`);

Advanced Strategies

Cross-Encoder Reranking

export const aktorCrossEncoderRerankingStrategy = createAktorFunctionAsync(
    'aktorCrossEncoderRerankingStrategy',
    async ({ query, results }) => {
        // Use transformer models for precise relevance scoring
        const queryResultPairs = results.map(result => ({
            query: query.processedQuery,
            text: result.content,
            originalResult: result
        }));
        
        const relevanceScores = await calculateCrossEncoderScores(queryResultPairs);
        
        return results.map((result, index) => ({
            ...result,
            relevanceScore: relevanceScores[index],
            rankingReason: 'cross_encoder',
            rerankedAt: new Date().toISOString(),
        })).sort((a, b) => b.relevanceScore - a.relevanceScore);
    }
);

Hybrid Search Strategy

export const aktorHybridSearchStrategy = createAktorFunctionAsync(
    'aktorHybridSearchStrategy',
    async ({ client, queryEmbedding, limit }) => {
        // Combine vector similarity with keyword search
        const vectorResults = await vectorSearch(client, queryEmbedding, limit);
        const keywordResults = await keywordSearch(client, queryEmbedding.query, limit);
        
        // Merge and deduplicate results
        const combinedResults = mergeSearchResults(vectorResults, keywordResults);
        
        return combinedResults.slice(0, limit);
    }
);

Best Practices

1. Query Preprocessing: Clean and normalize queries for consistent results
2. Embedding Consistency: Use the same embedding model for queries and documents
3. Metadata Design: Structure metadata for efficient filtering
4. Reranking: Use cross-encoder models for high-precision applications
5. Result Formatting: Include citations and highlights for transparency
6. Performance: Cache embeddings and use appropriate result limits

Next Steps

• Learn about Custom Strategy Development →
• Browse All Available Aktors →
• Explore the Embedding Pipeline →
• View Complete API Reference →