Skip to main content
Version: 2.6

@operaide/vector

The @operaide/vector package provides a comprehensive, strategy-based approach to document embedding and retrieval using vector databases. Built on the Aktor-Reaktor framework, it offers maximum flexibility through pluggable strategies while maintaining strong type safety and performance.

Architecture Overview​

The vectorization system consists of three main components:

  1. Data Schema: Simple 4-table hierarchy for files, documents, chunks, and vectors
  2. Embedding Pipeline: Processes documents from raw files to stored vectors (7 steps)
  3. Retrieval Pipeline: Searches and retrieves relevant documents (6 steps)
  4. File Serving: HTTP endpoint to serve stored files (PDFs, images, etc.) directly from databases

Both pipelines follow the same architectural principles:

  • Strategy Pattern: Each step can be customized with pluggable strategy functions
  • Aktor-based: Reactive, lazy-evaluated data processing
  • Type Safety: Full TypeScript typing with Zod validation
  • Default Implementations: Sensible defaults for immediate use

Core Architecture​

Key Features​

🔧 Extensible Strategy System​

  • Plugin architecture for every processing step
  • Default implementations for immediate use
  • Custom strategies for specialized requirements
  • Direct aktor access for advanced use cases

🚀 High Performance​

  • LibSQL vector extensions with DiskANN indexing
  • Batch processing and concurrent operations
  • Efficient similarity search with cosine distance
  • Comprehensive CRUD operations for all entities

🛡️ Type Safety​

  • Full TypeScript typing throughout
  • Zod schemas for runtime validation
  • Clear interfaces for all strategy types
  • Strong typing for all pipeline results

📊 Rich Metadata​

  • Document-level and chunk-level metadata
  • Flexible JSON storage for custom fields
  • Support for filtering and organization
  • Enhanced metadata through enrichment strategies

🔍 Advanced Retrieval​

  • Vector similarity search with full context
  • Metadata filtering and query optimization
  • Result reranking with multiple strategies
  • LLM-optimized output formatting
  • Direct vector search aktors for custom workflows

🔧 Comprehensive CRUD​

  • Full Create, Read, Update, Delete operations
  • Batch operations for efficient processing
  • Upsert logic for handling duplicates
  • Cascade deletion for data consistency
  • Error handling and recovery

Installation​

npm install @operaide/vector

Quick Start​

import { 
    // Pipeline functions
    aktorVektorEmbeddingPipeline,
    aktorVektorRetrievalPipeline,
    // Storage aktors
    aktorStoreMultipleEmbeddedChunkVectors,
    aktorSearchSimilarVectorsWithChunks,
    // CRUD aktors
    aktorCreateVector,
    aktorGetVectorsByDocumentId,
    aktorDeleteVectorsByDocumentId,
    // Optional strategies
    aktorAzureDocumentIntelligenceLoaderStrategy,
    aktorMarkdownChunkingStrategy,
} from '@operaide/vector';

// Store documents using the pipeline
const embeddingResult = await aktorVektorEmbeddingPipeline({
    file: singleFile,
    client: dbClient,
}).get();

console.log(`Stored ${embeddingResult.storedVectors.length} vectors`);

// Search documents using the pipeline
const searchResult = await aktorVektorRetrievalPipeline({
    client: dbClient,
    query: queryString,
    limit: limitNumber,
}).get();

console.log(`Found ${searchResult.results.length} relevant documents`);

// Direct vector operations
const vectors = await aktorGetVectorsByDocumentId({
    client: dbClient,
    documentId: 'doc_123',
}).get();

// Direct search operations
const searchResults = await aktorSearchSimilarVectorsWithChunks({
    client: dbClient,
    queryEmbedding: [0.1, 0.2, 0.3, /* ... */],
    limit: 5,
}).get();

Strategy Flexibility​

Each pipeline step can be customized:

// Use default strategies
const results = aktorVektorEmbeddingPipeline({
    file: singleFile,
    client: dbClient,
});

// Override specific strategies
const results = aktorVektorEmbeddingPipeline({
    file: singleFile,
    client: dbClient,
    documentLoaderStrategy: aktorAzureDocumentIntelligenceLoaderStrategy,
    chunkingStrategy: aktorMarkdownChunkingStrategy,
    embeddingStrategy: customOpenAIEmbedder,
});

Available Strategies​

Embedding Pipeline Strategies​

  • Document Loaders:
    • aktorDefaultDocumentLoaderStrategy - Basic file to document conversion
    • aktorAzureDocumentIntelligenceLoaderStrategy - Advanced document parsing with Azure AI
  • Text Cleaners: aktorDefaultTextCleanerStrategy - Removes unnecessary whitespace and formatting
  • Chunking Strategies:
    • aktorDefaultChunkingStrategy - Fixed-size chunking with overlap
    • aktorMarkdownChunkingStrategy - Advanced token-aware chunking for Markdown documents with multi-level splitting
  • Metadata Enrichers: aktorDefaultMetadataEnricherStrategy - Adds document and chunk metadata
  • Embedding Generators: aktorDefaultEmbeddingStrategy - OpenAI embeddings generation
  • Post Processors: aktorDefaultPostProcessorStrategy - Final processing and validation

Retrieval Pipeline Strategies​

  • Query Preprocessing: aktorDefaultQueryPreprocessingStrategy - Cleans and normalizes queries
  • Query Embedding: aktorDefaultQueryEmbeddingStrategy - Converts queries to embeddings
  • Vector Search: aktorDefaultVectorSearchStrategy - LibSQL vector similarity search
  • Metadata Filtering: aktorDefaultMetadataFilterStrategy - Filters results by metadata
  • Reranking: aktorDefaultRerankingStrategy - Reorders results by relevance
  • Post Processing: aktorDefaultRetrievalPostProcessingStrategy - Formats results for LLM consumption

Getting Started​

  1. Add Documents: Use the embedding pipeline to process and store documents
  2. Search Documents: Use the retrieval pipeline to find relevant content
  3. Customize Strategies: Override default behaviors with custom implementations

Next Steps​