Vector DB & Deep RAG Expert

English | Bahasa Indonesia

---

English

Purpose & Overview

Production-grade guidelines for Vector Databases (pgvector, Qdrant, Pinecone, Milvus), RAG indexing strategies, HNSW vector search, hybrid retrieval (dense vector embeddings + BM25 sparse keyword ranking), semantic document chunking, and RAG evaluation frameworks.

Key Capabilities

• pgvector & Hybrid Search: PostgreSQL pgvector HNSW indexing, cosine/L2 distance metric tuning, and BM25 hybrid re-ranking.
• RAG Architecture: Parent-Document retrieval, Hypothetical Document Embeddings (HyDE), and contextual compression.
• RAG Evaluation: Automated retrieval quality scoring using Ragas and TruLens.

import { sql } from 'drizzle-orm';

// Hybrid Search: Vector Cosine Similarity + Full Text Search
export async function hybridSearch(queryVector: number[], queryText: string, limit = 10) {
  const result = await db.execute(sql`
    SELECT id, title, content,
           (1 - (embedding <=> ${JSON.stringify(queryVector)}::vector)) * 0.7 +
           ts_rank(fts, websearch_to_tsquery('english', ${queryText})) * 0.3 AS score
    FROM documents
    ORDER BY score DESC
    LIMIT ${limit};
  `);
  return result;
}

Implementation Checklist

• Enable pgvector extension in PostgreSQL and create an hnsw index on the embedding column.
• Implement Semantic Chunking (breaking documents by semantic boundaries rather than fixed character lengths).
• Combine Vector Cosine Similarity with Full Text Search (BM25) using a weighted score (Hybrid Search).
• Generate Hypothetical Document Embeddings (HyDE) to improve retrieval recall.

Orchestration & Integration

• Integrates with: ai-llm-integration-expert, database-orm-expert, app-analyzer-optimizer.

---

Bahasa Indonesia

Deskripsi

Panduan tingkat produksi untuk Vector Database (pgvector, Qdrant, Pinecone, Milvus), arsitektur RAG, indeks pgvector HNSW, hybrid search (dense + BM25 sparse re-ranking), semantic chunking, dan evaluasi RAG.

Fitur Utama

• pgvector & Hybrid Search: PostgreSQL pgvector HNSW indexing, tuning jarak cosine/L2, dan re-ranking BM25.
• Arsitektur RAG: Retrieval Parent-Document, HyDE (Hypothetical Document Embeddings), dan kompresi kontekstual.
• Evaluasi RAG: Scoring kualitas retrieval otomatis menggunakan Ragas dan TruLens.

Checklist Implementasi

• Aktifkan ekstensi pgvector di PostgreSQL dan buat indeks hnsw pada kolom embedding.
• Terapkan Semantic Chunking (memecah dokumen berdasarkan batas semantik alih-alih panjang karakter tetap).
• Gabungkan Vector Cosine Similarity dengan Full Text Search (BM25) menggunakan skor berbobot (Hybrid Search).
• Hasilkan Hypothetical Document Embeddings (HyDE) untuk meningkatkan recall retrieval.

Integrasi Orkestrasi

• Terintegrasi dengan: ai-llm-integration-expert, database-orm-expert, app-analyzer-optimizer.