rag

rag

A Go library for multi-modal Retrieval-Augmented Generation with graph-enhanced retrieval, part of the saige SDK.

What it does

saige/rag models documents as hierarchical structures (Document -> Section -> ContentVariant) where each section can have multiple modality representations (text, image, table, audio). It provides a pluggable pipeline for ingesting documents, generating embeddings, performing hybrid search (vector + BM25 + graph), reranking, and assembling context with citations.

When to use

• Ingesting documents (PDF, text, images) into a searchable vector store
• Building RAG pipelines with multi-modal content support
• Combining vector search with knowledge graph traversal
• Evaluating retrieval and generation quality
• Deduplicating documents by content fingerprint

Usage

Install

go get github.com/urmzd/saige

Create a pipeline

import (
    "github.com/urmzd/saige/rag"
    "github.com/urmzd/saige/rag/pgstore"
    "github.com/urmzd/saige/rag/types"
    "github.com/urmzd/saige/postgres"
)

// Connect to PostgreSQL 18 (requires the pgvector and pg_search extensions).
pool, _ := postgres.NewPool(ctx, postgres.Config{URL: "postgres://localhost:5432/mydb"})
postgres.RunMigrations(ctx, pool, postgres.MigrationOptions{})

store, _ := pgstore.New(pgstore.Config{Pool: pool})
pipe, err := rag.New(rag.Config{Store: store},
    rag.WithContentExtractor(myExtractor),
    rag.WithEmbedders(myEmbedderRegistry),
    rag.WithRecursiveChunker(512, 50),
    rag.WithBM25(nil),
    rag.WithMMR(0.7),
)

Ingest a document

result, err := pipe.Ingest(ctx, &types.RawDocument{
    SourceURI: "https://example.com/paper.pdf",
    MIMEType:  "application/pdf",
    Data:      pdfBytes,
})

Search

results, err := pipe.Search(ctx, "attention mechanism", types.WithLimit(5), types.WithContextAssembly(4096))
fmt.Println(results.Context.Prompt) // context with citations

With knowledge graph integration

pipe, err := rag.New(rag.Config{Store: store},
    rag.WithContentExtractor(extractor),
    rag.WithGraph(kgGraph),  // enables entity extraction + graph retrieval
)

Key interfaces

Interface	Purpose
Pipeline	Orchestrate ingest, search, update, delete, reconstruct
Store	Document CRUD + vector search
ContentExtractor	Raw bytes -> structured Document
Chunker	Split long sections
EmbedderRegistry	Dispatch embedding by ContentType

Configuration options

Option	Purpose
WithStore(s)	Set the document store, or set rag.Config.Store (required)
WithContentExtractor(ext)	Set the content extractor (required)
WithRecursiveChunker(max, overlap)	Recursive text chunking
WithSemanticChunker(thresh, min, max)	Semantic similarity chunking
WithEmbedders(reg)	Set the embedder registry (optional, needed for search)
WithGraph(g)	Enable knowledge graph entity extraction (optional)
WithBM25(cfg)	Enable BM25 lexical retrieval. Over pgstore it searches the store's pg_search index; over a store without keyword search the index is in memory, so call rag.RebuildIndex(ctx, pipe) once after rag.New when the store already holds documents
WithParentContext()	Expand hits to parent section context
WithMMR(lambda)	MMR diversity reranking
WithCrossEncoder(scorer)	Cross-encoder reranking
WithHyDE(llm, n)	HyDE query expansion
WithCompression(llm)	LLM-based context compression
WithDedupBehavior(b)	Set dedup behavior: DedupSkip (default) or DedupReplace
WithStoreOriginals(true)	Persist raw document bytes

Search options: types.WithCandidatePool(k) sets how many fused candidates reach the reranker, and types.WithMinScore(s) is a threshold in [0, 1] on the normalized fused score (1.0 keeps only hits ranked first by every retriever), applied before reranking.

Evaluation

import "github.com/urmzd/saige/rag/eval"

// Retrieval metrics (pure functions).
precision := eval.ContextPrecision(hits, relevantUUIDs)
recall := eval.ContextRecall(hits, relevantUUIDs)
ndcg := eval.NDCG(hits, relevantUUIDs, 10)
mrr := eval.MRR(hits, relevantUUIDs)

// Generation metrics (require LLM).
faith, _, _ := eval.Faithfulness(ctx, response, contextText, llm)
correctness, _ := eval.AnswerCorrectness(ctx, response, groundTruth, llm)

Agent tool bindings

import "github.com/urmzd/saige/rag/tool"

tools := tool.NewTools(pipeline)
// rag_search, rag_lookup, rag_update, rag_delete, rag_reconstruct

CLI

saige knowledge ingest --db "$SAIGE_RAG_DB" --file paper.pdf --mime application/pdf
saige knowledge sync --db "$SAIGE_RAG_DB" --dir docs --include '**/*.md' --exclude 'drafts/**' --prune
saige knowledge search --db "$SAIGE_RAG_DB" --query "attention mechanism"
saige knowledge show --db "$SAIGE_RAG_DB" --uuid <variant-uuid>
saige knowledge delete --db "$SAIGE_RAG_DB" --uuid <doc-uuid>