Skip to content
rlm-rs

ADR-010: Switch to BGE-M3 Model

ADR-010: Switch to BGE-M3 Embedding Model

Section titled “ADR-010: Switch to BGE-M3 Embedding Model”

Status

Section titled “Status”

Accepted

Context

Section titled “Context”

Background and Problem Statement

Section titled “Background and Problem Statement”

The original embedding model choice (all-MiniLM-L6-v2) was made for its small size and fast inference. However, production usage revealed limitations:

  • 384 dimensions provide lower semantic resolution
  • ~512 token context often truncates larger chunks
  • Multilingual support is limited

BGE-M3 offers significant improvements at the cost of larger model size.

Current Limitations

Section titled “Current Limitations”
  1. Dimension mismatch: 384 dimensions limit semantic expressiveness
  2. Token truncation: ~512 token limit truncates chunks near the 2000-byte default size
  3. Multilingual gaps: MiniLM has limited non-English support

Decision Drivers

Section titled “Decision Drivers”

Primary Decision Drivers

Section titled “Primary Decision Drivers”
  1. Context coverage: BGE-M3’s 8192 token limit covers full chunks without truncation, provided MAX_CHUNK_SIZE is kept low enough for dense text (see “Update” section below)
  2. Embedding quality: 1024 dimensions provide richer semantic representation
  3. Consistency: Full chunk content is embedded, not truncated

Secondary Decision Drivers

Section titled “Secondary Decision Drivers”
  1. Multilingual support: BGE-M3 handles non-English content better
  2. Model maturity: BGE-M3 is well-established in the embedding community
  3. fastembed support: Both models supported by fastembed-rs

Considered Options

Section titled “Considered Options”

Option 1: Switch to BGE-M3

Section titled “Option 1: Switch to BGE-M3”

Description: Replace all-MiniLM-L6-v2 with BGE-M3 embedding model.

Technical Characteristics:

  • 1024 dimensions (vs 384)
  • 8192 token context (vs ~512)
  • ~1.3GB model size (vs ~90MB)
  • Stronger multilingual support

Advantages:

  • Full chunk coverage without truncation (given a chunk-size ceiling sized to the token limit — see “Update” section below)
  • Higher semantic resolution
  • Better multilingual embeddings
  • More accurate semantic search

Disadvantages:

  • Larger model download (~1.3GB vs ~90MB)
  • Slightly slower inference
  • Breaking change: requires schema migration
  • Existing embeddings must be regenerated

Risk Assessment:

  • Technical Risk: Low. fastembed-rs supports BGE-M3 well
  • Schedule Risk: Low. Simple model swap
  • Ecosystem Risk: Low. Well-established model

Option 2: Keep all-MiniLM-L6-v2

Section titled “Option 2: Keep all-MiniLM-L6-v2”

Description: Maintain current model.

Technical Characteristics:

  • 384 dimensions
  • ~512 token context
  • ~90MB model size

Advantages:

  • Smaller model download
  • Faster inference
  • No migration needed

Disadvantages:

  • Continued truncation issues
  • Lower semantic quality
  • Limited multilingual support

Disqualifying Factor: Token truncation undermines semantic search quality for typical chunk sizes.

Risk Assessment:

  • Technical Risk: None. No change
  • Schedule Risk: None. No change
  • Ecosystem Risk: Low. Status quo

Option 3: External Embedding API

Section titled “Option 3: External Embedding API”

Description: Switch to OpenAI or similar API embeddings.

Technical Characteristics:

  • API-based embedding generation
  • Higher quality models available
  • Requires network and API key

Advantages:

  • Access to latest models
  • No local model storage

Disadvantages:

  • Network dependency
  • Privacy concerns
  • API costs
  • Conflicts with offline-first design

Disqualifying Factor: Violates offline-first and privacy principles established in ADR-007.

Risk Assessment:

  • Technical Risk: Low. APIs are simple
  • Schedule Risk: Low. Easy integration
  • Ecosystem Risk: High. API dependency

Decision

Section titled “Decision”

Switch from all-MiniLM-L6-v2 to BGE-M3 as the default embedding model.

The implementation will:

  • Change EmbeddingModel::AllMiniLML6V2 to EmbeddingModel::BGEM3
  • Update DEFAULT_DIMENSIONS from 384 to 1024
  • Add schema migration (v2→v3) to clear incompatible embeddings
  • Keep model download silent (existing behavior)

Consequences

Section titled “Consequences”

Positive

Section titled “Positive”
  1. Full chunk coverage: 8192 tokens handles any reasonable chunk size, provided MAX_CHUNK_SIZE is kept low enough for dense text (see “Update” section below)
  2. Better search quality: 1024 dimensions capture more semantic nuance
  3. Multilingual improvement: Better handling of non-English content
  4. Future-proof: More headroom for chunk size increases

Negative

Section titled “Negative”
  1. Breaking change: Existing embeddings incompatible (different dimensions)
  2. Larger download: ~1.3GB model vs ~90MB
  3. Slower inference: Larger model has higher compute cost
  4. Migration required: Users must regenerate embeddings after upgrade

Neutral

Section titled “Neutral”
  1. Model download timing: Same lazy loading pattern as before

Decision Outcome

Section titled “Decision Outcome”

BGE-M3 provides a significant quality improvement for semantic search. The migration clears existing embeddings, requiring users to re-embed their content, but this is a one-time cost for lasting quality improvements.

Mitigations:

  • Schema migration (v2→v3) automatically clears old embeddings
  • Clear error messages guide users to re-embed
  • Document migration in CHANGELOG
  • Lazy loading preserves cold start for non-embedding operations

Update (2026-07-14, issue #313)

Section titled “Update (2026-07-14, issue #313)”

The “full chunk coverage without truncation” claims above assumed MAX_CHUNK_SIZE (the only enforced ceiling on chunk size, in src/chunking/mod.rs, enforced in UTF-8 bytes via text.len() and byte-offset slicing — equivalent to characters for ASCII text) was itself small enough to guarantee chunks stay under BGE-M3’s 8192-token limit. That was not the case: at the original MAX_CHUNK_SIZE = 50_000 bytes, dense text (source code, CJK) can consume more tokens per byte than the ratio assumed for English prose — this project’s own estimate_tokens_for_text heuristic (src/core/chunk.rs) weights non-ASCII text more heavily per character than ASCII — so chunks well under the 50k-byte ceiling could still exceed 8192 tokens and be silently truncated by fastembed’s tokenizer on embed — the same class of silent-truncation problem this ADR was written to eliminate, just recurring at a higher, less obvious threshold.

MAX_CHUNK_SIZE has been lowered to 24_000 bytes (~8k tokens at a conservative ~3 bytes/token), which comfortably covers source code and typical mixed-language content. This is a heuristic, not a hard guarantee: non-ASCII/CJK-dense content can consume more tokens per byte than ASCII, and could in principle still exceed 8192 tokens at the new ceiling. Closing that gap fully would require token-aware validation at chunk time (issue #313’s suggested fix option 1, using the existing Chunk::estimate_tokens()/estimate_tokens_accurate()), which was deliberately deferred in favor of this smaller, immediate fix. The “full chunk coverage without truncation” claims above should be read with that caveat.

Section titled “Related Decisions”
Section titled “Links”

More Information

Section titled “More Information”
  • Date: 2025-01-20
  • Source: Production usage feedback and quality analysis
  • Related ADRs: ADR-007, ADR-008

Audit

Section titled “Audit”

2025-01-20

Section titled “2025-01-20”

Status: Compliant

Findings:

FindingFilesLinesAssessment
BGE-M3 model configuredsrc/embedding/fastembed_impl.rsL66compliant
DEFAULT_DIMENSIONS = 1024src/embedding/mod.rsL27compliant
Schema version bumped to 3src/storage/schema.rsL6compliant
Migration clears embeddingssrc/storage/schema.rsL179-183compliant

Summary: BGE-M3 model switch fully implemented with schema migration.

Action Required: None