Struct DeduplicationService 

pub struct DeduplicationService<E: Embedder + Send + Sync, V: VectorBackend + Send + Sync> {
    config: DeduplicationConfig,
    exact_match: ExactMatchChecker,
    semantic: Option<SemanticSimilarityChecker<E, V>>,
    recent: RecentCaptureChecker,
    domain: Domain,
}

Service for deduplication checking.

Orchestrates three-tier deduplication:

1. Exact match (fastest) - SHA256 hash lookup
2. Semantic similarity - Embedding cosine similarity
3. Recent capture - In-memory LRU cache

Uses short-circuit evaluation: stops on first match.

Example

use subcog::services::deduplication::{DeduplicationService, DeduplicationConfig};
use subcog::services::recall::RecallService;
use subcog::embedding::FastEmbedEmbedder;
use subcog::storage::vector::UsearchBackend;
use std::sync::Arc;

let recall = Arc::new(RecallService::default());
let embedder = Arc::new(FastEmbedEmbedder::new());
let vector = Arc::new(UsearchBackend::in_memory(384));
let config = DeduplicationConfig::default();

let service = DeduplicationService::new(recall, embedder, vector, config);

let result = service.check_duplicate("Use PostgreSQL", Namespace::Decisions)?;
if result.is_duplicate {
    println!("Duplicate found: {:?} - {}", result.reason, result.matched_urn.unwrap());
}

Fields

config: DeduplicationConfig

Configuration.

exact_match: ExactMatchChecker

Exact match checker.

semantic: Option<SemanticSimilarityChecker<E, V>>

Semantic similarity checker (optional - may be disabled).

recent: RecentCaptureChecker

Recent capture checker.

domain: Domain

Domain for URN construction.

Implementations

impl<E: Embedder + Send + Sync, V: VectorBackend + Send + Sync> DeduplicationService<E, V>

pub fn new( recall: Arc<RecallService>, embedder: Arc<E>, vector: Arc<V>, config: DeduplicationConfig, ) -> Self

Creates a new deduplication service with all checkers.

Arguments

• recall - RecallService for exact match searches
• embedder - Embedder for semantic similarity
• vector - VectorBackend for semantic similarity searches
• config - Configuration including thresholds

pub fn without_embeddings( recall: Arc<RecallService>, config: DeduplicationConfig, ) -> Self

Creates a service without semantic checking.

Useful when embeddings are unavailable or disabled. Only performs exact match and recent capture checks.

Arguments

• recall - RecallService for exact match searches
• config - Configuration

pub fn with_domain(self, domain: Domain) -> Self

Sets the domain for URN construction.

fn domain_string(&self) -> String

Returns the domain string for URN construction.

fn check_exact_match( &self, content: &str, namespace: Namespace, domain: &str, start: Instant, ) -> Option<DuplicateCheckResult>

Performs exact match check.

fn check_semantic( &self, content: &str, namespace: Namespace, domain: &str, start: Instant, ) -> Option<DuplicateCheckResult>

Performs semantic similarity check.

fn check_recent( &self, content: &str, namespace: Namespace, start: Instant, ) -> Option<DuplicateCheckResult>

Performs recent capture check.

fn record_unique_check_metrics(&self, namespace: Namespace, duration_ms: u64)

Records final metrics for a unique check.

pub fn check( &self, content: &str, namespace: Namespace, ) -> Result<DuplicateCheckResult>

Checks if content is a duplicate.

Performs checks in order: exact match → semantic → recent capture. Returns early on first match (short-circuit evaluation).

Arguments

• content - The content to check
• namespace - The namespace to check within

Returns

A DuplicateCheckResult with match details.

Errors

Returns an error if a check fails. Individual check failures are handled gracefully (logged and skipped).

pub fn record_capture( &self, content: &str, memory_id: &MemoryId, namespace: Namespace, )

Records a successful capture for future duplicate detection.

Should be called after a memory is successfully captured to enable recent-capture detection.

Arguments

• content - The captured content
• memory_id - The ID of the captured memory
• namespace - The namespace the content was captured to

pub fn record_capture_by_hash( &self, content_hash: &str, memory_id: &MemoryId, namespace: Namespace, )

Records a capture by content hash.

Useful when the hash has already been computed.

Arguments

• content_hash - The pre-computed content hash
• memory_id - The ID of the captured memory
• namespace - The namespace the content was captured to

pub fn content_to_tag(content: &str) -> String

Returns the hash tag for content.

This tag should be added to the memory’s tags during capture to enable future exact-match detection.

pub fn hash_content(content: &str) -> String

Returns the content hash for the given content.

pub const fn is_enabled(&self) -> bool

Returns true if deduplication is enabled.

pub fn get_threshold(&self, namespace: Namespace) -> f32

Returns the configured threshold for a namespace.

Trait Implementations

impl<E: Embedder + Send + Sync, V: VectorBackend + Send + Sync> Deduplicator for DeduplicationService<E, V>

Implementation of the Deduplicator trait.

fn check_duplicate( &self, content: &str, namespace: Namespace, ) -> Result<DuplicateCheckResult>

Checks if content is a duplicate.

fn record_capture(&self, content_hash: &str, memory_id: &MemoryId)

Records a successful capture for recent-capture tracking.