Text Search Library

Full-text search library for GreyCat providing 15 search modes (plus a multi-field BM25F variant and a batch helper), multi-language support (33 languages), and C-accelerated hot paths. The main entry point is TextIndex<T>, a generic type where T is the value stored alongside each document.

Quick Start

// 1. Create an index. The default config is a sensible BM25 + stop words setup.
var index = TextIndex<String> { config: TextIndexConfig {} };

// 2. Add documents: add(text, value)
index.add("Machine learning is a subset of artificial intelligence", "doc1");
index.add("Neural networks learn patterns from data", "doc2");
index.add("Natural language processing enables text understanding", "doc3");

// 3. Build the index (required before searching)
index.build();

// 4. Search
var results = index.search_bm25("machine learning", 10);
for (var i = 0; i < results.size(); i++) {
    var r = results[i];
    info("${r.value} score=${r.score}");
}

Pipeline

Every usage follows the same three-step pattern:

1. Add documents with add(key, value), add_batch(entries), or add_fields(value) (typed multi-field with field references on the document type; supports T or node<T>)
2. Build the index with build() – computes IDF, TF caches, tries, and posting arrays
3. Search with any of the search methods listed below

Use add() after build() for incremental updates. For deletions/updates use remove(key) and update(key, value).

Choosing a Search Mode

For per-mode details, scoring formulas, and examples see Search Modes.

Key Features

• Faceted Search & Aggregations – Term facets, numeric range facets, metric/histogram aggregations
• Function Scoring & Curation – Decay functions, field value factors, document pinning/boosting/suppression, ranking rules
• Percolation (Reverse Search) – Register queries, match incoming documents against them
• Text Processing Utilities – Tokenizer, parser, chunker for preprocessing pipelines

Utility Methods

var _snippet = index.snippet("doc1", "machine learning", null);  // Snippet { text, highlighted }
var _explanation = index.explain("machine learning", "doc1");    // ScoreExplanation
var _suggestions = index.suggest("mach", 5);                     // Array<Suggestion>
var _correction = index.did_you_mean("machin lerning");            // DidYouMeanResult
var _similar = index.more_like_this("doc1", 10, null);             // Array<TextResult>
var _stats = index.stats();                                       // TextIndexStats

snippet() returns a Snippet { text, highlighted }. Use index.snippets(keys, query, options) for batched extraction.

See Utility Methods for full API details.

Configuration

TextIndexConfig is fully optional. All fields are nullable. Standalone fields stay at the top level; everything else is grouped into nested option blocks.

// Minimal: tweak BM25 only
var cfg = TextIndexConfig {
    bm25: BM25Options { k1: 1.5, b: 0.75 }
};

// Combine a few blocks
var cfg2 = TextIndexConfig {
    usePhonetic: true,
    synonyms: synonymMap,
    tokenization: TokenizationOptions {
        stemming: true,
        normOptions: NormOptions { stripHtmlTags: true, stripUrls: true, stripAccents: true }
    },
    stopWords: StopWordOptions {
        mode: StopWordMode::default,
        language: TextSearchLanguage::en
    },
    typoTolerance: TypoOptions { enabled: true }
};

Standalone fields: embed, synonyms, fields, deduplicateContent, fuzzyMaxTextLength, usePhonetic.

Nested option blocks: tokenization, stopWords, bm25, fusion, typoTolerance, edgeNgram, shortCircuit, diversify, chunking, dfr, lmDirichlet, highlight.

For hybrid fusion weights, build a map and pass it via fusion.weights:

var w = Map<SearchMode, float> {};
w.set(SearchMode::bm25, 0.7);
w.set(SearchMode::semantic, 0.3);
var _cfg = TextIndexConfig {
    fusion: FusionOptions { method: FusionMethod::rrf, weights: w }
};

See Configuration Reference for every field.

Presets

Static factories on TextIndexConfig produce ready-made configurations. Use them directly or as a starting point.

var _index = TextIndex<String> { config: TextIndexConfig::keyword() };

See Presets for full preset bodies and tuning notes.

Documentation

Supported Languages

33 languages for stop words and text processing:

Arabic, Bulgarian, Catalan, Chinese, Czech, Danish, Dutch, English, Finnish, French, German, Greek, Gujarati, Hebrew, Hindi, Hungarian, Indonesian, Italian, Japanese, Korean, Malay, Norwegian, Persian, Polish, Portuguese, Romanian, Russian, Slovak, Spanish, Swedish, Turkish, Ukrainian, Vietnamese.