Text Search Library

Full-text search library for GreyCat providing 15 search modes (plus a multi-field BM25F variant and a batch helper), multi-language support (33 languages), and C-accelerated hot paths. The main entry point is TextIndex<T>, a generic type where T is the value stored alongside each document.

Quick Start

@library("text_search", "0.0.0");

// 1. Create an index. The default config is a sensible BM25 + stop words setup.
var index = TextIndex<String> { config: TextIndexConfig {} };

// 2. Add documents: add(text, value) returns the allocated doc-id (int)
var id1 = index.add("Machine learning is a subset of artificial intelligence", "doc1");
var id2 = index.add("Neural networks learn patterns from data", "doc2");
var id3 = index.add("Natural language processing enables text understanding", "doc3");

// 3. Build the index (required before searching)
index.build();

// 4. Search
var results = index.search_bm25("machine learning", 10);
for (var i = 0; i < results.size(); i++) {
    var r = results[i];
    info("${r.value} score=${r.score} id=${r.id}");
}

Pipeline

Every usage follows the same three-step pattern:

1. Add documents with add(text, value) (returns the allocated doc-id int), add_batch(entries) (returns Array<int>), or add_fields(value) (typed multi-field with field references on the document type; supports T or node<T>)
2. Build the index with build() – computes IDF, TF caches, tries, and posting arrays
3. Search with any of the search methods listed below

Use add() after build() for incremental updates. For deletions/updates use remove(id) and update(id, value), where id is the doc-id returned by add(). Documents may also carry a caller-supplied identity: get_by_external_id(externalId) and remove_by_external_id(externalId) work by that key.

Choosing a Search Mode

Storage-efficiency gates. Some modes depend on optional index structures that are off by default (see Configuration). Phrase, Proximity, and Span require storePositions: true on the config — without it they return empty. Fuzzy is accelerated by buildTrigram: true, Prefix and trailing-wildcard by buildTrie: true, and leading-wildcard (*bar) by buildReverseTrie: true; without those flags the mode still returns the same results via a slower vocabulary scan. WAND block-max pruning requires buildBlockMax: true (speed only, result-identical).

For per-mode details, scoring formulas, and examples see Search Modes.

Key Features

• Faceted Search & Aggregations – Term facets, numeric range facets, metric/histogram aggregations
• Function Scoring & Curation – Decay functions, field value factors, document pinning/boosting/suppression, ranking rules
• Percolation (Reverse Search) – Register queries, match incoming documents against them
• Text Processing Utilities – Tokenizer, parser, chunker for preprocessing pipelines
• Filters, Sort & Cursor Pagination – Per-query termFilters / rangeFilters over typed document fields, stable multi-clause sort (overrides score order), and searchAfter cursor pagination — all on SearchOptions
• Multi-Fragment Highlighting – SnippetOptions.numFragments / fragmentSize surface several highlighted passages on Snippet.fragments
• Federated Merge – FederatedSearch::merge fuses ranked result lists from multiple shards/indexes into one top-k (dedup by id, RRF / linear)

Utility Methods

All document-addressed helpers take the integer doc-id returned by add().

var _snippet = index.snippet(id1, "machine learning", null);     // Snippet? { text, highlighted }
var _explanation = index.explain("machine learning", id1);       // ScoreExplanation?
var _suggestions = index.suggest("mach", 5);                     // Array<Suggestion>
var _correction = index.did_you_mean("machin lerning");            // DidYouMeanResult
var _similar = index.more_like_this(id1, 10, null);               // Array<TextResult>
var _stats = index.stats();                                       // TextIndexStats

snippet() returns a Snippet? { text, highlighted } (null when the id is unknown or has no match). Use index.snippets(ids, query, options) (ids: Array<int>) for batched extraction; it returns Map<int, Snippet> keyed by doc-id.

See Utility Methods for full API details.

Configuration

TextIndexConfig is fully optional. All fields are nullable. Standalone fields stay at the top level; everything else is grouped into nested option blocks.

// Minimal: tweak BM25 only
var cfg = TextIndexConfig {
    bm25: BM25Options { k1: 1.5, b: 0.75 }
};

// Combine a few blocks
var cfg2 = TextIndexConfig {
    usePhonetic: true,
    synonyms: synonymMap,
    tokenization: TokenizationOptions {
        stemming: true,
        normOptions: NormOptions { stripHtmlTags: true, stripUrls: true, stripAccents: true }
    },
    stopWords: StopWordOptions {
        mode: StopWordMode::default,
        language: TextSearchLanguage::en
    },
    typoTolerance: TypoOptions { enabled: true }
};

Standalone fields: embed, synonyms, fields, deduplicateContent, dedupBodies, fuzzyMaxTextLength, usePhonetic.

Body pooling (dedupBodies: bool?, default OFF): when on, identical normalized document bodies are pooled behind their SHA-256 so duplicates share one stored body node; when off (the default) each document gets its own body node and the per-add body hash is skipped. Search results are identical either way — turn it on only for duplicate-heavy corpora. remove() / update() note: with storePositions off (the default) there is no per-document forward index, so remove() unlinks a doc by scanning the vocabulary (O(total postings)); enable storePositions for O(doc-terms) removals if you churn heavily.

Nested option blocks: tokenization, stopWords, bm25, fusion, typoTolerance, edgeNgram, shortCircuit, diversify, chunking, dfr, lmDirichlet, highlight.

Storage-efficiency gates (all bool?, default OFF — build only what you query): buildTrigram, buildTrie, buildReverseTrie, buildBlockMax, storePositions, storeRawText, keepOriginalForm. These opt-in flags control whether optional index structures are built and persisted. A bare TextIndexConfig {} or a pure BM25-keyword index pays zero disk for fuzzy/prefix/wildcard/phrase/WAND scaffolding. Enabling a gate restores the full behavior for its feature (see the table below). Changing an existing index’s gates requires a rebuild (wipe + reindex).

For hybrid fusion weights, build a map and pass it via fusion.weights:

var w = Map<SearchMode, float> {};
w.set(SearchMode::bm25, 0.7);
w.set(SearchMode::semantic, 0.3);
var _cfg = TextIndexConfig {
    fusion: FusionOptions { method: FusionMethod::rrf, weights: w }
};

See Configuration Reference for every field.

Presets

Static factories on TextIndexConfig produce ready-made configurations. Use them directly or as a starting point.

var _index = TextIndex<String> { config: TextIndexConfig::keyword() };

See Presets for full preset bodies and tuning notes.

Documentation

Supported Languages

33 languages for stop words and text processing:

Arabic, Bulgarian, Catalan, Chinese, Czech, Danish, Dutch, English, Finnish, French, German, Greek, Gujarati, Hebrew, Hindi, Hungarian, Indonesian, Italian, Japanese, Korean, Malay, Norwegian, Persian, Polish, Portuguese, Romanian, Russian, Slovak, Spanish, Swedish, Turkish, Ukrainian, Vietnamese.