# Indexing Methods

Methods for adding, removing, updating, and building the `TextIndex<T>`.

---

## add()

Add a single document to the index.

```gcl
fn add(text: String, value: T): int
```

**Parameters**

| Parameter | Type | Description |
|-----------|------|-------------|
| `text` | `String` | Document text content to index (the text that will be searched against) |
| `value` | `T` | Associated data to store with the document (returned in search results, typically a document ID) |

**Returns:** The allocated doc-id — a stable, never-reused integer index into the index. (When `deduplicateContent` is enabled and the normalized body is a duplicate, the existing doc-id is returned and no new entry is created.)

**Example**

```gcl
// text = content to index, value = associated identifier
var docId1 = index.add("Machine learning algorithms", "doc1");
var docId2 = index.add("Natural language processing", "doc2");
```

**Notes**

- Normalizes text, tokenizes, builds inverted index
- With `dedupBodies` enabled (default OFF), identical normalized bodies are pooled behind a shared `node<String>` (the `textPool`); otherwise each document gets its own body node
- Embeds document if `embed` function configured
- Skips duplicates if `deduplicateContent` enabled
- Persists per-document positional arrays (`entryTerms` / `positionData` / `positionOffsets` / `positionCounts`) **only when `config.storePositions` is set**; the original-case body (`rawText`) **only when `config.storeRawText` is set**; and pre-stemmed term casing (`NormalizedTerm.originalForm`) **only when `config.keepOriginalForm` is set**. All three default OFF, so a bare/keyword config pays no disk for them. See [TextIndexConfig](./types.md) storage-efficiency gates.
- Must call `build()` before searching

To attach a caller-supplied identity to the document, use the `externalId`
field of `TextEntry` via `add_batch()`, or look documents up later with
`get_by_external_id()` / `remove_by_external_id()`.

---

## add_batch()

Add multiple documents in a single operation.

```gcl
fn add_batch(batchEntries: Array<TextEntry>): Array<int>
```

**Parameters**

| Parameter | Type | Description |
|-----------|------|-------------|
| `batchEntries` | `Array<TextEntry>` | Array of `TextEntry` with `key`, `value`, optional `externalId`, optional `vector` |

**Returns:** The allocated doc-ids, one per input entry (parallel to `batchEntries`).

**Example**

```gcl
var entries = Array<TextEntry> {};
entries.add(TextEntry { key: "doc1 text", value: "doc1", vector: null });
entries.add(TextEntry { key: "doc2 text", value: "doc2", externalId: "sku-42", vector: precomputedTensor });
var ids = index.add_batch(entries);
```

**Notes**

- Pre-computed vectors skip embedding inference
- `externalId` registers a caller-supplied identity for `get_by_external_id()` / `remove_by_external_id()`
- Useful for bulk imports

---

## add_fields()

Add a typed document. Field text is read off the document via the configured
`FieldConfig.f` references — or, if `config.fields` is null, auto-discovered
on the document type. `T` may be the document type directly or `node<T>`;
node resolution is transparent.

```gcl
fn add_fields(value: T)
```

**Parameters**

| Parameter | Type | Description |
|-----------|------|-------------|
| `value` | `T` | Typed document (or `node<T>`) to index |

**Example**

```gcl
type Article { title: String; body: String; tags: String?; }

var index = TextIndex<Article> {
    config: TextIndexConfig {
        fields: [
            FieldConfig { f: Article::title, weight: 3.0 },
            FieldConfig { f: Article::body,  weight: 1.0 },
            FieldConfig { f: Article::tags,  weight: 2.0 }
        ]
    }
};
index.add_fields(Article {
    title: "Machine Learning",
    body: "Algorithms for learning from data",
    tags: "ml ai data-science"
});
```

**Notes**

- When `config.fields` is null, `add_fields` auto-discovers every `String` /
  `String?` field on `T` at weight 1.0 on the first call.
- Empty `config.fields` raises `"TextIndexConfig.fields is set but empty"`.
- A `T` with no `String` fields and no explicit `config.fields` raises
  `"add_fields: no String fields found on T"`.
- Per-field text is read directly off `IndexEntry.value` at query time —
  no separate side index.

---

## remove()

Remove a document from the index by its doc-id.

```gcl
fn remove(id: int)
```

**Parameters**

| Parameter | Type | Description |
|-----------|------|-------------|
| `id` | `int` | Doc-id to remove (the value returned by `add()`) |

**Example**

```gcl
var docId = index.add("Machine learning algorithms", "doc1");
index.remove(docId);
```

**Notes**

- Tombstones the entry — the id stays stable and is never reused
- Unlinks all term associations. With `storePositions` on, this uses the per-document forward index (O(doc-terms)); with it off (the default) it falls back to a vocabulary scan (O(total postings)). Both are correct — enable `storePositions` for cheaper removals under heavy churn.
- Safe to call on an already-removed id (no-op)
- Updates statistics (totalEntries, avgTokenCount)
- IDF scores remain unchanged (call `build()` again for exact IDF)

To remove by caller-supplied identity instead, use
`remove_by_external_id(externalId: String)`.

---

## update()

Update a document in place by re-indexing under the same doc-id with a new value.

```gcl
fn update(id: int, value: T)
```

**Parameters**

| Parameter | Type | Description |
|-----------|------|-------------|
| `id` | `int` | Doc-id to update (the value returned by `add()`) |
| `value` | `T` | New value |

**Example**

```gcl
var docId = index.add("Machine learning algorithms", "doc1");
index.update(docId, "new_value");
```

**Notes**

- Re-indexes the document's text under the same doc-id (the id is preserved)
- Internally removes and re-adds the entry, reusing the original text and `externalId`
- No-op if the doc-id does not exist

---

## build()

Finalize the index after adding documents.

```gcl
fn build()
```

**Example**

```gcl
index.add("Machine learning algorithms", "value1");
index.add("Natural language processing", "value2");
index.build();  // Compute IDF, detect auto stop words
```

**Notes**

- **Required before searching**
- Computes IDF scores for all terms
- Auto-detects stop words (if configured)
- Computes average document length
- Safe to call multiple times: re-running `build()` after additional `add()` calls recomputes IDF and refreshes derived structures (TF cache, posting arrays, trigram/edge n-gram/phonetic/trie indices)
- Builds the optional vocabulary-scale structures **only when their gate is set**: the forward trie (`config.buildTrie`, for prefix/trailing-wildcard search), the reverse trie (`config.buildReverseTrie`, for leading-wildcard `*bar`), the trigram index (`config.buildTrigram`, for fuzzy candidate pre-filtering), and per-term block-max score arrays (`config.buildBlockMax`, for WAND pruning). All default OFF; when a gate is unset the corresponding feature falls back to a full vocabulary scan (trie/trigram) or plain BM25 without WAND skipping (block-max) — same results, just slower. See [TextIndexConfig](./types.md) storage-efficiency gates.

---

## Internal build helpers

`build()` orchestrates several lower-level helpers that are exposed on `TextIndex<T>` for advanced use cases (e.g. incremental rebuilds of a single derived structure). Callers should normally use `build()`; only reach for these directly when you need fine-grained control.

| Method | Description |
|--------|-------------|
| `build_internal(isRebuild: bool)` | Core build pass: computes `avgTokenCount`, detects auto stop words, pre-normalizes synonyms, computes IDF and `maxTermScore`, populates posting arrays and the BM25 TF cache |
| `build_trie_index()` | (Re)builds the forward trie used by prefix and wildcard search. `build()` invokes it only when `config.buildTrie` is set. |
| `build_reverse_trie_index()` | (Re)builds the reverse trie used for leading-wildcard patterns like `*ation`. `build()` invokes it only when `config.buildReverseTrie` is set. |
| `build_trigram_index()` | (Re)builds the trigram inverted index used by fuzzy pre-filtering. `build()` invokes it only when `config.buildTrigram` is set. |
| `index_edge_ngrams(token: String, ordinal: int)` | Indexes edge n-grams for a single token by its term ordinal; called from `add()` and `build()` when `edgeNgram.enabled = true` |