postal

Address parsing library based on libpostal, an NLP-based address parser trained on OpenStreetMap and OpenAddresses data.

@library("postal", "0.0.0");

Resource requirements

libpostal loads its NLP models on the first call to parse / parseStructured (a few seconds), not at startup — a program that never parses an address pays nothing. Once loaded, the models stay resident until the process exits.

Resource	Requirement
Disk	~1.4 GB downloaded, ~3 GB extracted (language models and address parser data)
RAM	~2–3 GB (after the first parse)

The model files are not shipped with the library — download them once and point GREYCAT_POSTAL_DATA_DIR at the model directory itself (the one holding transliteration/, address_parser/, …).

The data is made of three archives, versioned independently:

Archive	Version	Size	Contents
libpostal_data.tar.gz	v1.1.0	~10 MB	address_expansions/, numex/, transliteration/
language_classifier.tar.gz	v1.1.0	~50 MB	language_classifier/
parser.tar.gz	v1.2.0	~1.3 GB	address_parser/ — the updated parser model

Download them into the data directory:

mkdir -p <dir> && cd <dir>
curl -sSL https://public-read-libpostal-data.s3.amazonaws.com/v1.1.0/libpostal_data.tar.gz -o libpostal_data.tar.gz
curl -sSL https://public-read-libpostal-data.s3.amazonaws.com/v1.1.0/language_classifier.tar.gz -o language_classifier.tar.gz
curl -sSL https://public-read-libpostal-data.s3.amazonaws.com/v1.2.0/parser.tar.gz -o parser.tar.gz

Then extract them in place and export the directory:

tar -zxvf libpostal_data.tar.gz
tar -zxvf language_classifier.tar.gz
tar -zxvf parser.tar.gz
export GREYCAT_POSTAL_DATA_DIR=<dir>

Without the variable the library starts with the parser disarmed: it prints the commands above and every parse / parseStructured call raises.

Parsing into components

var components = PostalAddress::parse("4 Place de Strasbourg, L-2562 Luxembourg", null);
for (var i = 0; i < components.size(); i++) {
  var c = components[i]!!;
  info("${c.label}: ${c.value}");
}

Structured parsing

parseStructured returns a PostalParsedAddress with typed fields instead of a flat array of components.

var addr = PostalAddress::parseStructured("4 Place de Strasbourg, L-2562 Luxembourg", null);
info(addr.road);      // "place de strasbourg"
info(addr.postcode);  // "l-2562"
info(addr.country);   // "luxembourg"

Language and country hints

Both functions take an optional PostalOptions to hint the parser:

var addr = PostalAddress::parseStructured(
  "4 Place de Strasbourg, L-2562 Luxembourg",
  PostalOptions { language: "fr", country: "lu" }
);

Expanding abbreviations

expand normalizes an address (or a single field, e.g. just the street) into its canonical form(s) using libpostal’s expansion dictionaries — abbreviations (“St” -> “Saint”/“Street”, “Bd” -> “Boulevard”), ordinals (“3rd” -> “3”/“third”), and accents/punctuation. It’s cheaper than parse/parseStructured: it only loads the small (~10MB) address_expansions/numex/transliteration data, not the ~1.3GB parser model, so a program that only calls expand never pays for the parser.

Apply the same expand() call to both the text you index and the query you search with — that converges whichever form each one used (“Av.” vs “Avenue”) onto the same canonical token, and (unlike a search-time synonym map in a full-text engine) works uniformly regardless of which search mode reads the text.

var expansions = PostalAddress::expand("5 Av. St-Michel", PostalExpandOptions { languages: ["fr"] });
// ["5 avenue st-michel", "5 avenue saint michel", "5 avenue stmichel"]

var expansions2 = PostalAddress::expand("12 Bd du Général de Gaulle", PostalExpandOptions { languages: ["fr"] });
// ["12 boulevard du general de gaulle"]

A query almost always has several valid canonical readings (ambiguous abbreviations, alternate accent/spacing forms) — expand returns all of them, not just one, so index/search every element of the result rather than assuming [0] is “the” answer.

Passing no languages lets libpostal auto-detect the language, which additionally requires the language_classifier data (see above) — pass an explicit hint to skip that dependency, or when you already know the language (as in most single-tenant address datasets).

Note this does not replace splitting elision on the tokenizer side: expanding "Rue de l'Église" returns ["rue de l'eglise", "rue de l eglise"] — the elided article stays attached to the noun in both variants, it doesn’t produce a clean "eglise" token. Use expand() for abbreviation/ordinal canonicalization and a tokenizer-level fix (AddressIndex, below, or a hand-rolled TextIndexConfig::address() preset — makes the apostrophe a token separator) for the elision-splitting itself — the two are complementary, not alternatives.

AddressIndex: searchable addresses

AddressIndex (lib/postal/address_index.gcl) combines parseStructured + expand above with a text_search TextIndex to turn a corpus of raw address lines into something you can search — typo-tolerant, elision-aware, abbreviation-insensitive.

var hints = PostalOptions { language: "fr", country: "fr" };
var idx = AddressIndex<String> {
  index: TextIndex<AddressDoc> { config: address_index_config(TextSearchLanguage::fr) },
  lang: TextSearchLanguage::fr
};

idx.add("5 Avenue Saint-Michel, 75006 Paris", hints, "cust-1");
idx.add("3 Rue de l'Eglise, 67000 Strasbourg", hints, "cust-2");
idx.build();

var hits = idx.search("5 Av. St-Michel, Paris", 10, hints);
// idx.payload_of(hits[0])              -> "cust-1"
// idx.doc_of(hits[0])?.houseNumberText -> "5"

An AddressDoc keeps only what the index searches — the parsed components — never the line it was built from. A caller that wants the original line back stores it in payload, the slot the index carries untouched and never indexes.

This requires @library("text_search") as well — see “A note on the text_search dependency” below.

What a hit tells you

Nothing is filtered. Street, city, postal code and house number are all scored text, so a component libpostal misread costs a row its boost rather than its place in the results. Each hit carries two fields the caller reads:

• houseNumberMatch — true when the row carries the number that was asked for, false when it does not, null when the query named no number. A false row is still an answer — a client typing 5 may yet type 50 — but render it as an approximation, never as the address requested. The comparison reads the leading digits, so 5D still answers a query for 5; the suffix ranks rather than decides.
• level — address for the requested number on a street the query named, street for the same street at another number, area for a row sharing only a city or postal code. Geocoders report this as an “accuracy” or a layer.

A caller wanting only exact addresses filters on houseNumberMatch == true.

AddressDoc.houseNumberText keeps the number as libpostal read it, suffix and all. The address preset splits on space, apostrophe and hyphen only, so 5D is indexed as the single term 5d — a different term from 5, which is what puts the row that was asked for above the one next door. geoportail.lu ranks the same way: searching it for 5D, Rue de l'Industrie returns 5D first and the plain 5 ninth.

The trade is at the other end: because 5 and 5d are unrelated terms, a bare 17 gives 17A and 17B no boost from the number — they rank on street and city alone, where geoportail still pulls them up. Over the full national dataset that costs nothing observable (Bertrange has no plain 17 on that street, and 17A comes back first anyway); it shows only on a corpus where another commune holds a literal 17 on a street of the same name, which then outranks the local 17A.

Closing that gap means dropping edgeNgram.min from 3 to 1, so 17 becomes a term of its own. Measured over all 179,491 rows (address_profile::gram_min) it costs 21% on build(), 13% on disk and 11% on ingest, leaves ordinary query latency unchanged, and makes the house number 17 prefix-match the postcode 1752 — a worse number ranking than the one it was meant to fix. Not worth it.

Scoring rather than filtering matters on real input: libpostal reads the bare fragment "Bd Royal, Luxembourg" as road="bd", city="royal". Filtering on that city would exclude every genuine match; scored, “royal” lands in the query text and ranks Boulevard Royal first.

The word the client is still typing

A truncated final word — "5 rue de indus" — is not a word anyone indexed, and Levenshtein reads its missing tail as deletions ("indus" is 4 edits from "industrie"), which no sane budget reaches. address_index_config() sets edgeNgram.indexAsTerms, so every prefix from 3 characters up is a term of its own: "indus" has an idf and ranks like any other term, and search-as-you-type needs no separate pass.

One shape stays outside that: an elision typed without its apostrophe. The tokenizer splits the apostrophe a client types, not the one they omit, so "lindus" fuses into a term of nothing. address_fix_elided_tail() strips the leading article and rewrites the token when the vocabulary answers to the remainder.

What gets indexed is the libpostal EXPANSION of the street, not its printed name: "Z.I. Hahneboesch" is indexed as "zone industrielle hahneboesch", so it legitimately answers a query for "industr".

Two things worth knowing about libpostal itself. A house number or postal code anchors the parser, and a bare fragment often has nothing to split on — parseStructured("5, industrye", ...) returns every field null, so AddressIndex reads the leading number off the query text instead. And Luxembourg the city and Luxembourg the country share a name, so once a postal code anchors an address libpostal often does not tag the trailing “Luxembourg” as city; disambiguate by postal code there, not by city name.

How much of a typo is a typo

The edit budget is Elasticsearch’s fuzziness: AUTO, the yardstick anyone comparing this to geoportail.lu is holding it against: nothing below minWordLength, one edit up to 8 characters, two beyond. TypoOptions expresses that as minWordLength: 3, maxEdits1: 1, maxEdits2: 2.

Distance is plain Levenshtein, not Damerau, so an adjacent transposition costs two edits rather than one: "indsutrye" is 3 edits from "industrie" and does not resolve. A single transposition on a truncated word ("indsu" -> Rue de l’Industrie) still does, because the prefix it corrects onto is itself a term.

test/address_search_test.gcl, test/lu_open_data_test.gcl and test/geoportail_comparison_test.gcl are the suite — 29 tests, including some run against a real subset of Luxembourg’s official open address data and one direct comparison against geoportail.lu’s own search.

Benchmarks

Numbers below are from one dev box (i7-1165G7, 8 threads) over the national BD-Adresses export — 179,491 addresses, indexed in 17.6 s (~10k rows/s) with a 271 ms index build. They are a shape, not a spec: your hardware will differ.

greycat run address_bench::bench_full     # per-query ranks and timings
greycat run address_bench::stages_full    # where the milliseconds go

Where the time goes

Averaged over 21 query shapes, one search() call costs 516 µs, and almost all of it is the index, not the parsing:

stage	mean	share
libpostal parseStructured	22 µs	4%
expand + canonicalize	10 µs	2%
elided-tail rewrite	9 µs	2%
text_search passes	474 µs	92%
end-to-end	516 µs

libpostal is cheap once its models are resident; what a query costs is decided by how many candidate terms it opens up. That is why the spread across shapes is 10× while the parse cost barely moves.

Average speed per query shape

ours is the mean of 7 in-process runs at k=10. geoportail.lu is the median of 7 calls to its public fulltextsearch endpoint from the same machine — a network round-trip, not a comparable measurement: ~30 ms of it is DNS, TCP and TLS, and on a reused connection the same calls land in 10–20 ms. Read the column as “what a browser waits for”, not as the cost of their query.

query shape	query	ours	geoportail.lu
postcode only	1610	0.16 ms	42.8 ms
prefix, still typing	5 indus	0.22 ms	49.3 ms
full address	42 Avenue de la Gare, 1610 Luxembourg	0.32 ms	47.6 ms
number + typo	5 industrye	0.43 ms	46.4 ms
noise, no answer	zzzeldange	0.45 ms	46.2 ms
typo in one word	avenue de la garre luxembourg	0.49 ms	47.3 ms
abbreviated	av de la gare luxembourg	0.60 ms	48.2 ms
street + town	avenue de la gare luxembourg	0.61 ms	46.6 ms
elision, apostrophe omitted	5 rue de lindus	1.09 ms	47.8 ms
wrong house number	999 rue de l'industrie luxembourg	1.52 ms	45.9 ms

Best case is a postal code alone (0.16 ms): one rare term, a tiny candidate set. Worst is a house number nobody indexed on a street thousands of rows share (1.52 ms) — the number contributes nothing selective, so every row on the street stays in the running and the level pass then classifies all of them. The half-typed-elision cases (~1.1 ms) are the next worst for the same reason: a prefix term matches broadly before anything narrows it.

Two rows are worth reading for behaviour rather than speed. On 999 rue de l'industrie luxembourg geoportail returns nothing; this library returns the same-street rows flagged houseNumberMatch: false, which is the fallback described above. On zzzeldange both return nothing, which is the right answer.

How geoportail.lu’s search is configured

geoportail.lu answers https://api.geoportail.lu/fulltextsearch?query=...&limit=... with GeoJSON. Its implementation is public in Geoportail-Luxembourg/geoportailv3: one Elasticsearch index (5 shards), queried from views/fulltextsearch.py, mapped by lib/index_settings.json. It is a useful reference point because it solves the same problem from the other direction — analyzer chains rather than an address parser.

One field, indexed three ways. A document (poi type) carries label, the whole formatted address as one string — "5, Rue de l'Industrie, L-3895 Foetz" — plus a layer_name keyword. label is a multi-field:

field	analyzer	what it buys
label	whitespace + lowercase	exact word matches, accents intact
label.ngram	edge n-grams 1–12 over letters and digits, lowercase + asciifolding (searched with simplified)	the word still being typed
label.simplified	standard tokenizer + lowercase + asciifolding + elision	accent- and apostrophe-insensitive matches

The query. A bool with minimum_should_match: 2, filtered to type: poi, and two multi_match clauses per term: an exact one over label^2, label.ngram^2, label.simplified^2 with operator: and, and a fuzzy one over label.ngram, label.simplified with fuzziness — 1 by default, not AUTO, and overridable per request with &fuzziness=. Layer preference is expressed as scored wildcard clauses on layer_name rather than a filter:

Commune 10 · Localité 9 · Adresse 8 · lieu_dit 7 · nom_de_rue 2 · Parcelle 1 · FLIK 1 · editus_poi* -1.5

Those boost clauses sit in the same should array as the text clauses, so they count toward minimum_should_match. One detail worth knowing when reading the code: terms are split with query.split('%20') — the literal escape sequence, not a space — so a decoded query normally arrives as a single term.

How that differs from here. They reach abbreviation- and accent-insensitivity with analyzer chains (asciifolding, elision) and completion with edge n-grams; this library reaches them with libpostal’s expansions and prefix terms, and its index is address-only where theirs also holds communes, localities, street names, parcels and POIs. The consequential difference is the house number: geoportail has no address parser, so the number is just another token in label, which is why a number nobody indexed returns nothing at all rather than a flagged neighbour.

A note on the text_search dependency

Declaring AddressIndex inside postal itself means every consumer of postal now needs @library("text_search") too, even one that only ever calls PostalAddress::parse — GreyCat’s native “link” step resolves a library’s full module graph unconditionally, it doesn’t prune to what’s actually used. If you only need address parsing/expansion, nothing changes for you; you just carry a dependency you don’t call.

Declaration order matters. @library("postal") must come before @library("text_search") in project.gcl — the reverse order causes greycat codegen to silently drop postal’s own field-offset and native parameter macros (an empty diff where 36+ #define gc_postal_* lines should be), which then fails to compile with “undeclared identifier” errors in postal.c. This looks like a greycat codegen bug when two native libraries with a dependency between them are combined, not something specific to this code — order is currently the workaround.