A Query With Holes In It

Part I measured what vectors do well. Part II replaced vocabulary with grammar and found that structure adds distinguishability and subtracts semantic organization; every structural representation lost to plain unigrams at retrieval.

Both parts measured cosine similarity between pooled vectors. Every arc in a message gets hashed into a bucket, summed into a bag, and compared by dot product. Hashing loses subgraph identity and pooling loses which subgraph belonged to which sentence, so what those parts actually tested was a flat representation derived from structure.

That method cannot express this:

approve >nsubj $WHO >dobj $WHAT

There is no wildcard in a cosine, no slot and no binding. Unigrams win at “documents about invoices” and cannot separate “someone deleted a file” from “a file was deleted by policy” — same words, different argument structure. Keeping the grammar is only worth doing if the query can use it.

Choosing a pattern size

A pattern is usable as an index key if it matches at least two sentences and no more than 1% of them. Below that it identifies one sentence; above it, it filters nothing. Measured across all 4,773 sentences, by size in edges and by how much of the pattern is lexically bound:

edges	binding	usable	singleton	too broad
1	all words bound	47.0%	44.5%	8.4%
1	shape only	7.9%	0.7%	91.4%
2	all words bound	28.8%	69.1%	2.1%
2	verb bound, args free	38.1%	35.7%	26.1%
2	shape only	30.1%	5.1%	64.8%
3	all words bound	15.4%	84.1%	0.5%
4	all words bound	9.3%	90.6%	0.1%
4	shape only	55.5%	36.0%	8.5%

The usable band peaks at a different size for every binding level. Fully bound patterns peak at one edge, verb-bound at two, shape-only at four. Binding a content word is strongly selective, so two of them exhaust the corpus; shape is weakly selective, so more of it is needed before the pattern constrains anything.

The failure modes sit at the two corners. NOUN <nsubj VERB matches 91.4% too broadly. A four-edge fully-bound pattern is a singleton 90.6% of the time.

Star share — patterns where one node heads every edge, meaning an action with its participants — falls from 67% at two edges to 35% at three and 14% at four. Two edges is the last size where predicate-argument structure is the typical pattern rather than an occasional one. Past it, most connected subgraphs are chains describing embedding depth. RDF triples, semantic role labelling and knowledge graphs all use three nodes, and this is presumably why.

Two kinds of coordinate

Mailbox, thread and date are exact. They filter the posting list and take no part in scoring, since a message is either from 2001 or it is not.

The three-node star is what gets indexed.

A star has three bindable slots, so 2³ = 8 maskings, and the index stores all eight:

approve | nsubj:bob | dobj:invoice
approve | nsubj:*   | dobj:invoice
approve | nsubj:bob | dobj:*
*       | nsubj:bob | dobj:invoice
...

approve >nsubj $WHO >dobj invoice is then an exact hit on the mask with the subject slot free, rather than a scan or an approximate match. Relaxing a query becomes a second lookup instead of a search.

indexed 4,773 sentences into 330,420 keys in 599 ms
query latency: 0–3 µs typical

The relaxation ladder had one rung too many

The first version freed the head as a last resort. On the real corpus:

query	rung reached	hits
sign >nsubj $X >dobj $Y	freed head	1,657
delete >nsubj $X >dobj file	freed head	14

With the head freed, “who signed what” becomes “any verb with a subject and an object”, which is most transitive sentences in English. The module’s own documentation said the head is what the question is about, and the code freed it anyway.

The rule now is that every rung keeps at least one slot bound, enforced by a regression test. Both queries return zero instead.

Queries

Judged by reading the matched sentences, which is possible because pattern matching is deterministic:

query	hits	matches
send >nsubj $WHO >dobj $WHAT	42	all genuine sends
need >nsubj $WHO >dobj $WHAT	35	genuine
call >nsubj $WHO >dobj $WHOM	22	all genuine
$ANY >nsubj $WHO >dobj agreement	12	genuine; wildcard head
approve >nsubj $WHO >dobj $WHAT	6	all genuine
sign >nsubj $WHO >dobj $WHAT	2	genuine

Run them yourself. The index below is the same one, regenerated in the browser from the exported stars:

Query — HEAD >dep ARG >dep ARG

approve >nsubj $WHO >dobj $WHATcall >nsubj $WHO >dobj $WHOMsend >nsubj $WHO >dobj $WHAT$ANY >nsubj $WHO >dobj agreementdelete >nsubj $WHO >dobj filesign >nsubj $WHO >dobj $WHAT

A bare word binds a lemma. $NAME is a free slot. Relations are Universal Dependencies: nsubj,dobj, ccomp,xcomp, pobj, attr.

loading 205,526 indexed stars…

The ladder reports which rung produced a result:

delete >nsubj $WHO >dobj file
  relaxed to: delete >nsubj $X >dobj $X
  rung: freed dobj   hits: 2   [2 µs]
    · obviously , you can delete the smart - alecky last sentence .
    · i suggested that we delete it in the transaction documents , but …

Nobody in this corpus deleted a file. Two people deleted something else.

Against keyword search

The comparison that matters is not another vector, it is what you would do today: search for the verb. A keyword hit counts as relevant only if the sentence really does have that head governing both queried relations, which is checkable rather than judged.

intent	pattern	keyword	keyword precision	gain
who signed what	2	35	5.7%	17.5x
who wants what	17	117	14.5%	6.9x
who called whom	22	138	15.9%	6.3x
who reviewed what	11	57	19.3%	5.2x
who needs something	35	157	22.3%	4.5x
who sent what	42	104	40.4%	2.5x
who approved what	6	15	40.0%	2.5x
who received what	69	91	75.8%	1.3x
total	204	714

A keyword index returns 3.5x more material across the eight questions. Thirty-five sentences contain sign; two of them are someone signing something. The rest are signed pa, sign-off, signature, and imperatives with no subject.

Extracted bindings

who approved what   [approve >nsubj $WHO >dobj $WHAT]
              board  --approved-->  change
         commission  --approved-->  cut
                tco  --approved-->  overinjection
               they  --approved-->  500,000

A bag of words can report that a document mentions approving. It holds “board” and “change” as unrelated columns, so it cannot report that the board approved a change.

Three of every four extracted bindings are clean. The other 26.8% have a non-word filler — =20, =01, stray punctuation from Enron’s mangled quoted-printable encoding, which the parser assigns a dependency role like any other token.

Limits

Parse quality caps precision. An earlier run of “someone attached a file” returned nine hits, all variants of (see attached file : danny.jpg). Those are valid matches against attachment boilerplate.

let $X know returned nothing because I guessed dobj + ccomp and the parse uses different labels. The query was wrong, not the index. Writing correct patterns requires knowing the dependency scheme, which is an argument for having a model author them.

The mask approach costs about 69 keys per sentence. Odinson indexes dependency graphs into Lucene and matches a syntax-based traversal in 2.8 seconds over 134 million sentences, so the architecture scales; I have measured mine to five thousand.

What the three parts measured

	vector similarity	pattern query
“documents about invoices”	unigrams win, 8.57x	not expressible
“who approved what”	not expressible	6 hits, all correct

Neither method can express the other’s query. Precision is high wherever a pattern matches, which is a property that did not appear in any vector experiment across the first two parts. The conclusion there — that structure is weak for retrieval — was a conclusion about pooled cosine similarity.

Naming

The components have names. Structural and syntactic search are established; Semgrep, ast-grep and Comby match AST patterns with metavariables; Odinson and Semgrex match dependency patterns over text; backoff is old in language modelling.

I have not found a name for precomputing the full lattice of slot bindings so that relaxation is a lookup rather than a search, combined with exact coordinates filtering underneath. The eight masks form a Boolean lattice ordered by specificity and relaxing a query walks up it, so binding lattice index is at least accurate.

All counts come from the same 4,773-sentence corpus as Part II. The implementation passed 46 parser, indexing and matching tests; seed 20260807.