What Vectors Are Good At

Three parts, one corpus: 754 Enron messages, deduplicated, dependency-parsed. This part measures what a vector representation of them can do, because the next two try to replace it.

The separation theorems

The curse of dimensionality is the familiar half — distances concentrate, neighbourhoods empty out. Donoho’s 2000 lecture paired it with a blessing, and Gorban and Tyukin made the blessing precise. Concentration of measure says i.i.d. points pile into a thin shell. The stochastic separation theorems describe the structure of that shell: the points in it are individually linearly separable from all the others, for exponentially large sets.

A point x is α-Fisher separable from a set Y about centre c when

α (x − c, x − c) > (x − c, y − c)     for every y ∈ Y

The left side is x’s own squared radius from the centre; the right is how far each other point projects onto x’s direction. The separating hyperplane is fixed by x’s own position, so nothing is fitted. That property is why the applied literature calls these one-shot correctors: if a system errs on x, you attach the hyperplane normal to x − c and x is carved out without retraining.

Theorem 2 of Grechuk, Gorban and Tyukin (2021) bounds how many points survive this. For M points uniform in the unit ball of dimension n, all of them are α-Fisher separable with probability > 1 − δ provided

M < sqrt(2δ) (2α)^(n/2)

Exponential in n whenever α > ½. At α = 0.8 and n = 100 that permits around 10¹⁰ points.

Checking the instrument against the distribution the theorem covers

Uniform in the unit ball, M = 2000, α = 0.8, measured in Rust:

d	separable	RID
2	0.0000	2.03
4	0.0000	4.05
8	0.0055	7.90
16	0.7695	15.02
24	0.9890	>17.6
48	1.0000	>17.6

Separability saturates around d = 24–32, ahead of Theorem 2’s crossover at d ≈ 36. The theorem is a sufficient condition, so it is conservative by construction.

The third column is relative intrinsic dimensionality, from Sutton, Zhou, Gorban and Tyukin (2023), defined implicitly through a probability:

P( x ∼ D′, y ∼ D : (x − y, y − c) ≥ 0 ) = 1 / 2^(n + 1)

The paper states the definition and lists finite-sample estimation as open. Since the definition is a probability, the plug-in estimator is to sample pairs, measure the rate, and invert: n̂ = −log₂(p̂) − 1. On uniform-in-ball data it recovers ambient dimension to within 1 up to d = 16.

Past d ≈ 17 it reports >17.6 and stops. Resolving RID = n requires order 2ⁿ samples, so 400,000 pairs buys a ceiling at log₂(400000) − 1. That is a property of the definition rather than the implementation, and it is presumably why the paper leaves estimation open.

Enron

Hashed TF-IDF over the 754 messages. Corpus content fixed; only the number of hash buckets changes.

d	separable	RID	participation ratio
64	0.8655	14.44	48.4
256	0.9135	14.44	147.6
1024	0.9235	14.29	377.9
2048	0.9250	15.29	516.6

I expected clustered, Zipfian, non-log-concave data to fall well short of the isotropic case. 87–93% of messages are one-shot Fisher separable, and about 97% once near-duplicate reply chains are removed.

The sweep is the more useful column. A 32x increase in ambient dimension moved separability by 0.06. The participation ratio over the same sweep went 48 → 517, tracking d almost linearly, so the conventional effective-dimension measure is reporting hash buckets. RID stayed at 14–15 throughout, and so did separability.

Preprocessing does about as little. At d = 512: raw 0.9180, centered 0.9180, coordinate-scaled 0.9205, centered and renormalized 0.9195.

A control that did not fit

Loose Gaussian clusters at ambient d = 256 came back with RID above the estimator ceiling and 3% separability. High intrinsic dimension, no separability, which contradicts everything above.

Separability is a threshold on a per-point margin:

m(x) = max_y (x − c, y − c) / (x − c, x − c)

A point is separable exactly when m(x) < α, so the separable fraction is the left tail of a distribution, and reporting the fraction discards the rest of it:

structure	sep @ α=0.8	m p50	m p90	m max
isotropic	1.0000	0.205	0.237	0.308
clustered loose	0.0200	0.865	0.901	0.968
low-rank k=8	0.0393	1.265	1.853	3.651
enron	0.9267	0.302	0.700	1.000

The loose clusters have margins in a narrow band just above 0.8. If that reading is right, moving α slightly should move separability a lot. It does: 0.005 at α = 0.80, 0.920 at α = 0.90.

Four regimes come out of the margin quantiles that the fraction alone cannot distinguish. Isotropic data sits near 0.2 with wide headroom. Clustered data piles into a narrow band, so its separability is a step function of α and the reported number mostly records where α was placed. Low-rank data has broad heavy-tailed margins, and no choice of α rescues it. Enron behaves like isotropic data in the bulk with a genuine 7% tail.

Two corpora with the same separable fraction can therefore be in unrelated positions, which makes the fraction a poor summary on its own.

The baseline for parts II and III

Two capabilities, measured separately because the rest of the series separates them.

Telling documents apart: 92% one-shot separable, hyperplane requires no fitting.

Grouping documents by topic: using thread identity — the subject line with Re: stripped — hashed TF-IDF retrieves a thread-mate in the top 10 at 8.57x the chance rate.

All measurements use the fixed corpus and seed described above (20260807); the tables in this post are the published result record.