What the Winnex Madhava engine actually proves — and what it does not. Three public Kaggle notebooks, run with winnex-madhava==1.9.13 + winnex-ai-normalize==1.4.1 installed from PyPI, on real datasets. Every metric below is explained: what it measures, why it matters, and its honest limits.
Kaggle — Certificate (v2, legal+medical) Kaggle — Damaged dataset Kaggle — NDCG vs FAISS
Kernel: winnex-1913-certificado-v2-jus-med. This is the property no ANN index (HNSW, IVF, ScaNN) offers: for every document the engine discards, it can prove that document could not be in the true top-K.
For a query q and a document v, the Cauchy–Schwarz inequality gives an upper bound on their true similarity:
If UB(v,q) < threshold — where threshold is the exact score of the 10th result — then v mathematically cannot be in the true top-10. The engine records this at the moment of pruning: it is a witness, not a post-hoc judge. This is what a compliance layer can sign (WORM) and what a regulator can audit.
| Field | Meaning | Why it matters |
|---|---|---|
total_excluded_count | How many documents the bound proved outside the top-K | The scale of the provable pruning |
genuine_exclusions | Every sampled excluded doc has upper_bound < global_threshold | Confirms the certificate is not reporting false exclusions |
deterministic_commitment | Two calls with the same query return the identical sample | Reproducibility for audit |
count_match | commitment.total_excluded == audited.audit_excluded | The compact commitment agrees with the full audit |
recall_guarantee | pool_only vs exact_global | Honest scope: is the top-K global, or only best-within-the-pool? |
| Corpus | N | d | Mode | NDCG@10 | Recall@10 | Gen. excl. | Deterministic | Example exclusion |
|---|---|---|---|---|---|---|---|---|
| GloVe | 400,000 | 100 | default | 1.0000 | 1.0 | 100/100 | 100/100 | UB −0.012 < 0.604 |
| GloVe | 400,000 | 100 | audit_exhaustive | 1.0000 | 1.0 | 100/100 | 100/100 | exact_global |
| HN-OpenAI | 100,000 | 1536 | default | 0.9805 | 1.0 | 100/100 | 100/100 | — |
Read carefully: recall@10 = 1.0 here means the engine returns the exact top-10 of the same embedding space (verified against a numpy brute-force ground truth on the same vectors). It does not mean the embeddings capture human relevance — see the honesty section below. What the certificate adds is different and orthogonal: for every one of the ~37.8M accumulated exclusions on GloVe, the engine can prove it was not in the true top-K.
In legal e-discovery or medical triage, "we searched and didn't find it" is not acceptable if you cannot show why a document was set aside. An ANN index returns a fast guess with no way to prove a miss. Madhava returns the top-K and, for every discarded document, a signed bound proving it was outside the true top-K of the index. That is the difference between "the model thinks" and "the math proves". The notebook embeds real Supreme-Court judgments and medical transcriptions with multilingual / biomedical sentence transformers to exercise the certificate on domain text.
Kernel: winnex-1913-dataset-danificado. The question: what happens when the input is broken — not the engine?
Real-world corpora are dirty. The most instructive case is Word2Vec GoogleNews: 94.7% of its raw vectors are NaN — the file itself is corrupt. A naive pipeline indexes it, the engine reports bound_violations = 0 (true — the bound is internally consistent), and nobody notices the results are garbage. That silent failure is the danger this benchmark exists to expose.
We inject a controlled corruption (5–10% NaN rows into an otherwise clean corpus) and run both pipelines on the same broken data:
| Pipeline | What it does with NaN | Result |
|---|---|---|
| Winnex quality gate | Detects dataset.nan (FAIL) and blocks the corpus before indexing | Blocked — refuses to index garbage |
Winnex allow_unsafe=True | Indexes anyway (to show the alternative) | recall@10 = 0.0, NDCG = 0.0 |
| FAISS (FlatIP / HNSW) | No quality gate — indexes the NaN silently | recall@10 = 0.0 (FlatIP & HNSW) |
The allow_unsafe row is the honest counterfactual: if you force the engine past its own gate, recall collapses to 0.0. The bound is still "0 violations" — because the bound is about internal consistency, not data quality. That is exactly why the gate exists: to stop a 0.0-recall corpus from being indexed as if it were a 1.0.
This is why the normalize layer runs a quality gate before the engine — it is the input firewall. The damaged-data notebook demonstrates the firewall catching a real corruption that FAISS, having no such layer, indexes silently.
Kernel: winnex-1913-ndcg-vs-faiss. Recall@10 answers "are the right 10 present?" — it ignores order. NDCG@10 with graded relevance punishes a correct-but-misordered ranking. This is the metric that separates an exact ranking from an approximate one.
With binary relevance (a document is either in the top-50 or not) and a ground truth of exactly k relevant items, NDCG@k collapses toward recall — any ranking that places the k correct items in the top-k scores ~1.0 regardless of order. Graded relevance (rel = 1/(rank_in_GT + 1)) makes the position matter: putting the 3rd-best document first costs NDCG. Measured on synthetic data, a perfectly-reversed top-10 scores 0.56 under graded NDCG vs 1.0 under binary — the distinction the benchmark needs.
| Dataset | d | Method | NDCG@10 | Recall@10 | Build (s) | Latency/query |
|---|---|---|---|---|---|---|
| GloVe | 100 | Winnex (pca) | 1.0000 | 1.0 | 5.87 | 21.2 ms |
| FAISS FlatIP | 1.0000 | 1.0 | 0.08 | 6.7 ms | ||
| FAISS HNSW | 0.9959 | 1.0 | 56.8 | 0.38 ms | ||
| FAISS IVF | 0.9954 | 1.0 | 0.82 | 2.3 ms | ||
| MNIST | 784 | Winnex (pca) | 1.0000 | 1.0 | 18.6 | 7.4 ms |
| FAISS FlatIP | 1.0000 | 1.0 | 0.19 | 15.7 ms | ||
| FAISS HNSW | 1.0000 | 1.0 | 19.1 | 0.39 ms | ||
| HN-OpenAI | 1536 | Winnex (pca) | 0.9753 | 0.991 | — | 15.6 ms |
| FAISS FlatIP (exact) | 0.9786 | 0.991 | — | 48.3 ms | ||
| FAISS HNSW | 0.9731 | 0.99 | — | 1.08 ms | ||
| FAISS IVF | 0.9699 | 0.991 | — | 9.4 ms |
pool_only mode the top-K is exact within the k1 pool, not proven global. The recall_guarantee field says so. Nothing is rounded up.The recall and NDCG in these notebooks are measured against a numpy brute-force exact scan over the same vectors — a real external ceiling, not the motor's own search_exact. This avoids the circularity of "the engine scoring itself".
Honest limit: the ground truth is computed in the embedding space. It proves the engine retrieves the geometrically-correct neighbors; it does not prove the embeddings capture human relevance. Two documents that are near in vector space but unrelated semantically will still be "correctly" retrieved. Embedding quality is a separate question the engine does not and cannot solve.
They answer different questions. A benchmark that reports only recall can hide a misordered ranking; one that reports only latency hides the absence of proof.
It means: every document the engine pruned via the Cauchy–Schwarz bound was provably not in the exact top-K of the scored pool. It is a soundness guarantee about the pruning, reproduced on every run.
Honest limit: it is not a statement about data quality or semantic relevance. On a corrupted corpus you can have 0 violations and useless results — which is precisely why the quality gate (benchmark #2) exists as a separate, earlier layer.
pool_only vs exact_globalBy default the exact post-filter re-scores only the survivors of the k1 pool. The result is exact within that pool — the engine reports recall_guarantee = "pool_only". When a compliance workflow needs the global guarantee, audit_exhaustive=True forces the pool to cover the whole corpus (k3 == N) and reports "exact_global", at a predictable O(N·d) cost.
These notebooks run both modes and report both — the honest scope is never assumed.
Everything here is public and runnable: PyPI winnex-madhava==1.9.13 + winnex-ai-normalize==1.4.1, the exact notebook source in each Kaggle kernel, and full execution logs. If a number looks surprising, run the notebook — it will reproduce. We would rather show a 0.975 than claim a 1.0.
Winnex AI — deterministic vector search with mathematical proof. · GitHub · Kaggle · PyPI: winnex-madhava · Stack Whitepaper (PDF)
Honesty policy: all benchmarks are auto-executed public notebooks. Metrics report what was measured, including non-perfect numbers. Claims that could not be measured are not made.