CleverMemory

A pack build is a counting sort

2026-07-09T00:00:00+00:00

We compile Wikidata — 120 million entities, 1.76 billion statements — into a memory-mapped knowledge pack. The classic way to build something like that is a chain of external sorts: sort every statement by subject, merge-join the labels, re-sort by object, join again, sort once more into the final layout. Touch every record eight or ten times. Shuffle terabytes through the disk. Wait half a day.

We kept asking a question that sounds naive: copying 200GB takes under an hour — why should restructuring it take twenty? The answer turned out to be that comparison sorting was never actually required. It was a habit.

Order can be computed instead of discovered</h2>
Our keys aren't arbitrary strings. They're integers in a known, nearly dense range. And for keys like that, math has had a better offer on the table forever: counting sort — two linear passes, zero comparisons.
Here's the observation that reframed the whole build: constructing a CSR graph layout is counting sort. The offsets array every CSR needs is exactly the prefix sum of a histogram. So:

Pass one: stream the records, count edges per node. Cheap — it's just counts.</li>
Prefix-sum the counts. Now you know the final byte address of every single edge before writing anything.</li>
Pass two: stream again, write each edge straight into its final slot.</li> </ol>
The data lands already in order. There is no "end sort" because nothing was ever out of order. Order was computed, not discovered.
The joins dissolve the same way</h2>
Those expensive sorts existed to join labels onto statements — a statement needs the names of the entities it mentions, and those live elsewhere in the dump. But lay the label table out addressably — entity number times slot size — and the join becomes a positional read. No sort. No merge. Just look at the address where the label has to be.
One comparison sort survives: the key dictionaries. But those are small — about 130 million keys, a couple gigabytes — and that sorts in RAM in seconds. Which gives us a rule we now apply across the whole pipeline:

Comparison-sort only what fits in RAM. Distribute everything that doesn't. </blockquote>
Sometimes you don't even need two passes</h2>
One step further, and it's my favorite part. The source dump is entity-ordered — every entity's statements arrive together. So in plain arrival order, each node's edges are already physically contiguous. Append them as they come, record each node's `(offset, length)</code> in a compact addressing table, and the forward graph is built in one pass, at parse speed. CSR never required globally sorted rows — sortedness only bought us compressible offset arrays. An explicit addressing table buys that back for about a gigabyte, and deletes an entire sorting phase from existence.`
`None of this is exotic. Counting sort is undergraduate material. Log-structured storage and out-of-core graph systems have used these shapes for a decade. The lesson we keep relearning is quieter, and it's the one I'd put on a poster: when a pipeline feels slow, the biggest wins rarely come from making a phase faster. They come from noticing a phase never needed to exist.`

Proof or silence: why our memory layer refuses to answer 2026-07-09T00:00:00+00:00 Every retrieval system you've used has the same failure mode: ask it something it doesn't quite know, and it answers anyway. Vector search returns the nearest neighbor whether or not the neighbor is right. A language model sitting on top will happily narrate that near-miss into a confident paragraph. The industry calls the result "hallucination," like it's a glitch. It's not a glitch. It's the design. Similarity has no concept of wrong — only less similar. We built CleverMemory on the opposite premise: a wrong answer is worse than no answer, and the only answers worth giving are the ones the system can prove. What "prove" means here</h2> When text comes in, it doesn't become vectors. It becomes typed structure: entities linked to identities, events with typed roles for every participant, negation and modality kept as first-class marks ("might visit" is never stored as "visits"), time as intervals, every claim carrying provenance back to the exact bytes it came from. A question becomes structure too — a pattern with typed holes. "Who directed the film that won Best Picture in 2020?" is a small graph: an unknown person, tied by a directed edge to a film, tied by a won edge to an award, pinned to a year. Answering means matching that whole pattern into memory — every edge witnessed by a stored fact, every hole filled by something whose type fits. And here's the part I care about most: the match itself is the proof. Not a confidence score. A checkable object — for each edge of the question, the exact supporting fact; for each hole, the binding and why its type qualifies. When no match exists</h2> The system abstains — but not with a shrug. A failed match knows precisely which edge found no witness, so the abstention comes with a certificate: "I know a film that won Best Picture in 2020, but I hold no directed-by fact for it." When the question is ambiguous, the certificate names the tie: "Two films match — the 1997 original and the 2005 remake. Which one?" Sit with that last behavior for a second. A system that asks exactly the right clarifying question feels intelligent. But there's no cleverness in it. It's honesty, rendered. The tied candidates were sitting in the proof frontier all along — refusing to secretly pick one, and saying so instead, is the entire trick. Why this pairs with LLMs instead of competing with them</h2> Language models are extraordinary readers and writers with unreliable memories. This layer is the opposite: a perfect memory that can't write a paragraph. The division of labor is clean. The memory returns proven facts, cited spans, and honest gaps. The model composes prose from material it no longer has to invent. The model stops being asked to remember — the thing it was never good at. All of it is measurable, and we measure it: answer accuracy on benchmarks, and — just as important — abstention integrity: when the system declined, was declining correct? A memory you can trust is one whose silences you can trust too. Standing on giants: the open data behind the field (Foundations, part 8) 2026-07-08T00:00:00+00:00 Part 8 of Foundations: the sources we compile into one field of language and world knowledge — all of whom deserve the credit — and why licensing was a design decision, not paperwork. CleverMemory's substrate is a set of knowledge packs: language and world knowledge compiled into one queryable field. I want to be direct about something: none of it sprang from us. It's the accumulated, freely-given work of hundreds of researchers and thousands of volunteers. Credits on the record: Wikidata (Vrandečić & Krötzsch, and its volunteer editors) — a hundred-plus million entities and their relations, CC0 public domain. The backbone of world knowledge, right down to metadata describing its own properties.</li> PropBank (Palmer et al.) and VerbNet (Kipper Schuler) — verb argument structure and thematic roles. The data behind part 4's events.</li> UniMorph (Kirov, McCarthy et al.) — inflectional morphology across hundreds of languages. How "ran" relates to "run."</li> Universal Dependencies (Nivre, de Marneffe, and a huge consortium) — consistent grammar annotation for 100+ languages.</li> Princeton WordNet (Miller, Fellbaum) and the Open Multilingual Wordnet (Bond et al.) — word senses, multilingually linked.</li> Unicode CLDR and the Unicode Character Database — what text itself means. IANA tzdata (Olson, Eggert, and maintainers) — the world's timezone history. Quietly heroic public data.</li> QUDT — quantities, units, dimensions, machine-readable. The reason "3 km" and "3,000 m" can be the same fact.</li> GeoNames — the planet's places and what contains what.</li> GUM (Zeldes and Georgetown students) — richly annotated discourse.</li> ATOMIC 2020 (Hwang, Bhagavatula et al., AI2) — everyday commonsense, generously CC-BY.</li> Metamath (Megill and contributors) and mathlib (the Lean community) — formal math with every step checkable. Proof discipline, as data.</li> Project Gutenberg and the public domain — the corpus commons.</li> </ul> The licensing rule that cost us</h2> Early on we made a rule that hurt: friendly licenses only. Public domain, CC0, CC-BY, permissive. No share-alike, no non-commercial, no research-only — no matter how good the resource. Some famous datasets are missing from that list for exactly this reason. Excluded with regret, and with respect for their authors' right to set their terms. Why be that strict? Because a knowledge substrate is infrastructure, and infrastructure with murky terms poisons everything built on it. If you download our packs, you should be able to build a product, a paper, or another dataset without hiring a licensing archaeologist. That only works if every row is clean at the root. So every fact carries provenance to its source, and the whole set is auditable — walk any answer's citation back and you land on a friendly origin. Where the friendly set has gaps, we fill them the honest way: derive from clean sources, generate carefully and check before believing — never by quietly leaning on restricted data. Slower? Yes. Narrower at first? Also yes. Worth it, because here's the whole point: you can't build a common language for language on borrowed words. That closes Foundations. The two posts after this — on refusing to answer, and on building the packs fast — are where the story turns operational. The algebra of citations (Foundations, part 7) 2026-07-07T00:00:00+00:00 Part 7 of Foundations: why every answer — even a computed one — can carry its receipts, and the theory that guarantees it. Citations are easy when the answer is one stored fact: point at the fact. They get interesting when the answer is computed — a count, a set intersection, an "earliest," a roll-up across a whole document. What does it even mean to cite a number you derived from forty facts? Most systems answer: nothing. Here's the number. The provenance evaporates exactly when the answer got complicated enough to need it. Database theory got there first — again. Green, Karvounarakis, and Tannen (2007) unified years of lineage research with one beautiful move: tag every base fact, define how tags combine when facts are joined (both were needed) and when alternatives union (either sufficed), and query answers come out annotated with polynomials over their sources. They called it the provenance semiring framework. The polynomial is the citation — it says which facts contributed and how: jointly or alternatively, once or repeatedly. Here's the part that matters for building: this composes through any computation assembled from the right algebraic pieces. Filters, joins, unions, aggregations — define each one as a structure-preserving map and provenance threads through automatically. Citations stop being a per-feature chore you can forget. They become a property of the algebra. You can't forget to implement citations for your new operation, because the operation's definition is its citation behavior. I love results like this — the kind where doing it right is also doing it once. What it looks like from the outside</h2> Ask "how many distinct cities did I visit in 2024?" and you get: 7 — from these 11 mentions, deduplicated to these 7 identities, each with its source span</code>. Ask an intersection — "films that won Best Picture and are in my watchlist" — and every member arrives with both justifications. Ask something long and compositional and the evidence comes back as a tree: each step citing the step below it, all the way down to bytes in your documents. Two working rules keep the receipts real instead of decorative: No anonymous merging. Whenever facts combine, the result keeps back-pointers to its contributors. A deduplicated set remembers what it deduplicated. A "latest value" remembers what it superseded. And then a mechanical check enforces it — every merge either carries its contributors or explicitly justifies why not. Trust the algebra, verify the code. Bounded work is labeled work. Real systems cap searches for speed. Fine — but a cap may bound effort, never truth. If a scan got cut short, the result says so. An incomplete search can never dress up as "searched everything, found nothing." A citation that silently omits "…as far as I looked" isn't a citation. Notice how this snaps together with part 3: certain answers tell you when you may speak. Provenance polynomials are what you hand over when you do. Honesty about whether; receipts for what. The math has been sitting there since 2007, waiting for memory systems to take it literally. So we did. Time is two-dimensional (Foundations, part 6) 2026-07-06T00:00:00+00:00 Part 6 of Foundations: what temporal databases teach a memory about change, correction, and history. Two questions that look the same and aren't: "Where did I live in 2023?" and "What did you believe in 2023 about where I lived?" If that seems fussy, consider that the second one is what every audit, every "why did you tell me X last month," every debugging-of-an-old-decision actually asks. A memory that can't tell them apart will answer one with the other and call it recall. Database folks mapped this out years ago — Richard Snodgrass and the temporal-database community called it bitemporal data. Every fact gets two independent time axes. Valid time: when it was true in the world. Ingest time: when the system learned it. Two axes, because the world changes and your knowledge of the world changes, and those are different events. From there, a small catalog of correctness: Updates close intervals. They don't erase. When you move away from Denver, the old address doesn't become false — it becomes bounded: true until March. A memory that overwrites has amnesia about everything it ever correctly knew. Corrections aren't updates. "Actually, I moved in March, not May" repairs the past — valid time moves — while the record of when you said each thing stays put. So the system can honestly answer both "when did you move?" (March) and "when did I learn it was March?" (today). Succession isn't contradiction. Austin, then Denver — that's a history. Two claims about the same interval that disagree — that's a conflict, and it deserves a flag, not a silent latest-wins. Collapse this distinction and your system ends up believing whatever was said most recently. Sound familiar? For actually reading time out of language, we stand on more shoulders: James Allen's interval algebra (1983) — the thirteen ways two intervals can relate, turning before/during/overlaps into a calculus — and TimeML/TIMEX3 (James Pustejovsky and colleagues), the standard for normalizing "March 2024," "for three weeks," "every Tuesday" into typed dates, durations, and recurrences. Sounds mundane. It's genuinely hard, and it's solved because people spent years on it. That's the reason "every Tuesday" can be stored as a recurrence instead of a vibe. One more thing worth noticing: "no erasure" is also how accountants have kept books for five hundred years. Append-only ledgers, corrections as new entries referencing old ones. When the same design shows up independently in double-entry bookkeeping, temporal databases, and git, it's telling you what trustworthy records are. So: a real memory is an append-only, two-axis history. "What's true now" is one slice of it. "What was true then" is another. "What did we believe when" is always answerable. Time isn't metadata on the fact. It's half the fact. "Might" is not "is": storing what you actually said (Foundations, part 5) 2026-07-05T00:00:00+00:00 Part 5 of Foundations: modality, negation, and the quiet ways systems distort what you told them. Here's a failure so common it's almost invisible. You tell your assistant "I might move to Denver next year." Months later you ask where you live, and somewhere in the machinery your hedge has hardened into a fact: user is moving to Denver. Nothing lied, exactly. The representation just had no place to keep the might, so the might evaporated. Language is full of these little force-carrying words — modals (might, must, should, can, may), negation (not, never, no longer), conditionals (if, unless). And they're not decoration; they're the commitment level of the claim. Logicians and linguists took them seriously long ago: modal logic from C.I. Lewis through Kripke's possible-world semantics gave necessity and possibility a formal home, and Angelika Kratzer showed how natural-language modals systematically express flavors of commitment — possibility, obligation, ability, permission. This is settled science about what sentences claim. Most memory systems throw it away at the door. The distortion catalog</h2> Drop these markers and every loss is a truth distortion, not a lost nuance: Modality collapse: "might visit Paris" stored as "visits Paris." Possibility promoted to fact.</li> Negation blindness: "the deal did not close" retrieved as support for "did the deal close?" — because minus the not, they're nearly the same string. That's part 1's similarity problem at its sharpest.</li> Ability isn't action: "can access the server" is not "accesses the server." "May leave early on Fridays" is a permission, not a schedule.</li> Hedge laundering: a chain of systems each rounding "reportedly," "planned," "expected" a little closer to fact, until a rumor is a record.</li> </ul> The fix is representational, not clever</h2> We store force alongside content. A claim's polarity (asserted or negated) and its modal class (necessity, possibility, ability, permission, volition) are part of the stored structure — as inseparable from the fact as its subject. Then the question side respects them: "where do I live?" gets answered from asserted facts, never from possibilities. "Was I considering Denver?" finds the might on purpose. And when a hedged claim is all there is, the answer keeps the hedge: you said you might. Same discipline for reported speech — "Dana said the launch slipped" is a fact about Dana's claim, not about the launch — and for questions inside statements: "he asked whether it shipped" asserts an asking, not a shipping. Who committed to what is exactly the kind of thing a memory exists to keep straight. Notice what we're not doing: reasoning about possibility. That's the language model's job, upstairs. The memory's job is simpler and harder — preserve the commitment level of what was actually said, so whoever reasons later reasons from what you meant. Fidelity first. Cleverness can rent the space above it. The whole sentence informs its parts (Foundations, part 4) 2026-07-04T00:00:00+00:00 Part 4 of Foundations: events, roles, and why a sentence isn't a bag of triples. The tempting way to store meaning is subject–verb–object triples. "Maria sold the bakery to Tom in 2023" becomes maria — sold — bakery</code>. Quick question: where did Tom go? Where did 2023 go? Bolt them on as more triples and you've built yourself a puzzle — if Maria sold twice, which sold do they belong to? Linguistics solved this decades ago, and it's one of my favorite ideas we inherit. Donald Davidson (1967) said: action sentences are really about an event. There was a selling event. Maria was its agent, the bakery its theme, Tom its recipient, 2023 its time. Everything attaches to the event itself. Now "sold twice" is just two event nodes, each with its own participants. Puzzle gone. Charles Fillmore added the other half: verbs come with role frames. A selling has a seller, a thing sold, a buyer, maybe a price — knowing the verb tells you what participants to expect. And two landmark projects turned that insight into real, usable data we gratefully build on: PropBank (Martha Palmer and colleagues), which annotated verbs with numbered argument roles across a big corpus, and VerbNet (Karin Kipper Schuler), which organized verbs into classes with shared roles. Work like AMR (Banarescu et al.) and the precision-grammar tradition (English Resource Grammar, Flickinger; Minimal Recursion Semantics, Copestake et al.) proved you can produce these structures at scale with real discipline. What the event hub buys you</h2> Multi-participant questions just work. "Who did Maria sell the bakery to?" is a role lookup on the right event — not a prayer that "to Tom" survived being chopped into triples. The whole disambiguates the parts. Knowing this is a selling frame tells you the "to" phrase is a recipient, not a destination. Parts read in isolation go wrong exactly where language is most flexible. Parts read as members of a frame inherit the frame's discipline. Same lesson as part 2's questions, right? Wholes constrain parts — on both sides of memory. And it scales up. The same move repeats at every level: words in clauses, clauses in sentences, sentences in paragraphs, paragraphs in documents. Context flows down that tree — a pronoun resolves against its document, a date reads against its timeline. Structure rolls up it — a paragraph's topic is a function of its sentences' events. Meaning is fractal. The representation should be too. The engineering rule that falls out sounds simple and costs us real work to honor: never let a fragment travel alone. Every extracted piece of meaning keeps its connection to the event, the sentence, and the document that produced it. Because whatever context you drop at extraction time — that's exactly the context some future question turns out to need. Certain answers: the math of saying "I don't know" (Foundations, part 3) 2026-07-03T00:00:00+00:00 Part 3 of Foundations. Parts 1–2 covered structure and questions. This one's about making honesty precise. "Proof or honest abstention" sounds like a slogan. It's actually a fifty-year-old piece of mathematics, and I think it's the most underused idea in AI right now. Database researchers hit our exact problem in the 1970s: what should a system say when its data is incomplete? The landmark answer came from Imieliński and Lipski (1984), building on Codd's relational foundations and Reiter's logical reading of databases. Under the open-world assumption — what you know is true, but it isn't everything — the only defensible answers are the certain answers: the ones that hold in every possible world consistent with your data. Everything else is a guess. Might be right! A memory system has no business trading in "might be right." What you get for taking this seriously</h2> "I don't know" becomes a verdict, not a vibe. An abstention means: this candidate is possible but not certain. That's checkable. And it splits ignorance into useful kinds — no candidate at all, versus two candidates the data can't tell apart. In the second case the honest output is the tie itself: "the 1997 film or the 2005 remake — which one did you mean?" Saying "no" requires a license. Here's the subtle one. "Has the invoice been paid?" — answering no means knowing your records are complete for payments. Not finding a payment isn't the same as there being none. Systems that skip this distinction manufacture confident falsehoods out of missing rows every single day. Open-world honesty says: treat absence as absence, not as falsity, unless you've established completeness for that corner of the data. Counts inherit the rule. "How many meetings did I have with Dana?" is only certain if the underlying set is certainly complete. A count over maybe-partial data, presented as exact, is a subtle lie. The theory catches it mechanically. And there's a lucky break: for the conjunctive queries from part 2, certain answers are cheap to compute. The exact fragment natural questions live in is the fragment where honesty is affordable. When the math lines up that well, I stop believing in coincidence. Why a wrong answer is worse than none</h2> Worth saying plainly. An abstention costs you a lookup somewhere else. A wrong answer — delivered confidently, by the component whose whole job is being trusted — costs you the decision you built on it. And eventually it costs you your trust in every answer the system ever gave. A memory layer is infrastructure for trust. Certain-answer semantics is just what that looks like written down. So that's how our engine behaves: every answer is a certain answer with its derivation attached, and every abstention says whether the problem was absence, ambiguity, or your question assuming something the data doesn't support. The silences are as engineered as the speech. A question is a query with typed holes (Foundations, part 2) 2026-07-02T00:00:00+00:00 Part 2 of Foundations. Part 1 argued similarity isn't memory. This one's about what a question actually is. Ask "Who directed the film that won Best Picture in 2020?" and you haven't asked for text that looks like your words. You've drawn a shape: some unknown person, connected by a directed edge to a film, which is connected by a won edge to an award, pinned to a year. The question is a small graph with one piece missing — a hole. And the hole isn't anonymous. It's typed. Whatever fills it has to be a person. This idea has serious ancestry, and I want the credits on the record. Montague made the founding bet back in the 1970s: natural language is a formal language, translatable into logic by rules. The program stalled on coverage, but the bet was right. Semantic parsing — the whole line of work turning questions into database queries — carried it forward and made it practical. And database theory gave us the exact object a question becomes: the conjunctive query, a pattern of variables and relations that either maps into your data or doesn't. The result we lean on hardest is Yannakakis's algorithm (1981): for the tree-shaped queries natural questions almost always are, you can match efficiently and completely. No missed answers. No wasted search. Why typed holes change everything</h2> The whole question squeezes every part. In a query graph, every edge touching a variable narrows it. That person-hole doesn't get filled by scanning all people — it gets squeezed through the film, which gets squeezed through the award and the year. Systems that break questions into independent keyword lookups throw this away. That's why they hand you the 2005 remake when you asked about the 1997 original. The match is the proof. When the pattern maps into memory, the map itself tells you which stored fact witnessed each edge and what filled each hole. The answer shows up with its evidence attached. Not a score — a derivation. Failure tells you something. When no map exists, the algorithm knows exactly which edge couldn't be witnessed. "I know a film that won Best Picture in 2020, but I have no directed-by fact for it" beats silence by a mile. It tells you what's missing — which tells you what to feed the memory next. The genuinely hard part</h2> Getting from a sentence to that typed graph is the front half of the problem, and it's where most of our engineering lives. The same machinery that reads statements has to read questions — one reader, two purposes — so the question's math and the memory's math can never disagree about what words mean. A statement is a pattern with no holes. A question is the same pattern with some. That symmetry is the quiet thesis of this whole series: understanding a question and remembering a fact are the same act, pointed in opposite directions. Why similarity is not memory (Foundations, part 1) 2026-07-01T00:00:00+00:00 Part 1 of Foundations — a series on the ideas behind CleverMemory, where they came from, and why each one matters. The standard way to give an AI "memory" today is vector search. Embed your documents, embed the question, return the nearest neighbors. And honestly? It works great — for what it actually does. What it actually does is discovery. "Find me stuff that feels related to this" is a similarity question, and embeddings answer it brilliantly. Full credit to that whole lineage, from the distributional hypothesis (Harris, Firth — "you shall know a word by the company it keeps") through word2vec to modern encoders. Real breakthrough, genuinely useful. But here's the thing. Ask a memory question — "when does my contract renew?" — and similarity is the wrong tool wearing the right costume. Three things similarity can't see</h2> An embedding is a lossy squish into a space where nearby means related. Three things get squished out, and they're exactly the things memory exists for: Identity. "The renewal date is March 3rd" and "the renewal date is not March 3rd" are nearly identical strings, so they're nearly identical vectors. Similarity sees family resemblance. Memory needs to see contradiction. Direction. "Alice reports to Bob" and "Bob reports to Alice" — same bag of words, same neighborhood in vector space. The direction of that edge is the entire fact, and it's gone. Absence. This one bugs me the most. Vector search physically cannot say "I don't have that." It returns the nearest neighbor no matter how far away the nearest neighbor is. There's no distance where it says no. And a memory that can't say no will eventually say something false — confidently. So what do we do instead?</h2> We take the older, harder road: turn language into actual structure. Entities resolved to identities. Relations with direction. Negation and time as real, first-class marks. Then answer questions by matching that structure exactly. "Who reports to Bob?" either finds a proven reports-to edge arriving at Bob, or it doesn't — and either way, you get told the truth. To be clear, this isn't anti-embeddings. It's a division of labor. Similarity is great for finding candidate places to look, and for the fuzzy associative stuff language models do natively. Exactness is what you need the moment somebody's going to act on the answer. Most of the confident-nonsense problem of the last few years comes from one mistake: asking the similarity tool to do the exactness job. The rest of this series digs into what "meaning as structure" takes seriously: what a question really is (part 2), the math of honestly saying "I don't know" (part 3), events and roles (part 4), the word "might" (part 5), time (part 6), citations as algebra (part 7), and the public data that makes all of it possible (part 8). One sentence for the whole project: memory should be the part of an AI system that never guesses.