Most of the interesting decisions I have made recently came from a single rule I set at the start of a project: the binary may not make a network call. Not for a model download, not for telemetry, not on first run. Everything the thing needs to do its job has to already be inside the executable when you download it.
It sounds like a packaging preference. It is actually a design constraint, and almost every hard problem I hit afterwards falls out of that one sentence.
What the rule costs
The project is a local document search engine. It watches folders, extracts text, splits it into chunks, embeds each chunk, and answers questions with citations. The embedding step is the problem. Embedding means running a transformer, and running a transformer normally means Python, or an ONNX runtime, or at minimum a model file fetched on first launch.
All three break the rule. So the model had to be compiled in, which meant it had to be small, and it had to run without any native dependency at all.
Writing the model by hand
I ended up reimplementing MiniLM's forward pass in plain Go. Embedding sum, layer norm, six blocks of multi-head attention and a feed-forward network, mean pooling, L2 normalize. No matrix library, no bindings, about four hundred lines. Six layers, twelve heads, a hidden dimension of 384 and an intermediate of 1536.
This sounds like the wrong call and mostly it is. The reason it was the right one here is that any dependency with C in it forfeits CGO_ENABLED=0, and losing that means losing the ability to cross-compile to five platforms from one machine. The pure-Go version is slower than a tuned runtime. It is also the only version that ships.
Two details earned their place. The dot product uses four independent accumulators instead of one, which breaks the loop-carried dependency on the floating-point add and lets the CPU pipeline the work. That single change is worth roughly three times the throughput, and indexing is where all the time goes. The other is that each forward pass borrows about 4 MB of scratch memory from a pool rather than allocating it. Indexing a folder means thousands of passes, and thousands of 4 MB allocations is pure garbage collection churn.
There is also something that is not there. There is no attention mask, because sequences are never padded. Each text is embedded on its own and parallelism happens one goroutine per core, so the padding that normally forces a mask never exists.
Making it fit
Full-precision weights are 91 MB. That is too much to put in a binary people download casually, so the weights are quantized to int8: each row scaled by its own maximum absolute value over 127, stored alongside a small float array of those scales.
Not everything gets quantized. Anything under 100,000 elements stays float32, because biases, layer norms and the small embedding tables are not worth the rounding error they would pick up. The weights are expanded back to float32 once at startup, so inference speed is unchanged. The whole trade is about three times smaller for a one-time load cost and roughly 0.3% of weight rounding error.
Proving it is still the same model
Rewriting a model by hand is only defensible if you can show it produces the same numbers. So there is a small Python script that runs the real reference model over a set of deliberately awkward inputs and writes the outputs to a fixture the Go tests read back.
The inputs are chosen to break a tokenizer rather than to look representative. Accented text that has to be decomposed and stripped. Contractions and nested punctuation. A case number with digits and hyphens. A line of Chinese, where every ideograph becomes its own token. Mixed case with an emoji and runs of extra spaces. And one input that is a single word repeated four hundred times, purely to force truncation at the 256-token limit.
The two assertions are deliberately different. Token IDs must match the reference exactly, because any drift there is a bug in the tokenizer and will silently corrupt every downstream vector. Embeddings only have to match within 2e-4, because floating-point accumulation order legitimately differs between two implementations and demanding bit equality would be demanding the wrong thing.
That distinction took me a while to get right. Exactness is the correct bar in one place and the wrong bar three inches away.
When search finds nothing
The retrieval side has its own version of the same instinct: never fail, degrade. Keyword search and vector search are fused with reciprocal rank fusion, where each result scores 1/(k+rank) and k is 60. The appeal is that it needs no score normalization, and BM25 scores and cosine similarities live on completely unrelated scales.
Nearest-neighbour search has an annoying property: it always returns as many results as you ask for, no matter how weakly related the last one is. So results have to clear two floors, an absolute cosine of 0.25 and half the top score. Without them a one-word query drags in everything vaguely similar in the index.
But floors introduce a new failure. A question-shaped query can score under both while the corpus plainly contains the answer, and "no results" reads to a user as "nothing is indexed".
The last rung has a rule attached. The OR-matched search is only ever used when there is no semantic leg to catch paraphrases. As an input to the fusion step its one-word matches would rank stopword overlap and dilute genuinely good vector results.
Reflections
The rule was worth keeping. It is unusual to have a constraint sharp enough that it answers design questions for you, and "does this require a network call" answered a lot of them without debate.
What I underestimated is how much of the work is not the interesting part. The transformer was a weekend. Reconstructing paragraphs from PDF text positions, where you are inferring structure from the vertical gap between baselines and the ratio between font sizes, took considerably longer and involved far more staring at output. The model is the part people ask about, and it is not where the difficulty lives.
I would also be careful repeating this. Reimplementing a model by hand is justified by a specific constraint, and without that constraint it is just a worse version of a library someone else maintains. The version of this idea I would defend is narrower: pick one property you refuse to give up, make it non-negotiable, and let it force the architecture. Mine happened to be that the thing works on a plane.
The project is called Haypile, if you want to read the code.
Acknowledgements
The model is all-MiniLM-L6-v2 by the sentence-transformers team. I only reimplemented its forward pass; the weights and the architecture are theirs, and the reference implementation is what the golden tests check against.
Text extraction leans on PDFium, Chrome's PDF engine, and the Argon2 and SQLite builds that made the no-CGo rule survivable are hash-wasm and go-sqlite3.