Inside the Hot Path: What Building VOLLO Taught Us About Low-Latency AI Inference

Everyone in an electronic market sees the same event at the same instant. What happens in the next few hundred nanoseconds decides who captures it. This is our view, from the inside of that problem, of what an AI inference platform actually has to be to survive there — and why we built VOLLO the way we did.

We spend most of our time at myrtle.ai thinking about one very specific, very narrow problem: how to get a trained model’s decision out of silicon and onto a wire as fast as physically possible, with no surprises. It’s a strange thing to specialise in until you’ve watched what happens on the other side of that wire.

In a live market, every participant capable of receiving a price update receives it at effectively the same moment. There’s no head start on the raw information. Whatever happens next — who decides fastest, and who acts on that decision fastest — is the entire contest. And the contest resolves itself extremely quickly: the moment an order is sent, it changes the very condition that made it worth sending in the first place, so an opportunity that isn’t captured in the first few hundred nanoseconds isn’t captured at all.

The loop that actually matters

Strip away everything else and a trading system is running the same small loop, over and over, many millions of times a day: watch the market for something that changed, decide what that change means and what to do about it, and get a trade to the exchange before anyone else does.^[1] The industry has a name for the part of that loop under time pressure: the hot path. It starts with a trigger — typically a market data event — and ends the moment a responding order leaves the system. The time in between is trigger-to-order latency, and for a serious market participant, it’s the only latency number that really matters.^[1]

Inside that hot path, a handful of distinct jobs recur, each with its own budget. Prices need to be continuously requoted as the market moves. An opportunity needs to be recognised and aggressively taken before it’s gone. Once a trade fills, everything connected to it — every related price the fill just made stale — needs to be repriced and republished before a faster competitor trades against the old number. And a resting order that’s about to become a bad idea needs to be pulled before someone else takes the other side of it.^[1] A system can be excellent at three of those and still get picked apart through the fourth. That’s the property that makes this whole area unforgiving: it’s a worst-path problem, not an average-path one.

And measuring any of this is harder than it sounds. Once a system’s response time gets close to the physical limits of the hardware it’s running on, the noise in your instrumentation can be as large as the thing you’re trying to measure.^[1] That’s not a footnote — it shapes how seriously this kind of system has to be engineered, tested and monitored, long before AI enters the picture at all.

Why the engineering keeps ending up in the same place

Every account we’ve come across of how a firm actually gets its hot path fast follows a remarkably similar arc. It starts as an ordinary, well-written piece of software. It stays there until two things become the dominant source of delay: the operating system moving other work onto the same CPU core, and the general-purpose networking stack sitting between the network card and the application.^[1] The fix for the first is to pin that core to nothing but the trading process. The fix for the second is to bypass the kernel and talk to the network card directly, trading away the safety net a general-purpose OS provides for a large chunk of the latency it costs to have it.^[1]

Once both of those are done, the next bottleneck is the thing doing the actual computing: a general-purpose processor, executing instructions one after another, with all the overhead that implies. The next step down that path is to stop executing instructions altogether, and instead implement the decision logic directly as physical circuitry — a field-programmable gate array, rewired for exactly one job. The mechanism that makes this work is elegant and, once you’ve seen it, hard to unsee: enumerate in advance every pattern the logic might need to respond to, precompute the response for each one, hold all of those precomputed responses behind a fast piece of matching logic, and the instant a live pattern arrives, release the answer that’s already sitting there waiting for it.^[1] Nothing is being decided in the moment. The deciding already happened, at design time; what happens on the wire is just retrieval.

That is, not coincidentally, exactly the mechanism VOLLO automates. Where a hardware team would hand-write that logic in a hardware description language and burn it onto a chip, VOLLO takes a model you’ve already trained in PyTorch, TensorFlow or ONNX and compiles it directly into that same kind of fixed, cycle-accurate gate logic — no RTL, no FPGA toolchain, and no hardware team standing between a data scientist’s model and a deployable, deterministic circuit.^[2] We built it because the destination at the end of that latency progression was always going to be hardware; the only question was whether every team chasing it had to rebuild the same compiler from scratch.

It’s also why we stopped short of custom silicon. An application-specific chip pushes clock speed further into the gigahertz range than an FPGA ever will, but it does so by giving up the thing that makes an FPGA useful for a model that’s still being iterated on: the ability to recompile and redeploy in minutes rather than commit to a fabrication run measured in months.^[1] For a model that gets retrained as markets and strategies evolve, that flexibility isn’t a nice-to-have.

Latency became the floor. The model became the edge.

For a long time, the entire point of the exercise above was to be faster than the other side. If your response to a market event landed before theirs, you captured the mispricing before they could: the arms race was worth running because it paid for itself directly. That calculation changes once most serious participants have made the same investment. When everyone’s hot path is measured in low single-digit microseconds or less, being marginally faster than the next firm stops being enough on its own — latency turns into table stakes rather than an edge.^[1]

What we consistently hear from teams operating at this end of the market is that the effort has shifted accordingly: substantially more engineering time now goes into the quality of the models and data behind a decision than into shaving further latency off a system that’s already fast.^[1] That shows up as two recognisable patterns. One is a slower, richer model that has time to digest more data and identify a scenario worth acting on, paired with a fast trigger that fires the instant that scenario shows up live. The other is a smaller, faster model — often a distilled version of the first — running as close to the decision point as possible, watching signals in real time and acting on them directly.^[1]

Both patterns share the same dependency, and it’s the one we exist to remove: neither works if the inference step itself becomes the hot path’s new bottleneck. A slow model with a fast trigger only protects the opportunity it found if the trigger really is fast and its timing is predictable. A fast model at the edge only earns its keep if “fast” holds under load, not just on a good day. This is the gap between wanting a richer model on the hot path and actually being able to put one there — and it’s the specific gap VOLLO is built to close. Our most recent audited result on the STAC-ML Markets (Inference) benchmark — 1.89 microseconds at the 99th percentile for an LSTM model, the lowest yet recorded on that suite — is what “the inference step keeping up with the rest of the hot path” looks like when it’s measured rather than assumed.^[2]

Treat the inference path like it can lose you the firm

None of the above is worth much if it isn’t trustworthy. The industry’s own cautionary tale is a firm brought to the brink by a single bad software deployment; the lesson every serious trading operation drew from it is that a bug on the hot path isn’t a minor inconvenience, it’s an existential risk with no offsetting reward.^[1] The engineering response to that we’ve seen described is a simple discipline: hardware logic gets reviewed and tested exactly like software does, through the same automated pipelines, held to the same standard, because a change that lives in gates rather than in a source file is no less capable of taking the firm down if it’s wrong.^[1] And monitoring at this scale has to be sensitive enough to catch a regression as small as a cable that’s a few centimetres longer than it needed to be — the kind of change that looks like measurement noise right up until several days of data confirm it wasn’t.^[1]

We take the same view of what it means to compile a model onto FPGA silicon. A deterministic, audited, cycle-accurate execution path isn’t a performance feature bolted onto VOLLO after the fact; it’s the reason the whole approach exists. If a model’s behaviour on the hot path can’t be reasoned about, measured, and reproduced exactly, it doesn’t matter how good the model is underneath it.

Where this leaves us

Put the pieces together and the shape of the problem is fairly clean. The loop that decides who wins a trade is unforgiving of anything slower than its worst path. The engineering response to that has, for as long as this industry has existed, ended up in the same place: software first, then operating-system isolation, then a direct line to the wire, then hardware. And now that most serious participants have already made that investment, the thing worth spending engineering effort on has shifted to the model itself — provided whoever builds that model can still get its decision onto the wire fast enough, and reliably enough, that having a better model was worth the trouble.

That’s the specific, narrow problem VOLLO exists to solve: taking a model a data science team already trained, and compiling it directly into the same deterministic, gate-level execution that trading engineering teams have historically had to build in-house, one hardware description language line at a time. We’d rather that be a compiler you run than a team you hire.

FAQs

Sources and further reading

[1] This piece draws on general engineering patterns reported across the low-latency trading industry, including in: Orosz, G., Klaric, I. and Spevack, J., “Software engineering at a proprietary trading company: Optiver,” The Pragmatic Engineer, 11 August 2026. Specific facts about Optiver’s own organisation, people, hiring or financial results in that piece are not repeated or implied here; this article reflects myrtle.ai’s own perspective on industry-wide engineering patterns and on VOLLO.

[2] myrtle.ai, “Setting a New Record for Single-Digit Microsecond Latency Inference in Capital Markets,” 24 April 2026 (STAC Report, SUT ID MRTL260323).

VOLLO is a trademark of myrtle.ai. STAC and all STAC names are trademarks or registered trademarks of the Securities Technology Analysis Center, LLC. This piece reflects myrtle.ai’s own perspective, informed by general, publicly reported engineering patterns across the low-latency trading industry; it does not describe the internal practices, people, or performance of any specific firm named in its sources.