The application was receiving binary messages from the exchange over multicast, rebuilding state of the market, running various (simple) algorithms and responding with orders within 5 microseconds of the original message, at up to 10k messages per second.

With SBCL you can write a DSL and have ability to fully control the resulting instructions (through vops). It is just the question how dedicated you are to writing macros upon macros upon macros.

I used this to the fullest extent and the result was as good as any hand optimized C/assembly.

For example, the binary message parser would receive a stack of complicated specifications for the messages in the form of XML files (see here if you are curious: https://www.gpw.pl/pub/GPW/files/WSE_CDE_XDP_Message_Specifi...), converted XML to DSL and then, through magic of macros, the DSL was converted to accessors that allowed optimal access to the fields. Optimal here mans the assembly could not be improved upon any further.

Large parts of the application (especially any communication and device control as it was done with full kernel bypass) was written in ANSI C but the Foreign Function Interface makes integrating it into the rest of application a breeze.

I write all of this, because I frequently meet complete disbelief from people that Lisp can be used in production.

I personally think it is exactly the opposite. Lisps offer fantastic development environment. The problem are developers who are mostly unable to use Lisp for work effectively, I guess due to too much power, freedom and complete lack of direction on how to structure your application.

If some big shot corporation with weight in the industry doesn't shove it down the throat of devs, then it isn't possible.

And regarding Lisp, most tutorials keep ignoring that arrays, structures, stack allocation, deterministic allocation,... are also part of Common Lisp.

Well... don't give my application as an example.

I have worked around the GC problem by using CL as kind of compiler for the application -- I built high level DSL that was then compiled to binary supplemented with small blocks written in ANSI C.

If you think about it, SBCL already does this and so does JIT in JVM.

I will give you an example:

Before the order goes to the exchange, it needs to be validated against a set of rules. Like "do we have enough money to run this order?" or "How much of allotted limit this algorithm still has?" or "Is volatility on this market within bounds for this algorithm to be used?"

The rules were written in Common Lisp DSL, then Common Lisp code converted it to a decision tree, then optimized decision tree, then compiled optimized version of the decision tree to a binary function. That function itself had no longer anything to do with Common Lisp, it could have just as well been written in ANSI C.

Then Common Lisp code wired these functions to form the application.

While most of the application code was actually Common Lisp (and some 20% of ANSI C), if you were market order and observed what instructions are handling you, none of them would actually be Common Lisp.

After the initial setup, the Common Lisp application was running on only one core, and the constructed binary took over all over cores and was running without garbage collection on memory regions allocated outside of Common Lisp system.

I hope this description makes sense... I have never before or after met an application that was built this way. As far as I know it is only one of its kind.

So efficiency was an important consideration right from the start.

Other languages that most people use today grew at later times (borrowing from Lisp) when the resources wasn't such a big deal.

I consider languages like PHP, Ruby and Python peak examples of mainstream languages that were created with barely any consideration for efficiency. This coincides with 1990s and early 2000s where we saw dramatic increase of system resources and especially memory without much need to use it efficiently.

Nowadays it improved a little bit as we learn to run bigger loads so newer mainstream languages (like Rust) and some older (like Java, C# or JavaScript) are putting more effort on efficiency.

Working in the field, that seems overconfidently good, considering its faster than most wire to wire SLA of world class FPGAs doing market data translation.

5us is actually quite slow.

This is from 2014, around the time I worked on this project:

https://stackoverflow.com/questions/17256040/how-fast-is-sta...

They cite "750 to 800 nanoseconds" wire to wire latency and that their next platform is going to be even faster.

They were using FPGA and yes, FPGA is faster, but not as much faster as many people would think.

First, market events come in isolation. There are no concurrent messages. WSE guarantees 1/10000th of a secend between each message. So you have your entire machine dedicated to executing the order.

FPGAs are usually used to run multiple copies of same net to speed up a simple problem but that is not the case here.

Second, FPGAs are usually used as a shortcut to optimize the execution of the problem. With FPGA you say "rather than trying to solve this problem with generic instructions that add a lot of delays I will just design dedicated net that will not be bothered by the generic baggage".

So in generic assembly you may want to write a branch and the branch predictor may go the wrong way and that costs. On FPGA you design your net and so you just go straight to the point.

But it doesn't mean you can't design normal code to be fast. You just need to be aware of actual cost of every single instruction of the critical path.

And third, FPGAs are clocked slower. What this means you have to do a lot per clock cycle on FPGA just to be on par with x86 core.

That application I worked on it was not intended as HFT. 5 us was an arbitrary goal we wanted to reach knowing full well that it is way behind HFT-ers.

The largest influence were definitely LMAX articles and Disruptor pattern.

SBCL was comparatively easy. Basically, if anything caused problems I just moved it to C and called using FFI. Think in terms of writing a C program but using pieces of assembly when C is not enough for some reason.

Even though a large part of small pieces was moved to C, the application still felt like Lisp. It just orchestrated a large library of utilities to do various things. I still had fully functional REPL, for example.

The hard part of the project was full kernel bypass. After the initial setup, the application stopped talking to Linux kernel except for one CPU core that was devoted to running OS threads, some necessary applications and some non-performance-critical threads of the algotrading framework (like REPL, persistence, etc).

All except for one core were completely owned each by a single thread of the application and never did any context switch after initial setup.

I hadn't appreciated that level of thread isolation was possible in SBCL, but of course it makes sense. Presumably you had some kind of instrumentation to let you know if a stray syscall had slipped in?

Don't know if you can call it instrumentation... I wrote an extremely hacky patch for the kernel to detect when any piece of kernel is running on anything than core 0 after certain flag was set.

Remember, it is not just syscalls. Even something as simple as accessing memory can cause switch to kernel to resolve TLB entry if you don't set up your memory correctly to prevent this from happening.

Yeah... I know. Now a bunch of people will come and explain how this could be done the right way. I just didn't care at the time to invest more time than necessary to get this right.

I'm very interested in the kernel bypass technique to talk to the hardware, I'm assuming it was the NICs ring buffer.. how was this achieved in userspace?

And that is how plenty of polyglot devs can enjoy their favourite language.

Two things in the implementation are performance killers: Laziness and boxed maths.

- baseline: Taking your original implementation and running it on a sequence or 1e6 elements I generated, I start off at 1.2 seconds.

- Naive transducer: Needs a transducer of a sliding window which doesn't exist in the core yet[0], 470ms

- Throw away function composition, use peek and nth to access the vector: 54ms

- Map & filter -> keep: 49ms

- Vectors -> arrays: 29ms

I'd argue only the last step makes the code slightly less idiomatic. Might even say that aggressively using juxt, partial and apply is less idiomatic than simple lambdas

You can see the implementation here

[0] https://gist.github.com/nornagon/03b85fbc22b3613087f6

[1] https://gist.github.com/bsless/0d9863424a00abbd325327cff1ea0...

Edit: formatting

I also found a small bug, that you'll want to use `(elt s (+ x 7))`, not `(+ x 8)`. `elt` is 0-indexed, so first and last of an 8 element list will be 0 and 7.

Relevant Stackoverflow which contains a lot of simple suggestions to hopefully shore up the deficit compared to SBCL: https://stackoverflow.com/questions/14115980/clojure-perform...

To compare language compiler and runtime performance you should at least use similar data-structures and algorithms.

> but, you can generate very fast (i.e., more energy efficient)

I'm actually not sure this is true, I was surprised the other day to find a study on this and to find that the JVM was one of the most energy efficient runtime.

This was the link: https://thenewstack.io/which-programming-languages-use-the-l...

And what you can see is that execution time doesn't always mean more energy efficient. For example you can look at Go and see that it beats a lot of things in execution time, but loses to those in energy efficiency. Like how Go was faster than Lisp yet less energy efficient than Lisp.

The research showed that the Go version of the programs were in average faster in terms of execution time and consumed less memory than the alternate Lisp versions, yet the Lisp versions consumed less electricity and were thus more energy efficient.

So the interesting bit here is that better performance and lower memory consumption doesn't always mean more energy efficient.

Now, there is definitely a link between performance and energy use, the research did show that for the most part, faster execution often reflected in lower energy spent, but there were surprising variation within that range where it wasn't always clear cut.