10
$\begingroup$

Okay. This might be a pretty dumb question, but I really want to know what is it that the high frequency trading firms write in terms of services that requires C++.

Background

I am a Rust and TypeScript developer and I was having a conversation with a guy I met at a function. He was a C++ developer at a high frequency trading boutique. As we were discussing the merits and demerits of the languages he brought up that Rust will never replace C++ just because of the sheer speed.

Problem

I am a strong believer in Rust. And I do believe that it can tackle a lot of problems that other languages pose without compromising a lot on speed: benchmarks

Granted, C++ is still faster, and I am no one to argue its dominance in a sector I know jack all about.

But I really want to know how C++ is used that proves it to be faster for high frequency trading firms.

Bigger Problem

I know jack all about high frequency trading. But I really want to know.

So my question is, what kind of services or cronjobs or APIs do people write using C++ for high frequency trading firms. Forgive me for my ignorance, but I come from a more API centric background, so we generally use services/contracts and APIs like, get_balance. What is it? I can't seem to find anything on the open web. Or maybe I am not looking in the right place?

$\endgroup$
3
  • 5
    $\begingroup$ This Reddit thread is a bit old but the discussions in it can maybe answer some of your questions: Production deployment of Rust in HFT system at Korean financial company $\endgroup$
    – Alper
    Nov 14, 2021 at 18:48
  • 2
    $\begingroup$ I think you question is a "emacs vs vi" one. Of course other languages can be faster than C++... personally my preference would go to ocaml ;{)} stackoverflow.com/questions/1924367/… $\endgroup$
    – lehalle
    Nov 14, 2021 at 19:57
  • 1
    $\begingroup$ Stating that c++ is (generally) faster than rust is a bit naive I would claim. However, why does everyone write in English here? It's the same with coding, people use what most people understand (and what works well). $\endgroup$
    – AKdemy
    Nov 15, 2021 at 7:26

5 Answers 5

12
$\begingroup$

I ran all of the research and trading at a high-frequency market making firm with well over a million lines of C++. Now I'm at Databento, where we have about 1 line of Rust for every 1.5 lines of C++ and 0.5 lines of pure C, and most of our frontend is in TypeScript. Our stack at Databento is developed against much of the same hardware as the top trading firms today (e.g. Arista/Metamako layer 1 switches, Xilinx UltraScale+ VU5P FPGA, Xilinx NICs, Nexus 3548).

The main reasons to use C++ over other likely languages like Rust or Java are transparent performance and availability of developer resources.

  • The C++ community publishes more content on performance optimization than that of any other language community. So between C++ and any other language with manual memory management, it is often easier to understand performance in C++ through literature.
  • While concepts like latency measurement, memory profiling, false sharing, cache coherence, instruction pipelining, NUMA, SIMD, wait-free structures, fast hashing, kernel bypass networking etc. exist independent of language, most of the example implementations and explanations you'll find of these online are in C/C++.
  • Most proprietary or open source tools for electronic trading that you'll likely use will have C/C++ bindings, libraries, APIs, transpilers etc. Or will at least have more mature support for C/C++ than other languages.
  • Many high quality commercial and open source libraries in C/C++ you can rely on, like folly, SBE, Xilinx ef_vi, libvma etc.
  • C++ has existed for a longer time, so there's a lot of tribal wisdom passed down by electronic trading pioneers during the heydays like pre-2008; it's a lot easier to hire C++ engineers for a HFT role too.

As @experquisite said, the parts of a trading system that are commonly written in C++ include any part of the critical path of an order action:

  • Execution: order client, order router, internalization, order gateway etc.
  • Data: feed client, market data parser, normalization, book construction, data gateway etc.
  • Business logic: monetization strategy, position and order bookkeeping, features or signals, pre-trade risk etc.
  • Utilities: timestamping, instrument definition lookup, symbol hashing, interprocess or interthread communication, IP/UDP/TCP stack etc.

But also because of (i) the large volumes of data involved and how (ii) it's faster to go to production using the same code as research, many firms also use C++ for data exploration, backtesting and simulation etc.

There's nothing wrong with using Rust instead of C++ in all of these places. But in my experience, as of January 2023, here are some of the areas that will probably be more inconvenient to do from scratch in Rust:

  • Mathematical routines (regressions, decision trees, NNs etc.) and linear algebra.
  • Visualization, plotting or GUI frameworks.
  • Anything to do with layers 3-4, packet processing etc.
  • Latency-optimized data structures. (But some of Rust's de facto standards like crossbeam::channel are not bad, and the SwissTable port std::collections::HashMap is a good starting point compared to the deluge of worse options in the C++ standard library and Boost.)
$\endgroup$
6
$\begingroup$

I read your question here on stackexchange and was curious myself, so I did a little digging and came across this:

A thread on Quora

Where can I learn C++ for HFT?

Theodore Weld Smith (programming primarily in C++ and OCaml) answered:

One of the biggest distinctions from what C++ looks like for HFT versus in other applications is the latency aspect.

Writing good low latency code means

  • knowing how concurrency and parallelism works and how to use it
  • knowing what exactly the compiler is doing
  • knowing how memory is being dealt with
  • knowing how to write code for the specific hardware in use
  • knowing how the OS kernel the algorithm operates on works, and how to deal with it (e.g., kernel bypassing)

There are a bunch of things you can do in C++ that will “simulate” what HFT work is like; thus preparing you for it.

  • Writing device drivers (C or C++)
  • Using and learning about how frameworks such as CUDA work.
  • Learn about how the FIX protocol works"

And Daniel Wisehart (Longtime C/C++ Developer) answered:

Start by learning to write simple, clean C++ that is easy to understand by any competent developer with as little magic or reliance on the compiler to do the right thing as possible. Your intentions should be as clear to the compiler as they are to another developer. Some shops use libraries where they have no idea what is happening under the hood, but most prefer to roll their own for low-latency pathways. This means it is good for you to understand some of the issues confronted when writing those magic libraries, especially within the narrow set of problems you will deal with.

For example, a sorting library may have all sorts of optimizations for all sorts of initial conditions qsort will encounter, but if you only do insertion sorts—which is a common implementation in HFT—you really neither need nor desire a sorting library that handles the generic solution.

You will use a lot of hash map’s, but there are a lot of things hash maps can be optimized to do—such as merge or an apply—that you will likely never use. So you need to understand how to create a hash that is optimally sized for the number of entries you will deal with, statically allocate what you can and pool memory that will take care of hash collisions. Even if the firm already has an optimal hash for their use, you need to understand how it works, what choices were made and why other options were not pursued.

The more you can understand about the hardware you are running on, the better. With modern processors there are various rings of memory around the ALUs, and as you get further away from the ALU, the number of cycles to access something increases. We have a saying on this: in the old days, memory was fast, but if you had to go to disk you missed the trade because it was just too slow. Today, cache memory is fast, but if you have to go to main memory, you missed the trade.

We ran the entire Linux O/S in main memory on diskless machines—OK, there was a little bit of disk for local logging in case of a network failure, but no code was stored there—and we tried to keep everything in the trading code path in layer two cache or better. NICs with DCA (direct cache access) and user space data—aka kernel bypass—helped a lot.

If you have a good understanding of all the code—yours, the libraries and the O/S—that runs from network to application back to network, you will be a real asset as a C++ developer for an HFT firm.

Source: https://www.quora.com/Do-HFT-programmers-write-in-Assembly-and-Binary-code-to-further-optimise-their-systems

I know that might not be directly what you were looking for, but I hope it answers some of your questions.

Also, if you are interested to learn more about how to use Rust instead of C++ for quant trading, especially when trading on the blockchain, feel free to contact me on Discord. We are currently building a quant trading system in Rust. My discord is "Pierre#7374"

$\endgroup$
4
$\begingroup$

C++ is usually used for strategy logic and signals.

While FPGAs can get down to sub-100 ns tick-to-trade these days, most of that time is spent in serdes, TCP/UDP stack, which leaves not much time for complex signals.

Moreover, FPGA boards don't come with a lot of onboard memory.

So it's more typical that a strategy and its signals, written in C++, pre-compute the space of possible actions and populate a lookup table on the FPGA.

$\endgroup$
1
$\begingroup$

C++ is generally used for the “fast path” of making a decision and placing or cancelling orders as a result of it. This usually means code which is spinning in a tight loop, ideally all in L1 cache, monitoring memory for notifications/messages from the outside world upon which it will update its forecasts, risk model and take action. Often the action is updating trigger values in an FPGA networking card so that it can respond to the final event completely in hardware.

Since the entire “tick-to-trade” time from awareness of an event to having responded to the event should be under a microsecond, it becomes quite important to understand completely how the source code is getting compiled into machine code, and HFT developers spend quite a lot of time in perf and the disassembly of their work.

Rust (I believe) is perfectly fine, and I love it, but it’s just currently a little harder to be certain that it will continue to optimize your code the same way through compiler revisions, since fewer things are currently guaranteed in the language spec. Furthermore, for this specific style of code, there is never any dynamic memory allocation, so to some extent the benefits of Rust’s memory safety don’t help. That said, the language and type system is still far more pleasant than C++ IMO.

I believe we’ll start to see more and more Rust in trading firms, if only because it’s so pleasant to work with.

$\endgroup$
0
$\begingroup$

I don't have any experience with Rust. At a high enough level I believe the HFT community started using C++ because it was good at delivering close to metal performance with low (zero if you like) cost abstractions. Over the years, that's where the industry talent and existing code base has been. For another language to take over, I think the benefit would have to be significantly higher.

Anyway I would recommend watching these two videos to give you a sense of the speed required by HFT, and how it is delivered through C++. Maybe it would help you figure out how to deliver similar or better performance through Rust.

CppCon 2017: Carl Cook: When a Microsecond Is an Eternity: High Performance Trading Systems in C++

Core C++ 2019: Nimrod Sapir : High Frequency Trading and Ultra Low Latency development techniques

The ultimate goal for HFT is that you should be able to send your trade faster than your competition (or you'll get picked off by others - i.e. lose money). Let's call it a sub-microsecond timeframe.

Here are my not so well organized notes from the videos.

  1. Hotpath is exercised very infrequently (say 0.1% of the time). When it does, it has be superfast. An example could be "monitor the quotes on the exchanges. When this specific condition is met, send a quote / trade to the exchange"

  2. Critical code has to be fast not most of the times, but all the time. Jitter is unacceptable. Hence any garbage collection framework wouldn't work. Dynamic memory allocation doesn't work either (though it's okay to allocate memory on the heap during setup say before market opens)

  3. Operating systems, networks and hardware are generally designed with fairness in mind. Your code has to fight those natural impediments. Say for example by disabling all cores except one, disable hyperthreading, disable any OS processes that are not critical to running your application (basically everything else), preventing the kernel from doing anything (certainly not asking it to do something for you). Disabling other cores and hyperthreading would help reduce cache misses.

  4. Keep the data you need in the cache. Reading from memory would be too slow (and of course reading from file system is out of question). Also keep the cache warm. Since 99.9% of the times your code will not send an order, the branch predictor will be incorrect when you need the data. The technique generally used is to send the data to the (specialized) network card anyway (which would choose not to forward the data based on a flag).

  5. Be mindful of cache line size. Say if you need 4 bytes, and the cache line on your system is 64 bytes you can put other information in your data structure that you probably would've done a look up for anyway. For example you have instrument ticker and price in one structure, and market information in another structure. You can add the market info in the instrument struct. That de-normalized approach would be frowned upon elsewhere, but could give you extra efficiency here.

  6. Testing is the key to knowing what works better. For example you'll have to test different compilers, compiler flags, machine architecture, 3rd party libraries etc. Your hunch may be an okay place to start, but testing is how you would know what works better.

  7. Even though you would likely not code in assembly, you want to look at the generated assembly code for the critical path to make sure its reasonable (especially as you compile using different compilers, versions, flags etc). For example less jumps and fewer instructions are more desirable.

  8. Don't leave decisions until run time that can be handled at compile time. For example with template code as opposed to virtual functions. Template functions can even replace certain if conditions.

  9. Exceptions are okay to use, if they're truly exceptions. Because exceptions are zero cost unless you throw them. And if you do, you probably have some problem anyway (so you wouldn't be sending an order). Don't use them for control flow (which is a bad coding practice in my opinion anyway)

  10. If you have a series of function calls, each validating the input that's extra time. An alternative approach would be to assume that the prior function is returning the appropriate value. That way the program could fail ungracefully (which might be okay), but if it succeeds in sending the order to the exchange it would be fast (desirable)

This is just a short list to give a layperson an idea of the kind of tactics that are used in practice. In reality there would be 50 others - you just have to test and find out. With the understanding that every nanosecond counts.

$\endgroup$
1
  • 4
    $\begingroup$ Because you posted this as an answer, it would be much better if you summarized in sufficient detail the answer these videos provide to the question in case they are deleted or moved. $\endgroup$
    – Alper
    Jan 1, 2023 at 8:43

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.