What is the current lowest possible latency for TCP communication?

Question

I have two machines over a 10Gb network that need to communicate with each other through a TCP connection. In terms of technology, what is the current lowest latency possible for this to happen? What are the top players in the market using to bring their TCP latency down to the bare minimum?

Answer 1

For ultra-low-latency network applications it is mandatory to use a single-threaded, asynchronous, non-blocking network library. You can and should handle multiple TCP connections inside the same reactor thread which will be pinned to a dedicated and isolated cpu core.

To give you an idea of TCP latencies you can take a look on our benchmarks using CoralReactor, which is an ultra-low-latency, garbage-free, non-blocking, asynchronous network library implemented in Java.

Messages: 1,000,000 (size 256 bytes)
Avg Time: 2.15 micros
Min Time: 1.976 micros
Max Time: 64.432 micros
75% = [avg: 2.12 micros, max: 2.17 micros]
90% = [avg: 2.131 micros, max: 2.204 micros]
99% = [avg: 2.142 micros, max: 2.679 micros]
99.9% = [avg: 2.147 micros, max: 3.022 micros]
99.99% = [avg: 2.149 micros, max: 5.604 micros]
99.999% = [avg: 2.149 micros, max: 7.072 micros]

We are not aware of any software-based network library, even in C/C++, that can go below that.

Keep in mind that 2.15 micros is over loopback, so I am not considering network and os/kernel latencies. For a 10Gb network, the over-the-wire latency for a 256-byte size message will be at least 382 nanoseconds from NIC to NIC. If you are using a network card that supports kernel-bypass (Open OnLoad) then the os/kernel latency should be very low.

Disclaimer: I am one of the developers of CoralReactor.

Answer 2

You probably want more than a low "TCP latency". Anyway, what latency is it? The TCP layer to connect to your 10Go (i.e. The implementation of the TCP protocol only)? The implementation of a protocol to read the market data (direct native feed? FIX?)? Or to "move" the content of the feed to a zone of memory shared with your application? And what about the messages you want to send to the market? Don't you want them to go fast too?

There are generic answers in Market Microstructure in Practice (Laruelle, Lasnier, Pelin, Burgot and L).

The rule of thumb is if you are under the millisecond for all the layers I listed, you should be happy. There is not one provider for all of them, you have to investigate on your own. My advice is (from the book): if you really want to go fast, use an FPGA with a physical TCP/IP block, and put on it all the protocols. For simplicity you can keep the state machine of your trading logic outside of the hardware, but keep all the analytics inside it.

On the FPGA for datafeed side, you have firms like novasparks.