I need to persist every single FIX message received or sent by my order gateway for auditing purposes, however it takes more than 1 millisecond to write the bytes to disk. I tried to write in chunks of 64k but that did not help either. Was wondering what is the best practice to audit their trading gateway without introducing latency into their strategies.
$\begingroup$
$\endgroup$
4
-
$\begingroup$ This isn't really a finance question, but I'll try answering it anyway: have you tried a RAM disk? $\endgroup$– user59Nov 3, 2014 at 18:43
-
1$\begingroup$ Thanks for the comment. I understand the the main focus of this forum is quantitative finance, but I assume many people here are dealing with similar technology issues for trading technology. Would you recommend a different forum for this kind of question? $\endgroup$– Alfred WilkingsNov 5, 2014 at 14:17
-
$\begingroup$ I believe it to be of interest for (some) practitioners, so I like it and believe it to be on-topic. If you (the community) think otherwise, please use meta for discussion about the on-topicness of the IT aspects of Quantitative Finance. $\endgroup$– Bob Jansen ♦Nov 6, 2014 at 12:18
-
1$\begingroup$ You should also look into Chronicle, or perhaps the newly-released Aeron by Martin Thompson $\endgroup$– experquisiteNov 12, 2014 at 18:34
Add a comment
|
4 Answers
$\begingroup$
$\endgroup$
The correct way to do file I/O without introducing latency is to do it asynchronously, in other words, the logger thread just passes the message to another thread that is actually doing the disk I/O.
In the past, it was assumed that it was impossible to do it without creating garbage and lock-contention, but with the emergence of lock-free queues it is now possible to do it using pipelining for inter-thread communication with ultra-low-latency.
For example, CoralLog can easily log and persist messages in less than 100 nanoseconds, with throughput number above 4 million messages per second.
Disclaimer: I am one of the developers of CoralLog.
$\begingroup$
$\endgroup$
As suggested you could try a RAM disk or some other form of quick caching and then write to hard disk at intervals or even off-hours.
You could also introduce some multi-threading into your application and create a dedicated thread (or service) to handle the logging side exclusively.
$\begingroup$
$\endgroup$
1
RAM is super fast but not persistent. If you suffer an outage, such as a power loss, all your FIX records will be gone and your audit trail broken. Solid state disks (SSD) in a RAID configuration give a great combination of throughput and reliability.
In my day job I design database applications for a living. When considering system failure and recovery we have two concepts to keep in mind - recovery time objective (RTO) and recovery point objective (RPO). RTO is the estimated time to get a failed system up and running again. RPO is how much data loss you can accept as a consequence of the outage. From the wording of your question it sounds like your RPO is zero i.e. it is not acceptable for this system to loose any FIX records as a result of a system failure. If this is the case you just cannot use volatile storage (i.e. RAM) for this data. It has to go to persistent media immediately. The trick it to minimise the extra latency inherent in persistent storage. If you can, in fact, accept some small data loss during system failure this, needs to be agreed with your users and management and the system costed and designed accordingly.
There are low latency products specifically for persisting high volume data. RIAK is one that springs to mind, though the search engine of your choice may suggest others.
Another alternative would be to take this function out of the application and ask the network to log all traffic that looks likes a FIX message. There are third part tools and appliances which can do this out of the box.
-
$\begingroup$ The RAM persistence can easily be overcome by short intervals between the disk write and permanent storage (database or whatever). $\endgroup$ Nov 7, 2014 at 17:15
$\begingroup$
$\endgroup$
Don't write a log file. It's a very old fashioned way to achieve the result you want.
Look into packet capture using a span port on the switch to which your FIX engine is connected.
By running packet capture you grab all of the FIX messages sent and received with zero impact on your application - the packet capture just grabs the messages off the wire.
You then need to decode the captured packets. Depending on your FIX engine vendor this may be something that can be done trivially or it may require some work.
