suggestions for improving monitoring of trading bots?

Question

I'm simply looking for tips and ideas to make my system a little more professional.

The structure: The botting script is hosted on a VPS. There's a database hosted on yet another VPS. There's a webapp periodically hosted locally.

The botting script executes the analytics and trades and periodically updates the the database (MongoDB) with new values; performance metrics, positions and current statuses / errors.

The webapp connects to the MongoDB, pulls the latest data and displays it in a simple data-table along a few simple graphs (I've also added a simple feature that lets me pause the script for certain exchanges and accounts).

This is a pretty vague / ambigious question so it might get flagged for being too broad, but I'm looking for general suggestions for improving the system (I've been running it for almost two years and I'm simply looking for ways to improve it; not just for the sake of improving the performance, but also because I'd like to improve my skills and approach).

For starters, someone might suggest SQL over MongoDB (I'm not saving transaction data at the moment; this is because some of the exchanges I trade on lack sufficient API support for that - but I'm considering to begin as it will allow for better analytics at a later stage as I have more data.)

I'm also running the database and actual script on different servers, perhaps there's a way for me to simply run them together and thus improve performance a bit (right now I've structured it in such a way that the updating of the database values happens at a predictable interval and doesn't interrupt any of the cycles).

I'd be happy to answer any inquiries. Oh, I should note that I'm a mediocre programmer at best (mainly programming in Python).

Answer 1

There's no right way to do this. Some universal tips I can give are:

Operational vs strategy monitoring

On the first pass, you need to distinguish between these two as the tools for operational and strategy monitoring will be different.

Operational monitoring

Operational monitoring may include the following:

• A heartbeat to ensure that your VPS is still alive, some kind of notification system (could be a text messaging system with Twilio, sound alert, email notification with IMAP, chart on a web-based dashboard).

• Scheduling of automated journaling and persistence of your system logs in a centralized place. A larger operation may use tools like the ELK (Elasticsearch, Logstash, Kibana).

• A simpler operation may just use cron jobs to journal, compress and send system log files (e.g. in /var/log) to a logging server or your workstation. Tools and daemons frequently encountered here include fluentd, collectd, Nagios.

• You may also build your own database to take advantage of off-the-shelf persistence, replication and redundancy guarantees as well as connectivity components. Mongo is fine here, but Postgres also has great support for unstructured data. Redis has the advantage of native support for pub-sub semantics whereas in Mongo you may have to replicate that behavior with tailable cursors. InfluxDB also tends to be used in conjunction with typical tools here.
• Some kind of visualization and dashboard for CPU and memory utilization, and so on. Memory leaks in your application may take down your server, and VPSes tend to be resource-starved.

Strategy monitoring

A huge part of strategy monitoring is a matter of knowing what to log. A related problem is deciding on your serialization strategy both on disk and on the wire, where there's even more sensible options to choose from than database management systems.

Unfortunately this a bit of a chicken-and-egg problem where having seen it in a mature, production environment or knowing what your portfolio will look like at scale will give you a tremendous headstart in designing this. If you don't know what your strategies will look like when you're trading in the order of hundreds of thousands of orders per day and billions in notional exposure, then figuring it out by first principles will lose you a lot of time.

Some universal advice I can give here would be:

• Make sure your logging format has some backward compatibility.
• Don't prematurely optimize, every first year CS graduate has tried handcrafting his own proprietary binary serialization format.
• Human readability (json, unicode plaintext), a self-describing format (Avro), an interface description language (thrift), and multi-language support (protobuf) are all pretty desirable properties, decide what's most important for you.
• Download some free trading platform and use it for inspiration for what types of fields you would log, e.g. order side, timestamp, order quantity, time in force, symbol etc. are all fairly standard and you can compose more complex metrics from these ex post at the end of day or later.