3
$\begingroup$

I am planning building fully customizable backtesting trading engine in python from scratch as a open source project, the main features i am considering is,

  • It should be fully customizable from top to bottom
  • Customization is very easy and anyone can customize with a basic knowledge in python
  • It have a in built template engine for reports which is also customizable
  • Anyone can customize it as per their trading style

So what are the basic things which i have to consider for building a trading engine? Which are the python modules which is useful for this project So anyone know any material regarding this sharing a link will be very helpful....

$\endgroup$
7
  • 1
    $\begingroup$ Why not use zipline, instead of re-inventing it? $\endgroup$ Oct 26, 2014 at 3:06
  • 1
    $\begingroup$ Way too broad question. You emphasize customization requirements and that basically requires an incredible code stack that will be beyond any simple project. Just want you to be aware of the fact that you will most likely spend a year if you build from scratch. And I highly recommend not to attempt this project in Python but an OOP language that is much more performant, with a more stable and mature code base and that is actually suited to handle modules like broker API connectivity, OMS, PMS, parallel event based processing, and the like. Just my 2 cents... $\endgroup$
    – Matt
    Oct 27, 2014 at 3:25
  • $\begingroup$ I disagree on your latter statement: Python is more mature than say, Haskell (Tsuru, StanChart), Erlang (GSET) or OCaml (Jane St), so certainly, maturity isn't an issue. $\endgroup$
    – madilyn
    Oct 27, 2014 at 6:25
  • $\begingroup$ I don't see why parallelization is an issue either. The GIL in Python leans towards a multi-process rather than a multithreading paradigm for parallelization, but there's no reason to prefer the latter over the former. $\endgroup$
    – madilyn
    Oct 27, 2014 at 6:27
  • $\begingroup$ I also don't see why you can't do event-based processing in Python. You can write a TCP/IP stack in vanilla CPython if you wanted to, surely that's fast/robust enough considering some traders are using subpar trading platforms that don't even support basic order types such as HNS? $\endgroup$
    – madilyn
    Oct 27, 2014 at 6:31

2 Answers 2

11
$\begingroup$

Firstly, you'll probably be directed to consider Zipline. It's worth a look but I don't think that it's a good starting point, since:

  1. Quantopian's developers don't have a financial background and it shows through in the Zipline source code.

  2. Zipline is dreadfully slow if you compare it to any commercial platform with backtesting functionality in a compiled application, even the low-end retail trading platforms (e.g. NinjaTrader, Sierra, TradeStation).

  3. Zipline isn't very convenient for trading multiple products. I think the cheapest product that has that level of functionality is Deltix.

A modern processor should be able to backtest a moving average crossover strategy across an entire day of the OPRA feed (all products) without scheduling it overnight. Any less functionality or slower and you have poor developers. (I remember Goldman had 12-14 servers dealing with realtime OPRA in 2007-2008 and 2 persons rewrote the entire thing from scratch to target 128-bit architecture over a weekend. No reason why years of development on Zipline doesn't match up to 2 developers on a weekend before Stack Exchange existed.)

Here are some of the major considerations that you have to make before building your backtesting engine:

  1. How will you be storing/serializing your market data on disk and in memory?

    • One poor man's approach is to wrap it around a pandas dataframe, but this comes at the cost of abstraction and will slow down your backtesting engine. pandas is nice for data exploration, but not for a task that you will repeat many times.

    • How will you handle a data source whose size exceeds available memory?

    • How will you deal with unstructured market data?

  2. How will you be storing your outputs?

    • An obvious, naive problem is that you don't want to restart a backtest that took you 1 night to run if the application crashed midway. Another naive example is that you should be able to access old results from 6 months ago without repeating the backtest loop.
  3. What's your fill logic?

  4. What should your API expose? (e.g. Market orders, limit orders, instrument/price/volume queries, changes to fill logic)

$\endgroup$
3
  • $\begingroup$ I think pandas and HDF5 will be the good combination, But speed is still my biggest problem now. I am ready to do anything for make this super speed, have any idea? $\endgroup$
    – Jake
    Oct 28, 2014 at 19:47
  • $\begingroup$ In my mind one, the the worst lack of Zipline is the lack of bracket orders (Stop loss + Limit "child" orders attached to a parent order). $\endgroup$ Jan 1, 2015 at 17:38
  • $\begingroup$ +1 for "Quantopian's developers don't have a financial background and it shows through in the Zipline source code". $\endgroup$ Jan 7, 2017 at 23:35
0
$\begingroup$

There are some other opensource projects at github along with zipline which you can check for some additional inputs for your thoughts. Pyalgotrade is one such project where one can do back testing on their trading strategies

http://gbeced.github.io/pyalgotrade/

$\endgroup$

Not the answer you're looking for? Browse other questions tagged or ask your own question.