I am planning building fully customizable backtesting trading engine in python from scratch as a open source project, the main features i am considering is,

  • It should be fully customizable from top to bottom
  • Customization is very easy and anyone can customize with a basic knowledge in python
  • It have a in built template engine for reports which is also customizable
  • Anyone can customize it as per their trading style

So what are the basic things which i have to consider for building a trading engine? Which are the python modules which is useful for this project So anyone know any material regarding this sharing a link will be very helpful....

  • 1
    $\begingroup$ Why not use zipline, instead of re-inventing it? $\endgroup$ – Joshua Ulrich Oct 26 '14 at 3:06
  • 1
    $\begingroup$ Way too broad question. You emphasize customization requirements and that basically requires an incredible code stack that will be beyond any simple project. Just want you to be aware of the fact that you will most likely spend a year if you build from scratch. And I highly recommend not to attempt this project in Python but an OOP language that is much more performant, with a more stable and mature code base and that is actually suited to handle modules like broker API connectivity, OMS, PMS, parallel event based processing, and the like. Just my 2 cents... $\endgroup$ – Matthias Wolf Oct 27 '14 at 3:25
  • $\begingroup$ I disagree on your latter statement: Python is more mature than say, Haskell (Tsuru, StanChart), Erlang (GSET) or OCaml (Jane St), so certainly, maturity isn't an issue. $\endgroup$ – madilyn Oct 27 '14 at 6:25
  • $\begingroup$ I don't see why parallelization is an issue either. The GIL in Python leans towards a multi-process rather than a multithreading paradigm for parallelization, but there's no reason to prefer the latter over the former. $\endgroup$ – madilyn Oct 27 '14 at 6:27
  • $\begingroup$ I also don't see why you can't do event-based processing in Python. You can write a TCP/IP stack in vanilla CPython if you wanted to, surely that's fast/robust enough considering some traders are using subpar trading platforms that don't even support basic order types such as HNS? $\endgroup$ – madilyn Oct 27 '14 at 6:31

Firstly, you'll probably be directed to consider Zipline. It's worth a look but I don't think that it's a good starting point, since:

  1. Quantopian's developers don't have a financial background and it shows through in the Zipline source code.

  2. Zipline is dreadfully slow if you compare it to any commercial platform with backtesting functionality in a compiled application, even the low-end retail trading platforms (e.g. NinjaTrader, Sierra, TradeStation).

  3. Zipline isn't very convenient for trading multiple products. I think the cheapest product that has that level of functionality is Deltix.

A modern processor should be able to backtest a moving average crossover strategy across an entire day of the OPRA feed (all products) without scheduling it overnight. Any less functionality or slower and you have poor developers. (I remember Goldman had 12-14 servers dealing with realtime OPRA in 2007-2008 and 2 persons rewrote the entire thing from scratch to target 128-bit architecture over a weekend. No reason why years of development on Zipline doesn't match up to 2 developers on a weekend before Stack Exchange existed.)

Here are some of the major considerations that you have to make before building your backtesting engine:

  1. How will you be storing/serializing your market data on disk and in memory?

    • One poor man's approach is to wrap it around a pandas dataframe, but this comes at the cost of abstraction and will slow down your backtesting engine. pandas is nice for data exploration, but not for a task that you will repeat many times.

    • How will you handle a data source whose size exceeds available memory?

    • How will you deal with unstructured market data?

  2. How will you be storing your outputs?

    • An obvious, naive problem is that you don't want to restart a backtest that took you 1 night to run if the application crashed midway. Another naive example is that you should be able to access old results from 6 months ago without repeating the backtest loop.
  3. What's your fill logic?

  4. What should your API expose? (e.g. Market orders, limit orders, instrument/price/volume queries, changes to fill logic)

| improve this answer | |
  • $\begingroup$ I think pandas and HDF5 will be the good combination, But speed is still my biggest problem now. I am ready to do anything for make this super speed, have any idea? $\endgroup$ – Jake Oct 28 '14 at 19:47
  • $\begingroup$ In my mind one, the the worst lack of Zipline is the lack of bracket orders (Stop loss + Limit "child" orders attached to a parent order). $\endgroup$ – working4coins Jan 1 '15 at 17:38
  • $\begingroup$ +1 for "Quantopian's developers don't have a financial background and it shows through in the Zipline source code". $\endgroup$ – Homunculus Reticulli Jan 7 '17 at 23:35

There are some other opensource projects at github along with zipline which you can check for some additional inputs for your thoughts. Pyalgotrade is one such project where one can do back testing on their trading strategies


| improve this answer | |

Not the answer you're looking for? Browse other questions tagged or ask your own question.