7
$\begingroup$

Is there software (or Python / R / ... scripts) to generate (pseudo) tick data from candlestick data.

I have candlestick data (CSV format) from monthly timeframe (MN) to minute timeframe (M1) but time range can be different. Filenames are :

  • SYMBOL1.csv for M1 timeframe (1 minute)
  • SYMBOL5.csv for M5 timeframe (5 minutes)
  • SYMBOL15.csv for M15 timeframe (15 minutes)
  • SYMBOL30.csv for M30 timeframe (30 minutes)
  • SYMBOL60.csv for H1 timeframe (1 hour)
  • SYMBOL240.csv for H4 timeframe (4 hours)
  • SYMBOL1440.csv for D1 timeframe (1 day)
  • SYMBOL10080.csv for W1 timeframe (1 week)
  • SYMBOL43200.csv for MN timeframe (1 month)

I would like to feed software or script with every csv file. I would like to give start datetime and end datetime and software will output a csv file with ticks data.

Ticks data will be generated from the longest timeframe to the shortest timeframe (if data exists). For example the software will output ticks data from M1 candlestick timeframe if data exists for the requested time interval. If there is no data in M1 timeframe, software will try to find data in M5 timeframe to generate ticks.

I understand that the generated ticks will be generated using interpolation (so they won't be exacts)

Such ideas are implemented in Metatrader Strategy Tester http://www.metatrader5.com/en/terminal/help/tester/tester_using/tick_generation

$ head _FRA401.csv 
2010.11.16,08:01,3818.0,3820.0,3817.5,3820.0,41
2010.11.16,08:02,3820.0,3823.0,3801.0,3823.0,38
2010.11.16,08:03,3823.0,3825.0,3823.0,3823.5,28

$ tail _FRA401.csv 
2012.11.01,18:32,3477.0,3477.0,3474.0,3474.5,37
2012.11.01,18:33,3474.5,3476.0,3474.5,3475.5,25
2012.11.01,18:34,3475.5,3476.0,3471.5,3472.5,62

$ head _FRA4043200.csv 
2010.11.01,00:00,3818.0,3906.0,3589.5,3620.0,168480
2010.12.01,00:00,3629.0,3940.5,3618.5,3853.5,227760
2011.01.01,00:00,3848.0,4081.5,3605.5,4019.5,266725

$ tail _FRA4043200.csv 
2012.02.01,00:00,3310.5,3490.5,3305.5,3456.5,514738
2012.03.01,00:00,3441.0,3593.0,3343.0,3420.5,353738
2012.04.01,00:00,3428.0,3475.5,2974.5,3128.5,247351
Improve this question
$\endgroup$

3 Answers 3

Reset to default
2
$\begingroup$

We've done this before (*), but it is trivial: one tick for open, one tick for high, one tick for low, one tick for close. For tick timestamps, open is 08:01:00, close is 08:01:59.999, high is 08:01:20 and low is 08:01:40. (Or if that gives strange results, open is 08:01:00.001 and close is 08:02:00.000; it depends how your system makes bars.)

If you want volume to match too, then create N-4 ticks at the mean of open and close, and space out the ticks evenly throughout the minute. You could interpolate a straight line from open to close if you want; you could also set all N-4 ticks equal to high, etc. All choices are fabrications, so it doesn't matter which you choose. I like to fix at the mean, as anyone looking at it will instantly know it is artificial data.

*: This was with a system where we could only patch outages and backfill historical data with tick data; that system would then make the intraday bars. I.e. we couldn't patch intraday bars directly.

If you want it to look like real ticks, because your trading algorithm is only designed to work from the ticks, then you start with straight line interpolation from open to close, then jitter them. Then filter to make sure none exceed high and low, and then choose two ticks to be the high and low and alter their values.

Another variation, specific to R, is to put in the open/close ticks, and use NA for all the others. Then choose 2 ticks randomly to be high and low. Then use R's na.approx or na.spline function. Then jitter. Then filter for high and low again.

If doing this, I strongly suggest you make multiple data sets, each with a different random seed, and run your backtest on each data set. If different data sets get different results, your algorithms are sensitive to the random noise you've introduced. (I.e. your results are meaningless and you need to go and buy real tick data!)

Improve this answer
$\endgroup$
13
  • 1
    $\begingroup$ @Freddy I was just answering the question that was asked. There are systems that are built around ticks; if you want to use those systems for a lower frequency strategy, and you only have bar data, then of course you have to convert the bars to ticks. As mentioned in the first half of my answer, when I've done this I've done it in such a way that makes it obvious to anyone looking at it that the ticks came from bars. $\endgroup$
    – Darren Cook
    Nov 7, 2012 at 3:58
  • 2
    $\begingroup$ see this is what I strongly disagree with your answer: You CAN'T convert bars to ticks, simple as that. It makes zero sense to create ticks without knowing anything about them other than at which price the market traded at the start and end time of the bar and the highs and lows. Anything else is a wild guess and unless you are in the business of gambling you SHOULD NOT use semi-random pricing data in your strategy testing endeavor. $\endgroup$
    – Matt Wolf
    Nov 7, 2012 at 5:06
  • 2
    $\begingroup$ ...and risking to repeat myself, if your finest-grained pricing data are compressed bar-data but your strategy testing engine can only handle ticks then either you or your testing engine is in the wrong business. Sounds maybe harsh but I try to be honest, I work in this business for close to 14 years now. Nobody I ever came across with proven (positive) track record has ever perused data points to test ideas that he/she has no access to and in that regards are non-existent. Its as if I want to trade kospi volatility... $\endgroup$
    – Matt Wolf
    Nov 7, 2012 at 5:13
  • 1
    $\begingroup$ @Freddy My answer shows the mechanics to answer the OP's question; that does not deserve a downvote. (Unless you are disagreeing with part of those mechanics, rather than just when you should apply them.) $\endgroup$
    – Darren Cook
    Nov 7, 2012 at 5:49
  • 1
    $\begingroup$ @Freddy Just to be clear, I personally always use real tick data when I want to backtest a tick stategy, and use bar data when I want to backtest a strategy that uses bars. I think we're in full agreement that it would be foolish to do otherwise. $\endgroup$
    – Darren Cook
    Nov 7, 2012 at 5:49
4
$\begingroup$

You said:"I understand that the generated ticks will be generated using interpolation (so they won't be exacts)".

You are very optimistic, they will not only be far away from being exact, they (the tick data) will be completely removed from reality, the only parameters known for the tick data will be boundary conditions, such as open high low close. You make a completely wild guess about how many ticks each bar contains and where the market traded/quoted at which point in time within the limits of the candle open and close times and candle high and low. Of course you can do that and Python is most likely the best tool to get the job done quickly. (I recommend http://pandas.pydata.org/)

However, I highly question the usefulness of this approach. You obviously want to use the tick data for some sort of analysis or back test. However, the results will be completely random if you attempt to peruse such tick data for higher frequency analysis/back tests. If you are not interested in such high frequency testing then you should just stick to your (assumed) accurate compressed time series (1minute bars or what have you).

In summary I question the usefulness of generating higher frequency time series from lower frequency time series in general. The opposite obviously makes a lot more sense because you know exactly how 1-hour compressed data points look like from generating those from higher frequency time series such as 1-minute compressed data points.

Just my 2 cents, feel free to elaborate what you want to use the generated tick data for, maybe I miss something here.

Improve this answer
$\endgroup$
15
  • $\begingroup$ In fact I would like to do this because Metatrader expert advisors have init function and start function. Start function is executed every incoming tick. I would like to make my own backtest tool to test strategy (probably using Python). I want my Python code also have init and start function.So I need to generate "pseudo tick data"... this is exactly the way Metatrader strategy tester is working. Of course you can't use this kind of "pseudo tick data" for M1 (or shortest timeframe)... but for a M15 strategy (with interpolated M1 data), it should be not very far from reality. $\endgroup$
    – Femto Trader
    Nov 3, 2012 at 8:03
  • $\begingroup$ I think either the testing platform you mentioned is insufficient (in that the Start function is only called on each incoming tick instead of also on the first data point of compressed data) or you misunderstand how the testing platform really works. I am confused about your understanding of compressed data and their usage, possibly you want to go back and read about the basics of tick data and compressed data time series analysis. I am afraid you are missing some core idea which... $\endgroup$
    – Matt Wolf
    Nov 3, 2012 at 10:17
  • $\begingroup$ ...is that you CANNOT generate higher frequency time series data from lower frequency time series, at least I do not see any usage of pseudo-randomly generated tick data from compressed data time series. I strongly hint at you missing some very basic concepts here. $\endgroup$
    – Matt Wolf
    Nov 3, 2012 at 10:18
  • $\begingroup$ If I'm wrong I'm not alone ;-) stackoverflow.com/questions/9336444/… moreover I wonder if you really had a look at metatrader5.com/en/terminal/help/tester/tester_using/… $\endgroup$
    – Femto Trader
    Nov 3, 2012 at 12:37
  • $\begingroup$ @FemtoTrader, well you may not be alone, still I claim it makes zero sense to infer higher frequency data from lower frequency data because the data points in between are unknown. You make a random guess which benefits you nothing. I never claimed I had a look at Metatrader, its a platform that ripped off retail clients with a market maker auto quoting engine for which the company should have been sued and taken out of business a long time ago. Further more, if memory serves me right then ... $\endgroup$
    – Matt Wolf
    Nov 3, 2012 at 13:54
1
$\begingroup$

I do not know of anything that is already made and works out of the box, but I would recommend writing a small script in Python using csv.reader, this would probably be as fast as trying to plug something on your csv, especially if your format is a bit exotic ?

Improve this answer
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.