17
$\begingroup$

I want to generate synthetic forex data for the purpose of backtesting my trading algorithms. I have some rough ideas in mind on how to do this:

Start with a curve representing a trend, then randomly generate points around the curve according to a Gaussian or some other distribution. Then take the generated points and somehow generate the bar data (open, high, low, close) around those points; alternatively, add a time factor, and then randomly determine when a tick occurs and collect the data into bars afterward.

My question is: is this far off from the established methods for synthetic data generation? I suppose that raises the more basic question: are there any established methods for synthetic data generation? I can't seem to find any writing on this subject, be it a blog post or a research paper.

So in addition to a request for external resources on synthetic data generation, I'd like to know what sorts of distributions best model the relationships between open, high, low, close, or how to generate the appropriate intervals between ticks, the spread between ask and bid prices, etc.

Improve this question
$\endgroup$
3
  • 1
    $\begingroup$ You could use a stochastic process with a drift and volatility term. Perhaps an interest rate model such as Vasicek that has mean reversion, ES-VJ++ which is Vasicek with jumps, Hull-White, Black-Derman-Toy, etc. etc. The problem of course is that your trading algorithms (should) quickly identify the Guassian (or otherwise) assumptions from the model and beat the "market". Why not try sim trading with a broker that will give you live data? I have not seen writings on synthetic trading data. $\endgroup$
    – strimp099
    Commented Aug 15, 2011 at 1:46
  • $\begingroup$ Beating a fake market is a problem, hence my interest in existing research. You're right, I should and do test on live data from a broker as well. The problem is that live data takes time, and I can't control it. In addition to live testing, I want to have a suite of unit tests to run my strategy against, to serve as a sanity check that even in the ideal cases my strategy makes profit. Ideally, I would check a large number of cases with varying parameters. I can't guarantee that all such movements show up in live data, and historical data is unclean in addition to having the same problem. $\endgroup$
    – JeremyKun
    Commented Aug 15, 2011 at 4:30
  • $\begingroup$ So yes, maybe I want the data to be so simplistic that any reasonable strategy should beat it, but even so I want to model a real market as closely as possible... $\endgroup$
    – JeremyKun
    Commented Aug 15, 2011 at 4:31

4 Answers 4

Reset to default
7
$\begingroup$

For starters, I am not even sure why you need to ask this question. There is literally years of free tick data available for FX, just check out quant.SE's data wiki.

Having said that, a Gaussian is a very poor fit to high-frequency data, particularly FX. Your strategy for simulating data depends on the idea behind the simulation. If you wish to actually estimate any parameters or test methods on this data, and you will be accepting or rejecting ideas based on the results of the simulation, I urge you to stop right there and reconsider.

If, instead, you wish to model the volatility of the process and test the ability of your system to deal with changes to the parameters of the data generating process, you should consult chapter 5 and particularly page 122 of An Introduction to High Frequency Finance. They write:

The distributions of returns are increasingly fat-tailed as data frequency increases (smaller interval sizes) and are hence distinctly unstable...

Scaling laws describe mean absolute returns and mean squared returns as functions of their time intervals...

There is evidence of seasonal heteroskedasticity in the form of distinct daily and weekly clusters of volatility...

Some papers claim FX returns to be close to Paretian stable ones, for instance (McFarland et al., 1982; Westerfield, 1997); some to Student distributions that are not stable (Rogalski and Vinso, 1978; Boothe and Glassman, 1987); some reject any single distribution... Most researchers now agree that a better description of the data generating process is in the form of a conditional heteroskedastic model rather than being from an unconditional distribution.

Finally, if you wish to construct OHLC data, your best bet is to model the data generating process as above along with a tick frequency process (also discussed in the above reference) and construct OHLC bars from simulated tick data.

Improve this answer
$\endgroup$
6
  • 1
    $\begingroup$ I certainly don't mean to offend anyone by wanting synthetic data in addition to historical data. I just want to be able to isolate trends with specific volatility parameters, and hence give me a wider range of data to work with. Of course, the synthetic data would have a smaller weight on my overall decision about a strategy; but if I make some small change that has inadvertent side effects on some cases that don't happen to show up in my historical data, it would be better to know about it. I'm sure that if real data were as easy as a Gaussian, this forum wouldn't exist. $\endgroup$
    – JeremyKun
    Commented Aug 16, 2011 at 1:44
  • $\begingroup$ And of course, thanks so much for your specific reference! I'll check it out and maybe add another comment with what I find. $\endgroup$
    – JeremyKun
    Commented Aug 16, 2011 at 1:47
  • $\begingroup$ So, I guess that raises the question: what "conditional" process controls the heteroskedasticity? Are there any known models that seem to work similarly to a particular subset of real-world data? $\endgroup$
    – JeremyKun
    Commented Aug 16, 2011 at 2:02
  • $\begingroup$ @bean no one is offended. I just think that for FX simulating data is usually not worth the effort. But, if you want to go ahead anyway, I suggest you match the level of sophistication of your simulation to the sophistication of your trading algorithm. $\endgroup$
    – Tal Fishman
    Commented Aug 16, 2011 at 10:11
  • $\begingroup$ Say I'm not interested in simulating HFT. Assuming we look at weekly trends, perhaps the data fits some kind of known distribution/process in an easier way? Or at least the relationships of OHLC pieces is more tractable? $\endgroup$
    – JeremyKun
    Commented Aug 17, 2011 at 22:05
1
$\begingroup$

The method for generating synthetic data described here might be useful to you. Also I believe the meboot R package can be used for synthetic time series generation.

Improve this answer
$\endgroup$
2
  • $\begingroup$ This is interesting, but it doesn't look like the author of the blog post provides a method to change the market parameters. He just takes real tick data and modifies it slightly... And for those reading here's a link to the paper about meboot. I'll have to look more closely to see what that's about. jstatsoft.org/v29/i05/paper $\endgroup$
    – JeremyKun
    Commented Aug 16, 2011 at 1:55
  • $\begingroup$ I agree that there are some limitations and I have posted my adapted version here $\endgroup$
    – babelproofreader
    Commented Aug 16, 2011 at 14:56
0
$\begingroup$

You could start with some representative data (i.e. historical data for the period of interest) and then use bootstrapping to estimate the true distribution of that data. From there, you can use that distribution to generate representative data.

Improve this answer
$\endgroup$
5
  • $\begingroup$ So what if I can't easily find the particular trend I want to model in historical data? $\endgroup$
    – JeremyKun
    Commented Aug 17, 2011 at 22:05
  • $\begingroup$ If you can't find any examples, then how do you know your trend is a realistic scenario? $\endgroup$
    – G__
    Commented Aug 17, 2011 at 22:26
  • $\begingroup$ That's not the point. Just because a scenario hasn't happened before doesn't mean it can't happen in the future. Wouldn't you want your strategy to work in such cases too? $\endgroup$
    – JeremyKun
    Commented Aug 20, 2011 at 23:20
  • $\begingroup$ Indeed, but then you get into the realm of just making up data. How do you know that your scenario is possible? You can certainly invent a scenario, but it's going to miss "realistic" nuance and you don't really know where it lies in the true probability distribution, other than being pretty far down one tail (if nothing similar has happened before in historical data) $\endgroup$
    – G__
    Commented Aug 21, 2011 at 3:14
  • $\begingroup$ Right. That's the point of my question. How can I generate synthetic streams of data that are realistic to a reasonable degree, but have potentially never happened with exactly a particular set of parameters before? What if I want to plan for a European financial crisis? I can't very well look into history for that, but it's happening now. $\endgroup$
    – JeremyKun
    Commented Aug 21, 2011 at 15:31
0
$\begingroup$

I've been looking at the same question. I believe the data augmentation literature is relevant. And no doubt some ideas from Good-Turing frequency estimation and descendants can be adapted. Another idea is to transform one exchange rate so that it roughly coincides with another, and use a barrage of off-the-shelf classification algorithms to see if they can determine fake from real data. I think it all depends on which invariants in the data you believe you know well and which residual behavior is common across different time series.

Improve this answer
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.