1
$\begingroup$

To determine if a strategy is better than others, or to optimize the parameters of a model, the following statistical techniques are often employed, often one over the others instead of altogether. Their results are important in terms of training a model or strategy and ensuring it will retain predicted performance when applied to unseen test data, but what are all the differences between their procedures as well as weaknesses, applicability and strengths, given that many seem to do their own rendition of data resampling? To start, a brief description of their procedure might help for comparison.

  1. Backtesting
  2. Historical simulation
  3. Monte Carlo simulation
  4. Bootstrap replication
  5. Cross-validation
Improve this question
$\endgroup$

1 Answer 1

Reset to default
4
$\begingroup$

As with many things, particularly in machine learning and AI, I think you will find that these processes do not have a unique, logically or mathematically defined description. More so I would say that depending upon the context they can mean different things and might even mean the same thing. However, in my experience this is their most common usage.

Backtesting

In machine learning I have come across 'backtesting' in the domain of finance when a model or strategy has been defined and its purpose is to genuinely test the profitability of the actions on some real historical data. Therefore backtesting requires time series and real data and aims to test a model. If the model has been designed with the test data in mind, or has snooped then the test will not be reflective of real performance.

Historical Simulation

This is similar to the above except I would say it falls in a more general context. Not just necessarily when you have a profitability strategy to examine but possibly just to examine risk, or other factors, as well. For example a historical simulation of Value at Risk of a portfolio. Historical simulation requires time series and real data.

Monte Carlo

Monte carlo is the same as the above but rather than requiring real data it uses simulated data. How the simulator is defined will determine the success of the analysis, e.g. parametrically, non-parametrically or another random process. Monte carlo is used for a multitude of tasks not necessarily in finance or time series related.

Bootstrap Replication

Bootstrap replication is, I would describe, between the two above models. Bootstrap samples create a statistical sampling methodology where the underlying real data is used to some degree. Either it is repetitively sampled (non-parameterically) or a parametric model might be created from it which generates samples from a probability distribution. Although this might be quite similar to Monte Carlo, I think by definition this will be more closely related to the underlying historical data.

Cross Validation

When training a machine learning model there are often two types of parameters to determine: basic parameters and hyper parameters. Basic parameters are the underlying values needed for the model, for example a linear regression model needs coefficients. These are trained from the training data. However, when you train a model on the data itself it is very biased, and might be able to reproduce that data exactly. But when tested against new and unseen data it might perform very poorly. Therefore you often need hyper parameters to be trained to analyse how effective a 'trained' model can be on unseen data. Hyper parameters might be things how many nodes to use in a neural network, or how many clusters to use in k-means clustering. You might even consider it a hyper parameter to decide whether to use SVMs or Logistic Regression or a Decision Tree, for example. Cross validation often uses clever mixes of the data you have to determine a good set of basic parameters and hyper parameters. You can then test the final model on test data.

Improve this answer
$\endgroup$
3
  • $\begingroup$ i would not know how to implement them based on the descriptions provided solely on their usage of either real or artificial data. how do they differ in how they resample the data over and over again for multiple runs, slicing them differently each run? For traditional comparison, let's assume traditional K-fold cross-validation (instead of walk-forward cv), and how pairs bootstrap (instead of residuals bootstrap) typically is good for 250 replications. Compared to these, how do backtesting, historical simulation and Monte Carlo simulation resample the same dataset per run? $\endgroup$
    – develarist
    Commented Dec 2, 2019 at 1:02
  • $\begingroup$ "you often need hyper parameters to be trained to analyse" - the rest of the paragraph makes sense, but I think you just phrased this part incorrectly. You don't train hyper parameters to analyse anything, but rather to avoid overfitting. But you also need a way to evaluate the models generated when tweaking the hyper parameters, which is where cross validation comes in. $\endgroup$
    – NotThatGuy
    Commented Dec 2, 2019 at 1:51
  • $\begingroup$ @NotThatGuy yes fair enough using the word 'training' is not pedagogical since the two processes are different and training is usually reserved for the former. But the overall process of setting basic and hyper parameters to infer the most accurate model given the data you have is all part of 'buliding' or 'modelling' or 'training'. But if I were to re-write I would phrase differently to highlight more the distinction. $\endgroup$
    – Attack68 ♦
    Commented Dec 2, 2019 at 7:09

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.