26
$\begingroup$

A potential issue with automated trading systems, that are based on Machine Learning (ML) and/or Artificial Intelligence (AI), is the difficulty of assessing the risk of a trade. An ML/AI algorithm may analyze thousands of parameters in order to come up with a trading decision and applying standard risk management practices might interfere with the algorithms by overriding the algorithm's decision.

What are some basic methodologies for applying risk management to ML/AI-based automated trading systems without hampering the decision of the underlying algorithm(s)?

Update:
An example system would be: Genetic Programming algorithm that produces trading agents. The most profitable agent in the population is used to produce a short/long signal (usually without a confidence interval).

Improve this question
$\endgroup$
  • $\begingroup$ As it stands, I think this question cannot be answered. Please give more detail about the kind of "ML/AI-based trading system" that you're envisioning. Does it just say whether to go long/short, does it give a confidence interval, etc.? What kind of model? Is it really black-box (because many ML models can be interpreted)? $\endgroup$ – Shane Feb 1 '11 at 0:22
  • 1
    $\begingroup$ @Shane, I've updated the question... I think that a Genetic Programming model would be black-box, since the resulting "trading agents" are often difficult to understand (i.e. they contain "junk DNA" which usually occurs with evolution). $\endgroup$ – Kiril Feb 1 '11 at 0:49
12
$\begingroup$

The risks involved in trading is everywhere and always a multifaceted thing: it includes the volatility of the selected asset, the leverage and concentration of the porfolio, whether there is a stop loss, a hedge, etc. Also, risk management is frequently not tied to the "alpha model" directly (e.g. VaR, shortfall, and scenario testing).

For instance, one well known way of sizing a position is the Kelly formula:

$f^{*} = \frac{bp - q}{b}$

This makes no assumptions about the directional model that is used to enter the position. You can infer the values (e.g. probability of winning) from a historical simulation, regardless of whether the model is black-box, grey-box, or white-box.

Improve this answer
$\endgroup$
  • 2
    $\begingroup$ This form of the Kelly criterion seems only appropriate for binary outcomes. $\endgroup$ – shabbychef Feb 1 '11 at 3:32
19
$\begingroup$

ML/AI systems are susceptible to a number of risks not traditionally discussed in risk management:

  1. What I call 'backtest arbitrage'. In the process of automated model generation and testing, your machine learner may discover, exploit, and concentrate on irregularities in your backtesting system which do not exist in the real world. If, for example, your fill simulation is erroneous, you have not accounted for borrow costs, forgot to deal with dividends properly, etc., sufficiently powerful search techniques will find strategies which capture these nonexistent 'arbs'.
  2. If you sequentially generate, test, and refine many trading models, you run into the problem of 'datamining bias'. Here one has used the same data to simultaneously select the best model and estimate its performance via backtest. The estimate will be positively biased, and the size of the bias can be difficult to estimate if one has not kept careful track of all the strategies tested.
  3. Blackbox models are often subject to non-stationarities of the 'Grue and Bleen variety'. That is they may behave radically different out of sample due to non-stationarities of their input data and discontinuities in their processing of input data. An example would be an AI strategy which first checks if VIX is above 60, then trades one substrategy, otherwise it trades a different one. Your backtest period may contain little data in the 'over 60' regime, and you may find yourself in such a regime.

Regrettably many of these issues exist at the human level, and there is little one can do statistically to detect them or correct for them. They require great attention to process.

Improve this answer
$\endgroup$
  • $\begingroup$ @shabbychef, I was actually asking for risk management of the trades generated by the ML/AI systems, not the risks involved with developing such systems. $\endgroup$ – Kiril Feb 1 '11 at 6:10
  • 1
    $\begingroup$ @Lirik these are very serious risks. Beyond that, I am not sure one will have much success e.g. trying to reverse-engineer a black box ML/AI system in order to detect when it has gone haywire. If you are just receiving the trades out of the thing, I am afraid there is not much one can do beyond checking concentration limits and leverage constraints. $\endgroup$ – shabbychef Feb 1 '11 at 23:12
  • $\begingroup$ @shabbychef I think online machine learning can mitigate those risks for the most part. $\endgroup$ – Kiril Feb 2 '11 at 0:47
  • 1
    $\begingroup$ @shabbychef, online machine learning eliminates the use of back-testing. The point of online machine learning is that you never stop learning: your machine learner constantly generates viable candidates (i.e. strategies) as you continue to feed it the latest market data. This also nearly eliminates datamining bias as you don't have a fixed data set on which you can overfit your ML, the data set is constantly changing. $\endgroup$ – Kiril Feb 2 '11 at 5:34
  • 3
    $\begingroup$ @Lirik: no, I am thinking about what happens after 6 months of paper trading with mediocre results. Does one give up on finance and become a plumber? Or does one fiddle around with the algorithm, the data, etc? You always have only the data you have today, when you are deciding what to trade tomorrow. If you have any choice and the historical data guides that choice, you have datamining bias. $\endgroup$ – shabbychef Feb 3 '11 at 5:23
3
$\begingroup$

It depends on what the strategy does.

For a long/short signal on an equity symbol, one way is to look at the options prices / implied volatility for that symbol. Your system should give an expected timeframe and profitability, so the risk involved could be quantified by the price of buying options to insure yourself against losses compared to your expected returns.

For a more complicated symbol, you can attempt to approximate using a basket of options.

For a short-term microstructure trade (which is IMHO the arena in which ML/AI-type strategies are most useful, mainly because proper quant analysis often has little to say), there is very little you can in terms of principled risk estimates. Rather, you must rely on simulation, and backtesting. I especially recommend adding in simulations for totally disastrous fictional scenarios. For this kind of trade, any estimation is rough at best, so use a healthy dose of pessimism and superstition.

Improve this answer
$\endgroup$
1
$\begingroup$

The risk is not linked to the decision process but to your inventory, independently from the signals that triggered the buys and sells: you can monitor the inventory as usual.

If you are talking of taking into account the fact that you change your inventory more often because you use computer-based signals, it is more complex. You need to control the dynamics of your trading algorithm to be sure that it will not take positions in one millisecond that will dramatically increase your risk.

Improve this answer
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.