45

Because of: The (extreme) dominance of noise over signal The prevalence of non-repeating patterns (many of which we know are not going to repeat) A pathetic sample size for cross-validation Regime changes due to exogenous events. These are typically in the cross-val window which makes it even worse. (GFC, financial integration, trade law changes, interest ...


24

Two aspects of statistical learning are useful for trading 1. First the ones mentioned earlier: some statistical methods focused on working on live datasets. It means that you know you are observing only a sample of data and you want to extrapolate. You thus have to deal with in sample and out of sample issues, overfitting and so on... From this viewpoint, ...


24

First, let's speak about perceptrons in general: their input $X_0$ is a $K$-dimensional vector. So if you want to use $(P_{bid}(t),P_{ask}(t), Q_{bid}(t),Q_{ask}(t))$, it would mean that without any effort (but later we will see that is would be better to do some efforts, as usual): $$X_0(t)=(P_{bid}(t),P_{ask}(t), Q_{bid}(t),Q_{ask}(t))'\in\mathbb{R}^4$$ ...


15

Recently I attended a presentation by the first author of the following paper who gave us quite a creative and illuminating (kind of meta-)use of random forests in Quant Finance: All that Glitters Is Not Gold: Comparing Backtest and Out-of-Sample Performance on a Large Cohort of Trading Algorithms (March 2016) by Thomas Wiecki, Andrew Campbell, Justin Lent, ...


13

I would say that most ML methods risk overfitting and it depends very much on the asset class. The only area where more sophisticated ML methods such as deep learning appear to make a major difference is in cash equities, where the feature space is very rich (NLP, news and announcements, corporate earnings, other financials) and the data is relatively good, ...


11

Here's my favorite example of an intraday strategy on S&P500 futures that at least used to work: Intraday Share Price Volatility and Leveraged ETF Rebalancing I pull it out whenever people start talking about market efficiency. The strategy is very simple: if S&P500 futures are up or down more than 2% on the day with two hours left until close, ...


10

First, we are few quants and academics to use the full toolkit of machine learning: stochastic algorithms, to optimal trading. Here are at least two papers: Optimal split of orders across liquidity pools: a stochastic algorithm approach, Sophie Laruelle (PMA), Charles-Albert Lehalle, Gilles Pag├Ęs (PMA) Optimal posting distance of limit orders: a stochastic ...


9

I was just like you when I started out: I had learned a lot about machine learning (mainly neural networks and genetic algorithms/programming) and used it heavily. I also had learned about classic statistics but not nearly as much as about ML. The problem with ML is - as I see it today - that you are often taking a sledgehammer to crack a nut, meaning: ...


8

One excellent resource is to try Kaggle and to examine some of the competitions, some of which are specifically on the application of machine learning to credit scoring. https://www.kaggle.com/c/GiveMeSomeCredit You wil see that the winning solution is made public, including source code and output. https://github.com/IdoZehori/Credit-Score/blob/master/...


7

do input attributes need to be scaled? No. It is not required. It is only a heuristic [1]. It is primarily motivated because of the following: From the Feature Scaling article: Since the range of values of raw data varies widely, in some machine learning algorithms, objective functions will not work properly without normalization. For example, the ...


7

People seem to think that using ML is going to circumvent the process of actually learning to trade, it doesn't. ML can be used to refine trading ideas, but it doesn't generate them, you need to use your brain for that.


7

One potential use I could imagine would be identifying paradigm shifts / regime change. Just as a quick toy example, maybe you're interested in how gold is often considered a hedge against downturns in the stock market. Say you are building a trading strategy based on that intuition, but want your model to be more flexible by identifying different regimes ...


7

There is at least one clear area of application of ML in Q quant finance, it is the LSM algorithm invented by Longstaff, Schwartz and Carriere in the late 1990s for the valuation of callable exotics in the context of Monte-Carlo simulations, and widely adopted for more recent bank-wide risk calculations like CVA. In order to estimate the continuation value ...


6

As with many machine learning technologies, you can run a separate training and testing phase before deploying it live for prediction. All it does is build a collection of decision trees based on the parameters you give it - if the output field is a factor, you get classification (a finite enumerated set of values); if it's numeric, you get prediction. One ...


6

Such a complex question... Geometric Brownian Motion (GBM) will not typically work to aid one finding strategies based on technicals, as the pursuit of the technical trader is to find market deviations from a random walk. However, some strategies, for example a "take profit/stop loss" strategy can work, (or at a minimum one can change the risk/reward ...


6

'Machine learning' describes a very broad spectrum of algorithms. Just briefly here are a few conceptual areas; Neural networks Reinforcement learning Genetic algorithms and genetic programming Particle swarm optimisation (PSO) Regression models Optimisation routines Markov models Wavelet transforms and Fourier Transform and Spectral Analysis. Clustering ...


6

The code below is written in Wolfram Mathematica. For example, we have some training data. And we are trying to predict: long (1) or short (0). SeedRandom[0]; n = 10000; X = RandomReal[{-1, 1}, {n, 100, 5}]; Y = RandomInteger[{0, 1}, n]; net = NetChain[ { LongShortTermMemoryLayer[64], SequenceLastLayer[], ElementwiseLayer[Ramp], LinearLayer[...


5

I have not used random forests myself but I know of a guy who applied this classification technique to machine learning algorithms applied to pattern recognition. Thus I think its advantages over classic regression approaches can be applied to discern patterns in financial data, though I get the impression that it vastly overfits the data and thus you end ...


5

Sorry, but despite being used as a popular example in machine learning, no one has ever achieved a stock market prediction. It does not work for several reasons (check random walk by Fama and quite a bit of others, rational decision making fallacy, wrong assumptions ...), but the most compelling one is that if it would work, someone would be able to become ...


5

One possibility worth exploring is to use the support vector machine learning tool on the Metatrader 5 platform. Firstly, if you're not familiar with it, Metatrader 5 is a platform developed for users to implement algorithmic trading in forex and CFD markets (I'm not sure if the platform can be extended to stocks and other markets). It is typically used for ...


5

Speech recognition signal processing is complex and possibly similar to the complexity of financial markets. They are similar as per characterictics the non stationarity, noise types and other aspects such us the existence of a cepstrum etc conceptual frequency and the grammar to construct and articulate concepts is not evenly and randomly distributed; so ...


5

Machine learning could be integrated into anyone of these strategies. Order flow prediction strategies would be the "easiest" of these examples, specifically for integrating neural networks and machine learning. The most widely used method of AI in this field is regression, here are some examples of it in the high frequency field with LOB prediction. ...


5

"I need to get an algo or a formula to determine to right quantity to trade each time I place the pair (limit_buy_order, limit_sell_order)." Actually, you need a formula for determination of the optimal prices, not quantities. For example, if the market goes down and you have long positions in inventory, you should reduce ask price to attract more buy ...


4

These 2 sites are relevant: - The Whole Street (research aggregation) - Oxford Capital Strategies (strategy reviews)


4

I was going to comment but it turned out to be quite elaborate. My experience with certain AI/ML methods is that they're not deterministic. Take RBM for instance, a very wide-spread paradigm. To train such a machine you have two approaches, backpropagation or Kullback-Leibler divergence. Both require you to initialise the machine to a random state. And ...


4

From what I have read, there are 3 popular algorithms for financial time series. Random Forests and SVMs, then followed by Neural Network Architectures. There are a couple of good papers, to name a few: Empirical Asset Pricing via Machine Learning Deep neural networks, gradient-boosted trees, random forests: Statistical arbitrage on the S&P 500 An ...


4

R. The others in-play are Python and, increasingly, Scala. But if you're trying to create or test a machine learning algorithm for a new problem, it's R. Update July 2, 2017: This answer came up in my feed because of an upvote, so I suppose its worth updating. These days there are a few key deciding factors in what language I choose for a problem. It ...


4

I have been through your confusion myself for the last five years. Until recently, my account started to get some consistent performance. First, I started with Technicals, Spent $$$ on a automated trading platform. From there I created common strategies. The results is not promising. The strategy doesn't consists parameters and if one strategy works on one ...


Only top voted, non community-wiki answers of a minimum length are eligible