Take the 2-minute tour ×
Quantitative Finance Stack Exchange is a question and answer site for finance professionals and academics. It's 100% free, no registration required.

There is a large body of literature on the "success" of the application of evolutionary algorithms in general, and the genetic algorithm in particular, to the financial markets.

However, I feel uncomfortable whenever reading this literature. Genetic algorithms can over-fit the existing data. With so many combinations, it is easy to come up with a few rules that work. It may not be robust and it doesn't have a consistent explanation of why this rule works and those rules don't beyond the mere (circular) argument that "it works because the testing shows it works".

What is the current consensus on the application of the genetic algorithm in finance?

share|improve this question

8 Answers 8

up vote 22 down vote accepted

I've worked at a hedge fund that allowed GA-derived strategies. For safety, it required that all models be submitted long before production to make sure that they still worked in the backtests. So there could be a delay of up to several months before a model would be allowed to run.

It's also helpful to separate the sample universe; use a random half of the possible stocks for GA analysis and the other half for confirmation backtests.

share|improve this answer
4  
Is that a different process than you would use before trusting any other trading strategy? (If so, it is not clear to me what you gain from making a GA model using data to time t, then testing until t+N before trusting it, versus using data to time t-N, testing from t-N to t, and using it immediately.) –  Darren Cook Nov 23 '11 at 2:08

The late Thomas Cover , (likely the leading "Information Theorist" of his generation), considered "Universal" approaches to things like data compression and portfolio allocations as true genetic algorithms.

Evolution has no parameters to fit or train. Why should true genetic algorithms?

Universal approaches make no assumptions about the underlying distribution of data. They make no attempt to predict the future from patterns or anything else.

The "theoretical" effectiveness of Universal approaches (they present significant implementation challenges see my recent question: Geometry for Universal Portfolios?) follow from them doing what evolution demands. The fastest, smartest, or strongest don't necessarily survive in the next generation. Evolution favors that gene, organism, meme, portfolio, or data compression algorithm positioned to most easily adapt to whatever happens next.

Also, because these approaches make make no assumptions and operate non-parametrically, one can consider all tests, even on all historical data, as out-of-sample.

Certainly they have limitations, Certainly they can't work for every kind a problem we face in our domain, but gee, what an interesting way to think about the things.

share|improve this answer

There's a lot of people here talking about how GAs are empirical, don't have theoretical foundations, are black-boxes, and the like. I beg to differ! There's a whole branch of economics devoted to looking at markets in terms of evolutionary metaphors: Evolutionary Economics!

I highly recommend the Dopfer book, The Evolutionary Foundations of Economics, as an intro. http://www.cambridge.org/gb/knowledge/isbn/item1158033?site_locale=en_GB

If your philosophical view is that the market is basically a giant casino, or game, then a GA is simply a black-box and doesn't have any theoretical foundation. However, if your philosophy is that the market is a survival-of-the-fittest ecology, then GA's have plenty of theoretical foundations, and it's perfectly reasonable to discuss things like corporate speciation, market ecologies, portfolio genomes, trading climates, and the like.

share|improve this answer
    
At short time scales it is more a casino. Like nature, in fact. –  quant_dev Apr 5 '11 at 20:46
    
@quant_dev, the problem with this is that GA-- like any other quantitative methods-- only works with short time scale, if I'm not mistaken. So if stock market is more like a stock market, then GA would be completely useless. –  Graviton Apr 6 '11 at 8:56
2  
@Graviton... There's no inherent reason why one couldn't program a GA to do analysis at longer time spans. The time domain of a GA is measured in generations, not years or days. So, one would simply need to define a population containing individuals whose generations are years or decades long (ie. corporations). There has definitely been some work that approaches defining corporate 'genomes' by their production processes. In such a model, one would optimize for an efficient corporate business model, given a particular market climate. It's not a stock price portfolio model, however. –  BioinformaticsGal Apr 6 '11 at 15:12

Well, the goal of a genetic algo is to find the best solution without going through all the possible scenarios because it would be too long. So of course it is curve fitting, that's the goal!!!

share|improve this answer
    
But there is a significant difference between overfitting the sample (bad) and fitting the population (good). That is why many suggest you cross-validate your algorithm with out-of-sample testing. –  Joshua Jul 17 '13 at 2:34

I've applied GA to all sorts of things. I had some success in the deterministic world where a pattern actually existed and I knew that some physical structure existed (seismic analysis, vibration analysis, inventory calcs, etc). After I found a GA model that behaved, the real work started....figuring out why it behaved.

I also generated a lot of GA garbage from financial data that "worked" looking backward, but was worthless looking forward.

Techniques aren't the issue in finance, it's the structure. And, of course, never enough data (useful data).

share|improve this answer

Assuming you avoid data-snooping bias and all the potential pitfalls of using the past to predict the future, trusting genetic algorithms to find the "right" solution pretty much boils down to the same bet you make when you actively manage a portfolio, whether quantitatively or discretionary. If you believe in market efficiency then increasing your transaction costs from active management is illogical. If, however you believe there are structural & psychological patterns or "flaws" to be exploited and the payoff is worth the time and money for researching and implementing a strategy the logical choice is active management.

Running a GA derived strategy is an implicit bet against market efficiency. You're basically saying "I think there are mis-valuations that occur from some reason" (masses of irrational people, mutual funds herding because of mis-aligned incentives, etc.) and "running this GA can sort this mass of data out way quicker than I can."

share|improve this answer
    
manually managing an active portfolio involves using all the information we have and derive a logical conclusion about the market and then execute strategies on it; this is a rational activity. OTOH, using GA is using a black-box tool; we can't explain the result derived from it from any accepted principles. I'm not too sure whether these two are really the same. –  Graviton Feb 18 '11 at 15:57
4  
@Graviton Yes but consider the similarities between GA's and how us humans learn about markets, develop strategies, learn from mistakes, and adapt to changing market conditions. When you research what winning and losing stocks have in common, or what volume and price patterns create good trades, or which model is the most accurate for valuing derivatives what you are doing is data-mining the past in a way. When market conditions change you either trade new strategies or eventually go out of business. If there are exploitable edges in the market then the only difference between you and a GA is –  Joshua Chance Feb 18 '11 at 16:46
    
@Graviton (in a very broad sense) is that you have a narrative, a story to go with your strategy. Us humans risk finding a seemingly recurring pattern and then rationalizing it and creating a narrative. GA's risk the same thing, only their potentially false model doesn't use words, they use math and logic. –  Joshua Chance Feb 18 '11 at 16:54
2  
@Graviton.... as posted below, there are plenty of accepted principles for accepting the results of a GA, if one is using a framework of evolutionary economics. If one is just using monte-carlo style models, then sure, a GA is a black-box, at best. But if your economic perspective is grounded in evolutionary science, the results can be readily interpreted according to principles of ecology and population genetics (i.e. corporate speciation and market ecologies). –  BioinformaticsGal Apr 5 '11 at 15:53

I think the biggest problem that genetic algorithms have are overfitting, data snooping bias and that they are black boxes (not so much like Neural Networks but still - it depends on the way they are implemented).

I think they are not used very much. I guess there are a few hedge funds out there that use it but all in all they were hyped and then busted. (But they are still useful for getting a paper accepted ;-)

BTW: There is never a real consensus in finance - everybody tries to outsmart everybody else. This is why it is so interesting. (Or put another way: this is why there are still buyers AND sellers - a real consensus is a crash ;-)

share|improve this answer
2  
Regarding data-snooping, if a GA is implemented correctly, that shouldn't be a concern. Mutation functions are specifically included to randomly search through the problem space, and avoid data snooping. That being said, finding the right mutation levels can be something of an art; and if the mutation levels are too low, then it's as if the function wasn't implemented in the first place. –  BioinformaticsGal Apr 6 '11 at 16:28
4  
@BoinformaticsGal I don't understand how the inclusion of mutation functions allows us to avoid data snooping. After the search, there's a fitness function which makes each generation 'fit' the data ever more. Or am I not understanding you correctly? –  Vishal Belsare Sep 15 '11 at 18:05

if you backtest properly your GA (using only past data to generate the time serie of indicator), then you can trust the result.

But I agree with you that genetic algorithms are purely empirical and thus I don't feel very comfortable using them.

share|improve this answer
    
but backtesting is also problematic for GA because the rules that GA selects are the rules that work very well in backtesting. –  Graviton Feb 18 '11 at 9:30
3  
@Graviton: It is also what we (humans) do when we backtest –  Zarbouzou Feb 18 '11 at 10:04
    
you select your strategy using past data, then you apply it to future data! That way you are absolutely not sure that an optimised GA will perform! It will work only if there is persistence –  RockScience Feb 18 '11 at 16:29
2  
That could be said about so many things in finance and prediction in general –  Steve Feb 20 '11 at 0:57
    
@Graviton, one would assume that you reserved some out of sample data from your training set. Backtesting the trained model should occur on this out of sample data, not on the data one used to generate the GA. –  Louis Marascio Sep 8 '11 at 12:38

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.