Given $G=\frac {C_n} {C_0}$, with $C_0$ the initial capital and $C_n$ the final capital after $n$ trades,
the Kelly criterion derives the optimal fraction of capital to invest in each trade, by maximising the expectation $\mathbb{E}[\log G]$.
Why did Kelly choose to maximise $\mathbb{E}[\log G]$ rather than simply maximising $\mathbb{E}[G]$? What is the rationale behind this choice?
Maximizing $E[\log(G)]$ which corresponds to a concave utility function is a subtle way of incorporating risk aversion in the utility.
Maximizing $E[G]$ is basically saying that you have linear utility which corresponds to infinite risk appetite because as soon as you have positive expectation you are willing to bet as much capital as possible no matter the variance of the resulting portfolio. In practice it can easily lead to ruin almost surely if the game is played long enough although the linear utility function is blind to that.
On the other hand, since $\log$ is concave, the second order term ensures that there will be a balance found between expectation and variance.
The short answer is that:
Maximizing expected return results in betting everything on your highest expected return investment. Repeatedly doing that over time typically leads to polarized outcomes: you become fantastically wealthy or (more likely) destitute.
Kelly's original problem was related to repeated betting on binary outcomes, but the expected log wealth objective of his optimization problem connects more generally to what's called a growth optimal portfolio. Let $\mathcal{S}$ denote the feasible set of portfolio returns (where each return $R \in \mathcal{S}$ is a random variable denoting a gross return like $\frac{X_{t+1}}{X_t}$). The idea is that by choosing different investments, we can obtain any return $R \in \mathcal{S}$ for our overall portfolio. A return $R^* \in \mathcal{S}$ is called a growth optimal portfolio return if it solves:
$$ \max_{R \in \mathcal{S}} \operatorname{E}[\log R] $$
Like many good ideas, the objective $\operatorname{E}[\log R]$ can be justified philosophically in multiple ways:
As @Dave Harris in his answer points out, it is the intuition of the latter, the attractive properties of maximizing the geometric mean, that initially motivated the expected log objective.
Note also the geometric mean return is the exponential of the (arithmetic) mean logarithmic return: $$\left(\prod_{i=1}^t R_i \right)^\frac{1}{t} = \exp\left( \frac{1}{t} \sum_{i=1}^t \log R_i \right)$$ When you maximize the log return or log wealth, you're maximizing the expected geometric growth rate.
Assume an IID setting where we repeatedly choose a stochastic return $R$ (from a set $\mathcal{S}$). Log wealth at time $t$ is given by the $\log W_t = \sum_{i=1}^t \log R_i$. Log wealth for the growth optimal portfolio is $\log W^*_t = \sum_{i=1}^t \log R^*_i$. We have $\operatorname{E}[\log R^*] \geq \operatorname{E}[\log R]$ by the optimality of $R^*$. Hence by the strong law of large numbers:
$$ \lim_{t \rightarrow \infty} \frac{1}{t} \left( \log W^*_t - \log W_t\right) \geq 0 \text{ almost surely}$$
Log is a monotonic transformation so if log wealth is almost surely higher, wealth is almost surely higher. Something to realize too is that if $\operatorname{E}[\log R] < 0$, you will go broke almost surely over the long run.
Perhaps unintuitively, maximizing expected return can easily lead to wealth heading towards $0$ in the repeated context instead of growing towards $\infty$.
For example, in the classic Kelly problem the problem is to choose what fraction $f \in [0, 1]$ of wealth to allocate to a risky, binary bet. Imagine you gain $bf$ with probability $p$ and lose $f$ with probability $1-p$. Maximizing expected return gives the problem:
$$\max_{f \in [0, 1]} p(1 + b f) + (1-p)(1-f)$$
This is equivalent to:
$$\max_{f \in [0, 1]} (p(b+1) - 1)f$$
Thus $f^* = 1$, you bet all your wealth, if the wager has positive expectation (i.e. $p(b+1) > 1$). In the limit as $t \rightarrow \infty$, this leads you to go broke almost surely! After $t$ bets, you have $1 - p^t$ probability of having 0 wealth. The expected log return is $- \infty$.
Maximizing the expectation of the logarithm gives the problem: $$\max_{f \in [0, 1]} p \log (1 + b f) + (1-p) \log (1-f)$$
Assuming the bet has positive expectation, the constraints $f \in [0, 1]$ don't bind and the solution is given by the first order condition:
$$ \frac{pb}{1 + bf} = \frac{1-p}{1 - f} $$
which simplifies to
$$ f^* = \frac{p(b+1) - 1}{b} $$
While the growth optimal portfolio has many attractive properties, at least two general points of caution are in order:
Models over the joint distribution of security returns are highly imprecise. You can get into extremely dangerous garbage in, garbage out problems when computing some notion of an optimal portfolio based upon highly uncertain inputs.
The original paper was concerned with optimizing the long run geometric return. In fact, he does not explicitly optimize either $\mathbb{E}(G)$ or $\mathbb{E}(\log(G))$. He also assumes the probabilities are known. The expectation is implicit in his assumption that $$G=\lim_{N\to\infty}\frac{1}{N}\log\left(\frac{V_N}{V_0}\right).$$
He notes that $$V_N=(1+\mathcal{l})^W(1-\mathcal{l})^LV_0.$$ The use of logs is from the obvious multiplicative relationship. Since $V_N$ is the only controllable variable and its structure is multiplicative, taking the logs simplifies the calculus with no loss of generality. This obviously reduces to $$G=\frac{W}{N}\log(1+\mathcal{l})+\frac{L}{N}\log(1-\mathcal{l})$$
The introduction of logarithms has only to do with the fact he is optimizing the long run geometric growth rate.
A quote from the original paper may give you an answer to your question as it has nothing to do with risk aversion, at least in Kelly's mind.
The gambler introduced here follows an essentially different criterion from the classical gambler. At every bet he maximizes the expected value of the logarithm of his capital. The reason has nothing to do with the value function which he attached to his money, but merely with the fact that it is the logarithm which is additive in repeated bets and to which the law of large number applies. Suppose the situation were different; for example, if the gambler's wife allowed him to bet one dollar each week but not to reinvest his earnings. He should then maximize his expectation (expected value of capital) on each bet. He would bet all his available capital (one dollar) on the event yielding the highest expectation.
This is indeed a very good question!
There were (and still are) very hefty debates, where even academic champions like Paul Samuelson were involved!
A very good starting point to get some main arguments is the following chapter 4 from the book "Fortunes Formula" by William Poundstone:
The gist is that "in the long run" using the Kelly criterion would ensure an optimal outcome growth-wise "on average". But as we all know "in the long run we are all dead" and "on average" doesn't mean that it is the best outcome for You. In general the Kelly criterion is criticized for taking too much risk (i.e. generating too much volatility) which is why methods like e.g. "Half Kelly" are being used.
In a way it is not only a mathematical debate but touches also on more philosophical and psychological points. As I said, it is a very interesting question which can lead you down all kinds of different paths.
For me it was one of the starting points of QuantFinance and I am still fascinated by it.
Kelly maximises the geometric return in each period. I believe this is equivalent to maximising expectation of log wealth in the next period.
If you maximise the arithmetic return, your expected wealth will be higher. However the actual return you will experience is -100% i.e. you will go bust with probability 1.
If you maximise the geometric return, then with probability 1 you will achieve a higher wealth than anyone following any other strategy.
These statements may seem contradictory, but they are not, for more debate see the St Petersburg Paradox.
Both statements are only true in the limit of an infinite series of bets. I have seen the credit for point 2 given to Leo Breimann's 1960 paper Investment policies for expanding businesses optimal in a long‐run sense, but as Matthew Gunn notes above, the proof is a trivial consequence of the law of large numbers.
In a nutshell, the final wealth is a monotonically increasing function $f^+$ of the average log return $$ W = W_0 (1+r_1)(1+r_2) ... (1+r_n) = W_0e^{\sum_n{ln(1 + r_i)}} = f^+\left(\sum\limits_n{ln(1 + r_i)}\right) $$
Since this is an arithmetic sum, the law of large numbers applies to it (therefore maximising $\mathbb{E}(ln(1+r_i))$ maximises the sum), and since the function is monotonically increasing, maximizing the sum maximises final wealth $W$. Note also that $r_i$ here are random variables.
I got interested in the Kelly Criterion (KC) as a method of optimizing position sizes for intraday trading, and consider one of my primary references as to what it is, how it works, and how to apply it to be "The Kelly Criterion in Blackjack, Sports Betting, and the Stock Market" by Edward Thorp, where he derives the basic formula for the case of an "even money" coin toss using a biased coin, i.e., one that favors landing on one side more than the other, and where the player either wins or loses the same amount of money that he bets. That basic formula can then be extended to "uneven money" cases where the payouts and losses are not necessarily equal to the amount bet (or each other), or to what Thorp calls the "Continuous Approximation", which adapts it to stock market price behavior over time.
The formula for the first case is f = p - q, where "f" is the fraction of the player's bankroll that is bet, and p and q are the probabilities of winning and losing, respectively. For the second case, it's f = p/a - q/b, where "b" and "a" are the fraction (or multiple) of "f" that the player wins or loses, respectively. (The first case can then just be treated as special case of the second case, where a = b = 1.) For the third case, it's f = m/(s^2), where "m" is actually "dm/dt" (Thorp calls it the "drift rate") and "s^2" is actually d(s^2)/dt (which Thorp calls the "variance rate). A lot of people appear to assume that "m" and "s^2" are simply the mean and variance, but they are not. They are the slopes of the mean and variance over time, not the mean and variance at one particular instant in time. Big difference, so we shouldn't be surprised at how many people speak negatively about the KC when the fact is, they probably never learned correctly it in the first place.
But getting back to your question, why log(G) and not just G? This seems to be another instance where people assign an aura of mystique to the matter, but from a practical point of view, it's simply about how to solve for "f*" in any of the three KC versions discussed.
From the point of view of a first-year calculus student, it's just about how to solve for "f*" (in case 1), the optimal fraction of the player's bankroll, given that: G(f) = (1+f)^p * (1-f)^q and p + q = 1 Since p and q are fractions of 1 and not integers, G(f) can't be solved for in the conventional way that you would if they were integers. To solve this, you have to take the logarithm of the function, preferably using natural logarithms to the base "e" because it's just cleaner that way. So this ends up being g(f) = ln[G(f)] = ln[(1+f)^p * (1-f)^q] = pln(1 + f) + qln(1 - f), or g(f) = pln(1 + f) + qln(1 - f)
So now that we have g(f) = ln[G(f)] we can solve for "f". Consider that the range of "f" is just from 0 to 1, because it represents the fraction of the player's bankroll to bet, all leverage aside. The function itself starts at (0,0) moves in the positive direction and reaches a peak somewhere between 0 and 1, then moves back in the negative direction and crosses the horizontal axis somewhere before it reaches 1, then proceeds to negative infinity, i.e., it has a vertical asymptote at 1. That means that it has only one peak, which occurs somewhere between 0 and 1, and that peak represents f*, the maximum value of f, or "maxima" in calculus terms. This is what the Kelly Criterion calculates - the value of f* where the expected value of ln(G), and consequently G, are at their maximum.
Since the plot of g(f) is simply that of the logarithm of G(f), both of these plots will peak at the same value of f. They will just be on different scales, but their peaks will occur in the same place. Thanks for reading this!
Really Short Answer:
Simple returns are not additives while log-returns are. By maximizing the log-returns, with guarantee you are maximize the return over many periods of time (because of the aggregation). By maximizing the simple return, you maximize only that outcome.
Example:
Let's say you can play either of two heads or tails game, how may times you want, but you must bet all your money each time:
The first game has a simple average return of 1.5x, while the second has 1.375x. By maximizing the simple return, you should pick game 1.
But notice, by playing many times, you only need to lose once to lose all your money. By playing more games, you are more likely to get a tail. In the limit, the probability to go broke is 100%, therefore the expected wealth is 0.
Looking at log returns:
The second game is expected to gain around 20% per game in the long run, while the first guarantee to lose all.
