The short answer is that:
- Maximizing the expected logarithm leads to more wealth almost surely in the long run.
- In contrast, maximizing expected return can easily lead to going broke almost surely in the long run!
Maximizing expected return results in betting everything on your highest expected return investment. Repeatedly doing that over time typically leads to polarized outcomes: you become fantastically wealthy or (more likely) destitute.
Discussion of growth optimal portfolios
Kelly's original problem was related to repeated betting on binary outcomes, but the expected log wealth objective of his optimization problem connects more generally to what's called a growth optimal portfolio. Let $\mathcal{S}$ denote the feasible set of portfolio returns (where each return $R \in \mathcal{S}$ is a random variable denoting a gross return like $\frac{X_{t+1}}{X_t}$). The idea is that by choosing different investments, we can obtain any return $R \in \mathcal{S}$ for our overall portfolio. A return $R^* \in \mathcal{S}$ is called a growth optimal portfolio return if it solves:
$$ \max_{R \in \mathcal{S}} \operatorname{E}[\log R] $$
Like many good ideas, the objective $\operatorname{E}[\log R]$ can be justified philosophically in multiple ways:
As @Dave Harris in his answer points out, it is the intuition of the latter, the attractive properties of maximizing the geometric mean, that initially motivated the expected log objective.
Note also the geometric mean return is the exponential of the (arithmetic) mean logarithmic return: $$\left(\prod_{i=1}^t R_i \right)^\frac{1}{t} = \exp\left( \frac{1}{t} \sum_{i=1}^t \log R_i \right)$$
When you maximize the log return or log wealth, you're maximizing the expected geometric growth rate.
In the long run, a growth optimal portfolio almost surely has higher wealth
Assume an IID setting where we repeatedly choose a stochastic return $R$ (from a set $\mathcal{S}$). Log wealth at time $t$ is given by the $\log W_t = \sum_{i=1}^t \log R_i$. Log wealth for the growth optimal portfolio is $\log W^*_t = \sum_{i=1}^t \log R^*_i$. We have $\operatorname{E}[\log R^*] \geq \operatorname{E}[\log R]$ by the optimality of $R^*$. Hence by the strong law of large numbers:
$$ \lim_{t \rightarrow \infty} \frac{1}{t} \left( \log W^*_t - \log W_t\right) \geq 0 \text{ almost surely}$$
Log is a monotonic transformation so if log wealth is almost surely higher, wealth is almost surely higher. Something to realize too is that if $\operatorname{E}[\log R] < 0$, you will go broke almost surely over the long run.
Maximizing expected return has deeply problematic consequences
Perhaps unintuitively, maximizing expected return can easily lead to wealth heading towards $0$ in the repeated context instead of growing towards $\infty$.
For example, in the classic Kelly problem the problem is to choose what fraction $f \in [0, 1]$ of wealth to allocate to a risky, binary bet. Imagine you gain $bf$ with probability $p$ and lose $f$ with probability $1-p$. Maximizing expected return gives the problem:
$$\max_{f \in [0, 1]} p(1 + b f) + (1-p)(1-f)$$
This is equivalent to:
$$\max_{f \in [0, 1]} (p(b+1) - 1)f$$
Thus $f^* = 1$, you bet all your wealth, if the wager has positive expectation (i.e. $p(b+1) > 1$). In the limit as $t \rightarrow \infty$, this leads you to go broke almost surely! After $t$ bets, you have $1 - p^t$ probability of having 0 wealth. The expected log return is $- \infty$.
Kelly criterion:
Maximizing the expectation of the logarithm gives the problem:
$$\max_{f \in [0, 1]} p \log (1 + b f) + (1-p) \log (1-f)$$
Assuming the bet has positive expectation, the constraints $f \in [0, 1]$ don't bind and the solution is given by the first order condition:
$$ \frac{pb}{1 + bf} = \frac{1-p}{1 - f} $$
which simplifies to
$$ f^* = \frac{p(b+1) - 1}{b} $$
A note of caution on growth optimal portfolios
While the growth optimal portfolio has many attractive properties, at least two general points of caution are in order:
- Log utility isn't magic. Other specifications, possibly with higher risk aversion, can be justified.
- Estimation errors can lead the estimated growth optimal portfolio to be substantially more risky than a naive analysis might suggest.
Models over the joint distribution of security returns are highly imprecise. You can get into extremely dangerous garbage in, garbage out problems when computing some notion of an optimal portfolio based upon highly uncertain inputs.