I have read about something like Kelly criterion for long term expectation maximization assuming a fixed starting bankroll. But if one can assume unlimited leverage, and one has a signal for a price movement..how could one go about deciding optimal trading sizes? It would seem to me like there are a lot of factors involved (things like transaction costs, inventories, etc..) - in practice would one come up with some intuitive model for this and run historical simulations for it? Or how would one tackle this? Is there any literature which I can read up on to understand this problem better?
There are few things to consider.
The investor's holdings is a consequence of an investor's utility function interacting with the investor's perceived trading opportunity subject to constraints. (Indeed, the Kelly criterion is also utility maximizing.)
We produced trades by re-balancing -- that is to say, we have new expectations of alpha or risk and the optimal portfolio net of these factors and transactions costs produces a new set of optimal portfolio weights. The difference between the new target set of weights and current weights multiplied by the portfolio value produces a trade-schedule.
A utility function would likely include the following terms i) contribution from alpha (in % returns terms, or risk-adjusted terms), a ii) a penalty for taking on portfolio risk defined in some way (variance, cVaR, etc.), a iii) a penalty for incurring transactions costs (again, in % of the portfolio terms). The last term in particular would account for existing holdings, liquidity, bid-ask spreads, and market impact.
Each one of these terms is a well-defined function of portfolio weights. For example, (i) is the cross-product of expected alphas and security weights; (ii) might be the variance of the portfolio given a covariance matrix and weights. Since the utility is a function of portfolio weights, an optimizer can be used to identify the utility-maximizing portfolio(s).
Each portfolio objective would also have its own lambda associated with it based on the relative priority of the objectives. For example, in mean-variance optimization a frontier of efficient portfolios is identified and the investor's risk-aversion as expressed by the lambda identifies the optimal portfolio along the frontier.
Each function can also be tested and evaluated in isolation. For example, using real trade data you can update your transactions costs model to improve your overall portfolio construction process. So focusing on developing effective models at the objective level would be a reasonable research program before chaining them together in a utility function.
There are several ways to implement these multiple goal objectives. Polynomial goal programming for example. Or you can find a optimizer that uses the method of lagrange multipliers and gradients to identify the optimal portfolio, or genetic algorithms.