I have read about something like Kelly criterion for long term expectation maximization assuming a fixed starting bankroll. But if one can assume unlimited leverage, and one has a signal for a price movement..how could one go about deciding optimal trading sizes? It would seem to me like there are a lot of factors involved (things like transaction costs, inventories, etc..) - in practice would one come up with some intuitive model for this and run historical simulations for it? Or how would one tackle this? Is there any literature which I can read up on to understand this problem better?
There are few things to consider.
Trading moves the price, to minimize market impact and maximize return it is generally optimal to split an order in several child orders. See the Kyle model.
Splitting optimally dependents on specific assumptions that you make. The simplest (and first) approach is that of Berstsimas and Lo (Optimal Control of Execution Costs). Almagren improves on it considering more realistic price impact fuctions. You can find much of his work at his homepage. More recently, the focus has shifted to optimal submission strategies in the limit order book (e.g.: Obizhaeva and Wang) and to latency cost (for example see here).
From a consistency perspective, the work of Jim Gatheral (No dynamic arbitrage and market costs) details some price impact functions that don't allow dynamic arbitrage (for example, "pump and dump" strategies). The high brow approach to these ideas is contained in the Econometrica paper by Stanzl and Huberman (Quasi-arbitrage and Price Manipulation). This area is expanding rapidly as several groups are working on expanding these results to limit order book markets.
The investor's holdings is a consequence of an investor's utility function interacting with the investor's perceived trading opportunity subject to constraints. (Indeed, the Kelly criterion is also utility maximizing.)
We produced trades by re-balancing -- that is to say, we have new expectations of alpha or risk and the optimal portfolio net of these factors and transactions costs produces a new set of optimal portfolio weights. The difference between the new target set of weights and current weights multiplied by the portfolio value produces a trade-schedule.
A utility function would likely include the following terms i) contribution from alpha (in % returns terms, or risk-adjusted terms), a ii) a penalty for taking on portfolio risk defined in some way (variance, cVaR, etc.), a iii) a penalty for incurring transactions costs (again, in % of the portfolio terms). The last term in particular would account for existing holdings, liquidity, bid-ask spreads, and market impact.
Each one of these terms is a well-defined function of portfolio weights. For example, (i) is the cross-product of expected alphas and security weights; (ii) might be the variance of the portfolio given a covariance matrix and weights. Since the utility is a function of portfolio weights, an optimizer can be used to identify the utility-maximizing portfolio(s).
Each portfolio objective would also have its own lambda associated with it based on the relative priority of the objectives. For example, in mean-variance optimization a frontier of efficient portfolios is identified and the investor's risk-aversion as expressed by the lambda identifies the optimal portfolio along the frontier.
Each function can also be tested and evaluated in isolation. For example, using real trade data you can update your transactions costs model to improve your overall portfolio construction process. So focusing on developing effective models at the objective level would be a reasonable research program before chaining them together in a utility function.
There are several ways to implement these multiple goal objectives. Polynomial goal programming for example. Or you can find a optimizer that uses the method of lagrange multipliers and gradients to identify the optimal portfolio, or genetic algorithms.