It depends on what you want to optimize with transaction costs:
- liquidation
- hedging
- allocation
The two best reference I have in mind are:
Gökay, S., Roch, A., Soner, 2011. Liquidity models in continuous and discrete time. In: Di Nunno, G., Øksendal, B. (Eds.), Advanced Mathematical Methods for Finance. Springer Berlin Heidelberg, pp. 333-365. URL http://dx.doi.org/10.1007/978-3-642-18412-3_13 - is a review paper covering:
- optimal liquidation
- hedging
for portfolio allocation, you have a nice review paper here: Kolm, P., Dec. 2009. Stochastic optimal control and dynamic portfolio analysis. In: Workshop on High-Frequency Finance and quantitative strategies. Courant Institute, NYU. URL http://www.slideshare.net/pkolm/60-years-of-portfolio-optimization-practical-challenges-and-current-trends
You will not find in the literature a paper covering exactly a specific problem.
Nevertheless my viewpoint is that the difficulties in your case (i.e. mixing combinatorial constraints, robust optimization and transaction costs) comes from the way to insert transaction costs in existing frameworks.
As it is explained into the Soner et al. paper: there is one paper by Almgren dealing with "when should I end liquidation?". It is an important question since trading infinitesimally can be perceived as a solution to market impact minimization. With M Labadie, we extensively covered this aspect of discretizing a liquidation process in "Optimal starting times, stopping times and risk measures for algorithmic trading: Target Close and Implementation Shortfall".
The question "when should I end selecting lines for a portfolio?" (i.e. not that far from asset count constraints) is in fact close to the previous one.
Not deep enough to justify an academic paper according to me, since once you choose:
- a utility function (i.e. a criterion),
- a market impact (i.e. transaction cost) model,
- the parameters or model features you want to be robust to
the result should come straightforward thanks to a combination of techniques described in the papers I cited.