They are a lot of open problems in market microstructure. To have an idea of the whole landscape, have a look at Market Microstructure in Practice, 2nd Edition, by L and Laruelle.
I would split them in
From the viewpoint of exchanges
Optimal fee schedules to "attract" liquidity (and hence efficient market makers), have a look at Optimal make-take ...