Since order book granularity backtesting is challenging, as you've pointed out, I recommend first deciding your business requirements:
Can you rely on a third party execution desk? I do not recommend modeling the transaction costs as a static number of bps, but the one exception is if all you're doing is dropping a large parent order and someone else is coming up with the execution trajectory on your behalf (this could be some other team within a company that is focused on execution research OR a bank's algo execution service).
Does your strategy need to use passive orders? A strictly liquidity-taking strategy is easier to backtest.
If your strategy uses passive orders, how important are your passive fills relative to the alpha that you're capturing?
Then, the most common 4 approaches are described below, in increasing order of difficulty.
Approach 1 (liquidity-taking only)
The best scenario is if you are only running a liquidity-taking strategy. Then, you could skirt the issue even with only top of book data: the practical solution is to backtest the strategy with a min size that is available at touch, put the strategy into production, then ramp up the size by hand, and calibrating for capacity based on real world PnL.
Another reason this tends to work well is that size has a self-fulfilling effect for liquidity-taking strategies - modulo liquidity replenishment effects, your PnL should get better with larger size because you're dislocating the spread further in your trade direction.
Approach 2 (conservative fills with MBP or consolidated top of book)
If you have to use passive orders, then the next best approach is to assume that the price that your order is on must trade through completely, i.e. a trade must post on the next level, to simulate a fill.
This takes into account the effect of hidden or max show orders etc. This is imperfect, since it is conservative on the fill probability. If the alpha that you're capturing is mostly from providing liquidity, then this usually doesn't work since it will underestimate the fills with adverse selection. On the other hand, if you have some longer horizon alphas and operate on a lower frequency, and passive orders are only being used to reduce slippage, this still works fairly well for backtesting and in practice.
Approach 3a (conservative MBP simulation)
To go one step further in sophistication, you'd need market-by-price (MBP) data. Then, you can just assume that all adds go behind your order while all cancels go to the backmost of a level.
This works well in practice because orders closer to the front have more optionality and tend not to be canceled.
Approach 3b (optimistic MBP simulation by modeling queue position)
Rather than conservatively assuming that all cancels go to the back, you can get more sophisticated by modeling your queue position from MBP data.
This is more sophisticated than 3a, but sophistication is not always a good thing - and I don't recommend this approach because it is hard to get right, and for the time you spend on it, you might as well invest it in Approach 4.
An anecdote in recent history can be found on CME, which was one of the most notable markets to shift from MBP to MBO. Some sophisticated trading firms had an edge from modeling queue position on MBP data, but that trade almost instantly vanished after MBO was introduced. This goes to show why Approach 4 is superior to Approach 3b. Otherwise, 3b is a fun academic project or the sort of thing you'd assign college interns at a prop trading firm.
Approach 4 (using MBO data)
Finally, you can just get your explicit queue position by backtesting with MBO data.
The most important thing to keep in mind here is that only a few vendors provide MBO data with the right timestamping techniques for this kind of simulation. (Disclaimer: The firm I work for, Databento is one of them.)
Most firms will market "high-frequency TAQ data" that is simply top of book from the SIPs, because it's much cheaper to distribute the SIPs. Other firms will market "L2" data that is just MBP data. And for the remaining firms that do provide MBO data, they often don't use appropriate timestamping techniques.
This then shifts your initial attention to the implementation of a LOB structure, which is the trivial part.
What most of your attention then goes to is (i) estimating matching engine round trip times and (ii) calibrating your estimate based on real world fills. The right timestamping techniques come into play especially because of (i).
