When fees are not symmetric, to take fees into account on orderbook needs to know if you want to provide or consume liquidity: you have in fact two different views (ie two ranking) on the same orderbook:
Say you are a buyer, and do the calculation for the first limit only. $P^B(i)$ and $P^A(i)$ are respectively the prices at the bid and ask on venue $i$, where marker fees (ie for liquidity providers) are $f^m(i)$ and taker fees (for liquidity consumers / removers) $f^t(i)$. Remember that for rebates, you have to put a minus sign in the fees.
- if you you want to provide liquidity, you have to compare
- if you want to consume liquidity the game is taking place at the ask: compare
(you add fees to the price because you are buying, hence you will spend more money; for a seller you will get less money, so you have to put a minus sign in front of the fees).
But the reasoning cannot stop at this point:
- For liquidity consumption, it is simple: just go to the cheapest venue.
- But for liquidity provision, you need to account for the probability to obtain a transaction.
Say for instance that the best bid of venue $i$, once fees are taken into account, is cheapest than the the best bid of venue $j$, even that it is equal to the second bid of venue $j$. How can you be sure that consumers (ie seller) will not go to venue $j$ first? since the fees are not the same, the order can be inverted for them: it may be more attractive for consumers on venue $j$... In such a case you will wait on venue $i$, expecting a better net price, but nobody will ever come to trade there with you...
To avoid this configuration, the price improvement to wait on venue $i$, ie
$$\Delta P^m(i|j):=P^B(j)\cdot(1+f^m(j)) - P^B(i)\cdot(1+f^m(i))$$should have the same sign as the price improvement to come and consume on this venue, ie
$$\Delta P^t(i|j):=P^B(i)\cdot(1-f^t(i))- P^B(j)\cdot(1-f^t(j)).$$
This is somehow a theoretical equilibrium, thus you can have a look at the rate at which market orders are consuming your side, at your price limit to assess the probability to obtain e transaction. You can do this on a long term historical basis (estimating the Markov chain of rates given the state of the orderbook on several days), or in real-time.
For details have a look at Market Microstructure in Practice (1st or 2nd edition).