How does a bank assess the liquidity of OTC derivatives such as swaps, options, and forwards, for which there isn't public data regarding trading volume?
Assuming you're using "bank" in generic reference to any buy-side account (not a market marker), I'd say there are 3 primary ways to asses liquidity. I think it's important to note that in OTC derivatives space, when we assess liquidity in a certain instrument, we're really assessing the liquidity in whatever instrument or instruments the counterparty will use to hedge their exposure once a trade is on.
If we're trading an OTC option on Amazon equity for example, if our CP has access to ample liquidity to delta/gamma hedge the option, then certainly the option contract itself will appear to be similarly "liquid". If we're opening a swap on an esoteric basket of equities that will be hard for the CP to source, then again, because the hedge or back-to-back trade is "illiquid", the swap trade itself will appear to be similarly "illiquid".
Bid/Ask Spread As one commenter alluded, the spread between bid quotes and ask quotes is probably the best way to asses liquidity in an OTC market. Wide bid/ask will pretty much always be a good barometer of "illiquidity" and a CP demanding more spread to open exposure.
Break fees Another variable I'd look at is break fees and/or trade covenants. Trading swap s this indicator is particularly useful. High break fees or even stipulations that make swap contract unbreakable are great gauges of liquidity. If a counterparty is going to have a lot of trouble sourcing their hedge at inception of a contract, they'll make it really expensive or impossible to force them into a position where they have to offload that hedge as well.
Trade Tenor/Maturity Another tell-tale sign of illiquid markets is difficulty in sourcing longer tenor trades: long-dated options, swaps past 6 month maturity, forwards that extend over multiple IMM roll dates etc. When risk is difficult to manage on the CPs end, they'll generally express that difficulty by managing the longevity of any one trade.
A caveat to all of these points is that the nature of OTC derivatives almost by definition makes liquidity difficult to measure. Because risk is managed bilaterally by counterparty, there are an infinite number of idiosyncratic conditions that may influence a counterparties' willingness to take risk and they might reflect that willingness in bid/ask, break fees etc. Best practice would always have us quote at least three desks on a trade so that we can attempt to distinguish between counterparty specific risk appetite and general liquidity in a certain product.