How Crypto Exchanges handle overlapping orders when self trading is disabled?

Question

I am unsure how a typical "matching engine" handles overlapping orders (in my case, Deribit). I would be just as interested in how any crypto exchange handles the situation. The user has self-trading set to disabled (like most users).

Given the following very simple orderbook...

SELL $504.... amount=20000.... owner=ALICE
SELL $503.... amount=15000.... owner=BOB
------
BUY. $497.... amount=15000.... owner=BOB

If BOB sends through a new GTC limit order (not post)...

BUY... $505... amount=30000... owner=BOB

My understanding is if BOB had sent through an amount of 20000 or less, the order would execute entirely against ALICE and the status be set to "filled" without any errors (despite the order overlap and self-trading being set to disabled).

But what happens to BOB's new overlapping order (with amount=30000)?

(1) Rejected because BOB's order did not completely fill requested amount=30000 so an "order_overlap" error is returned and filled_amount=0.

(2) Partially filled and status set to "cancelled" but with filled_amount=20000

(3) Partially filled and status set to "open" with filled_amount=20000 with price adjusted down (to 502.50 after executing at $504.00 against ALICE)

Answer 1

Generally in a regulated exchange, you are not allowed to skip price points. In the case you have shown, BOB's new Buy order will skip the price point at 503 and execute at $504. Therefore the order will get cancelled/expired/rejected even if the time constraint of the order is not IOC/FAK.

So in both scenarios you mentioned (whether the order quantity is 20,000 or 30,000), the regulated orders will not execute since they are skipping a price point.

The important point to remember here is that the exchanges are supposed to provide a price-time order book. Take the following scenario:

SELL ALICE 10000 @ $503 at 17:00:00.000 UTC
SELL BOB 5000 @ $503 at 17:00:00.010 UTC
SELL DAVE 1000 @ $503 at 17:10:00.025 UTC
SELL JANICE 1000 @ $504 at 16:58:00.000 UTC

In this, you can see while JANICE has sent the order earlier on, it is at the back of the queue because the price is worse. Now let's suppose BOB submits a new BUY order 11000 @ $503, this will first match 10000 with ALICE. If self-trade-prevention is on, then it will skip BOB's own order and will match with DAVE.

Let's suppose the order size was 15000 instead. In a regulated exchange, it will match with ALICE and DAVE and the remaining 4000 will be expired.

Do note that some exchanges provide additional controls over the order such as FOK (Fill the entire order, Or kill it. No partial fills), min fill sizes etc. So depending on that your experience may slightly vary.