I am unsure how a typical "matching engine" handles overlapping orders (in my case, Deribit). I would be just as interested in how any crypto exchange handles the situation. The user has self-trading set to disabled (like most users).
Given the following very simple orderbook...
SELL $504.... amount=20000.... owner=ALICE SELL $503.... amount=15000.... owner=BOB ------ BUY. $497.... amount=15000.... owner=BOB
If BOB sends through a new GTC limit order (not post)...
BUY... $505... amount=30000... owner=BOB
My understanding is if BOB had sent through an amount of 20000 or less, the order would execute entirely against ALICE and the status be set to "filled" without any errors (despite the order overlap and self-trading being set to disabled).
But what happens to BOB's new overlapping order (with amount=30000)?
(1) Rejected because BOB's order did not completely fill requested amount=30000 so an "order_overlap" error is returned and filled_amount=0.
(2) Partially filled and status set to "cancelled" but with filled_amount=20000
(3) Partially filled and status set to "open" with filled_amount=20000 with price adjusted down (to 502.50 after executing at $504.00 against ALICE)
"The matching engine DOES NOT allow for self-trading. If an order is sent to the exchange that would execute immediately with an order from the same account in the order book, this order will be rejected. Note that the order will only be rejected if it would actually execute against another order in the same account. So an order overlapping with other orders in the same account might still be accepted and executed if it would execute against other orders in the order book, not from the same account."
Their documentation suggests they aim to maximize volume (this is good) by matching BOB's orders with ALICE's orders regardless of BOB's orders overlapping... but the documentation is very ambiguous. The second and third sentence suggests they may not try maximize volume in some cases. Surely BOB's orders will ALWAYS execute against ALICE's order if they overlap? (Assuming the exchange aims to maximize volume rather than always throwing an error due to BOB's orders overlapping).