I am unsure how a typical "matching engine" handles overlapping orders (in my case, Deribit). I would be just as interested in how any crypto exchange handles the situation. The user has self-trading set to disabled (like most users).
Given the following very simple orderbook...
SELL $504.... amount=20000.... owner=ALICE
SELL $503.... amount=15000.... owner=BOB
------
BUY. $497.... amount=15000.... owner=BOB
If BOB sends through a new GTC limit order (not post)...
BUY... $505... amount=30000... owner=BOB
My understanding is if BOB had sent through an amount of 20000 or less, the order would execute entirely against ALICE and the status be set to "filled" without any errors (despite the order overlap and self-trading being set to disabled).
But what happens to BOB's new overlapping order (with amount=30000)?
(1) Rejected because BOB's order did not completely fill requested amount=30000 so an "order_overlap" error is returned and filled_amount=0.
(2) Partially filled and status set to "cancelled" but with filled_amount=20000
(3) Partially filled and status set to "open" with filled_amount=20000 with price adjusted down (to 502.50 after executing at $504.00 against ALICE)