Best i could find was:
((BP1 x N1) + (CP * N2)) / (N1+ N2) = AVG
BP1 = initlal price paid
N1 = intial shares bought
CP = current price
AVG is your desired average cost
Put the numbers in and solve for N2
Generally in a regulated exchange, you are not allowed to skip price points. In the case you have shown, BOB's new Buy order will skip the price point at 503 and execute at $504. Therefore the order will get cancelled/expired/rejected even if the time constraint of the order is not IOC/FAK.
So in both scenarios you mentioned (whether the order quantity is 20,...