Anyone knows why the OIS leg for basis swaps pays the average rate instead of the geometric average compounding rate as expected for a regular OIS swap leg?
As far as I know, it's a market convention. The two products, namely OIS swap (fixed vs floating) and Fed Fund Libor basis swap, are developed differently, so they follow different conventions.
My only guess is that it's because of the difference in maturity and period: OIS swap is typically a single-period swap (i.e. zero coupon swap) on short-end (< 2 yrs) while FF Libor basis swap is multiple period swap on long-end (> 2 yrs). The geometric compounding is for mimicking the daily reinvestment, so it makes sense for the single period. However, the daily reinvestment is not in line with multiple period swap because the notional is reset at every period, so you don't have to stick to the geometric compounding. In fact, arithmetic averaging is a much simpler computing choice to layman.