I have a binary classification model that predicts BUY (1) and SELL (-1) with an out of sample F1 score of 71% (precision is 65% and recall is 80%). The model's output is a probability of a BUY label occurring, which is then used in a bet sizing formula to bet cash (similar to the kelly criterion). The model was also trained taking commission and conservative slippage into account.
Now that I have the trained model, I am backtesting on out of sample data. However, it seems that the execution is losing money in the long term. The model is performing as expected - even on the backtest data, it still performs with an F1 score of 71%, but it seems commission and slippage are eroding gains.
Are there any tips / rules / literature one should follow when backtesting and simulating execution? I find it strange that the model performs so well on out of sample data yet still loses money - again, the model was trained with conservative estimates on slippage, higher than would ever occur in actual trading.
EDIT: As suggested in the comments, I will expand a bit more on what the model is doing. The data is simple OHLCV time-series with some feature engineering applied (e.g. CDF values for the distribution of returns, making the prices stationary through the backshift operator, normalizing data between 0 and 1). The method for labelling data is taken from Lopez's Advances in Financial Machine Learing, specifically the Triple Barrier Labelling method; first you calculate percentage returns from close price, then calculate a EWMA of standard deviation (std) of these returns - this is like an implied volatility. The upper barrier is a multiple of the EWMA of std of returns and similarly with the lower barrier, the 3rd vertical barrier is a fixed window of time later (say 60 minutes, or 5 days). At each time
i, calculate the returns between
i + j. If this return hits (is greater than or equal to) the upper barrier (upper EWMA of std of returns at
i), it is labelled BUY (1) at time
i. If the return hits the lower barrier (lower EWMA of std of returns at
i), it is labelled SELL (-1) at time
i. If the return has not hit either upper or lower barrier by the time the fixed window time has elapsed, it is labelled a SELL (-1) at time
The backtest data is distributed the same as the training data, as shown by a chi-square test and the fact that the model achieves similar F1 scores between test data and backtest data (both out of sample).