A simplified example. Given:
- asset's price time series
- fixed distances to stop and target.
A function of these inputs has two possible output values: $1$ if price is likely to hit the target earlier than stop and $0$ otherwise. This function is implemented by means of machine learning, e.g. a neural network.
For reasonably and equally sized target and stop, the feasible win rate will be close to $\frac{1}{2}$. Most of times the outcome is not predictable and the right choice is not to open a position. From time to time the odds are marginally better than $\frac{1}{2}$; this is when I want the algorithm to output a $1$. Cherry picking.
When putting this up as a binary classifier, it appears that the widely used binary cross-entropy loss
- strongly penalizes confident mis-classifications
- weakly rewards being rightfully more confident
and is not necessarily appropriate here.
To be trainable by means of stochastic gradient descent, this neural network needs to produce a continuous output $\in(0;1)$, and the objective function must have a useful derivative. This is why simply counting winners and losers wouldn't help. The objective function needs to weigh network's confident and less confident decisions differently.
How would you deal with it?
Here is a similar question, probably better written.