IMHO the 'definition' you mention is not a mathematical definition per se, but rather an approximation used by some practitioners.
Mathematically, it is $N(d_2)$ in the BS formula which figures the conditional probability that the terminal asset price $S_T$ will finish above the strike level $X$ given the information we possess today (represented by $S_t$), and that, under a risk-neutral measure which uses the risk-free money market account as numéraire. Omitting discounting, $N(d_2)$ is indeed the price of a binary or digital option assuming GBM.
The delta (omitting dividend payments) is on the other hand given by $N(d_1)$ not $N(d_2)$. Only for extremely short time to maturity or low volatility can $d_1$ be approximated by $d_2$ and hence $\Delta$ be approximated by $N(d_2)$ (again assuming no divs).
Still, we could also say that $\Delta$ corresponds to a probability of finishing in the money, but in an equivalent martingale measure which uses the asset itself as numéraire.
It therefore depends on the probability measure in which you are working/interpreting your results. In the sell-side we like to think in terms of risk-neutral probabilities (or binary options).
See the definition of $d_1$ and $d_2$ here for instance.