This is the whole problem with the FF3 model, and indeed the very reason it was ever concocted in the first place!
CAPM 101 argues that the cost of equity is constant riskless plus market beta. Except this doesn't work, because there are clear value and size; or value, size, and momentum effects also in operation. In statistical parlance, the null hypothesis that these style factors have a zero risk premium can be really disproved.
However, proving these have not been not zero does not tell you what value beyond zero they are, let alone "should" be. The factors simply exist to help correct for the divergence of CAPM v1.0 from observed reality. These style effects can be measured ex-post. However, that offers no intuitive ex-ante explanation for why the size effect should be 0.5%, 1%, 2% or 4% per annum; and likewise for value and momentum.
There's a vague and intuitive logic for why the size and value effects exist, with positive risk premia. Momentum is more difficult here. But nobody - to my knowledge - can sensibly suggest an intuitive economic model to suggest what the "fair value" of any of those incremental risk premia is, or should be.
As such, they've always been crimes against theory. Like oh so many elementary particles in physics, they exist to make the equations continue to work. Except that just means we need them for theory's sake. It doesn't mean we actually understand them ;-)