EMH says that one can not earn excess return using some information. This is known as joint-hypothesis problem: to test for market efficiency one have to determine first what is "normal" market return, i.e. what type of information is normally priced by the market. Usually to test for EMH they use CAPM or 3-factor Fama-French model (which is a kind of CAPM-on-crutches), or 4-factor Carhart, of Stambaugh, of 5-factor Fama-French. CAPM, in turn, is a model for asset price in the world where everyone is using Markowitz-style optimization. Other models are built on CAPM. Authors who created it tried to overcome CAPM inconsistencies with empirical evidence.
Now back to Markowitz. Markowitz just recommends what you have to do, if you (1) want to be optimal, (2) have some expectations of return and risk. It EMH holds, your expectations wouldn't earn you excess return (but would earn you some normal). If not, then you'll earn "normal" + excess return. That's why Markowitz had won his Nobel prize: his theory is good both for efficient and non-efficient market, whatever you define "efficiency".
Now back to your question. "If we want to be compliant with EMH, shouldn't we believe that those estimates make no sense?" EMH doesn't stop you from having your own views. It just says that whatever your views would be, you can not use it and earn return in excess of return predicted by some market model (CAPM, FF3F, etc.) But if you have views, you have to organise portfolio according to Markowitz.