I can think of three reasons.
First, and simplest, is that people care about variance.
Second, if you really do care about draw-downs, if returns are close to normally distributed, the distribution of draw-downs is just a function of the variance, so there's no need to include draw-downs explicitly in your portfolio construction objective. Minimizing variance is the same as minimizing expected draw-downs.
Third, (and here's the main point) if returns are very non-normal and you really do want to find portfolio weights that minimize expected draw-downs, you still wouldn't choose weights that minimize historical draw-down. Why? Because minimizing historical draw-down is effectively the same as taking all your returns that weren't part of a draw-down, and hiding them from your optimizer, which, as Tal mentioned, will lead to portfolio weights that are a lot less accurately estimated than if you let your optimizer see all the data you have. Instead, you might just include terms in your optimization objective that penalize negative skew and penalize positive kurtosis.
Or, if you wanted to get fancy, you could use all your historical data to fit your favorite fat-tailed distribution. As far as I know, none of those distributions have both finite variance and are closed under linear combinations, so your portfolio returns wouldn't have the same distribution. So to get optimal portfolio weights, you'd have to simulate some very large number of returns from each of these distributions, and then calculate (let's say) the 99th percentile portfolio draw-down for each set of portfolio weights, and use that function in your optimization. I would think that would give you more robust results. Though since you're trying to estimate a tail event, it would probably require a huge amount of simulated data and take a long long time to run. After you finish, you might fairly conclude that the entire exercise wasn't worth the effort, and just go back to mean variance optimization...