Multiple sources say that you should find the optimal hedging ratio between two stocks in a pairs trade by conducting 2 linear regressions (with each stock as the independent variable), and using whichever beta value yields a highest ADF [Augmented Dickey Fuller] test statistic . Does this actually give you the optimal hedging ratio? If so, what are the mathematics behind it? It seems like a very arbitrary procedure.
The error series will be different in two cases and their stationarity will be different as well. Hence more likely than not, one of them will be less stationary than the other one.
You are right that it is a "very arbitrary procedure". More charitably it is a "hack" that gives a practical solution without addressing the fundamental issue.
The very fact that when you do an OLS regression of x vs y you get a different result than when you regress y vs x tells you that OLS regression is probably not the right tool to construct a hedged portfolio from 2 assets. How do you decide if x or y is to be considered the "independent asset"?
In the early 2000's a quant at Merrill Lynch named Mary Ann Bartels suggested that it makes more sense to use a Total Least Squares (TLS) regression to find the hedge ratio. At least TLS regression is symmetric. I cannot find her published report right now, although the idea is still floating around and has apparently been re-discovered by at least one person (http://quantdevel.com/public/pdf/betterHedgeRatios.pdf) who does not acknowledge her prior work. It seems to me the idea has merit. (Someone told me that TLS will be less numerically stable than OLS but I don't know if that is a valid criticism).
But you are right that choosing the regression with the better ADF is not justified and probably statistically biased (an example of "p-hacking").