Multiple sources say that you should find the optimal hedging ratio between two stocks in a pairs trade by conducting 2 linear regressions (with each stock as the independent variable), and using whichever beta value yields a highest ADF [Augmented Dickey Fuller] test statistic . Does this actually give you the optimal hedging ratio? If so, what are the mathematics behind it? It seems like a very arbitrary procedure.
You are right that it is a "very arbitrary procedure". More charitably it is a "hack" that gives a practical solution without addressing the fundamental issue.
The very fact that when you do an OLS regression of x vs y you get a different result than when you regress y vs x tells you that OLS regression is probably not the right tool to construct a hedged portfolio from 2 assets. How do you decide if x or y is to be considered the "independent asset"?
In the early 2000's a quant at Merrill Lynch named Mary Ann Bartels suggested that it makes more sense to use a Total Least Squares (TLS) regression to find the hedge ratio. At least TLS regression is symmetric. I cannot find her published report right now, although the idea is still floating around and has apparently been re-discovered by at least one person (http://quantdevel.com/public/pdf/betterHedgeRatios.pdf) who does not acknowledge her prior work. It seems to me the idea has merit. (Someone told me that TLS will be less numerically stable than OLS but I don't know if that is a valid criticism).
But you are right that choosing the regression with the better ADF is not justified and probably statistically biased (an example of "p-hacking").