In addition to @madilyn's answer, there is one point that needs to be addressed and that is often called an unfair advantage although it is merely a competitive advantage.
Take the US Equities market. There are now several venues on which the same symbols are traded. If one HFT acquires information about one symbol in one venue - e.g. due to a limit order being filled - it might try to act upon that information very quickly on the other venues on which it is also active. That may result in surprising behavior to people with little knowledge of market microstructure. For instance, quotes on other venues may be cancelled before every chunk of a single order sent to a broker (and naïvely routed) has reached its venue, resulting in partial fills or slippage.
HFT firms pay dearly for connectivity and colocation and work on very tight profit margins. In return, they get the possibility of making more efficient use of new information than most other market participants. If you take speed away, the vast majority of venues (and the most active ones) do not offer any particular advantage to an HFT than to any other participant. As for speed, it is a matter of access like in any other field, and criticizing HFTs because of that is like arguing that floor traders had an unfair advantage in the days when pit trading was the only way to trade actively. There is a service being provided, people investing time and money to do that, competition driving profits down and costs up. Nothing unfair to it, really.