How is historical data for forex collected or computed?

Question

I'm looking at four sources of forex data, as compiled in the question, What data sources are available online? And I think I must be misunderstanding something, perhaps something fundamental, but I'm not sure what. Given my ignorance, it's hard to pin down my confusion as a single question, so I'll express a few, in hopes someone can pick up on the source of my confusion and shed light.

• The tick data from DukasCopy shows 5 columns: Time, Ask, Bid, AskVolume, and BidVolume. Am I correct to believe that this data shows no information about actual executions—only changes to the quotes? (Am I correct to believe that the AskVolume and BidVolume columns describe only the quantity offered at the Ask and Bid prices, respectively?)

• The 1-second data from DukasCopy shows 6 columns: Time, Open, High, Low, Close, and Volume. Am I correct to believe that this data does show information about actual executions, i.e. Volume being the quantity traded in the given time period of 1 second?

• OANDA only provides daily and weekly data, the forexforums.com links only provide 1-minute data (resembling DukasCopy's 1-second data), and GAIN Capital seems to provide tick data (resembling DukasCopy's tick data) without the bid and ask volumes. I have looked at many other sources as well, but can't seem to find tick data on trades, i.e. the time, price, and quantity at every trade. Am I looking for something that doesn't actually exist? If so, then why does it not exist?

• I suspect it could have something to do with the non-centralized nature of the forex markets. But then, what exactly does this historical data, e.g. DukasCopy's, mean? Do these numbers only represent the quotes (tick data) and trades (1-second data) handled by DukasCopy (who I understand to be a broker)? Or, do they indeed come from some centralized aggregation of quotes and/or trades?

I have looked for explanations regarding the data on each of the four websites, but to no avail. I apologize for asking such a basic question; I'm very new to forex, and I'm surprised by how different it is from the the NYSE/NASDAQ markets, which I'm slightly more familiar with.

Answer 1

The fx market, contrary to most other asset classes is an almost entirely fragmented over-the-counter market, aside the very small number of fx futures that are trading at dismal liquidity levels. Therefore, you will not encounter a single serious liquidity provider that will take a stab at estimating total traded volume in any of the currency pairs. Having said that, there are some brokers that may show traded volume that crosses their own books, only. You cannot compare fx markets with any listed cash equity, futures, or options market. Why this is the case? Simple, historical reasons and traders and market participants really are against change unless of course it changes their own bottom lines. That is why many initiatives that were pushed by risk managers, back and middle offices died for lack of front office support, such as trade settlement at T+0. A decentralized market in fx works well enough for most players to see no need to change much about it. ECNs and aggregation frameworks saw the void and now provide quite a chunk of liquidity but in the end all they do is, well, aggregate. The actual liquidity still comes from various houses. In a sense it is funny if you really think about it because a 200 million block gets in the end passed from one hand to the other, while each hand retains a small chunk until it is split up sufficiently for the remaining hand to hold. So the same 200 million or part of it may pass the same brokers and order books several times before it is swallowed up by those the risk eventually ended up with.

To your bullets above (1-4):

1) you are right. And its only the quantity that the broker/ecn itself sees.

2) I cannot confirm that but I would logically assume that the volume refers to trades but again only at the broker. I have not come across many brokers that show trading volume or average traded prices or any other related trade data but that does not mean it does not exist. Some ECNs do show traded volume.

3) See 2, regarding trading data at each ecn or broker, it does exist but traditionally most market participants have not expressed much interest in such data because it is pretty meaningless. You can't meaningfully extrapolate to the market as a whole.

4) No centralized fx trading, hence no centralized aggregation unless you refer to ECNs that centralize/aggregate a chunk of total trading interest.

Answer 2

Correct explanation by Freddy. Retail investors and even most institutional investors don't have access to trading, bid, or ask volumes. The reason is that there is no centralized body who would aggregate data. Would it be possible to put it in place? Certainly, but the big fx players (handful of the really big banks) would suffer.

Although not even the big banks don't see all the flows (only flow going through their books), they see a good amount of it and use it to trade. Just to get an idea how concentrated the market is, in the US in 2010, 7 banks account for 75% of fx turnover (see BIS Triennial Central Bank Survey). For example, it is a very valuable information to know how many billion of USD/JPY are sitting on the bid/ask because it will take the market longer to eat through all of it and this will act as support/resistance.

Answer 3

How about:

1. Each tick is a bid or ask order for some amount. Dukascopy could have order feeds into a number of FX platforms and aggregate the orders found there. They could be iceberg orders. When there's an execution the liquidity is removed from the order book to the trade book. The time this takes to propagate through execution venues can create liquidity mirages. Alternatively orders are simply cancelled.
2. In one second you could have tens (or more) of orders across a range of prices. So during the second you have the open, high, low, close and a volume number which is probably the gross turnover of orders that were seen on the Dukascopy (or aggregate) book.
3. Data on trades exists but it's in a different book to an order book. Plus it may involve proprietary positions (ie inventory, see http://www.math.nyu.edu/faculty/avellane/HighFrequencyTrading.pdf). Then again, regulations these days require most FX (except spot) trades to be reported to trade repositories. You could try those (ie http://www.regis-tr.com/)
4. I would say it's a mixture between the proprietary order flow and order flow across several well known FX platforms (EBS, Reuters, leading banks).