3
$\begingroup$

I am looking to store equity price data in a hdf5 table. The use will be purely as a historical archive, not as day-to-day data source.

Options

  1. One option would be to store base10 significand and exponent separately as e.g. uint64 and uint8. The downside is that it is fairly awkward to handle especially as int do not come out of the box with NaN handling for missing values.
  2. The other option would be to use float64 which is easier to handle and has NaN support built-in.

My question: Does float64 have sufficient precision to store price data? What is the experience of the number of significant digits required for a price archive?

Note: float64 seems to have 15-17 "significant decimal digits" precision. Not sure whether this means "significant digits" or whether this only refers to the decimal digits.

Improve this question
$\endgroup$

3 Answers 3

Reset to default
5
$\begingroup$

As background, Floating point precision is a way of storing numbers such that the precision is relative to the largest digit. For instance, the number $0.00123$ stored in fixed precision needs 6 digits of precision (3 zeros and the 3 non-zero numbers). However, this same number stored as floating point precision $1.23 \cdot 10^{-3}$ needs only 3 significant decimal digits to store. Floating point is generally a more efficient way to store numbers that have many different orders of magnitude but more importantly they are stored in a form where the computer can do efficient basic calculations like multiplication and in your case is much easier to work with than option 1.

Even the deepest markets (treasuries, currencies) need only 6-7 digits of floating precision to store price data. There are some important limitations meaning the prices stored won't always be exact but maybe approximated at the 16th or 17th digit. If the prices are used in calculations later (we are on quant finance after all) it would be shocking if for numbers with 6-7 digits of precision rounding to the 16th/17th digit mattered at all. On the corner case that this approximation matters you can look into option 3 which is fixed point storage.

Improve this answer
$\endgroup$
5
  • $\begingroup$ As a cheap plug, if you are using Python the Pandas package has wonderful time series manipulation with NaN handling and interacts very well with hdf5. $\endgroup$
    – rhaskett
    Commented Nov 17, 2014 at 18:37
  • $\begingroup$ Thanks. I am using pandas but NaN is only supported for floats not for ints as I pointed out in my original post. Unless I am doing something very wrong. Thus my question... $\endgroup$
    – Max
    Commented Nov 17, 2014 at 20:36
  • $\begingroup$ No, you're correct. NumPy does not support NaN for int types. Only for floats. Note that this is not some universal property of NaN's -- it was a choice made by the authors from NumPy not to support NaN for int. $\endgroup$
    – Olaf
    Commented Nov 17, 2014 at 22:51
  • 1
    $\begingroup$ Also, like rhaskett said, what you described in Option 1 is essentially how floating point numbers are stored. $\endgroup$
    – Olaf
    Commented Nov 17, 2014 at 22:53
  • $\begingroup$ Return calculations should be fine. Though adding multiple returns could accumulate errors after many, many calculations. You can always test your particular calculations by comparing float calculations to Decimal calculations for prices that are very close. $\endgroup$
    – rhaskett
    Commented Nov 17, 2014 at 23:44
1
$\begingroup$

As a practical viewpoint: having habits from float32-by-default time, I designed my db with Numeric type (aka Decimal, ie fixed-precision). In this case, it was important. In most case, I took a maximum 8 digits precision.

But now with float64 + Numpy (on which Pandas is based) not handling Decimal but float64, I'm converting my db schema to float64 (ie double on postgresql). Moving back-and-forth from float64 (for Numpy processing) and Decimal (for storage) is just too much pain...

Being also a market operator, I hardly see any value-added above 10 digits, markets bid/ask being what they are.

Improve this answer
$\endgroup$
1
  • $\begingroup$ I had the same setup, fixed-precision on db and float64 for computations (also because numpa/pandas does not recognize Decimal type). we don't have use of more than 10 digits, same conclusion as you reached. it's just boring to have 0.40000001 values stored in db... $\endgroup$
    – comte
    Commented Feb 28, 2018 at 18:45
0
$\begingroup$

In my case, it really depends if you are using any software at all. I usually get my Discount Factors with a 15 decimals precision. So im always storing at least 15 digits for DF, and yields up to 7-8 as rhaskett said.

For equities, i guess it's pretty much the same as for yields, or even less. Nevertheless i store any return, price or ratio with a 7-8 digit precision.

Improve this answer
$\endgroup$
2
  • $\begingroup$ What is the reasoning that you are storing returns with a potentially lower precision? If return = price(t+1)/price(t)-1 and the two prices are very close, the algebra eats up the significant digits really quickly. Unless you remember to round every time, follow-on calculation can accumulate substantial errors, no? $\endgroup$
    – Max
    Commented Nov 17, 2014 at 20:41
  • $\begingroup$ Depends on the original precision of the prices. Prices of many asset classes can often have only a few digits of precision. So, storing with 7-8 digits is not really a problem. Still, my feeling is that storage space is generally cheap, so I'm happy to store at whatever precision that I do my calculations. $\endgroup$
    – rhaskett
    Commented Nov 17, 2014 at 23:39

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.