-1
$\begingroup$

I am trying to make a histogram in numpy but numpy.histogram seems to really hate NaN values. I have tried removing NaN values from a list called data in three different ways and Quantopian doesn't let me use any of those three ways: 1.) TypeError: only integer arrays with one element can be converted to an index

data = data[~np.isnan(data)]

2.) So then I tried using pandas.dropna() and it threw: ValueError: The truth value of an array with more than one element is ambiguous. Use a.any() or a.all()

datapd = pd.Series(data)
hist, bins = np.histogram(datapd.dropna(), density=True, bins = 'auto')

3.) And when that didn't work I tried removing them on my own:

i = 0
    while(i < len(y)):
        if(float('nan') == y[i]):
            y[i] = 0
        i = i + 1

It threw the same error: ValueError: The truth value of an array with more than one element is ambiguous. Use a.any() or a.all()

How do you remove NaN values in quantopian? I tried these methods in a python console and it worked I have no clue as to why it wouldn't work in quantopian.

$\endgroup$
3
$\begingroup$

In general you should probably have a look at the following answers:

However, I'd like to address number 3.

  1. You should use numpy to check for NaNs, not use the equals operator. np.isnan(...).
  2. You shouldn't use a while loop. It's very unpythonic.
  3. Your if statement shouldn't have a parenthesis.
  4. The indentation of while part should make this script fail.
  5. Remember you're working with rows. You shouldn't get the ambiguous truth value error if your code was implemented the way you showed.

Maybe you wrote something like this:

i = 0
while(i < len(y)):
    if np.isnan(y[i]):
        y[i] = 0
    i = i + 1

This produces:

ValueError: The truth value of an array with more than one element is ambiguous. Use a.any() or a.all()

What is happening here is that the if statement gets a list of bools, and doesn't know what to do with it. Is it true when all of them are true? Or, is it true when any of them are true? You could resolve it as follows:

i = 0
while(i < len(y)):
    if np.isnan(y[i]).any():
        y[i] = 0
    i = i + 1

Or more pythonic:

for i in range(0, len(y)):
    if np.isnan(y[i]).any():
        y[i] = 0

Also be careful about the zero assignment. I don't know how pandas data frames are implemented, but typically with normal collection types you'd be replacing a row with the number zero (0).

If we weren't working with special pandas data frames, and it was simply lists I'd go for:

y = [row for row in y if not np.isnan(row).any()]

... if you wanted to delete a row with any NaN values.

Or, if you simply want to set any element which is NaN to zero:

y = [[0 if np.isnan(elm) else elm for elm in row] for row in y]

With test data:

import numpy as np
y = [[0,0],[1,1],[2,float('nan')],[3,3]]
print(y)
x = [row for row in y if not np.isnan(row).any()]
print(x)
z = [[0 if np.isnan(elm) else elm for elm in row] for row in y]
print(z)

This produces the following output:

[[0, 0], [1, 1], [2, nan], [3, 3]]
[[0, 0], [1, 1], [3, 3]]
[[0, 0], [1, 1], [2, 0], [3, 3]]
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.