1
$\begingroup$

suppose we have volumes every minute like below

100, 200 , 19,  0 , 200 , 12 , 100

I want to convert all these numbers to less than 10 , where 10 is max and 1 is min.

I can do this with normalisation but problem occurs when there is some sudden high volume comes like below

100, 200 , 19,  0 , 200 , 12 , 20000

where when I use normalization for past past 100 volumes, , this 20000 is affecting all other volumes.

Is there something I can do by taking averages of volumes and do normalisation for that or something ?

$\endgroup$
1
$\begingroup$

Since it is your model you can do anything. What I would do is use some dynamic outlier exclusion. For example in this case you know the min is zero.

One method (of many) might be to evaluate the median (since it might be more robust that the standard deviation or mean) and use 2 x median as your upper limit:

>>> arr = np.array([100,200,19,0,200,12,20000])
>>> upper_lim = np.median(arr) * 2
>>> arr_adj = np.where(arr>upper_lim, upper_lim, arr) / upper_lim
>>> arr_adj *= 10
>>> arr_adj
array([5, 10, 0.95, 0, 10, 0.6, 10])
| improve this answer | |
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.