Take the 2-minute tour ×
Quantitative Finance Stack Exchange is a question and answer site for finance professionals and academics. It's 100% free, no registration required.

I'm trying to do a regression on my panel data. Say I have T=3500 days of data and N=125 firms.

Since Matlab get's major memory issues (which I try to prevent by the usual solutions as seen on the Mathworks site), because my panel was too big, I decided to only look at 700 days of data and to take averages of a number of firms to get N=5.

I discovered that the way I take these averages/medians heavily influences my regression results. If I first sort the firms on the size of the dependent variable and then make 5 groups to average them, I'll get less significant results than when I used another (default) sorting (almost alphabetic).

So what is the way to go in these kind of situations?

share|improve this question
    
Doesn't this rather fit to stats.stackexchange.com ? –  Richard Nov 5 '13 at 16:59
    
@Richard The OP might get a better answer from there (especially if she makes it more clear what she's trying to do). More generally, so-called big data issues are important for quants (and if she had phrased the question about unbalanced panel methods with huge financial datasets, then that might be a good question here). –  John Nov 5 '13 at 17:31
    
Maybe but the answer might depend on the use case. I believe it would be best to not average at all but if someone can offer some smart approach I'd be interested. –  Bob Jansen Nov 5 '13 at 17:32
    
Can't help directly, but can comment: 1 if you have major differences between mean and median based results - your data has too much skew. You need to cover your basics - are there outliers, is there a skew in population or in your grouping? Can you transform your data (e.g. power transform, or some ratio) to make it more normal? If 2 My Matlab usually can handle this size of data. Are you using 32 or 64 bit matlab? What kind of regression model do you use? –  onlyvix.blogspot.com Nov 5 '13 at 19:57
    
@Cindy88, Praise the Lord for bell-curve(s) is all that comes to mind (but then I focused on your profile not your question). Sorry, just could not help it. And back to the topic: I love to help but can you please provide more details. Similar to variance reduction techniques on the Monte Carlo side, much more details are needed what you exactly try to achieve in order to decide on the best algorithm optimization. Care to share the type of regression you try to run? –  Matt Wolf Nov 6 '13 at 14:43
show 2 more comments

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Browse other questions tagged or ask your own question.