2
$\begingroup$

I'm trying to do a regression on my panel data. Say I have T=3500 days of data and N=125 firms.

Since Matlab get's major memory issues (which I try to prevent by the usual solutions as seen on the Mathworks site), because my panel was too big, I decided to only look at 700 days of data and to take averages of a number of firms to get N=5.

I discovered that the way I take these averages/medians heavily influences my regression results. If I first sort the firms on the size of the dependent variable and then make 5 groups to average them, I'll get less significant results than when I used another (default) sorting (almost alphabetic).

So what is the way to go in these kind of situations?

Improve this question
$\endgroup$
7
  • $\begingroup$ Doesn't this rather fit to stats.stackexchange.com ? $\endgroup$
    – Richi Wa
    Nov 5, 2013 at 16:59
  • $\begingroup$ @Richard The OP might get a better answer from there (especially if she makes it more clear what she's trying to do). More generally, so-called big data issues are important for quants (and if she had phrased the question about unbalanced panel methods with huge financial datasets, then that might be a good question here). $\endgroup$
    – John
    Nov 5, 2013 at 17:31
  • $\begingroup$ Maybe but the answer might depend on the use case. I believe it would be best to not average at all but if someone can offer some smart approach I'd be interested. $\endgroup$
    – Bob Jansen ♦
    Nov 5, 2013 at 17:32
  • $\begingroup$ Can't help directly, but can comment: 1 if you have major differences between mean and median based results - your data has too much skew. You need to cover your basics - are there outliers, is there a skew in population or in your grouping? Can you transform your data (e.g. power transform, or some ratio) to make it more normal? If 2 My Matlab usually can handle this size of data. Are you using 32 or 64 bit matlab? What kind of regression model do you use? $\endgroup$
    – onlyvix.blogspot.com
    Nov 5, 2013 at 19:57
  • $\begingroup$ @Cindy88, Praise the Lord for bell-curve(s) is all that comes to mind (but then I focused on your profile not your question). Sorry, just could not help it. And back to the topic: I love to help but can you please provide more details. Similar to variance reduction techniques on the Monte Carlo side, much more details are needed what you exactly try to achieve in order to decide on the best algorithm optimization. Care to share the type of regression you try to run? $\endgroup$
    – Matt Wolf
    Nov 6, 2013 at 14:43

0

Reset to default

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Browse other questions tagged or ask your own question.