I’m working in the area of Data Mining and have come up with the following idea for my Masters project.The text may not be the best structured but it’s a working draft to give you a quick idea.

Basic Hypothesis,

  • Can a combination of (Weather Data + Agricultural Data + Social Media (twitter, etc) data + other relevant data) be used to aid an investor to buy futures of product / commodity ‘X’
  • I plan to focus on testing the hypothesis on ‘Soy Bean Futures’. The core idea is to test the approach, even if I fail, its fine. My method / approach must be correct.


  • potential target audience could be investor, govt agencies or agricultural industries


  • Focus on Soy Bean Futures in USA (Worlds largest Soy producer) to narrow down my problem scope
  • more specifically on State of Illinois (leads Soy production in US) to zoom in even further


  • Understand how the model for pricing of Futures works
  • Find Historical trading data on Soy Futures in Illinois [from Quandl?] I still don’t know how I will match Soy Future trading data to where it was produced so that’s an issue, I think
  • Weather [temp, humidity, sea pressure, etc] & Agricultural [yields, farm sizes etc] data is easy to get & analyze
  • Do some number crunching / data mining to test “IF Weather in Illinois affects Soy yields/ production which is turn affects Soy Futures prices” ; I still need to refine the technique but its a rough idea

Your Input,

  • what do you think about the whole idea? totally nuts? not realistic? I need to be Math God to figure this out?
  • if you think, that this is even remotely feasible, what are my must do’s, must NOT do’s?
  • is my technique fully flawed? what am I missing? under-estimating? How can I improve my technique?
  • $\begingroup$ Didn't want to put it in my answer but are you sure that's enough for a Masters project? I might be blinded by my daily job but this seems more like a few hours of work (given the right tools and just moderate computer power). Or are you planning on developing some kind of production-quality trading strategy? $\endgroup$ – hroptatyr Sep 18 '14 at 6:12
  • $\begingroup$ thank you for your reply. It"s a 3 month project (for a 1 yr masters program) no plans for production strat but just intrigued but this topic...i used weather data for electricity prediction in my previous masters and it was really explaining a lot of vairance.. also I have a more of Comp Sci background...with absolutely no knowledge of finance,agriculture , getting the right data, cleaning it - takes a lot of time, writing a thesis etc so need to build that up too if i am to get anything done $\endgroup$ – user3491422 Sep 20 '14 at 9:12
  • $\begingroup$ Even if you put together the knowledge to tackle such a project, my experience suggests there is not enough data points to infer the effects of external information on agricultural commodities. The first thought that comes to mind is to increase the sampling rate. But the real issue is that the responses to say weather changes are observed in long time frames, so frequent sampling doesn't bring in additional information. $\endgroup$ – Cowboy Trader Sep 22 '14 at 6:34
  • $\begingroup$ where did you get your weather data? $\endgroup$ – Dez Udezue Nov 23 '15 at 21:08
  • $\begingroup$ I'm thinking about doing something similar for my master thesis. How did it go for you? Just reading the comments I realized I'm a bit naive or just very early in my research and the search for my topic. Thanks a lot for any experience you can share. $\endgroup$ – Lukas R. Feb 19 '18 at 22:02

Rather than "data mining", I'd try to have a more structured approach to the selection of variables / factors: the key drivers in ag prices are, as always in economics, supply and demand.

Supply is, as you say, determined by planted acreage and yields, which in turn depend on weather and other factors (for example fertilization - can't run heavy on Nitrogen for many years on a row). I'd start from the USDA website.

From what I read in your question, demand and its drivers are missing. Luckily, demand is relatively stable when it comes to food consumption. However, corn ethanol in the past years was a big swing factor resulting in a corn madness.

A more structured approach would help you understand and interpret your results, imo. I think it is a great project, but a big project. You could theoretically invest your entire career on this subject.

  • $\begingroup$ really appreciate your input, you hit the right note on the Supply & Demand,thats an aspect I've been struggling with...as of now, I dont have a concrete idea on what level (day, week,month) my predictions are going to be in & also for which Soy Futures [S1, S2,...S10]. Acreage, yields are on a yearly level, so I"m not sure if I can just divide yield/12 & use it in my model...same goes for demand side variables I was considering to use crude oil spot prices, corn ethanol prices in the model for 1st stage, I'm planning to use only Illinois Agri + weather data & see how much var it explains $\endgroup$ – user3491422 Sep 20 '14 at 9:21
  • $\begingroup$ No, you can't divide yield/12. Your model needs to take into account planting seasons and crop rotation. F.ex, corn is planted and harvested only once a year, while in certain crop rotation schemes wheat could also serve as an overwintering cover crop. Cash crops acreages and yields are monitored closely, even surveyed before planting by the USDA: this is huge news. Crop rotation schemes are very important: farmers need to minimize the depletion of N,P and K nutrients. This means you have to alternate different types of crops over the time, otherwise yields might get heavily impacted. $\endgroup$ – pincopallino Sep 20 '14 at 11:48

I think this is a good idea; instead of if I'd ask: to what degree though.

Futures prices reflect a lot of factors, so naturally you will only see some correlation which will change over time and even over different expiries (e.g. due to different liquidity in the back months).

My advice therefore: Make sure you understand how the markets work because you will have to explain a lot of, say, oddities on the way. Other than that I think the approach is sound.


Your Answer

By clicking "Post Your Answer", you acknowledge that you have read our updated terms of service, privacy policy and cookie policy, and that your continued use of the website is subject to these policies.

Not the answer you're looking for? Browse other questions tagged or ask your own question.