I've only recently begun exploring and learning R (especially since Dirk recommended RStudio and a lot of people in here speak highly of R). I'm rather C(++) oriented, so it got me thinking - what are the limitations of R, in particular in terms of performance?

I'm trying to weigh the C++/Python/R alternatives for research and I'm considering if getting to know R well enough is worth the time investment.

Available packages look quite promising, but there are some issues in my mind that keep me at bay for the time being:

  • How efficient is R when it comes to importing big datasets? And first of all, what's big in terms of R development? I used to process a couple hundred CSV files in C++ (around 0.5M values I suppose) and I remember it being merely acceptable. What can I expect from R here? Judging by Jeff's spectacular results I assume with a proper long-term solution (not CSV) I should be even able to switch to tick processing without hindrances. But what about ad-hoc data mangling? Is the difference in performance (compared to more low level implementations) that visible? Or is it just an urban legend?
  • What are the options for GUI development? Let's say I would like to go further than research oriented analysis, like developing full blown UIs for investment analytics/trading etc. From what I found mentioned here and on StackOverflow, with proper bindings I am free to use Python's frameworks here and even further chain into Qt if such a need arises. But deploying such a beast must be a real nuisance. How do you cope with it?

In general I see R's flexibility allows me to mix and match it with a plethora of other languages (either way round - using low level additions in R or embed/invoke R in projects written in another language). That seems nice, but does it make sense (I mean like thinking about it from start/concept phase, not extending preexisting solutions)? Or is it better to stick with one-and-only language (insert whatever you like/have experience with)?

So to sum up: In what quant finance applications is R a (really) bad choice (or at least can be)?

  • $\begingroup$ I suppose this would fit SO (or maybe stats.SE?) as well, but I assumed new questions are of more use in here. I hope it's sufficiently on topic here though. $\endgroup$ Commented Mar 15, 2011 at 18:25
  • $\begingroup$ I work with txt files > 300M in R with no problem. $\endgroup$ Commented Mar 16, 2011 at 7:28
  • $\begingroup$ you can develop pyhton gui that calls R for statistical calculus... $\endgroup$ Commented Mar 16, 2011 at 7:30
  • 1
    $\begingroup$ @Karol Piczac since i have migrated to R-Evolution i have no more problem with big data file, have a look at this (revolutionanalytics.com/products/revolution-enterprise.php) $\endgroup$
    – Beer4All
    Commented Aug 5, 2011 at 10:40
  • 3
    $\begingroup$ This question has hit the front page of Hacker News. $\endgroup$ Commented Oct 5, 2011 at 22:24

13 Answers 13


R can be pretty slow, and it's very memory-hungry. My data set is only 8 GB or so, and I have a machine with 96 GB of RAM, and I'm always wrestling with R's memory management. Many of the model estimation functions capture a link to their environment, which means you can be keeping a pointer to each subset of the data that you're dealing with. SAS was much better at dealing with large-ish data sets, but R is much nicer to deal with. (This is in the context of mortgage prepayment and default modeling.)

Importing the data sets is pretty easy and fast enough, in my experience. It's the ballooning memory requirements for actually processing that data that's the problem.

Anything that isn't easily vectorizable seems like it would be a problem. P&L backtesting for a strategy that depends on the current portfolio state seems hard. If you're looking at the residual P&L from hedging a fixed-income portfolio, with full risk metrics, that's going to be hard.

I doubt many people would want to write a term structure model in R or a monte-carlo engine.

Even with all that, though, R is a very useful tool to have in your toolbox. But it's not exactly a computational powerhouse.

I don't know anything about the GUI options.

  • $\begingroup$ Thanks for bringing this up. I haven't thought that memory management could be such an issue. And great example selection. Just what gets into my field of interests. ;-) At least I shall know what not to expect from R. But as you say, it's probably nice to at least get acquainted with it, so as to know when it may come in handy. Forgive my question blatantly arising from my personal interests and lack of knowledge in the matter at the same time, but how appropriate is R in machine learning applications and particularly Bayesian network inference? $\endgroup$ Commented Mar 15, 2011 at 22:23
  • 1
    $\begingroup$ @Karol Piczak: R is certainly used for machine learning applications, but I don't know about Bayesian network inference in particular. $\endgroup$ Commented Mar 16, 2011 at 12:06
  • 2
    $\begingroup$ Well, Bayesian statistics is a the prime example for mixing of C++ (for speed) with R (for ease of analysis), see the Bayesian Stats Task View. Also, packages like ff and bigmemory deal with large memory, see the HPC Task View. These issues can be addressed quite well in a hybrid manner as I indicated in my answer above. $\endgroup$ Commented Mar 16, 2011 at 14:42
  • 1
    $\begingroup$ I have spent some time during the last weeks to infer parameters (calibrate) of a vol-sto model (hence using the classical Baysian inference theory). I used C# interfaced with R, no memory problem, even with a parallelized implementation.. $\endgroup$
    – Beer4All
    Commented Aug 5, 2011 at 11:04
  • 1
    $\begingroup$ SAS appears faster b/c its data step encourages you to processes data one line at time, whereas R encourages loading everything into memory-- however that isn't the only way to do it. My (biological) datasets are 300gb+ each, and I have no particular problem processing them with a Perl->R pipeline with intermediate data placed in sqlite databases, never loading more than 1gb into memory. $\endgroup$
    – user1481
    Commented Oct 5, 2011 at 19:21

Don't have to switch -- it's not either / or after all.

Use either where it has an advantage: R for explorations, modeling, ... and C++ for industrial-strength and high-performance implementations (at a possible cost in terms of time to code).

And (with the obvious Rcpp plug coming) you can even combine both of them.

  • $\begingroup$ Yeah, I suppose so. But this part "at a possible cost in terms of time to code" is exactly my problem. I indeed like C++, but at times it's really hard to get something up and running quickly. So I'm looking for a tool that would allow me to crash test my concepts more rapidly and see if they are worth implementing in, as you name it, industrial-strength at all. But I wouldn't like to shoot myself in the foot either and get locked in a point where it's not appropriate for production use nor preliminary research. $\endgroup$ Commented Mar 15, 2011 at 22:36

I am not an R advocate, but can witness that R is trivially very, very good at data analysis. It is essentially a LISP-like functional language domesticated enough to make you productive in one afternoon. It is unbeatable at getting data in your system, analyzing them, and producing high-quality output, be it latex reports or charts. I have used several languages (from SAS to Python), and none is close to R's productivity when it comes to advanced data analysis. It has an unparalleled suite of packages, with great redundancy: there are 4 packages on Kalman filter, and almost 10 packages on various regularized regression alone. Very often, the packages complement papers that are just being published, thus giving you access to the latest technology. It's not a problem to consume datasets of 100M rows or more, given sufficient memory. Those complaining about memory management in R should try MATLAB. Sure, it's slow, but consider this:

  • Linear algebra is as fast as C++, and interfacing to LAPACK is a whole lot easier in R than in C++;
  • there are APIs to specific DBs, key-value stores, and ODBC);
  • many packages are optimized for speed and written in C or Fortran;
  • For 99% of applications it is fast enough;
  • For the remaining 1% use in which you are not bottlenecked by computation or data management, you can speed up things in C, C++, Fortran, Java.

I would maintain that well-coded R can be faster and more robust than poor C++ code. R can be used in production, although with some care, and not as the main language. It is definitely not suited for doing GUIs.

I thought the plug for Python was a bit off-topic, but I'll say that Python is without doubt among the most versatile and easy languages (I mean it as a compliment), and Cython is a great asset. Still, I believe languages, like people, should be judged based on what they're best at, not on what they're good enough at. I'd assert that R is best at data analysis, and that its syntax is slighly better than Python for this purpose. It'll be a while until Python has the domain-specific packages and visualization packages of R, and most importantly the people behind them. But I'll agree on one very important point: most quant hedge funds do relatively elementary data analysis, and Python+Numpy+Pandas is a sensible choice as single language.

  • 2
    $\begingroup$ In fairness, the OP mentioned weighing "C++/Python/R" alternatives so it wasn't that off-topic. $\endgroup$
    – wesm
    Commented Oct 10, 2011 at 2:58

As a disclaimer, I'm a noted advocate of using Python to build production systems for quant finance (old talk but: http://python.mirocommunity.org/video/1531/pycon-2010-python...). I've been very successful at doing it and largely as the result of my example many other quant shops have chosen the Python route to excellent results. The pandas Python library (http://pandas.sourceforge.net) is an open-source outgrowth of my proprietary work.

I see a another fan of my work has already posted here =)

My question is: why program in C++? I don't think anyone will argue it's an insanely low productivity language relative to Python or R. But Python and R are slow for iterative, procedural code. The near panacea for Pythonistas is to use Cython (http://cython.org) to develop C speed code but take maybe only 1.5-2x longer than writing Python code (to get all the type declarations right etc.). You can also directly call methods in C / C++ libraries using Cython, so it really is the best of both world in my experience.

I think in general that hybrid systems are best avoided if at all possible since debugging across "the bridge" is a thorny problem. You typically end up with more code than you planned in the higher-productivity language (e.g. R). I like Python because Python is good at all the things that R is not good at. Yes, Python's statistics libraries are very weak (though we're making progress in http://statsmodels.sourceforge.net) compared with CRAN, but in quant finance it turns out that 90% of the modeling and data analysis that you actually do isn't that statistically sophisticated. It's largely a relational data manipulation and time series processing problem (which pandas takes care of in spades-- has much better integrated data alignment features than just about anything in R, too).

Python is also excellent for building GUIs. I've used wxPython and PyQt and found that I could hack together a GUI in an afternoon that would have taken a week or more to do in Java or C++.

  • $\begingroup$ Python/Pandas has a great mix of ease in development and ability to handle and manipulate large datasets. Python will also be much more flexible for low level integration than R, but the ability to easily build a GUI and manipulate datasets in a single language is not to be underestimated. $\endgroup$
    – rhaskett
    Commented Apr 9, 2014 at 22:58
  • $\begingroup$ The link to the PyCon talk is dead, but I found it at the following URL: pyvideo.org/video/305/python-in-quantitative-finance-158 $\endgroup$ Commented Jan 8, 2015 at 10:45

The greatest weakness and greatest strength of R is that it is not a strongly typed language. Therefore easy tasks in strongly typed languages such as re-factoring, auto-compiler checks, unit testing, etc. can be more difficult in R.

On the other hand, one can rapidly prototype in the R language. R is an interpreted language -- it will dynamically convert types. R is also an excellent tool for visualization and analysis (GGplot2 library). There is also a wonderful community of R developers that are creating new solutions for problems all the time.

The R Inferno is an essential read before you develop production code with R.


Getting something up and running quickly -- i.e. data manipulation and exploration are activities R are adept at, and there are a plethora of packages to help you. Flexibility and speed (of research) are R's primary strengths. I feel memory and computing power are less expensive than the thought cycles used to explore an idea.

If you're entering a production level arms race, obviously R is not the answer. However, I find R acceptable for production -- enough to plug it into an institutional order management system. As long as your investment strategy is based on predictive market analytics, I don't see a drastic need for speed.


Is it worth it to learn R?

Switching from C/C++ to R has increased my productivity and shrunk code line-counts for similar tasks by roughly an order of magnitude. To give a small flavor of why, consider one of the most common patterns of iteration over some collection, selection and action:

    declare iterator for collection
    for element in collection
        if (element meets condition)
            do something with element

In R this construct typically shrinks to:

    collection[condition] <- new_values

Where new_values is a vector expression as well. And this is not only limited to arrays/vectors.

In R, there's no need to declare iterator-variables and write loops to iterate over them because when acting on vectors (or higher dimensional data-structures) the loops are implied. Similarly, there's no need to use if because subsetting via [ ] implies a condition.

Add to this the fact that off-by-one bugs, outside-array and null dereferences are no longer an issue, that hashes (using $ list member references) are compact and part of the language, that visualizing data by turning it into a chart (see my avatar as an example of visualizing a 3-dimensional continuous-value table) is trivial, and you can start seeing the tremendous productivity jump.

Add ~4000 libraries in CRAN covering state-of-the-art statistics, data-mining and machine-learning libraries mean you often don't need to write code at all, just use what's out there. Many of these libraries (where it matters) are already written in a compiled language (C or Fortran) so efficiency is largely taken care of.

Are 1 million values in R an issue?

The OP question mentions working on data-sets of 0.5 million values and whether this may be an issue with R. Let's run a quick check:

R> million.random.values.array <- rnorm(1e6, mean=5, sd=2)
R> mean(million.random.values.array)
[1] 5.000186
R> sd(million.random.values.array)
[1] 2.000551

The above 3 lines of R code complete instantaneously on my desktop. Memory consumption of all of R loaded with the session shows 120 MB virtual and 37 MB resident, so the answer is that ~1M element size of data-sets shouldn't generally be an issue. I've used 1B-items data-set sizes on big memory 64-bit machines. Reading data from a database using RODBC for example, or from a flat data file (csv, tsv, text), using read.table/read.csv or similar, instead, is trivial as well.

Inherent inefficiencies in R & possible remedies

Having said that, it is easy to write inefficient R code, both in terms of memory and speed. The most common cause I've seen is building a data-frame iteratively (adding one column at a time, using cbind() or similar) because the copy-arguments on every call makes this $O(n)$ process become an $O(n^2)$ process. Similarly, passing big data-structures as function arguments, like a full data-frame, when you only want to pass one or a subset of the columns, has its (pass by value copy) undesired cost. If the last issue is your #1 slowness cause, you want to look at data.table library which allows passing args by reference using :=, and learning about the <<- operator.

Reading Patrick Burns "The R inferno" (124 pages, available for free as PDF) is an excellent time investment as mentioned by Quant Guy above if you're serious about learning R and avoiding the pitfalls.

Also seconding Dirk's comment that by using Rcpp it is possible to avoid the above pitfalls where it matters and write the low level critical loop code in C/C++ where needed.

Bottom line:

Yes, time spent learning R is well worth it, but there's no one hammer to fit all needs. Use the tool that's best for the job.

  • $\begingroup$ Very complete answer! $\endgroup$ Commented Dec 30, 2013 at 19:37

If I were starting fresh, I would choose

You can get a one-click install from Enthought, http://enthought.com/. [2015 edit: anaconda is a better bet these days]

Python is hot on the heels of R as an exploratory data analysis solution for finance, and it's a heck of a lot more fun to write code in (imho:). Plus, python tends to play well with others within a larger software ecosystem.


What about customizing KNIME (Open source aswell) for this particular problem? I am no expert, just my two cent. KNIME comes with Weka(I dont mean complete Weka, but a basic machine learning functions) & R integration aswell(Importing/Exporting R codes are easy), Text Mining, Neural Nets etc.

I reckon, KINME(www.knime.org) with it's enterprise edition(EE) can do wonders, but it costs 2000euros.:(

For BigData R seems to not perform well. Instead use Hadoop or Apache Mahout or KNIME. My bets are on KNIME. Currently I am working on a similar trading system as a personal project using these tools.

Apologies if I am not so coherent, my first language is not english.

Thanks Roshan Daniel [email protected] Bangalore


What are the limitations of R?

The limitations of the R kernel are well-documented in Sridharan and Patel (2014):

  • Spends more than 85% time in processor and memory stalls
  • High rate of cache misses (about 90% on linear regression and k-means tasks)
  • Triggers garbage collection very frequently
  • Creates a large number of unnecessary temporary objects, resorting to swap space quickly even for datasets that should have fit main memory

C++ is not necessarily "faster than R". It makes limited sense to talk about faster languages, only faster implementations.

C++ overcomes the above issues and gives you more room for optimization by giving you deterministic control over low level facilities that allow you to manually manage memory and access patterns. That's all. Chances are, a naive C++ implementation of native R routines will be slower than using the R implementation.

What quant finance applications is R a (really) bad choice (or at least can be)?

The two most obvious things are:

  • When you need stable, 24/7 operation of the program without program crashes
  • When you have an application-specific task that needs to be repeated very often, e.g. feed handling, order routing, volatility modeling etc.

For me, the questions are

  1. who is using R? (large companies?)
  2. Who is hiring R programmers and does it look like a language of choice in a field (finance, medical, automotive) with a strong future?

I've been developing in C/C++ for almost 17 years now and it has easily kept me gainfully employed. I started out writing small DOS applications, did a fair amount of CRUD GUI Windows development, and I'm currently in the embedded field doing work on ARM hardware.

For me, I see a very bright future in embedded. ARM chips are now multi-core and this hardware, in my opinion, is showing up in everything from automobiles to TV sets to medical equipment.

I recently did an interview for Bloomberg (what a joke) and the whole interview was about testing my knowledge of C++.

I don't know much about R but what I can say, in looking back at my 17 years in this field and in working for some very large companies (Fotune-5) and very small companies that the languages I see the most often are C/C++, J2EE and C# as the top languages. And under these I see utility type languages in use such as Perl; mostly Perl in fact.

My compass check might be to do a DICE or MONSTER search and see who is looking for R programmers.

  • $\begingroup$ This is a quant finance site. People often have to answer questions by applying statistical tests. For example, to find out if a time series is stationary, one applies the Dickey-Fuller test [ en.wikipedia.org/wiki/Dickey%E2%80%93Fuller_test ]. We use other tests to determine if a random variable is gaussian, or if two time series are cointegrated, etc. These tests are readily available in R, Python, and MATLAB, but are not in C++ or Java. $\endgroup$
    – wsw
    Commented Oct 18, 2015 at 6:01
  • $\begingroup$ @wsw you may have a good point, IDK. What I hear you saying is that some stats library is available in R and others but not C++, the language. Perhaps you're conflating language and libraries as I don't think the Python you gave as counter point example has stats, as a language, either. A quick google search found many stats C++ libraries. Side bar: At least as far has my experience has shown, you can't always depend on the work of others either. (faster, bug free, etc) $\endgroup$
    – Eric
    Commented Oct 20, 2015 at 13:47

There are three languages : c++, R , python.

Performance wise : c++ > python >> R

Packages available : R >> python >> c++

Ease of manual exploratory analysis : R > python >> c++

Ease of adding GUI like features and interactivity: python > c++ >> R

Ease of programming small projects : R > python >> c++

Ease of programming large projects : c++ > python >> R

Scalability in performance : c++ > python >> R

Ease of learning : python > R > c++

  • 2
    $\begingroup$ Ease of adding GUI like features and interactivity: python > c++ >> R.... Are you aware of shiny and rCharts, among others, for R? Also, would add Exploration(Visualization): ? > ? > ? $\endgroup$ Commented Jun 14, 2014 at 16:29
  • 1
    $\begingroup$ One word for R with data interactivity and ease of adding GUIs: Shiny. Very fast application development, very powerful for building ideas. Those inequalities should be flipped. $\endgroup$ Commented Jul 8, 2014 at 7:16
  • $\begingroup$ I'll echo the previous commenters - with the advent of Shiny, R is now the most convenient language for interactive data visualization. Can your language do all these things in a couple dozen lines of very readable code? $\endgroup$
    – Paul
    Commented Jul 26, 2015 at 13:25

With regard to memory scalability and parallel processing, it is opportune to update this discussion to 2016 taking into account the existence of the Microsoft R and Spark R for Apache.

"SparkR, an R package initially developed at the AMPLab, provides an R frontend to Apache Spark and using Spark’s distributed computation engine allows us to run large scale data analysis from the R shell." (by Shivaram Venkataraman Posted in ENGINEERING BLOG June 9, 2015)

With reference to Microsoft-R, In this article of August 2016, the author presents a set of solutions for open and corporate environments. For the enterprise they offer Disk scalability Operates on bigger volumes & factors, Full parallel threading & processing.

enter image description here

In her words

By using and extending open source R, Microsoft R Server is fully compatible with R scripts, functions and CRAN packages, to analyze data at enterprise scale. We also address the in-memory limitations of open source R by adding parallel and chunked processing of data in Microsoft R Server, enabling users to run analytics on data much bigger than what fits in main memory. And since R Server is built on top of Microsoft R Open, you can use any open source R packages to build your analytics.

Microsoft R Server delivers enterprise class performance and scalability for your R-based applications with libraries that allow you to write once and deploy across multiple platforms with minimal effort, whether on-premises or in the cloud.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.