Switching from C++ to R - limitations/applications

Question

I've only recently begun exploring and learning R (especially since Dirk recommended RStudio and a lot of people in here speak highly of R). I'm rather C(++) oriented, so it got me thinking - what are the limitations of R, in particular in terms of performance?

I'm trying to weigh the C++/Python/R alternatives for research and I'm considering if getting to know R well enough is worth the time investment.

Available packages look quite promising, but there are some issues in my mind that keep me at bay for the time being:

• How efficient is R when it comes to importing big datasets? And first of all, what's big in terms of R development? I used to process a couple hundred CSV files in C++ (around 0.5M values I suppose) and I remember it being merely acceptable. What can I expect from R here? Judging by Jeff's spectacular results I assume with a proper long-term solution (not CSV) I should be even able to switch to tick processing without hindrances. But what about ad-hoc data mangling? Is the difference in performance (compared to more low level implementations) that visible? Or is it just an urban legend?
• What are the options for GUI development? Let's say I would like to go further than research oriented analysis, like developing full blown UIs for investment analytics/trading etc. From what I found mentioned here and on StackOverflow, with proper bindings I am free to use Python's frameworks here and even further chain into Qt if such a need arises. But deploying such a beast must be a real nuisance. How do you cope with it?

In general I see R's flexibility allows me to mix and match it with a plethora of other languages (either way round - using low level additions in R or embed/invoke R in projects written in another language). That seems nice, but does it make sense (I mean like thinking about it from start/concept phase, not extending preexisting solutions)? Or is it better to stick with one-and-only language (insert whatever you like/have experience with)?

So to sum up: In what quant finance applications is R a (really) bad choice (or at least can be)?

Answer 1

R can be pretty slow, and it's very memory-hungry. My data set is only 8 GB or so, and I have a machine with 96 GB of RAM, and I'm always wrestling with R's memory management. Many of the model estimation functions capture a link to their environment, which means you can be keeping a pointer to each subset of the data that you're dealing with. SAS was much better at dealing with large-ish data sets, but R is much nicer to deal with. (This is in the context of mortgage prepayment and default modeling.)

Importing the data sets is pretty easy and fast enough, in my experience. It's the ballooning memory requirements for actually processing that data that's the problem.

Anything that isn't easily vectorizable seems like it would be a problem. P&L backtesting for a strategy that depends on the current portfolio state seems hard. If you're looking at the residual P&L from hedging a fixed-income portfolio, with full risk metrics, that's going to be hard.

I doubt many people would want to write a term structure model in R or a monte-carlo engine.

Even with all that, though, R is a very useful tool to have in your toolbox. But it's not exactly a computational powerhouse.

I don't know anything about the GUI options.

Answer 2

Don't have to switch -- it's not either / or after all.

Use either where it has an advantage: R for explorations, modeling, ... and C++ for industrial-strength and high-performance implementations (at a possible cost in terms of time to code).

And (with the obvious Rcpp plug coming) you can even combine both of them.

Answer 3

I am not an R advocate, but can witness that R is trivially very, very good at data analysis. It is essentially a LISP-like functional language domesticated enough to make you productive in one afternoon. It is unbeatable at getting data in your system, analyzing them, and producing high-quality output, be it latex reports or charts. I have used several languages (from SAS to Python), and none is close to R's productivity when it comes to advanced data analysis. It has an unparalleled suite of packages, with great redundancy: there are 4 packages on Kalman filter, and almost 10 packages on various regularized regression alone. Very often, the packages complement papers that are just being published, thus giving you access to the latest technology. It's not a problem to consume datasets of 100M rows or more, given sufficient memory. Those complaining about memory management in R should try MATLAB. Sure, it's slow, but consider this:

• Linear algebra is as fast as C++, and interfacing to LAPACK is a whole lot easier in R than in C++;
• there are APIs to specific DBs, key-value stores, and ODBC);
• many packages are optimized for speed and written in C or Fortran;
• For 99% of applications it is fast enough;
• For the remaining 1% use in which you are not bottlenecked by computation or data management, you can speed up things in C, C++, Fortran, Java.

I would maintain that well-coded R can be faster and more robust than poor C++ code. R can be used in production, although with some care, and not as the main language. It is definitely not suited for doing GUIs.

I thought the plug for Python was a bit off-topic, but I'll say that Python is without doubt among the most versatile and easy languages (I mean it as a compliment), and Cython is a great asset. Still, I believe languages, like people, should be judged based on what they're best at, not on what they're good enough at. I'd assert that R is best at data analysis, and that its syntax is slighly better than Python for this purpose. It'll be a while until Python has the domain-specific packages and visualization packages of R, and most importantly the people behind them. But I'll agree on one very important point: most quant hedge funds do relatively elementary data analysis, and Python+Numpy+Pandas is a sensible choice as single language.

Answer 4

As a disclaimer, I'm a noted advocate of using Python to build production systems for quant finance (old talk but: http://python.mirocommunity.org/video/1531/pycon-2010-python...). I've been very successful at doing it and largely as the result of my example many other quant shops have chosen the Python route to excellent results. The pandas Python library (http://pandas.sourceforge.net) is an open-source outgrowth of my proprietary work.

I see a another fan of my work has already posted here =)

My question is: why program in C++? I don't think anyone will argue it's an insanely low productivity language relative to Python or R. But Python and R are slow for iterative, procedural code. The near panacea for Pythonistas is to use Cython (http://cython.org) to develop C speed code but take maybe only 1.5-2x longer than writing Python code (to get all the type declarations right etc.). You can also directly call methods in C / C++ libraries using Cython, so it really is the best of both world in my experience.

I think in general that hybrid systems are best avoided if at all possible since debugging across "the bridge" is a thorny problem. You typically end up with more code than you planned in the higher-productivity language (e.g. R). I like Python because Python is good at all the things that R is not good at. Yes, Python's statistics libraries are very weak (though we're making progress in http://statsmodels.sourceforge.net) compared with CRAN, but in quant finance it turns out that 90% of the modeling and data analysis that you actually do isn't that statistically sophisticated. It's largely a relational data manipulation and time series processing problem (which pandas takes care of in spades-- has much better integrated data alignment features than just about anything in R, too).

Python is also excellent for building GUIs. I've used wxPython and PyQt and found that I could hack together a GUI in an afternoon that would have taken a week or more to do in Java or C++.

Answer 5

The greatest weakness and greatest strength of R is that it is not a strongly typed language. Therefore easy tasks in strongly typed languages such as re-factoring, auto-compiler checks, unit testing, etc. can be more difficult in R.

On the other hand, one can rapidly prototype in the R language. R is an interpreted language -- it will dynamically convert types. R is also an excellent tool for visualization and analysis (GGplot2 library). There is also a wonderful community of R developers that are creating new solutions for problems all the time.

The R Inferno is an essential read before you develop production code with R.

Answer 6

Getting something up and running quickly -- i.e. data manipulation and exploration are activities R are adept at, and there are a plethora of packages to help you. Flexibility and speed (of research) are R's primary strengths. I feel memory and computing power are less expensive than the thought cycles used to explore an idea.

If you're entering a production level arms race, obviously R is not the answer. However, I find R acceptable for production -- enough to plug it into an institutional order management system. As long as your investment strategy is based on predictive market analytics, I don't see a drastic need for speed.

Answer 7

Is it worth it to learn R?

Switching from C/C++ to R has increased my productivity and shrunk code line-counts for similar tasks by roughly an order of magnitude. To give a small flavor of why, consider one of the most common patterns of iteration over some collection, selection and action:

    declare iterator for collection
    for element in collection
        if (element meets condition)
            do something with element

In R this construct typically shrinks to:

    collection[condition] <- new_values

Where new_values is a vector expression as well. And this is not only limited to arrays/vectors.

In R, there's no need to declare iterator-variables and write loops to iterate over them because when acting on vectors (or higher dimensional data-structures) the loops are implied. Similarly, there's no need to use if because subsetting via [ ] implies a condition.

Add to this the fact that off-by-one bugs, outside-array and null dereferences are no longer an issue, that hashes (using $ list member references) are compact and part of the language, that visualizing data by turning it into a chart (see my avatar as an example of visualizing a 3-dimensional continuous-value table) is trivial, and you can start seeing the tremendous productivity jump.

Add ~4000 libraries in CRAN covering state-of-the-art statistics, data-mining and machine-learning libraries mean you often don't need to write code at all, just use what's out there. Many of these libraries (where it matters) are already written in a compiled language (C or Fortran) so efficiency is largely taken care of.

Are 1 million values in R an issue?

The OP question mentions working on data-sets of 0.5 million values and whether this may be an issue with R. Let's run a quick check:

R> million.random.values.array <- rnorm(1e6, mean=5, sd=2)
R> mean(million.random.values.array)
[1] 5.000186
R> sd(million.random.values.array)
[1] 2.000551

The above 3 lines of R code complete instantaneously on my desktop. Memory consumption of all of R loaded with the session shows 120 MB virtual and 37 MB resident, so the answer is that ~1M element size of data-sets shouldn't generally be an issue. I've used 1B-items data-set sizes on big memory 64-bit machines. Reading data from a database using RODBC for example, or from a flat data file (csv, tsv, text), using read.table/read.csv or similar, instead, is trivial as well.

Inherent inefficiencies in R & possible remedies

Having said that, it is easy to write inefficient R code, both in terms of memory and speed. The most common cause I've seen is building a data-frame iteratively (adding one column at a time, using cbind() or similar) because the copy-arguments on every call makes this $O(n)$ process become an $O(n^2)$ process. Similarly, passing big data-structures as function arguments, like a full data-frame, when you only want to pass one or a subset of the columns, has its (pass by value copy) undesired cost. If the last issue is your #1 slowness cause, you want to look at data.table library which allows passing args by reference using :=, and learning about the <<- operator.

Reading Patrick Burns "The R inferno" (124 pages, available for free as PDF) is an excellent time investment as mentioned by Quant Guy above if you're serious about learning R and avoiding the pitfalls.

Also seconding Dirk's comment that by using Rcpp it is possible to avoid the above pitfalls where it matters and write the low level critical loop code in C/C++ where needed.

Bottom line:

Yes, time spent learning R is well worth it, but there's no one hammer to fit all needs. Use the tool that's best for the job.

Answer 8

If I were starting fresh, I would choose

• IPython
• Numpy+Scipy
• Pandas

You can get a one-click install from Enthought, http://enthought.com/. [2015 edit: anaconda is a better bet these days]

Python is hot on the heels of R as an exploratory data analysis solution for finance, and it's a heck of a lot more fun to write code in (imho:). Plus, python tends to play well with others within a larger software ecosystem.

Answer 9

What about customizing KNIME (Open source aswell) for this particular problem? I am no expert, just my two cent. KNIME comes with Weka(I dont mean complete Weka, but a basic machine learning functions) & R integration aswell(Importing/Exporting R codes are easy), Text Mining, Neural Nets etc.

I reckon, KINME(www.knime.org) with it's enterprise edition(EE) can do wonders, but it costs 2000euros.:(

For BigData R seems to not perform well. Instead use Hadoop or Apache Mahout or KNIME. My bets are on KNIME. Currently I am working on a similar trading system as a personal project using these tools.

Apologies if I am not so coherent, my first language is not english.

Thanks Roshan Daniel [email protected] Bangalore

Answer 10

What are the limitations of R?

The limitations of the R kernel are well-documented in Sridharan and Patel (2014):

• Spends more than 85% time in processor and memory stalls
• High rate of cache misses (about 90% on linear regression and k-means tasks)
• Triggers garbage collection very frequently
• Creates a large number of unnecessary temporary objects, resorting to swap space quickly even for datasets that should have fit main memory

C++ is not necessarily "faster than R". It makes limited sense to talk about faster languages, only faster implementations.

C++ overcomes the above issues and gives you more room for optimization by giving you deterministic control over low level facilities that allow you to manually manage memory and access patterns. That's all. Chances are, a naive C++ implementation of native R routines will be slower than using the R implementation.

What quant finance applications is R a (really) bad choice (or at least can be)?

The two most obvious things are:

• When you need stable, 24/7 operation of the program without program crashes
• When you have an application-specific task that needs to be repeated very often, e.g. feed handling, order routing, volatility modeling etc.

Answer 11

For me, the questions are

1. who is using R? (large companies?)
2. Who is hiring R programmers and does it look like a language of choice in a field (finance, medical, automotive) with a strong future?

I've been developing in C/C++ for almost 17 years now and it has easily kept me gainfully employed. I started out writing small DOS applications, did a fair amount of CRUD GUI Windows development, and I'm currently in the embedded field doing work on ARM hardware.

For me, I see a very bright future in embedded. ARM chips are now multi-core and this hardware, in my opinion, is showing up in everything from automobiles to TV sets to medical equipment.

I recently did an interview for Bloomberg (what a joke) and the whole interview was about testing my knowledge of C++.

I don't know much about R but what I can say, in looking back at my 17 years in this field and in working for some very large companies (Fotune-5) and very small companies that the languages I see the most often are C/C++, J2EE and C# as the top languages. And under these I see utility type languages in use such as Perl; mostly Perl in fact.

My compass check might be to do a DICE or MONSTER search and see who is looking for R programmers.

Answer 12

There are three languages : c++, R , python.

Performance wise : c++ > python >> R

Packages available : R >> python >> c++

Ease of manual exploratory analysis : R > python >> c++

Ease of adding GUI like features and interactivity: python > c++ >> R

Ease of programming small projects : R > python >> c++

Ease of programming large projects : c++ > python >> R

Scalability in performance : c++ > python >> R

Ease of learning : python > R > c++

Answer 13

With regard to memory scalability and parallel processing, it is opportune to update this discussion to 2016 taking into account the existence of the Microsoft R and Spark R for Apache.

"SparkR, an R package initially developed at the AMPLab, provides an R frontend to Apache Spark and using Spark’s distributed computation engine allows us to run large scale data analysis from the R shell." (by Shivaram Venkataraman Posted in ENGINEERING BLOG June 9, 2015)

With reference to Microsoft-R, In this article of August 2016, the author presents a set of solutions for open and corporate environments. For the enterprise they offer Disk scalability Operates on bigger volumes & factors, Full parallel threading & processing.

In her words

By using and extending open source R, Microsoft R Server is fully compatible with R scripts, functions and CRAN packages, to analyze data at enterprise scale. We also address the in-memory limitations of open source R by adding parallel and chunked processing of data in Microsoft R Server, enabling users to run analytics on data much bigger than what fits in main memory. And since R Server is built on top of Microsoft R Open, you can use any open source R packages to build your analytics.

Microsoft R Server delivers enterprise class performance and scalability for your R-based applications with libraries that allow you to write once and deploy across multiple platforms with minimal effort, whether on-premises or in the cloud.