# Is R being replaced by Python at quant desks?

I know the title sounds a little extreme but I wonder whether R is phased out by a lot of quant desks at sell side banks as well as hedge funds in favor of Python. I get the impression that with improvements in Pandas, Numpy and other Python packages functionality in Python is drastically improving in order to meaningfully mine data and model time series. I have also seen quite impressive implementations through Python to parallelize code and fan out computations to several servers/machines. I know some packages in R are capable of that too but I just sense that the current momentum favors Python.

I need to make a decision regarding architecture of a subset of my modeling framework myself and need some input what the current sentiment is by other quants.

I also have to admit that my initial reservations regarding performance via Python are mostly outdated because some of the packages make heavy use of C implementations under the hood and I have seen implementations that clearly outperform even efficiently written, compiled OOP language code.

Can you please comment on what you are using? I am not asking for opinions whether you think one is better or worse for below tasks but specifically why you use R or Python and whether you even place them in the same category to accomplish, among others, the following tasks:

• acquire, store, maintain, read, clean time series
• perform basic statistics on time series, advanced statistical models such as multivariate regression analyses,...
• performing mathematical computations (fourier transforms, PDE solver, PCA, ...)
• visualization of data (static and dynamic)
• pricing derivatives (application of pricing models such as interest rate models)
• interconnectivity (with Excel, servers, UI, ...)
• (Added Jan 2016): Ability to design, implement, and train deep learning networks.

EDIT I thought the following link might add more value though its slightly dated [2013] (for some obscure reason that discussion was also closed...): https://softwareengineering.stackexchange.com/questions/181342/r-vs-python-for-data-analysis

You can also search for several posts on the r-bloggers website that address computational efficiency between R and Python packages. As was addressed in some of the answers, one aspect is data pruning, the preparation and setup of input data. Another part of the equation is the computational efficiency when actually performing statistical and mathematical computations.

Update (Jan 2016)

I wanted to provide an update to this question now that AI/Deep Learning networks are very actively pursued at banks and hedge funds. I have spent a good amount of time on delving into deep learning and performed experiments and worked with libraries such as Theano, Torch, and Caffe. What stood out from my own work and conversations with others was that a lot of those libraries are used via Python and that most of the researchers in this space do not use R in this particular field. Now, this still constitutes a small part of quant work being performed in financial services but I still wanted to point it out as it directly touches on the question I asked. I added this aspect of quant research to reflect current trends.

• I am not sure but definitively there are some adventages for python in regards to the development of packages in some areas. – Barnaby May 19 '15 at 8:26
• You are a highly respected member of this community but I am getting a worse and worse feeling about this question. One of the examples of questions that we don't want on this site is "What programming language should I use?" (quant.stackexchange.com/help/on-topic). When you look at the discussions in the comments you can see why: They are getting more and more contentious - and you seem to have made up your mind anyway. I think if somebody with less rep had asked this question it would have got closed right away. I think best would be to close this question. Do you see my point? – vonjd May 21 '15 at 15:35
• @vonjd, I raised this on meta, thanks for suggesting this: meta.quant.stackexchange.com/questions/1452/… – Matt May 22 '15 at 5:04
• @vonjd, no I have not yet made a decision. But I am much better informed thanks to some of the answers and my spending more time with packages such as data.table and rcpp. It does not change my impression of bits and pieces being "glued together" in R in order to run more performant computations (Rcpp is in effect a bridge to run compiled C++ code and data.tables is a highly indexed data structure which should not be compared with solutions that make no use of indexing). My main concern at this point is that I will end up with code bases in multiple languages to achieve ... – Matt Jun 8 '15 at 3:45
• ...performance that matches or exceeds what can be done purely in Python. For example, any statistical or numerical techniques that cannot be vectorized require me to essentially maintain a C++ code base to beat code operations in Python. Similar applies to visualizations: Most dynamic visualizations or visuals that allow me to pan/zoom or otherwise manipulate rendering during run-time requires knowledge of .js and/or D3.js. Python on the other hand allows me to more easily interface with existing visualization libraries I already peruse. But as said, I have not yet come to a final conclusion – Matt Jun 8 '15 at 3:51

My deal is HFT so what I care about is

2. perform very efficient data-munging operations (group,transform)
3. visualize easily the data

I think is is pretty clear that 3. goes to R, graphics and ggplot2 and others allow you to plot anything from scratch with little effort.

About 1. and 2. I am amazed reading previous post to see that people are advocating for python based on pandas and that no one cites data.table The data.table is a fantastic package that allows blazing fast grouping/transforming of tables with 10s million rows. From this bench you can see that data.table is multiple time faster than pandas and much more stable (pandas tend to crash on massive tables)

Example

R) library(data.table)
R) DT = data.table(x=rnorm(2e7),y=rnorm(2e7),z=sample(letters,2e7,replace=T))
R) tables()
NAME       NROW NCOL  MB COLS  KEY
[1,] DT   20,000,000    3 458 x,y,z
Total: 458MB
R) system.time(DT[,.(sum(x),mean(y)),.(z)])
user  system elapsed
0.226   0.037   0.264

R)setkey(DT,z)
R)system.time(DT[,.(sum(x),mean(y)),.(z)])
user  system elapsed
0.118   0.022   0.140


Then there is speed, as I work in HFT neither R nor python can be used in production. But the Rcpp package allows you to write efficient C++ code and integrate it to R trivially (literally adding 2 lines). I doubt R is fading, given the number of new packages created every day and the momentum the language has...

EDIT 2018-07

A few years latter I am amazed by how the R ecosystem has evolved. For in-memory computation you get unmatched tools, from fst for blazing fast binary read/write, fork or cluster parallelism in one liners. C++ integration is incredibly easy with Rcpp. You get interactive graphics with the classics like plotly, crazy features like ggplotly (just makes your ggplot2 interactive). For trying python with pandas I honestly do not understand how there could even be a match. Syntax is clunky and performance is poor, I must be too used to R I guess. Another thing that is really missing in python is litterate programming, nothing comes close to rmarkdown (the best I could find in python was jupyter but that does even come close). With all the fuss surrounding the R vs Python langage war I realize that vast majority of people are simply uninformed, they do not know what data.table is, that it has nothing to do with a data.frame, they do not know that R fully supports tensorflow and keras.... To conclude I think both tools can do everything and it seems that python langage has very good PR...

• Hmm I guess I need to disagree with you here regarding visualizations. R packages are still lightyears behind efficient and especially dynamic visualization. Every first year IT student can chart a time series from scratch. What people want and need is visualization of millions of data points that a charting app can down sample. Fast zooming and panning and handling of annotations. I have not seen anything in R that comes even remotely close. – Matt May 28 '15 at 20:07
• Secondly datatables in R are very very slow. Throw a few million time series data points at it and data frames go to their knees. The only thing I have seen that was fast was an implementation that perused memory mapping. But one could argue this is just an interface R peruse ...as soon as you actually grab the data and run R functions over it becomes very slow. Caveat here: I have not looked at any new developments over the past 8 months in R space. If there is anything new I would be happy to be pointed to it. – Matt May 28 '15 at 20:10
• @statquant, another issue with data.table is that their founders and followers are somewhat too protective (although similar in the case of pandas too) and any sign of mentioning issues with data.table you can expect your post on SO to get rapidly downgraded. At least in the case of pandas, (1) the source code is available on github (you can do an easy search on web without downloading) versus downloading the sources code from CRAN, (2) you can easily overwrite pandas to customize your own subclass of pandas dataframes (I use two such specialized subclasses for my work). – uday May 28 '15 at 20:26
• @statquant, I cannot comment on what we decided in the end to peruse but I can comment on why we decided not to choose R. The biggest reason was time-consuming procedures to hook up R code with our existing research and trading framework. While everything can be interconnected some way or the other connectivity often proved quirky and cumbersome. To name just two examples: To up R's performance regarding large data sets one can use packages such as data table and Rcpp among others but then the question begged why using R in the first place. Another example is visualizations: None of the... – Matt Aug 8 '15 at 13:27
• ...R packages provided visualization capabilities that we demanded. Even D3 is not performant enough. We decided to stay with C# in regards to our front-ends that encapsulate visualizations. We hardly use any of the R packages for algorithms and statistical/mathematical computations hence we are entirely independent from those. In the end we wondered what R is particularly good at in comparison with other choices and we believe R is a great all-round tool that performs reasonably well but does not come out on top in any of the categories. Hope this explains our decision making process a bit. – Matt Aug 8 '15 at 13:30

Instead of wild guesses about R's/python's future in the community, here some facts:

The following query on StackExchange Data Explorer counts the number of questions that have <r> or <python> tags. If you scroll down on one of the three webpages provided below, you can see a graph with data on a monthly basis. You can easily run this query on databases for other sites as well (just go to "Switch sites" right below the query).

The results:

• In absolute terms, R has more hits for both stats.stackexchange.com and quant.stackexchange.com (the latter having very few data points). Python has more hits for stackoverflow.com.

• In relative terms, the gap between R and python is closing for stackoverflow.com (ratio approx 1 to 3 at the moment). The ratio between R and python tags on stats.stackexchange.com is more or less stable since mid/end 2013 (roughly a factor 10 or a little above).

I really do think that the tag statistics in the stackexchange universe are a good indicator of the current interest in a particular programming language - probably even more so for its future popularity.

All-in-all, I am confident that the present data makes a strong case against Matt Wolf's hypothesis that "R might be obsolete in 3-4 years". ;)

Update: So now it's been 6 months since my initial answer. We still have to wait another 2.5-3.5 years to definitely see whether R has become obsolete. :) In the meantime, a quick addition due to Matt Wolf's comment. Here are variations of the above queries that give you the tag ratios (that's what I have been referring to in the second point of my answer). All ratios are python tags divided by R tags.

stats

http://data.stackexchange.com/stats/query/421036/r-versus-python-tags-quotient-py-r#graph

I do not see a clear trend here. The Py/R ratio is around 0.07 (there was a spike to 0.095 in November though). Since mid 2013, the ratio varies between 0.04 and 0.11. So I would call it relatively stable.

SO

http://data.stackexchange.com/stackoverflow/query/421032/r-versus-python-tags-quotient-py-r#graph

There was indeed a short term trend in favor of Python since Jul 15 (Py/R ratio went from 3.1 to 3.5). So the statement that "R is closing the gap wrt the Py/R ratio" could be called obsolete at the moment.

quant

http://data.stackexchange.com/quant/query/421042/r-versus-python-tags-quotient-py-r#graph

Still very noisy. Python did seem to catch up a little bit the last few months. But hard to tell with that little data.

• Three cheers and an up vote to bringing empirics into a decade full of conjectures. – Dirk Eddelbuettel Aug 15 '15 at 19:34
• @cryo111, not sure I would call this stable, tags spreads have continuously increased since the starting date of your query. Also, StackExchange does not seem to be the main platform of exchange for Python power users. I have not mentioned it in my original post but a majority of research work in AI (ML and especially Deep Learning Networks) is performed with Python as wrapper, all major tools such as Theano or Torch provide Python but no R libraries. So, I hold on to my hypothesis, which I feel even strengthened today vs 7 months ago. – Matt Jan 8 '16 at 10:41
• @MattWolf In my above post, I have been referring to the tag ratio. Maybe I have been unclear with that. Will add 3 more queries for the ratios. Wrt to these, things seem relatively stable (apart from quant.stackexchange where volume is low and therefore very noisy). What is the main platform for python users? I am not a python programmer, so I don't know to be honest. But I agree with you that R probably won't overtake python in terms of absolute user numbers [which I have never been claiming :)]. – cryo111 Jan 8 '16 at 12:04
• @cryo111, not sure we are looking at the same data (and I have not run time series queries over StackExchange data myself but just looked at simple tag counts). Python currently stands at 517027 tags counted while R tags amount to 119847. Not sure how you can get to a quotient of Py/R ever being smaller than 1...(source: api.stackexchange.com/docs/…). Now if the most recent monthly counts strongly point in the opposite direction... – Matt Jan 9 '16 at 4:48
• @MattWolf No, you are not looking at the same data. Your numbers are the sums over all tags (since the beginning of the SE universe) whereas my numbers are per-month aggregations. I have chosen a time-series representation because I wanted to see the trend. The Py/R quotient smaller than 1 comes from stats.stackexchange.com. I have also included SO and quant SE, as these seem to be the most interesting for quants. I don't know of any other SE sites that might be relevant. But if you know other sites, you can easily switch the above queries to one of these. – cryo111 Jan 10 '16 at 14:10

This is interesting because I see another trend: Matlab is being replaced by R, but I guess this is another story.

I use R for my academic (I am also teaching this stuff) as well as my consulting work (I am mainly working in the $$\mathbb{P}$$ area, with some excursions into $$\mathbb{Q}$$). I tried Python but it didn't work for me. I think the main reasons I will stick with R are:

• especially in the area of statistics and analytics there is such a huge amount of high quality packages with sometimes even very recent methods which is unrivalled by any other language out there
• for me R has the right mixture of low level capabilities of e.g. (re-)organizing data and high level commands (e.g. even k-means in the core package)
• the speed is ok for me because I am not working in the area of HFT and there are many possibilities of speeding up code (vectorization, parallelization, good connectivity with C asf)
• the community is really very much into the kind of stuff I am interesting in whereas with Python it is really everybody and his dog doing all kinds of stuff I am not interested in... I guess this is also about the mindset how to approach some problems, I don't know.

I think in general one should focus: I wouldn't try to build a webpage or a game with R but when it comes to statistics and analytics I think Python is no real competitor and I would strongly recommend R as your future setup.

Edit
I also wrote a blog post with additional points about why R is better suited for data science than Python: http://blog.ephorie.de/why-r-for-data-science-and-not-python

• I agree the available packages that pertain to stats, math, and financial math are quite numerous in R. Though the current rate of new packages that target the above areas seems to be a lot higher in Python than R these days. I got the impression that R might be obsolete in 3-4 years due to so much that is done or ported over to Python right now and that is what caused me to ask this question, to gauge whether others share those observations. Thanks for your input on this. – Matt May 19 '15 at 15:18
• I got the impression that R might be obsolete in 3-4 years. I take the other side on that bet. I actually watch what packages get added every day and I don't see this as stagnating at all. – Dirk Eddelbuettel May 20 '15 at 3:06
• I'd take the other side of that bet too, Matt. These things take long time. The last time I checked many of our academic brethren were still enamored with Fortran. In all seriousness though R is alive and growing just maybe not the best for the broader use case you describe above. – rhaskett May 20 '15 at 21:21
• I stand by my own estimate but I did not intend to flame or cause discontent. Sorry if that above number rattled some cages. I just hear a lot of new quant projects get started in Python rather than R which got me thinking and caused me asking this question. R has the strength of an existing library repository but the growth momentum seems to be on the Python side. – Matt May 21 '15 at 4:03
• I'm more-or-less with @vonjd on this. I've used a bunch of languages, but I'm most productive for statistics and analytics in R. Python is a great language and numpy and pandas are a fantastic combination. However, the community for R is just so much better. I once spent several hours over a few days to get ipopt to work on Python, and then it just worked without any real effort with R. I also don't like the conda package manager because I can't get it to work behind a corporate firewall. – John May 21 '15 at 15:32

I've used both R and Python with Pandas in a professional quantitative financial work to do both large and small scale projects. I would strongly recommend Python with Pandas over R for most new projects in the field especially in time series analysis.

While I don't dispute vonjd in that you will find more libraries in R with algorithms on the bleeding edge of statistical research, the libraries in Python are very robust and fleshed out in that area. Also, I find in my work and the work of my colleagues that we are grabbing libraries from electrical engineering, computer vision, big data and more. People in these fields mostly have libraries in Python, not R.

However, the main advantage of Python over R in this field is workflow. The workflow with R tended to be that you used Perl/Python for data cleaning, preparation database work because R was too slow awkward for large complicated datasets though this is getting better. You then build the statistical model in R taking advantage of its libraries. Afterwards, the R model was rewritten in C for speed, control, interface, parallelization and error handling for production.

Python can handle this full workflow start to finish. All the inter-connectivity steps surrounding the main research projects is much more robust and a lot of time is saved in development when using the same language throughout. Also, with Pandas the even the core research portion and data handling is now easier and cleaner in my opinion.

In general, if you are just focusing only on advanced statistics/data-mining time series research then R and Python with Pandas are interchangeable at least for now. However it sounds like from your question that you are also are worried also about inter-connectivity and architecture for that Python is far superior.

Edit for 2018: It's amazing how much easier it is to get into data munging in Python these days compared to when I first wrote this. Try Anaconda for those that would like to check out Python/Pandas without any fuss.

• yes, that is another trend I am seeing, for time series analysis a lot of academic courses nowadays seem to have switched from R to Python as teaching and demonstration tool. I am not generalizing but a lot of students with Master's degrees I recently interviewed seem to have a much better grasp at Python than R. But one thing that makes me not yet want to fully embrace Python is: What libraries are exactly out there that assist in time series analysis, derivatives pricing, modeling, applying machine learning techniques aside the generalized Pandas, SciPy, ... packages? – Matt May 21 '15 at 4:11
• On statistics, Python just doesn't have as much developed as R does on those fronts, but statsmodels is my goto. One option is to just call R functions with Python (RPy, but not apparently well tested with Windows). On derivatives, pyql seems to be more developed than the R version. On machine learning, scikit.learn. One other benefit of Python is that iPython Notebook is better than Rstudio. However, the Jupyter project looks interesting (and should work with R). – John May 21 '15 at 15:40
• @rhaskett, regarding inter-connectivity, I think it is extremely important to be able to efficiently interface with other modules, hardware, software applications. I believe it to be a myth that most who seriously perform data analyses do not need inter-connectivity. In that I find Python a lot more capable and it provides more efficient means to, for example, fan out computations to other hardware instances. – Matt May 24 '15 at 7:27
• In reference to @statquants answer, the OP didn't mention HFT but I completely agree for HFT you pretty much always end up in C. I know of some that feel the need to write their own C for an added boost over RCpp but your mileage may and almost certainly will vary. – rhaskett Oct 11 '15 at 6:10

For data analysis, particularly for large data analysis project, pretty much most of the top quant hedge funds and a lot of the banks are using Python (over R) for a couple of reasons but many still have bits and pieces of R for specific packages or functions (I work at a bank and interface with quite a few quant hedge funds on data analysis):

1. Earlier Python 2 used to have a lot of backward compatibility issues, but Python 3 is more stable between versions. Even Pandas versions since 0.13 are very stable between versions. No one wants to use a language for which they have to revisit and rewrite significant codes sometime in the future.

2. People needed same codes to run on both Linux and Windows. Installing, compiling packages in Python can be a super pain, whether Linux or Windows. A lot of people did not wanted to do any new project in Python 2 as sometime in the future one would need to move to Python 3 and they stuck to R for quite a while. Also for a while, Python 3 was available only with WinPython distro and WinPython used to work only on Windows. Anaconda, which is leading Python disto for Linux (& Mac), came out with Python 3 support sometime in 2014, which then caused a huge migration.

(i) Raw speed is the biggest motive (allowing you to do way more statistical data analysis in the same time)

(ii) Pandas can read csv files very fast (one of the reasons why many folks moved from Matlab to R at some point)

(iii) Cython is more flexible than RCpp (at least my experience)

(iv) organize code files neatly into logical directories and classes within files (classes in R are an oversight) and the project looks much better

(v) As of 2015, PyCharm is a significantly better IDE than RStudio (although RStudio is better than Spyder). Tools matter

(i) The big issue with Pandas used to be that it didn't have its own binary data format. R's RData format is a huge edge. PyData's HDF5 based storage is not compressible easily, gives a lot of errors every now and then, and for big data it was a hindrance. Pickle, and other formats didn't just cut it. After years of Python-vs-R exploration, most ended up writing their own custom binary data format (to store Pandas data frame) or using significant modifications of PostgreSQL for big data storage.

Statistical packages are generally great with both languages.

I have projects in R that took 4 hours to run every day (over night). Now, in Python, they take a total of 20 minutes (with much less use of Cython codes than RCpp codes in R). That's the speed difference for you.

• acquire, store, maintain, read, clean time series: Python is better

• perform basic statistics on time series, advanced statistical models such as multivariate regression analyses, etc.: both Python and R

• performing mathematical computations (fourier transforms, PDE solver, PCA) visualization of data (static and dynamic): both Python and R

• pricing derivatives (application of pricing models such as interest rate models) : both Python and R

• interconnectivity (with Excel, servers, UI): Python is better

• Thank you for sharing your experience and providing pros and cons, I appreciate the balanced thought sharing. Though regarding backward compatibility, does Python 3.x not break backward compatibility? And in terms of mathematical and statistical features of packages, R clearly still has the lead here, imho. At the same time, however, I do not see much value in 90%+ R packages because they target a very specific statistical approach and the implementation is not modularized and not extendible, so that functionality remains very limited, almost to the degree of single time usage. – Matt May 24 '15 at 7:30
• If you migrate from R to Python 3.x, you have less to worry about backward compatibility, but if you migrate from R to Python 2.x, you will have to worry about backward compatibility if you later decide to switch from Python 2.x to Python 3.x – uday May 24 '15 at 18:51
• also, regarding R packages or Python packages, a lot of real world stuff involves modifications that standard packages can't handle. For example, let's say if you want to do create a tradable ICA or PCA from tradable time-series (e.g. time-series of stock prices or futures prices), you wanted to a liquidity-weighted ICA or liquidity-weighted PCA to avoid the top ICA or PCA factors to loading up some penny cap stocks etc. So, you will end up looking a the source codes of ICA or PCA in either R or Python and rewrite your own codes – uday May 24 '15 at 18:57
• what I meant was backward compatibility within the Python stack. Is it true that using Python 3.x unables me to use packages that target 2.x? – Matt May 28 '15 at 0:59
• packages that were originally written for 2.x and aren't made compatible for handling both 2.x and 3.x will unlikely work without errors in 3.x. But the list of such packages is really very small - vast majority of the packages like numpy, pandas, etc., work well with both 2.x and 3.x. – uday May 28 '15 at 18:59

For the tasks listed, both Python and R perform very well. There are some packages in Python not in R and vice versa. My solution for this is to simply call R from Python. This allows for the best of both worlds.

It is also important to note I do not write any R code other than calling an R library from Python.

Calling Python from R does not work equally across all major OSes as well.

Also in the high frequency / medium frequency field here.

I received a "mixed" consensus regarding the use of R and its prevalence in the field (specifically HFT). Speaking with someone who works in the equity option industry at a relatively small proprietary firm in San Francisco, I was told, "R is a legacy language".

However, speaking with someone who formerly was leading a HFT team at Goldman Sachs, I was told it is still the best language for time series analysis, statistics and especially latency sensitive projects. For libraries, the following were mentioned:

1. Quantmod (See Quantmod)
2. Caret (See Caret)
3. Zoo (See Zoo)
4. XTS (See XTS)
5. highfrequency (See highfrequency: tools for high frequency data analysis)
6. The popular open source QuantLib library also has an R version, which can be found here.

And to reiterate on other answers to this question, given how heavily dependent the HFT field is on speed, R cannot be integrated into production HFT systems. However, the R C++ Package is a popular tool which makes the integration to the HFT system both practical and easy.

I would not say R is dying, but it also does not have a monopoly for data analysis in the field of quantitative finance in general. Python and matlab are of great use in this field as well (I seem to be a minority in my use of matlab but it is great).

• Thanks for sharing your experience. My prediction was not that R would be dying but that by 2019 Python would by far have outpaced R's utility in quant space. I think it is safe to conclude this has already occurred by today. I occasionally see the odd R port at attempting to link up R to ML or deep learning but the overwhelming majority of development happens in Python space, whether regarding quant, ML, deep learning, or hft, I am strictly speaking of research and development not deployment. – Matt Jun 12 '18 at 4:15
• @Matt There are none so blind than those who will not see. Deep Learning is well supported by python, as to hft (for as much as it makes sense to relate hft and such tools ) I do not know of any serious firm that uses python. – statquant Oct 24 '18 at 18:21
• @statquant, perhaps then opening one's own eyes might be the perfect recommendation. If you hear of any recent interviewees who got hired into quant roles at hedge funds, hft firms, and banks and were not asked and most likely tested on Python skills then do let me know. And DL is an area that is strongly pursued by all entities listed above where Python is the language of choice by far. I do appreciate you are a R power user, all the blessings to you, but averages do not work by extrapolating from oneself. – Matt Oct 25 '18 at 23:49

The major advantage of Python (w/ pandas) over R is that Python supports OOP (object-oriented programming). It makes sense to organize a large code base using a hierarchy of classes. Python also supports the notion of polymorphism so that we can use well-known design patterns (e.g., Strategy, Observer, etc.) in our code.

• First R is also object-oriented, second the problem I have with python is that you never know when to use a function or a method, see e.g.: stackoverflow.com/q/8108688/468305 – vonjd Nov 6 '18 at 15:05