1
$\begingroup$

This is a soft, and probably also an opinion based question.

Suppose I am writing a library for numeric linear algebra for the purposes of working with financial data. My goals are:

  1. Since I work with big data, I want to make it as fast as possible by using parallel computing.
  2. I want my colleagues to use this library without understanding/tweaking the parallelism underneath the functions.

The resulting questions are:

  1. Is this a good idea to make parallelism the default behaviour? For instance sum(vector) to parallelise the summation without asking the user.
  2. If yes, is there any rule of thumb that would cover the default behaviour of the task split between the processors? I.e. how many processors I should use?

Asking on QuantFinance since I am especially interested on how people in the industry tackle this. Thanks in advance.

$\endgroup$
2
$\begingroup$

I'm a big fan of data.table. The authors of data.table put a lot of thought into performance on big data sets and thanks to its popularity and age have had a lot of feedback and gained much experience in weighing the alternatives. I would definitely recommend that you familiarize yourself with their thinking.

They allow setting the number of threads using the function setDTthreads():

Set and get number of threads to be used in ‘data.table’ functions that are parallelized with OpenMP. The number of threads is initialized when ‘data.table’ is first loaded in the R session using optional envioronment (sic, PR made) variables. Thereafter, the number of threads may be changed by calling ‘setDTthreads’. If you change an environment variable using ‘Sys.setenv’ you will need to call ‘setDTthreads’ again to reread the environment variables.

This is specific to working with an R package but I think the principle applied carries over. Make it configurable at runtime and through environment variables and set a sensible default variable. In the case of data.table the default is quite greedy which make sense for people doing analysis on their own machine. It's also smart in that it detects being forked: if it detects that the process was forked, parallelism is removed because it's likely that parallelism is applied at a higher level and trying to use all the cores in all the threads would lead to very bad contention.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.