Some years ago I used to work for a large institution that was not a hedge fund and intercated with some folks who were worked on data cleanup. I'll share some observations that I hope may help understand what they do.

They focused on two kinds of data: securities indicative data (stock dividends, nond maturities and coupons), and market data (prices, rates). I think these days, "alternative" data (things like the number of people who visited a particular mall on a given date) has also grown more prominent.

Some of the people developing the processes and procedures for cleanup had PhDs. I'm pretty certain that no one executing the operational procedures did.

Typical examples of indicative data cleanup - Bloomberg is missing local identifier, amortization schedule, exotic coupon formula... all of these need to be amended in the internal databases.

Typical examples of indicative data cleanup - some value is an outlier, accoring to some criteria, then it needs to be investigated with the source (vendor or internal), and possibly replaced with a "missing" value. An unexpectedly "missing" value needs to be investigated with the source and hopefully populated.

As you see, such example are seldom quantitative, and much of it could be replaced with AI / automation.

Some years ago I used to work for a large institution that was not a hedge fund and interacted with some folks who worked primarily on data cleanup. I'll share some observations that I hope may help understand what they do.

They focused on two kinds of data: securities indicative data (stock dividends, nond maturities and coupons), and market data (prices, rates). I think these days, "alternative" data (things like the number of people who visited a particular mall on a given date) has also grown more prominent.

Some of the people developing the processes and procedures for cleanup had PhDs. However I'm pretty certain that no one executing the operational procedures did.

Typical examples of indicative data cleanup - Bloomberg is missing local identifier, amortization schedule, exotic coupon formula... all of these need to be amended in the internal databases.

Typical examples of indicative data cleanup - some value is an outlier, accoring to some criteria, then it needs to be investigated with the source (vendor or internal), and possibly replaced with a "missing" value. An unexpectedly "missing" value needs to be investigated with the source and hopefully populated.

As you see, such examples are seldom quantitative, and much of it could be replaced with AI / automation.