Back to Posts

How hedge funds can clean up their 'big data mess'

Posted in Insights

Even before “big data” took its place at the top of the long list of business buzzwords, hedge funds and other investment firms were in constant pursuit of ways to extract useful insights from the data available to them. While tools built around artificial intelligence and machine learning technology have come a long way in the last few years to make it easier to separate the signal from the noise, hedge funds still struggle to use big data in a way that consistently improves their trading strategies.

Data comes from everywhere, but quality is uneven



A 2016 Bloomberg article by Saijel Kishan titled, “Big Data Is a Big Mess for Hedge Funds Hunting Signals,” traced the expansion of what qualifies as big data for hedge funds. It’s much more than financial data – fund managers are looking at things like user behaviors on social media and satellite images that track foot traffic at shopping centers to drive trading strategies.

Hedge funds are leaving no rock unturned to develop differentiated strategies and they’re tapping more sources of data than ever before. But as with most things, quantity does not equal quality. For example, Kishan pointed out that one fund received a large set of credit card data from Cracker Barrel restaurants, but later found out that transactions for the Nutcracker ballet were also included in the dataset.

“A lot of the data is useless and even the good stuff needs to be laboriously cleaned of erroneous bits and duplicates,” Kishan wrote.

Another issue that funds face is the legality of the data they obtain and use. If a vendor sells a fund data that it did not have the right to sell without consumer consent, there will be major ethical and legal concerns about using it. With data becoming an even more important resource, new vendors are popping up all the time looking to capitalize on the trend. It’s critical for funds to have their researchers vet these vendors and the data they sell before using it.

Kishan concludes his article with a call to action for funds: Give your data scientists and quants more sway in the overall management of the fund. For all the lip service paid to the importance of big data, too many funds still have silos between managers and quants. Making intelligent use of data means having people who truly understand how to gather, clean, and interpret it.


Elsen nPlatform helps hedge funds clean their big data messes



Bridging the divide between quants and managers will be key in unlocking the true potential of big data, and that’s been a core focus of what we’ve build with the Elsen nPlatform. We’ve made data more accessible and understandable for users without the programming and statistical chops of a true quant.

One of the greatest benefits of Elsen nPlatform is that it helps hedge funds avoid the major issues they’ve had with data quality. nPlatform’s Data Store eliminates the need for researchers to spend weeks or even months cleaning huge datasets. It includes thousands of pre-loaded datasets from only reputable, premium financial data vendors that are precleansed and normalized so customers can access and use them almost instantly.

All of that data is stored in a vendor-compliant, multi-tenant cloud environment reducing spin-up time, lowering costs, and making it possible for us to allow for real-time row-level permissions so only qualified users can access and manipulate the data.

Harnessing big data is a challenge even for the most talent-rich hedge funds. Using solutions built on the nPlatform, managers can get on the quants’ level by turning chaotic data into an organized and usable resource.

Day-to-day, Jim is responsible for managing business development with channel partners, promoting positive customer engagements, and driving operational excellence across the company. He makes sure Elsen can scale successfully as it grows. Before joining the Elsen team, Jim gained extensive experience in building technology businesses in both large and startup companies such as EMC/DELL, Akorri, NetApp, Sun MicroSystem, and Pirus. Jim is most excited about how Elsen is creating a paradigm shift in how financial services use modern technology to significantly accelerate their research. When he’s not at Elsen, Jim is likely playing guitar, biking, enjoying good food and wine, or playing poker. Jim holds a B.S. in Electrical Engineering from Oregon State University.