Last week I attended the North American Trade Architecture Summit (NATAS), a Waters conference on high-frequency trading, real-time risk and of course Big Data. I was a panelist on a session entitled: “Overcoming the Big Challenge of Big Data in Finance”. From the size of the crowd it was the highlight of the day-long conference.   I have to give credit to the panel organizers for bringing together a diverse group of experts; my fellow panelists from the big institutions of Credit Suisse and BNY Mellon, a small quant fund Pan Alpha Trading, technology vendors and a consulting firm.  How the benefits, challenges and opportunities of Big Data are manifested is quite varied from the viewpoint of a Tier 1 bank as compared to a quant fund or the variety of clients that vendors witness.

The Opportunity of Big Data… in Finance

There has been an explosion of hype surrounding Big Data. In the past 12-months, the use of the term Big Data in the U.S. has increased over 1,200% on the internet. The term Big Data has largely been associated with loosely-structured content, originating from web search companies and social media.

Social Big Data is about gleaning meaningful value from unstructured content. It is judging the mood of the human psyche, which could be Twitter, Facebook, websites commentary and blogs.  In the hunt for business benefit, any one or group of data points cannot be valued as accurate or inaccurate only factors that influence behavior.  The science of social data is the alchemy of behavioral targeting involving not only what to keep, but what to throw away.

Conversely, in capital markets the need for reliable and accurate data is the driving force. Drilling down to the individual constituent is a mandate, be it a stock symbol, option strike, currency pair or futures contract.  In the fiercely competitive trading industry it is tighter spreads, thinner margins and a lower risk appetite that evoke a wider hunt for alpha as firms look to cross border, cross asset trading models and a reenergized focus on controlling costs. This has exponentially increased demand for deep data over longer time periods across global markets; equities, futures, options and currencies.  In the Options industry, trading volume last year was four and a half billion contracts, OPRA peaked at four million messages a second. The Options market is a quintessential example of Big Data’s volume and velocity definition.

Financial Big Data is about the capture and storage of this deep data and linking disparate data sets under some common thread to tease out an intelligible answer, a diamond from a mountain of coal. It’s the quantitative research to find the cross asset correlations or understanding how to best hedge a position to offset risk.

Firms demand confidence in the resulting analytics derived from Big Data… those used in determining the profitability profile of new models, optimizing parameters through backtesting of existing models and re-balancing portfolios. For this they all depend on accurate, clean market data across all their tradable markets and the capture of order executions. This big data dump is the fuel that drives the engine of the trade life cycle from:

• Alpha Discovery and Research
• Strategy Development
• Strategy Backtesting and Optimization
• Portfolio Valuation and VaR Management, Transaction Cost Analysis

It’s Big Data and the assurance of its accuracy, reliability and timeliness that powers this entire process, the engine of the trading industry.

The Challenge of Big Data… in Finance

Yet Big Data is messy. When it comes to financial data, market data of all stripes comes in many shapes, sizes and encodings. It continually changes and requires corrections and an occasional tweak. Market practitioner’s worst fears are spending more time processing and cleaning data than analyzing it. To focus on discovering new alpha and optimizing existing strategies, finding that diamond in the rough so-to-speak, demands a confidence in the resulting derived analytics. It means dealing with ….

• The vagaries of multiple data sources
• Mapping ticker symbols across a global universe  of exchanges & geographies
• Managing dynamic (rolling) symbologies (i.e. futures contracts)
• Ingesting cancelations and corrections to create clean prices
• Applying corporation action price and symbol changes
• Detecting (missing) gaps in history, that could influence outcomes
• Coping with exchange calendars, venue trading hours and holidays
• Managing complex order book structures

These are huge data management obstacles to overcome not easily solved by the common storage engine.  Normalizing and cleaning data is a major obstacle to ensure accuracy.

The competitive forces driving Big Data’s accuracy, reliability and timeliness are many. Within the next three years 80% of Hedge Funds will be trading algorithmically, creating an ever increasing number of firms chasing after an ever diminishing pot. Looming over the industry like a rain cloud is regulatory action, will there be order cancelation fees or minimum resting periods imposed?   These forces will drive deeper liquidity analysis across fragmented markets researching order book dynamics in that hunt for alpha.

The Prospects of Big Data… in Finance

Where are heading with Big Data? Data volumes will continue to grow across all asset classes and firms will continue to reach deeper to weather that perfect storm of thinner trade margins, regulatory actions and cost containment.

Disparate data sources will begin to play a greater role in trade analysis. The analytics behind linking market data and social data will expand beyond news sentiment. We will begin see to firms try to understand that to their advantage.

We’ll begin to will see non-financial vendors enter the space – primarily on the analytics side. As they try to capitalize on their knowledge and skills in social analytics.

Marketplace providers and exchanges will broaden their offering to include more than market data as social content becomes more relevant in trade decision modeling.  Just as in the retail industry where user opinion/product reviews are highly valued as a decision tool for buyers, the combination of market, news and social content will drive a new style of trade model.

As the hype subsides, the Big Data definition and understanding will clarify. But unlike hardware technology which always seems subject to commoditization the business value of data will always remain and only get higher.

Once again thanks for reading.
Louis Lovas

For an occasional opinion or commentary on technology in Capital Markets you can follow me on  twitter, here.