The Wisdom of Crowds: Open Source for Middleware

Feargal O’Sullivan and Jamie Hill of NYSE Technologies discuss OpenMAMA, the open source middleware Agnostic Messaging API they hope will expedite innovation in services, reduce vendor lock-in and minimize implementation time and cost.

Solving a Problem
Choosing a market data vendor because of their API alone is not sound practice. The issue of how to come up with a standard way of accessing market data that allows clients to select a vendor for any range of reasons – other than the API that the vendor happens to offer – has been a struggle for a long time. Something that should be low on any decision-making tree has unfortunately tended to be much more important. There are a number of different consolidated market vendors, including some obvious names like Thomson Reuters or Bloomberg and there is also a range of direct feeds or ticker plant vendors, where instead of going to a consolidator, feeds are accessed directly from an individual exchange.
In selecting a vendor, users must write all their code to suit that vendor’s particular way of accessing the data. Changing to a different vendor requires opening up the source code and altering everything to match how this other vendor wants to access the market data. With a consolidated feed for broad international access and a direct feed for low latency algo trading in US equities, for example, many users have to write according to two to four different APIs. This has been a significant problem for the industry and with OpenMAMA we are trying to drive the industry towards a standard.
User Base
This API is an eight-year-old standard that was initially developed by NYSE Technologies as the Middleware Agnostic Messaging API (MAMA), and it is quite heavily deployed in the financial services industry; close to 200 clients already use this API in their custom applications, so today it has an established installed base. We have opened that up and made it a standard by taking the source code for the APIs these firms are using today and provided it to The Linux Foundation, which will physically host the code as a neutral body.
During this process we worked with multiple parties that would not ordinarily use our API. Since the launch of OpenMAMA on 31 October 2011, one of the key factors to this being taken seriously as an open installation, was getting the right level of adoption. Before we launched, we approached a number of customers, other vendors and competitors, out of whom we established our launch partners J.P. Morgan, Bank of America Merrill Lynch, Exegy, Fixnetix and EMC. These launch partners, along with NYSE Technologies, formed a steering committee to drive the direction and the future of OpenMAMA.
From that point forth, each of those organizations who are part of that committee has a stake in Open MAMA. The API is open source under the LGPL 2.1 licence, so it is now owned by the open source community. With participation from Interactive Data, Dealing Object Technologies and TS-Associates as well, we now have a group ten strong and it is a global mix comprising different industries. Whereas before the API was driven largely by NYSE Technologies and our commercial use cases, now it is being driven forward as an industry standard. The more people we have to adopt and participate, the higher the likelihood of achieving that.
Adoption
OpenMAMA is very consistent with the models of development used by subscribers of market data applications, so porting to this API is straight forward. The strong response from competitors who wish to join this initiative is proof that they understand its value. The measure of success is allowing firms not to feel locked-in with any particular vendor based on the technical hassles of changing, but more importantly to spark innovation.
No one vendor is going to be the solution to every particular need. By opening up this API we bypass the wasted cycles of designing and writing new APIs that only get used by one firm and so we are encouraging the industry to innovate beyond basic plumbing and to start to create new services that offer a community environment.
Enabling Innovation
OpenMAMA enables firms that are creating applications to turn those applications into an object and make them a serviceoriented application. Much like what FIX did for transactions, we hope to standardize application development for the front office. This has the potential to offer significant cost reductions to developers with new event-driven applications for analytics, algorithms, historical back testing or intelligent risk checks.
Typically most vendors have to build a more complex system  because they have their own proprietary API as well as those used by other service providers. Standardizing this API allows firms to focus on offering the best analytics without concerns about compatibility with market data vendors and order management systems, linking the application fluidly with existing OMS, desktop applications, Excel plugins, etc. This drastically simplifies the adoption of small but useful applications. Everyone involved has identified that OpenMAMA simplifies creating event-driven applications; there is a single API and no vendor lock-in.
At the moment, we have a preliminary road map brought together by each of the committee members. Some of the projects we are collaborating on include an open-standard market data model, standardizing monitoring interfaces and standard interfaces for entitlements. We have also had a number of prominent players in the Advanced Message Queuing Protocol (AMQP) space who have offered to build and contribute back a bridge support for that too. We are talking about building guaranteed messaging query APIs with all open source payloads.
All of this is still in the preliminary stages as the steering committee has now met twice, while discussions go on in between. Underneath, the steering committee comprises various working groups, which contribute technical resources. The beauty of this is that we have collaboration between vendors and end users, both working together to develop future-proof software.
Balancing Neutrality and Throughput
There are three levels of low latency trading: ordinary trading is measured in milliseconds, low latency trading is measured in hundreds of microseconds and ultra low latency trading is measured in tens of microseconds end to end. At the ultra low latency level, firms must customize the normalization into their application by learning each individual feed, processing it in their application space, FPGA programming and generally become an expert about everything related to that market. As a result, it is difficult to build a generic interface for the ultra low latency traders because they have a competitive edge by being different.
For the low latency space, however, OpenMAMA is ideal because latency applies not only  to how quickly market data is processed but also how quickly strategies are moved in production. Programming that FPGA can slow implementation, and a fast final product may be eclipsed by another participant who started trading the strategy six months earlier.
Next Steps
The next important area we are working on is the Data Model. Every exchange has its own data model for how it sends and receives market data. We hope to use the same committee to standardize the data model by harmonizing fields among various market data vendors and we have a very good head start, with over 200 feed handlers represented. The API already has an install base, interest and momentum, so we are very confident it will be adopted. The Data Model takes longer for vendors to standardize, but it is not difficult. After those two projects, our remaining goal will be to address symbology.
When we converted MAMA into open source in October 2011, we opened up the C portion of the MAMA API, and since then we have been working to open source all the other components within MAMA. We have a roadmap of contributions into the open source repository, and by the end of April 2012, the entire API will be made available, including the MAMDA API and a series of wrappers for support for any other programming languages.
Working groups were formed according to the roadmap and each group will go through a few iterations before they publish any findings. Having vendors and end users involved together means that discussions are productive and we expect new features to be added to the API through the end of 2012. We are also hosting the Linux End Users Conference at the end of April to discuss the future of OpenMAMA and Linux. Currently, we are working on the structure and talking to the key players in the financial community. As more people adopt OpenMAMA and consider joining the steering committee or even contribute through the open source community, it is our hope that this project will become more than the sum of its parts.