Removing Systemic Risk

1. Build continuity into systems with the greatest number of adjacencies

Redundancy and high-availability are critical requirements for all components in a mission critical system, but more so for those that interconnect other components.  This would include liquidity connectivity, network infrastructure and messaging middleware.  It’s important to note that building a mesh infrastructure as a means to resiliency actually has the opposite effect because risk increases when five nodes are interconnected in this manner.  This frequently occurs at both the network and messaging levels.  Redundancy does not mean plugging two systems into a risk-prone infrastructure – it doesn’t make the risk go away. The underlying paths must mitigate the “adjacency effect.”

2. Eliminate unnecessary integration

Much like a great team, each player in the trading infrastructure must play his part and play it well.  Too often, technology is adopted that allows systems to do many things.  The challenge is that individual component behavior is obscured and surprisingly at risk along with the other items with which they are comingled. Only do what you need to do.  This often generates conflicts because business units prefer to build out a separate infrastructure rather than embrace the economies of scale proposed by a centralized technology group.  If the latter could guarantee a specific risk profile, that would be an ideal scenario.  Otherwise, the savings may not outweigh potential operational issues.

3. Optimize performance characteristics

Major technology evolution occurs every five to seven years.  When it does, the results are dramatic.  Given that the benefits are often a magnitude higher and the cost, a fraction of the previous incarnation, adoption at the right time is important. This is most often when the technology has been in the market (not a lab) for about nine months.  Competitive pressures also influence this number.  We saw this with networks evolving to hardware; processors adding more cores; and now messaging moving to silicon.  It’s a continual evolution requiring ongoing education and evaluation for even the savviest of firms.  

4. Effectively provision management of multiple information streams

A great deal of market data and order flow is moving through the organization.  It’s critical that these streams do not all come together at under-provisioned rendezvous points, because volatility and micro-bursting can turn these into information dams, slowing data flow and causing risk-inducing latency.  This is a key consideration in messaging systems for market data distribution and order routing: software-based systems will need distributed streams while hardware-based systems can handle aggregated flows.  In this case, the risk profile is matched to the infrastructure capabilities.

Achieving operational excellence

Architecture is one side of the coin; operations is the other.  Operational excellence is not a one-time activity; it requires regular tuning and modification.  Quantified data and the ability to measure it are critical success factors.  The following figure highlights the reconciliation of risk with some of the requisite operational elements.

1. Establish operational checkpoints

Gone are the days when high-performance systems were compromised by monitoring capabilities.  Establish checkpoints at the system level to mitigate the risk of component failure.  Establish checkpoints at major ingress and egress points to mitigate the risk of systemic failure. Make sure that the checkpoints can report independently of being polled, especially when baseline conditions are breached.  Checkpoints need to be managed as well.  Too many checkpoints - if improperly set up - can adversely affect the trading cycle by slowing systems down or creating excess network traffic.

2. Measure, measure, measure

Having the checkpoint in place is one thing, but the performance expectations both individually and in the aggregate must also be considered.  Not knowing is not an excuse.  Set the criteria, establish benchmarks, validate regularly, and warn when operational thresholds are compromised. Measure data volume - not just averages but peaks as well.  Measure latency across the entire trading cycle in addition to specific execution points.  Measure end-to-end system performance and compare with benchmarks and trending curves.  Measure server utilization rates.  Measure your service providers and your trading partners.  Aggregate what you measure and perform regular statistical analysis.

Even with the proper planning, problems are inevitable.   If components can be dynamically isolated while maintaining overall system performance, that’s a major step forward.  The slow consumer problem described earlier is an excellent example from the messaging arena.  This problem is being solved by today’s contemporary messaging platforms because they have the intelligence to be self-isolating.  High-availability and resiliency are important, but have historically impacted performance, at least in the short-term.  Far better to add the requisite intelligence to prune systems (with notification of course) while ensuring consistent high-performance.

Continuity is key

The cost of a lapse in operational continuity in today’s high-performance trading infrastructures is too high to leave things to chance.   Numerous and diverse systems create complex interdependencies with complicated risk profiles.  The most effective means to model this risk is with chaos theory, but it becomes impractical in real-world environments. Capriciously adding new technology may not mitigate the perils either.

Fortunately, risk can be driven down substantially by addressing continuity factors in key, individual components, especially those that have a large number of adjacencies.  Architectural improvement can be done on both new and, more likely, existing trading platforms.  Major technology innovations play a key role here.  Architecture progression in isolation is not enough; operational controls and metrics must be established as well.  Though we’ll never entirely remove all risk, we can certainly reduce the chance of systemic failure by orders of magnitude.  That will keep savvy firms both in and ahead of the market.

Rob Ciampa

Tags: messaging (5) systemic risk (1) 

Comments (0)

Mission-critical, high-performance trading infrastructures remain extremely vulnerable to constituent failures, while burgeoning complexity and increasing system co-dependencies exacerbate risk and time-to-resolution. Today’s competitive environment leaves little room for operational mishaps, yet the foundation of the trading business – technology – is more susceptible than ever to failure.  This blog entry examines key technology hazards and a subsequent entry explains how to proactively mitigate danger without negatively impacting the performance equation.

The Sky is Falling

Recently, a major financial services institution specializing in mutual funds suffered a 48-hour outage in a critical business unit resulting in over $20 million in losses and numerous litigation issues.  The cause was the inability of algorithmic trading servers to effectively process market data, forcing the data producers to continually rebroadcast information and bring the underlying network down.

Another leading international financial firm recently saw its large New York City trading floor experience numerous outages that resulted in a suspended order flow of $90 billion and losses of $3 million per hour.   Data volumes wreaking havoc in the middle office and back office triggered network and system outages that brought down the front office.

The trading world continues to evolve as firms adjust their trading strategies to profitably exploit market opportunities before their competitors.  But neither of these variables – trading strategies nor market opportunities – is discrete, autonomous or effectively measurable in real-time.  Instead, they are part of an ecosystem of complex systems with sophisticated inter- and intra-dependencies.    In addition to diverse algorithmic trading strategies, firms must also manage trade structure, order routing and execution costs across diverse asset classes, markets and liquidity venues.  All of this will run on diverse IT infrastructures with diverse Service Level Agreements (if any) and have disparate controls and authority.  It’s critical in high-performance trading environments that all the components in the value chain are understood and their inherent interdependencies and risks are quantified.

The path to interdependent systems

Trading infrastructures will continue to evolve as long as algorithmic and technological advances keep yielding positive financial benefits.  Challenging economic and market conditions inherently present numerous and diverse execution opportunities.  Alternative execution venues, direct market access (DMA), algorithmic containers, electronic communication networks (ECNs), smart order routers, exchange co-location, multi-core processing, distributed caches, low-latency networks  and messaging systems are just some of the ingredients that must be combined into the optimal trading recipe.

Building a high-performance trading infrastructure from scratch is just not a practical option for several reasons.  Specific expertise is rare. There are too many niche components. Integration schedules won’t match the market opportunity window. And, finally, the operational risk of a monolithic system can knock a firm out of the market, which is far too common with many legacy systems today. Hedge funds, market markers, new liquidity venues, algorithmic trading houses and high-frequency trading firms are but a sampling of organizations either leading the charge into high-performance trading or being dragged into it.  They understand that the status quo won’t do.

The most overriding factor and the foundational driver for competitive advantage with interdependent systems is effective and efficient automation of all critical components of the trade lifecycle which includes market data processing and distribution, risk management, order routing, etc.  Historically, a good deal of emphasis has been on straight-through processing (STP) to automate and integrate information flow within a firm, but contemporary demands emphasize other critical path trade routes.  For example, market data distribution has been a critical challenge for firms, especially as market data volumes continue to grow.  In its latest year-end report on market data capacity, the Financial Information Forum (FIF) reported significant volume growth across all feeds and market centers.  The implications for firms are that any processing challenges in this area will only be exacerbated down the line.  Similar parts of the trading ecosystem have challenges as well, requiring a look at the overall risk.  Our goal is to keep our overall risk profile within an acceptable range.

The risk of complexity

With so many disparate systems in the trade lifecycle, the risk equation changes greatly.  If there is one system, it can be readily modeled for risk.  If there are two systems, symbiotic models can be produced to sufficiently define the risk profile.  If there are greater than two systems, it involves a level of complexity that is increasingly difficult to model.  If there are three systems, for example, – A, B and C – then there are 10 risk factors: A, B, C, AB, BC, AC, ABC, AB on C, BC on A, AC on B.  Beyond that the model becomes significantly more complex, as does the risk profile.  What elements are part of the modern trading equation?  Networks (switches, routers, WAN links), messaging systems (producers, consumers, middleware, queues, servers), OMS, and EMS are some of the numerous elements of this equation.  The assertion is that merely adding a single system to the trading ecosystem increases the risk factor not by the autonomous risk profile of that system, but potentially by magnitudes as that system interacts with other systems.

A simple example is warranted - we’ll call it “The Slow Consumer Problem.” Assume market data is being distributed to 30 systems using the conventional message-oriented middleware of an Ethernet network.  Efficiencies have been built by using multicast technology so that a single message can be delivered to all systems simultaneously.  When one system falls behind in processing the market data (the slow consumer) requests that the producer retransmit information.  This, in turn, slows the producer down and forces all the other consumers to process the retransmitted information (which they will likely discard).  As a result, all systems – producers and consumers – lose valuable processing cycles and the network bandwidth will begin to erode. We have research that shows the slow consumer issue is not an isolated, one-time event.  Excessive requests may ultimately and dangerously impact the producer of information.  This cascades down to all the other components in the trading ecosystem, ultimately resulting in such conditions as price slippage and the inability to trade profitably and effectively.  Many trading environments are that fragile.

Given the less-than-desirable operational profile, there is an effective way to model the risk but there are also distinct challenges.  Chaos theory is the most appropriate model.  IT media company TechTarget provides an excellent definition:

In a scientific context, the word chaos has a slightly different meaning than it does in its general usage as a state of confusion, lacking any order. Chaos, with reference to chaos theory, refers to an apparent lack of order in a system that nevertheless obeys particular laws or rules; this understanding of chaos is synonymous with dynamical instability, a condition discovered by the physicist Henri Poincare in the early 20th century that refers to an inherent lack of predictability in some physical systems…The two main components of chaos theory are the ideas that systems - no matter how complex they may be - rely upon an underlying order, and that very simple or small systems and events can cause very complex behaviors or events.

In his book, Chaos Theory in the Financial Markets, Dimitris N. Chorafas analyzes in great depth the role of nonlinear systems, volatility, risk and cumulative exposure, as well as cognitive models for financial operations.  Dr. Chorafas states:

An overriding need in any business is the ability to represent problem information in such a way that the full complexity and dynamic nature of the underlying structures is captured.  Financial systems are no exception … As the information and the tools needed to solve prediction problems becomes more complex, it is increasingly more challenging to foresee and represent the evolving real-world situation.  From risk management to generation of profits, flexibility is the cornerstone of a successful prediction process.

 
Dr. Chorafas’ assertions address the external dynamics of financial markets.  The premise of this blog entry is that organizations must use the same scrutiny on their internal trading systems, understanding fully that a symbiotic relationship exists between these two environments.  Several challenges exist.  First, seldom do firms assign their quants to develop models for internal systems.  Next, the operational methods of most of these systems are not quantifiable.  Even the application of chaos theory to weather systems has greater measurable components. Last, even if these two challenges were overcome, it would be extremely difficult to bridge the operational – and political – compartments involved in the trading infrastructure.

Mitigation strategies

Due to the complexity of tackling risk at the systemic level, addressing component risk first is both a pragmatic, tactical move and an effective, long-term strategy.  Purists may argue that systems should be engineered from the ground up for effective operational risk mitigation, but this is nearly impossible in practice, especially given the continual changes in technology and infrastructure.  Furthermore, the risk probability curve rises precipitously as components are added, so factoring out hazards in individual components will have the inverse, beneficial effect.

Is there an optimum number of systems for a trading infrastructure?  If they were all built the same way, then the answer would be an emphatic “yes.”  However, reality quickly sets in.  One international broker-dealer we work with was able to reduce the number of FIX engines on legacy middleware from 24 down to two (with redundancy) on a modern messaging platform.  Outages dropped by 87 percent while performance increased by over 300 percent.  In another example, one of our hedge fund clients detected network (and latency) deterioration after just five direct mesh connections on options processing servers.  They moved to a centralized communication system yielding predictable and consistent sub-100 microsecond latency.  In both of these examples, many factors were at play including data volumes, intersystem communication, excess messaging traffic, etc.

Before going through the steps, what types of risk are critical to remove from the internal trading infrastructure?  There are several.  The most dangerous is operational risk, the failure of a critical component in the information flow.  Next is performance risk, the executional degradation of a critical component in the order flow which may include processing slowdowns and system malfunctions that produce errant behavior.  Finally, there is flexibility risk, the inability of systems to dynamically adapt to diverse market conditions.  All of these ultimately aggregate risk implications in the overall systemlike being behind the market or entirely out of the market.

Also exacerbating these risk conditions are fault detection and isolation.  The more complex the ecosystem, the more difficult the root cause analysis and return to execution.  Understand that a system that comes back up will likely be hit, for example, by the market data deluge that sunk it in the first place, much like the mutual fund company discussed at the beginning of this blog entry.

Rob Ciampa

Tags: dimitris m. chorafas (1) high-performance trading (1) systemic failure (1) dma (1) ecn (2) stp (1) ems (1) henri poincare (1) 

Comments (0)

I recently came across a commentary on technology appliances in the data center where the author was asking a large number of rhetorical questions. Underlying his questions was a premise that companies (except for the top .001%) can solve all their business problems with  general purpose computing devices. Perhaps it’s time to throw out those Cisco routers, those Juniper firewalls, those IBM intrusion detection systems? Perhaps not.

Appliances, like other products, evolve through a lifecycle influenced by data conditions, operational economics, technology maturation and market acceptance. Early adopters of appliances are often large firms, but over time smaller firms and consumers dominate in broader, deeper markets. Importantly, though, what is an “appliance”? My internet access device at home now consists of a router, a switch, a firewall, a wireless access point and an intrusion detection system – and it’s the size of book. In time, I expect my next one to have server capabilities, audio and video storage, etc.

• Level 1 – Economics. Yes, you can exceed Moore’s Law. Adding more general-purpose computing elements may not be economically or physically viable.  Enter the appliance.
• Level 2 – Integration. As the appliance model solidifies, adjacent capabilities are added, yielding a more holistic solution.
• Cycle 3 – Simplicity. Unified configuration and management drive the user experience. Automation also begins to take hold.
• Cycle 4 – Defacto.  Changes to form factor continue. Integration into broader offerings and services become the norm.
  

Regards,
=rob.ciampa

Tags: maturity model (1) appliance (1) cisco (1) juniper (1) ibm (1) moores law (1) 

Comments (0)

My radar goes off when legacy vendors with legacy products announce non-legacy performance numbers. “Wow,” I say, “I’m very interested in hearing more about the innovation.” Often, however, the innovation is in the testing and not in the architecture.  Recently, one of the “big boys” called out Tervela when they touted the performance numbers of their new-and-improved-legacy-messaging.

I decided to check out their testing methodology. I couldn’t help but scratch my head on the lab and testing.

Years ago when I was a smarter man (in engineering), I had some excellent labs that would benchmark and test infrastructure performance and reliability: WANs, LANs, apps, etc. The challenge my group had was not in setting up in infrastructure, but rather in establishing flows that would test real world scenarios, edge cases, etc.  This was hard and – at times - non-scientific. The interdependencies and varying load conditions made it nearly impossible to model. That didn’t stop us and we called it dirty water testing. Clean water testing was just the opposite: simple, deterministic, and unrealistic.  We did both, but the dirty water mimicked real-world. We had no surprises when we went into production.

I want to applaud the big boys for calling us out after their clean water test. As a courtesy, please check out our testing methodology and the results for messaging infrastructure testing. It’s dirty water and it’s why a Tervela messaging infrastructure works as advertised. We’ll discuss their approach later. (And their less-than-green footprint, too.)

If you can't see the whole messaging infrastructure testing article, email me and I'll send you a copy.

Regards,
=rob.ciampa

Tags: tervela (2) messaging infrastructure testing (1) 

Comments (0)

I was in meeting recently where the topic of liquidity came up. It naturally moved to liquidity venues, their business models and the technical underpinnings of contemporary exchanges. I decided to pull an excerpt from a piece that Barry Thompson and I wrote several months ago for Banking Technology. Tough to separate superior business performance from technical excellence.

The ECN Assault

The baby is all grown up. BATS, the electronic communication network founded in 2005, has received exchange status from the SEC, making it the first ECN to reach this level organically. And it's not going unnoticed.

This means the threat level for traditional exchanges has just hit red alert. Infamous for highlighting weaknesses in traditional exchanges while advocating its price, technology and latency advantages, BATS has taken over 10% of the US equities market since its launch. As an exchange, BATS no longer has to publish quotes via other exchanges, which reduces overall trading latency vis-à-vis traditional exchanges. What's more, its newness means there is no legacy infrastructure dragging down performance. Instead investments in the latest technologies provide lower latency and more efficient use of human capital required to support internal systems.

The proliferation of ECNs - or multi-lateral trading facilities as they are known in Europe - also threatens the evolution of traditional liquidity pools around the world. Seeded by some of the largest market makers from exchanges like NYSE/EuroNext and Deutsche Börse, these investments are more than financial. They bring immediate order flow, enabling ECNs to quickly build up liquidity. Like BATS, they also are able to easily leverage new technologies to maintain advantage and can be more aggressive when it comes to pricing structure.

The "maker-taker" model, originally popularized by the equity and options trading industries, was also adopted by the former US Futures Exchange. The pricing model offers rebates for market makers while charging fees to customers who remove liquidity from the exchange. It brings together favorable market conditions and supports market growth by encouraging market makers to add liquidity, which results in tighter bid-ask spreads. The difference in price between the two becomes profit for the execution venue. This is where the lower cost base of the ECN model proves a key competitive differentiator over established exchanges.

The exchange strikes back

In today's volatile market, traditional exchanges are seeing order flow and liquidity being threatened. As such, they are actively seeking new ways to improve service, increase product 'stickiness' and remain competitive.

CME Group, which operates the CME, CBOT and NYMEX, has leveraged its innovation in technology to rapidly grow its business on the CME Globex trading platform from less than 15% of total volume in 2000 to nearly 90% total volume today. With more than two billion trades taking place at the exchange in 2007 (worth a notional value of $1.2 quadrillion), CME Group has differentiated itself from other exchanges through a vast distribution network. They have customers in more than 85 countries with access to the exchange through Globex nearly 24 hours every trading day and offer speeds of approximately 10 milliseconds within the CME network.

In August 2008 CME Group completed its acquisition of NYMEX and has publicly stated that the over the counter success of the ClearPort platform will figure into its future. Volume on the ClearPort platform is already up nearly 40% to 470,000 contracts a day over last year. CME Group also has strategic partnerships with several other key exchanges, building on the strength of Globex, including BM&FBovespa in Brazil, KRX in Korea, the Dubai Mercantile Exchange and the Green Exchange.

Interdependencies within the trading technology domain

Central to high-velocity trading, stability and resiliency are not traits synonymous with legacy architectures. The primary impediment to innovating through the addition of new asset classes or integrating new applications to bring about competitive advantage is a sagging foundation. Many traditional exchanges have built their market data distribution frameworks on outmoded software-based messaging infrastructures. When these systems break down the traditional mechanisms they use to compensate only exacerbate the problems.

Any time you are dealing with the sheer physics of information movement within complex systems - especially in a high-volume liquidity environment - your foundation needs to accelerate and integrate the entire trading ecosystem. The capacity requirements for this kind of trading are way beyond what software alone can provide. Only a hardware-based system can yield the substantial headroom and horsepower to handle large data bursts and unforeseen peaks in volume.

Because messaging systems are at the core of the vast array of financial applications running on an exchange backbone, they are often the prime culprit when system failures occur. With emerging trends like the proliferation of algorithmic trading influencing technology requirements on a constant basis, exchanges must ensure that they have a solid foundation on which they can layer the adjunct services they need to remain competitive.

Technology is the Path

Having the most stable and resilient underlying technology is critical when building out services to keep clients and attract new ones. Having close physically proximity to an exchange is important as is deploying latency- tuned applications that add value and attract liquidity. Market demands are fueling the need for smarter technology architectures so that migrating applications and adding new services no longer requires a complete overhaul of the existing system. The liquidity venue with the most modern and scalable infrastructure is always going to win.

Regards,
=rob.ciampa

Tags: ecn (2) bats (1) sec (1) nyse/euronext (1) deutsche börse (1) cme (1) cbot (1) nymex (1) globex (1) 

Comments (0)

Good article in today's New York Times on high-frequency trading.

These systems are so fast they can outsmart or outrun other investors, humans and computers alike. And after growing in the shadows for years, they are generating lots of talk.

Regards,
=rob.ciampa

Tags: high-frequency trading (1) new york times (1) 

Comments (0)

Interesting discussion thread occurring on LinkedIn regarding Kevin McPartland’s recent Tabb Group Report “Hardware Acceleration: Traders and Teraflops.” I’ve echoed my comment below because it’s worth repeating: effective trades depend on several factors and the proper application of hardware can have a significant impact. Note the use of “proper application” i.e. the right architecture. Here’s the note:

There is no silver bullet for the high-performance financial transactions and order flow. Our market deals with ever-changing feeds, algos, distribution requirements, etc., making it difficult to apply one set of processing logic holistically. Add in bus contention, OS optimization, multiple cores, caching, threading, and network flows/QoS and you have some serious, complex tuning to do. Not an easy task, especially since some may be outside one’s control and operational domain.

ASICs, FPGAs, network processors, GPUs and processors are additional parts of the equation and each one does certain things well. When we designed our message switches, we chose to do incorporate a blend of several of the components listed above, allowing us to deliver optimal performance because each element is doing what it does best. That’s the equation. I still hear many technically astute people say “I’m just going to put it in an FPGA.” Think again – an FPGA is only one variable and you have to solve for several more.

Regards,
=rob.ciampa

Tags: kevin mcpartland (1) tabb group (1) hardware acceleration (1) message switch (1) 

Comments (0)

We announced the Tervela TMX-500 Message Switch last week at SIFMA to a great deal of fanfare.  Attendee response to the new offering was entirely positive, many echoing the word “wow.”  We had it running and opened up for all to see. Based on some questions at the show, I wanted to take a couple of minutes to share what was behind the announcement and illuminate our design philosophy.

Planning for the TMX-500 began in 2008, some time after the TMX-1000 was publicly available and in-production at top-tier financial services firms: investment banks, hedge funds, broker-dealers, etc.  We had a good deal of experience to work from.  It’s worth noting that we saw demand for a complementary, smaller message switch from Tervela, so now we offer both the TMX-500 and the TMX-1000.

We run a continual and disciplined product management process at Tervela, one that is very much market-driven.  We spend a great deal of time with customers (business leaders and technical architects), third-party thought leaders, standards groups, ISVs, systems integrators, and perhaps – most importantly – people who would never buy a hardware-accelerated messaging solution.   Over my career, I have found this last group to be a treasure trove of product requirements.  They also, ironically, become the largest purchasing group of the products they say they’d never buy.

So what were the drivers?  What did the market want? What did customers and prospects want?

They wanted hardware-accelerated messaging, but in a smaller form factor that gave them deployment options for workgroups, data centers, co-lo facilities, etc. We brought it down to 2U (1U = 1.75 inches).

They wanted deployment flexibility.  We can multi-home to diverse networks through 16 x 1 Gigabit Ethernet ports or trunk through 4 x 10 Gigabit Ethernet ports.

They wanted to scale linearly without pain.  We can connect 1 to 16 TMX-500s into a single unified fabric.

They wanted to reduce the data center footprint.  Our first TMX-500 customer order reduces 2+ racks of messaging servers by 92% (!) to 5U (2 x 2U TMX-500s + a 1U TPM management platform).

They wanted to cut power consumption, too. The TMX-500 tops off at 250 watts, but runs steady state in the 100s.  This is a big deal.  More on it in a subsequent post.

They wanted real economics.  We gave them great CapEx, OpEx and TCO with the TMX-500.  Software solutions can’t match it.

They wanted the Tervela differentiators: high-performance, low-latency, scalability, and a seamlessly integrated fabric.  We didn’t compromise.  In fact, we made them better.

They wanted insane reliability.  We went solid state, added more environmental sensors, variable-rate N+1 fans, redundant power, ECC protected memory, etc.

They wanted future-proofing.  The combination of programmable ASICs and Tervela’s operating system for messaging, TVOS, ensures future support without any compromise.

They wanted to beat their competition.  We gave them the TMX-500.

There’s much, much more.  Please let me know if you’d like additional details.

Regards,
=rob.ciampa

Tags: tmx-500 (2) sifma (2) tervela (2) 

Comments (0)

Now that we’ve recovered from SIFMA 2009, I wanted to share some thoughts from the floor.  For accuracy, it’s the 2009 Securities Industry and Financial Markets Association Technology Management Conference & Exhibit.  Whew.  Let’s stick with “SIFMA.”

First off, SIFMA 2009 was different from SIFMA 2008, which was different from all other previous ones as well.  That’s how tradeshows go:  each one has a different personality, much like the children in a family.  Fewer attendees?  Yes.  Third floor of the venue closed?  Yes.  But so what?  We knew this going in, adjusted our plan accordingly and had an outstanding event.  What does that mean?  We had twice the number of scoped project discussions on enterprise messaging systems than we had last year.  A 100% increase, not to mention the number of follow-on meetings.  We didn’t have many tire kickers or “tourists” as a press person remarked to me.  Pete Harris from A-Team Group expressed similar positive observations in his recent post.

Next, we didn’t go there half-cocked with a crap booth and no story to tell.  It’s just not our style.  We introduced the Tervela TMX-500 Message Switch, had the hottest booth at the show (literally or pun intended) and staffed it with our team members who do real-world messaging projects.  I’ll save the TMX-500 discussion for another post, but I was either part of or witness to some of the most poignant discussions on messaging philosophy and architecture that I’ve heard in some time.  That’s why we were there.  That’s why people attended.

Finally, what’s the scoop with this “Wicked Hot Message Sauce” theme?  Candidly, it came to me several months ago during an early, weekend morning caffeine infusion at Starbucks.  I was reflecting on a comment made by an IT executive last year about his desire to “spice up” his application performance.  The rest, of course, is obviously history.  I did take some grief from my New York colleagues about the word “wicked” and my inability to stray from my Boston roots.  Frankly, that wasn’t even in the equation.  I used it in the context of something more powerful than “freakin’.”  I’ll close, however, with some of my Bostonian vernacular:  check out Tervela's TMX-500; it’s wicked awesome.

P.S. To all the great attendees, press, analysts, friends, partners and vendors who stopped by our booth:  Thank you for making the show wonderful.

Cheers,
=rob.ciampa

Tags: sifma (2) securities industry and financial markets association (1) pete harris (2) tmx-500 (2) wicked hot messa (1) 

Comments (0)

But it worked in the lab... A deep and profound (please read it a couple of times) post by Kirk Wylie.  He claims that he's not trying to be down on best effort messaging, but he does have some valid points that we often hear from customers.

Here's a quote:

Best Effort = Development Win, Production Fail

 

Because the problem is that Best Effort systems seem much better than that in development. In development, you can run days without dropping a single message, no matter what size it is. In testing, you can run many test iterations without being able to force a message drop [2]. More importantly, you really can't force a message drop, at least not without implementing a lossy Decorator on top of the MOM interfaces. And that's a problem.

Best effort messaging is meant to be fast.  That's good.  But hardware (with some decent architecture) can make guaranteed messaging fast.  Check out our last post.  The reality is that there are many types and qualities of service when it comes to messaging.  Pick the one that makes the most sense for the applications you're running.  And be realistic.  To Kirk's point: a little bit of diligence in the lab will go a long way to avoiding surprises in production.

=rob.ciampa

Tags: best effort messaging (1) guaranteed messaging (2) kirk wylie (1) 

Comments (0)

Guaranteed messaging.  Guaranteed Delivery.  Durable Consumer.  Reliable.  Blocking.  Non-blocking.  Persistent.  Lots of descriptors attached to the concept of guaranteed messaging.

But what does guaranteed mean?  Where do the guarantees occur?  Is it really a guarantee or is it just statistical?  What price must I pay for a guarantee?  What’s my time window for assurance? How different is a guaranteed messaging flow from a best effort flow?  Are there different types of guarantees?

We’ve spent a good deal of time and R&D dollars on guaranteed messaging.  Though we’ve had for some time a guaranteed connected quality of service that seamless overcomes temporary operational issues, we needed something for major outages and debilitating breaks in continuity.  We opted, however, not to cobble something together that would perform like a sick dog and treat the term “guarantee” lightly.  Our base architecture was designed anticipating the need to have multiple qualities of services that were unified, seamless and fully functional.  Guaranteed messaging was not going to incur a tax.

Today we announced our “Real-Time Guaranteed Messaging Service” - but we claimed it was the “Industry’s First.”  Why?  Because it is both real-time and real-guaranteed.  Together for the first time.  What were our objectives?

• Give customers true assurance without the debilitating deficiencies of traditional guaranteed service
• Be the highest performing guaranteed messaging on the market
• Provide true guarantees, not statistical likelihoods
• Scale cleanly from workgroup to enterprise
• Leverage architectural differentiators that our competitors just can’t match

It’s a different market and information is far too important to implement guaranteed messaging the same old way.  It’s all about the architecture.  That's what drives business advantage.

=rob.ciampa

Tags: guaranteed messaging (2) quality of service (1) durable consumer (1) 

Comments (0)

In a previous post, I mentioned the Options Industry Conference 2009.  Given the importance of messaging to business performance, the conference organizers put together a panel on Friday, May 1, 2009 entitled: The Tech Arms Race (Messaging: The Key to Being Faster).  Barry Thompson, founder of Tervela will be speaking, so no doubt it will be pithy, relevant and opinionated.

Tervela is also at the show in Booth E160.  We'll have copies of our latest research and deployments of messaging in the options markets: Options Advantage Through Infrastructure Transformation - Driving Profitability and Market Leadership.  I'll be posting it to the Tervela site soon.

=rob.ciampa

Tags: options industry conference 2009 (2) messaging (5) 

Comments (0)

The Options Industry Conference 2009 begins today in Florida.  What I like about this event is that it's a great venue for looking at the business and technical aspects of the razor's edge in financial services: options processing.  One of our hedge fund customers summed it up best:

You can't approach options processing with an equities mentality.

I have corollary:

You can't approach contemporary data-intensive business challenges with a legacy messaging middleware mentality.

Some other thoughts:

New business demands and archaic messaging systems.  Massive growth in options market data, amplified by greater order processing and algorithmic demands for low-latency and fan-out are crippling messaging systems, resulting in lost data, information slippage and system failures.  These all equate to position and market exposure.

More complexity = more risk.  Disparate systems drive complexity and, ironically, impact other systems.  Problems in one area frequently affect other areas, making the tuning and debugging complex and dangerously time-consuming.  Because these systems are so tightly-coupled, recovery becomes non-deterministic and predictability goes away.

Adding more resources doesn’t solve problem.  In fact, it creates more problems with complexity, rack space, operations, and cost.  This is the old model: throw more systems at the problem; partition the symbols even more; add more routers and switches; and hope it improves.  The net result is more stuff to manage and more things to go wrong.

Manageability is not an option.  Lack of control and visibility is a legacy shortcoming of traditional messaging systems.  Not having the insight in the critical messaging function is an unacceptable excuse when a firm is out of the market.  You can’t manage what you can’t see.

=rob.ciampa

Tags: options industry conference 2009 (2) messaging (5) options processing (1) messaging middleware (1) 

Comments (0)

So here we are sitting on the dock, watching the sunset and reminiscing about the good old days of Lehman, Bear, and - geez - even Madoff.  We watch the tide go out and get lost in the moment.  In the old days, this would be a full page ad in some retirement magazine.  (Given the state of our 401(k)s, we'll probably have a walker next to us on the dock.) Suddenly, the wind changes direction and the tide starts coming in, awaking us from our respite.

Like the tide, the market is now coming back and firms are suddenly awakening and taking notice.  This time, however, it may be a rather high tide.  Many firms that we work with spent the past several months making some much needed architectural and infrastructure changes - fixing up the dock if we follow our coastal analogy.  Other organizations, well, find themselves a bit behind after wrestling with budget and all sorts of other organizational issues, some of which now seem to be abating.  They must be careful, though, not to find themselves caught off guard, sitting on a distant sandbar while the tide rushes in.  Perhaps it's time to look for a quick return to shore.

I'm not sure what prompted this beach imagery; perhaps it was the unseasonably warm weather in the Northeast these past few days.  I do, however, blame Melanie Rodier and her recent piece "Financial Firms Face Massive Market Data Infrastructure Challenges" in Wall Street & Technology for the tide analogy.  She couples her journalism with some recent research by Adam Honoré at Aite Group.  Here is a snippet:

Market data volumes are expected to continue to grow exponentially, further straining market data infrastructures, according to a new report from Aite Group.

 

If U.S. equities message volumes continue to grow at their current pace, they should average 1.2 billion messages per day by 2011, the Boston-based firm said.

 

Storing all Level 1 and Level 2 data for U.S. equities is approaching a requirement of three terabytes per month.

 

For all asset classes, across U.S. market data rates, data volumes nearly doubled on an annual basis over the last two years.

We'll shortly see if the tide hits the seawall.

=rob.ciampa

Tags: market data (4) melanie rodier (1) wall street & technology (3) adam honoré (1) aite group (1) 

Comments (0)

We recently made an announcement about our joining the Advanced Message Queuing Protocol (AMQP) working group.  For anyone who has worked with standards bodies, this may be akin to a B-grade soap opera sprinkled with politics, protagonists, antagonists and an ever-changing plot line.  I’m sure there will be some of that (though there are some great people involved), but our focus is on the end game: fundamentally changing middleware.  AMQP represents a movement in that direction.  Like comparable endeavors, this is and will be a continual process.  Expect more changes as companies adopt and AMQP is road tested, which, of course, is a normal part of the process.

We’ve made the case here many times against legacy messaging architectures, but it’s just as true with their queuing system alter-egos.  We also continue to argue that messaging has to go through the same critical transition to hardware as network systems did.  AMQP (no pun intended) is queued up for that.  The biggest challenge is not whether adoption will occur, but how quickly it will happen.  That’s an important variable.   Hyper-efficiency and blazing speed may well be the recipe for faster market adoption, but it’s going to take some hardware acceleration to bake it.

=rob.ciampa

Tags: amqp (1) queuing systems (1) middleware (2) 

Comments (0)

Recently, I had the opportunity to attend the quarterly Financial Information Forum (FIF) event entitled “Doing More with Less” at Thomson Reuters in Times Square in New York.  The session opened (in typical FIF fashion) with market data statistics, which were presented by Chris Perry of Thomson Reuters and Manisha Kimmel of FIF.  Recession aside, we we’re still seeing impressive market data growth.  (Can someone please tell the market we’re in a damn recession and stop producing so much data?  Kidding, our course.  We at Tervela thrive on this.)

The follow-on panel was moderated by the esteemed Tom Jordan, President & CEO, of Jordan & Jordan.  Practitioners in the hot seats, whom I compliment on their honesty, included:

• Brett Redfearn, Global Head of Liquidity & Algorithmic Trading, J.P. Morgan
• Madalena Sheehan, Executive Director, Global Wealth Management, Morgan Stanley
• Dan Weingarten, Senior Vice President and Co-Director of Global Sales and Marketing, Penson Worldwide

My takeaway was about efficiency: in how we operate, in how we process data, in how we work with our partners, in how we continue to make our customers happy. Though certainly an important tactic, I’d much rather be an organization that’s already up the efficiency curve.   As AIG and GM are proving, it’s not a great time to retool.  But we also heard how volatility and increasing volumes are having an adverse effect on trading systems.  The panelists described the unerasable link between business and technology.  They discussed their concerns about the risk of technical outages.  Think about the link:  technical outage = business outage. Obvious?  Of course, but just make sure your movement to efficiency doesn’t leave you vulnerable.

=rob.ciampa

Tags: financial information forum (2) fif (1) jordan & jordan (1) market data (4) efficiency (1) 

Comments (0)

The latest issue of Wall Street & Technology (January/February 2009 Issue) came late.  I thought I was dumped because I was honest on the subscription and didn’t say I was buying $3.2 billion of Complex Event Processors.  Regardless, the current issue features a special report on “The Modern IT Organization.”  The lead article and the supporting pieces are a very good read.  There is a good deal of ink on cloud computing, but my comments on that are for another day.  In the meantime, the intro quote was telling:

For the first time in recent memory, senior technology leaders’ No. 1 mandate is not to build technology to prepare for business growth.  Instead the top IT priority across Wall Street is to manage technology efficiently.

But…  having just read the 2008 Financial Information Forum year-in-review that showed the market data for EVERY asset class increasing, I have to wonder whether we’re setting ourselves up for a market data crisis.  Not sure the cloud computing junkies are going to give us a sunny day on that one.

=rob.ciampa

Tags: wall street & technology (3) it (1) cloud computing (1) market data (4) financial information forum (2) 

Comments (0)

Hardware-accelerated messaging is now in fashion!  It’s the new black! 

I’m impressed by the recent rash of announcements on hardware-accelerated messaging, FPGAs, etc.  I decided to let the noise settle down before I chimed in, so now is a good time to look under the covers of this very intriguing topic.

As the innovators in this market, Tervela would like to share a dirty little secret about hardware-accelerated messaging: it’s all about the ARCHITECTURE you accelerate.  Unfortunately, architectural deficiencies in performance abound in message-oriented middleware, queuing systems, ESBs, EAI, etc.  It’s even in networking.  In doubt?  Ask any competent IT architect about requisite data flows and handshakes necessary to post a single event.  Ask a network engineer about the joy of IP multicast management in an environment where it’s actually used by applications, not just routing protocols.

The point is this: you want to hardware accelerate the right architecture.  Applying hardware to legacy frameworks is questionable as a strategic play; even in the short-term it’s tenuous at best.  And, importantly, not all hardware is created equal.  At Tervela, we’ve taken our architecture, our hardware and our software very seriously.  Given recent news, though, let’s talk about hardware.

Tervela was started to overcome critical inefficiencies in legacy middleware and networking paradigms.  We introduced an architecture that represented a fusion of middleware and networking, yet had the flexibility to work with both legacy and emerging applications.

• We separate the data path from the control path to ensure predictable, consistent, high-performance and low-latency messaging.  Control information does not interfere with data.
• A separate data path allows Tervela message switches to readily connect into a resilient message network without sacrificing performance.
• Our data path is implemented entirely in hardware with no need to “go up and down the stack” or suffer an operating system context change.
• Being “network aware” and realizing that underlying network conditions are critical to performance, the Tervela messaging platform dynamically adjusts processing in response to network circumstances and consumption patterns.
• Tervela incorporates different types of hardware depending on the processing requirements.  Concurrent with the operations to be performed, we employ different types of hardware within the flow.  Not all hardware is created equal;  we use both FPGAs and high-performance programmable ASICs where it makes sense and delivers optimum performance.
• Our hardware is programmable, ensuring both performance and functional flexibility.
• In addition to our own hardware, we also utilize advanced, virtualized, multi-processing systems to address the diverse computational needs of our customers.
• We designed our own hardware, employ best-of-breed silicon and are currently finishing our third-generation messaging architecture.

We are thrilled to have other companies join us in the hardware-accelerated messaging space.  For Tervela, it’s all about the architecture, which allows us to quickly and transparently ride the ever-improving hardware innovation curve without having to reinvent our approach. Given that we’ve been doing this for five years, we thought we’d provide some closing insight:

• FPGAs are a means to an end, not an end in itself.  Putting legacy protocols in FPGAs might not be the optimal way to maximize messaging performance.
• If a legacy protocol is inherently inefficient, putting it in hardware doesn’t make it any more efficient.
• If the legacy application is heavyweight, it will remain so.  Rethinking messaging is critical for emerging, high-performance applications.
• The right architecture is about flexibility and the future.
• Designing for performance is nice, but manageability and serviceability are non-negotiable.

=rob.ciampa

Tags: hardware-accelerated messaging (1) message-oriented middleware (2) esb (1) eai (1) fpga (1) 

Comments (0)

Many years ago, I was on a boat on the Cooper River in Charleston, South Carolina with an extremely interesting ecosystem researcher.  He was studying the behavior of aquatic life in the brackish water of Charleston Harbor.  Brackish water is a mix of fresh water and salt water, often found when fresh water rivers meet the salt water ocean.  In my case, it was the Cooper River meeting Charleston Harbor.  Brackish water ecosystems, though apparently resilient (salinity changes frequently), are actually rather fragile.

We have similar challenges in our middleware ecosystems.   We’ve built many systems around queuing models and frameworks, but we’re challenged by massive (and growing) data volumes and demanding low latency requirements.  We’ve built peer-to-peer messaging systems that perform well with a few connections, but deteriorate when scaled. The two don’t necessarily blend.  It appears as if we’ve been unable to sustain a brackish ecosystem of messaging: you either have salt water or fresh water – you either have queuing or peer-to-peer.  Kirk Wylie alluded to this in a recent post on AMQP.
 

We need to embrace both frameworks. We need to support a brackish messaging ecosystem, one that can give us the scale and robustness of queuing systems, but with the hyper performance of peer-to-peer systems.  That was and still remains an architectural goal at Tervela.  It’s simply not a matter of throwing hardware at the problem; it’s about looking at things differently – looking at things “brackishly.”

=rob.ciampa

Tags: message queueing (1) peer-to-peer messaging (1) low latency messaging (1) brackish water (1) charleston harbor (1) 

Comments (0)

Recently, one of our senior systems engineers and I presented the topic of “High Performance Messaging Systems” to a Unix/Linux/BSD user group at the Cooper Union in New York.  Full disclosure: I’m a former Unix kernel engineer whose research was deterministic communication flow.  Admittedly, that was several years ago, but (again full disclosure) I currently run several Linux servers and desktops at home with various types of middleware, queuing systems, applications, etc.  I thought product marketing and strategy work would cure me of my technical tendencies, but alas they have not.

We expected our presentation to focus on the inflection point of when/where peer-to-peer messaging systems (and sockets) deteriorate and should yield to a more formalized message network, but – surprise – many of the questions had to do with server tuning.  This, of course, makes sense because if data producers and consumers are performance dogs, then one might as well put token ring between them. However, tuning end points is still a bit art and still a bit science: network interface card selection, interrupt coalescence, thread management, etc.  Sorry for the technical complexity but the point is this: if end systems are the bottleneck, fix ‘em first.  The bottleneck will naturally move to the middleware and to the network.  That’s the new battleground (which bandwidth won’t fix) and a discussion for another day when we look at holistic end-to-end performance.

For the curious: when do peer-to-peer system begin to break down?  Around 5 end points.  No kidding.  That, too, is a discussion for another day.

=rob.ciampa

Tags: linux (1) unix (1) bsd (1) bottleneck (1) cooper union (1) 

Comments (0)

No one would have crossed the ocean if he could have gotten off the ship in the storm.

Tags: guy lombardo (1) john maynard keynes (1) charles f. kettering (1) 

Comments (1)

Talk about a buzz kill.  Wall Street layoffs.  AIG executive incentives. Troubled Assets Relief Program (TARP). Bernie "Alleged Ponzi" Madoff Investment Securities. Bear Stearns.  Lehman Brothers. Washington Mutual. Lions. Tigers. Bears. Oh my!

Time for a reality check, please: the markets ARE NOT shutting down.  In fact, there is a great deal of business happening and some folks are actually making money.  It is good to see that there are some level-headed people who don't have their heads buried in the sand and are looking objectively at markets, technology and opportunity.  Let's review some recent, notable analysis and coverage.

A recent piece by Leslie Kramer in Wall Street & Technology captured the essence in its title and subhead:

Demand for Low-Latency Infrastructure in Financial Markets Continues Despite Current Turmoil. There is a race underway in the financial markets, with technology vendors and service providers vying to offer the lowest possible latency in the delivery of market data and trade execution information.

Kramer comments on recent research by Rik Turner of Datamonitor.  Turner states "...the current volatility in the markets actually makes low-latency infrastructure even more critical"  The gist is this: current market volatility underscores the need the technology, especially for systems that remove performance impediments.  (It's now moving across more asset classes...)

Another great piece was from John Barr of the 451 Group in his report "2009 preview - Financial Services IT."  John states:

2008 was (hopefully) a once-in-a-lifetime event in financial markets. In 2009, we expect the markets to pick themselves up, dust themselves off, and start planning for the future. Consolidation, cuts and cost savings have made recent headlines – but business goes on, the value of being competitive has not gone away, the low-latency race is still being run, and the European Multilateral Trading Facilities (MTFs) must compete on both business model and technology if they are to survive to the end of 2009.

Poignant. Pithy. Think about it.  When we get through the current market challenges - and we will - the firms that will lead are the ones taking action NOW.

=rob.ciampa

Tags: low-latency infrastructure (1) wall street & technology (3) datamonitor (1) the 451 group (1) 

Comments (2)

There is a rather interesting event in Denver this week called the Defrag 2008 Conference.  Check out the agenda - much discussion on content, information, flow, computing, applications, etc.  I received several commentaries over twitter from Ian Glazer, senior analyst with the Burton Group.  One I found particularly poignant.

We are in an asymmetric conflict with information, fighting a modern fight with cavalry and sword.

Profound? Yes. Obvious? When we consider it. Actionable? I'll let you know when I'm caught up on email, blogs, magazines, books, whitepapers...

=rob.ciampa

Tags: defrag 2008 conference (1) ian glazer (1) information conflict (1) 

Comments (0)

The Rob Daly piece from Waters that I referenced last week really got me thinking.  I responded to the piece in Waters and will reprint it below, but the topic has been a source of ongoing debate. Recently, I was out with several customers where we spent dinner discussing the physics of information movement within complex systems.  The gist was that we continually tweak our market data systems to ensure low latency, yet we just don't have the time (or the insight) to look at it more systematically.  This is a problem.

The Waters Letter

Rob Daly's September column, "Stop the Latency Insanity," is a harbinger for a much broader discussion on a systematic approach to addressing and analyzing latency across the entire trade-execution spectrum. The ever-elusive latency equation is inherently complex and consists of diverse variables whose individual characteristics are constantly changing. Mr. Daly covers several of these, including network bandwidth, protocol stack bypass, application architecture, and so forth, but we need to look at the entire chain, not just the pieces.

Unfortunately, we struggle as an industry to fully grasp the entire latency spectrum. Why? The reasons are as diverse as the variables. Trade execution spans departmental and organizational boundaries, preventing integrated and holistic analysis; latency tools only cover a limited number of discrete systems; volatility and the non-deterministic nature of market data inhibit accurate modeling; algorithmic execution generates non-deterministic computational and network loads.

The semblance to chaos theory is warranted. We're dealing with complex, inter-related, dynamic systems, which is why latency is so hard and-stealing from Mr. Daly's point-insane. While chaos theory puts much emphasis on initial conditions (market open), today's trading ecosystems introduce more stimuli all the time (Fed announcements, earnings reports). The mathematical modeling of latency around chaos theory may be a good concept (and a reasonable PhD dissertation), but it's just not pragmatic in the real world.

That shouldn't, however, stop us from attempting to solve the latency equation-which also needs to include stability, predictability and scale. There isn't a choice, especially in today's markets. Not having a grasp on broader, systemic latency will not only put our trading systems at risk, but erase any market advantage as well.

=rob.ciampa

Tags: low latency (4) chaos theory (1) 

Comments (0)

What a way to end the month of October 2008.  Let's look at the VIX, Chicago Board Options Exchange Volatility Index.

=rob.ciampa

Tags: vix (1) volatility index (1) 

Comments (0)

Rob Daly of Dealing with Technology fame recently wrote a piece in Waters Magazine about the incessant search for faster trade execution and our obsessive quest for the holy grail: low latency.  He admonishes us to "Stop the Latency Insanity."  Rob then goes on:

But please, people, get a grip-there is only so much that physics can do with most firms' existing architecture...While reducing latency is an ever-present concern for firms, it must be addressed in a rational manner, balancing the potential return with its potential costs.

I agree and I disagree.  Perhaps that's an indication of insanity.  Perhaps there are more ways to look at the latency problem.  Perhaps we'll continue this discussion.

=rob.ciampa

Tags: low latency (4) rob daly (0) trade execution (0) insanity (0) 

Comments (0)

I really like industry events: the intensity; the competitiveness; the pithy debates; the BS; the incredible concentration of opportunity. The good ones have the pulse and cacophony of a third-world bazaar. The evening parties aren’t too bad either, especially when they’re in New York or Las Vegas.

Fortunately, we had High Performance on Wall Street 2008 in New York this week. It was at the Roosevelt Hotel, La Grande Dame from La Belle Époque. I felt out of place without my frock suit and top hat, but mentally returned to the 21^st century and enjoyed the healthy debates on in-line risk calculations, low latency, complex event processing, co-lo architecture, etc.

Candidly, with all the market animation this week, it was tough to gauge what the atmosphere would be going in to the event. With rare exceptions (“Hey, where’s the Lehman dude on the panel?”) things went rather smoothly. Hats off to Pete Harris for keeping things lively and moving.

I was able to catch some very good panel discussions as well. I’m always leery of panels because … well … many (regardless of the event) are truly awful. Often panelists are inarticulate, rambling or just too timid to have an opinion. And moderators? They’re running the show but often forget their role and responsibility to us not seated on the dais. Far too typically moderators go through the panelists in sequential order asking softball questions. For most of the audience this proves (often after the ubiquitous high-carb lunch) a quick route to slumberland. When I’m in the audience, I want to be entertained and mentally challenged. I want Jerry Springer meets Harvard. Hey Mr./Ms. Moderator: pick a fight, force a debate, enlighten me and stop wasting my time.

Off my soapbox and back to the panels (because this was not much of an issue at this week’s event). I very much enjoyed the session “Gaining First Mover Advantage with a Low Latency Market Data Solution.” I skipped the concurrent “Low Latency Market Data Distribution – Software and Hardware Approaches” panel because I work with Barry Thompson and get to eat, drink and debate with him all the time. What struck me about the first panel was a very poignant comment made by an excellent panelist. He commented on the market obsession with low latency and the collective failure to account for devastating effect of outliers – late, non-low-latency information. Algos just don’t like outliers and it screws many of them up. Outliers are low-latency impurity. 99.44% pure is good for Ivory Soap, but it’s not good enough for today’s environments.

This subject deserves much more scrutiny. Outliers undermine low latency. Maybe a debate on this at the Roosevelt next year?

=rob.ciampa

Tags: low-latency (1) outliers (1) pete harris (2) high performance on wall street (1) 

Comments (0)

Every time I hear the phase “OSI Stack” I cringe. Perhaps it’s residual animus from my youthful days at Apollo Computer where those of us doing “second-rate” protocols like TCP/IP and SNA were mocked by elitist colleagues working on the Open Systems Interconnect model. I’m surprised they had time to look down upon us, especially given the demands of going to Europe to attend all the standards meetings. I sat in a dark office, reading RFCs, writing code, drinking crappy coffee and trying to make stuff work over this other “inferior” thing called Ethernet. My colleagues, meanwhile, were sipping espresso on Lake Geneva.

Amazing how time changes everything. My memories were stirred by a recent article in Network World by Steve Taylor and Jim Metzler “Why it's time to let the OSI model die.” I think it’s been six feet under for some time now, but I still think Steve and Jim made their new 3-layer model too complex. Perhaps it’s only 2 layers: application and network. We know company XYZ has an application group and a network group, don’t we? And, like two siblings, those two groups always get along, don't they? Maybe we do need a seven layer model again, somewhat like The Waltons of technology. And they may get along better, too. I’m imagining my new world… “Rob, I think you’ll need to speak with my dear friend in the layer 6 group. I think he’s drinking espresso in the cafeteria with the layer 1 and 3 guys.”

OSI. It’s dead. Go bury it on Walton's Mountain. So, where's middleware? Layer 2.5. At least it's stillbetter than 3.

G’night John-Boy.

=rob.ciampa

Tags: osi stack (1) network world (1) the waltons (1) espresso (1) middleware (2) 

Comments (0)

I had the opportunity to attend a great panel discussion at this week’s Gartner Event Processing Summit 2008 in Stamford, CT. It was moderated by Roy Schulte, Gartner VP Distinguished Analyst and included not just the usual suspects in the messaging space (read software), but some of the newer players (read hardware acceleration) including (gratuitous reference) Tervela. Roy was even-handed, yet challenged all of us on our solution approach, differentiation and target market.

So, what was the motivation for this panel entitled Ultra-Low Latency Messaging Panel Discussion? Events, messages and low-latency communications are critical components of what we do. Roy commented that events are becoming more widely deployed in diverse business settings. More importantly, those events have increasing time sensitivity. Messaging, of course, is the foundation for event processing now and moving forward.

So, what was my take? Aside from the typical “some speakers were good and some were less so,” one thing was clear: the world of messaging is splitting into software and hardware. To say one is better than the other would be naïve – and we don’t even position ourselves that way. But one thing we do know: when the volumes start getting high, software systems just can’t keep up. (When was the last time you used a software-based LAN switch?) Is this new? No, but with the rise of events and their underlying messages, we’re starting to redline the performance tachometer.

Where is this all going? Just look to the financial services industry; it’s been a rather good harbinger for the broader enterprise.

=rob.ciampa

Tags: event processing (1) ultra-low latency (1) gartner (1) roy schulte (1) 

Comments (0)

My phone started ringing early this morning.

“The LSE is down,” said the voice on the line.
“You’re kidding me,” I responded. “Please not today. The Fed announced the Fannie Mae and Freddie Mac rescue. Great day to be in the market.”
“Exactly. Lots of volatility.”
“Any idea of the cause?”
“Nope. It’s been called a ‘connectivity failure’ by the LSE.”

Lots of questions around this outage. I’m sure the LSE is not too happy either.

From the Wall Street Journal:

The connectivity failure by the London Stock Exchange Group PLC, the exchange operator, is an "absolute nightmare," said one trader. "We are just not trading on what should have been a big trading day."

Was it an application failure?
Was it a server crash?
Was it a network failure?
Was it a middleware issue?

We’ll get more information over the coming days, but rest assured we’re seeing the results of building complex trading ecosystems with interdependencies that have their own, often significant risk surface areas.  It's not just an LSE issue.

=rob.ciampa

Tags: london stock exchange (0) lse (0) outage (0) 

Comments (0)

A recent article by Fayazuddin Shiraz in Chief Executive turned me onto a post by Jonathan Schwartz, CEO of Sun, lauding the company’s recent announcement of 1 million messages per second for RMDS (Reuters Market Data System). Congratulations to Sun and Intel on the worthwhile collaboration. There is, however, an interesting implication: what happens to the network when you source such a vast amount of market data to a large number of very hungry electronic consumers? (Who may, of course, generate derivative traffic.) Rhetorical question perhaps, but this volume of messaging data can destroy even the most well-provisioned infrastructure regardless of the bandwidth ceiling.

Some time ago (in the pre-steroids era), we had message traffic flowing on a separate network. It was the right thing to do. Later, that network was collapsed into the “new and improved” high performance switched infrastructure. All the data lived happily together for a while – until message volumes went up and multicast trashed the network. And everyone screamed about low latency. Now we’re seeing the pendulum swing back to separate networks for messaging again. We don’t have time to manage one network and soon we’ll have two.

=rob.ciampa

Tags: rmds (1) market data (4) low latency (4) 

Comments (0)

I was reviewing some notes from a discussion I had with Kevin McPartland from Tabb Group. We talked about messaging volumes in the global equities and options markets. His numbers were an astonishing 7 billion (Yes, Virginia, that's a "B") messages per day in 2007. More interesting, though, were the projections: 128 billion messages per day by 2010. Wow. And low latency, too.

=rob.ciampa

Tags: message (2) messaging (5) low latency (4) 

Comments (0)

Marc recently had a very good post on his Magmasystems Blog regarding a recent announcement by a legacy messaging vendor that they were entering the hardware game. As a hardware vendor in the messaging space, we welcome the market validation but we have several questions:

Why is this not akin to a John Kerry flip flop?

Most recent noise on the The Street had said vendor saying no, no, no to hardware. We, of course, argue that it’s inevitable. Look at what the networking world taught us.

Should this legacy messaging be hardware accelerated?

It’s not about messaging; it’s about a messaging architecture that supports acceleration. More lessons here, this time from the automobile world. Shelby SuperCars introduced a 1,183 hp rocket last year that hit 257.41 mph on the road. It wasn’t just about the engine, though. The rest of the car was architecturally matched to it. Putting that engine in the classic Toyota Camry could produce much different – and possibly dangerous - results. You be the judge.

Will this make other problems go away, too?

Again, it’s about architecture. Nothing like a slow messaging consumer to bring down a peer-to-peer framework – even one that’s hardware accelerated. You need a new architecture to fix this. Period.

What about all this XML stuff?

I’ll answer with a question: How prevalent is XML in the high-performance world? Before anyone barks – I’m an XML fan – but for the right applications.

Welcome to the high performance world of hardware. May your engine bolts be fastened properly and your drive train sufficiently matched.

=rob.ciampa

Tags: hardware-based messaging (1) tibco (1) xml (1) shelby supercars (1) 

Comments (0)

OK, we were a bit unorthodox in our approach at SIFMA. When you’re dealing with 7,000 attendees and 300 vendors, you have to be. Perhaps our booth felt like the Haight-Ashbury district of San Francisco in 1967, but we have a big issue with “the establishment” – the messaging establishment. What we’re calling into question is the architecture of legacy messaging systems, message-oriented middleware, etc. Are we stuck with messaging frameworks that are about as contemporary as the tie-dye VW bus? In doubt? Ask a messaging architect about the last outage with a software-based messaging solution. The new revolution is about hardware, architecture and rethinking messaging. It’s time for the message network.

=rob.ciampa

Tags: message network (1) message revolution (1) messaging systems (1) message-oriented middleware (2) haight-ashbury (1) 

Comments (0)

Welcome to the Tervela Blog. In our sanitized world of corporate communications, we often have to “play within the lines.” Corporate blogs in particular scare us. Why? Because they tend to be regurgitated marketing speak. If people want that, they’ll go to our web site and get it undigested. So why are we doing this? It’s simple: we believe the messaging industry is at a critical inflection point – and many have strong opinions that we want to discuss and debate. We’d be foolish not to, especially given how strongly we believe messaging is the foundation for future computing platforms, SOA, Web 3.0, etc… (That wasn’t a typo: you be the judge on Web 2.0). Thought the messaging card was played in the 90s? That was the pre-season. You ain’t seen nothin’ yet. We look forward to your thoughts and opinions.

=rob.ciampa

Comments (0)