When the Chicago Board Options Exchange (CBOE) opened its doors in 1973 it was the first institution in the United States to offer standardized stock options trading. On its first day of trading, it processed 911 contracts on 16 stocks. Two years later, it introduced computerized price reporting. It was a rather prescient move, as otherwise keeping up with its rapid expansion would have been difficult.
In 2005, the CBOE processed 468,249,301 contracts on nearly 2,000 stocks, indexes, and funds. This is a 30 percent increase in the number of transactions compared to the previous year. In the past three years alone, the number of price quotes grew nearly 3,000 percent, from 11 million a day to 320 million.
Making this high-volume possible is a trading system called CBOE Hybrid, which rolled out in June 2003 running on an Amdahl (now Fujitsu) mainframe and eight Fujitsu PrimePower server clusters.
"We have had great success with this system, which has allowed us to keep up with the crazy growth in capacity we were up against," says Curt Schumacher, the CBOE's CTO of systems operations.
Knocking Out the Delays
CBOE Hybrid is the latest in a long line of systems the CBOE has developed over the years to automate trading. In 1984, CBOE launched its Retail Automatic Execution System (RAES) to facilitate electronic order execution; in 1993, it introduced hand-held terminals on the trading floor; and in 2001, it started CBOEdirect, a screen-based trading system initially used for extended-hours trading.
"We were already very electronic, but as competition grew, we went out and built this system called CBOEdirect based on Solaris, Java, CORBA, and other technologies," says Schumacher. "As we were developing it, we built it on the Sun hardware platform."
The system worked well initially, but as the number of traders using it grew from 50 to 800, the system started experiencing capacity problems. The CBOE continued to grow the system horizontally, but that wasn't enough to ensure service levels. According to Schumacher, options traders are price shoppers and thus monitor prices at several exchanges. If they can get a better price from another exchange, that exchange gets the contract.
"That is why it is so important that there be no latency or delay in price quotes and the market input to our books," he says. "To remain competitive, we have to be able to handle more quotes and trades, faster than any of our competitors."
This led to the creation of CBOE Hybrid, a system that marries the open outcry system used on the trading floor with the CBOEdirect system. It continued to use the Amdahl S/390 mainframe for order routing, running the IBM Transaction Processing Facility (TPF) code. But the CBOE wasn't satisfied with Sun's road map at that time, so it began looking at other vendors.
"Sun's product line was not offering much in terms of expansion at that point," says Schumacher. "So we got back with Amdahl which turned into Fujitsu and they came to us with PrimePower."
Since PrimePower servers run Solaris, it was easy for CBOE to port over its existing code and start using the new servers. Another thing Schumacher likes about the Fujitsu servers is their clear product development plan, which enables the CBOE to upgrade to faster processors as they become available.
CBOE uses primarily PrimePower 650 servers in its clusters, along with PrimePower 250 and 1500 servers. PrimePower is a line of Unix servers powered by the SPARC64 V processors. The PrimePower 250 is a workgroup server with up to two processors. The PrimePower 650 is a midrange 8U server with up to 8 processors and 64 GB of main memory. The PrimePower 1500 is an enterprise-class pedestal server with up to 32 CPUs and 256 GB of RAM. The eight clusters are spread throughout the Exchange's 45,000 square foot trading floor, servicing about 5,000 clients.
"We went from a 30-server architecture in June 2003 to 290 servers in our CBOEDirect system," says Schumacher. "90 percent of them are from Fujitsu."
The traders have two options for accessing and entering data on the trading floor. One is to use a Fujitsu Stylistic ST5000 tablet PC. Schumacher says these are preferable to standard notebook computers since their light weight and small form factor make them ideal for working in crowded sections of the floor. These handhelds are serviced by a 5 GHz Cisco Aeronet wireless network called Mobile PARs. Before rolling out CBOE Hybrid, about 800 traders were using the wireless system. That figure is now down to a several hundred since the Hybrid model requires more bandwidth. Consequently, many of the brokers have gone back to using wired terminals. For remote access, brokers can use PrimePower 250 servers at their own offices.
Once the order is entered it passes to the Amdahl mainframe, which routes it to one of the PrimePower clusters. These clusters then interact with another PrimePower server to process the order.
As it is for other financial institutions, it is exteremly important CBOE maintain secure backups of all data. In this case, the transaction data is stored in a data warehouse on an IBM pSeries Server running AIX.
"We built our big data warehouse on the IBM pSeries due to the high-performance backplane," Schumacher says. "We needed a lot of throughput."
From there, data passes to a three-tiered storage system Tier One is an EMC Symmetrix storage system, Tier Two is an LSI storage system (now Engenio Information Technologies), and Tier Three is a StorageTek tape system for archiving. These, together with hardware from Hitachi Data Systems and Sun, contain more than 600 TB of data.
The CBOE hybrid system has worked well so far, but further capacity is needed. Given the way peak demand keeps expanding, from 2,000 quotes per second to 40,000 quotes per second during the past three years, Schumacher is already looking ahead to the next generation of hardware. The Amdahl mainframe will be supported only for a few more years, and the PrimePower servers stop with the 2.16 GHz processor. CBOE hasn't upgraded to these CPUs because they don't offer a significant enough performance boost to justify the cost. Instead, Schumacher is waiting to see what will happen when Sun and Fujitsu merge their server lines into the Advanced Product Line (APL) later this year.
CBOE is also using some IBM computers running SUSE Linux, but Schumacher would prefer to stay with Solaris so the code need not be ported to a new operating system and he can continue to leverage the expertise gained over the years.
"We are very interested in the whole AMD Opteron line, and we have brought someone in from Sun and like their Solaris 10," he says. "I am not against using Linux, but we have the expertise and experience with Solaris."
Plus, he values the support he has gotten and continues to get from Fujitsu, so he would like to run Solaris 10 on Fujitsu's Primergy line. Primergy is Fujitsu's Intel-architecture-based server line. It includes blade, rackmount, and pedestal servers with Xeon, Itanium, or Opteron processors.
"We have been meeting with Fujitsu, and I encouraged their president to certify Primergy on Solaris 10," he continues. "We like the support and the whole relationship we have built with Fujitsu. It has been very successful, and we look forward to continuing that relationship as we move to our next generation of services."