Properly Evaluating Trading Systems

Due to the exponential growth of both the futures and personal computer industries, the trading system software business has exploded over the last 20 years, with any number of programs available to the public at prices ranging from under a $100 to over $15,000. Unfortunately, this expansion has not been accompanied by an increased availability of unbiased consumer information for potential buyers.

The software in question is not charting or technical analysis varieties common to most traders - Tradestation, Supercharts, Metastock - but rather systems that tell someone when to buy or sell a particular commodity automatically, often referred to as a "Black Box" system. There are numerous systems for sale that are in fact solidly researched, contain statistically valid trading ideas, and offer users a high probability of making impressive returns in the futures markets.

Although any systems profitability can change in different market environments, other systems have little value. How does the public differentiate virtual reality from truth? Fact from fiction? Robust performance from misleading hyperbole?

A company called Futures Truth in Henderson, N.C., has carved out a niche for itself as "The Morningstar" of the futures industry. The industry's only independent third party system tester, testing thousands of publicly sold systems over the past decade. (Its ranking of the top performing systems appears every other month in Futures' Computerized Trader column.) Founder John Hill started the company in 1985 noting that "there was a crying need in the industry" for such a service. The amount of information makes it difficult for buyers to know with any degree of certainty if a system has merit before purchasing it. Vendors can make outlandish claims without backing them up, ultimately to the detriment of the industry as a whole, and more importantly to those vendors who do sell viable trading systems.

A Paradigm Shift. In the early 1980’s, pure black box systems often sold for thousands of dollars, a true leap of faith for someone who doesn’t even know why he’s being told to buy or sell. As software consumers have become more sophisticated and savvy, prices have come down and more vendors have turned to selling "open" systems in which customers are disclosed how the system works. George Pruitt, the System Analyst at Futures Truth, estimates that 90% of the software he reviews is completely open, as is 70% of the software available to the public. "When you say it’s a black box, that reduces the potential sales that you’ll make. A lot of people are scared, they don’t know how the system handles a drawdown."

While some vendors test their systems on real-time data, others derive performance statistics from the same backtesting methods used by most other traders who develop their own methodologies. Historical prices are fed into the system, and the computer spits out how the system would have performed over a given period of time. Obviously this method presents some problems. First, it is susceptible to "curve-fitting", that is, adjusting a systems parameters in hindsight to compensate for losing periods (drawdowns) or other anomalies to create a system that looks amazing in test results. Unfortunately, over optimization of this sort almost inevitably results in a system performing very poorly in real-time trading.

Second, the time period used to test the system will affect its simulated performance: The shorter or more selective the time period, the easier it is to generate positive results. The type of test data used also will color results. Continuous data, for example, changes the impact of real-life market events like rollovers that impact a systems performance.

Adding to the problem is the dilemma a trader faces after he buys a system. Purchasing a black box system may seem like a perfectly logical choice for someone with a full-time job, but when it comes down to trading, many people find it impossible to put their faith and money into someone else’s ideas. To make money with a mechanical system, you absolutely have to follow it religiously, and for most people that’s just not plausible. Jack Schwager, author of the renowned "Market Wizards" books, remarked that "he thought it was highly improbable to make money buying someone else’s trading system because typically the buyer loses confidence during a drawdown period and abandons the system entirely."

Some systems have modifiable parameters enabling users to customize the system to a degree, but this element of discretion also can lead to frustration on the part of both users and vendors. Unfortunately most buyers don’t want to think. They want to work their regular eight hours a day job and have the system turn $10,000 into $100,000 overnight. Their performance expectations are totally unrealistic and the results are usually dismal.

People who purchase mechanical systems must have the discipline to trade a "second hand" approach and accept total responsibility for any element of discretion a particular system may require. Even a solid system won’t work if it’s misapplied or not followed.

System Test Requirements. The success of any computer trading program depends on the following 5 necessary requirements:

1. Long-Term Historical Testing - Any trading idea method or strategy must be verified over long periods of previous data, to ensure that the strategy is sound. Test periods of 3 months, 6 months, or even 2-3 years are simply not long enough. A minimum of 5 years is necessary to build a successful model. Test periods greater than 5 years will further increase the future reliability of the method.
   
2. Real-Time or Out-of-Sample Testing - After the Historical Testing has been completed, and potential variables have been selected, the method must then be tested on new data going forward in "Real-Time" or at least other data not included in the Historical Test sample data. If Real-Time tests yield favorable results, then the method can be relied on to trade with real money in the markets.
   
3. Walk-Forward Testing - This technique is a rigorous test of a method’s validity. Walk-Forward Testing is very similar to programs that use Artificial Intelligence or Neural Networks. This method tests a moderate time period, such as 1-2 years, and automatically comes up with a good set of variables to use on the next contract, which is new or "Real-Time" data. A trading method that produces profits under these test conditions is very valuable, and can be used in the markets with a high degree of confidence.
   
4. Limited Number of Conditions - Any trading method could be 100% accurate, and display impressive profits if an unlimited number of system rules, or conditions are used. Theoretically, a different rule could be used for each trade, thus guaranteeing success during historical testing. However, the method would almost certainly fail on tests of new or "Real-Time" data. Therefore, it is important that the number of rules or variables be kept to a minimum.
   
5. Dynamic Variables - For every rule that a trading method uses, there is a potential range of numbers or variables that can be associated with the rule. A Higher Close Pattern Rule could be based on a moving average of either the last 1,2,3,4 ...8, 9 ...etc. Closes. Although many of the numbers or variables will work better than others, if only a few of the variables work, then the method is fragile, and may not hold up under rigorous and changing market conditions. All successful trading methods must have rules that can be incremented over a wide range of variables, and continue to produce profitable results.
   
6. Where To Turn? The important question for many is whether a system lives up to its claims. Will system X really return 100% in 12 months? One source of information is Futures Truth. In a bi-monthly newsletter, Futures Truth tracks the performance of anywhere from 150-200 publicly offered systems in its Master Performance Table and provides detailed performance characteristics of individual systems.
   

Futures Truth employs a standardized testing methodology using a proprietary software program and tests at least a decade of prices using actual contract data to give the most realistic portrayal possible of a system. It signs a non-disclosure agreement with vendors and test contracts initially selected by the vendors which cannot be altered for the remainder of the tracking period.

Users must take into consideration the leverage and money necessary to trade a system properly. It might be possible to make $100,000 with a system but only by accepting unreasonable imprudent drawdowns. To get a realistic feel for how a system with only a hypothetical track record might actually trade, you should expect real-time returns to be approximately half the hypothetical returns and real-time maximum drawdown to be twice the hypothetical maximum drawdown.

What goes into the software out on the market? Although its disclosure agreement prevents it from revealing the logic of any system it tests, Futures Truth has been in a position to find out the most used indicators and techniques used in their systems. Not surprising, they are tools most savvy and sophisticated traders are familiar with. They generally fall into some variation of a trend-following or breakout system. There are certain standard parameters that exist in almost all trading systems.

Alternative Sources. A few other groups have emerged in response to the void of credible trading system information sources. One of the more notable is Club 3000, a newsletter edited by Bo Thunman in Augusta, MI. Thunman was trading stocks, options, and futures for himself when he started Club 3000 in 1981 as a user group for a particular piece of trading software. From a circle of 10 members who traded information on one piece of software, Club 3000 has grown to over 1500 members including traders, analysts, and system vendors who discuss a wide range of compelling trading topics such as new systems, advanced software products, and trading psychology.

The main forum is the Club 3000 newsletter in which Thunman assembles various commentary from members in an open discussion format. Members candidly praise or criticize systems; vendors respond and promote their services. It makes for a very lively and healthy exchange. Thunman also provides the names and numbers of persons who are interested in sharing information about a particular piece of software so traders can directly contact others with their first hand direct experiences.

Other sources include Commodity Traders Club News based in Flagstaff, AZ. (Editor Dave Green also writes and sells his own trading programs). Two other system developers who promote and sell their systems are Joe Krutsinger and Murray Ruggiero’s firm, Ruggiero Associates.

The internet is another valuable information source. A number of new financial and trading orientated services and news groups have formed recently, and although most are still in the embryonic stages, net surfers can still find useful information if they’re willing to sit through promotional materials from system vendors and other industry professionals.

It’s Well Worth It. Consumers who take the time to properly educate themselves will clearly have a competitive advantage in terms of finding rigorously tested, reliable and profitable trading systems.