ML Newsnew | past | best | rss | submit about
Mistakes when Applying Computational Intelligence & ML to Stock Market Modeling (arxiv.org)
73 points by jnazario on Nov 12, 2012 | hide | past | pdf | 15 comments on HN

In plain words: A review of stock-prediction projects pinpoints five recurring errors — too little data, wrong scaling, mishandled time order, vague targets, and misleading scores — and shows how each skews results. Correcting these steps is the way to make such market models trustworthy for real trading firms.

Abstract · Common Mistakes when Applying Computational Intelligence and Machine Learning to Stock Market modelling

For a number of reasons, computational intelligence and machine learning methods have been largely dismissed by the professional community. The reasons for this are numerous and varied, but inevitably amongst the reasons given is that the systems designed often do not perform as expected by their designers. The reasons for this lack of performance is a direct result of mistakes that are commonly seen in market-prediction systems. This paper examines some of the more common mistakes, namely dataset insufficiency; inappropriate scaling; time-series tracking; inappropriate target quantification and inappropriate measures of performance. The rationale that leads to each of these mistakes is examined, as well as the nature of the errors they introduce to the analysis / design. Alternative ways of performing each task are also recommended in order to avoid perpetuating these mistakes, and hopefully to aid in clearing the way for the use of these powerful techniques in industry.

E. Hurwitz, T. Marwala
arXiv:1208.4429 · stat.AP, cs.CY, q-fin.GN · submitted Aug 22, 2012
abstract · pdf · 5 pages

add comment on HN
Also discussed: Mar 2015 (1 point, 0 comments)
gfodor on Nov 12, 2012 [–]
The paper focuses on a number of practical mistakes but overlooks some higher-level ones that I think are also important:

- Most ML algorithms perform well when there is some underlying phenomenon whose characteristics are being statistically inferred by the model. However, the stock market is now essentially a large network of computers trying to model one another. You can imagine how this might break some of the underlying assumptions that good ML results rely upon. We can see the results of this in the increasing frequency of "flash crashes" caused by over-leveraged quant hedge funds all tripping over sell triggers and/or getting margin calls at the same time.

- In practice, asset class/trading strategy correlations tend to change dramatically over time. Nowadays we talk about the market being 'risk-on' or 'risk-off' modes, since we're used to seeing market-wide selloffs and buyins. Your ML model can only do so much if the entire market is selling everything, unless you're going to dip your toes in derivatives or short selling to which I say: good luck! :)

- A major, major issue is execution. You can have the greatest and most accurate model in the world, but actually trading it is another beast entirely. Bid/ask spreads and market movements from your trading, particularly if you are dealing with a non-trivial amount of money or trading securities that have less than ideal liquidity, is usually going to eat up any alpha you might have. In my own backtests of fairly straightforward trading algorithms even a minor 0.05% spread on bid-ask spreads for a weekly trading algorithm can eat your lunch, nevermind if you're planning on doing intra-day trading or trading anything other than the most popular funds/stocks.

- Beyond any of these risks, you're going to have to inevitably suffer through downdrafts. I don't know about you, but watching my money disappear as it's being controlled by a trading algorithm/model that is subject to all kinds of mistakes and bugs is well beyond my own intestinal fortitude.

reply
ChuckMcM on Nov 13, 2012 | parent [–]
"Most ML algorithms perform well when there is some underlying phenomenon whose characteristics are being statistically inferred by the model. However, the stock market is now essentially a large network of computers trying to model one another."

This is one of the insights that I think deserves more ink than it gets. Historically system's analysis has focused on predominantly linear systems where underlying factors that were unseen but capable of being modeled as FSM combined into emergent behaviors which were more complex but ultimately still linear across the region of analysis.

That sort of analysis falls down however when the modelling system is a part of the system it is modelling. The result can easily become non-linear (or turbulent or chaotic depending on when you were introduced into systems analysis :-)) and the calculus for those conditions is a lot harder to tease out.

The feedback algorithms become signals to other feedback algorithms and you get hard to predict changes which don't track the measured data, they track the response to the measured data, which is itself changing in response to the response.

From a practical viewpoint (for me) its of interest in discovering entities that use ML techniques to game search engine rank which is itself being derived by ML algorithms.

reply
gfodor on Nov 13, 2012 | parent [–]
Indeed -- it opens up a (scary) need for models to be "self-aware" of competitors' models and their effects on the training data. Above my pay grade, for sure.
reply
redwood on Nov 13, 2012 | parent [–]
Great point... I've been thinking lately how it must be possible to hack the systems now. E.g. there must be machine analysis of press coverage on equities, and the machines must deduce, quickly, "positive" or "negative". This means fake stories have a lot of power. Now this was always true, of course... but traditionally you had humans vet what seemed legit. Now I bet you can keyword stuff articles, among other things, to shift the market... etc who knows but sky's the limit.

Total BS mess is what this has become

reply
ramana10 on Nov 13, 2012 [–]
I work in a quant fund. Can I just say that the longer I spend in the industry, the more I find that ML/AI techniques are useless (in general). It almost seems like collective self-delusion in building ever more complex systems that attempt to use the latest fad in the field without first appreciating that your dataset is almost always just noise...

Me, I just do my boss's bidding like a good little solider and code up whatever the fund wants. My personal portfolio is a vanilla asset allocation model. Guess which one has done better for the past 2 years?

reply
tzs on Nov 12, 2012 [–]
There was a really nice example of a mistake applying ML to currency exchange rate forecasting given in the "Data Snooping" section of Yaser Abu-Mostafa's "Learning from Data" course at Caltech (http://work.caltech.edu/telecourse.html).

Here's the relevant lecture: http://work.caltech.edu/library/173.html (the currency exchange example starts around 51:38).

reply
dave_sullivan on Nov 13, 2012 [–]
Re: the insufficient data sets problem, I've been surprised how hard it is to find large quantities of historical financial data. Decent sources of the data seem very expensive, and I'm surprised there isn't a startup out there that is tackling this well since I'd be willing to pay a decent amount of money for minute by minute or hour by hour tick data--but am finding most subscription prices crazy. Is this data really as hard to come by as it seems?
reply
jayro on Nov 13, 2012 | parent [–]
Here are two relatively inexpensive options for tick-by-tick market data:

   * IQFeed
   http://www.iqfeed.net/dev/

   * Nanex
   http://www.nanex.net/historical.html
The Nanex data is more expensive and harder to work with, so you might want to start with the IQFeed data.
reply
gfodor on Nov 13, 2012 | parent [–]
It's a situation where your average hedge fund can shell out the thousands of dollars per month for a full-access feed while individual investors are left hanging. Zack's used to offer reasonable prices for accessing fundamental data via their API but have since changed their pricing model to be more prohibitive to small timers.

Intra-day data is still a bitch but YCharts recently added a premium plan for $200/mo that seems to hit a nice sweet spot for individual investors that do serious trading for accessing fundamental data & analysis tools.

reply
justincormack on Nov 13, 2012 | parent [–]
It is expensive as you know. The data is generally owned by exchanges and they see it as a major revenue source. On the other hand if you want it it is just a cost to play. Open data is not big in finance.
reply
hooande on Nov 12, 2012 [–]
This could be renamed "Common Machine Learning Mistakes". Great advice all around, of course. Insufficient data, lack of or incorrect data prep, and poorly defined success criteria are problems that plague all forms of machine learning.

As other commentors pointed out, the biggest problem with stock market modeling is with trade execution and automated trading. A skilled ML practitioner will know how to deal with the size of the data sets and the normalization of the data. Placing the trades properly and taking into account the myriad sources of information available will give even experts a hard time.

reply
alexholehouse on Nov 13, 2012 [–]
Nice, accessible paper, but I find it galling that the images are so low quality. This paper is by no means alone in suffering from low image quality - I was reading a Nature paper earlier today where the graphics had compression artifacts.
reply
raverbashing on Nov 13, 2012 [–]
Pet peeve: reading two-column pdfs on screen is awful!

Article is nice, sometimes when using machine learning people forget how it works, its limitations, etc, and sometimes a solution that fits better a certain range is not the best solution

reply
acqq on Nov 13, 2012 | parent [–]
On typical screens made only to display movies, sure. On the 10 inch retina iPad the article page fits perfectly the screen in portrait mode and everything is perfectly readable. If there's a perfect use case for that 10 inch device it's this.
reply
raverbashing on Nov 13, 2012 | parent [–]
Actually the biggest problem is coming from the bottom of the 1st column to the top of the 2nd column (and even on paper this can be bothersome sometimes, well, it's their standart)

If the pages fits entirely on the iPad screen, great!

reply
About | RSS | RSS (all) | HN arXiv