Ready to learn Machine Learning? Browse Machine Learning Training and Certification courses developed by industry thought leaders and Experfy in Harvard Innovation Lab.
How AI helps traders make better decisions & improve high-frequency trading
Summary
Trading is a gruesomely competitive world. And with AI being painted as the new wonderweapon for everything, it’s understandable that there’s a huge amount of interest in discovering how to use AI for trading.
Disclaimer
AI does play an important role in trading – but maybe not in the way you’d expect. Unfortunately, AI can’t be used to power a superhuman trading machine that steals human traders’ lunches in every market. Not even Renaissance Technologies has that capability. At least not yet.
Why not?
The short answer: human competition – more about that below.
Meanwhile, the battles AI actually wins are much more incremental – but still significant. AI is more a trader’s side-kick than their replacement.
Below are the top four ways AI applies to trading today:
Analysing Sentiment
Humans can’t process all the information out there – but machines can get close. With AI, machines can now do much more – including analysing and summarising texts.
They can summarise things like sentiment: scrapers collect daily news, tweets, and other social media posts about a particular topic, and then AI algorithms (in particular Natural Language Processing) summarise whether positive or negative views are stronger. They can even categorise texts into topics and build human-readable text summaries automatically.
This is highly valuable information for traders, who have to be up to date on what’s happening as quickly as possible.
Forecasting real-world data
Traders also use AI to improve the reliability of input data forecasts – elements in the real world that help traders succeed.
Like the prediction of:
- Weather patterns in the northeast over the next two weeks;
- Solar energy supply in Europe;
- The outcome of a political election.
These forecasts are based on other algorithms developed by other companies – but that doesn’t mean these predictions can’t be improved.
One handy trick is to train an algorithm to combine multiple expert forecasts into another prediction – which is then more accurate than any of the forecasts it was based on. This is called ensembling, and it works pretty well.
Finding patterns
Trading is about identifying localised patterns – which are often limited in time and space – and then guessing how to exploit them. The process of finding patterns is arduous and time consuming.
But AI algorithms are basically pattern-finding machines. If an analyst suspects irregularities in a particular dataset, they can save time by using AI to find them.
So AI can find useful patterns, as long as it’s guided by an experienced analyst who knows what to look for. These patterns are then used by traders, who mix them with their experience and intuition, then apply them. Or you can use them to design automated trading machines – see the next section.
Tuning high-frequency trading machines
In high-frequency trading – as the name suggests – machines execute thousands or millions of trades per day, trying to take advantage of inefficiencies that only exist over very short time spans.
Human beings can’t make these trades – there are simply too many – but humans define the rules by which these machines operate.
However, because the market keeps changing, these machines need to be adjusted constantly. That takes a lot of time and effort.
AI can automate these recalibrations – and do a lot of the repetitive statistical work that analysts would otherwise need to do.
Warning signs – Things to watch out for
The promise of finding a miracle algorithm that can really print money is so alluring, lots of smart people have bought into it. Here are the most important traps to watch out for:
AI today is NOT “smarter” than human beings
In truth, even today’s most advanced AI algorithms are very naive compared to human brains.
When an algorithm beats a human at chess or GO, it’s like a car beating a human runner in a quarter-mile race: yes, the machine is faster, but that doesn’t make it superior. It just means that we built a machine that can do a very narrowly defined task – under certain narrowly defined conditions – really well.
It might seem like it at first, but trading is not such a narrowly defined task. Why? Because in trading you are competing with other humans – who will use all their brainpower trying to outsmart you.
Published trading strategies often don’t work in real life
There is a lot of research and lots of blog articles out there promising a profitable AI-based trading algorithm. But these models don’t tend to work in real life, for several reasons.
Faulty setups
A surprising number of papers actually make mistakes in how they set up their training and testing framework. They use variables that wouldn’t have been available at the time when the AI needed to make a decision (data leakage) or evaluate the predictions against the current price, not the future price. Surprising mistakes – but time-series datasets are a complicted thing for human brains to handle.
Selection bias
Fund managers are often critiqued for attributing their better returns to superior skill rather than luck. But if a lot of fund managers make a lot of random guesses, then in the end there will be some who made a few good guesses. The losers close down, and we never hear about them – so it looks like there are lots of fund managers who have the skills to beat the market.
In reality though, the number of fund managers who beat the market is exactly in line with what you would expect based on random guesses.
The same is true for research papers. If you try a lot of algorithms, you’ll eventually find one that seems to produce certain profits. If you don’t tell everyone how many experiments you had to run to get there, it will seem like you just stumbled on a superior approach.
However, there is absolutely no guarantee that this strategy will work outside the particular data you tested it on.
Transaction fees and slippage
Actually, building a trading strategy that outperforms the market is often quite simple – IF you forget about the real-world costs of doing trades. Transaction fees (the fees you pay for every trade) and slippage (the fact that the price might change between the time you make your order and the trade going through) eat up a lot of profit. And in almost every case, that’s enough to delete the profit you saw in simulation.
Patterns change over time
One of the most important concepts in Machine Learning is finding patterns in past data and using them to make correct forecasts of the future.
However, this doesn’t work in trading. Other traders are competing to find the same patterns – so patterns get found, exploited, and then disappear. That means patterns rarely exist for long, and you have to constantly find new ones.
This requires immense adaptability – something human beings are currently much better at than machines.
Algorithms alone will never give you an advantage
It’s easy to get carried away and focus on the algorithm as the main competitive advantage between one trading strategy and another.
This is essentially what companies like Numerai are proposing:
- Combine lots of good models into a super model.
- Beat the stock market.
But this does not work. Why? Because data beats algorithms. The data you give to your algorithm has a much larger effect on your model’s performance than how good the algorithm is. The data Numerai gives you is fixed – you can’t add to it. So the forecasts they make will always be worse than those of traders, who aren’t limited in the data they can use – traders who have access to an open pool of data, who can try, test, and add new data points to their algorithms continuously.
A bit of hope – Inefficent markets might still be fertile ground
Markets with few trading participants, high barriers to entry, a limited trading volume, and few players able to use Machine Learning might offer some opportunity for pure-play AI trading success.
In these markets, automated trading – especially the use of Machine Learning – is still just beginning, and traders who build automated trading engines could score enough of an edge to produce a good profit.
