In Flash Boys, Michael Lewis explores the US stock market’s use and acceptance of high-frequency trading (HFT). Lewis—a best-selling author and journalist—argues that HFT exploits regular investors, creates unfairness in the stock market, and is a concern for any investor.
Published in 2014, Flash Boys devotes much of its attention to the experiences of Brad Katsuyama, a Canadian trader who investigated HFT tactics and was a key player in exposing the practice to the financial world. Lewis examines Katsuyama’s investigation in detail, using it as a lens through which to understand the history and effects of HFT on the market and economy.
In this guide, we’ll first discuss how the stock market adapted to technology and the new features of trading that resulted from these changes. Then we’ll explore HFT tactics that create problems for average traders and investors, as well as how Wall Street responded to HFT tactics. Finally, we’ll explain how Katsuyama and his team found solutions to the problems generated by these high-frequency trading tactics and their efforts to create a fairer stock market.
Along the way, we’ll look at how other experts have weighed in on the practice and some differing views on whether or not these changes have been beneficial. We’ll also provide updated information about trading research and regulations that have come into effect since the book’s publication.
Lewis describes high-frequency trading (HFT) as a type of electronic trading platform that uses automated computer algorithms to very quickly buy and sell large quantities of stocks. High-frequency (HF) traders and firms own these high-powered computers and algorithms. These HFT algorithms gather market data and use this information to buy and sell stocks, completing this process in microseconds, or one-millionth of a second, thus using a different strategy than long-term buy-and-hold investing positions. This speed advantage makes HFT algorithms faster than any human trader, giving them a powerful edge by enabling them to accrue billions of dollars in tiny profits from HFT trades that traditional investors and traders can’t.
(Shortform note: While Lewis doesn’t detail the history of high-frequency trading, who started it, or when it started, this may be because experts don’t agree on how it started. Experts have traced the practice back to the 1830s, 1930s, the 1970s, and 2005. Some researchers attribute HFT’s creation to former finance professor David Whitcomb, while others believe the SEC’s deregulation of the stock markets started the practice.)
As Lewis explains, a Canadian trader named Brad Katsuyama working for the Royal Bank of Canada (RBC) was one of the first people to raise the alarm about HFT. In the late 2000s, Katsuyama noticed changes in the stock market that clued him in to systemic problems caused by HFT, motivating him to build a team to investigate why he was losing money in his trades. While Katsuyama wasn’t the first person to learn about HFT and its tactics, he was one of the first to identify and raise awareness of how HFT was a problem. Lewis learned about HFT through Katsuyama’s findings and focuses much of the book on him.
(Shortform note: In 21 Lessons For The 21st Century, Yuval Noah Harari argues that as algorithms like HFT software provide increasingly accurate suggestions, their convenience is almost irresistible—but relying on algorithms to make your decisions denies you the freedom and ability to make your own choices, which may explain why Katsuyama felt he couldn’t trade successfully anymore. In addition to threatening jobs, technology threatens human liberty, as algorithms learn so much about people that they gain an immense power to influence and manipulate.)
Lewis outlines a few ways by which HFT uses its unique characteristics to unfairly disadvantage regular investors:
Speed is a crucial factor in stock trading. Traders who can lower their latency times, or the time between sending and receiving a signal about market data or a stock order, have an advantage over other traders.
Latency times can differ between exchanges and investors depending on how close to an exchange an investor physically is, since the further the electric signal has to travel through wiring, the longer it takes to execute an order. HF traders work so quickly that they can buy or sell orders in microseconds, so if a stock has changed its price on one exchange but there’s a lag of fractions of a second before it changes its price on another exchange, the HF trader can take advantage of that difference and make a risk-free profit.
Because of this almost-instantaneous purchasing power, HF traders can detect and act upon an order placed by a regular investor and profit off the market movements prompted by that order, lowering the profit for that regular investor.
(Shortform note: While Lewis asserts that speed is an important factor in high-frequency trading, critics argue that speed has always been an important factor in any trading. This may be one reason that HFT has still gone unregulated, as regulators haven’t identified where to draw the line when it comes to speed and whether it’s even appropriate to punish certain traders solely because they deployed better technology to decrease the latency times of their trades.)
The rise of electronic trading allowed for the development of dark pools—private stock exchanges set up by large banks where buying and selling orders can be executed separately from the wider market. Since dark pools are private, banks don’t have to immediately report to the public what happens inside their dark pool. Because of this, traders outside the dark pool have access to less information than traders within the dark pool, giving the latter the ability to not only move faster on pricing changes, but also to anticipate them. HF traders profit from this discrepancy in information in ways regular investors can’t.
(Shortform note: Critics of dark pools believe the lack of transparency is a problem and that dark pools aren’t being used for their intended purpose: to reduce market impact when placing large trade orders. Indeed, investors who abuse this secrecy can face serious consequences. Bill Hwang was arrested for fraud and racketeering in connection with his family office, Archegos, after a risky investment strategy cost him $20 billion. Since Archegos was investing inside dark pools, other banks weren’t aware of his trades or the terms of those trades. Had they known this, it may have deterred them from doing business with him or prevented him from building such a large market position. In response to the Archegos scandal, regulators have expressed interest in implementing new disclosure rules to prevent this kind of trading activity.)
Electronic trading led to changes in the rebate system. Rebates are when stock exchanges charge a small fee to one broker—generally the buyer—of a trading order and pay a small fee (a rebate) to the other broker—usually the seller.
At first, exchanges continued this traditional rebate system in their electronic trades, but HF traders encouraged them to change their rules so that they paid the brokers—generally the buyer—instead of charging them a fee. HF traders used this change to predict the flow of trades—if a broker could be paid for a trade she’d usually be charged for, it’s in her best interest to send it to the exchange where she’d be paid. Thus, HF traders could anticipate and monitor where brokers would send orders and could use that knowledge to get there first.
Incentives Don’t Always Work the Way We Expect
We can understand rebates—a kind of trading incentive—through the lens of mechanism design, the process of designing rules to accomplish specific outcomes. Mechanism design is a tricky business—since rule adjustments have such a significant impact on the game’s outcome, small unforeseen side effects can have extreme consequences, as in the case of rebates and HFT. In Freakonomics, Steven Levitt and Stephen Dubner describe how mechanism design can go wrong.
The Freakonomics authors categorize incentives into three types: economic, social, and moral. Economic incentives are when you save money by behaving well, social incentives are when you can maintain a good reputation by behaving well, and moral incentives are when you can satisfy your conscience by behaving well. The authors argue that economic incentives have the potential to backfire by inadvertently removing social and moral incentives. This principle applies to HFT, which as an algorithm, doesn’t operate using social or moral decisions. So the system of rebates, which was originally intended to reward good trading behavior, ended up incentivizing people to try to game the market.
As the stock market changed due to technology, stock market regulations changed with it. In 2007, the Securities and Exchange Commission (SEC) enacted a new rule that would have lasting effects on the market—the Regulation National Market System (or Reg NMS). This rule stated that traders must buy stock at the lowest market price for investors. So if stock for a certain company ranged from $10.00 and $10.08, the trader is legally obligated to buy it at whichever exchange sells it for $10.00. Reg NMS was intended to ease investors’ worries that traders weren’t getting them the best deals.
However, the new regulation ended up benefitting HF traders more than ordinary investors, because it allowed them to more quickly identify price changes at any given time.
(Shortform note: Like Lewis, other experts are concerned about the SEC’s regulations and their impact on markets. Some experts urge the SEC to reevaluate and update Reg NMS, while others recommend a complete overhaul of their rules. While Reg NMS may have positive effects on fair pricing and market competition, critics note that the regulations have led to further market fragmentation, difficulty executing orders quickly, and pricing inefficiency, showing that financial experts can’t agree on the effects of the regulations. Some experts have even proposed scrapping the idea of government oversight altogether, recommending self-regulation as an alternative to SEC regulation.)
Another consequence of Reg NMS was to allow HF traders to front run trades—to use advance knowledge of an investor’s intentions for profit. Under Reg NMS, traders must buy stock at the lowest price, but sometimes, the exchange with the lowest price won’t have all the shares the investor wants to buy. Thus, a broker might have to go to multiple exchanges to fulfill an investor’s entire order. When she does this, her order tips off HF traders of her intentions to buy or sell a certain stock. The HF traders can then race to other exchanges to buy the stock she wants, only to sell it to her at a higher price.
(Shortform note: Other experts agree with Lewis’s argument that traders have found ways to exploit well-intentioned regulation, such as Reg NMS. Some experts even recommend replacing Reg NMS with entirely new—albeit more relaxed—regulation. One suggestion is to outline several factors—not just price—that a broker must consider when executing customer trade orders, such as the price of the stock or how secure the market is at the time of purchase.)
Lewis describes order types as sets of instructions determining how traders place stock orders. For instance, a trader can use an order type to stipulate that her order goes through only if she’ll receive a rebate or only if the price doesn’t exceed a certain number. As electronic trading became more popular, the number of order types increased from three to over 150 and became so complex that very few people could understand them—typically, only people whose job it specifically was to understand them.
Lewis argues that most of these order types were actually unnecessary and gave HF traders more opportunities to exploit investors by tapping into their trading strategies and intentions. HF traders could examine order types to know what conditions investors were looking out for and then use that information to get to trades first, increasing their own profit at the expense of other investors. HF traders also used these order types to push their orders ahead of regular investors, so that, for example, if an ordinary investor tried to buy 100 shares of a certain stock, an HF trader's standing order could be activated to scoop up those shares first.
(Shortform note: Another way to think of HFT and order type baiting is through the lens of game theory. In The Undercover Economist, Tim Hartford explains that game theory is a discipline adjacent to economics and mathematics, where a “game” is defined as an activity in which predicting another’s actions affects your own actions. By analyzing and predicting the actions of investors through order type baiting, HF traders—algorithmic, rational decision-makers—use this data to maximize their payoffs.)
These high-frequency trading tactics capitalized on the precedents set by electronic trading. Lewis explains that anyone involved in the stock market—brokers, investors, hedge fund managers, and so on—felt the effects of HFT, which were exacerbated by two factors: Wall Street’s greed and the “flash crash.”
Lewis argues that the Wall Street firms and banks, whose proper role is to help investors, adopted HFT techniques despite knowing that HFT made trading unfair for average investors. They did this because they were making so much money. One firm made over a billion dollars in one year from its HFT department alone.
(Shortform note: Lewis is no stranger to exposing Wall Street’s greed: In The Big Short, he argues that greed and short-sightedness were the prime drivers of the financial crisis. All major stakeholders in the ecosystem were fueled by the desire for profit, which made it easy to overlook systemic problems.)
After years of unregulated high-frequency trading, Wall Street experienced a “flash crash” in 2010. Lewis defines a flash crash as a stock market crash that happens quickly—stock prices drop dramatically before recovering within a few minutes. But during the 2010 flash crash, 20,000 stocks traded at dramatically different stock values.
According to Lewis, no one could pinpoint a reason behind the flash crash. Katsuyama couldn’t access the data that could prove HFT caused the crash, since this information wasn’t public—it belonged to the HFT firms and exchanges—but he knew that the flash crash was a result of HFT.
The flash crash marked a turning point in the investing world. Investors were interested in what was going on in the market—and why it had dropped by 600 points so suddenly. People called Katsuyama and his team to set up informational meetings where he educated investors and executives about HFT.
Who Is Responsible for the Flash Crash?
While Lewis blames the trillion-dollar flash crash on high-frequency traders, the US courts found another reason: a self-taught stock market trader named Navinder Singh Sarao. He was convicted of “spoofing”—creating a huge number of fake buy or sell orders. High-frequency trading firms program their algorithms to get out of the market when it gets too volatile, and when Sarao’s spoofing operation created artificial volatility, all the algorithms pulled out at once, resulting in a crash. He capitalized on the changing prices, earning about $900,000 from the market manipulation.
While the Commodity Futures Trading Commission (CFTC) found Sarao responsible for the flash crash, some experts contend that blaming Sarao doesn’t get to the heart of the issue—that both human and algorithmic traders still have the ability to manipulate markets despite existing regulations.
After discovering these tactics, Katsuyama and his team decided to fight HFT rather than mimic its tactics to make money, so they quit their jobs at RBC to create their own stock exchange. They called it the Investors Exchange, or IEX. The purpose of IEX was to make investing fair by preventing the predatory behavior of high-frequency traders.
When creating IEX, Katsuyama and his team knew they couldn’t ban HFT on their exchange. But they could take preventative measures against the tactics HF traders used to exploit investors. Lewis discusses the team’s solutions addressing each of the HFT tactics:
(Shortform note: While Lewis praises IEX for fighting HFT, some critics note that in doing so, he perpetuates the idea that stock exchanges will correct themselves without regulation. Critics believe that regulatory bodies like the SEC—not exchanges or traders—should be responsible for keeping up with changes in stock trading, especially for something as influential as HFT, to better protect investors. Experts have suggested a small tax on every trade, called the Tobin Tax, that would have little effect on regular investors but would deter HF traders by further reducing the margins on their already-small profits per trade.)
IEX increased the amount of time it would take for any investor—including HF traders—to access the exchange’s market. To do this, they moved their "point of presence,” or a point where traders connect to an exchange, 10 miles away from their matching engine in New Jersey and coiled the cable between the two locations to lengthen the time it took to get a signal from one to the other. In doing so, IEX created a 350-microsecond delay that slowed down HF traders accessing their exchange and therefore allowed IEX to interact with other exchanges at the same time as HF traders, instead of behind them.
(Shortform note: While Lewis claims IEX’s tactics (the point of presence and 350-microsecond delay) make trading fairer for regular investors, some experts object to IEX slowing down traders’ orders. They believe that IEX is effectively using latency arbitrage against HF traders since the exchange is using old prices on other exchanges before traders on those exchanges can update their own prices, thus making trading with IEX unfair for HF traders. They believe these tactics only benefit big investors and portfolio managers. However, this argument seems to acknowledge that using “old” prices to perform latency arbitrage is unfair—it’s just a question of whether humans or technology are on the receiving end of it.)
Lewis explains that IEX wanted to prevent the banks from allowing customers’ orders to sit unfulfilled in the dark pool. IEX thus encouraged investors to request that their orders be sent to IEX.
(Shortform note: While encouraging investors to ask their bank to send their orders to IEX was a solution, it didn’t completely fix IEX’s problem since it put the responsibility on the investor to make sure their bank cooperated and sent an investor’s order to IEX. To address this problem, in 2016, the SEC approved IEX’s application to become an official exchange, thus requiring brokers to send their orders to IEX when it offers stocks at the best price. This made it no longer up to the investor to make sure their order went to IEX.)
As their next solution, IEX eliminated rebates. Instead, IEX charged the people involved in both sides of the trade $0.009 per share. This approach differed from the previous model, which charged one side of the trade and paid the other a rebate, thus removing HF traders’ ability to use rebates to predict where investors would sell orders and then get there first.
(Shortform note: Following IEX’s decision to remove rebates from its trading process, the SEC reexamined whether or not rebates are an effective quality of the exchange. Although the SEC still allows rebates, it has admitted that rebates may create conflicts of interest when brokers choose where to send their orders. The SEC has considered testing how markets would function without rebates. NYSE and Nasdaq sued to stop this experiment before it went into effect.)
To reduce information baiting, Katsuyama and his team limited the number of order types to three, as opposed to the 150 offered by other exchanges at the time. Lewis argues that fewer order types made trading more straightforward, encouraging investors who actually wanted to buy and sell stocks to trade with IEX. By limiting the number of order types, IEX prevented HF traders from baiting information out of investors: HF traders had fewer ways of determining which stocks an investor wanted or how much an investor was willing to pay.
(Shortform note: The exchange has increased the number of order types it offers since the publication of the book. IEX now has three classes of order types—ones for standard and retail investing, as well as what IEX calls their Signal Series order types, which are supposed to offer “enhanced price protection.”)
To counteract the secrecy of dark pools, IEX published the trading rules they use for their matching engine. They also showed what order types they allowed on IEX, as well as whether or not any other investors were given special access to their exchange. They hoped being transparent would build investor trust and encourage other exchanges to do the same.
(Shortform note: Lewis believes IEX fostered trust with investors through being honest and transparent about what was happening in their trades, which is a strategy other experts recommend. In Algorithms to Live By, Brian Christian and Tom Griffiths note that one simple way to improve the equilibrium of a competitive game, such as stock trading, is to change the rules to make honesty and transparency the optimal strategy. The authors argue that when you design a game in which players are incentivized not to hide their intentions, it eases the burden on everyone. Players no longer have to stress about predicting what others are going to do, or what others think they’re going to do.)
In Flash Boys, best-selling author Michael Lewis explores the US stock market’s use and acceptance of high-frequency trading (HFT). He argues that HFT exploits regular investors and creates unfairness in the stock market. Therefore, he believes any investor should be concerned about high-frequency trading.
Lewis focuses much of the book on the experiences of Brad Katsuyama, a Canadian trader who investigated HFT tactics and was a key player in exposing the practice to the financial world. Lewis examines Katsuyama’s investigation in detail, using it as a lens through which to understand the history and effects of HFT on the market and the economy.
In examining the role HFT plays in the investment world, Lewis explains six features of the stock market and how HFT algorithms exploit each of them to unfairly tilt the playing field in their favor. He explains how these tactics create problems for average traders and investors. Lewis also explores how Katsuyama and his team found solutions to these HFT tactics and their efforts to create a fairer stock market.
Michael Lewis is a financial journalist and best-selling author. He writes for publications like The Guardian, Bloomberg, and The New York Times, covering finance, economics, and business to reveal complex and unfair systems. While critics believe he oversimplifies these topics, his books are generally well-received. Lewis also hosts a podcast called Against the Rules, which explores fairness in society and is known for its skepticism of authority.
Lewis grew up in New Orleans and attended college at Princeton for art history. He later received his masters in economics from the London School of Economics. In the 1980s, he was a bond salesman on Wall Street, which prompted him to write his first book, Liar’s Poker. His other books include Moneyball, The Big Short, The Blind Side, and most recently, The Premonition.
Connect with Michael Lewis:
Flash Boys was published by W. W. Norton & Co. in 2014, making it Lewis’s 15th book. The two books that preceded Flash Boys—The Big Short and Panic—both explore elements of the US financial system. Like Lewis’s other books, Flash Boys interweaves narrative with explanations of technical subjects.
Lewis wrote Flash Boys in response to the increase in high-frequency trading throughout the 2000s and 2010s. Several events raised awareness of high-frequency trading, including a 2009 SEC testimony on the practice, a Wall Street programmer’s arrest for stolen trading code, and the “flash crash” of 2010.
The increased awareness of high-frequency trading caused experts to research these trading tactics, resulting in mainstream investigation and reporting:
Flash Boys addresses and expands on the issues brought up in these publications, while also discussing the people involved in understanding high-frequency trading.
Flash Boys made a big impact on the financial industry, fueling interest in the topic of high-frequency trading. A day after the book was published, the FBI announced an investigation into high-frequency trading practices. People speculated that the FBI’s decision was motivated by Flash Boys, but the FBI claimed to have been investigating high-frequency trading for over a year prior to the book’s publication.
Flash Boys was a No. 1 New York Times best-seller. Reviewers praised Lewis’s ability to combine explanations of complex topics with an enjoyable narrative. Others noted that the book is an essential read for anyone interested in economics, investing, and finance.
But critics of the book claimed it was one-sided, since Lewis only presented the argument against high-frequency trading. They also believe Lewis oversimplified the issue, criticizing him for a lack of research—beyond interviews—for the book.
The book jumps around in time, place, and narrative focus. Lewis examines high-frequency trading through the perspective of Brad Katsuyama, a Canadian trader, as well as other people who join his team. Lewis also explores the trial of Sergey Aleynikov, a Russian programmer arrested for stealing code. This approach, paired with Lewis’s conversational tone, makes the book easy to read. Some reviewers noted that the book read like a detective novel because of this writing style.
Lewis breaks down complex concepts to make them accessible to readers without a finance background. Even still, readers with stock market and investing knowledge may better understand the book.
In this guide, rather than focusing on the narrative elements, such as Lewis’s chronicling of a Russian programmer’s trial for stealing code and Katsuyama’s process of learning about high-frequency trading, we’ll primarily focus on the concepts Lewis discusses: changes in the stock market, high-frequency trading tactics, and the solutions Katsuyama’s team developed to address these problems. In our commentary, we’ll address some of the counterarguments to Lewis’s claims that arose after the book’s publication.
In Flash Boys, Lewis exposes Wall Street and its use of high-frequency trading to exploit investors. He argues that high-frequency trading makes money using advantages that regular investors don’t have access to, meaning anyone with a retirement account, investments, or plans to invest is at a disadvantage.
Lewis focuses the narrative on a Canadian trader named Brad Katsuyama, who investigated high-frequency trading after noticing changes in his ability to trade. After learning about high-frequency trading, Katsuyama and his team used this knowledge to found the Investors Exchange (IEX), a new stock exchange designed specifically to prevent unfair high-frequency trading tactics and take action against Wall Street’s unfairness.
Lewis is a journalist and bestselling author of over 30 books, revealing unfair systems in finance, economics, and business. He wrote Flash Boys in response to the increase in high-frequency trading throughout the 2000s and 2010s.
In this guide, we’ll briefly explain what high-frequency trading is and how both Lewis and Katsuyama learned about it. Then we’ll discuss how changes to the stock market and electronic trading created the six conditions for high-frequency trading to exist. Then we’ll examine the six tactics that resulted from high-frequency traders exploiting each of those conditions. Finally, we’ll discuss the solutions to these tactics that Katsuyama and his team developed to make stock trading fair for investors.
Along the way, we’ll look at how other experts have weighed in on the practice and some differing views on whether or not these changes have been beneficial. We’ll also we’ll provide updated information about trading research and regulations that have come into effect since the book’s publication.
In Part 1, we’ll start by examining what high-frequency trading is, how Lewis was introduced to it, and who first brought attention to the problems it was creating for the stock market.
Lewis describes high-frequency trading (HFT) as a type of electronic trading platform that uses automated computer algorithms to very quickly buy and sell large quantities of stocks. High-frequency (HF) traders and firms own these high-powered computers and algorithms. These HFT algorithms gather market data and use this information to buy and sell stocks, completing this process in microseconds, or one-millionth of a second, thus using a different strategy than long-term buy-and-hold investing positions. This speed advantage makes HFT algorithms faster than any human trader, giving them a powerful edge over traditional investors and traders by enabling them to accrue billions of dollars through HFT trades that traditional investors and traders can’t.
(Shortform note: While Lewis doesn’t detail the history of high-frequency trading, who started it, or when it started, this may be because experts don’t agree on how it started. Experts have traced the practice back to the 1830s, 1930s, 1970s, and 2005. Some researchers attribute HFT’s creation to former finance professor David Whitcomb, while others believe the SEC’s deregulation of the stock markets started the practice.)
Lewis learned about high-frequency trading following the arrest of a Russian immigrant named Sergey Aleynikov. His trial received media attention and boosted public interest in HFT. Aleynikov was an HFT programmer for Goldman Sachs and was considered one of the best programmers on Wall Street. His job was to make HFT software faster, but he was unaware of the full effect of his work. When he left Goldman Sachs in 2009, the firm accused him of stealing HFT code and he was arrested by the FBI.
Aleynikov was put on trial and sentenced to eight years in prison. After a year in prison, he was acquitted, but his trial exposed Wall Street’s questionable trading practices and sparked public interest—as well as Lewis’s—in HFT.
Another Programmer Was Arrested for Stealing Code
Not long after Aleynikov was charged for stealing code, another programmer named Samarth Agrawal was also arrested for replicating HFT software after leaving his Paris-based HFT bank for a hedge fund. But while Aleynikov was eventually acquitted, the US courts ruled differently for Agrawal, who was sentenced to three years in jail. So why was Aleynikov acquitted and not Agrawal?
While Agrawal cited Aleynikov’s case, the jury found a small difference between the two cases. In Aleynikov’s trial, the court considered the code itself as the stolen product for commerce. But in Agrawal’s trial, the jury argued that the stocks traded by the bank—not the HFT code—were the products of commerce. Because Agrawal’s HFT code helped facilitate those traded stocks, Agrawal was found guilty. The courts also noted that Agrawal printed out the code he stole, thus transforming it into a tangible stolen product under the National Stolen Property Act. The decision—which seems largely influenced by the medium of the code—may support Lewis’s argument that the legal system has done little to keep up with or understand changes in technology.
In the late 2000s, in the years leading up to Aleynikov’s trial, a Canadian trader with the Royal Bank of Canada (RBC), Brad Katsuyama, was starting to notice changes in the stock market that clued him in to systemic problems caused by HFT. While Katsuyama wasn’t the first person to learn about HFT and its tactics, he was one of the first to identify and raise awareness of how HFT was a problem.
While working at RBC, Katsuyama noticed that he couldn’t trade successfully anymore, because in the time between sending out his customers’ orders and actually buying stocks, the stock market would change, forcing him to pay more money. Katsuyama asked RBC executives for permission to investigate why he was losing money in his trades, and they agreed.
(Shortform note: In 21 Lessons For The 21st Century, Yuval Noah Harari argues that as algorithms like HFT software provide increasingly accurate suggestions, their convenience is almost irresistible—but relying on algorithms to make your decisions denies you the freedom and ability to make your own choices, which may explain why Katsuyama felt he couldn’t trade successfully anymore. In addition to threatening jobs, technology threatens human liberty, as algorithms learn so much about people that they gain an immense power to influence and manipulate.)
Lewis explains that Katsuyama first learned about HFT when HFT executives asked RBC to provide them with access to the Canadian stock market. Katsuyama thought this was a bad idea, since he couldn’t find anyone who could explain HFT to him, and he suspected that HF traders might be the reason why he was paying more for his trades.
Katsuyama assembled a team to help him investigate the practice. He recruited Ronan Ryan, a telecommunications expert who helped trading firms increase their signal speed. After pooling their knowledge, Katsuyama and Ryan met with different Wall Street executives, educating them about how HFT tactics exploited the stock market.
Katsuyama also added John Schwall to his team. Schwall was a product manager, the intermediary between programmers and traders. Soon after joining the team, Schwall investigated the history of the stock market and its regulations. (We’ll discuss these findings in Part 2 of the guide.) This information supplemented Katsuyama’s knowledge of the stock market, its conditions, and how HFT exploited those conditions.
As part of their investigations, Katsuyama’s team ran experiments to understand how HFT worked. As a result of these experiments, Katsuyama came to understand the six tactics of HFT and how it impacts the market, as we’ll discuss in Part 3.
Qualities of a Great Team
By building a good team, Katsuyama was able to research and find effective solutions to problems. But what makes a great team? Experts identified three key qualities of effective teams:
Intellectual Diversity: when the members of your team have a range of different experiences, backgrounds, and education, thus leading to more creative solutions. Gender and racial diversity also fall under this category.
Psychological Safety: when teammates feel safe enough to share their thoughts and ideas. When all members of the team are comfortable speaking up, the team can work better together as a whole.
Shared Purpose: a goal that the whole team believes is worth fighting for. A shared purpose can also help create a shared identity.
These characteristics fit Katsuyama’s team: They came from different intellectual backgrounds, brought different individual strengths and experiences to the company, and felt comfortable enough to honestly share their ideas. Most importantly, each was committed to finding out what had changed in the stock market and how they could make trading fair.
To understand the HFT tactics Katsuyama and his team found, we should discuss the stock market characteristics—and the regulations that Schwall learned about—that set the stage for HF traders to exploit individual investors. First, we’ll provide a brief overview of stock market fundamentals. Then we’ll discuss the early phases of electronic trading that preceded high-frequency trading and how it changed the stock market. Then we’ll explain the six conditions of the electronically-based stock market that HFT exploited. (We’ll discuss how HFT exploited these precedents in Part 3.)
The stock market is where investors buy and sell stock, or partial ownership in companies. The stock market is composed of many different stock exchanges, which are individual marketplaces where stocks are traded. The National Association of Securities Dealers Automated Quotations (Nasdaq), the New York Stock Exchange (NYSE), and Better Alternative Trading System (BATS) are all examples of stock exchanges.
Traders and brokers trade stocks on behalf of an investor at a stock exchange, typically doing so as part of investment firms, brokerage firms, or banks. Traders are the financial intermediaries between investors and stock exchanges, and they’re responsible for executing their customers’ orders.
(Shortform note: While Lewis mostly uses “trader” and “broker” interchangeably, there is a slight difference between the two. A trader usually buys or sells securities based on the requests of a portfolio manager at an investment firm to create an investment strategy that best fits their client. Brokers work directly with clients to buy and sell securities, and they may act as financial planners for their clients, such as by helping with a retirement plan or portfolio diversification.)
Lewis explains that these basic elements of the stock market—exchanges, brokers, and investors—have stayed the same over the years. But stock traders no longer buy and sell stocks in person on a stock exchange floor–-today, traders buy and sell stocks electronically through computer programs. The stock exchange buildings still exist but are now used to house stock exchange computers, called “matching engines,” which pair stock buy and sell orders.
(Shortform note: While Lewis uses “the stock market” or “Wall Street'' to broadly refer to financial processes, Thomas Sowell argues in Basic Economics that a market is just humans engaging in transactions among themselves. Sowell asserts that when the stock market is treated instead as a personified, third-party entity, rhetoric is allowed that takes away people’s freedom to transact on mutually agreeable terms. His argument extends to stock exchanges transitioning to electronic platforms if technology is viewed as a tool, not as a third-party entity.)
Today, anyone with a computer can access a stock exchange, since stocks are traded electronically, using computer technology to pair buyers and sellers in a virtual market. Lewis explains how both human and computer traders go through the same three steps of an electronic trade:
Lewis believes that humans are good at this process but asserts that technology is better. Computer programs can complete these steps more thoroughly and much faster than a person can, sending and receiving signals about market data in a fraction of the time it takes you to blink your eyes. If a trader has an extra split second to evaluate stock information, they have a slight advantage over other traders, as we’ll discuss in more detail later.
Electronic Versus Algorithmic Versus High-Frequency Trading
While Lewis discusses electronic trading, he doesn’t differentiate between electronic trading, algorithmic trading, and HFT. Electronic trading involves creating an online brokerage account to facilitate electronic transfers between you and the brokerage. After placing an order, the brokerage uses its technology to interact with stock exchanges to execute trades.
While someone can make an electronic trade, someone can only write the code for an algorithmic trade. With algorithmic trading (also called automated trading, black-box trading, or algo-trading), a computer algorithm—a defined set of instructions—places a trade. The trade, in theory, can generate profits at a speed and frequency that is impossible for a human trader to do—this is HFT, or the most common kind of algorithmic trading. Lewis’s focus is HFT, which is probably why he doesn’t detail the other kinds of algorithmic trading.
Next we’ll discuss six characteristics of electronic trading that HFT later exploited.
Lewis explains that in stock trading, speed is a crucial factor. Traders are always looking to lower their latency times, or the time between sending and receiving a signal about market data or a stock order so that their orders are sent and received faster than everyone else’s. A faster trader can buy and sell stocks at better prices. Lewis explains that a lower latency also gives traders an extra split second to evaluate stock information, which can help them make better decisions. It’s the same concept that would give you an advantage if, for example, you’re taking a test and you get the questions five minutes before everyone else.
Lewis describes three factors that affect the latency of an electronic trade:
How Much Does Speed Matter?
While Lewis asserts that speed is an important factor in HFT, critics argue that speed has always been an important factor in any trading. This may be one reason that HFT has still gone unregulated, as regulators haven’t identified where to draw the line when it comes to speed and whether it’s even appropriate to punish certain traders solely because they took advantage of better technology to decrease the latency times of their trades.
Critics also argue that HFT doesn’t matter for retail trading since it’s not time-sensitive. Most regular investors use a long-term buy and hold strategy, so paying a few pennies or dollars more—due to HFT’s fast and frequent trades—when placing the trade isn’t significant in the overall return or risk.
But HFT’s speed does impact big investors, as a quick change in stock price can result in large losses. If, for instance, news of your pending purchase or the special knowledge you have hits the market before your trade and drives shares up even four or five dollars, the impact on your trade would be $4-5 million.
Lewis explains that traders have found ways to reduce the latency associated with each of these three factors through three methods: co-location, better code, and fiber-optic cable, each of which we’ll describe below.
To speed up their electronic trading—and thus reduce their latency time—financial traders use co-location. Lewis explains that co-location is when exchanges place trading computers as close as possible to an exchange’s matching engine, which pairs buy and sell orders for a given stock. When the computer is closer to the matching engine, it can send and receive signals milliseconds faster, thus reducing its latency time. Think of it like running a race where one runner starts five feet ahead of everyone else.
Co-location is so important that trading firms have gone to great lengths to get closer to matching engines. Lewis describes how one firm moved its computers from Kansas City to Nutley, New Jersey (closer to the NYSE engines), reducing their latency times from 43 milliseconds to 3.8 milliseconds.
Exchanges and other data centers took advantage of this trend and made millions of dollars by renting out space near their matching engines to HFT firms. Then, these firms paid even more money to move their machines closer—even by a few inches—within that space, showing the enormous profit potential of reducing latency times by even a few milliseconds.
Alternative Data Dissemination Systems Ease Co-Location Advantages
Lewis notes that co-location gives certain traders a speed advantage, and some exchanges have taken steps to address this issue, such as by changing their system of data dissemination. Most co-location facilities use a system that sends data from the exchange server to clients sequentially based on when the trader connected to the server. Due to co-location, whichever trader is at the front of this virtual line has the speed advantage.
But since 2014, the National Stock Exchange of India (NSE) has shifted away from this system and started using the Multicast TBT (MTBT) system at its co-location facility. In the MTBT system, the exchange server sends data to a switch, which each trader connects to independently. Then the switch sends the data to the traders and acts as a middleman between the traders and the server’s data. Experts have noted that switching to the MTBT system may have helped eliminate preferential access.
Lewis explains that another way to reduce latency times is by creating faster code—stronger and speedier algorithms that facilitate trades. When your code can process information faster, it can make quicker market decisions, thus executing orders faster. This has led to trading firms seeking out talented programmers to improve their trading algorithms.
Humans vs. Algorithms
We can consider trading algorithms as a way to forecast, or to analyze and predict future events—in this case, which stocks will be profitable to buy or sell—with an accuracy better than chance. In Superforecasting, authors Philip Tetlock and Dan Gardner question why we should rely on humans for forecasting at all instead of an advanced computer algorithm. For situations where a well-validated statistical algorithm exists and has been proven reliable, computer predictions are almost always more accurate than those of human forecasters—but very few such algorithms exist.
Tetlock and Gardner believe that as technology advances, we may develop more of these algorithms, or refine the ones that already exist, such as the fast software used for HFT. But even then, the judgment of human forecasters will not be obsolete. Artificial intelligence (AI) is mostly immune to human cognitive biases, but it can’t interpret the results of its own predictions and create new meaning. Therefore, the authors argue, the future of forecasting will be a combination of skilled forecasters and powerful algorithms, so the solution to HFT won’t eliminate the need for human insight.
Lewis explains that the third method of decreasing latency times is to improve the cable that transmits the data between locations. To do this, firms use fiber-optic cable, which offers the most speed. These cables can be used between computers within the exchanges, but they’re also used for telecommunications routes that transmit data across the country, like the ones used by Verizon and AT&T.
Wall Street firms go to great lengths to improve their fiber-optic cable. Lewis describes a stock trader named Dan Spivey who saw an opportunity to improve latency times by building his own $300 million telecom route. The fastest existing line was 14.65 milliseconds, meaning it took 14.65 milliseconds for information from Point A to reach Point B. But Spivey’s would be 13 milliseconds.
To create this faster line, Spivey made the line out of fiber-optic cables and constructed it in as straight a line as possible between Chicago (where the Chicago Mercantile Exchange is) and Newark, New Jersey (where the Nasdaq data center is). Unlike other telecom companies, Spivey minimized the number of curves and turns, which would slow down the transmission of data. This required drilling through mountains and laying their cable through parking lots.
But all of this effort benefitted Spivey: He sold use of the line to trading firms for millions of dollars, showing how far traders were willing to go to gain a 1.65-millisecond speed advantage.
(Shortform note: In 2017, Spread Networks took another step in capitalizing on their line: A company called Zayo bought Spread Networks for $127 million.)
Hollow-Core Fiber Is the New Fiber-Optic Cable
Since the publication of the book, HF traders have improved upon fiber-optic cable to transmit market data. Some traders are experimenting with hollow-core fiber, which takes a third of the time to transmit data. Trading firms use hollow-core fiber for short distances, such as to connect a data center to a nearby communication tower, which reduces the transmission time by a billionth of a second.
Hollow-core fiber is expensive to manufacture and doesn’t work well over long distances since the data is lost more quickly with hollow-core fiber than with traditional fibers. Manufacturers are working on improving long-distance transmission, and they report that dozens of HFT firms have contacted them about the improved fiber, and some firms have already started using hollow-core fiber. Other experts have expressed interest in using the hollow-core fiber between London and New York once the technology improves, thus representing the next step of extreme measures traders will go to improve speed.
The rise of electronic trading also allowed for the development of dark pools. Lewis defines dark pools as private stock exchanges set up by large banks. Investors send their orders to the brokers at these banks, and the bank routes the order to its dark pool to be fulfilled instead of looking for buyers or sellers in the wider market.
(Shortform note: While Lewis discusses dark pools as a relatively new market phenomenon, the first dark pool was started in 1979 due to a regulation that allowed securities listed on an exchange to be traded off the exchange, offering more privacy than the public exchanges. For the next 20 years, dark pool trading accounted for only 3-5% of trades—until electronic trading platforms made setting up a dark pool easier.)
Since dark pools are private, banks don’t have to immediately report to the public what happens inside them, which means they don’t show real-time market changes. Because of this, traders outside the dark pool have access to less information than traders within the dark pool, which enables the latter to not only move faster on pricing changes, but also to anticipate pricing changes, which they can then profit from.
Lewis explains that if a trader sells a large number of shares on the public market, the price naturally goes down since other traders would see the supply of that stock increase. If a trader sells a large number of shares in a dark pool, however, traders in the public market don’t see that sale happening, and so they don’t know that the price for that stock is about to decrease. This secrecy benefits the dark pool trader, who can sell the shares before the price drops on the public market, thus making more of a profit.
Archegos: Dark Pool Secrecy Backfires on Risky Trading
Experts note that the secrecy of dark pools Lewis describes can be beneficial by facilitating favorable prices for retirement accounts as they rise in value, thus helping people with 401(k) or pension accounts. But critics of dark pools argue that the lack of transparency is a problem and that dark pools aren’t being used for their intended purpose: to reduce market impact when placing large trade orders.
Indeed, investors who abuse this secrecy can face serious consequences. Bill Hwang was arrested for fraud and racketeering in connection with his family firm, Archegos, after a risky investment strategy cost him $20 billion. Since Archegos was investing inside dark pools (so his large trades wouldn’t impact market prices), other banks and institutions weren’t aware of his trades or the terms of those trades, which may have deterred them from doing business with him or prevented him from building such a large market position. Regulators have expressed interest in implementing new disclosure rules to prevent this kind of trading activity.
Eventually, HFT led to changes in the rebate system. Lewis explains that stock exchanges would often charge a small fee to one broker—generally the buyer—of a trading order and pay a small fee (a rebate) to the other broker—usually the seller.
At first, traders followed this traditional rebate system in their electronic trades, but some exchanges catered to HF traders by changing their rules to pay the brokers—generally the buyer—who were usually charged a fee. HF traders used this change to predict the flow of trades—if a broker could be paid for a trade she’d usually be charged for, it’s in her best interest to send it to the exchange where she’d be paid, thus allowing an HF trader to anticipate and monitor where brokers would send orders and could use that knowledge to get there first.
Incentives Don’t Always Work the Way We Expect
We can understand rebates—a kind of trading incentive—through the lens of mechanism design, the process of designing rules to accomplish specific outcomes. Mechanism design is a tricky business—since rule adjustments have such a significant impact on the game’s outcome, small unforeseen side effects can have extreme consequences, as in the case of rebates and HFT. In Freakonomics, Steven Levitt and Stephen Dubner describe how mechanism design can go wrong.
The Freakonomics authors categorize incentives into three types: economic, social, and moral. Economic incentives are when you save money by behaving well, social incentives are when you can maintain a good reputation by behaving well, and moral incentives are when you can satisfy your conscience by behaving well. The authors argue that economic incentives have the potential to backfire by inadvertently removing social and moral incentives. This is certainly true with HFT—which, because it’s algorithm-based, doesn’t operate using social or moral decisions.
For example, a study of ten daycares in Israel discovered that when parents were fined $3 as a penalty for arriving late to pick up their children, the number of late parents doubled. Levitt and Dubner theorize that the fine caused parents to see the obligation as economic instead of social and moral—after paying a few bucks, they no longer felt guilty for being late, thus showing how difficult it is to design proper incentives.
In the same way, the system of rebates, which was originally intended to reward good trading behavior, ended up incentivizing people to game the market.
As the stock market changed due to technology, stock market regulations changed with it. In 2007, the Securities and Exchange Commission (SEC) enacted a new rule that would have lasting effects on the market—the Regulation National Market System (or Reg NMS). This rule stated that traders must buy stock at the lowest market price for investors. So if stock for a company ranges from $10.00 and $10.08, the trader has to buy it at whichever exchange sells it for $10.00. Reg NMS was intended to ease investors’ worries that traders weren’t getting them the best deals.
The SEC then created the Securities Information Processor (or SIP) to make sure investors had access to the same market prices so they could abide by Reg NMS. The SIP collects the stock prices at all thirteen stock exchanges and consolidates this information into one data feed. Thus, it calculates the lowest market price so traders can follow the regulation. Both the new regulation and the new processor would change the way HF traders operated, allowing them to more quickly identify price changes at any given time.
Pros and Cons of Reg NMS
Like Lewis, other experts are concerned about the SEC’s regulations and their impact on markets. Some experts urge the SEC to reevaluate and update Reg NMS, while others recommend a complete overhaul of their rules. While Reg NMS may have positive effects on fair pricing and market competition, critics note that the regulations have led to further market fragmentation, difficulty executing orders quickly, and less efficient prices, showing that financial experts can’t agree on the effects of the regulations. Some experts have even urged scrapping the idea of government oversight altogether and moving toward a system of self-regulation.
Additionally, the increase in regulation price visibility may explain why large investors have moved toward dark pool trading, rather than on the public markets, which would exacerbate the problems Lewis attributes to dark pools (as we’ll discuss later).
Lewis describes order types as sets of instructions determining how traders place stock orders. For instance, a trader can use an order type to stipulate that her order goes through only if she’ll receive a rebate or to cancel her order if a larger order was about to go through.
Because of HFT, exchanges created unusual order types with complicated stipulations that would essentially place HF traders ahead of regular investors in priority, so that, for example, if an ordinary investor tried to buy 100 shares of a stock, an HF trader's standing order could be activated to scoop up those shares first.
HF traders came up with complex order types that hardly anyone understood except for people whose job it specifically was to figure them out. As electronic trading became more popular, the number of order types increased from three to over 150. Lewis believes that most of these order types were actually unnecessary and gave HF traders more opportunities to exploit ordinary investors by gaining insight into their trading strategies and intentions.
Additional Order Type Strategies
Lewis defines order types but doesn’t explore an important related aspect of order types: order-flow trading (or tape reading), which is a trading strategy that involves analyzing the trades that other brokers place, as well as the impact of these trades on the prices of stocks. In order-flow trading, data—such as information about large orders being executed, the volume of buyers and sellers, imbalances between buyers and sellers of certain stocks, and types of orders placed—is published in a “footprint chart.” Brokers can then analyze this data to determine smart short-term strategy trades. Comparing certain order types can help traders trade more effectively. While humans can analyze this data, algorithms are inherently faster.
Another feature of electronic trading was the rise of proprietary traders—also known as prop traders or prop shops—who traded on behalf of a bank or independent financial firm, not the firm’s individual customers. Lewis explains that prop shops used the firm or bank’s own money to trade, rather than relying on the revenue from customer orders. Banks got a small commission from customer orders, but the profit margins weren’t large, so prop trading helped them make more money.
Lewis reveals that because the banks weren’t required to use the same technology for their customers as they did for prop trading, they used a faster telecom connection—such as Spivey’s fiber-optic line—for proprietary trading, but not for their individual customers. Thus their customers’ orders used slower cables, had higher latency times, and were at a speed disadvantage—all without their customers knowing about it.
(Shortform note: Since the publication of Flash Boys, the Federal Reserve has enacted more regulations affecting banks and their proprietary trading. One such regulation is known as the Volcker Rule, which limits what kinds of dealings and investments banks can make with hedge funds and private equity funds. Before the Volcker Rule, many banks had HFT departments, but after its implementation, banks weren’t allowed to have proprietary trading desks or hedge fund investments. Consequently, banks have closed their HFT shops, but many face accusations of past HFT-related misconduct.)
Lewis argues that the above six characteristics of an electronic-based stock market allowed high-frequency traders to exploit individual investors. In Part 3, we’ll explore two of the main ways they do this: arbitrage and front-running. Within that discussion, we’ll describe six specific strategies that give HF traders an unfair advantage, allowing them to profit off the market in ways that are unavailable to regular investors.
Lewis explains that most HF firms make their money from different forms of arbitrage, which is when traders buy and sell a stock at two different prices at two different exchanges. Traders who can effectively time their purchases and sales profit from the temporary differences in stock prices.
Lewis notes that stock values are constantly changing—prices shift multiple times in a second. Every time a stock is bought or sold, its value changes slightly. Usually its value doesn’t change very much—by fractions of a cent—but this small difference in price is enough for HF traders to take advantage of arbitrage.
To illustrate the concept, imagine you buy an old typewriter at a local antique store for $35. You then go to a different antique store, where they offer you $40 for your typewriter. You accept and pocket the $5. This is similar to how HF traders make money from arbitrage, but HF traders complete this process in milliseconds. For example, at NYSE, an HF trader buys 100 shares of stock priced at $200 and sells these stocks at Nasdaq for $200.10, thus profiting off the difference in price.
Arbitrage and Market Efficiency
Lewis argues that the way that HF traders use arbitrage is bad for investors, but other experts point out that arbitrage itself is not inherently bad. In fact, it’s encouraged in the US. Experts believe that arbitrage contributes to market efficiency since it rebalances price discrepancies in stocks and minimizes the spread in stock prices.
This price rebalancing is known as the Law of One Price, which is an economic concept that states that the price of a stock will be the same price in a market free of transaction costs, legal restrictions, and currency exchange rates. The only way to achieve the Law of One Price is through arbitrage. By buying a stock at a low price and selling it at a higher price, market prices equalize.
The problem seems to stem from when certain parties can take advantage of arbitrage because of technological superiority, while other parties can’t. This leads to the imbalances Lewis discusses.
Lewis explains that the most widespread tactic HFT uses is slow-market arbitrage, or latency arbitrage. This is when HF traders use different latency speeds to capitalize on changes in stock prices. HF traders see changes in stock prices on one exchange and buy or sell those orders on other exchanges that haven’t caught up to the changes in stock prices. Slow-market arbitrage is why trading firms care so much about their latency speeds and why it matters to be a microsecond faster than their competitors.
(Shortform note: While Lewis discusses slow-market arbitrage within the US. stock markets, it can also be used in foreign markets. Stock prices on foreign exchanges are sometimes slower to adjust to exchange rates. Traders will purchase these stocks on foreign exchanges for lower prices (before it’s had time to update) and sell them at a higher price in the US market. According to a 2020 study, slow-market arbitrage costs regular investors about $5 billion dollars each year.)
Additionally, Lewis explains that the SIP (the program that determines the best market price for a stock) has a high latency, so it’s slower to obtain and then transmit market data. HFT firms developed their own faster versions of the SIP—which gathers and condenses stock prices from all the exchanges—giving them a slight sneak peek of what the market will do. Then the HF traders use this advance information to capitalize on the price difference.
Because HF traders have access to faster infrastructure than investors, they can see changes in the markets before anyone else. They can then use that advance knowledge to game the market—buying a stock at one price on one exchange and then reselling it for a higher price on another exchange where the price hasn’t updated to the lower price yet.
It’s the same idea as if you went into a popular clothing store chain and bought a shirt on sale. You leave that store with your new shirt and go to the same store in a different location. The second location hasn’t marked down its sale items yet, so you return the shirt for full price, pocketing the difference.
SIP Timestamps Indicate a Lack of Latency Arbitrage
While Lewis believes latency arbitrage is a problem for investors, other experts disagree. In 2015, the SEC changed the SIP data to show the time by the microsecond, as well as showing when exchanges transmit data to their feeds. By comparing these two timestamps, traders can identify the latency of trades.
According to a study analyzing nine months of trading data, traders who used the SIP data actually earned three cents per 100 shares rather than losing money as Lewis claims. The study also found that around 97% of trades happen when data from both timestamps match, indicating that most trades don’t suffer from a lag in latency.
Lewis names another HFT tactic: dark-pool arbitrage, which is similar to slow-market arbitrage. Dark-pool arbitrage is when HF traders make money buying and selling stocks using the price differences between dark pools and other exchanges. Dark-pool arbitrage generates more than a billion dollars a year.
As discussed in Part 2, banks aren’t required to immediately publish what goes on in their dark pool, or private stock exchange, thus allowing investors to trade more discreetly. But because they don’t immediately publish this information, there is a time lag between dark pools and public stock exchanges. Lewis explains that this time lag means dark pools are slow to reprice stocks when their values change on public stock exchanges, thus allowing HF traders to commit dark-pool arbitrage.
For example, say Jordan has an order to buy Tesla stock waiting in a dark pool. Then Tesla stocks decrease in value on the public markets from $300.00 to $299.99. A dark pool is slow to reprice Tesla’s stock from $300 to $299.99 on their exchange. An HF trader buys Tesla stocks on the public exchanges for $299.99 and sells them to Jordan at $300 before the dark pool reprices the stock to $299.99. Thus, Jordan loses money to the HF trader.
European Regulations Discourage Dark-Pool Arbitrage
In response to concerns about dark-pool arbitrage and misuse, US regulators are still exploring the possibility of dark-pool regulation, but European regulators responded more quickly by introducing MiFID II in 2018, which encouraged more transparency and expanded on existing stock regulations from 2007. Under MiFID II, only 8% of a stock’s average trading volume within a year can be completed in dark pools. The regulation also set minimum limits for the size of trades to exclude HF traders, who generally buy and sell small-sized orders, and encourage investors to use dark pools as originally intended: to allow investors to make large trades without impacting market prices. Experts note that these regulations have made dark-pool arbitrage less of a concern in their markets, which could provide a framework for US regulators.
HF traders found ways to exploit different exchange rules, including the changes in the rebate system. As discussed earlier, different exchanges have different rebate rules—sometimes an exchange rewards takers with a rebate and charges makers a fee, while other exchanges do the opposite. To make money off of these new rebate rules, HF traders use rebate arbitrage, which is when HF traders capitalize on differences in rebate rates at different exchanges. HF traders profit from the rebates, not from the stock itself. This tactic doesn’t add or take away stock from the overall market supply—instead, the stock just moves from exchange to exchange.
In rebate arbitrage, HF traders buy a stock for which they receive a rebate. Then they immediately sell the stock for the same price on an exchange where they’ll also receive a rebate for selling that stock or on an exchange where the fee to trade is less than the rebate they’ve already received. Thus the money from buying and selling the stock cancels out, but the HF trader pockets the rebates. Lewis explains that the rebates are cents or fractions of a cent per stock, but HF traders buy and sell large quantities of stocks in a short amount of time, meaning these tiny rebates add up and result in large profits.
For example, BATS might offer a $0.03 rebate for each share bought, and NYSE might offer a $0.05 rebate for each share sold. An HF trader could buy a $50 stock from BATS and receive the $0.03 rebate. Then it could immediately sell that $50 stock at NYSE and receive the $0.05 rebate. The HF trader made $0.08 for moving the stock around, but it didn’t actually help the market. If the HF trader bought and sold 100,000 shares of this stock, they’d make $8,000 from the rebates.
Car Rebates as a Potential Framework for Stock Rebates
While Lewis criticizes rebate arbitrage, other experts aren’t concerned by this tactic, saying HFT should be rewarded for providing market liquidity, or the ease of buying or selling stock. This divide in opinion may explain why US regulators haven’t yet decided whether or not to keep rebates.
Other industries can provide guidance on how to address rebate arbitrage. For instance, car dealerships sometimes offer rebates to consumers, which are usually applied to the down payment of the vehicle. These rebates come from the car manufacturer, and the dealership is required by law to use the rebate—if they withhold any portion of the rebate, they risk legal action. This framework could possibly be used for stock trading in that the rebate can be applied to the stock price, offering a better deal upon purchase, rather than as a kind of additional payment. The car industry also indicates that legal regulation might not discourage market activity.
Lewis also describes front-running, when traders use advance knowledge of an impending trade to capitalize on that trade. The main goal of front-running is to gather information—about stocks, the market, and investors—that other traders don’t know and then to use that information to make smarter trading decisions.
Traders who know a stock is about to be bought or sold can buy or sell that stock for themselves and can buy or sell that stock for a better price. For example, if you know that another trader is about to buy a large quantity of Apple stock, you’ll know that the value of Apple stock is about to increase in price. The demand has increased, meaning the price will probably increase as well. You use this knowledge to buy Apple stock before the other trader, while the stock is at a lower price. Then, when the price rises, you can sell it for a profit.
Lewis contends that HF traders use versions of front-running to gather information about regular investors and other traders. Then they use this information to make educated decisions about profitable stock orders. We’ll discuss three versions of HFT front-running: rebate baiting, electronic front-running, and information baiting.
Gray Areas of Front-Running
While Lewis describes front-running as using insider knowledge to make unfairly advantageous trades, the issue isn’t as black and white as he makes it seem. There are types of front-running that aren’t illegal, such as index front-running.
Index front-running happens when companies are added or removed from an index fund, such as the S&P 500, which manages large collections of stocks. When this information is announced, traders can buy or sell stocks of that company that will be beneficial to them. It’s not illegal to do this since the information is available to anyone looking for it.
This is why some critics don’t agree that HF traders are front-running simply because they receive public information faster than other traders. However, reports revealed that HF traders bought direct access to market data, giving them advance knowledge over other traders, which could be considered a form of front-running.
Lewis discusses another tactic the new rebate rules allowed for was rebate baiting, which is when an HF trader monitors an exchange where human brokers are more likely to send their orders first because they’ll get a rebate. The HF traders use this information to front-run the order to other exchanges that haven’t received it.
(Shortform note: Because rebates may create a conflict of interest for brokers when sending out customer orders, regulators are reevaluating rebate effectiveness. Under Europe’s MiFID II, most forms of rebates are banned to increase transparency about what investors pay for when working with brokers. This regulation could inspire US regulators to also eliminate rebates, as the absence of incentives may not negatively affect markets and could reduce the chance of rebate baiting.)
Lewis notes that while the new trading regulation (Reg NMS) was intended to protect investors by offering the best stock prices, it allowed HF traders to use the rules against investors through electronic front-running, which is when an HF trader spots a broker trying to trade at one exchange and—using this information—races to another exchange to beat him to that particular stock. Then the HF trader buys or sells it to the broker—who isn’t as fast as the HF trader—for a profit.
Lewis notes that Reg NMS mandates that traders buy stocks at the best (lowest) market price. But the regulation doesn’t guarantee that the exchange with the lowest stock price will have the exact number of shares an investor wants. Thus, a broker might have to go to multiple exchanges to fulfill an investor’s entire order. When he does this, his order tips off HF traders of his intentions to buy or sell a certain stock. The HF traders can then race to other exchanges to buy the stock he wants, only to sell it to him at a higher price.
For example, an investor wants to buy 200 shares of Company X. At NYSE, there are 50 shares available at a price of $50.00. At BATS, there are 100 shares available at $50.02. At Nasdaq, there are 200 shares available at $50.01. Normally, a broker would buy the 200 shares (on behalf of the investor) from Nasdaq, since she is guaranteed to get the number of shares she wants at a decent price. But because of Reg NMS, the broker is required to purchase the 50 shares for $50.00 from NYSE, since this is the lowest market price. Seeing this purchase, an HF trader now knows that Company X’s shares are in demand. It uses this information to race to BATS and Nasdaq and buy those shares. Then when the broker’s order for the remaining 150 shares gets to Nasdaq or BATS, the HF trader sells the broker the shares at $50.03. Because of the HF trader’s electronic front-running, the broker pays more than if she’d bought the 200 shares from Nasdaq in the first place.
Could Replacing Reg NMS Solve Electronic Front-Running?
Other experts agree with Lewis’s argument that traders have found ways to exploit well-intentioned regulation, such as Reg NMS, providing HFT with avenues to front-run investors. Some experts even recommend replacing Reg NMS with entirely new—albeit more relaxed—regulation. One suggestion is to outline several factors—not just price—that a broker must consider when executing customer trade orders, such as:
The price of the stock
How secure the market is at the time of purchase
The size of the transaction
The type of transaction
The number of markets researched
How easily the transaction can be fulfilled
Experts argue that by considering more factors of a trade, investors would get a more satisfactory deal, as their trade would be customized to them and their preferences instead of just price. However, as with Reg NMS, this well-intentioned regulation may lay the groundwork for other kinds of investor exploitation as traders find new ways to scheme the markets.
Lewis explains that another front-running strategy is order-type baiting, or when an HFT examines different order types—and the parameters that accompany them—to figure out the intentions of an investor. So, for example, an HF trader’s order type might alert an HF trader that an investor withdraw an order if another broker doesn’t act on it. If a broker does act on the order, HFT uses this information to front-run their order to other exchanges, forcing the investor to pay a higher price.
Because of order-type baiting, HFT accounted for 99% of all orders made, but only half of all completed trades. Lewis believes that most of the orders HF traders made were used to get as much information about investors (and their intentions to buy or sell) as possible, thus hurting ordinary investors.
Order-Type Baiting as Game Theory
Another way to think of HFT and order-type baiting is through the lens of game theory. In The Undercover Economist, Tim Hartford explains that game theory is a discipline adjacent to economics and mathematics, where a “game” is defined as an activity in which predicting another’s actions affects your own actions. Many everyday situations, like driving, are games. When you’re behind the wheel, you drive based on the rules of the road but also based on what behavior other cars on the road are exhibiting. If a car is driving erratically, or too quickly, you’ll likely switch into a more defensive driving mode. If a car in front of you is driving too slowly, you’ll attempt to pass. By analyzing and predicting the actions of investors through order-type baiting, HF traders—algorithmic, rational decision-makers—use this data to maximize their payoffs.
Reflect on all you’ve learned about high-frequency trading.
Do you find HFT to be an unfair presence in the stock market? Why or why not?
Which HFT tactic, if any, do you find the most unfair? Why?
Are there any HFT tactics you think are fair? Why or why not?
These six HFT tactics capitalized on the precedents set by electronic trading. Lewis explains that anyone involved in the stock market—brokers, investors, hedge fund managers, and so on—felt the effects of HFT. In this section, we’ll discuss how Wall Street responded to HFT, as well as the three strategies Wall Street firms adopted to keep up with their tactics. We’ll also describe the “flash crash” that caused investors to question what was happening with trading.
Lewis argues that the Wall Street firms and banks, who were supposed to help investors, adopted HFT techniques despite knowing that HFT made trading unfair for average investors. They did this because they were making so much money. One firm made over a billion dollars in one year from its HFT department alone.
(Shortform note: Lewis is no stranger to exposing Wall Street’s greed: In The Big Short, he argues that greed and short-sightedness were the prime drivers of the financial crisis. All major stakeholders in the ecosystem were fueled by the desire for profit, which made it easy to overlook the other systemic problems below, as he similarly argues about HFT.)
Three ways in which investment firms and banks took advantage of HFT practices at the expense of regular investors were trading against customers, manipulating dark pool orders, and selling access to dark pools:
Lewis asserts that in response to HFT, Wall Street banks used proprietary trading (or “prop trading”) to trade against their customers in their dark pools, thus committing their own form of dark-pool arbitrage. Like HF traders, the banks’ prop traders—traders who work on behalf of the bank instead of on behalf of customers—used dark pools to see customer orders, buy stocks on other exchanges, and sell that stock at a higher price to the customer in the dark pool. Thus, rather than trying to fix the problems caused by HFT, the banks made the problem worse by participating in the behavior that harmed their own customers.
(Shortform note: When trading against their customers, banks also misled customers about what happened with their trades and failed to disclose that their own affiliates were profiting from their orders. The SEC has taken action against some of these banks for misleading customers, resulting in a collective $154 million settlement from Credit Suisse and Barclays PLC after the banks didn’t regulate their dark pools by giving customers inaccurate information and breaking SEC pricing rules. Another bank paid $37 million to stop a similar investigation into their mistreatment of customers.)
Wall Street also took advantage of the dark pool arbitrage made possible by HFTs. When sending an order to a bank and its dark pool, the investor had to trust that the broker (and the bank the broker works for) acted in her best interest. But Lewis believes the banks often didn’t act in her best interest—sometimes, instead of sending her order on to other exchanges where it could be executed, the bank would leave the order in its own dark pool, unfulfilled. So a customer’s order to buy Microsoft stock might sit in a bank’s dark pool, waiting to be paired with a seller's order to sell Microsoft stock. If a seller never enters the dark pool, the buy order would never be completed.
Lewis explains that banks sometimes wouldn’t send on customers’ orders because (1) they made commissions and avoided public exchange fees when they completed orders in their own dark pools, and (2) they wanted to make their dark pool statistics look good because they could then use their good dark pool statistics to advertise to outside investors that they were a good bank to trade with.
(Shortform note: While Lewis argues that the banks’ manipulation of orders poses a problem for investors, there could be a solution: smart routers, which are programs that determine where to send orders based on cost-effectiveness. For instance, a smart router might break up an investor’s large order and send it to a dark pool and an exchange that offers a rebate, thus giving the investor the best deal for their trade. But smart routers do come with disadvantages, such as increased latency and additional complexity required to complete an order. And just as the technology of HFT had consequences, smart router technology may eventually have unintended effects too.)
Banks also made money from selling access to their dark pool to HF traders, despite telling their customers dark pools were safe from HFT. Lewis explains that HF traders were willing to buy access because they could commit dark pool arbitrage, which generates more than a billion dollars a year. Selling dark pool access benefitted both HF traders and the firm at the expense of the individual investor, who (by nature of a dark pool) had no idea how the private trading worked.
So, when banks boasted that they had a large number of trades executed within their dark pool, the high number of transactions was often due to HF traders exploiting investors.
(Shortform note: In 2018, the SEC attempted to address the issues Lewis outlines by passing new regulations for dark pools, forcing banks to be more transparent about completed trade prices, fees, matching processes, and whether or not other traders have priority in their dark pool system. While this was a good first step in regulation, the SEC has expressed interest in implementing further transparency and disclosure surrounding the nature of dark pool operations, indicating the issues it set out to address in 2018 have persisted.)
After years of unregulated high-frequency trading, Wall Street experienced a “flash crash” in 2010. Lewis defines a flash crash as a stock market crash that happens quickly—stock prices drop dramatically before recovering within a few minutes. But during that time, 20,000 stocks traded at dramatically different stock values.
According to Lewis, no one could pinpoint a reason behind the flash crash. Katsuyama couldn’t access the data that could prove HFT caused the crash, since this information wasn’t public—it belonged to the HFT firms and exchanges—but he knew that the flash crash was a result of HFT.
The flash crash marked a turning point in the investing world. Investors were interested about what was going on in the market—and why it had dropped by 600 points so suddenly. People called Katsuyama and his team to set up informational meetings where he educated investors and executives about HFT.
Who Was Responsible for the Flash Crash?
While Lewis blames the trillion-dollar flash crash on HF traders, the US courts found another reason: a self-taught stock market trader named Navinder Singh Sarao. He was convicted of “spoofing,” or creating a huge number of fake buy or sell orders. HFT firms program their algorithms to get out of the market when it gets too volatile, and Sarao made the market appear so volatile that all the algorithms pulled out at once, creating a crash. He capitalized on the changing prices, earning about $900,000 from the flash crash.
While the Commodity Futures Trading Commission (CFTC) found Sarao responsible for the flash crash, some experts contend that blaming Sarao doesn’t get to the heart of the issue—which is the fact that both human and algorithmic traders still have the ability to manipulate markets despite existing regulations.
After discovering these HFT tactics and Wall Street’s greedy response to it, Katsuyama and his team decided to fight HFT rather than mimic its tactics to make money. In this section, we’ll discuss his team’s first attempt at fighting HFT: a trading program called Thor. Then we’ll explore their second attempt: a new stock exchange. We’ll also discuss the six features the team implemented to make trading fair against HF traders’ six tactics.
As their first step to addressing HFT techniques, Katsuyama’s team developed a trading software called Thor. Lewis explains that Thor slowed down a broker’s order so that it arrived at all exchanges simultaneously. While it may seem counterintuitive to go slower, this tactic actually helped Katsuyama’s team successfully fill their orders and saved them money. Slowing the orders down prevented an HF trader from front-running it to another exchange and changing the price of the stock.
For example, without Thor, a broker’s order arrived at BATS in three milliseconds and NYSE in six milliseconds. In the three milliseconds before his order got to NYSE, an HF trader saw his order at BATS, thus revealing that there was a known buyer for a certain stock. The HF trader raced to NYSE, bought that stock, and sold the stock to the broker at a higher price.
But Thor slowed down the speed of the broker’s BATS time, meaning his orders arrived at both BATS and NYSE in six milliseconds. The HF trader then wouldn’t have a time advantage, removing the opportunity to front-run the trader’s order.
The Tactical Approach of Removing Momentum (Speed)
As a first attempt to fight HFT, Thor took a counterintuitive approach by slowing down their latency rather than increasing their speed to match or beat HFT. This tactic inverts a strategy that Sun Tzo outlines in The Art of War, which states that momentum is the life force of any conflict. When momentum is on your side, you have the advantage. You can create this advantage by manipulating your enemy into action.
But by recognizing that they could never match or beat HFT’s momentum (speed), Katsuyama’s team avoided conflict (in this case, HFT’s ability to front-run) by reducing their own momentum (slowing down some of their stock orders), thus manipulating their enemy (HFT) into inaction (not frontrunning their orders) rather than action.
Next, Katsuyama and his team quit their jobs at RBC to create their own stock exchange. They called it the Investors Exchange, or IEX. The purpose of IEX was to make investing fair by preventing the predatory behavior of HF traders.
Lewis believes that they succeeded—IEX ranked number one on a list evaluating exchanges on how well they adhere to trading regulations. IEX also had larger trade orders than other exchanges, as well as more random trades. So if a stock changed price in another market, IEX was less likely to execute trades because of it.
When creating IEX, Katsuyama and his team knew they couldn’t ban HFT on their exchange. But they could take preventative measures against the tactics HF traders used to exploit investors. Lewis discusses the team’s five solutions addressing each of the HFT tactics.
IEX vs. Regulation
While Lewis praises IEX for fighting HFT, some critics note that in doing so, he perpetuates the idea that stock exchanges will correct themselves without regulation. Critics believe that regulating bodies like the SEC—not exchanges or traders—should be responsible for keeping up with changes in stock trading—especially for something as influential as HFT—and taking actions to protect investors. Experts have suggested a small tax on every trade, called the Tobin Tax, that would have little effect on regular investors but deter HF traders by reducing the margins they make on their frequent trades.
However, Lewis and critics seem to agree on the fact that while the SEC should regulate HFT more, they’re not—and if they are, they’re acting too slowly. In 2021—seven years after the book’s publication—the SEC began considering new mandates for HFT firms that would require them to report all of their trades and to be subject to periodic exams. Lewis praises the IEX team for their actions in spite of the SEC’s failure to do so.
As discussed earlier, HF traders have speed advantages that result in slow-market arbitrage (when HF traders watch for stock price changes and then buy a stock at one exchange at a lower price, then sell it at another exchange before the second exchange has updated its prices).
To address this problem, IEX increased the amount of time it would take for any investor—including HF traders—to access the exchange’s market, which erased any speed advantages for HF traders. To do this, they moved their "point of presence,” or a point where traders connect to an exchange, 10 miles away from their matching engine in New Jersey and coiled the cable between the two locations to lengthen the time it took to get a signal from one to the other.
In doing so, IEX created a 350-microsecond delay that slowed down HF traders accessing their exchange and therefore allowed IEX to interact with other exchanges at the same time as HF traders, instead of behind them.
Is IEX Being Unfair to HF Traders?
While Lewis claims IEX’s tactics (the point of presence and 350-microsecond delay) make trading fairer for regular investors, some experts object to IEX slowing down traders’ orders. They believe that IEX is effectively using latency arbitrage against HF traders since the exchange is using old prices on other exchanges before traders on those exchanges can update their own prices, thus making trading with IEX unfair for HF traders. They believe these tactics only benefit big investors and portfolio managers.
However, this argument seems to acknowledge that using “old” prices to perform latency arbitrage is unfair—it’s just a question of whether humans or technology are on the receiving end of it.
Because IEX couldn’t control what happened in banks’ dark pools, they couldn’t directly address dark pool arbitrage. But Lewis notes that the team found a way to indirectly address the issue: They encouraged investors to request that their orders be sent to IEX.
As we discussed earlier, the banks weren’t sending customers’ orders on to other exchanges where the order could be fulfilled, thus allowing an HF trader—who had paid the bank for access to the dark pool—to commit dark pool arbitrage (profiting off small differences in price between dark pools and other exchanges).
Lewis explains that IEX wanted to prevent the banks from allowing customers’ orders to sit unfulfilled in the dark pool. IEX thus encouraged investors to request that their orders be sent to IEX.
(Shortform note: While Lewis claims HFT exploits investors, some experts believe that HFT only really affects wealthy investors, such as the ones managing hedge funds, who charge their retail investors more than what they lost to HFT. Critics accused Lewis of creating victims to make his story more interesting.)
However, many banks initially resisted these requests, as they didn't want to send IEX the business nor did they want to decrease trading volumes in their own dark pools, as high volumes improved their statistics.
To solve this problem, Katsuyama met with a group of influential investors and explained to them exactly how banks had been exploiting them so that they would put pressure on their banks to send orders to IEX. He also met with Goldman Sachs executives and showed them that they would never be able to compete with HFT firms and their advanced algorithms. As a result, Goldman Sachs managers began directing their orders to IEX.
Because of his efforts to educate investors and banks about the advantages of using his (fairer) exchange, IEX soon gained a reputation as an exchange that could solve the problems that HFT had created and make trading more equitable again.
(Shortform note: While encouraging investors to ask their bank to send their orders to IEX was a solution, it didn’t completely fix IEX’s problem since it put the responsibility on the investor to make sure their bank cooperated and sent an investor’s order to IEX. To address this problem, in 2016, the SEC approved IEX’s application to become an official exchange, thus requiring brokers to send their orders to IEX when it offers stocks at the best price. In doing so, it’s not up to the investor to make sure their order goes to IEX.)
As their next solution, IEX eliminated rebates. Instead, IEX charged the people involved in both sides of the trade $0.009 per share. This approach differed from the previous model, which charged one side of the trade and paid the other a rebate.
Lewis explains that without rebates, HF traders couldn’t commit rebate arbitrage or rebate baiting by moving stocks between exchanges to profit off of rebates. In addition, regular brokers didn’t have the incentive of a rebate to send orders to IEX first, which allowed HF traders to front-run her order. By charging both the maker and the taker a fee, traders didn’t have an ulterior motive—the appeal of a rebate—to trade at IEX.
Lewis believes that as a result, traders and investors who truly wanted to buy or sell stocks—and not just receive rebates—were encouraged to trade at IEX. If someone truly wanted to buy or sell a certain stock, they’d be more likely to pay the small fee to do so.
Reevaluating Rebate Effectiveness
Following IEX’s decision to remove rebates from its trading process, the SEC reexamined whether or not rebates are an effective quality of the exchange. Although the SEC still allows rebates, it has admitted that rebates may create conflicts of interest when brokers choose where to send their orders. The SEC has considered testing how markets would function without rebates. NYSE and Nasdaq sued to stop this experiment before it went into effect.
However, the SEC identified advantages to the current rebate model, stating that rebates might make it easier for brokers to identify fair stock prices, while also encouraging trading on exchanges. It also noted that taking away rebates could make stock prices worse for investors. While IEX hasn’t faced any of these issues, their analysis may hold true if all exchanges eliminated rebates.
To reduce information baiting, Katsuyama and his team limited the number of order types to three, as opposed to the 150 offered by other exchanges at the time. Lewis argues that fewer order types made trading more straightforward, encouraging investors who actually wanted to buy and sell stocks, instead of just use order types to glean information, to trade with IEX. By limiting the number of order types, IEX prevented HF traders from baiting information out of investors: HF traders had fewer ways of determining what stocks an investor wanted or how much an investor was willing to pay.
(Shortform note: While Lewis notes that IEX has three order types, the exchange has increased the number since the publication of the book. IEX now has three classes of order types—ones for standard and retail investing, as well as what IEX calls their Signal Series order types, which are supposed to offer “enhanced price protection.”)
To counteract the secrecy of dark pools, IEX decided to publish their trading rules that they use for their matching engine. Lewis explains that they also showed what order types they allowed on IEX, as well as whether or not any other investors were given special access to their exchange. They hoped being transparent would build investor trust and encourage other exchanges to do the same.
(Shortform note: Lewis believes IEX fostered trust with investors through being honest and transparent about what was happening in their trades, which is a strategy other experts recommend. In Algorithms to Live By, Brian Christian and Tom Griffiths note that one simple way to improve the equilibrium of a competitive game, such as stock trading, is to change the rules to make honesty and transparency the optimal strategy. The authors argue that when you design a game in which players are incentivized not to hide their intentions, it eases the burden on everyone. Players no longer have to stress about predicting what others are going to do, or what others think they’re going to do.)
Reflect on all you’ve learned about the solutions to high-frequency trading.
Do you think HFT should be regulated? Why or why not?
If you think HFT should be regulated, whose responsibility should it be? The exchanges? The HF traders? The SEC? An algorithm like Thor? Why?
Consider the solutions Katsuyama’s team implemented. Do you think these strategies effectively address the tactics HFT uses? Why or why not? Can you think of any other solutions?