Thoughts on Wall Street Technology

Last week I went on vacation and missed the news that a software error at a major market maker caused almost half a billion dollar worth of trading errors, which required a rescue by a group of investors. Mainstream media is quick to point out that high frequency trading should be more regulated, but as a software engineer I find that this has more to do with process and management than with the trading technology or software itself.

As I’ve been gathering my thoughts on this topic for the past few days I came across an article by Robert Dewar, one of my most respected professors at NYU Courant Institute where I did my masters. Dr. Dewar’s main point is that safe and trustworthy software is possible if the company has proper processes and a proper culture in place.

Wall St. companies could learn from this. Companies so dependent on technology should not be run by traders — they should be run by technologists. From CEO level down to the humblest engineers, these companies must be staffed with technically minded people who have the right training, the right attitude, and the right thought processes to build reliable and trust worthy software (but such developers are never on the market, so this creates a conundrum, right?)

How many Wall Street job openings have you seen that proclaim “fast pace” as one of the exciting aspects of the job ? To be clear, when a recruiter contacts me about such a job opening I immediately brush it off. It is this “fast pace” of development, this requirement that something requested this morning must be in production by midnight today, is what time and time again gets Wall St. technology into a pickle. As one of the commenter’s on Dewar’s article said, senior management and sales should not under any circumstances be allowed to rush software development process. Human mind is simply not capable of accelerating its cognitive processes, and therefore forcing developers to build software faster is counterproductive.

Following this line of reasoning that developers are humanly unable to think faster than they do, production support and active development must be thought of as two distinct areas that should be staffed by different people. Under no circumstances should engineers be forced into a situation where they are expected to deliver working code — while also responding to phone calls from traders and customers every fifteen minutes. Bug fixes are too a distinct activity from software development.

Dr. Dewar concludes his article:

High-frequency automated trading is not avionics flight control, but the aviation industry has demonstrated that safe, reliable real-time software is possible, practical, and necessary. It requires appropriate development technology and processes as well as a culture that thinks in terms of safety (or reliability) first. That is the real lesson to be learned from last week’s incident. It doesn’t come for free, but it certainly costs less than $440M.

NASA’s overall budget is about $10 billion per year. Wall Street technology spending is about $40 billion, depending on who you talk to. NASA is able to find developers and subcontractors who can program a robot from 250 million miles apart, or safely launch humans into orbit. Wall Street could benefit from recruiting the same kind of engineers and managers and offering them meaningful career paths.