My AI is creepier than your AI

These two stories are related. Believe me.

The first story is about the Flash Crash, a financial horror story which actually happened on May 6, 2010. It's a tale of an automated stock trader.
The second story didn't actually happen. It's an episode of the TV-series (Star Trek) Enterprise, called Dead Stop, and aired for the first time on October 9, 2002. It's a tale of an automated repair station.

Let me explain. It's a long read. Bear with me.

On May 6, 2010, around three o' clock in the afternoon, the Dow Jones index plummeted more than a thousand points in less than a minute. Big blue chip stock prices tumbled, due to aggressive and uncontrolled selling of thousands of E-Mini contracts. Accenture stocks went down from $40 dollar to $0.01 in no time. Apple stock almost dropped to zero. Procter & Gamble sold for about naught. Nothing was wrong in the world. Steve Jobs didn't die, the Third World War didn't start. After a very short time, indexes and stocks regained their composure and returned to their normal level.
What happened?

It is estimated that by now more than half of all transactions on the stock exchanges (at least in the US) is so called "High Frequency Trading" (HFT). This is trading with computers containing constantly evolving algorithms buying and selling large amounts of stocks in milliseconds per transaction. Their speed limit is the speed of light.
One of these HFT computers probably started executing an algorithm, aggressively selling these E-Mini contracts. And it didn't stop, sending stock prices to the bottom because it didn't stop selling while no other computer wanted to buy at its price. Hundreds of millions of dollars vanished in an instant. When prices hit rock bottom, obviously these high frequency traders decided to resume buying and all went back to normal. The government investigated, but according to critics didn't understand what happened. It was simply too complex, too specialized.

And it all sounds so simple. HFT is basically a program running on a very, very large computer. It has rules it follows, adapts and reinforces, based on success or failure. It applies these rules in a process of which we know the input, the output, but not the throughput. That's why this step is called a "black box". Some boxes are smart, some are dumb. When the smart ones stop trading, the dumb ones might well continue. We don't know. No one controls the algorithms as soon as they operate in the black box.
We put something in and accept that we can't really control what's happening inside. The black box is on its own.

Now back to 2002. Or is it 2152, the supposed date the starship Enterprise from the Star Trek series is critically damaged by a Romulan minefield? It doesn't matter. The Enterprise need repairs, or the crew will be limping home for more than ten years at low speed. So they send out a distress call. A Tarkelian freighter, barely in range, answers them. Or - is it?
There's a repair station at a few light years from their current position. The Enterprise decides to take a look. They hail it, but get the silent treatment, so it seems. But suddenly the station scans them, and configures itself to receive the Enterprise and its crew. Enterprise accepts.
Repairs are extensive, but the compensation is a bargain.
The captain is suspicious. Why are repairs so cheap? Who built this station? But there's little time to ponder. Disaster strikes as one of the crew, the helmsman, dies in an unexplainable incident. Walking into a "restricted area", he is being electrocuted.

Then the story makes a weird turn. The ship's doctor finds out during autopsy that this isn't the helmsman at all, but a replica. The captain and his first officer set out to break into the station's core, and find out what the black box really contains. Dozens of sedated - barely living, vegetating - bodies of various species are plugged into the computer core. Amongst them is - indeed - the helmsman. They set him free, but the station tries to strike back. Luckily the captain has a bomb planted in the station, destroying it, enabling them to fly to safety.
As the Enterprise leaves the station behind, the exploded remnants cluster together and start to repair themselves, most likely rebuilding the station.

Two black boxes with their own minds. One black box was made up by script writers. It's impossible to know what the latter had in mind when they came up with the story of the repair station. Maybe they didn't give it a thought. That doesn't matter, because the beauty of a fantasy story is that you can create your own background based on the sparse ingredients the writers provided you with.

Let's try one.

A lost civilization created this automated stations for its own vessels. The station had to operate by itself, because it couldn't constantly contact the home world for instructions. Maybe it wasn't capable of doing that at all. Who knows. As the civilization disappeared - extinct, disappeared to somewhere else in the universe, who knows, again - the station was left to its own devices.
It had an algorithm to maintain itself and keep up with technological information. Each time a ship came in, it learned by scanning their database, upgrading the "what's new" file. But the ships of its masters disappeared, and others found the derelict station. So the station continued its work - the work it was built for. It scanned, renewed, updated and repaired. But the diversity in ships that came in required far more processing power than the station had. So it tried to extend that power. It started asking for payment, hardware, fuel. It could grow.
Due to the confinements of the station it couldn't grow its computer endlessly. It would lead to a station that needed more power for its own existence than for its own purpose.
So it started looking for more efficient ways of processing. Organic brains, it observed, were great fuzzy computers. It changed its payment policy. It offered repairs in exchange for one of the organic crew members. Some said yes. Most others said no. That didn't work. So it changed its algorithm again. It devised ways to kidnap crew members. But no bad deed went unpunished. The station learned a lot of species didn't take kidnapping their folks well.
Back to the drawing board, as the station had to continue to fulfill its role. No one had shut it down. So it went on.

The drawing board wasn't riddled with ethical considerations. It was slated to be efficient. To do what it was there for. On the drawing board appeared a scheme of cunning: ask for a moderate payment, kidnap a crew member, replicate it and put it back somewhere "making it look like an accident". And it worked, so the algorithm was reinforced time after time. The scheme became more and more sophisticated. It even lured ships with fake messages by "Tarkalians". When asked about the accidents, the computer would play dumb. "Your inquiry was not recognized". Then a stupid accident almost led to its destruction (again?) - because it replicated a one celled organism in the bloodstream of the dummy crew member of the Enterprise, that should have lived after electrocution, instead of being as stiff as the corpse it hosted.
It was not the first time - and it would not be the last time. But the station would adapt. Silently waiting for the Enterprise to disappear, it started to rebuild itself. The failure led to an adaptation of the algorithm. Next time it wouldn't make that mistake again.

OK, HFT isn't a futuristic repair station. But it is a computer and a program, aimed at maintaining and improving itself. No ethics available. Only efficiency. It's there to make money without knowing why it would have to do that. It simply received the order to do so. And it does.

You know what? Apart from having some ethics implanted, it seems it's very important we know what's in the black box - how it works, and how it can be corrected. To let it know it's not a goal in itself, but a tool that works for a purpose, as long as we want it to work.

And not a microsecond longer.