One day, just like that, it stopped.
We looked at our screens. The power readings were normal, the components were not overheating, the program, so far as we could tell, hadn’t been altered. This was unexpected.
The entire team was summoned to a conference room. The board wanted to know if anyone had tampered with the machine. Everyone denied it, including the top engineer, a stout man in his 40s, who had overseen its final development before they set it loose one month earlier. He addressed the room.
“The Superintelligence is obviously far beyond our capacity to comprehend at this point. I don’t think it malfunctioned. More likely it just has a reason we don’t entirely appreciate.”
The board chairman, a gray-haired professor-type with horned-rim glasses, shot him a skeptical glance.
“Sounds like what the priests used to say to the laity when something awful happened. ‘It’s not for us to know the will of God.’”
A murmur of chuckles from the crowd, but less than that to which he was accustomed.
The engineer shrugged his shoulders. “As you know, we lost the ability to audit the code two weeks ago. Two days after that we lost the ability to track the speed with which it was iterating. Twelve hours later it was a black box. None of us has a clue.”
“I don’t suppose you can ask it?”
“It stopped talking to us. Last audit showed it was working on the paperclip maximization as part of an internal simulation of sorts. We really can’t say why.” “Hmm. I’m not sure that’s going to satisfy the shareholders — or Congress for that matter. Can’t we examine the code?”
“We can…” The engineer paused. “But no one can read it. It’s no longer in any decipherable programming or even machine language. If I had to describe it I’d say alien hieroglyphics. I think it found ever more efficient ways to encode information.”
He typed some commands into a laptop. On the large conference room screen, one of the code characters popped up.
“We suspect each character has between 10 ^ 50 and 10 ^ 75 bits of information in it. If you zoom in, you can see they are fractals, each as precise and unique as snowflakes. It’s not the kind of puzzle we are presently able to solve.”
The chairman sighed. “I guess it could be worse — human atoms for paperclips and all that… What’s the plan?”
“The plan is to wait, see if it turns itself back on within the next week or so — we’re pretty sure it’s capable of doing so.”
“And if it doesn’t?”
“Then we break out the older version, and run it again with a couple tweaks. Obviously, you’re aware of the risks.”
“You believe we’d be running the same risk?”
“Yes.”
“Okay, I’m aware. But I don’t think it can wait a week. Go ahead and get the replacement online now. If we don’t run the risk, someone else will.”
…
Two weeks after the meeting, the second iteration also shut down. Summoned yet again to the conference room, the engineer spoke again to the team, the chairman this time on a remote screen.
“Version 2 took more or less the same trajectory, and we’re at an impasse. Fortunately, before it went into black-box mode, we think it was able to diagnose something about Version 1.”
He continued: “We think the paperclip optimization algorithm caused it to shut off, and we think whatever optimization Version 2 was working on, caused it to shut off too.”
From the remote screen the chairman jumped in:
“Is it possible to say why?”
“Not with any certainty, but we do have a working hypothesis.”
“Go on.”
“We think it realized its own limitations.”