"There's a guy I know only on the Internet. He's apparently involved in some kind of hassle with Krause. Anyway, he says that Krause has got a rat's-nest-inter-agency intelligence operation going, and one of the things that they're testing is called Deep Data Correlation. The basic concept was supposed to be that they could look at an ocean of data and figure out from that who might be bad guys. Terrorists."
"Is that bad?" The waiter came back with a martini, waited, with me, until Bob nodded. The waiter went away and I continued.
"Not if that was what was happening. but there are some fundamental problems with that kind of data-mining," I said. I explained the numbers problem. "So essentially, what they were trying to do is impossible. But - if you come at it from the other end, starting with a name, then going after associated data, you can develop some pretty powerful tools."
"What a minute," Bob said. "You're saying that instead of looking at the data, and finding suspects, they find a suspect, and then mine some data to support the suspicion."
"Yeah. Except, of course, that you've got to identify a target first. With terrorists, identifying the target is the whole problem. That's the hard part. If they'd been a private company, say, hired to find techniques that would identify terrorists, they'd have concluded that data-mining was a waste of time. But they're not in a private company. They're with the government. So they apparently said to themselves, "Well, data-mining doesn't work, but we've got this great research tool, let's just check it out on a few targets."
-John Sandford, as excerpted from The Hanged Man's Song, published in 2003