Taking Spying to Higher Level, Agencies Look for More Ways to Mine Data
E-Mail This
Printer-Friendly
Reprints
Save Article


By JOHN MARKOFF
Published: February 25, 2006
PALO ALTO, Calif., Feb. 23 — A small group of National Security Agency officials slipped into Silicon Valley on one of the agency's periodic technology shopping expeditions this month.


Software for Spies


A Technique for Spotting Connections
On the wish list, according to several venture capitalists who met with the officials, were an array of technologies that underlie the fierce debate over the Bush administration's anti-terrorist eavesdropping program: computerized systems that reveal connections between seemingly innocuous and unrelated pieces of information.

The tools they were looking for are new, but their application would fall under the well-established practice of data mining: using mathematical and statistical techniques to scan for hidden relationships in streams of digital data or large databases.

Supercomputer companies looking for commercial markets have used the practice for decades. Now intelligence agencies, hardly newcomers to data mining, are using new technologies to take the practice to another level.

But by fundamentally changing the nature of surveillance, high-tech data mining raises privacy concerns that are only beginning to be debated widely. That is because to find illicit activities it is necessary to turn loose software sentinels to examine all digital behavior whether it is innocent or not.

"The theory is that the automated tool that is conducting the search is not violating the law," said Mark D. Rasch, the former head of computer-crime investigations for the Justice Department and now the senior vice president of Solutionary, a computer security company. But "anytime a tool or a human is looking at the content of your communication, it invades your privacy."

When asked for comment about the meetings in Silicon Valley, Jane Hudgins, a National Security Agency spokeswoman, said, "We have no information to provide."

Data mining is already being used in a diverse array of commercial applications — whether by credit card companies detecting and stopping fraud as it happens, or by insurance companies that predict health risks. As a result, millions of Americans have become enmeshed in a vast and growing data web that is constantly being examined by a legion of Internet-era software snoops.

Technology industry executives and government officials said that the intelligence agency systems take such techniques further, applying software analysis tools now routinely used by law enforcement agencies to identify criminal activities and political terrorist organizations that would otherwise be missed by human eavesdroppers.

One such tool is Analyst's Notebook, a crime investigation "spreadsheet" and visualization tool developed by i2 Inc., a software firm based in McLean, Va.

The software, which ranges in price from as little as $3,000 for a sheriff's department to millions of dollars for a large government agency like the Federal Bureau of Investigation, allows investigators to organize and view telephone and financial transaction records. It was used in 2001 by Joyce Knowlton, an investigator at the Stillwater State Correctional Facility in Minnesota, to detect a prison drug-smuggling ring that ultimately implicated 30 offenders who were linked to Supreme White Power, a gang active in the prison.

Ms. Knowlton began her investigation by importing telephone call records into her software and was immediately led to a pattern of calls between prisoners and a recent parolee. She overlaid the calling data with records of prisoners' financial accounts, and based on patterns that emerged, she began monitoring phone calls of particular inmates. That led her to coded messages being exchanged in the calls that revealed that seemingly innocuous wood blocks were being used to smuggle drugs into the prison.

"Once we added the money and saw how it was flowing from addresses that were connected to phone numbers, it created a very clear picture of the smuggling ring," she said.

Privacy, of course, is hardly an expectation for prisoners. And credit card customers and insurance policyholders give up a certain amount of privacy to the issuers and carriers. It is the power of such software tools applied to broad, covert governmental uses that has led to the deepening controversy over data mining.

In the wake of 9/11, the potential for mining immense databases of digital information gave rise to a program called Total Information Awareness, developed by Adm. John M. Poindexter, the former national security adviser, while he was a program manager at the Defense Advanced Research Projects Agency.

More at NY Times