Boston Globe, Carolyn Y. Johnson

We are what we click.

In an age where everything from friendships to financial planning has moved online, the pulse of pop culture, the ups and downs of the housing market, and even the subjects of fifth-grade homework assignments can all be measured by tallying search terms and mouse clicks.

"You can use online data to predict what consumers are doing across every facet of their lives," said Stephen DiMarco, chief marketing officer of Compete Inc., a Boston firm that tracks Web traffic. "The Web is so mainstream and so ingrained . . . this is kind of like the dawn of a new age."

It also has created a new frontier for companies vying to be the Nielsen of the Net. From Nielsen//NetRatings to companies such as Compete, several companies are scrutinizing the way people surf the Net. Clients such as the US Postal Service pay for Web analyses when they want to know if receiving a catalog makes a person more likely to shop online. Telecommunications firms sign up to see whether their customers are checking out rival services.

Some Web analysis companies buy their data - stripped of information that would identify individual users - from Internet service providers. Others depend on data from people who download an application that shares their Web history, without their names. Some also use surveys, or blend data from multiple sources.

The companies have created a viable business out of measuring what people do online for marketers, car makers, hoteliers, and other businesses. Analytics firm comScore, for example, reported revenues last year of $66 million and went public this summer. Another, Hitwise, was purchased this year by Experian for $240 million. In August, Compete raised $10 million in funding from investors.

Findings about Web trends can vary from company to company because of their different approaches to collecting data, and the Interactive Advertising Bureau, a trade group for the interactive advertising industry, has been advocating for greater transparency in online audience measurements. But there is no question that the clickstream created as people log into online bank accounts, social networks, or shopping sites can give businesses sharper insight into what their customers are like, and whether efforts to reach them are working.

Bill Tancer, general manager of global research at analytics firm Hitwise, was surprised to find that searches for "prom dresses" were high in January. He later learned that advertisers had stretched the prom advertising season earlier, and the search spike was a sign the message was reaching its audience.

Hedge funds and investment banks buy data from Web analytics companies, seeking guidance from clicks as they issue reports and analyze their portfolios.
"What we do with the data is we look for trends," said Robert Peck, a managing director at Bear Stearns who uses data from comScore. It "gives us an edge on calling the quarter versus some of our competitors."

Gian Fulgoni, chairman and cofounder of comScore, said his first hint of the power hidden in his data came shortly after he noticed that online book sales dropped off after Sept. 11, 2001. The decline caught his eye, but he ignored it until weeks later, when book retailers began predicting shortfalls in revenue - suffering from a decline in the economy.

ComScore began developing other ways to use the online world as a periscope into reality. In a study for the US Postal Service, the company found that people who receive catalogs make 16 percent more visits, view 22 percent more pages, and spend 15 percent more time at retailers' websites. The Postal Service published the results to encourage retailers to ship more catalogs.

Compete worked with Carlson Hotels Worldwide to figure out when in the booking process online visitors abandoned their real-world reservations. It advised the company to tweak the website, and Carlson saw 56 percent more of its online visitors become real-world guests over a two-year period.

Honda Motorcycles used Hitwise data to figure out where its online visitors were coming from, how their website compared with competitors', and discovered new ways to drive visitors to the website and measure its marketing campaigns.

"There is nothing I know of in the offline world that is as broad, as fast, as granular as this stuff," Fulgoni said. "No research company could afford to maintain a panel of 2 million people in the offline world."

Online data have their pitfalls. Trying to discern patterns in much online activity seems futile. Google's Hot Trends, a free service that provides a glimpse into the collective curiosity, for instance, found queries ranging from the news of the day - "chrysler strike" - to apparently random terms, such as "tchotchkes," or "celebrity twins."

Even when clear patterns appear, they don't necessarily predict overall public preferences. If candidate website hits were votes, dark-horse Republican candidate Ron Paul would be running ahead of Mitt Romney, John McCain, and John Edwards, according to Compete's data during September. Tancer noted last month that searches for "Ron Paul" outweighed those for most of his Republican rivals, too, and suggested that Paul's audience may not be captured by traditional polling.

And all searches are not equal. Last year, Tancer dissected American Idol search data, finding that Taylor Hicks and his competitor Katharine McPhee drove Web searches. But McPhee's name was paired with words like "wardrobe malfunction" or "yellow dress" - so he picked Hicks as the likely winner on his blog, and was right.

But the power of Web analytics extends beyond reality television, too. By tracking visits to state unemployment websites and search terms, Tancer created a tool to help predict the direction that the unemployment numbers will take - before they are reported by the US Department of Labor. By watching search terms like "homes for sale" or tracking building and construction websites, he has created tools to predict a downturn or upswing in the housing market.

With the intense attention given to such indicators of economic health, real-time insights into the market could be useful to the investors, banks, economists - even anthropologists.
While the data are never linked to individuals, analysts see humanity in the stream of data.

When AOL search data were mistakenly published last year, personal details emerged, from the mundane - someone with a Scion XB needed brake pads - to the serious query, "how to tell your family you're a victim of incest?"

Marketers used to ask "Will it play in Peoria?" Today, they don't need to settle for the residents of one archetypal city to tailor their pitches.

Marketers "understand you a lot better than you want," said Robin Hanson, an economics professor at George Mason University. "Somehow, you think you're on equal ground. You're buying your third stereo - he's sold 3,000."