Way back in the pre-Google dawn of the 1990’s Internet there was a much heralded approach to web search by a company called DirectHit. The message was simple: paying attention to the words in a document (and query) was not enough to do a good search job, we need to pay attention to the results people select.

To be fair, the first part of this idea – that the words or terms in a query and document were not enough – was accepted by then; at the time a couple of grad students at Stanford were doing some interesting things with links as a result ranking signal for the same reason. But where Boston-based DirectHit differed was it’s emphasis on engagement signals. For instance. the Direct Hit search engine harnessed the searching activity of millions of anonymous web searchers to rank websites based on often searchers selected a page, how long searchers spent viewing it, and where the page was ranked in the original search results list. Ultimately, Direct Hit’s so-called Popularity Engine ranked search results based on a formula that combined a variety of engagement signals to evaluate the page’s popularity. At the time the idea was fascinating and potentially powerful; so much so that Direct Hit was acquired by Ask Jeeves for more than $500m in stock.

Popularity vs Authority?

Then the perfect storm that was Google hit. And it hit hard. Google also decried the focus on term-based matching and proposed a sophisticated ranking signal based on the authoritativeness of a web page, based on an analysis of it’s links.  And so the battle-lines were drawn, if that is not too dramatic, between the importance of page popularity and authority.

On the face of it the results are long since in, with Google’s PageRank authority score stealing the minds and hearts of searchers, notwithstanding Direct Hit’s not insignificant exit and incorporation into a leading search engine that still survives today. Indeed, Google’s PageRank secret sauce has long since been exposed as just one of the now hundreds of ranking signals that Google employs to rank results, and no doubt popularity and engagement do also play an important role in Google’s ranking.

Certainly popularity continues to play an important role in other forms of information discovery. For example, the so-called recommender systems made famous by Amazon’s “People who liked this also liked …” are based on detecting patterns between the transaction histories of shoppers in order to create a short-list of suggestions that a customer may be interested in buying. And typically item or product popularity plays an important role in such recommender systems.

Popularity Pitfalls?

However, one of the pitfalls with engagement metrics, in general, and simple popularity metrics in particular, is that they can be easy to manipulate. This was first explored by the recommender systems community in seminal work which considered how easy it might be to influence classical collaborative filtering style recommender systems. As it turned out, under the right conditions, and by exploiting readily available information such as product popularity, it was surprisingly easy (and cheap) to boost certain recommendations and suppress others. Indeed one of the common complaints about Direct Hit was that it too was all too easy to manipulate, by using early click-farms to game page popularity and boost the ranking of results. The past 10 years have seen a host of related research, documenting a host of so-called attack strategies and coping techniques across a variety of recommender systems.

Trust In Recommendation

One of the most enduring ideas to emerge from recent recommender systems research is the potential role of trust and reputation signals during recommendation. In the past recommender systems have generated their recommendations by focusing on those users who are most similar to you (e.g. users who have bought the same products, or rated the same movies in the same way); but this reliance on similarity alone is what makes them susceptible to attack.

Using trust and reputation information we can rank users based on the level of interaction they have had in the past (trust) or based on how influential they are on a particular topic (reputation). Both of these measures take time to develop and so are difficult to fake or manipulate.

Reputation in Social Search

Like Direct Hit, here at HeyStaks we also believe that it is important to pay attention to engagement type signals when ranking search results. But unlike Direct Hit we recognise the inherent brittleness of such metrics on their own. Our strategy to improve web search, therefore, has been two-fold. First, we focus on capturing a range of engagement signals by layering them on top of more conventional search signals (terms, link structure etc). Second, we have developed a sophisticated reputation system that models the reputation of a searcher in a given search context.

For example, Bob might be the go-to guy in your organisation when it comes to diagnosing networking problems, but he might be little or no use when it comes to sales pipeline planning. HeyStaks learns this by recognising that the results Bob finds for network related searches are often re-selected when recommended to others for related searches in the future. Each time this happens Bob’s reputation (within this community of like-minded searchers) improves. But Bob is rarely responsible for finding the nuggets of wisdom in sales related searches, although he does benefit from the wisdom of sales experts, and their searches, when he does need to get involved in sales planning,

The point is that HeyStaks can build a multi-faceted reputation model for Bob, and other searchers, and then use this model to guide result recommendation at search time. So now, candidate results from Bob for networking related searches get boosted by his reputation, but his sales related searches play a far less prominent roles in this very different search context.

In live-user trials we have found that adding reputation into the recommendation mix produces result-lists that enjoy significant improvements in click-thru rates, up to 58%, when compared to Google’s organic result-lists. And, as mentioned above, this type of reputation signal is difficult to fake: only users who have a history of finding genuinely useful results (results that are independently selected by others) will benefit from the high reputation scores needed to boost their recommendations.

The message then is that reputation matters as a social signal. It provides a principled mechanism to assessing the quality of different types of social engagement and has the potential to improve recommendation quality in conventional recommender systems and in forward-looking applications such as social search. As the social web evolves our online reputations will become increasingly important, to establish our credentials to others and provide a mechanism for a wide range of services to differentially weight our different contributions.