Today I took part in a lively panel about enterprise search at ACM's SIGIR (Special Interest Group for Information Retrieval) conference in Boston, along with Sue Feldman of IDC and Whit Andrews of Gartner. There was much debate, but none as prominent and lengthy as the one on relevance.

Whit recounted a story of a grocer who sells milk. He sells six different kinds of milk. Which should come up first in the search results? Which milk is the most.....milky? Whit pointed out that such results are often financially-driven, and that the milk that turns up at the top is likely whomever is paying. "Relevance is money," he said, "And high ranks in search results based on money are ok if you're transparent about it."

Before Sue Feldman could finish the simple phrase "I disagree" (thank you, Sue), I launched into a tirade. I pretty much lept down Whit's throat. It's not ok if you want to select your milk based on different criteria, such as organic or hormone-free, but all that determines relevance is money. And in an enterprise search scenario, is one department going to pay the developer tweaking the index, so their document comes out on top? What if the most relevant result on the Web has nothing to do with a financial deal? Have we really become so bereft of concern for facts, quality and context that we'd rather just nod and embrace whatever or whomever has the most marketing dollars? Well, in many cases, yes -- that's exactly what people do, many without realizing it. They just click on the first result.

Whit, a charming southerner who seems to love "stirring the pot" on a panel even more than I do, later specified that his example only applied to certain scenarios. Still, the issue is one all searchers should be aware of. When you search for something and look at the results list, do you know what's driving relevance ranking, and is it in line with your best interests? If ranking is determined by who pays, do users know that? I'd disagree that a company is being "transparent about it" if the notice about paid relevance ranking is in the fine print on the "About Us" page.

Later in the day, a panel of vendors was asked, "What makes the relevance ranking of your engine unique?" The answers weren't about specific criteria used to determine relevance, but rather about the flexibility that allows search engine implementers to hone relevance ranking based on content and context. "Control, exploration, flexibility, tunability," were the answers expounded by representatives of Microsoft, Endeca, and Vivisimo. Relevance is in the eye of the beholder, but relevance ranking is driven by the search engine. Know what criteria are driving the ranking of the results you're looking at, or at least, be skeptical of them.