I have been meaning to write a post on Stowe Boyd’s Conversation Index for awhile, but for whatever reason didn’t get around to it. Then late last week I read that a media monitoring firm had started using it as a metric to track the amount of conversation a blog entry generates, and now I feel compelled to comment. As background, here’s Stowe Boyd’s description of the Conversation Index that started it all:

“While working at Corante, I had the opportunity to peer at the stats for all sorts of blogs that we had going. And one thing that became really obvious is that sucessful blogs — ones that were currently viable and vibrant, and those that were on a growth trajectory from their start — shared a common characteristic: The ratio between posts and comments+trackbacks (posts/comments+trackbacks) was less than one. Meaning that there was more conversation — as indicated by the number of comments and track backs offered by readers — than posting articles. I will call this the Converation Index, just to put a handle on it.”

I think its fantastic if you want to use the Conversation Index as an internal metric of your own success at generating conversation on your blog. Watch how it changes over time. Compare the ratio of different posts to each other. Go for it.

However, the Conservation Index is not valuable as a universal measure for comparing blogs (and blog posts) to each other. Why? Generally speaking, because measuring the conversation generated by a blog post is a lot more complicated than that. Specifically, because there isn’t a univeral method for dealing with trackbacks and comments. You end up comparing apples to watermelons:

(1) Not all blogs accept comments. And some don’t take trackbacks. Technorati’s most popular blog, Boing Boing, doesn’t allow comments or trackbacks in the traditional sense. Other prominent blogs that don’t take comments include Instapundit (#16), the official Google blog (#11), Andrew Sullivan (#72) and Michelle Malkin (#12). And those are just from memory. How do you accurately show the conversation generated by these influential sites using the Conversation Index?

(2) Some blogs moderate comments and others require registration to comment. I know Micro Persuasion (#69) moderates. I’m sure other popular (and influential) blogs do as well. I’ve also abandoned the idea of leaving comments many times when I was asked to register (Personal Democracy and Doc Searls jump to mind). These tactics employed by bloggers to prevent spam/bad language/thread hijacking lead to less conversation. These kinds of sites will have artificially low ratings.

(3) Different categories of blogs attract different levels of participation. Some blogs ask open ended questions that invite a lot of discussion. Others don’t. Personal blogs often attract more comments than more professionally oriented sites, as Rohit Bhargova of Ogilvy has pointed out. In these cases, the level of conversation (particularly the number of comments) says more about the kind of blog it is than its influence.

There is more I could write. Are comments and trackbacks really of the same value (I place more value on trackbacks)? Aren’t links to an article more important than trackbacks (lots of bloggers don’t use trackbacks)? How do you account for comment and trackback spam? How do you deal with sites that have massive open threads that attract comments about what people had for breakfast?

There are too many holes for this to be used as a universal metric in blog monitoring. At its best, it provides an antecdotal measurement of the amount of conversation generated. At its worst, it could lead a client to a false conclusion about the true impact of a post or blog.

Utlimately, I think measuring the conversation generated by a blog is more complicated than adding a few numbers together. I think Tom Foremski over at Silicon Valley Watcher got it right in a recent post:

“Finding the right metrics to measure a blog’s value as an influencer will never be as simple as measuring numbers of links, comments, trackbacks, Alexa rank, Technorati rank, etc. Because you have to understand the context of each blog and how it fits into its online communities. And you can only do that by being involved in those communities, online and offline.”

Disclosure: The Bivings Group has its own media monitoring product, ImpactWatch. We don’t use the Conversation Index.