How Useful is NLP for Sentiment Scoring? Is Brand Monitoring a Realistic Goal?

Matt O’Hern just posted about yet another brand monitoring service (this time from Vocus) who claims to track sentiment.

“Vocus spreads your name to the top bloggers and news sites, but it also offers Sentiment Analysis, a tool that tracks press, positive or negative, regarding your company — in real time.”

After reading the inside story of Monitor 110’s demise, and having developed our own sentiment tagging system, I immediately said to myself, “‘Real time’ huh? Looks like another system where one out of four items is scored wrong…”

Thankfully, the comments contained the real meat of this story — and validated my impressions. First Martin Edic of Techrigy pounced, and injected a dose of reality:

“Be very careful in writing about claims of sentiment analysis. Unless (and this is a big unless) they are using actual humans to read each instance of apparently negative/positive commentary the sentiment is only a very rough indicator. No algorithm, NLP or other machine reading analysis can be very accurate with today’s technology. Machines are simply not good at context semantics, sarcasm, irony, etc.
When I talk to people about our sentiment analysis I’m very careful to make this clear. It is just an indicator until you drill down on a granular level to make the determination yourself…One more thing: my note above and our experience is based on performing sentiment analysis on over 500 million social media search results of all types over the past six months.”

Then no less than Kye Strance, the Director of Product Management at Vocus stepped in:

“NLP and automated sentiment analysis is certainly a useful tool for those looking at trending and real time analysis.”

(emphasis mine)

“We found that people who are looking for up to date analysis in real time, were comfortable with NLP technology that provided 80% accuracy. For those that time is critical and not able to wait until weeks or months end for human analysis, being able to see a trend throughout the day or over any period of time, this is a great approach.

I completely agree with Martin in that if you are looking to send an article straight to your CEO that has been analyzed by an engine thinking it is a positive article, you would certainly want to review it before hand. But I have also seen articles reviewed by humans that should have also been reviewed before being sent as well to save embarrassment.

While it is not able to detect sarcasm and irony, the one thing it can do is be consistent in analysis. A sentiment engine does not show up for work in a bad mood or get tired through out the day, so if you are looking for high level analysis that is consistent, NLP analysis is a great tool to do the major leg work for you.”

My question is what items do you NOT have to review before the CEO reads it? A negative could be neutral or positive, a neutral could be negative or positive etc. etc.

Sounds to me like traditional NLP is what you need for general trends (financial/investing,) but is not that useful for brand monitoring. Considering what happened to Monitor 110, it may be not even that good for seeking alpha…

This is why we think we have a winner with our non-traditional automated text analysis service that:

* Provides human-level (90 percent plus accuracy) scoring.
* Can detect context semantics, sarcasm, and irony.
* Provides low/zero latency (immediate) results 24 hours a day, 7 days a week.
* Never shows up for work in a bad mood or gets tired through out the day.
* Doesn’t require that a VC achieves hurdle rate for their money.