At its core, sentiment analysis tries to answer the following question: “Is this content positive, negative or neutral toward Topic A?” Manual sentiment asks people this question directly, while automated sentiment tries to answer the way a person would. However, both approaches require a set of posts separated into positive, negative and neutral groups by people. It is this need for human-classified data that causes instability in common sentiment analysis. There is an inherent subjectivity in the sentiment question that causes much disagreement among social media users, making it an unreliable method.

And this is exactly what I want to talk about at next years SXSW event! In my session I will explain how the inherent subjectivity of sentiment analysis can lead to unstable sentiment algorithms. We will also demonstrate how companies can reduce the amount of subjectivity in the sentiment question and implement a more effective alternative to sentiment measurement called Hierarchal Topic Discovery.

But I need your help to vote for my session! So if this sounds good to you I would appreciate it if you would click here and vote for my session!

In a recent blog post called The Problem with Automated Sentiment Analysis from Fresh Networks, a social media agency, they evaluated a few sentiment tools and their results are quite similar to what we’ve found in a number of our own experiments:

-       About 80% of posts are neither positive nor negative.

-       Sentiment tools “accuracy” of 70% to 80% is largely driven by their ability to correctly label neutral posts.

-       “In our tests when comparing with a human analyst, the tools were typically about 30% accurate at deciding if a statement was positive or negative”

From the blog comments, it’s clear that the companies in this space are doing their best to obfuscate the truth. To some’s credit, they do state that sentiment alone is not enough information to derive any conclusions.

However it’s NOT better than nothing, it’s actually worse than doing nothing because you are getting INCORRECT information.

With sentiment there is no such thing as accuracy, there is only agreement.  The technology can’t become more accurate, it can only agree with people more often.  And, “sentiment” does not mean the same thing to all people in all situations.  You can’t get more “accurate” at “sentiment” because what you are actually talking about is trying to solve hundreds or thousands of slightly different problems with one tool.  Until we can map the human brain into a program or electronic circuits, I just don’t think that is going to happen.

I completely believe that having inaccurate sentiment is worse than having nothing.  Here is a good example.  In posts about “Blackberry” that have been classified 3 different times by hand, about 32% of posts are positive (with a majority vote).  When we take that same data set and have each post classified 10 times, now about 10% of posts are positive (with a majority vote).  And, if we only consider the posts we are confident in, only about 3% of posts are positive.

So, which is it: do 30% of people like “Blackberry” or do 3% of people, because that’s a BIG difference.  Of course, the answer is probably neither because we aren’t actually measuring how many people like “Blackberry”.  Unfortunately, that’s how it can be interpreted.  Hence, bad information can be worse than no information.

Marketers need to be ware that a lot of these companies say they do monitoring and provide analytics like sentiment but in reality they are really keyword-focused listening platforms with limited analysis capability. If you really want to go beyond sentiment analysis you need to use semantic analysis. With semantic analysis marketers can better understand the conversations about their brand or product category– here is a white paper that compares Semantic vs Sentiment Analysis and can help you make a more informed decision about when and how to use Sentiment.

For years, sentiment has been a widely used measure of how customers view a company’s products and services. But sentiment analysis has inherent flaws. First is what it cannot tell you because it only considers a small amount of the available data. Only about 25 percent of posts actually contain sentiment, either positive or negative, which means three out of four posts are neutral, revealing no sentiment, and are effectively being ignored by the analysis. Thus, decisions are being based on what only a quarter of the posts are saying.

Another problem with sentiment is statistical confidence in the data. Simply stated, all methods of sentiment analysis rely on example data that, whittled down, reveals a low level of confidence about the sentiment being identified, either positive or negative. Data with such low confidence is a poor foundation for sentiment analysis.

Bottom line — sentiment analysis is not inherently bad. In fact, for particular types of questions, it may be the right approach. But if you use it, you need to be sure that it’s the right tool for the job, that valuable data is not being ignored, and that the method of sentiment analysis is built on sound data.

A much more statistically reliable approach is semantic analysis — a way to distill and create structure around mountains of unstructured data, such as blog posts, social network chatter, tweets and more, without preconceived ideas of whether or how they are related.

Semantic analysis allows you to cluster different data elements based on similarity, rather than preset classifications such as positive, negative and neutral. This helps you uncover important information like what exactly people are saying about your product or service; where and how they use it; and enhancements or new offerings they’re interested in. This type of valuable information can drive product development, new revenue streams and strategies for marketing, advertising and media planning.

Social media information is expanding at such an amazing pace, organizations need new and innovative ways to capture “nuggets” of intelligence it can reveal. Semantic analysis can help harness and make sense of all that information — in exponentially more powerful ways than sentiment analysis. In fact, a valuable type of semantic analysis is topic discovery: the summarization of large amounts of text by automatically discovering the topics and themes within. Networked Insights’ new Topic Discovery Engine (TDE) is a semantic analysis system finely tuned to discover topics in social media posts.

Most exciting, with semantic analysis you have tremendous latitude about how you approach the analysis. You don’t have to be certain about what you’re looking for. How many times in your life can you say that lack of certainty gives you a leg up? With semantic analysis, you can let the social media speak for itself, revealing to you amazingly accurate and important information that can inform critical decisions.

Click here to read our full report on Semantic vs Sentiment Analysis.

This is a great debate and is definitely worth checking out, along with Nathan Gilliatt’s response. I would like to add a few points that may have been under emphasized. In particular, I would like to address misconceptions about the volume of data we’re talking about, the use of statistics, validity vs. power, and human bias.

Data Volume

The main issue around data volume was covered in my last blog post, “Does monitoring provide the confidence and omniscience you need? One of the main points of that article is that there is simply too much data out there for humans to analyze effectively, in many cases.

As a concrete example (which I used in the last article), a search of only Google Groups for “Macintosh OR Mac -cheese -Fleetwood” returns over 17,000,000 posts over the last three months (more than twice as many as I found while writing my last article). That averages out to over 180,000 posts a day for one brand, from only the sources searched by Google! (While the number of sites covered by Google is impressive, it is not exhaustive.)

The point is, to perform analytics around one topic or brand (such as Macintosh), let alone multiple, means analyzing staggeringly large volumes of data! And, it is only increasing with time. It is important to keep the magnitude of data in mind as we move forward.

Statistics

The response to the above point is often something similar to Mike Daniels response: “But there’s an unstated assumption behind the technology promise: that it is necessary to analyse all or a very large percentage of these conversations in case we miss something”[sic].

Do we need to analyze a large percentage of posts? Yes and no. There are really two use cases that we are talking about.

1. Finding the PR or marketing emergencies that require immediate action
2. Understanding trends

For the first use case, finding emergencies, we really do need to analyze a large percentage of posts, and this is why the volume of data is relevant (This point was also covered by my last blog post). For example, let’s say you can only analyze 10,000 posts per day because we are using humans, and if we are receiving 180,000 posts per day, that gives us less than a 6% chance of finding an emergency post when it happens. How much are you willing to pay for a 6% chance?

For the second use case, understanding trends, it is true that we do not need to analyze a large percentage of posts. We have well founded statistical methods of sampling a subset of posts, analyzing them, and then relating our results back to the whole in a valid way.

However, saying that we don’t have to measure all posts is an abuse of statistics. Statistics is not meant as a shortcut to avoid processing more data points. Instead, it is a way to still derive some value when you CANNOT process all the data points (for whatever reason). Statistics is a fall back, it is a safety net. As a general rule we should process as many data points (posts) as we can and only use statistics when we are unable to process all of the points (due to constraints of time, money, feasibility, etc.)

Validity vs. Power

This discussion of statistics feeds directly into my next point: analytic validity and power. This is really the heart of the matter. While in some cases we may not need to process a large percentage of posts (as I discussed above), we do want to process as many posts as possible.

In analytics, we talk about “validity” and “power”. Statistics provides methods and rules for finding valid results when you can not process all the data points. Analytic power comes from processing more data points. “Power” in this sense is the ability to detect a difference, trend, etc. when it is exists. So, yes, we don’t need to process a large percentage of the posts to be valid. But, with more posts comes more analytic power and, hence, more value. Thus, there is much to be gained from taking advantage of machines to perform analytics.

What about humans, then?

Human Bias

Even if we could use humans to analyze all of the post we have, we still may not want to. Computers have the ability to put new data points in context with all of the other data under consideration. Humans can’t do that (especially when there are 180,000 other data points); we put things in context with all of the other things we know, which is very different. This is where bias can creep in.

There is a natural flow in analytics from data, to information, to knowledge, to decisions. We begin with data, organize it, analyze it, and put it into context with the other data to produce information. Then, a human consumes that information and combines it with what she knows to produce new knowledge. This knowledge can then drive decisions.

Regardless of whether your analytics are performed by human, machine, or other, this is the general flow; there is human knowledge used at some point. But, it is important to insert this knowledge into the flow at the right time. Human knowledge, in the form of expectations, inserted too early into in the the process I’ve described can drastically bias the results.

To sum up, I think Mike Daniels and Mark Westaby both bring up some great points, as do many of the people commenting on their post. At the end of the day, both humans and machines are needed, one cannot proceed along the data, information, knowledge, decision chain without them. The trick is to use the right tool for the right job. Machines are great at processing large amounts of data, putting it into context, and producing information while people are unequaled in our ability to create knowledge and use it to drive decisions in situations where the answer cannot be reduced to a one or a zero.

Until recently, the best solution for many of us has been something like Google Alerts. You enter your search terms (“Macintosh OR Mac -cheese -Fleetwood” and so on) and you get tens, hundreds, or even thousands of items a day; you know everything.

Or do you?  How capable are you, as a human being, at finding the most important information in a sea of data? Let’s take a shot at figuring that out.

On the plus side, people are very sophisticated text processors. We are highly skilled at reading a piece of text, understanding its meaning, and placing it in context with other information about the brand we are monitoring. We are very good at knowing what’s important.

But how do those skills scale at the volume we’re dealing with on the web? The problem isn’t “how skilled are you?” but “how much can you read?”.

Let’s say that you need to monitor the Macintosh brand. A quick search of Google Groups for “Macintosh OR Mac -cheese -Fleetwood” returns about 5,750,000 posts over the last 3 months; that’s about 64,000 posts per day. So, if you read 10 posts an hour, for 16 hours a day, for three months straight, you’d cover less than 0.3% of the posts about Macintosh computers!

Even with your tremendous ability to identify important content, you would be missing up to 99.7% of the posts concerning your brand. And, don’t forget, your time has been completely monopolized by one brand, so you are completely ignoring 100% of the rest of your brands (not to mention your family, social life, and general hygiene).

So, if you truly want confidence and omniscience, you do not want a service that gives you a bunch of posts to read, you do not want a monitoring solution. What you need is a system capable of processing all of your posts and finding the important information for you. This would free up your time to perform the important in-depth analysis for which there is only one tool: your human brain!