Appen is so fast. Using their platform, we could do overnight what used to take us a month. Appen is wonderfully efficient.
– Rick Britt, Vice President of AI, CallMiner
The Company
Founded in 2002, CallMiner is the pioneer of the artificial intelligence (AI)-powered speech analytics space. In the years since, the company has continued to advance its AI software to provide organizations with invaluable insights into all customer interactions, answering questions like “Why do my customers call customer service?” and “How do my customers feel about my product?” Using conversational data analysis (in the form of customer service calls, email, chat, and other types of media), CallMiner distills massive amounts of data into key customer insights, understanding, and metrics across sentiment, emotions, and more.
The Challenge
The majority of the data CallMiner works with is audio from customer service calls, which is then processed through a speech recognizer. The CallMiner Research Lab is tasked with training AI models to understand those conversations, including sentiment and emotions, and other relevant insights between organizations and their customers. In one of their largest projects to date, the CallMiner Research Lab worked on accurately identifying and analyzing the sentiment in customer interactions, negative, positive, and neutral.
There were several challenges in this initiative. Sentiment analysis, in particular, presented a hurdle. For example, “That’s just great!” can be classified positively, but it’s negative when said sarcastically. Understanding the nuances in customer interactions was difficult for machines to learn. In many cases, the calls were mostly neutral, which could bias the machine learning model toward thinking all of the calls were neutral, which wasn’t accurate.
To handle these potential issues, the team needed a large dataset across an array of organizations to parse out the truly negative moments. Annotating training data with high accuracy was critical for AI models. It became obvious the effort was greater than the resources that were available, and the project needed outside help.
The Solution
I really appreciate Appen’s flexibility and the amount of control that I have over my projects. I want to go in and make the page look exactly how I want it to look. And I want to present the questions exactly as I envisioned. I have that amount of fine-grained control with Appen’s platform.
– Micaela Kaplan, Data Scientist and Ethics in AI Lead, CallMiner
CallMiner sought a partner who could help scale its annotation efforts. Our annotation platform at Appen not only fit CallMiner’s needs, but also our commitment to responsible AI practices aligned with CallMiner’s values. Since the start of the partnership, CallMiner has been using our platform to annotate sentiment and emotion of call center data. Notably, we already had security-compliant annotators ready and able to tackle the massive data annotation effort required, which freed up CallMiner’s research team, allowing them to focus on tasks more relevant to their roles. Our high levels of security and compliance enabled efficient and safe handling of data.
With us as the data partner, CallMiner now has a large-scale annotation solution. The research team no longer has to do manual calculations in Excel; instead, they use our reporting features for easy access to key performance indicators.
The Result
Before we settled on Appen, I wondered how I could know what my agreement metrics were—that had been a huge part of the workload when we were doing it ourselves. The fact that Appen has it all right there, with a project as subjective as emotion, I can be really confident in what we’re getting back and have those numbers to back up the work that we’re doing.
– Micaela Kaplan, Data Scientist and Ethics in AI Lead, CallMiner
Before working with us, the CallMiner Research Lab could only parse through 3,000 or so data samples in a few months. Now, they’ve analyzed tens of thousands of data samples. Our platform lets them process more calls faster and with more accuracy, enabling them to expand their customer base and explore new types of conversation insights with the extra time saved.
By leveraging our annotators, CallMiner gained a greater diversity of perspectives, helping the research team to expand definitions in new ways to achieve better analysis. Our platform also supports CallMiner’s efforts in documenting the decision-making process behind its models, an essential step toward contributing to the explainability and overall responsible use of the model.
