Search is the most personal touchpoint in the online buying process. It’s the place where your customer is telling you what they want. If the experience is a bad one – you will not make the sale and you may never see that customer again. And yet, most online retailers are still relying on legacy technologies like keyword matching and behavioral data to power their search. As a result, the relevance of search results is not as good as it should be because search engines are not able to make use of all of the information provided by the customer. But, conversion rates through site search can be 2-3 times higher than the average. Companies that fail to invest in best-in-class search are leaving money on the table.
The Problem with Keyword Matching.
There are real limitations with keyword search, and companies are feeling the frustration users have been experiencing for years. The source of frustration stems from the fact that keyword search relies on text matching rather than understanding the text’s actual meaning.
Keyword search deliberately ignores functional words such as “and,” “without,” “with,” and “not” – which are often crucial for understanding the user’s intent. A search engine that cannot distinguish between a customer looking for a “red dress with sleeves” and “red dress without sleeves” is a search engine significantly lacking in “meaning”.
Keyword searches also have a tough time distinguishing between words that are spelled the same way but mean something different (i.e. “red dress” and “red dress shirt”). This often leads to results that are completely irrelevant to your customer’s query.
And, sometimes, real knowledge is required to match a user query to the products on sale. For example, a customer might search for “best dress for summer wedding at the beach”. This information is not explicitly specified in the product description, so it must be inferred from what we know about weddings (i.e. usually a formal event), summer (i.e. that is it hot) and the beach (there will be sand and wind). Keyword matching cannot support searches that require this kind of inferential knowledge.
Customers want answers instantly by way of asking naturally, not by using a series of guesses to determine what combination of words will magically cause a search engine to return relevant results. This is why your customers are so frustrated.
How Natural Language Search Solves the Problem of Keyword Matching.
Natural language is what we use everyday to communicate with each other. Unlike keyword matching, natural language search focuses on real meaning and the natural way people phrase themselves when looking to buy or sell products. Natural language search is concept-based, which means it returns search results that are “about” the product your customer is looking for, even if the words in the product description are different from the words used in the query.
By letting users express themselves in their own words, as they would when speaking to a store clerk, you have the foundation to create a customer journey that delivers inspiration, service and a best-in-class experience – across all touchpoints.
If you believe what you read, everyone today is doing NLP. But most vendors, and even most internal engineering teams, are only using “shallow statistical” NLP techniques to automatically learn correlations between queries and relevant products, or between pairs of matching products, based on collecting and annotating many examples. Shallow statistical NLP will improve your overall relevance score, but there are a number of limitations with this approach:
- It is not possible to provide a representative data set for the “long tail” of more rare phenomena and longer queries.
- It is difficult to force machine learning models and algorithms to learn what we want them to learn.
As with keyword matching, precise understanding of complex expressions is difficult, if not impossible, to achieve using the shallow modeling common to statistical NLP techniques.
Twiggle, on the other hand, uses its proprietary Universal Product Ontology and sophisticated NLP techniques to drive human-like understanding of customer queries and products.
Twiggle is the first and only company to build a Universal Product Ontology specifically designed for use in product search. Our ontology reaches a level of granularity not modelled by existing taxonomies and drives the semantic understanding of both your product data and user queries.
The ontology fundamentally differs from existing taxonomies in a number of significant ways:
- Universal: Our ontology is a universal and “customer-independent” model of each product domain.
- Independent of Browsing: Most taxonomies are specifically designed for user browsing and are not always suitable for more accurate analysis of information in product descriptions and user queries
- Deep: Our ontology reaches a level of detail that is significantly higher than existing taxonomies. This manifests in the high level of granularity of concepts, attributes, possible attribute values, constraints, and relations between entities in the ontology.
- Complex: Our ontological infrastructure supports unique capabilities that do not exist in common ontologies, and were designed specifically for phenomena encountered in the products world. For example, our ontology supports:
- Hierarchical values (colors, materials, finish).
- Taxonomy concepts
- Qualifiers defined in terms of soft properties
- Probabilities of synonyms of model items
- Value distributions