Querying Integration

Overview

Integrating the Twiggle API in the Querying Process

In order to integrate the Twiggle API as part of your search engine querying process, use the /queries endpoint to produce semantic interpretations for your shoppers’ search queries and incorporate these interpretations into your search engine queries. Before starting the integration, Twiggle configures the /queries endpoint to translate its query interpretations to your search engine query language and return them in the form of Translated Queries (translated from Twiggle’s internal query language to your search engine query language). A Translated Query represents the user's intent as a query against Twiggle Feature Fields which were previously indexed in your search engine. Translated Queries should be integrated into your final search engine queries alongside any other search signal you find useful.

Interpret Search Queries

Set the Engine Version

Whenever you send a GET /queries request, you need to declare which Engine version you are using. In order to assure consistency between listings' Feature Fields and Translated Queries, it's necessary to always use the Engine version that was used to produce the Features Objects stored in the catalog index you use in production.

Note that during a full catalog indexing cycle, POST /listings requests may be using a newly released Engine version, which is different than the version used when creating your production index. In this case, you should continue sending GET /queries requests using the Engine version which matches your production index. Only after successfully completing an indexing cycle with the new Engine version and pushing the newly-built index into production, should you switch to the latest Engine version in GET /queries requests.

Send Search Queries, Get Interpretations

Whenever a user performs a search, you should pass the query forward to the Twiggle API through the /queries endpoint. The Twiggle API will interpret the search query to produce a structured representation of the user's intent, and translate the interpretation to your search engine query language to produce a Translated Query. A Translated Query represents the user's specifications as search engine query instructions which apply to Twiggle Feature Fields previously added to you catalog index.

To interpret a search query, submit an HTTP GET request to the /queries endpoint along with the following query parameters:

Parameter Name	Required?	Data Type	Description
q	Required	String	The user's raw textual query.
engine_version	Required	String	The Engine version that was used to create the Features Objects stored in your index in production.
corrected_q	Optional	String	A version of the search query that underwent spelling correction. This helps Twiggle's spell corrector yield more precise corrections.

For example, an interpretation request for the query "red dress without sleeves" would look like this:

GET https://<your_partner_id>.api.twiggle.com/v1.6/queries?q=red+dress+without+sleeves&engine_version=201711011216

A successful request will yield a response with status code 200 and a body of content type application/json with the following fields:

Field Name	Data Type	Description
`_id`	String	The response's unique identifier.
`engine_version`	String	The Engine version with which the call was performed.
`translated_query`	Object	Twiggle’s query interpretation, translated to your search engine query language. This query represents the user's search specifications as search engine instructions which apply to Twiggle Feature Fields. It contains a single field (`q` for Solr, `query` for Elasticsearch) which holds the search engine query. This query should be incorporated into your main search engine queries alongside your additional search signals.
`strict_translated_query`	Object	Twiggle’s strict query interpretation, translated to your search engine query language. This query represents the user's search specifications as strict search engine instructions which apply to Twiggle Feature Fields. It contains a single field (q for Solr, query for Elasticsearch) which holds the search engine query (please see “Fallbacks” section below).
`query_text`	String	The textual search query that was interpreted, potentially after spelling correction.
`original_query_text`	String	The original textual search query that was sent for interpretation – before spelling correction.
`is_query_corrected`	Boolean	Returns `true` if the search query was modified by Twiggle's spelling corrector, `false` if it was left unchanged.
`search_text`	Array[String]	A list of substrings from the search query which are recommended for free-text keyword matching.
`test_variant`	String	The A/B testing bucket of the query (optional).

Example response body:

ElasticSearch Solr

{
  "_id": "5a9efea1ea3911e7bb5842010a810606",
  "engine_version": "201712111017",
  "is_query_corrected": false,
  "original_query_text": "red dress without sleeves",
  "query_text": "red dress without sleeves",
  "translated_query": {
    "query": {
      "bool": {
        "should": [
          {
            "bool": {
              "must": [
                {
                  "term": {
                    "$concept_sm": {
                      "value": "Dress"
                    }
                  }
                }
              ],
              "should": [
                {
                  "term": {
                    "$color_sm": {
                      "value": "Red"
                    }
                  }
                },
                {
                  "term": {
                    "sleeving_o.$concept_sm": {
                      "value": "Sleeveless"
                    }
                  }
                }
              ]
            }
          }
        ]
      }
    }
  },
  "search_text": [
    "red dress"
  ],
  "test_variant": "A"
}

{
  "_id": "5a9efea1ea3911e7bb5842010a810606",
  "engine_version": "201712111017",
  "is_query_corrected": false,
  "original_query_text": "red dress without sleeves",
  "query_text": "red dress without sleeves",
  "translated_query": {
    "q": "+$concept_twg_sm:Dress apparel_sleeves_twg_bm:false $color_twg_sm:Red"
  },
  "search_text": [ "red dress" ],
  "test_variant": "A"
}

Fallbacks

A common way to improve search quality is to favor precision over recall when the result set is large enough and to favor recall over precision when the search query does not retrieve enough results. This approach can be implemented using the search fallbacks pattern: Try using a "strict" precision-driven search query first. If it yields more results than some predefined threshold, return these results. Otherwise, use a "relaxed" recall-driven search query.

The Twiggle API can be configured to support the fallbacks pattern. When fallbacks are enabled, targeting the /queries endpoint will yield a response which includes two interpretations — a strict interpretation and a relaxed interpretation:

The strict interpretation is constructed to favor precision over recall by boosting only products which match all of the interpreted specification in the shopper's query, namely the specified product type as well as all specified attributes. For example, the strict interpretation for the query "red dress with a floral pattern" will only boost dresses which are red and have a floral pattern. Strict interpretations are held by the field strict_translated_query.
The relaxed interpretation favors recall over precision by boosting any product that matches the specified product type and assigning an additional boost for every matched attribute. For example, the relaxed interpretation for the query "red dress with floral pattern" will boost all dresses and will further boost red colored dresses which have floral patterns. Relaxed interpretations are held by the field translated_query.

Here's an example for how a response for the query "red dress with floral pattern" would look when fallbacks are enabled:

ElasticSearch Solr

{
  "_id": "abddbfb90e774d929e95f37d5dcfc36b",
  "query_text": "red dress with floral pattern",
  "original_query_text": "red dress with floral pattern",
  "is_query_corrected": false,
  "engine_version": "201807050000",
  "strict_translated_query": {
    "query": {
      "bool": {
        "must": [
          {
            "term": {
              "features.$concept_sm": "Dress"
            }
          },
          {
            "term": {
              "features.$color_sm": "Red"
            }
          },
          {
            "term": {
              "features.design_patterns_om.$concept_sm": "Floral"
            }
          }
        ]
      }
    }
  },
  "translated_query": {
    "query": {
      "bool": {
        "must": {
          "term": {
            "features.$concept_sm": "Dress"
          }
        },
        "should": [
          {
            "term": {
              "features.$color_sm": "Red"
            }
          },
          {
            "term": {
              "features.design_patterns_om.$concept_sm": "Floral"
            }
          }
        ]
      }
    }
  },
  "search_text": [
    "red dress with floral pattern"
  ]
}

{
  "_id": "e207a96a188642f4b7f4083f263483d0",
  "query_text": "red dress with floral pattern",
  "original_query_text": "red dress with floral pattern",
  "is_query_corrected": false,
  "engine_version": "201806270000",
  "strict_translated_query": {
    "q": "(+twiggle_features.$concept_twg_sm:Dress +twiggle_features.$color_twg_sm:Red +(+twiggle_features.design_patterns_om.$concept_twg_sm:Floral))"
  },
  "translated_query": {
    "q": "(+twiggle_features.$concept_twg_sm:Dress twiggle_features.$color_twg_sm:Red (+twiggle_features.design_patterns_om.$concept_twg_sm:Floral))"
  },
  "search_text": [
    "red dress with floral pattern"
  ]
}

External Features

External Features are search document fields that may be referred to in query interpretations, but are not included in Features objects. Instead, it is assumed that these fields are represented in your original listing data.

Currently, Twiggle supports two external features:

A numeric external feature for the listing price.
A boolean external feature if the listing is on sale.

Price

Twiggle assumes that you already maintain an up-to-date price for each of the listings indexed in your search engine. Since prices are highly dynamic by nature, using your existing price field within query interpretations would yield more accurate results than using a price Feature Field by Twiggle.

In order to utilize users' price-related search specifications in query interpretations, you need to inform Twiggle of the name of your original price field (if you have multiple price fields, choose the field which best matches a typical price-related search intent). This will enable Twiggle to map its price External Feature to your price field, and set the relevant parts in query interpretations to apply to your price field.

For example, let's assume that your price field is named price, and consider the query "table under $200". An interpretation request for this query would yield a Translated Query which includes the user's price-related intent by boosting listings whose price field value is less than $200:

ElasticSearch Solr

{
  "bool": {
    "must": {
      "term": {
        "twiggle.$concept_sm": {
          "value": "Table"
        }
      }
    },
    "should": {
      "range": {
        "price": {
          "to": 200
        }
      }
    }
  }
}

($concept_twg_sm:Table)^3.0 +(+Table_om.$concept_twg_sm:Table)^2.0 price:[* TO 200.0]

On Sale (Discounted Items)

Similarly to Price, Twiggle can utilize users' discount-related search specifications. The listings should have a field which marks whether the listing is currently on sale.

This can be achieved in the following ways:

A boolean field which indicates whether an item is on sale or not.
A numerical field which indicates what is the discount percentage (For example: The field value is 10 if the item is 10% off).

For example: Let’s assume that there exists a field ‘discount’ that marks the discount percentage of listings. Consider the query “ table on sale”. The interpretation response for this query will yield a Translated Query which includes the user’s discount related intent by boosting listings who have a discount percentage equal or greater than 1.

ElasticSearch Solr

{
  "query": {
    "bool": {
      "should": [
        {
          "bool": {
            "must": [
              {
                "constant_score": {
                  "filter": {
                    "term": {
                      "features.$concept_sm": {
                        "value": "Table",
                        "boost": 10
                      }
                    }
                  }
                }
              }
            ],
            "should": [
              {
                "constant_score": {
                  "filter": {
                    "range": {
                      "discount": {
                        "from": 1,
                        "include_lower": true,
                        "include_upper": true
                      }
                    }
                  },
                  "boost": 10
                }
              }
            ],
            "disable_coord": true
          }
        }
      ]
    }
  }
}

+((((twiggle_features.$concept_twg_sm:Table)^=10)^3 +((twiggle_features.Table_om.$concept_twg_sm:Table)^=10)^2 (discount:[1.0 TO Infinity])^10.0))

Error Handling

API Errors

GET /queries requests can result in the following errors:

Error Type	Status Code	Description
`Bad Request`	`400`	Missing or invalid query parameters.
`Unauthorized Request`	`401`	Missing or incorrect authentication credentials.
`Internal Server Error`	`500`	Server error on the Semantic API end.

Whenever one of these errors occurs, you should revert to using an existing search engine query without the Twiggle API. These errors require offline inspection, and should not result in an automatic attempt to resend the request.

It is recommended that you preconfigure a timeout limit to GET /queries requests. In case of a timeout, you should revert back to your original search logic.

Integrating the Twiggle Search Signal

Twiggle’s query interpretations are designed to be added as a search signal which complements your current search signals—not replace them. There are a few ways in which Twiggle’s query interpretations can be used to enhance your existing search logic. As part of the integration process, Twiggle’s team collaborates with your team to identify core business and technical needs as well as any constraints to design a logical flow that combines Twiggle’s query interpretations with your existing search signals.

The below is an example of a simplistic signal combination strategy wherein Twiggle’s Translated Queries are combined with a preexisting simple free-text query. This is just one example of how Twiggle’s Search Signal can be used in addition to other search signals to enhance the relevance of search results.

ElasticSearch Solr

The example above merely demonstrates the concept of search signal combination. Real-world search and navigation systems are typically more complex and require a preliminary integration design phase before integrating Twiggle's query interpretations. As part of this process, our technical teams work intimately with your team to build a search signal combination strategy that caters to your specific business and technical requirements.

Twiggle’s solution can scale as your business grows and the Twiggle API and signal combination strategy can always be improved to support advanced conditions. Just contact the team and we’ll be happy to make recommendations and provide guidance on any search topic.