Source

Keyword Search

• Goal → Use keyword search to answer questions using a database.
• Keyword search is the method most commonly used to build search systems.
• Let’s say you want to search the following query from an archive shown below:

• A common way to do keyword search is to count the number of common words between the query and the documents.

Code

• In the example below, we connect to a DB and do some keyword search on it:

  • Connect to Weaviate DB:

    • Weaviate → An open source database, it has keyword search capabilities, but also vector search capabilities that rely on language models.
    • The demo database (https://cohere-demo.weaviate.network/) contains 10 million records from Wikipedia.
      • 1M is in English and the other 9M in other languages.

    import weaviate
    auth_config = weaviate.auth.AuthApiKey(
        api_key=os.environ['WEAVIATE_API_KEY'])
    
    #WEAVIATE_API_KEY="76320a90-53d8-42bc-b41d-678647c6672e"
    
    client = weaviate.Client(
        url=os.environ['WEAVIATE_API_URL'],  # the public DB for this course: <https://cohere-demo.weaviate.network/>
        auth_client_secret=auth_config,
        additional_headers={
            "X-Cohere-Api-Key": os.environ['COHERE_API_KEY'],
        }
    )
    
    client.is_ready() 
    

    • Create the Keyword Search Function
      • bm25 → it’s a keyword (lexical) search algorithm commonly used, and it scores the documents in the DB against the query based on specific formula that looks at the count of the shared words between the query and each document. Read more

    def keyword_search(query,
                       results_lang='en',
                       properties = ["title","url","text"],
                       num_results=3):
    
        where_filter = {
        "path": ["lang"],
        "operator": "Equal",
        "valueString": results_lang
        }
        
        # Calling Weaviate API
        response = (
            client.query.get("Articles", properties)
            .with_bm25(
                query=query
            )
            .with_where(where_filter)
            .with_limit(num_results)
            .do()
            )
    
        result = response['data']['Get']['Articles']
        return result
        
    query = "What is the most viewed televised event?"
    keyword_search_results = keyword_search(query)
    print(keyword_search_results)
    

Keyword Search Internals

• We can think of search systems in two stages:
  1. Retrieval stage
  2. Reranking stage → in case we want to involve additional criteria other than text relevance.

Retrieval Stage - Inverted Index

• This stage commonly uses BM25 algorithm to score the documents against the query.
• The implementation often contains using this idea of an inverted index:
  • The inverted index contains keywords and the document IDs of documents that the keyword is present in.
  • This is done to optimize the speed of the search.
  • In practice, in addition to document ID, the frequency of how many times the keyword appears is also added.