Bidirectional Encoder Representations from Transformers (BERT)

Bidirectional Encoder Representations from Transformers (BERT)

Understanding searches better than ever before

Bidirectional Encoder Representations from Transformers (BERT) Google’s innovative algorithm for bots to understand natural language

October 25, 2019, Google announced its new and biggest search algorithm change since Rank Brain, it started rolling out last week of October 2019 and initially it is aimed for the English language only, it's said it would expand to other languages in the near future. According to Google, it will impact featured snippets.

BERT is Google’s neural network-based technique for natural language processing (NLP), more precisely BERT is able to help search engines to understand the language a bit more like humans do.

Google said BERT helps better understand the nuances and context of words in searches and better match those queries with more relevant results. It is also used for featured snippets, as described above. This technology enables anyone to train their own state-of-the-art question answering system.

For featured snippets, we’re using a BERT model to improve featured snippets in the two dozen countries where this feature is available, and seeing significant improvements in languages like Korean, Hindi, and Portuguese.

Transformers are models that process words in relation to all the other words in a sentence, rather than one-by-one in order. BERT models can therefore consider the full context of a word by looking at the words that come before and after it—particularly useful for understanding the intent behind search queries.

In one example, Google said, with a search for “2019 brazil traveler to the USA need a visa,” the word “to” and its relationship to the other words in query are important for understanding the meaning. Previously, Google wouldn’t understand the importance of this connection and would return results about U.S. citizens traveling to Brazil. “With BERT, Search is able to grasp this nuance and know that the very common word “to” actually matters a lot here, and we can provide a much more relevant result for this query,” Google explained.

Search is about understanding language. It’s our job to figure out what you’re searching for and surface helpful information from the web, no matter how you spell or combine the words in your query. While we’ve continued to improve our language understanding capabilities over the years, we sometimes still don’t quite get it right, particularly with complex or conversational queries. In fact, that’s one of the reasons why people often use “keyword-ese,” typing strings of words that they think we’ll understand, but aren’t actually how they’d naturally ask a question.

Let’s look at another query: “do estheticians stand a lot at work.” Previously, our systems were taking an approach of matching keywords, matching the term “stand-alone” in the result with the word “stand” in the query. But that isn’t the right use of the word “stand” in context. Our BERT models, on the other hand, understand that “stand” is related to the concept of the physical demands of a job, and displays a more useful response.




BERT will impact around 10% of queries. It will also impact organic rankings and featured snippets. So this is no small change!

Lots of the major AI companies are also building BERT versions:

  • Microsoft extends on BERT with MT-DNN (Multi-Task Deep Neural Network).
  • RoBERTa from Facebook.
  • SuperGLUE Benchmark was created because the original GLUE Benchmark became too easy.

NLP is the technology behind such popular language applications as:

  • Google Translate
  • Microsoft Word
  • Grammarly
  • OK Google, Siri, Cortana, and Alexa