What does "masking" refer to in the context of transformers?

In the context of transformers, "masking" specifically refers to the technique of hiding future tokens during the attention calculation in sequence models, particularly in autoregressive tasks. This ensures that the model does not have access to future information when predicting the current token. For example, during the training of a language model, when predicting the next word, it is crucial that the model only uses the words that have been seen up to that point in the sequence to prevent any information leakage that could skew the learning process. By masking future tokens, the transformer can maintain the correct causal structure where it generates text one token at a time, from left to right.

This approach is vital for tasks such as language modeling where the sequence is inherently directional, and the model needs to learn to predict the next token based solely on the previous context. Thus, the correct understanding of masking in transformers not only aids in ensuring the integrity of the prediction process but also aligns with the key architectural principles governing models built using this technology.