BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding

Jacob Devlin Ming-Wei Chang Kenton Lee Kristina Toutanova

Google AI Language {jacobdevlin,mingweichang,kentonl,kristout}@google.com

Abstract There are two existing strategies for apply-ing pre-trained language representations to down-We introduce a new language representa-stream tasks: feature-based and ﬁne-tuning. The tion model called BERT, which stands for Bidirectional Encoder Representations from feature-based approach, such as ELMo (Peters

Transformers. Unlike recent language repre-et al., 2018a), uses task-speciﬁc architectures that sentation models (Peters et al., 2018a; Rad- include the pre-trained representations as addiford et al., 2018), BERT is designed to pre- tional features. The ﬁne-tuning approach, such as train deep bidirectional representations from the Generative Pre-trained Transformer (OpenAI unlabeled text by jointly conditioning on both GPT) (Radford et al., 2018), introduces minimal left and right context in all layers. As a re-task-speciﬁc parameters, and is trained on the sult, the pre-trained BERT model can be ﬁne-downstream tasks by simply ﬁne-tuning all pretuned with just one additional output layer to create state-of-the-art models for a wide trained parameters. The two approaches share the range of tasks, such as question answering and same objective function during pre-training, where language inference, without substantial task- they use unidirectional language models to learn speciﬁc architecture modiﬁcations. general language representations.

BERT is conceptually simple and empirically We argue that current techniques restrict the powerful. It obtains new state-of-the-art re- power of the pre-trained representations, espesults on eleven natural language processing cially for the ﬁne-tuning approaches. The matasks, including pushing the GLUE score to jor limitation is that standard language models are 80.5% (7.7% point absolute improvement), unidirectional, and this limits the choice of archi-MultiNLI accuracy to 86.7% (4.6% absolute improvement), SQuAD v1.1 question answer-tectures that can be used during pre-training. For ing Test F1 to 93.2 (1.5 point absolute im-example, in OpenAI GPT, the authors use a left-toprovement) and SQuAD v2.0 Test F1 to 83.1 right architecture, where every token can only at-(5.1 point absolute improvement). tend to previous tokens in the self-attention layers of the Transformer (Vaswani et al., 2017). Such re-

1 Introduction

strictions are sub-optimal for sentence-level tasks,

Language model pre-training has been shown to and could be very harmful when applying ﬁnebe effective for improving many natural language tuning based approaches to token-level tasks such processing tasks (Dai and Le, 2015; Peters et al., as question answering, where it is crucial to incor-2018a; Radford et al., 2018; Howard and Ruder, porate context from both directions. 2018). These include sentence-level tasks such as In this paper, we improve the ﬁne-tuning based natural language inference (Bowman et al., 2015; approaches by proposing BERT: Bidirectional Williams et al., 2018) and paraphrasing (Dolan Encoder Representations from Transformers. and Brockett, 2005), which aim to predict the re- BERT alleviates the previously mentioned unidilationships between sentences by analyzing them rectionality constraint by using a “masked lanholistically, as well as token-level tasks such as guage model” (MLM) pre-training objective, innamed entity recognition and question answering, spired by the Cloze task (Taylor, 1953). The where models are required to produce ﬁne-grained masked language model randomly masks some of output at the token level (Tjong Kim Sang and the tokens from the input, and the objective is to De Meulder, 2003; Rajpurkar et al., 2016). predict the original vocabulary id of the masked word based only on its context. Unlike left-to- These approaches have been generalized to right language model pre-training, the MLM ob- coarser granularities, such as sentence embedjective enables the representation to fuse the left dings (Kiros et al., 2015; Logeswaran and Lee, and the right context, which allows us to pre- 2018) or paragraph embeddings (Le and Mikolov, train a deep bidirectional Transformer. In addi- 2014). To train sentence representations, prior tion to the masked language model, we also use work has used objectives to rank candidate next a “next sentence prediction” task that jointly pre- sentences (Jernite et al., 2017; Logeswaran and trains text-pair representations. The contributions Lee, 2018), left-to-right generation of next senof our paper are as follows: tence words given a representation of the previous sentence (Kiros et al., 2015), or denoising auto-

• We demonstrate the importance of bidirectional encoder derived objectives (Hill et al., 2016). pre-training for language representations. Un-ELMo and its predecessor (Peters et al., 2017, like Radford et al. (2018), which uses unidirec-2018a) generalize traditional word embedding retional language models for pre-training, BERT search along a different dimension. They extract uses masked language models to enable pre-context-sensitive features from a left-to-right and a trained deep bidirectional representations. This right-to-left language model. The contextual repis also in contrast to Peters et al. (2018a), which resentation of each token is the concatenation of uses a shallow concatenation of independently the left-to-right and right-to-left representations. trained left-to-right and right-to-left LMs. When integrating contextual word embeddings

• We show that pre-trained representations reduce with existing task-speciﬁc architectures, ELMo the need for many heavily-engineered task- advances the state of the art for several major NLP speciﬁc architectures. BERT is the ﬁrst ﬁne- benchmarks (Peters et al., 2018a) including questuning based representation model that achieves tion answering (Rajpurkar et al., 2016), sentiment state-of-the-art performance on a large suite analysis (Socher et al., 2013), and named entity of sentence-level and token-level tasks, outper- recognition (Tjong Kim Sang and De Meulder, forming many task-speciﬁc architectures. 2003). Melamud et al. (2016) proposed learning contextual representations through a task to pre-

• BERT advances the state of the art for eleven dict a single word from both left and right context NLP tasks. The code and pre-trained mod-using LSTMs. Similar to ELMo, their model is els are available at https://github.com/ feature-based and not deeply bidirectional. Fedus google-research/bert. et al. (2018) shows that the cloze task can be used to improve the robustness of text generation mod-

els.

There is a long history of pre-training general language representations, and we brieﬂy review the

2.2 Unsupervised Fine-tuning Approaches

most widely-used approaches in this section.

As with the feature-based approaches, the ﬁrst

2.1 Unsupervised Feature-based Approaches

works in this direction only pre-trained word em-

Learning widely applicable representations of bedding parameters from unlabeled text (Colwords has been an active area of research for lobert and Weston, 2008). decades, including non-neural (Brown et al., 1992; More recently, sentence or document encoders Ando and Zhang, 2005; Blitzer et al., 2006) and which produce contextual token representations neural (Mikolov et al., 2013; Pennington et al., have been pre-trained from unlabeled text and 2014) methods. Pre-trained word embeddings ﬁne-tuned for a supervised downstream task (Dai are an integral part of modern NLP systems, of- and Le, 2015; Howard and Ruder, 2018; Radford fering signiﬁcant improvements over embeddings et al., 2018). The advantage of these approaches learned from scratch (Turian et al., 2010). To pre- is that few parameters need to be learned from train word embedding vectors, left-to-right lan- scratch. At least partly due to this advantage, guage modeling objectives have been used (Mnih OpenAI GPT (Radford et al., 2018) achieved preand Hinton, 2009), as well as objectives to dis- viously state-of-the-art results on many sentencecriminate correct from incorrect words in left and level tasks from the GLUE benchmark (Wang right context (Mikolov et al., 2013). et al., 2018a). Left-to-right language model-

Figure 1: Overall pre-training and ﬁne-tuning procedures for BERT. Apart from output layers, the same architectures are used in both pre-training and ﬁne-tuning. The same pre-trained model parameters are used to initialize models for different down-stream tasks. During ﬁne-tuning, all parameters are ﬁne-tuned. [CLS] is a special symbol added in front of every input example, and [SEP] is a special separator token (e.g. separating questions/answers).

ing and auto-encoder objectives have been used for pre-training such models (Howard and Ruder, 2018;Radford et al.,2018;Dai and Le,2015). 2.3 Transfer Learning from Supervised Data There has also been work showing effective trans- fer from supervised tasks with large datasets, such as natural language inference (Conneau et al., 2017) and machine translation (McCann et al., 2017). Computer vision research has also demon- strated the importance of transfer learning from large pre-trained models, where an effective recipe is to ﬁne-tune models pre-trained with Ima- geNet (Deng et al.,2009;Yosinski et al.,2014). 3 BERT We introduce BERT and its detailed implementa- tion in this section. There are two steps in our framework: pre-training and ﬁne-tuning. Dur- ing pre-training, the model is trained on unlabeled data over different pre-training tasks. For ﬁne- tuning, the BERT model is ﬁrst initialized with the pre-trained parameters, and all of the param- eters are ﬁne-tuned using labeled data from the downstream tasks. Each downstream task has sep- arate ﬁne-tuned models, even though they are ini- tialized with the same pre-trained parameters. The question-answering example in Figure1will serve as a running example for this section. A distinctive feature of BERT is its uniﬁed ar- chitecture across different tasks. There is mini-

mal difference between the pre-trained architec- ture and the ﬁnal downstream architecture. Model Architecture BERT’s model architec- ture is a multi-layer bidirectional Transformer en- coder based on the original implementation de- scribed inVaswani et al.(2017) and released in 1 the tensor2tensor library. Because the use of Transformers has become common and our im- plementation is almost identical to the original, we will omit an exhaustive background descrip- tion of the model architecture and refer readers to Vaswani et al.(2017) as well as excellent guides 2 such as “The Annotated Transformer.” In this work, we denote the number of layers (i.e., Transformer blocks) as L, the hidden size as 3 H, and the number of self-attention heads as A. We primarily report results on two model sizes: BERTBASE (L=12, H=768, A=12, Total Param- eters=110M) and BERTLARGE (L=24, H=1024, A=16, Total Parameters=340M). BERT was chosen to have the same model BASE size as OpenAI GPT for comparison purposes. Critically, however, the BERT Transformer uses bidirectional self-attention, while the GPT Trans- former uses constrained self-attention where every 4 token can only attend to context to its left. 1 https://github.com/tensorﬂow/tensor2tensor 2 http://nlp.seas.harvard.edu/2018/04/03/attention.html 3 In all cases we set the feed-forward/ﬁlter size to be 4H, i.e., 3072 for the H = 768 and 4096 for the H = 1024. 4We note that in the literature the bidirectional Trans-

Input/Output Representations To make BERT In order to train a deep bidirectional representahandle a variety of down-stream tasks, our input tion, we simply mask some percentage of the input representation is able to unambiguously represent tokens at random, and then predict those masked both a single sentence and a pair of sentences tokens. We refer to this procedure as a “masked (e.g., 〈 Question, Answer 〉) in one token sequence. LM” (MLM), although it is often referred to as a Throughout this work, a “sentence” can be an arbi- Cloze task in the literature (Taylor, 1953). In this trary span of contiguous text, rather than an actual case, the ﬁnal hidden vectors corresponding to the linguistic sentence. A “sequence” refers to the in- mask tokens are fed into an output softmax over put token sequence to BERT, which may be a sin- the vocabulary, as in a standard LM. In all of our gle sentence or two sentences packed together. experiments, we mask 15% of all WordPiece to-We use WordPiece embeddings (Wu et al., kens in each sequence at random. In contrast to 2016) with a 30,000 token vocabulary. The ﬁrst denoising auto-encoders (Vincent et al., 2008), we token of every sequence is always a special clas- only predict the masked words rather than reconsiﬁcation token ([CLS]). The ﬁnal hidden state structing the entire input. corresponding to this token is used as the ag- Although this allows us to obtain a bidirecgregate sequence representation for classiﬁcation tional pre-trained model, a downside is that we tasks. Sentence pairs are packed together into a are creating a mismatch between pre-training and single sequence. We differentiate the sentences in ﬁne-tuning, since the [MASK] token does not aptwo ways. First, we separate them with a special pear during ﬁne-tuning. To mitigate this, we do token ([SEP]). Second, we add a learned embed- not always replace “masked” words with the acding to every token indicating whether it belongs tual [MASK] token. The training data generator to sentence A or sentence B. As shown in Figure 1, chooses 15% of the token positions at random for we denote input embedding as E, the ﬁnal hidden prediction. If the i-th token is chosen, we replace vector of the special [CLS] token as C ∈ R^H, the i-th token with (1) the [MASK] token 80% of and the ﬁnal hidden vector for the i^t^h input token the time (2) a random token 10% of the time (3) as T_i ∈ R^H. the unchanged i-th token 10% of the time. Then, For a given token, its input representation is T_i will be used to predict the original token with constructed by summing the corresponding token, cross entropy loss. We compare variations of this segment, and position embeddings. A visualiza- procedure in Appendix C.2. tion of this construction can be seen in Figure 2. Task #2: Next Sentence Prediction (NSP)

3.1 Pre-training BERT Many important downstream tasks such as Ques-

Unlike Peters et al. (2018a) and Radford et al. tion Answering (QA) and Natural Language Infer-(2018), we do not use traditional left-to-right or ence (NLI) are based on understanding the relaright-to-left language models to pre-train BERT. tionship between two sentences, which is not di-Instead, we pre-train BERT using two unsuper-rectly captured by language modeling. In order vised tasks, described in this section. This step to train a model that understands sentence relais presented in the left part of Figure 1. tionships, we pre-train for a binarized next sentence prediction task that can be trivially gener-

Task #1: Masked LM Intuitively, it is reason- ated from any monolingual corpus. Speciﬁcally, able to believe that a deep bidirectional model is when choosing the sentences A and B for each prestrictly more powerful than either a left-to-right training example, 50% of the time B is the actual model or the shallow concatenation of a left-to- next sentence that follows A (labeled as), IsNext right and a right-to-left model. Unfortunately, and 50% of the time it is a random sentence from standard conditional language models can only be the corpus (labeled as). As we show NotNext trained left-to-right or right-to-left, since bidirec- in Figure 1, C is used for next sentence predictional conditioning would allow each word to in- tion (NSP).⁵ Despite its simplicity, we demondirectly “see itself”, and the model could trivially strate in Section 5.1 that pre-training towards this predict the target word in a multi-layered context. task is very beneﬁcial to both QA and NLI. ⁶ former is often referred to as a “Transformer encoder” while The ﬁnal model achieves 97%-98% accuracy on NSP.

the left-context-only version is referred to as a “Transformer The vector C is not a meaningful sentence representation decoder” since it can be used for text generation. without ﬁne-tuning, since it was trained with NSP.

Figure 2: BERT input representation. The input embeddings are the sum of the token embeddings, the segmentation embeddings and the position embeddings.

The NSP task is closely related to representation- (4) a degenerate text-∅ pair in text classiﬁcation learning objectives used in Jernite et al. (2017) and or sequence tagging. At the output, the token rep-

Logeswaran and Lee (2018). However, in prior resentations are fed into an output layer for tokenwork, only sentence embeddings are transferred to level tasks, such as sequence tagging or question down-stream tasks, where BERT transfers all pa- answering, and the [CLS] representation is fed rameters to initialize end-task model parameters. into an output layer for classiﬁcation, such as en-tailment or sentiment analysis.

Pre-training data The pre-training procedure Compared to pre-training, ﬁne-tuning is relalargely follows the existing literature on language tively inexpensive. All of the results in the pamodel pre-training. For the pre-training corpus we per can be replicated in at most 1 hour on a sinuse the BooksCorpus (800M words) (Zhu et al., gle Cloud TPU, or a few hours on a GPU, starting

2015) and English Wikipedia (2,500M words). from the exact same pre-trained model. We de-

For Wikipedia we extract only the text passages scribe the task-speciﬁc details in the correspondand ignore lists, tables, and headers. It is criti-ing subsections of Section 4. More details can be cal to use a document-level corpus rather than a found in Appendix A.5. shufﬂed sentence-level corpus such as the Billion

Word Benchmark (Chelba et al., 2013) in order to 4 Experiments extract long contiguous sequences.

In this section, we present BERT ﬁne-tuning re-

3.2 Fine-tuning BERT sults on 11 NLP tasks.

Fine-tuning is straightforward since the self- 4.1 GLUE attention mechanism in the Transformer al-The General Language Understanding Evaluation lows BERT to model many downstream tasks— (GLUE) benchmark (Wang et al., 2018a) is a colwhether they involve single text or text pairs—by lection of diverse natural language understanding swapping out the appropriate inputs and outputs. tasks. Detailed descriptions of GLUE datasets are

For applications involving text pairs, a common included in Appendix B.1. pattern is to independently encode text pairs be-To ﬁne-tune on GLUE, we represent the input fore applying bidirectional cross attention, such sequence (for single sentence or sentence pairs) as Parikh et al. (2016); Seo et al. (2017). BERT as described in Section 3, and use the ﬁnal hid-instead uses the self-attention mechanism to unify

den vector C ∈ RH corresponding to the ﬁrst

these two stages, as encoding a concatenated text input token ([CLS]) as the aggregate representa-pair with self-attention effectively includes bidi-tion. The only new parameters introduced during rectional cross attention between two sentences.

ﬁne-tuning are classiﬁcation layer weights W ∈

For each task, we simply plug in the task-

RK×HK

speciﬁc inputs and outputs into BERT and ﬁne-pute a standard classiﬁcation loss with C and W, tune all the parameters end-to-end. At the in-i.e., log(softmax(CW ^T)). put, sentence A and sentence B from pre-training are analogous to (1) sentence pairs in paraphras-For example, the BERT SQuAD model can be trained in around 30 minutes on a single Cloud TPU to achieve a Dev ing, (2) hypothesis-premise pairs in entailment, (3) F1 score of 91.0%. question-passage pairs in question answering, and ⁸See (10) in_h_t_p_s_:_/_g_l_u_e_b_e_n_c_h_m_a_r_k_._c_o_m_/_f_a_q.

Table 1: GLUE Test results, scored by the evaluation server (https://gluebenchmark.com/leaderboard).
System	MNLI-(m/mm)	QQP	QNLI	SST-2	CoLA	STS-B	MRPC	RTE	Average
	392k	363k	108k	67k	8.5k	5.7k	3.5k	2.5k	-
Pre-OpenAI SOTA	80.6/80.1	66.1	82.3	93.2	35.0	81.0	86.0	61.7	74.0
BiLSTM+ELMo+Attn	76.4/76.1	64.8	79.8	90.4	36.0	73.3	84.9	56.8	71.0
OpenAI GPT	82.1/81.4	70.3	87.4	91.3	45.4	80.0	82.3	56.0	75.1
BERT_B_A_S_E	84.6/83.4	71.2	90.5	93.5	52.1	85.8	88.9	66.4	79.6
BERTLARGE	86.7/85.9	72.1	92.7	94.9	60.5	86.5	89.3	70.1	82.1

The number below each task denotes the number of training examples. The “Average” column is slightly different than the ofﬁcial GLUE score, since we exclude the problematic WNLI set.⁸ BERT and OpenAI GPT are singlemodel, single task. F1 scores are reported for QQP and MRPC, Spearman correlations are reported for STS-B, and accuracy scores are reported for the other tasks. We exclude entries that use BERT as one of their components.

We use a batch size of 32 and ﬁne-tune for 3 Wikipedia containing the answer, the task is to epochs over the data for all GLUE tasks. For each predict the answer text span in the passage. task, we selected the best ﬁne-tuning learning rate As shown in Figure 1, in the question answer-(among 5e-5, 4e-5, 3e-5, and 2e-5) on the Dev set. ing task, we represent the input question and pas-

Additionally, for BERT we found that ﬁne- sage as a single packed sequence, with the ques-LARGE tuning was sometimes unstable on small datasets, tion using the A embedding and the passage using so we ran several random restarts and selected the the B embedding. We only introduce a start vecbest model on the Dev set. With random restarts, tor ∈ R^H ∈ R^H S and an end vector E during we use the same pre-trained checkpoint but per- ﬁne-tuning. The probability of word i being the form different ﬁne-tuning data shufﬂing and clas- start of the answer span is computed as a dot prodsiﬁer layer initialization. uct between T_i and S followed by a softmax over Results are presented in Table 1. Both all of the words in the paragraph: ∑eSj^·_S^Tⁱ_·_T_j.

P_i =

BERT and BERT outperform all sys- ^j^S^e^S^·^T^j BASE LARGE The analogous formula is used for the end of the tems on all tasks by a substantial margin, obtaining answer span. The score of a candidate span from

4.5% and 7.0% respective average accuracy im-

ijS·Ti+E·Tj,

provement over the prior state of the art. Note that

and the maximum scoring span where j ≥ i is

BERT and OpenAI GPT are nearly identical BASE used as a prediction. The training objective is the in terms of model architecture apart from the at-sum of the log-likelihoods of the correct start and tention masking. For the largest and most widely end positions. We ﬁne-tune for 3 epochs with a reported GLUE task, MNLI, BERT obtains a 4.6% learning rate of 5e-5 and a batch size of 32. absolute accuracy improvement. On the ofﬁcial

Table 2 shows top leaderboard entries as well GLUE leaderboard, BERT obtains a score LARGE as results from top published systems (Seo et al., of 80.5, compared to OpenAI GPT, which obtains

2017; Clark and Gardner, 2018; Peters et al., 72.8 as of the date of writing.

2018a; Hu et al., 2018). The top results from the

We ﬁnd that BERT signiﬁcantly outper-LARGE SQuAD leaderboard do not have up-to-date public forms BERT across all tasks, especially those BASE 11 system descriptions available, and are allowed to with very little training data. The effect of model use any public data when training their systems. size is explored more thoroughly in Section 5.2.

We therefore use modest data augmentation in our system by ﬁrst ﬁne-tuning on TriviaQA (Joshi

4.2 SQuAD v1.1

et al., 2017) befor ﬁne-tuning on SQuAD.

The Stanford Question Answering Dataset Our best performing system outperforms the top (SQuAD v1.1) is a collection of 100k crowd-leaderboard system by +1.5 F1 in ensembling and sourced question/answer pairs (Rajpurkar et al., +1.3 F1 as a single system. In fact, our single

2016). Given a question and a passage from BERT model outperforms the top ensemble sys-

9 tem in terms of F1 score. Without TriviaQA ﬁne-The GLUE data set distribution does not include the Test labels, and we only made a single GLUE evaluation server submission for each of BERT and BERT. ¹¹QANet is described in Yu et al. (2018), but the system BASE LARGE

Footnotes

10 https://gluebenchmark.com/leaderboard has improved substantially after publication.

Table 2: SQuAD 1.1 results. The BERT ensemble is 7x systems which use different pre-training checkpoints and ﬁne-tuning seeds.
System		Dev	Test
	EM	F1	EM	F1
	Top Leaderboard Systems (Dec 10th, 2018)
Human	-	-	82.3	91.2
#1 Ensemble - nlnet	-	-	86.0	91.7
#2 Ensemble - QANet	-	-	84.5	90.5
	Published
BiDAF+ELMo (Single)	-	85.6	-	85.8
R.M. Reader (Ensemble)	81.2	87.9	82.3	88.5
	Ours
BERT_B_A_S_E (Single)	80.8	88.5	-	-
BERT_L_A_R_G_E (Single)	84.1	90.9	-	-
BERT_L_A_R_G_E (Ensemble)	85.8	91.8	-	-
BERT_L_A_R_G_E (Sgl.+TriviaQA)	84.2	91.1	85.1	91.8
BERT_L_A_R_G_E (Ens.+TriviaQA)	86.2	92.2	87.4	93.2

Table 4: SWAG Dev and Test accuracies. Human performance is measured with 100 samples, as reported in the SWAG paper.
System	Dev	Test
ESIM+GloVe	51.9	52.7
ESIM+ELMo	59.1	59.2
OpenAI GPT	-	78.0
BERT_B_A_S_E	81.6	-
BERTLARGE	86.6	86.3
Human (expert)^†	-	85.0
Human (5 annotations)^†	-	88.0

··.sˆi,j=maxj≥iSTi+ETj

answer when sˆ_i_,_j > s_n_u_l + τ, where the threshold τ is selected on the dev set to maximize F1.

We did not use TriviaQA data for this model. We ﬁne-tuned for 2 epochs with a learning rate of 5e-5

Table 3: SQuAD 2.0 results. We exclude entries that use BERT as one of their components.
System	Dev		Test
	EM	F1	EM	F1
	Top Leaderboard Systems (Dec 10th, 2018)
Human	86.3	89.0	86.9	89.5
#1 Single - MIR-MRC (F-Net)	-	-	74.8	78.0
#2 Single - nlnet	-	-	74.2	77.1
	Published
unet (Ensemble)	-	-	71.4	74.9
SLQA+ (Single)	-		71.4	74.4
	Ours
BERT_L_A_R_G_E (Single)	78.7	81.9	80.0	83.1

and a batch size of 48.

The results compared to prior leaderboard entries and top published work (Sun et al., 2018;

Wang et al., 2018b) are shown in Table 3, excluding systems that use BERT as one of their components. We observe a +5.1 F1 improvement over the previous best system.

4.4 SWAG

The Situations With Adversarial Generations

(SWAG) dataset contains 113k sentence-pair com-

pletion examples that evaluate grounded commonsense inference (Zellers et al., 2018). Given a sentence, the task is to choose the most plausible con-

tuning data, we only lose 0.1-0.4 F1, still outper-

tinuation among four choices.

forming all existing systems by a wide margin.

When ﬁne-tuning on the SWAG dataset, we construct four input sequences, each containing

4.3 SQuAD v2.0

the concatenation of the given sentence (sentence

The SQuAD 2.0 task extends the SQuAD 1.1

A) and a possible continuation (sentence B). The

problem deﬁnition by allowing for the possibility

only task-speciﬁc parameters introduced is a vec-

that no short answer exists in the provided para-

tor whose dot product with the [CLS] token repgraph, making the problem more realistic. resentation C denotes a score for each choice

We use a simple approach to extend the SQuAD which is normalized with a softmax layer. v1.1 BERT model for this task. We treat ques-

We ﬁne-tune the model for 3 epochs with a

tions that do not have an answer as having an an-

learning rate of 2e-5 and a batch size of 16. Re-

swer span with start and end at the [CLS] to-sults are presented in Table 4. BERT out-LARGE ken. The probability space for the start and end performs the authors’ baseline ESIM+ELMo sysanswer span positions is extended to include the tem by +27.1% and OpenAI GPT by 8.3%. position of the [CLS] token. For prediction, we compare the score of the no-answer span: s_n_u_l =

5 Ablation Studies

S·C + E·C to the score of the best non-null span

In this section, we perform ablation experiments

Footnotes

12 The TriviaQA data we used consists of paragraphs from over a number of facets of BERT in order to better TriviaQA-Wiki formed of the ﬁrst 400 tokens in documents, that contain at least one of the provided possible answers. understand their relative importance. Additional

Table 5: Ablation over the pre-training tasks using the BERT architecture. “No NSP” is trained without BASE the next sentence prediction task. “LTR & No NSP” is trained as a left-to-right LM without the next sentence prediction, like OpenAI GPT. “+ BiLSTM” adds a randomly initialized BiLSTM on top of the “LTR + No NSP” model during ﬁne-tuning.
Dev Set
Tasks	MNLI-m	QNLI	MRPC	SST-2	SQuAD
Tasks	(Acc)	(Acc)	(Acc)	(Acc)	(F1)
BERT_B_A_S_E	84.4	88.4	86.7	92.7	88.5
No NSP	83.9	84.9	86.5	92.6	87.9
LTR & No NSP	82.1	84.3	77.5	92.1	77.8
+ BiLSTM	82.1	84.1	75.7	91.6	84.9

results are still far worse than those of the pretrained bidirectional models. The BiLSTM hurts performance on the GLUE tasks. We recognize that it would also be possible to train separate LTR and RTL models and represent each token as the concatenation of the two models, as ELMo does. However: (a) this is twice as expensive as a single bidirectional model; (b) this is non-intuitive for tasks like QA, since the RTL model would not be able to condition the answer on the question; (c) this it is strictly less powerful than a deep bidirectional model, since it can use both left and right context at every layer.

5.2 Effect of Model Size

ablation studies can be found in Appendix C.

In this section, we explore the effect of model size

5.1 Effect of Pre-training Tasks on ﬁne-tuning task accuracy. We trained a number of BERT models with a differing number of layers, We demonstrate the importance of the deep bidi-hidden units, and attention heads, while otherwise rectionality of BERT by evaluating two pre-using the same hyperparameters and training protraining objectives using exactly the same pre-cedure as described previously. training data, ﬁne-tuning scheme, and hyperparameters as BERT: BASE Results on selected GLUE tasks are shown in Table 6. In this table, we report the average Dev

No NSP: A bidirectional model which is trained Set accuracy from 5 random restarts of ﬁne-tuning. using the “masked LM” (MLM) but without the We can see that larger models lead to a strict ac-“next sentence prediction” (NSP) task. curacy improvement across all four datasets, even LTR & No NSP: A left-context-only model which for MRPC which only has 3,600 labeled trainis trained using a standard Left-to-Right (LTR) ing examples, and is substantially different from LM, rather than an MLM. The left-only constraint the pre-training tasks. It is also perhaps surpriswas also applied at ﬁne-tuning, because removing ing that we are able to achieve such signiﬁcant it introduced a pre-train/ﬁne-tune mismatch that improvements on top of models which are aldegraded downstream performance. Additionally, ready quite large relative to the existing literature. this model was pre-trained without the NSP task. For example, the largest Transformer explored in This is directly comparable to OpenAI GPT, but Vaswani et al. (2017) is (L=6, H=1024, A=16) using our larger training dataset, our input repre- with 100M parameters for the encoder, and the sentation, and our ﬁne-tuning scheme. largest Transformer we have found in the literature We ﬁrst examine the impact brought by the NSP is (L=64, H=512, A=2) with 235M parameters task. In Table 5, we show that removing NSP (Al-Rfou et al., 2018). By contrast, BERT BASE hurts performance signiﬁcantly on QNLI, MNLI, contains 110M parameters and BERT con-LARGE and SQuAD 1.1. Next, we evaluate the impact tains 340M parameters. of training bidirectional representations by com- It has long been known that increasing the paring “No NSP” to “LTR & No NSP”. The LTR model size will lead to continual improvements model performs worse than the MLM model on all on large-scale tasks such as machine translation tasks, with large drops on MRPC and SQuAD. and language modeling, which is demonstrated For SQuAD it is intuitively clear that a LTR by the LM perplexity of held-out training data model will perform poorly at token predictions, shown in Table 6. However, we believe that since the token-level hidden states have no right- this is the ﬁrst work to demonstrate convincside context. In order to make a good faith at- ingly that scaling to extreme model sizes also tempt at strengthening the LTR system, we added leads to large improvements on very small scale a randomly initialized BiLSTM on top. This does tasks, provided that the model has been sufﬁsigniﬁcantly improve results on SQuAD, but the ciently pre-trained. Peters et al. (2018b) presented

Table 7: CoNLL-2003 Named Entity Recognition results. Hyperparameters were selected using the Dev set. The reported Dev and Test scores are averaged over 5 random restarts using those hyperparameters.
System	Dev F1	Test F1
ELMo (Peters et al., 2018a)	95.7	92.2
CVT (Clark et al., 2018)	-	92.6
CSE (Akbik et al., 2018)	-	93.1
Fine-tuning approach
BERTLARGE	96.6	92.8
BERT_B_A_S_E	96.4	92.4
Feature-based approach (BERT_B_A_S_E)
Embeddings	91.0	-
Second-to-Last Hidden	95.6	-
Last Hidden	94.9	-
Weighted Sum Last Four Hidden	95.9	-
Concat Last Four Hidden	96.1	-
Weighted Sum All 12 Layers	95.5	-

mixed results on the downstream task impact of increasing the pre-trained bi-LM size from two to four layers and Melamud et al. (2016) mentioned in passing that increasing hidden dimension size from 200 to 600 helped, but increasing further to 1,000 did not bring further improvements. Both of these prior works used a featurebased approach — we hypothesize that when the model is ﬁne-tuned directly on the downstream tasks and uses only a very small number of randomly initialized additional parameters, the taskspeciﬁc models can beneﬁt from the larger, more expressive pre-trained representations even when

downstream task data is very small.

5.3 Feature-based Approach with BERT

All of the BERT results presented so far have used the ﬁne-tuning approach, where a simple classiﬁcation layer is added to the pre-trained model, and

layer in the output. We use the representation of

all parameters are jointly ﬁne-tuned on a down-

the ﬁrst sub-token as the input to the token-level stream task. However, the feature-based approach, classiﬁer over the NER label set. where ﬁxed features are extracted from the pre-To ablate the ﬁne-tuning approach, we apply the trained model, has certain advantages. First, not feature-based approach by extracting the activaall tasks can be easily represented by a Trans-tions from one or more layers without ﬁne-tuning former encoder architecture, and therefore require any parameters of BERT. These contextual em-a task-speciﬁc model architecture to be added. beddings are used as input to a randomly initial-

Second, there are major computational beneﬁts ized two-layer 768-dimensional BiLSTM before to pre-compute an expensive representation of the the classiﬁcation layer. training data once and then run many experiments with cheaper models on top of this representation. ^L^A^R^G^E Results are presented in Table 7. BERT performs competitively with state-of-the-art meth-In this section, we compare the two approaches ods. The best performing method concatenates the by applying BERT to the CoNLL-2003 Named token representations from the top four hidden lay-

Entity Recognition (NER) task (Tjong Kim Sang

ers of the pre-trained Transformer, which is only

and De Meulder, 2003). In the input to BERT, we

0.3 F1 behind ﬁne-tuning the entire model. This

use a case-preserving WordPiece model, and we

demonstrates that BERT is effective for both ﬁne-

include the maximal document context provided

tuning and feature-based approaches.

by the data. Following standard practice, we formulate this as a tagging task but do not use a CRF

6 Conclusion

Table 6: Ablation over BERT model size. #L = the number of layers; #H = hidden size; #A = number of attention heads. “LM (ppl)” is the masked LM perplexity of held-out training data.
Hyperparams				Dev Set Accuracy
#L	#H	#A	LM (ppl)	MNLI-m	MRPC	SST-2
3	768	12	5.84	77.9	79.8	88.4
6	768	3	5.24	80.6	82.2	90.7
6	768	12	4.68	81.9	84.8	91.3
12	768	12	3.99	84.4	86.7	92.9
12	1024	16	3.54	85.7	86.9	93.3
24	1024	16	3.23	86.6	87.8	93.7

Recent empirical improvements due to transfer learning with language models have demonstrated that rich, unsupervised pre-training is an integral part of many language understanding systems. In particular, these results enable even low-resource tasks to beneﬁt from deep unidirectional architectures. Our major contribution is further generalizing these ﬁndings to deep bidirectional architec-

tures, allowing the same pre-trained model to suc-

cessfully tackle a broad set of NLP tasks.

References Kevin Clark, Minh-Thang Luong, Christopher D Man-ning, and Quoc Le. 2018. Semi-supervised se-

Alan Akbik, Duncan Blythe, and Roland Vollgraf. quence modeling with cross-view training. In Pro-

2018. Contextual string embeddings for sequence ceedings of the 2018 Conference on Empirical Methlabeling. In Proceedings of the 27th International ods in Natural Language Processing, pages 1914–

Conference on Computational Linguistics, pages1925.

1638–1649.

Ronan Collobert and Jason Weston. 2008. A uniﬁed

Rami Al-Rfou, Dokook Choe, Noah Constant, Mandy architecture for natural language processing: Deep

Guo, and Llion Jones. 2018. Character-level lan- neural networks with multitask learning. In Proguage modeling with deeper self-attention. arXiv ceedings of the 25th international conference on preprint arXiv:1808.04444. Machine learning, pages 160–167. ACM.

Rie Kubota Ando and Tong Zhang. 2005. A framework Alexis Conneau, Douwe Kiela, Holger Schwenk, Loı̈c for learning predictive structures from multiple tasks Barrault, and Antoine Bordes. 2017. Supervised and unlabeled data. Journal of Machine Learning learning of universal sentence representations from

Research, 6(Nov):1817–1853. natural language inference data. In Proceedings of the 2017 Conference on Empirical Methods in Nat-ural Language Processing, pages 670–680, Copen-Luisa Bentivogli, Bernardo Magnini, Ido Dagan, hagen, Denmark. Association for Computational Hoa Trang Dang, and Danilo Giampiccolo. 2009. Linguistics. The ﬁfth PASCAL recognizing textual entailment challenge. In TAC. NIST. Andrew M Dai and Quoc V Le. 2015. Semi-supervised sequence learning. In Advances in neural informa-

John Blitzer, Ryan McDonald, and Fernando Pereira.tion processing systems, pages 3079–3087.

2006. Domain adaptation with structural correspondence learning. In Proceedings of the 2006 confer-

J. Deng, W. Dong, R. Socher, L.-J. Li, K. Li, and L. Feience on empirical methods in natural language pro-Fei. 2009. ImageNet: A Large-Scale Hierarchical cessing, pages 120–128. Association for Computa-Image Database. In CVPR09. tional Linguistics.

William B Dolan and Chris Brockett. 2005. Automati-

Samuel R. Bowman, Gabor Angeli, Christopher Potts, cally constructing a corpus of sentential paraphrases. and Christopher D. Manning. 2015. A large anno- In Proceedings of the Third International Workshop tated corpus for learning natural language inference. on Paraphrasing (IWP2005).

In EMNLP. Association for Computational Linguis-tics. William Fedus, Ian Goodfellow, and Andrew M Dai.

2018. Maskgan: Better text generation via ﬁlling in

Peter F Brown, Peter V Desouza, Robert L Mercer, the. arXiv preprint arXiv:1801.07736.

Vincent J Della Pietra, and Jenifer C Lai. 1992.

Class-based n-gram models of natural language. Dan Hendrycks and Kevin Gimpel. 2016. Bridging

Computational linguistics, 18(4):467–479. nonlinearities and stochastic regularizers with gaus-sian error linear units. CoRR, abs/1606.08415.

Daniel Cer, Mona Diab, Eneko Agirre, Inigo Lopez-

Gazpio, and Lucia Specia. 2017. Semeval-2017 Felix Hill, Kyunghyun Cho, and Anna Korhonen. 2016. task 1: Semantic textual similarity multilingual and Learning distributed representations of sentences crosslingual focused evaluation. In Proceedings from unlabelled data. In Proceedings of the 2016

Conference of the North American Chapter of the of the 11th International Workshop on Semantic

Evaluation (SemEval-2017), pages 1–14, Vancou-Association for Computational Linguistics: Human ver, Canada. Association for Computational Lin-Language Technologies. Association for Computaguistics. tional Linguistics.

Jeremy Howard and Sebastian Ruder. 2018. Universal Ciprian Chelba, Tomas Mikolov, Mike Schuster, Qi Ge, language model ﬁne-tuning for text classiﬁcation. In Thorsten Brants, Phillipp Koehn, and Tony Robin-ACL. Association for Computational Linguistics. son. 2013. One billion word benchmark for measur-ing progress in statistical language modeling. arXiv Minghao Hu, Yuxing Peng, Zhen Huang, Xipeng Qiu, preprint arXiv:1312.3005. Furu Wei, and Ming Zhou. 2018. Reinforced mnemonic reader for machine reading comprehen-

Z. Chen, H. Zhang, X. Zhang, and L. Zhao. 2018.sion. In IJCAI.

Quora question pairs.

Yacine Jernite, Samuel R. Bowman, and David Son-

Christopher Clark and Matt Gardner. 2018. Simple tag. 2017. Discourse-based objectives for fast unand effective multi-paragraph reading comprehen- supervised sentence representation learning. CoRR, sion. In ACL. abs/1705.00557.

Mandar Joshi, Eunsol Choi, Daniel S Weld, and Luke Matthew Peters, Mark Neumann, Luke Zettlemoyer,

Zettlemoyer. 2017. Triviaqa: A large scale distantly and Wen-tau Yih. 2018b. Dissecting contextual supervised challenge dataset for reading comprehen- word embeddings: Architecture and representation. sion. In ACL. In Proceedings of the 2018 Conference on Empiri-cal Methods in Natural Language Processing, pages

Ryan Kiros, Yukun Zhu, Ruslan R Salakhutdinov,1499–1509.

Richard Zemel, Raquel Urtasun, Antonio Torralba, and Sanja Fidler. 2015. Skip-thought vectors. In Alec Radford, Karthik Narasimhan, Tim Salimans, and

Advances in neural information processing systems, Ilya Sutskever. 2018. Improving language underpages 3294–3302. standing with unsupervised learning. Technical re-port, OpenAI.

Quoc Le and Tomas Mikolov. 2014. Distributed rep-resentations of sentences and documents. In Inter- Pranav Rajpurkar, Jian Zhang, Konstantin Lopyrev, and national Conference on Machine Learning, pages Percy Liang. 2016. Squad: 100,000+ questions for

1188–1196. machine comprehension of text. In Proceedings of the 2016 Conference on Empirical Methods in Nat-Hector J Levesque, Ernest Davis, and Leora Morgen-ural Language Processing, pages 2383–2392. stern. 2011. The winograd schema challenge. In

Aaai spring symposium: Logical formalizations of Minjoon Seo, Aniruddha Kembhavi, Ali Farhadi, and commonsense reasoning, volume 46, page 47. Hannaneh Hajishirzi. 2017. Bidirectional attention ﬂow for machine comprehension. In ICLR. Lajanugen Logeswaran and Honglak Lee. 2018. An efﬁcient framework for learning sentence represen-Richard Socher, Alex Perelygin, Jean Wu, Jason tations. In International Conference on Learning Chuang, Christopher D Manning, Andrew Ng, and

Representations. Christopher Potts. 2013. Recursive deep models for semantic compositionality over a sentiment tree-Bryan McCann, James Bradbury, Caiming Xiong, and bank. In Proceedings of the 2013 conference on Richard Socher. 2017. Learned in translation: Con-empirical methods in natural language processing, textualized word vectors. In NIPS. pages 1631–1642.

Oren Melamud, Jacob Goldberger, and Ido Dagan. Fu Sun, Linyang Li, Xipeng Qiu, and Yang Liu. 2016. context2vec: Learning generic context em-2018. U-net: Machine reading comprehension bedding with bidirectional LSTM. In CoNLL. with unanswerable questions. arXiv preprint

Tomas Mikolov, Ilya Sutskever, Kai Chen, Greg S Cor- arXiv:1810.06638. rado, and Jeff Dean. 2013. Distributed representa-tions of words and phrases and their compositional-Wilson L Taylor. 1953. Cloze procedure: A new ity. In Advances in Neural Information Processing tool for measuring readability. Journalism Bulletin,

Systems 26, pages 3111–3119. Curran Associates, 30(4):415–433.

Inc. Erik F Tjong Kim Sang and Fien De Meulder.

Andriy Mnih and Geoffrey E Hinton. 2009. A scal-2003. Introduction to the conll-2003 shared task: able hierarchical distributed language model. In Language-independent named entity recognition. In

CoNLL.

D. Koller, D. Schuurmans, Y. Bengio, and L. Bot-tou, editors, Advances in Neural Information Pro-Joseph Turian, Lev Ratinov, and Yoshua Bengio. 2010. cessing Systems 21, pages 1081–1088. Curran As-Word representations: A simple and general method sociates, Inc. for semi-supervised learning. In Proceedings of the

Ankur P Parikh, Oscar Täckström, Dipanjan Das, and48th Annual Meeting of the Association for Compu-

Jakob Uszkoreit. 2016. A decomposable attention tational Linguistics, ACL’10, pages 384–394. model for natural language inference. In EMNLP. Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob

Jeffrey Pennington, Richard Socher, and Christo- Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz pher D. Manning. 2014. Glove: Global vectors for Kaiser, and Illia Polosukhin. 2017. Attention is all word representation. In Empirical Methods in Nat- you need. In Advances in Neural Information Proural Language Processing (EMNLP), pages 1532– cessing Systems, pages 6000–6010.

1543.

Pascal Vincent, Hugo Larochelle, Yoshua Bengio, and

Matthew Peters, Waleed Ammar, Chandra Bhagavat- Pierre-Antoine Manzagol. 2008. Extracting and ula, and Russell Power. 2017. Semi-supervised se- composing robust features with denoising autoenquence tagging with bidirectional language models. coders. In Proceedings of the 25th international

In ACL.conference on Machine learning, pages 1096–1103.

ACM.

Matthew Peters, Mark Neumann, Mohit Iyyer, Matt

Gardner, Christopher Clark, Kenton Lee, and Luke Alex Wang, Amanpreet Singh, Julian Michael, Fe-

Zettlemoyer. 2018a. Deep contextualized word rep- lix Hill, Omer Levy, and Samuel Bowman. 2018a. resentations. In NAACL. Glue: A multi-task benchmark and analysis platform

for natural language understanding. In Proceedings • Additional details for our experiments are of the 2018 EMNLP Workshop BlackboxNLP: An- presented in Appendix B; and alyzing and Interpreting Neural Networks for NLP, pages 353–355.

• Additional ablation studies are presented in

Appendix C. Wei Wang, Ming Yan, and Chen Wu. 2018b. Multi-granularity hierarchical attention fusion networks We present additional ablation studies for for reading comprehension and question answering. BERT including: In Proceedings of the 56th Annual Meeting of the As-sociation for Computational Linguistics (Volume 1:

– Effect of Number of Training Steps; and

Long Papers). Association for Computational Lin-guistics. – Ablation for Different Masking Proce-dures.

Alex Warstadt, Amanpreet Singh, and Samuel R Bow-man. 2018. Neural network acceptability judg- A Additional Details for BERT ments. arXiv preprint arXiv:1805.12471.

A.1 Illustration of the Pre-training Tasks

Adina Williams, Nikita Nangia, and Samuel R Bow-man. 2018. A broad-coverage challenge corpus We provide examples of the pre-training tasks in for sentence understanding through inference. In the following.

NAACL.

Masked LM and the Masking Procedure As-

Yonghui Wu, Mike Schuster, Zhifeng Chen, Quoc V suming the unlabeled sentence is_m_y_d_o_g_i_s Le, Mohammad Norouzi, Wolfgang Macherey, _h_a_i_r_y, and during the random masking procedure Maxim Krikun, Yuan Cao, Qin Gao, Klaus

Macherey, et al. 2016. Google’s neural ma-we chose the 4-th token (which corresponding to chine translation system: Bridging the gap between hairy), our masking procedure can be further ilhuman and machine translation. arXiv preprint lustrated by arXiv:1609.08144.

• 80% of the time: Replace the word with the Jason Yosinski, Jeff Clune, Yoshua Bengio, and Hod Lipson. 2014. How transferable are features in deep [MASK] token, e.g., my dog is hairy → neural networks? In Advances in neural information my dog is [MASK] processing systems, pages 3320–3328.

• 10% of the time: Replace the word with a

Adams Wei Yu, David Dohan, Minh-Thang Luong, Ruirandom word, e.g., my dog is hairy → my

Zhao, Kai Chen, Mohammad Norouzi, and Quoc V dog is apple Le. 2018. QANet: Combining local convolution with global self-attention for reading comprehen-

• 10% of the time: Keep the word unsion. In ICLR. changed, e.g., my dog is hairy → my dog

Rowan Zellers, Yonatan Bisk, Roy Schwartz, and Yejin. The purpose of this is to bias the is hairy

Choi. 2018. Swag: A large-scale adversarial dataset representation towards the actual observed for grounded commonsense inference. In Proceed-word. ings of the 2018 Conference on Empirical Methods in Natural Language Processing (EMNLP). The advantage of this procedure is that the

Yukun Zhu, Ryan Kiros, Rich Zemel, Ruslan Salakhut- Transformer encoder does not know which words dinov, Raquel Urtasun, Antonio Torralba, and Sanja it will be asked to predict or which have been re-Fidler. 2015. Aligning books and movies: Towards placed by random words, so it is forced to keep story-like visual explanations by watching movies and reading books. In Proceedings of the IEEE a distributional contextual representation of evinternational conference on computer vision, pages ery input token. Additionally, because random

19–27. replacement only occurs for 1.5% of all tokens (i.e., 10% of 15%), this does not seem to harm

Appendix for “BERT: Pre-training of

the model’s language understanding capability. In

Deep Bidirectional Transformers for

Section C.2, we evaluate the impact this proce-

Language Understanding”

dure.

We organize the appendix into three sections: Compared to standard langauge model training, the masked LM only make predictions on 15% of

• Additional implementation details for BERT tokens in each batch, which suggests that more are presented in Appendix A; pre-training steps may be required for the model

Figure 3: Differences in pre-training model architectures. BERT uses a bidirectional Transformer. OpenAI GPT uses a left-to-right Transformer. ELMo uses the concatenation of independently trained left-to-right and right-toleft LSTMs to generate features for downstream tasks. Among the three, only BERT representations are jointly conditioned on both left and right context in all layers. In addition to the architecture differences, BERT and OpenAI GPT are ﬁne-tuning approaches, while ELMo is a feature-based approach.

to converge. In SectionC.1we demonstrate that MLM does converge marginally slower than a left- to-right model (which predicts every token), but the empirical improvements of the MLM model far outweigh the increased training cost. Next Sentence Prediction The next sentence prediction task can be illustrated in the following examples. Input = [CLS] the man went to [MASK] store [SEP] he bought a gallon [MASK] milk [SEP] Label = IsNext Input = [CLS] the man [MASK] to the store [SEP] penguin [MASK] are flight ##less birds [SEP] Label = NotNext A.2 Pre-training Procedure To generate each training input sequence, we sam- ple two spans of text from the corpus, which we refer to as “sentences” even though they are typ- ically much longer than single sentences (but can be shorter also). The ﬁrst sentence receives the A embedding and the second receives the B embed- ding. 50% of the time B is the actual next sentence that follows A and 50% of the time it is a random sentence, which is done for the “next sentence pre- diction” task. They are sampled such that the com- bined length is ≤ 512 tokens. The LM masking is applied after WordPiece tokenization with a uni- form masking rate of 15%, and no special consid- eration given to partial word pieces. We train with batch size of 256 sequences (256 sequences * 512 tokens = 128,000 tokens/batch) for 1,000,000 steps, which is approximately 40

epochs over the 3.3 billion word corpus. We use Adam with learning rate of 1e-4, β = 0.9, 1 β = 0.999, L2 weight decay of 0.01, learning 2 rate warmup over the ﬁrst 10,000 steps, and linear decay of the learning rate. We use a dropout prob- ability of 0.1 on all layers. We use a gelu acti- vation (Hendrycks and Gimpel,2016) rather than the standard relu, following OpenAI GPT. The training loss is the sum of the mean masked LM likelihood and the mean next sentence prediction likelihood. Training of BERT was performed on 4 BASE Cloud TPUs in Pod conﬁguration (16 TPU chips 13 total). Training of BERT was performed LARGE on 16 Cloud TPUs (64 TPU chips total). Each pre- training took 4 days to complete. Longer sequences are disproportionately expen- sive because attention is quadratic to the sequence length. To speed up pretraing in our experiments, we pre-train the model with sequence length of 128 for 90% of the steps. Then, we train the rest 10% of the steps of sequence of 512 to learn the positional embeddings. A.3 Fine-tuning Procedure For ﬁne-tuning, most model hyperparameters are the same as in pre-training, with the exception of the batch size, learning rate, and number of train- ing epochs. The dropout probability was always kept at 0.1. The optimal hyperparameter values are task-speciﬁc, but we found the following range of possible values to work well across all tasks: • Batch size: 16, 32 13 https://cloudplatform.googleblog.com/2018/06/Cloud- TPU-now-offers-preemptible-pricing-and-global- availability.html

Learning rate (Adam): 5e-5, 3e-5, 2e-5 To isolate the effect of these differences, we per-
Number of epochs: 2, 3, 4 form ablation experiments in Section 5.1 which

demonstrate that the majority of the improvements

We also observed that large data sets (e.g., are in fact coming from the two pre-training tasks

100k+ labeled training examples) were far less and the bidirectionality they enable. sensitive to hyperparameter choice than small data sets. Fine-tuning is typically very fast, so it is rea-A.5 Illustrations of Fine-tuning on Different sonable to simply run an exhaustive search over Tasks the above parameters and choose the model that The illustration of ﬁne-tuning BERT on different performs best on the development set. tasks can be seen in Figure 4. Our task-speciﬁc models are formed by incorporating BERT with A.4 Comparison of BERT, ELMo,and one additional output layer, so a minimal num-OpenAI GPT ber of parameters need to be learned from scratch.

Here we studies the differences in recent popular Among the tasks, (a) and (b) are sequence-level representation learning models including ELMo, tasks while (c) and (d) are token-level tasks. In OpenAI GPT and BERT. The comparisons be- the ﬁgure, E represents the input embedding, T_i tween the model architectures are shown visually represents the contextual representation of token i, in Figure 3. Note that in addition to the architec- [CLS] is the special symbol for classiﬁcation outture differences, BERT and OpenAI GPT are ﬁne- put, and [SEP] is the special symbol to separate tuning approaches, while ELMo is a feature-based non-consecutive token sequences. approach.

B Detailed Experimental Setup

The most comparable existing pre-training method to BERT is OpenAI GPT, which trains a

B.1 Detailed Descriptions for the GLUE

left-to-right Transformer LM on a large text cor-Benchmark Experiments. pus. In fact, many of the design decisions in BERT were intentionally made to make it as close to The GLUE benchmark includes the following

GPT as possible so that the two methods could be datasets, the descriptions of which were originally minimally compared. The core argument of this summarized in Wang et al. (2018a): work is that the bi-directionality and the two pre-MNLI Multi-Genre Natural Language Inference training tasks presented in Section 3.1 account for is a large-scale, crowdsourced entailment classiﬁthe majority of the empirical improvements, but cation task (Williams et al., 2018). Given a pair of we do note that there are several other differences sentences, the goal is to predict whether the secbetween how BERT and GPT were trained: ond sentence is an entailment, contradiction, or neutral with respect to the ﬁrst one.

• GPT is trained on the BooksCorpus (800M words); BERT is trained on the BooksCor- QQP Quora Question Pairs is a binary classiﬁpus (800M words) and Wikipedia (2,500M cation task where the goal is to determine if two words). questions asked on Quora are semantically equiv-alent (Chen et al., 2018).

• GPT uses a sentence separator ([SEP]) and classiﬁer token ([CLS]) which are only in-QNLI Question Natural Language Inference is troduced at ﬁne-tuning time; BERT learns a version of the Stanford Question Answering [SEP], [CLS] and sentence A/B embed- Dataset (Rajpurkar et al., 2016) which has been dings during pre-training. converted to a binary classiﬁcation task (Wang et al., 2018a). The positive examples are (ques-

• GPT was trained for 1M steps with a batch tion, sentence) pairs which do contain the correct size of 32,000 words; BERT was trained for answer, and the negative examples are (question,

1M steps with a batch size of 128,000 words. sentence) from the same paragraph which do not contain the answer.

• GPT used the same learning rate of 5e-5 for all ﬁne-tuning experiments; BERT chooses a SST-2 The Stanford Sentiment Treebank is a task-speciﬁc ﬁne-tuning learning rate which binary single-sentence classiﬁcation task consistperforms the best on the development set. ing of sentences extracted from movie reviews

Figure 4: Illustrations of Fine-tuning BERT on Different Tasks. with human annotations of their sentiment ( Socher RTE Recognizing Textual Entailment is a bi- et al. , 2013 ). nary entailment task similar to MNLI, but with 14 much less training data ( Bentivogli et al. , 2009 ). CoLA The Corpus of Linguistic Acceptability is a binary single-sentence classiﬁcation task, where WNLI Winograd NLI is a small natural lan- the goal is to predict whether an English sentence guage inference dataset ( Levesque et al. , 2011 ). is linguistically “acceptable” or not ( Warstadt The GLUE webpage notes that there are issues et al. , 2018 ). 15 with the construction of this dataset, and every trained system that’s been submitted to GLUE has STS-B The Semantic Textual Similarity Bench- performed worse than the 65.1 baseline accuracy mark is a collection of sentence pairs drawn from of predicting the majority class. We therefore ex- news headlines and other sources ( Cer et al. , clude this set to be fair to OpenAI GPT. For our 2017 ). They were annotated with a score from 1 GLUE submission, we always predicted the ma- to 5 denoting how similar the two sentences are in jority class. terms of semantic meaning. MRPC Microsoft Research Paraphrase Corpus 14 Note that we only report single-task ﬁne-tuning results consists of sentence pairs automatically extracted in this paper. A multitask ﬁne-tuning approach could poten- tially push the performance even further. For example, we from online news sources, with human annotations did observe substantial improvements on RTE from multi- for whether the sentences in the pair are semanti- task training with MNLI. cally equivalent ( Dolan and Brockett , 2005 ). 15 https://gluebenchmark.com/faq

C Additional Ablation Studies and ﬁne-tuning, as the [MASK] symbol never ap-pears during the ﬁne-tuning stage. We report the C.1 Effect of Number of Training Steps Dev results for both MNLI and NER. For NER,

Figure 5 presents MNLI Dev accuracy after ﬁne- we report both ﬁne-tuning and feature-based aptuning from a checkpoint that has been pre-trained proaches, as we expect the mismatch will be amfor k steps. This allows us to answer the following pliﬁed for the feature-based approach as the model

questions:			tions.
1. Question:		BERT	really	need	such
a	large	Masking Rates			Dev Set Results
a	words/batch * 1,000,000 steps) to achieve	MASK	SAME	RND	MNLI	NER
high ﬁne-tuning accuracy?		MASK	SAME	RND	Fine-tune	Fine-tune	Feature-based
Answer:	Yes, BERT_B_A_S_E achieves almost	80%	10%	10%	84.2	95.4	94.9
		100%	0%	0%	84.3	94.9	94.0
	1.0% additional accuracy on MNLI when
		80%	0%	20%	84.1	95.2	94.6
	trained on 1M steps compared to 500k steps.	80%	20%	0%	84.4	95.2	94.7
		0%	20%	80%	83.7	94.8	94.6
	2. Question: Does MLM pre-training converge	0%	0%	100%	83.6	94.9	94.6
	slower than LTR pre-training, since only 15%

of words are predicted in each batch rather Table 8: Ablation over different masking strategies. than every word? Answer: The MLM model does converge The results are presented in Table 8. In the table, slightly slower than the LTR model. How- MASK means that we replace the target token with ever, in terms of absolute accuracy the MLM the [MASK] symbol for MLM; SAME means that model begins to outperform the LTR model we keep the target token as is; RND means that almost immediately. we replace the target token with another random token.

C.2 Ablation for Different Masking The numbers in the left part of the table repre-

Procedures sent the probabilities of the speciﬁc strategies used during MLM pre-training (BERT uses 80%, 10%, In Section 3.1, we mention that BERT uses a 10%). The right part of the paper represents the mixed strategy for masking the target tokens when Dev set results. For the feature-based approach, pre-training with the masked language model we concatenate the last 4 layers of BERT as the (MLM) objective. The following is an ablation features, which was shown to be the best approach study to evaluate the effect of different masking in Section 5.3. strategies. From the table it can be seen that ﬁne-tuning is Note that the purpose of the masking strategies surprisingly robust to different masking strategies. is to reduce the mismatch between pre-training However, as expected, using only the MASK strategy was problematic when applying the feature-based approach to NER. Interestingly, using only the RND strategy performs much worse than our ⁸² strategy as well.

Footnotes

80
78 BERT (Masked LM) BASE
76 BERT (Left-to-Right) BASE ₂₀₄₀₆₀₈₀₁_,₀ Pre-training Steps (Thousands) Figure 5: Ablation over number of training steps. This shows the MNLI accuracy after ﬁne-tuning, starting from model parameters that have been pre-trained for k steps. The x-axis is the value of k.

BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding

Jacob Devlin Ming-Wei Chang Kenton Lee Kristina Toutanova

1 Introduction

2 Related Work

2.2 Unsupervised Fine-tuning Approaches

2.1 Unsupervised Feature-based Approaches

3.2 Fine-tuning BERT sults on 11 NLP tasks.

4.2 SQuAD v1.1

Footnotes

4.4 SWAG

4.3 SQuAD v2.0

5 Ablation Studies

Footnotes

5.2 Effect of Model Size

5.3 Feature-based Approach with BERT

6 Conclusion

A.1 Illustration of the Pre-training Tasks

Appendix for “BERT: Pre-training of

Deep Bidirectional Transformers for

Language Understanding”

B Detailed Experimental Setup

B.1 Detailed Descriptions for the GLUE

Footnotes