Advance BERT model via transferring knowledge from Cross-Encoders to Bi-Encoders | by Chien Vu | Jan, 2021


In this experiment, I will be able to introduce a demo on how to observe AugSBERT with other situations. First, we want to import some programs

Scenario 1: Full annotated datasets (all categorized sentence-pairs)

The major goal of this situation is extending the categorized dataset by the directly ahead information augmentation methods, subsequently, we can get ready teach, dev, take a look at dataset at the Semantic Text Similarity dataset (link) and outline batch measurement, epoch, and model title (You can specify any Huggingface/transformers pre-trained model)

Then, we can insert phrases by our BERT model (you’ll observe any other argumentation method as I discussed within the Technique spotlight segment) to create a silver dataset.

Next, we outline our Bi-encoders with imply pooling with each(gold + silver) STS benchmark dataset.

Finally, we can assessment our model within the take a look at STS benchmark dataset.

Scenario 2: Limited or small annotated datasets (few categorized sentence-pairs)

In this situation, we can use Cross-encoders that had been skilled at the restricted categorized dataset (gold dataset) to smooth label the in-domain unlabeled dataset (silver dataset) and teach Bi-encoders in each datasets (silver + gold). In this simulation, I additionally use once more STS benchmark dataset and create new pairs of sentences by pre-trained SBERT model. First, we can outline Cross-encoders and Bi-encoders.

Step 1, we can get ready teach, dev, take a look at like ahead of and fine-tune our Cross-encoders

Step 2, we use our fine-tuned Cross-encoders to label unlabeled datasets.

Step 3, we teach our Bi-encoders in each gold and silver datasets

Finally, we can assessment our model within the take a look at STS benchmark dataset.

Scenario 3: No annotated datasets (Only unlabeled sentence-pairs)

In this situation, the entire steps are very an identical to situation 2 however in a special area. Because of the aptitude of our Cross-encoders, we can use a generic supply dataset (STS benchmark dataset) and switch the knowledge to a selected goal dataset (Quora Question Pairs)

And teach our Cross-encoders.

Labeling Quora Question Pairs dataset (silver dataset). In this situation, the duty is classification so now we have to convert our rating to binary ratings.

Then, practising our Bi-encoders

Finally, comparing on take a look at Quora Question Pairs dataset

AugSBERT is a straightforward and efficient information augmentation to make stronger Bi-encoders for pairwise sentence scoring duties. The thought is in keeping with labeling new sentence pairs by the usage of pre-trained Cross-encoders and mixing them into the educational set. Selecting the appropriate sentence pairs for soft-labeling is the most important and essential to make stronger the efficiency. The AugSBERT means can be used for area adaptation, by soft-labeling information at the goal area.

You can touch me if you wish to have additional dialogue. Here is my Linkedin

Enjoy!!! 👦🏻

[1] Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language figuring out.

[2] Thomas Wolf, Victor Sanh, Julien Chaumond, and Clement Delangue. Transfertransfo: A switch studying means for neural network-based conversational brokers.

[3] Pierre-Emmanuel Mazare, Samuel Humeau, Martin Raison, and Antoine Bordes. Training tens of millions of customized discussion brokers.

[4] Nils Reimers and Iryna Gurevych. SentenceBERT: Sentence Embeddings the usage of Siamese BERTNetworks.

[5] Samuel Humeau, Kurt Shuster, Marie-Anne Lachaux, and Jason Weston. Poly-encoders: Architectures and pre-training methods for speedy and correct multi-sentence scoring.

[6] Nandan Thakur, Nils Reimers, Johannes Daxenberge, and Iryna Gurevych. Augmented SBERT: Data Augmentation Method for Improving Bi-Encoders for Pairwise Sentence Scoring Tasks.

[7] Giambattista Amati. BM25, Springer US, Boston, MA.


Please enter your comment!
Please enter your name here