Migrate model card from transformers-repo
Browse filesRead announcement at https://discuss.huggingface.co/t/announcement-all-model-cards-will-be-migrated-to-hf-co-model-repos/2755
Original file history: https://github.com/huggingface/transformers/commits/master/model_cards/mrm8488/bert-uncased-finetuned-qnli/README.md
README.md
ADDED
@@ -0,0 +1,62 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
1 |
+
---
|
2 |
+
language: en
|
3 |
+
thumbnail:
|
4 |
+
---
|
5 |
+
|
6 |
+
# [BERT](https://huggingface.co/deepset/bert-base-cased-squad2) fine tuned on [QNLI](https://github.com/rhythmcao/QNLI)+ compression ([BERT-of-Theseus](https://github.com/JetRunner/BERT-of-Theseus))
|
7 |
+
|
8 |
+
I used a [Bert model fine tuned on **SQUAD v2**](https://huggingface.co/deepset/bert-base-cased-squad2) and then I fine tuned it on **QNLI** using **compression** (with a constant replacing rate) as proposed in **BERT-of-Theseus**
|
9 |
+
|
10 |
+
## Details of the downstream task (QNLI):
|
11 |
+
|
12 |
+
### Getting the dataset
|
13 |
+
```bash
|
14 |
+
wget https://raw.githubusercontent.com/rhythmcao/QNLI/master/data/QNLI/train.tsv
|
15 |
+
wget https://raw.githubusercontent.com/rhythmcao/QNLI/master/data/QNLI/test.tsv
|
16 |
+
wget https://raw.githubusercontent.com/rhythmcao/QNLI/master/data/QNLI/dev.tsv
|
17 |
+
|
18 |
+
mkdir QNLI_dataset
|
19 |
+
mv *.tsv QNLI_dataset
|
20 |
+
```
|
21 |
+
|
22 |
+
### Model training
|
23 |
+
|
24 |
+
The model was trained on a Tesla P100 GPU and 25GB of RAM with the following command:
|
25 |
+
|
26 |
+
```bash
|
27 |
+
!python /content/BERT-of-Theseus/run_glue.py \
|
28 |
+
--model_name_or_path deepset/bert-base-cased-squad2 \
|
29 |
+
--task_name qnli \
|
30 |
+
--do_train \
|
31 |
+
--do_eval \
|
32 |
+
--do_lower_case \
|
33 |
+
--data_dir /content/QNLI_dataset \
|
34 |
+
--max_seq_length 128 \
|
35 |
+
--per_gpu_train_batch_size 32 \
|
36 |
+
--per_gpu_eval_batch_size 32 \
|
37 |
+
--learning_rate 2e-5 \
|
38 |
+
--save_steps 2000 \
|
39 |
+
--num_train_epochs 50 \
|
40 |
+
--output_dir /content/ouput_dir \
|
41 |
+
--evaluate_during_training \
|
42 |
+
--replacing_rate 0.7 \
|
43 |
+
--steps_for_replacing 2500
|
44 |
+
```
|
45 |
+
|
46 |
+
## Metrics:
|
47 |
+
|
48 |
+
| Model | Accuracy |
|
49 |
+
|-----------------|------|
|
50 |
+
| BERT-base | 91.2 |
|
51 |
+
| BERT-of-Theseus | 88.8 |
|
52 |
+
| [bert-uncased-finetuned-qnli](https://huggingface.co/mrm8488/bert-uncased-finetuned-qnli) | 87.2
|
53 |
+
| DistillBERT | 85.3 |
|
54 |
+
|
55 |
+
|
56 |
+
|
57 |
+
|
58 |
+
> [See all my models](https://huggingface.co/models?search=mrm8488)
|
59 |
+
|
60 |
+
> Created by [Manuel Romero/@mrm8488](https://twitter.com/mrm8488)
|
61 |
+
|
62 |
+
> Made with <span style="color: #e25555;">♥</span> in Spain
|