julien-c HF staff commited on
Commit
0992775
·
1 Parent(s): 3f1bf3c

Migrate model card from transformers-repo

Browse files

Read announcement at https://discuss.huggingface.co/t/announcement-all-model-cards-will-be-migrated-to-hf-co-model-repos/2755
Original file history: https://github.com/huggingface/transformers/commits/master/model_cards/mrm8488/bert-uncased-finetuned-qnli/README.md

Files changed (1) hide show
  1. README.md +62 -0
README.md ADDED
@@ -0,0 +1,62 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ ---
2
+ language: en
3
+ thumbnail:
4
+ ---
5
+
6
+ # [BERT](https://huggingface.co/deepset/bert-base-cased-squad2) fine tuned on [QNLI](https://github.com/rhythmcao/QNLI)+ compression ([BERT-of-Theseus](https://github.com/JetRunner/BERT-of-Theseus))
7
+
8
+ I used a [Bert model fine tuned on **SQUAD v2**](https://huggingface.co/deepset/bert-base-cased-squad2) and then I fine tuned it on **QNLI** using **compression** (with a constant replacing rate) as proposed in **BERT-of-Theseus**
9
+
10
+ ## Details of the downstream task (QNLI):
11
+
12
+ ### Getting the dataset
13
+ ```bash
14
+ wget https://raw.githubusercontent.com/rhythmcao/QNLI/master/data/QNLI/train.tsv
15
+ wget https://raw.githubusercontent.com/rhythmcao/QNLI/master/data/QNLI/test.tsv
16
+ wget https://raw.githubusercontent.com/rhythmcao/QNLI/master/data/QNLI/dev.tsv
17
+
18
+ mkdir QNLI_dataset
19
+ mv *.tsv QNLI_dataset
20
+ ```
21
+
22
+ ### Model training
23
+
24
+ The model was trained on a Tesla P100 GPU and 25GB of RAM with the following command:
25
+
26
+ ```bash
27
+ !python /content/BERT-of-Theseus/run_glue.py \
28
+ --model_name_or_path deepset/bert-base-cased-squad2 \
29
+ --task_name qnli \
30
+ --do_train \
31
+ --do_eval \
32
+ --do_lower_case \
33
+ --data_dir /content/QNLI_dataset \
34
+ --max_seq_length 128 \
35
+ --per_gpu_train_batch_size 32 \
36
+ --per_gpu_eval_batch_size 32 \
37
+ --learning_rate 2e-5 \
38
+ --save_steps 2000 \
39
+ --num_train_epochs 50 \
40
+ --output_dir /content/ouput_dir \
41
+ --evaluate_during_training \
42
+ --replacing_rate 0.7 \
43
+ --steps_for_replacing 2500
44
+ ```
45
+
46
+ ## Metrics:
47
+
48
+ | Model | Accuracy |
49
+ |-----------------|------|
50
+ | BERT-base | 91.2 |
51
+ | BERT-of-Theseus | 88.8 |
52
+ | [bert-uncased-finetuned-qnli](https://huggingface.co/mrm8488/bert-uncased-finetuned-qnli) | 87.2
53
+ | DistillBERT | 85.3 |
54
+
55
+
56
+
57
+
58
+ > [See all my models](https://huggingface.co/models?search=mrm8488)
59
+
60
+ > Created by [Manuel Romero/@mrm8488](https://twitter.com/mrm8488)
61
+
62
+ > Made with <span style="color: #e25555;">&hearts;</span> in Spain