Not able to reproduce results

[ashim95]

First, thank you for releasing your code.

I am trying to reproduce results of your paper. I am running NLI_KD_training.py for MRPC with DEBUG=True.

The setting I am running is argv = get_predefine_argv('glue', 'MRPC', 'finetune_teacher').

After completing the training for 4 epochs, I get following results :

05/10/2020 19:09:30 - INFO - __main__ -   ***** Eval results *****
05/10/2020 19:09:30 - INFO - __main__ -     acc = 0.27942028985507245
05/10/2020 19:09:30 - INFO - __main__ -     acc_and_f1 = 0.13971014492753622
05/10/2020 19:09:30 - INFO - __main__ -     eval_loss = 3.8775325307139643
05/10/2020 19:09:30 - INFO - __main__ -     f1 = 0.0

Also the eval_log has the following :

epoch,acc,loss
1,0.8259803921568627,0.35975449818831223
2,0.8700980392156863,0.3205762528456174
3,0.8774509803921569,0.3944101127294394
4,0.8578431372549019,0.4749428268808585

-- which means training is probably correct but there is something wrong with test evaluation.

I have referred to the hyperparameter files that are provided in results_summary but I am not sure what might be wrong.