Add to MSMarco-Doc/CovidQA Replication Log

[justinborromeo]

CovidQA

Able to replicate CovidQA experiment with identical T5 results but different BM25 results.

BM25 NL Question

precision@1	0.15384615384615385
recall@3	0.21865889212827985
recall@50	0.7208778749595076
recall@1000	0.7582928409459021
mrr	0.25329970378011524
mrr@10	0.23344131303314977

BM25 Keyword Query

precision@1	0.15384615384615385
recall@3	0.21865889212827985
recall@50	0.7208778749595076
recall@1000	0.7582928409459021
mrr	0.25441237140238665
mrr@10	0.23493413238311195

MSMarco-Document

Able to replicate MSMarco-Document experiment with identical results to doc.

After 1st Half

2020-09-08 03:27:22 [INFO] evaluate_document_ranker: precision@1 0.16
2020-09-08 03:27:22 [INFO] evaluate_document_ranker: recall@3    0.44
2020-09-08 03:27:22 [INFO] evaluate_document_ranker: recall@50   0.84
2020-09-08 03:27:22 [INFO] evaluate_document_ranker: recall@1000 0.88
2020-09-08 03:27:22 [INFO] evaluate_document_ranker: mrr         0.33663
2020-09-08 03:27:22 [INFO] evaluate_document_ranker: mrr@10      0.33171

After 2nd Half

2020-09-09 15:38:46 [INFO] evaluate_document_ranker: precision@1 0.24
2020-09-09 15:38:46 [INFO] evaluate_document_ranker: recall@3    0.32
2020-09-09 15:38:46 [INFO] evaluate_document_ranker: recall@50   0.76
2020-09-09 15:38:46 [INFO] evaluate_document_ranker: recall@1000 0.88
2020-09-09 15:38:46 [INFO] evaluate_document_ranker: mrr         0.31052
2020-09-09 15:38:46 [INFO] evaluate_document_ranker: mrr@10      0.29133

Hardware:

GeForce GTX 960M
Intel i7 6700

Software:

Python 3.7
PyTorch 1.6.0
CUDA Toolkit 11.0
Nvidia Driver 450

[ronakice]

#78 seems to have run into BM25 replication issues, did you?

[justinborromeo]

Actually, yeah. I didn't notice because I was focussing on the T5 numbers but yeah, my BM25 numbers align with what Yuxuan got.

[justinborromeo]

Updated to reflect the differences in CovidQA and add msmarco-doc replication log entry. Should I be adding to the CovidQA replication log if the BM25 results were different?

[ronakice]

LGTM thanks Justin!