Replication experiement results for CovidQA, MSMARCO document and MSMARCO subset

[jhuang265]

Colab Environment:
OS: Ubuntu-18.04.3 LTS
Java: 11.0.8
Python: 3.6.9
GPU: Tesla P100

CovidQA

With Random

NL Question:

❯ python3 -um pygaggle.run.evaluate_kaggle_highlighter --method random \
                                                    --dataset data/kaggle-lit-review-0.2.json \
                                                    --index-dir indexes/lucene-index-cord19-paragraph-2020-05-12
precision@1     0.0
recall@3        0.0199546485260771
recall@50       0.3247165532879819
recall@1000     1.0
mrr             0.03999734528458418
mrr@10          0.020888672929489253

Keyword Query:

❯ python3 -um pygaggle.run.evaluate_kaggle_highlighter --method random \
                                                     --split kq \
                                                     --dataset data/kaggle-lit-review-0.2.json \
                                                     --index-dir indexes/lucene-index-cord19-paragraph-2020-05-12
precision@1     0.0
recall@3        0.0199546485260771
recall@50       0.3247165532879819
recall@1000     1.0
mrr             0.03999734528458418
mrr@10          0.020888672929489253

With BM25:

NL Question:

❯ python3 -um pygaggle.run.evaluate_kaggle_highlighter --method bm25 \
                                                    --dataset data/kaggle-lit-review-0.2.json \
                                                    --index-dir indexes/lucene-index-cord19-paragraph-2020-05-12
precision@1    0.15384615384615385
recall@3       0.21865889212827985
recall@50      0.7208778749595076
recall@1000    0.7582928409459021
mrr            0.25329970378011524
mrr@10         0.23344131303314977

Keyword Query:

❯ python3 -um pygaggle.run.evaluate_kaggle_highlighter --method bm25 \
                                                     --split kq \
                                                     --dataset data/kaggle-lit-review-0.2.json \
                                                     --index-dir indexes/lucene-index-cord19-paragraph-2020-05-12
precision@1    0.15384615384615385
recall@3       0.21865889212827985
recall@50      0.7208778749595076
recall@1000    0.7582928409459021
mrr            0.25441237140238665
mrr@10         0.23493413238311195

With MonoT5:

NL Question:

❯ python3 -um pygaggle.run.evaluate_kaggle_highlighter --method t5 \
                                                    --dataset data/kaggle-lit-review-0.2.json \
                                                    --index-dir indexes/lucene-index-cord19-paragraph-2020-05-12
precision@1     0.2789115646258503
recall@3        0.41854551344347257
recall@50       0.92555879494655
recall@1000     1.0
mrr             0.417982565405279
mrr@10          0.4045405463772811

Keyword Query:

❯ python -um pygaggle.run.evaluate_kaggle_highlighter --method t5 \
                                                     --split kq \
                                                     --dataset data/kaggle-lit-review-0.2.json \
                                                     --index-dir indexes/lucene-index-cord19-paragraph-2020-05-12
precision@1     0.24489795918367346
recall@3        0.38566569484936825
recall@50       0.9231778425655977
recall@1000     1.0
mrr             0.37988285486956513
mrr@10          0.3671336788683727

MSMARCO Document

First Half:

❯ python -um pygaggle.run.evaluate_document_ranker --split dev \
                                                --method t5 \
                                                --model castorini/monot5-base-msmarco \
                                                --dataset data/msmarco_doc_ans_small/fh \
                                                --model-type t5-base \
                                                --task msmarco \
                                                --index-dir indexes/index-msmarco-doc-20200527-a1ecfa \
                                                --batch-size 32 \
                                                --output-file runs/run.monot5.doc_fh.dev.tsv
precision@1 0.16
recall@3  0.44
recall@50  0.84
recall@1000 0.88
mrr     0.33663
mrr@10   0.33171

Second Half:

❯ python -um pygaggle.run.evaluate_document_ranker --split dev \
                                                --method t5 \
                                                --model castorini/monot5-base-msmarco \
                                                --dataset data/msmarco_doc_ans_small/sh \
                                                --model-type t5-base \
                                                --task msmarco \
                                                --index-dir indexes/index-msmarco-doc-20200527-a1ecfa \
                                                --batch-size 32 \
                                                --output-file runs/run.monot5.doc_sh.dev.tsv
precision@1 0.24
recall@3  0.32
recall@50  0.76
recall@1000 0.88
mrr     0.31052
mrr@10   0.29133

MSMARCO Passage Subset:

With monoBERT:

❯ python -um pygaggle.run.evaluate_passage_ranker --split dev \
                                                --method seq_class_transformer \
                                                --model castorini/monobert-large-msmarco \
                                                --dataset data/msmarco_ans_small/ \
                                                --index-dir indexes/index-msmarco-passage-20191117-0ed488 \
                                                --task msmarco \
                                                --output-file runs/run.monobert.ans_small.dev.tsv
precision@1     0.2761904761904762
recall@3        0.42698412698412697
recall@50       0.8174603174603176
recall@1000     0.8476190476190476
mrr     0.41089693612003686
mrr@10  0.4026795162509449

With monoT5:

❯ python -um pygaggle.run.evaluate_passage_ranker --split dev \
                                                --method t5 \
                                                --model castorini/monot5-base-msmarco \
                                                --dataset data/msmarco_ans_small \
                                                --model-type t5-base \
                                                --task msmarco \
                                                --index-dir indexes/index-msmarco-passage-20191117-0ed488 \
                                                --batch-size 32 \
                                                --output-file runs/run.monot5.ans_small.dev.tsv
precision@1     0.26666666666666666
recall@3        0.4603174603174603
recall@50       0.8063492063492063
recall@1000     0.8476190476190476
mrr     0.3973368360121561
mrr@10  0.39044217687074834

Comments

No issues were encountered, but for the CovidQA dataset, it might be beneficial to add a section on the data preparation. I figured out how to get the indexes/lucene-index-cord19-paragraph-2020-05-12 file, but it required some poking around the project which could probably be summarized quite quickly.

I'll also try replicating the entire dev set for MS MARCO Passage, but since it needs to run for such a long time I'll try to fit that in later.