Submission Guideline

Please submit your runs and be the top performer!

Training

In the NTCIR-12 MobileClick-2, we provide training data that were extracted from the NTCIR 1CLICK-2 test collection. Please go to the data page after log in.

Currently, you may upload iUnit ranking subtask runs for the training data, and share your scores with the other participants.

Steps

  1. Log in (or sign up)
  2. Obtain training data
  3. Develop an iUnit ranking system after reading task specification
  4. Generate run files that satisfy the output format in task specification
  5. Upload your runs at new submission page

Rules

Note that these rules are applied when you are submitting runs for the training queries (i.e. 1C2-*-queries.tsv)

  • Please do not manually make a run
  • You can utilize any data for making iUnit ranking systems, but do not use iUnit importance ground truth data (i.e. 1C2-*-weights.tsv) (of course, you may use it for self evaluation or training for test runs)
  • Do not redistribute provided training data

Test

Only runs for the test data will be recorded as official results. While there is no restriction to submit runs for the training data, there are some rules for submitting runs for the test data to avoid over-tuning.

Steps

  1. Log in (or sign up)
  2. Obtain test data
  3. Develop iUnit ranking and summarization systems after reading task specification
  4. Tune your systems with the training data
  5. Upload your runs at new submission page

Rules

Note that these rules are applied when you are submitting runs for the test queries (i.e. MC2-*-queries.tsv)

  • Runs can be submitted at weekly intervals for avoiding heavy tuning on the test data
    • Your team cannot submit the same type of runs without a week interval, but can submit different types of runs (e.g. English iUnit ranking and English iUnit summarization).
    • Organizers will send you "NTCIR-12 MobileClick-2 Group Registration" E-mail after registration at the NTCIR website, and ask you to link your account with your group. Only user accounts who belong to authorized teams can submit runs.
  • Please do not manually make a run
  • You can utilize any data for developing iUnit ranking and summarization systems
  • Do not redistribute provided test data