This page exposes the rules relative to upload of data in view of automatic evaluation, pertaining only to the 1st LPT and DA challenge data. For submission rules relative to the 2nd LPT and DA challenge, please visit its dedicated page.

Accessibility

  1. Data are available for download to anyone, with no need of account creation.
  2. Anyone can test their methods on the data and submit their results for automatic evaluation. Publication of the obtained results on the website is not mandatory.
  3. Submission of results first requires the creation of a user account, which must include the user affiliation (institution). Note that it is not allowed to register several times with different email addresses. Furthermore, the email address of your account should match that of your institution.
  4. In the result upload process, further information on the algorithm used to obtain the results need to be indicated: the algorithm's short-name / acronym, its full name, and, optionally, a web link to a publication or webpage describing it.
  5. Shortly after result upload, a summary of performance evaluation will be accessible to the user on a private page, whose link will be sent by email. The user will then be given the choice to include or not their performance charts in the general benchmark charts:
    • Upon dismissing, both the results and the performances will be deleted from the website and the servers
    • Upon acceptance, only the performances will be kept on the website

User data policy

The only user data that is kept on our website is the performance evaluation of a published result. We do not keep any of the uploaded data, and delete any performance that has not been published.

Submission rules

  1. In order to avoid excessive overfitting of methods to the computed performances, the number of submissions per user and per test case is limited to 1 every 15 days, irrespective of publication decision.
  2. As long as a user has not decided on accepting or discarding publication of their results, no further result submission is possible to the same data set, even after 15 days. However, the user still has the possibility to upload results to other data sets.
  3. If no decision about publication occurs following 30 days of a result upload, results and performance evaluation will be automatically deleted.

Publication rules

  1. A given algorithm may appear only once in the public result tables of a data set. In the case of a new publication of the same algorithm on the data set, results will be replaced.
  2. A user may publish the results of up to 5 different algorithms on a given data set.
  3. In case several publications have been done with too few differences in the algorithm name (i.e. differences only in parameter setting), you will be asked to indicate the only publication that should be considered in the charts.
  4. There is no frequency constraint pertaining to publication for a data set, other than that linked to submission of results (see "Submission rules").