The steadily increasing amount of multimedia data poses challenging questions on how to index, visualise, organise, navigate, or structure multimedia information. The benefit of different automated approaches is often not clear, as usually they are evaluated on different datasets with different performance measures. Evaluation campaigns aim at establishing an objective comparison between the performance of different approaches by posing well-defined tasks including datasets, topics, and measures.
The Visual Concept Detection and Annotation Task (VCDT) was part of the evaluation initiative »ImageCLEF« and focuses on the evaluation of multi-modal image annotation approaches in consumer photos with respect to user requirements on photo collection organisation. The task challenges the participants to deal with an unbalanced number of annotations per image, an unbalanced number of images per concept, the subjectivity of concepts such as boring, cute, or fancy, and the diversity of images belonging to the same concept. Additionally, textual and multi-modal runs have to cope with images lacking EXIF data and/or Flickr user tags. The VCDT was organised within the research programme THESEUS in the ImageCLEF evaluation cycles of 2009, 2010, and 2011, and in the ImageCLEF@ICPR 2010 contest by Fraunhofer IDMT.
The test collections of the different VCDT cycles are now available for download for research purposes. The photos used in the benchmark are a subset of the MIRFlickr collection which comprises 25,000 photos from Flickr.com that were published under the Creative Commons license. The creator information as well as the exact license type and image title is distributed together with the images. Further, the dataset includes Flickr user tags and EXIF information. Many thanks to the University of Leiden for collecting and providing this dataset. The set used for the VCDT in the ImageCLEF cycles contains 18,000 Flickr photos in total. In all evaluation cycles, the VCDT test collection was fully assessed with relevance judgements for the annotation task. The assignment of the ImageCLEF IDs to the MIRFlickr IDs can be found here.
If you use the VCDT test collections in your work, please cite the overview paper of the corresponding evaluation cycle.