The Challenge Problem for Automated Detection of 101 Semantic Concepts in Multimedia

We introduce the challenge problem for generic video indexing to gain
insight in intermediate steps that affect performance of multimedia
analysis methods, while at the same time fostering repeatability of
experiments. To arrive at a challenge problem, we provide a general
scheme for the systematic examination of automated concept detection
methods, by decomposing the generic video indexing problem into 2
unimodal analysis experiments, 2 multimodal analysis experiments, and 1
combined analysis experiment. For each experiment, we evaluate generic
video indexing performance on 85 hours of international broadcast news
data, from the TRECVID 2005/2006 benchmark, using a lexicon of 101
semantic concepts. By establishing a minimum performance on each
experiment, the challenge problem allows for component-based
optimization of the generic indexing issue, while simultaneously
offering other researchers a reference for comparison during indexing
methodology development. To stimulate further investigations in
intermediate analysis steps that influence video indexing performance,
the challenge offers to the research community a manually annotated
concept lexicon, pre-computed low-level multimedia features, trained
classifier models, and five experiments together with baseline
performance, which are all available at
http://www.mediamill.nl/challenge/.