Combinação de áudio e sequência de imagens para recuperação de momento de vídeo por linguagem natural

Abstract

The video moment retrieval with natural language area aims to locate the segment (moment) of the video most relevant to a textual description (natural language). However, existing methods are based only on the analysis of the image sequence and neglect the information derived from the audio. Thus, the main objective of this study is to combine both features to make the recovery more comprehensive and robust. For this, a model is built on audio and image sequence extractors aligned that relate to the textual description to recover the desired moment of the video. Results demonstrate that the proposed model outperforms the current state-of-the-art in the metric mloU by more than 27%, in addition to decreasing the response time of the video moment retrieval (reducing the computational complexity from polynomial to linear).