Improving Natural Language Person Description Search from Videos with Language Model Fine-Tuning and Approximate Nearest Neighbor
4
item.page.dc.date.issued
2022-12-01
item.page.dc.type
item.page.dc.identifier.eissn
25042289
item.page.dc.identifier.doi
item.page.dc.identifier.scopus
2-s2.0-85144601088
item.page.oaire.citation.title
Big Data and Cognitive Computing
item.page.oaire.citation.volume
6
item.page.oaire.citation.issue
4
item.page.dc.rights.holder
SCOPUS
item.page.dc.identifier.citation
Big Data and Cognitive Computing Vol.6 No.4 (2022)
punsarn.item.page.citation
Yuenyong S., Wongpatikaseree K. Improving Natural Language Person Description Search from Videos with Language Model Fine-Tuning and Approximate Nearest Neighbor. Big Data and Cognitive Computing Vol.6 No.4 (2022). doi:10.3390/bdcc6040136 Retrieved from: https://repository.li.mahidol.ac.th/handle/123456789/83958
item.page.dc.title
Improving Natural Language Person Description Search from Videos with Language Model Fine-Tuning and Approximate Nearest Neighbor
item.page.dc.contributor.author
item.page.oairecerif.author.affiliation
item.page.dc.contributor.other
item.page.dc.description.abstract
Due to the ubiquitous nature of CCTV cameras that record continuously, there is a large amount of video data that are unstructured. Often, when these recordings have to be reviewed, it is to look for a specific person that fits a certain description. Currently, this is achieved by manual inspection of the videos, which is both time-consuming and labor-intensive. While person description search is not a new topic, in this work, we made two contributions. First, we improve upon the existing state-of-the-art by proposing unsupervised finetuning on the language model that forms a main part of the text branch of person description search models. This led to higher recall values on the standard dataset. The second contribution is that we engineered a complete pipeline from video files to fast searchable objects. Due to the use of an approximate nearest neighbor search and some model optimizations, a person description search can be performed such that the result is available immediately when deployed on a standard PC with no GPU, allowing an interactive search. We demonstrated the effectiveness of the system on new data and showed that most people in the videos can be successfully discovered by the search.
