Visual Search Tool
The Computer Vision group develops automatic methods to process and evaluate large image collections, including a retrieval technique for objects, which was implemented in a publicly accessible visual search tool. The system was designed in collaboration with computer scientists and art historians to assist with art historical research and considered requirements imposed by humanities' research. The following aspects were relevant: user interaction, easy/comprehensive navigation, allow to evaluate image collections of different media and genre etc and search for multiple objects, thus enabling an iconographic analysis. Eventually, the group has applied this search engine to various image collections and demonstrated its efficiency.
Image displays a collection of still lives on the visual search tool
Interface for Visual Search
The system holds various datasets, which are uploaded by users. A visual search is performed within these collections based on single and multiple object queries (up to five), as marked by the user. A search process is triggered, where the underlying algorithm retrieves identical as well as similar regions to the query. Results are displayed in a new window with decreasing similarity. In order to refine the classifier and improve the precision of results, the group implemented a feedback system, where the user selects positive and negative retrievals. In contrast to other search tools, the presented system solely operates on visual input, which is especially useful when we work with unlabelled data.
The visual search tool and its function have been developed with regards to demands, imposed by humanities' research (tasks, usability etc). It has been successfully applied to diverse image corpora including portraits, medieval manuscripts, architectural drawings, paintings and photographs. These projects, some of which are described in the publications listed below, demonstrate the efficiency of said tool.
Get more information
More information on an unrestricted image search based on user-selected exemplars can be found here.
To get more information on the group's work in the digital humanities click here.
All Videos
Search example: the video illustrates a visual search using the tool; the user was interested in three image regions, which have been successfully found by the algorithm. Feedback after the first retrieval round retrained the model and led to better results in the second iteration.
Publications (selection)
Lang, Sabine and Ommer, Björn (2018): Attesting Similarity: Supporting the Organization and Study of Art Image Collection with Computer Vision. Digital Scholarship in the Humanities. Oxford, Oxford University Press, 33:4, pp. 845-856.
Lang, Sabine and Ommer, Björn (2018): Reconstructing Histories: Analyzing Exhibition Photographs with Computational Methods. Arts, Computational Aesthetics, 7:64, pp. 1-21.
Takami, Masato, Bell, Peter and Ommer, Björn (2014): An Approach to Large Scale Interactive Retrieval of Cultural Heritage. Eurographics Workshop on Graphics and Cultural Heritage. The Eurographics Association.