Given the basic data and query model, a user query can then be processed by undergoing three steps. In vitrivr, each of these steps is handled by a different component.
Query Formulation: Users of a retrieval system must be able to express their search intent. Often, these are just textual inputs (e.g., keywords). However, in multimedia retrieval, other modes of operation are possible. Examples include (but are in no way limited) to query by example (e.g., an example image), query by sketch (e.g., a sketched input of a 2D image) or even more exotic modes, such as, query by humming (e.g., to search for a familiar song). This formulation step is handled by the different vitrivr user-interfaces – a web-based one and a VR search system.
Query Translation: The query input created by the user must then be translated into a feature representation that can be processed. This is the conversion is handled by the middleware Cineast, which delegates this to one of the many different feature modules that know, for a particular implementation, how this translation takes place. Cineast (see ) then uses the resulting representation(s) to orchestrate the database lookup.
Database Lookup: Once a representation has been obtained, the nearest neighbour search must take place in the data collection. In vitrivr, we use a specialized database called Cottontail DB (see ), that is geared towards these types of proximity-based searches. The search can either be executed in a brute-force fashion, by obtaining the distance between query and all the vectors in the database or using specialized index structures.
The resulting items are then ranked and sent back to the user interface for presentation. Since vitrivr has been built to be modular, any of the components can of course be adapted and replaced, which is an advantage for the XReco project, which will rely on its own user interface.