A person searching a digital collection may know what an object looks like without knowing its catalogue terminology. They may also need results from a particular period or with a particular documented relationship. Combining these needs is the focus of REEVALUATE-listed research on multimodal retrieval.
Connecting visual search with structured questions
The ESWC 2026 publication presents a system that combines an adapted CLIP model, which connects images and language, with Text2SPARQL, which translates natural-language requests into queries over a knowledge graph. A fusion method brings the results together.
This allows visual information and structured knowledge to contribute to the same search process. An illustrative request might combine a visual description with a date constraint, rather than requiring the user to run unrelated searches.
Testing the approach in practice
The researchers integrated the system into a cultural heritage platform through a backend API. Their evaluation combined quantitative measures with qualitative studies involving museum curators.
The reported results indicate strong retrieval performance from the adapted visual model, with knowledge-based querying further improving precision. The paper also discusses limitations encountered when bringing these methods into an operational environment.
Why this matters for REEVALUATE
For a curator, researcher or creative professional, discovery is often the first step towards a new use of a collection. Search interfaces therefore need to accommodate the ways people describe their interests.
The value of this research lies in exploring a more flexible route into collections while keeping documented relationships available to the search process. A useful result should help someone find an object and continue investigating why it is relevant.
Read the publication
Knowledge-Enhanced Multimodal Retrieval over Cultural Heritage Knowledge Graphs
Xuemin Duan; Federico D’Asaro; Ruben Peeters; Giacomo Blanco; Tommaso Monopoli; Giuseppe Rizzo; Anastasia Dimou
ESWC 2026, The Semantic Web, Springer, 2026.
REEVALUATE is funded by the European Union under Grant Agreement No. 101132389. Views and opinions expressed are those of the authors and do not necessarily reflect those of the European Union or the European Research Executive Agency. Neither the European Union nor the granting authority can be held responsible for them.





