NeoDescriber: An image-to-text model for automatic style description of neoclassical architecture

Wenke Qin , Lang Chen , Boyi Zhang , Weiya Chen , Hanbin Luo

Abstract

Architectural style description is important for the renewal and reuse of historical monuments. However, the identification of Neoclassical buildings is more difficult than that of other European architectural styles owing to the revival of classicism and its numerous variants. This paper proposes the image-to-text model NeoDescriber for the automatic identification and description of Neoclassical buildings based on the combination of dedicated visual classification and detection modules organized by expert knowledge in architecture. NeoDescriber takes building façade images as input and follows a coarse-to-fine description logic to generate structured annotations covering the whole building and key building elements. We built several building image datasets composed of more than 300 historical monuments to train and validate different visual modules. The results showed the acceptance performance of both classification and detection modules, with average precisions ranging from 73.6% to 95.2%. The generated text descriptions were then compared with results from BLIP-2 by qualitative and quantitative analyses. With human descriptions as a reference, the results showed that NeoDescriber’s generated descriptions were more precise and detailed, having a higher BERTScore for Neoclassical monuments. Thus, NeoDescriber could successfully recognize and describe Neoclassical buildings.

https://www.sciencedirect.com/science/article/pii/S0957417423012083