We plan to design, develop and test unsupervised or weakly supervised learning methods that will allow us to carry out large-scale automatic searches adapted to tillustrated scientific corpora from the Middle Ages and the modern period. These methods will be based on the detection of iconographic similarities (between images, in order to identify the copying and borrowing processes, in particular) and textual similarities (between images and captions or associated texts, in order to identify, e.g., different images describing similar textual content). This approach will allow a strong integration of the intertextual and intericonic dimensions in historical study of scientific illustration.
We present hereafter some first particularly encouraging results obtained with an automatic extraction and alignment algorithm applied to two first sets of illustrations, respectively from two of the four project corpora :
- the Physiologus, written around the 2nd century AD in Alexandria and preserved in about 80 medieval manuscripts (13 illustrated). The tests were carried out mainly on the following manuscripts: Venezia, Bibl. Naz. Marciana, gr. IV. 35 (coll. 1383); Vaticano, BAV, Barb. gr. 438].
- the De materia medica of Dioscoride composed around 77 AD and preserved in 65 manuscripts, made between the 6th and 16th centuries. It is a pharmacological treatise containing plants, minerals and animals that were used in the preparation of remedies. The tests were carried out mainly on the following manuscripts: Wien, Österreichische Nationalbibliothek, Med. gr. 1; New York, Pierpont Morgan Library, Ms. M 652; Vaticano, BAV, Chigianus 53 (F.VII.159).
The algorithm developed by the Imagine team of the LIGM (ENPC) enables to extract images from each manuscript and then to apply to these extracted images an alignment method based on a self-supervised similarities computing method (see http://imagine.enpc.fr/~shenx/ArtMiner/), pre-trained on other data, and on an alignment optimization method.
As can be seen from the two examples below, the results obtained on the two manuscripts of the Physiologus (for more results, see http://imagine.enpc.fr/~kaouar/html/P2_P4_best_matches.html) and on the manuscripts of the De materia medica with more complex iconography (for more results, see http://imagine.enpc.fr/~kaouar/html/D3_D4_best_matches.html) are very conclusive and encouraging.
These results are a first step towards the development of self-supervised or weakly supervised learning methods able to automatically perform alignments on large corpora between images with few common features. This next step will allow the identification of new alignments and new iconographic series, thus renewing in depth the classical means of analysis of iconographic representations of scientific knowledge and its circulation in history.
Extracts of results on two Physiologus manuscripts
Extracts of results on two De Materia Medica manuscripts
Kaoua, R., Shen, X., Durr, A., Lazaris, S., Picard, D. and Aubry, M. (2021). “Image Collation: Matching illustrations in manuscripts”, International Conference on Document Analysis and Recognition (ICDAR) [More information]