On the usefullness of Scholarly Digital Editions

During a workshop at the University of Saint-Quentin-en-Yvelines on digital text editions, we had the pleasure to make a provocative pun. Scholarly Digital Editions, a concept largely defined by our friends James Cummings and Franz Fischer (cf. their “Criteria for Reviewing Scholarly Digital Editions” online : http://www.i-d-e.de/publikationen/weitereschriften/criteria-version-1-1/) and Elena Pierazzo (cf. her book DIgital Scholarly Editing: Theories, Models, Methods, available on HAL-SHS), are not only thought for the Humanists, but also for the machines. Indeed the better the edition is, the better it is to teach computer to read reading skills on medieval documents to beginners, i.e. Computers. We demonstrated that and how electronic editions can have a second life as a teacher for Artificial Intelligences, far from their original intent. In fact, our “ground-truth” is mostly based on the electronic edition by Ecole de Chartes of chancery documents published by Paul Guerin more than a hundred years ago: but to be good teachers, they need to be enhanced and upgraded to a more elaborate scholarly level.

Of course, this is not all. They also serve, along with indexes and inventories, to measure the accuracy and recall, and we all have to think about what it means to index (and not to edit) for the remnant parts and what conclusions we can draw from uncertain questions. That was the rest of our talk 😉

Old and new tools for retrieving information in a massive textual corpus

Seeking information in the registers of the French royal chancery was never an easy task. During the conference Administrer par l’écrit au Moyen Âge held at the IRHT in Paris on June 1st, 2016, our colleague Olivier Canteaut pointed out during our joint paper, it also applied to medieval researchers. Indeed, even in the Middle Ages, reading registers cover to cover was not excluded for retrieving a particular document because tables of contents and rubrics were more than often not sufficient to find the information that was needed. And to avoid such a tedious task, the king was sometimes gathering living witnesses of his predecessor’s act rather than searching for a copy in his registers! Still today, making research in those registers is daunting, especially for the time period following the reign of Jean II (1350-1364) where modern inventories are either partial or difficult to access.

A list is provided in the Virtual Inventory Room of the French National Archive [direct link to JJ inventories]

Screenshot of the list of inventories to access the JJ series in the French National Archive
Screenshot of the list of inventories to access the JJ series in the French National Archive

 

Therefore, as the main technological challenge of the HIMANIS project is to develop effective new tools for retrieving information in a massive document corpus like the chancery registers, how about merging ancient and new technologies?

Auto-indexation of the chancery register will provide keyword spotting across all the documents kept in the chancery registers. Combining this with all the existing materials (converted to electronic format), we aim at allowing simultaneous browsing between the auto-indexed digital images, document level segmentation, existing inventories, edited documents and indexed places names and persons.

IRHT's interface proposal to access the Chancery corpus: aligning and merging early modern inventories (tables of contents), 20th c. inventories and indexes, partial editions, and the result of automated indexing performed in the HIMANIS project
IRHT’s interface proposal to access the Chancery corpus: aligning and merging early modern inventories (tables of contents), 20th c. inventories and indexes, partial editions, and the result of automated indexing performed in the HIMANIS project

 

Text-Image Alignment: from alignment to enhancement

The first batch of our ground-truth and images has been processed. That are 489 pages, 7100 lines, 132’700 words! All on HD images provided by the French National Archive and based on the text edition by Paul Guérin. We now are able to see the text on the image :
Aligned text and image at word level on Paris, Archives Nationales, JJ 35, fol. 18v

From then on, we are now working on
1) the validation/correction of the alignment or segmentation… and here we have to acknowledge that our “ground-truth” was far from being 100% accurate (some links to images were wrong, some text paragraphs were omitted by Paul Guérin, and all sorts of funny discoveries)
2) the enhancement of the text edition, esp. by enriching it with the abbreviations as to create better models and a better ground-truth. For that, we have played with a little tool, which indicates which abbrevations are possible for a given word :

Web interface to select which abbreviations are present in a given word

Named entities and ground-truth: progress report

The research project started on November 1st, 2015. We are now describing the first steps undertaken during the first 3 months of the HIMANIS project are the following: the partners created the CHANCERY corpus (Archives nationales, JJ 35 to JJ 91) and associated the ground-truth with the images. They also acquired the necessary linguistic resources and formatted them in order to apply them to the automated indexing. At the same time, they created a dictionary of named entities with a dual objective: on the one hand, allow for a better indexing, and, on the other hand, provide a tool for accuracy measure. This dictionary is mainly based on the diverse regional systematic surveys : Gascogne

Languedoc and Rouergue

Berry, Blésois, Chartrain, Orléanais, Touraine

While working on these resources, we are moving from a linear and self-contained work and go to a direct index to spot on which page of which register, one can find a specific named entity.

The ground-truth is drawn from the large collection of Paul Guérin, Recueil des documents concernant le Poitou contenus dans les registres de la Chancellerie de France, 12 vol.

as it has been digitally edited by Ecole nationale des chartes:
http://corpus.enc.sorbonne.fr/actesroyauxdupoitou/tome1/

In parallel, they started the automated segmentation per line and words of their systems as well as the automated indexing, either based on first optical models or based on text-image alignment.

A great start

frchanjj_jj035_0006r_l_extr

Paris, Archives Nationales, JJ 35, fol. 5r (en ligne sur le site des Archives nationales). Reproduction by permission (see conditions on the website: http://www.archives-nationales.culture.gouv.fr/web/guest/mentions-legales)

This week, we had our kick-off meeting! It is a great fun to gather all the partners from the Netherlands, Spain and France.

This is the occasion to give some specific information about our corpus, aka “the Chancery corpus” or short “Chancery”. It already encompasses the high resolution digitized images of the registers from the French royal chancery, covering the period from 1211 to 1363, with the shelfmarks Paris, Archives nationales, JJ 7 to JJ 91. The full inventory is online : http://www.archivesnationales.culture.gouv.fr/chan/chan/fonds/EGF/SA/SAPDF/egfn_j.pdf and a fair part of this collection is already freely available to the public on the website of the French National Archive with more detailled descriptions for each volume from JJ 7 to JJ 56 (see http://www.culture.gouv.fr/documentation/archim/tresor-chartes.html).

For our team, the core corpus only covers the period 1302-1363 with the registers stricto sensu, but wouldn’t it be a pity to let the first part of the digitized material without indexation?

We are very happy to thank the French National Archive for its very generous and ongoing support. Not only they provide us with full resolution images for our scientific research, but they keep digitizing. In two years from now, the HIMANIS team and the Archives nationales will be glad to let everyone search directly in the text of this wonderful resource!

As for now, how should we start? Is there a better way than quoting the first charter of the first volume (JJ 35, fol. 5r), the reform order of 1303, whose preamble is all about starting a new way of life?

Nos, Philippus Dei gratia Francorum rex, pro reformatione regni nostri, quod retroactis temporibus gravatum extitit adversitatibus temporum et guerrarum ac pluribus aliis contrariosis eventibus, deliberatione provida fecimus aliquas ordinationes et statuta utilia et salubria pro gubernatione ac bono statu regni predicti, pro pace etiam ac transquillitate subjectorum nostrorum, sicut inferius continetur…

 

Himanis

Manuscripts are among the most important witnesses to our European shared cultural heritage. In recent years, large quantities of historical handwritten documents are being scanned and made available through web portals. Yet, the wealth of information conveyed by the text captured in these images remains largely inaccessible. General users and researchers more and more expect to query handwritten resources in plain text like printed books, but current handwritten text recognition technology is still far from offering sufficiently accurate transcripts on historical documents. This situation raises the need of searching approaches specifically designed for text images. HIMANIS aims at developing cost-effective solutions for querying large sets of handwritten document images. To this end, innovative keyword spotting, indexing and search methods will be developed, tested, adapted and/or scaled up to meet the real world conditions required. HIMANIS associate Computer Science (A2iA, UPVLC and RUG), Humanities (IRHT) and Cultural Heritage (The European Library, the Archives Nationales of France and Bibliothque Nationale de France) institutions and produce technology to generate new, research-based knowledge from historical manuscripts. As a challenging and particularly interesting case study, the large collection of registers produced by the French royal chancery (14th-15th c.) will be used as a test bed. This will allow sustainable use and management of this important cultural resource. Implementing user feedback will also increase our understanding of the meanings that it holds for people and how they perceive, use and interpret it, thereby meeting societal challenges in promoting its use and re-use. The expected outcomes of this project are: a) a new indexing/searching technology for historical manuscripts, brought to the market by A2iA; b) a new paradigm to study our historical heritage, as conveyed by manuscripts, by using full text search technology. c) a new vision of the raise of nation states in Europe via a new study of the corpus under this paradigm.

HIstorical MANuscript Indexing for user-controlled Search