Institute for Informatics and Telematics
Italian National Research Council (CNR)
ArchEle
An acquisition, search and retrieval system based on Zope/Plone
M. Andreini, C. Lucchesi, M. Martinelli, G. Vasarelli
EuroPython - 2004
Göteborg, June 7-9

Agenda

Electronic Archiving Systems: Requirements

Goals

Architecture

The acquisition phase

Scalability

Case Study

Future Developments

Electronic Archiving System: Requirements

An Archiving system should allow to:

insert and delete documents;

associate personalized metadata to stored documents;

validate the metadata insertion;

integrate metadata with existing archive;

search and retrieve the archived documents by means of predefined or personalized method;

insert, edit and retrieve documents by different people having different roles;

Web Electronic Archiving system

Electronic Archiving through the Web is:

platform independent;

no need of further applications in order to insert, search and retrieve documents;

accessible through any Web browser;

based on standard protocols and tecnologies (HTTP, HTML, CSS, XML, PNG, etc);

easiness to use.

ArchEle Goals

ArchEle is born as a system for document archiving through the Web that allows to:

acquire paper documents by means of one or more computer equipped with scanners, and to catalogue them;

catalogue electronic documents (PDF, MsWord,...);

define the metadata associated to the documents;

define the integration between metadata and external data;

obtain a good scalability even for millions of documents.

General architecture

Python + C

ZOPE

ArchEle

Plone

Archetypes

Isaac

Application

Framework

XHTML/CSS

Interfaces

Types

Definition

Documents

Management

SQL

FileSystem

Documents

Management

ArchEle Overview

Document Repository

Archele

Application

Server

WEB

Metadati

+

Metadata

PNG

PNG

insert/edit

search/retrieve

Interaction with the scanner

How can Plone interact with the scanner?

We need:

an acquisition client to:

acquire the images by means of scanners;

send them to the server.

some methods server-side to:

run the acquisition;

fetch the data sent by the acquisition client and store them into the Document Repository.

Client side: the Isaac client

The Isaac client:

is written in Python;

uses wxPython for the graphic interface;

currently uses the Python Twain Module to interact with the scanner (Windows only) but, anyway it could be easily replaced with SANE under Linux;

has been compiled using py2exe to produce a small binary.

How the acquisition phase works

ArchEle Client

HTTP/HTTPS

TWAIN

Archele Application Server

1

1. Acquisition Request

2

2. Response with application/x-isaac

3

3. Start of ArchEle Client

4

4. Sending request of acquisition

5

5. Client receive the images

6

6. Processing of the images (BMP-PNG)

7

7. Authenticated sending of the images

Server Side developments

A Zope product has been created to contain the needed skin and other parts of code;

Use ExtImage/ExtFile to store the document/images acquired by the scanners;

Use BTreeFolder to manage large number of objects in the same folder.

System scalability (1)

What about 1 million of documents to be acquired?

160KB for each archetype object insertion;

ZODB grows by up to 250MB per day (with ~1200 objects/day);

archetypes objects are about 6KB in packed size, of which only a small part, around 300 bytes, are relevant metadata;

using the SQLStorage the object size does not change significantly.

System scalability (2)

Steady increase of the ZODB

Main problems

considerable load due to elaboration of a large number of objects during each operation;

this led to long waiting-periods of the operators;

the server load average was always high (~3);

we realized that the elaboration time depended on the increase of the ZODB objects number.

New objects creation time

grafico del creation time/ZODB object

Obtaining the scalability (1)

Our Requirement:

keep the code compatible with archetypes and thus to follow future developments;

How SQL Storage works

UID is the key of the table containing the values of all sql fields;

for each object the key (UID) is stored in the ZODB;

Obtaining the scalability 2

For a large collection of document our solution is:

use a FactoryObject for each collection to manage the objects (like in the FlyWeight pattern);

redefine the UID method to maintain the association between the instances and the rows in the tables, so it can depend directly on the session (cookie);

create a new Archetypes Storage based on PostgreSQLStorage to fit the necessary work.

Store the images

we've defined a wrapper class to store the objects directly in the filesystem;

the UID is used to obtain the association between the filesystem and the objects instances in Zope;

the storing of these images is still usable - with limitations – with ZEO and NFS.

Case study

ArchEle has been successfully used at the Italian Registry of the “.it” ccTLD, managed by IIT-CNR.

Using this application were filed:

200000 faxes (Changes of Provider/Maintainer) with 180000 associated images for a total of 18GB

213000 Letters of domain requests with 503000 associated images for a total of 28GB

It's estimated that soon will be filed:

about 400000 Changes of Provider for a total of ~35GB of images

about 700000 Letters of domain requests for a total of ~92GB of images

Future developments

Client side:

digital signature to the acquired documents;

negotiation of the digital certificates to permit accreditation of the users;

Isaac Client for Linux, Mac OS X.

Server side:

move to another, more object oriented, storage: APE, ...;

assign a unique and official protocol number to be compliant with the Public Administration Information Protocol.

Thanks for your attention

Good morning. My name is Cristian Lucchesi and i'm going to illustrate a work I did in co-authoring with my colleagues Marco Andreini, Giuseppe Vasarelli and Maurizio Martinelli. We work at the Institute for Informatics and Telematics of The Italian National Research Council. The work I’m going to present now is about an acquisition, search and retrieval system based on Zope /Plone. We named it ArchEle which stands for Electronic Archiving.

The agenda of what we will see is:

The requirements of an electronic archiving system

Goals of these systems

The general architecture of our solution

The acquisition phase

The scalability and main problems

A case study

Future Developments and improvements

First of all we'll look at what an Electronic Archiving system should allow to do:

insert and delete documents;

associate personalized metadata to stored documents;

integrate metadata with existing archives;

search and retrieve the archived documents by means of predefined or personalized methods;

insert, edit and retrieve documents by different people having different roles;

In addition an Electronic Archiving system may be used through the Web. Main features are:

platform independent, almost on the client side;

It doesn't need further applications in order to insert, search and retrieve documents;

It is accessible through any Web browser;

It is based on standard protocols and tecnologies (HTTP, HTML, CSS, XML, PNG, etc.)

It is easy to use

Let's see the main ArchEle goals.

When designing the system the following requirements were considered:

acquire paper documents by means of one or more scanner, an operation that should be integrated as much as possible with the insertion of metadata in the application; metadata could be for example the title, the authors, the date, etc of a document;

catalogue both paper documents and electronic documents such as PDF, MsWord, etc.;

personalize documents type, defining the metadata associated to the document itself;

if necessary, the system should be able to be integrated with other applications or external data sources;

obtain a good scalability, both for GBs of images and millions of documents;

obtain a good scalability both for insertion and retrieval in order to reduce the access time.

Everyone knowing Plone and hearing the above mentioned requirements and goals surely is thinking to it!. Yes we have used Plone!

We've used Zope 2.7 as framework, Plone 2.0 as layer for developing the application and Archetypes 1.3 for managing dynamics types.

This choices allow us to have:

the user and groups management with its Access Control List;

Template Attributes Language for dynamic interfaces;

multi-language interfaces;

integration with others data sources (RDMBS, Web services and so on);

type definition through Archetypes;.

This picture illustrates a general overview of our web acquisition system: people acquire the paper documents by means of a computer equipped with a scanner and store them in the Document Repository by means of the ArchEle Application Server. Data are sent through the Web. Operators that file the documents associate them with metadata useful for their retrieval. Data are acquired and converted to PNG format to permit the visualization of the images through the browser.

One or more persons, having the right permissions can search and retrieve the stored documents.

In order to acquire documents through the web from the scanner and send them to the server, a small software program was developed.

Some methods server-side were also created in order to run the acquisition, fetch data sent by the acquisition client, and store them into the Document Repository. Now we will see what the client is.

The client's name is Isaac.

The client is written in Python.

To develop the graphical part we adopted the framework wxPython which permits, with a bit of additional code, to build portable interfaces.

To acquire images from the scanner we used the Twain Module for Python which can be interfaced, in a Windows environment, with nearly all scanner types on the market but anyway it could be easily replaced with SANE under Linux;

We also took into consideration that the application could acquire fax or other paper documents by means of a scanner with Automatic Document Feeder.

In order to speed up the acquisition phase the client has been compiled with py2exe.

we would not be here

we would not be here

This picture shows how the acquisition phase works. Acquisition of images by scanner consists of the following steps:

the operator selects “acquire” or “save and acquire” in the web interface to start the acquisition;

the application provides an http response with an ad hoc Content-Type (i.e. (id est), application/x-isaac) containing the credentials of the current user, the required version of the client and the URI to which the acquired images should be sent;

thanks to the addition of an entry into the Windows Registry, the browser automatically runs the client, providing also a temporary file containing the ArchEle response;

the client ask to the scanner the acquisition of images;

using Twain, the client acquires the images from the scanner, available in Bitmap format;

the client converts the images from Bitmap to PNG (ping).

in the last phase the client send the PNG images to the URI provided by the application server. Together with the images it also sends the credentials that had been sent to it initially.

What about the server side?

A Zope product has been developed to contain:

methods for activating the acquisition;

methods for evaluating the client’s credentials – i.e. (id est) the authorizations;

Methods for the acquisition in the modification and content management template; the buttons “acquire” and “save and acquire” we talked about in the previous slide.

In order not to overload the ZODB, we used ExtFile to store the acquired images ;

To store a large number of objects in the same folder, BTreeFolder is surely a good choice.

What about 1 million of documents to be acquired?

We performed a series of surveys on various indicators:

Inserting each document increases the size of the ZODB by approximately 160 KB;

the ZODB grows by up to 250 MB per day (with 1 thousand two hundreds objects inserted per day, which makes a daily “pack” necessary;

the Archetypes objects – even those with small fields – are about 6 KB after the “pack”, of which only a small part (around 300 Bytes) are the metadata while the other ones are internal Archetypes structures;

using the SQLStorage the object size does not change significantly.

In this figure we show the steady increase of ZODB size (packed) according to the number of objects contained. From this, we can assume the future size of ZODB and therefore the load of the entire system.

In our system we had:

considerable load due to elaboration of a large number of objects during each operation;

this led to long waiting-periods of the operators;

the server load average was always high, around 3;

we realized that the elaboration time depended on the increase of the ZODB objects number.

The graphic represents the time for the creation of new objects and it is composed of three parts.

The first one, in red, is when the objects are stored in the ZODB. The creation time is:

less than 1 second –and therefore it's good: when the objects stored are less than 25000 (twenty ...);

more than 5 seconds – therefore it's ugly: when the objects are between 25000 and 130000.

The second part in orange is a transitional phase;

The third part in yellow is when the objects are more than 130000, and the documents are stored completely out of ZODB. In this case the creation time is less than 1 second.

We will explain how we've completed this task in the next slides.

To obtain the requested scalability we keep in mind to maintaining the code compatible with archetypes in order to follow its future developments.

In Archetypes all fields with SQL storage depend on the UID, that is an object method.

In practice, the UID is the key of a table containing the values of various fields with SQL storage. For each object the UID key stored in the ZODB.

In the Zope database there is only one Archetypes object for each type of document, utilized to manage the objects of its collection, like in the FlyWeight pattern.

A class was implemented to redefines the UID method, so that it can depend directly on the session (cookie), in order to select one of the elements of the collection.

A new storage system was also created starting from the PostgreSQLStorage to fit the necessary work.

A further step was the development of a class that permits storing directly in the filesystem using a variable method – usually the UID method – to obtain the physical path of the documents in the filesystem.

This class is implemented using the path.py module and re-uses many of the constructs already found in Zope.

The storage of these images is still usable in an architecture consisting of a ZEO server and many Zope instances, providing shared access to the same filesystem, we used NFS in our solution.

And now our real case study.

ArchEle has been successfully used at the Italian Registry of the “.it” ccTLD. managed by our institute.

Filed using this application were more than 200000 fax for Changes of Provider/Maintainer and more than 200000 Letters of Assumption of responsibility for a total of more than 35GB of images.

It's estimated that soon will be filled more than 1 million of documents for a total of more than 130GB of images.

Future developments.

On the client side:

adding the digital signature to the acquired documents using Isaac Client

adding the negotiation of the digital certificates to permit accreditation of the users

On the server side:

moving to another, more object oriented, storage: APE, ...

affixing an univocal protocol number introduced by the italian legislation for the information protocol in public administration.

ArchEle @ europython

2004-04-20T13:36:00