This page is in the 'old' namespace, and was imported from our previous wiki. We recommend checking for more up-to-date information using the search box.
Some notes about the code for incremental building.
archiveinf-src.gdb - is a reverse lookup between import file and oid that it gets used for. One file may end up with several oids, eg if its a database file of many records.
archiveinf-doc.gdb stores a list of oids. For each one, it stores the following
[HASHe407a49647d425a685f959] <sort-meta> <doc-file>HASHe407.dir/doc.xml <src-file>/research/kjdon/home/gsdl/collect/incre4/import/1-1.jpeg <assoc-file> <meta-file> <index-status>I
src-file is the primary import document that generated this greenstone document. doc-file is the file in archives sort-meta is a metadata for sorting the build (if -sortmeta was specified for import)
assoc-file and meta-file store auxiliary files that were used for this document, eg txt files for a pagedimage, OAI metadata file.
For a metadata file to be listed in these databases, then an entry in extrametafile must be added.
$extrametafile->{$filename_for_metadata}->{$file} = $filename_full_path;
$filename_for_metadata is the primary import file, $file is a short version of the meta file, and $filename_full_path is the full path to the metadata file.
Note, metadata.xml files and metadatacsv files do not do this. Instead, during the file_block_read, they are marked as metadata files:
$block_hash->{'metadata_files'}->{$filename_full_path} = 1;
It seems that metadata_read is always carried out, so if a doc is changed it will always get metadata. If a metadata files is changed, then the whole folder must be reimported, as we don't know if new docs have metadata added to that file.
Because OAI files currently only give metadata to one document, I am assuming that the document that it refers to will never change, and so we do a lookup for the file it belongs to, like for associated files. We don't reimport the whole folder.
Plugins need to call associate_source_file for any auxiliary files in import that they are using (eg images in HTML page, image and text files in pagedimage etc). This is to record these files as associated files and this info will go into the two databases.
HTMLPlugin and PagedImagePlugin are sometimes used to process converted files - eg PDF converted to HTML/item, then processed by HTML/PagedImage plugin. In this case, we do not want to record the auxiliary files as they are in temp dir and will not be modified directly. When loading up the secondary plugin, the main plugin must set -tmp_file_process option, and these plugins will not store these auxiliary files.
If a doc or one of its associated files change, then the doc is marked for reimporting, and the old record is marked for deletion (the doc may end up with a new id after reimporting.)
Some annoying/leftover things: