This version (2014/11/03 13:27) is a draft.
Approvals: 0/1

OAI

The Open Archives Initiative Protocol for Metadata Harvesting (OAI-PMH) allows for interoperability between document repositories.

  • Data Providers expose their metadata using OAI-PMH
  • Service Providers or harvesters access this metadata by making service requests.

Greenstone allows you to both harvest metadata provided by others and make the metadata for your own collections accessible via OAI-PMH.

Harvesting OAI records using Greenstone

Downloading records from an OAI repository

Greenstone can download records from an OAI repository and build them into a collection. The downloading can either be done from the Download panel of the GLI or from the command line. The following options are available for downloading via OAI:

OptionDescription
Source URL (-url <string>)(REQUIRED) OAI repository URL
Metadata prefix (-metadata_prefix <string>)The metadata format used in the exported metadata, e.g. oai_dc, qdc, etc. (Default: oai_dc)
Restrict to set (-set <string>) Restrict the download to the specified set in the repository
Get document (-get_doc)Download the source document if one is specified in the record
Only include file types (-get_doc_exts <string>)Permissible filename extensions of documents to get (Default: doc,pdf,ppt)
Max records (-max_records <int>)Maximum number of records to download. If not specified, will download all records.

In the GLI, clicking Server Information will cause the following request to be sent to the OAI data provider specified by the Source URL argument:

 <url>?verb=Identify

The response is shown in a popup window. You can use the returned server information to help fill out the arguments, for example, the set name and metadata prefix.

If you are using the GLI, you can view the downloaded files on the Gather panel. On the left-hand side of the panel, double click the Downloaded Files folder to expand its content. The subfolders are named by the OAI server URL. At the lowest level of each subfolder are the metadata files, which are organized by the specified set name. These metadata files are physically stored in a temporary cache directory. You can build a collection using these downloaded metadata files, using the OAIPlugin.

Downloading source documents

If you select the option to get document, then Greenstone will check the value of dc.identifier. If it is a URL (starts with http, https, ftp), then Greenstone will check whether the file extension matches those listed in the Only include file types argument. If so, the file is downloaded. If the extension does not match those listed and it is an HTML file, then Greenstone downloads the page and scans though it looking for href's that match the specified file extensions, and downloads these.

Serving OAI Data using Greenstone

Greenstone comes with a built-in OAI data provider, called oaiserver. A configuration file provides options for the set up of the server. Collections can opt in or out of the server, and each collection will be advertised as an OAI set. Multiple collections can be grouped into a single OAI set using Greenstone's OAI super set mechanism.

Testing

Once you have your OAI service in place, testing can be done via online validation facilities such as the following: http://www.openarchives.org/data/registerasprovider.html or http://re.cs.uct.ac.za/.

The former only verifies the Identify command, while extensive testing can be performed via the later one (called Repository Explorer).

The Greenstone OAI server must be publically accessible over the Internet to use these validation tools.

Additional Resources