Downloading from the command line

Greenstone allows you to download files from the internet using a variety of protocols:

Web: downloads web pages and files via HTTP and FTP.
MediaWiki: downloads web pages and files via HTTP from a MediaWiki website.
OAI: downloads metadata records (and optionally documents) from an OAI-PMH (Open Archives Initiative) server.
Z39.50: downloads MARC records that match a particular search criterion from a Z39.50 server.
SRW/SRU: downloads MARCXML records that match a particular search criterion from a Search/Retrieve via URL (SRU) server.

This can be done from either the Download panel of the GLI, or directly from the command line.

Greenstone3

On Windows:

gs3-setup

On Linux/Mac:

source gs3-setup.bash

Greenstone2

On Windows:

setup

On Linux/Mac:

source setup.bash

GLI uses a perl script, downloadfrom.pl, to download files. This can be run on the command line, outside of GLI. The following options are available for all methods of download (Web, MediaWiki, OAI, Z3950, and SRW) using downloadfrom.pl:

Option	Description
`-download_mode <enum>`	(REQUIRED) The type of server to download from; allowable values: `Web`, `MediaWiki`, `OAI`, `Z3950`, and `SRW`
`-cache_dir <string>`	The location of the cache directory
`-gli`
`-info`	Print information about the server, rather than downloading
This information is also available from the command line: `perl -S downloadfrom.pl -h`

There are also several options available if you are using a proxy:

Option	Description
`-proxy_on`	Indicates you are using a proxy connection
`-proxy_host <string>`	Proxy host
`-proxy_port <string>`	Proxy port
`user_name <string>`	Proxy username
`user_password <string>`	Proxy password

Each download mode also has its own set of additional options, which are outlined on their respective pages in the documentation (Web, MediaWiki, OAI, Z39.50, SRW/SRU) . These options are the same as those available on the GLI Download panel and can also be viewed by running perl -S downloadinfo.pl <download-module>. For example, to get information and options for downloading via OAI protocol, you would run:

perl -S downloadinfo.pl OAIDownload

The download modules names are:

WebDownload
MediaWikiDownload
OAIDownload
Z3950Download
SRWDownload (for SRU/SRW downloads)

Once aware of all of the options, you are able to run the download script. An example download would be:

 perl -S downloadfrom.pl -document_mode OAI -url http://www.nzdl.org/cgi-bin/oaiserver.cgi -set demo -max_records 5

This will try to download 5 records from the set demo at the nzdl.org's OAI server.

The records (and optionally documents) will be downloaded into the folder the script is run from. To change this, use the -cache_dir <full-path-to-folder> option.

Additional Resources

There are several tutorials on downloading files using various protocols: