Differences

This shows you the differences between two versions of the page.

Link to this comparison view

en:user_advanced:command_line_download [2014/04/14 11:52] (current)
Line 1: Line 1:
 +====== Downloading from the command line======
  
 +Greenstone allows you to download files from the internet using a variety of protocols:
 +  * **[[en:​filetype:​web|Web]]:​** downloads web pages and files via HTTP and FTP.
 +  * **[[en:​filetype:​mediawiki|MediaWiki]]:​** downloads web pages and files via HTTP from a MediaWiki website. ​
 +  * **[[en:​user_advanced:​oai|OAI]]:​** downloads metadata records (and optionally documents) from an [[http://​www.openarchives.org/​pmh/​|OAI-PMH (Open Archives Initiative)]] server. ​
 +  * **[[en:​user_advanced:​z3950#​Download through Z39.50|Z39.50]]:​** downloads MARC records that match a particular search criterion from a Z39.50 server. ​
 +  * **[[en:​user_advanced:​z3950#​Download through SRU|SRW/​SRU]]:​** downloads MARCXML records that match a particular search criterion from a Search/​Retrieve via URL (SRU) server. ​
 +
 +This can be done from either the [[en:​gli:​download_panel|Download panel]] of the GLI, or directly from the command line. 
 +
 +
 +<TABAREA tabs="​Greenstone3,​Greenstone2">​
 +<TAB>
 +On Windows:
 +<​code>​
 +gs3-setup
 +</​code>​
 +\\
 +On Linux/Mac:
 +<​code>​
 +source gs3-setup.bash ​
 +</​code>​
 +</​TAB>​
 +<!-- GS2 -->
 +<TAB>
 +On Windows:
 +<​code>​
 +setup
 +</​code>​
 +\\
 +On Linux/Mac:
 +<​code>​
 +source setup.bash ​
 +</​code>​
 +</​TAB>​
 +</​TABAREA>​
 +
 +
 +GLI uses a perl script, **downloadfrom.pl**,​ to download files. This can be run on the command line,
 + ​outside of GLI. The following options are available for all 
 +methods of download (Web, MediaWiki, OAI, Z3950, and SRW) using ''​downloadfrom.pl'':​
 +
 +^Option^Description^
 +|''​-download_mode <​enum>''​|(REQUIRED) The type of server to download from; allowable values: ''​Web'',​ ''​MediaWiki'',​ ''​OAI'',​ ''​Z3950'',​ and ''​SRW''​|
 +|''​-cache_dir <​string>''​|The location of the cache directory|
 +|''​-gli''​| ​  |
 +|''​-info''​|Print information about the server, rather than downloading|
 +|//This information is also available from the command line: ''​perl -S downloadfrom.pl -h''//​||
 +
 +There are also several options available if you are using a proxy:
 +
 +^Option^Description^
 +|''​-proxy_on''​|Indicates you are using a proxy connection|
 +|''​-proxy_host <​string>''​|Proxy host|
 +|''​-proxy_port <​string>''​|Proxy port|
 +|''​user_name <​string>''​|Proxy username|
 +|''​user_password <​string>''​|Proxy password|
 +
 +Each download mode also has its own set of additional options, which are outlined on 
 +their respective pages in the documentation ([[en:​filetype:​web|Web]],​ [[en:​filetype:​mediawiki|MediaWiki]],​ [[en:​user_advanced:​oai|OAI]],​ [[en:​user_advanced:​z3950#​Download through Z39.50|Z39.50]],​ [[en:​user_advanced:​z3950#​Download through SRU|SRW/​SRU]]) . These options are the same as 
 +those available on the GLI Download panel and can also be viewed by running ​
 +''​perl -S downloadinfo.pl <​download-module>''​. For example, to get information ​
 +and options for downloading via OAI protocol, you would run:
 +<​code>​
 +perl -S downloadinfo.pl OAIDownload
 +</​code>​
 +
 +The download modules names are:
 +  * WebDownload
 +  * MediaWikiDownload
 +  * OAIDownload
 +  * Z3950Download
 +  * SRWDownload (//for SRU/SRW downloads//​)
 +
 +Once aware of all of the options, you are able to run the download script. An example download would be:
 +<​code>​
 + perl -S downloadfrom.pl -document_mode OAI -url http://​www.nzdl.org/​cgi-bin/​oaiserver.cgi -set demo -max_records 5
 +</​code>​
 +
 +This will try to download 5 records from the set //demo// at the nzdl.org'​s OAI server.
 +
 +The records (and optionally documents) will be downloaded into the folder the script is run from. To change this, use the ''​-cache_dir <​full-path-to-folder>''​ option.
 +
 +
 +
 +
 +===== Additional Resources =====
 +There are several tutorials on downloading files using various protocols:
 +
 +<TABAREA tabs="​Greenstone3,​Greenstone2">​
 +<TAB>
 +  * [[http://​wiki.greenstone.org/​wiki/​gsdoc/​tutorial/​gs3-current/​en/​downloading_from_internet.htm|Downloading files from the web]]
 +  * [[http://​wiki.greenstone.org/​wiki/​gsdoc/​tutorial/​gs3-current/​en/​OAI_downloading.htm|Downloading over OAI]]
 +</​TAB>​
 +<TAB>
 +  * [[http://​wiki.greenstone.org/​wiki/​gsdoc/​tutorial/​gs2-current/​en/​downloading_from_internet.htm|Downloading files from the web]]
 +  * [[http://​wiki.greenstone.org/​wiki/​gsdoc/​tutorial/​gs2-current/​en/​OAI_downloading.htm|Downloading over OAI]]
 +</​TAB>​
 +
 +</​TABAREA>​