Greenstone Wiki

This page is read only. You can view the source, but not change it. Ask your administrator if you think this is wrong.
====== Command Line Building ======
<!-- id:26 -->It is possible to create and build collections directly from the command line. This
page provides the basic information on building Greenstone collections on the command line. The full instructions
are provided for Windows users. If you are on a MacOS/Linux, the steps are the same, but some of the
commands themselves are slightly different. These differences are listed in the [[#MacOSX/Linux]] section. 

This page goes through the complete steps for creating a new collection and building it on the command line.
However, in some cases you may like to create a collection using the Librarian Interface and just rebuild it on the command line. In this case, you'll need to open the terminal, set up the environment, and then go straight to the building step for your existing collection.

<TABAREA tabs="Greenstone3,Greenstone2">
<TAB>

===== Windows =====
==== Open a terminal ====
On Windows, there are several different ways to open a DOS terminal (a black console screen known as the DOS Prompt). Do one of the following:
  * ''Start -> All Programs -> Accessories -> Command Prompt''
  * Under the Start menu, type ''cmd'' into the search box and press Enter
  * Hold down your keyboard's Windows key and press the key for letter r. (The Windows key is located between the Ctrl and Alt keys on your keyboard.) In the Run dialog that appears, type ''cmd'' in the textfield and press the OK button. 
  * In any Windows Explorer, hold down Shift and right click in an empty area in the window. Select ''Open command window here'' from the menu.


==== Setup the Environment ====

In order to build collections in Greenstone (or run any other Greenstone scripts from the 
command line), you must first setup the terminal's environment for Greenstone. To do this, first
change into the directory where Greenstone has been installed. 
Assuming Greenstone was installed in its default location (and your user name was "jsmith"), you can move there by typing:

<code>
cd C:\Users\jsmith\Greenstone3
</code>
//**Note** if the path to your Greenstone installation includes spaces (e.g. Program Files), you **must**
put quotations around the path. For example: ''cd "C:\Program Files\Greenstone3"'' and ''cd "%GSDL3HOME%\sites\localsite\dlpeople"''.//

Next, at the prompt type:

<code>
gs3-setup
</code>

This batch file (which you can read if you like) tells the system where to 
look for Greenstone programs.

//Note: On Windows 95/98 systems running ''gs3-setup''
 may fail with an **Out of environment space** error. 
If this happens, you should edit your system's ''config.sys''
 file (normally found at ''C:\config.sys'') and add the line 
''shell=C:\command.com /e:4096 /p'' (where ''C:'' is your system 
drive letter) to expand the size of the environment table. You'll
 need to reboot for this change to take effect, and then repeat the steps above for Greenstone.//

If, later on in your interactive session at the DOS prompt, 
you wish to return to the top level Greenstone directory you can accomplish this by typing 
''cd %GSDL3SRCHOME%''. 

**//If you close your DOS window and start another one, 
you will need to invoke ''gs3-setup'' again.//**

<!-- id:33 -->Now you are in a position to make, build and rebuild collections. 

==== Create a collection ====

The first program we will look at is the Perl program ''mkcol.pl'', 
whose name stands for “make a collection.” Typing ''perl —S mkcol.pl'' will provide 
the full list of options, which you can also view [[script_options#mkcol.pl|here]]. 

//(If your Windows environment is set up to associate the Perl application with 
files ending in ''.pl'', you can leave off the ''perl -S'' for all of these scripts.)//

To create a new collection:
<code>
perl -S mkcol.pl [options] collection-name
</code>

<!-- id:34 -->For example, to create a collection named //dlpeople// in ''localsite''
with the creator's email address of //me@cs.waikato.ac.nz//, type

<code>
perl —S mkcol.pl -site localsite —creator me@cs.waikato.ac.nz dlpeople
</code>
\\
//(Since Greenstone3 allows you to have multiple [[en:user:sites]], you must always specify in which site the 
collection is in. The default site is called ''localsite''.)//

<!-- id:36 -->To view the newly created files, move to the newly created 
collection directory by typing

<code>
cd %GSDL3HOME%\sites\localsite\collect\dlpeople
</code>


<!-- id:38 -->You can list the contents of this directory by typing ''dir''.
 There should be six subdirectories: 
  * //etc//
  * //images//
  * //import//
  * //script//
  * //style//

==== Add documents ====

<!-- id:39 -->Now we must populate the collection with sample documents. To do this, 
we copy documents into the collections ''import'' folder. Assuming your documents are in the folder
''C:\Users\jsmith\dldocuments'', you can either:

<!-- id:40 -->select the contents of the ''dldocuments'' directory
and drag them into the ''dlpeople'' collection's ''import'' directory.

<!-- id:41 -->Or, you can type the command

<code>
xcopy /s C:\Users\jsmith\dldocuments\* import
</code>

==== Edit the Config file ====

<!-- id:42 -->In the collection's ''etc'' directory there is a file called ''collectionConfig.xml''.
Any modifications that you can make in the GLI, can also be achieved by manually editing the
''collectionConfig.xml'' file. Simply open it using your favorite text editor, 
e.g. Notepad or Wordpad, make changes and save it. You can learn more about the Collection configuration file [[en:user:configuration files#Collection configuration files|here]]. 

==== Import the collection ====


<!-- id:43 -->Now you are ready to “import” the collection. 
This is the process of bringing the documents into the Greenstone system, 
standardizing the document format, the way that metadata is specified, 
and the file structure in which the documents are stored. 
Type ''perl —S import.pl'' at the prompt to get a list of all the options for the import program, 
or view them [[script_options#import.pl|here]].

<code>
perl —S import.pl -site localsite dlpeople
</code>

<!-- id:44 -->Don't worry about all the text that scrolls past—it's just reporting 
the progress of the import. Note that you do not have to be in either the 
//collect// or //dlpeople// directories when this command is entered;
 because ''%GSDL3SRCHOME%'' is already set, the Greenstone software can work out where 
the necessary files are.

==== Build the collection ====

<!-- id:49 -->The next phase is to “build” the collection, 
which creates all the indexes and files that make the collection work. 
Type ''perl —S buildcol.pl'' at the command prompt for a list of
 collection-building options, which are also listed [[script_options#buildcol.pl|here]]. 
For now, stick to the defaults by typing

<code>
perl —S buildcol.pl -site localsite dlpeople
</code>

<!-- id:50 -->Again, don't worry about the “progress report” text that scrolls past.

==== Make the collection live ====

Finally, we need to make the collection "live" by replacing the collection's old ''index'' folder
with the contents of the ''building'' folder. We can do this in two ways:

In an explorer window (i.e. outside of the terminal) simply select 
the contents of the //dlpeople// collection's ''building''
directory and drag them into the ''index'' directory.

<!-- id:53 -->Alternatively, you can remove the ''index'' directory 
(and all its contents) by typing the command
<code>
rd /s index            # on Windows NT/2000
deltree /Y index       # on Windows 95/98
</code>

<!-- id:54 -->and then change the name of the ''building'' directory to ''index'' with

<code>
ren building index
</code>

It is important that these commands are issued from the correct directory 
(unlike the Greenstone commands ''mkcol.pl'', ''import.pl'' and ''buildcol.pl'').
 If the current working directory is not //dlpeople//, type 
''cd %GSDL3HOME%\sites\localsite\collect\dlpeople'' before going through the 
''rd'', ''ren'' and ''mkdir'' sequence above.

<!-- id:57 -->If your Greenstone server is already running, 
you should be able to access the newly built collection 
from your Greenstone homepage. You will have to reload the page 
if you already had it open in your browser, or perhaps even close 
the browser and restart it (to prevent caching problems).


<!-- id:59 -->In summary then, the commands typed to produce the //dlpeople// collection are:

<code>
cd C:\Users\jsmith\Greenstone3 # assuming default location
gs3-setup
perl —S mkcol.pl -site localsite —creator me@cs.waikato.ac.nz dlpeople
cd %GSDL3HOME%\sites\localsite\collect\dlpeople
xcopy /s C:\Users\jsmith\dldocuments\* import
perl —S import.pl -site localsite dlpeople
perl —S buildcol.pl -site localsite dlpeople
rd /s index           # on Windows NT/2000
deltree /Y index      # on Windows 95/98
ren building index
</code>

<!-- LINUX ###################################################################################-->

=====MacOSX/Linux =====

Running Greenstone from the command line on MacOSX and Linux is very similar to doing it
on a Windows. Some of the commands are just a bit different. First change into the directory where Greenstone has been installed. 
For example, if Greenstone is installed under its default name 
at the top level of your user account you can move there by typing

<code>
cd /home/jsmith/Greenstone3
</code>
\\
To set up the Greenstone environment:
<code>
source ./gs3-setup.bash 
</code>

\\
To create a collection:
<code>
mkcol.pl -site localsite —creator me@cs.waikato.ac.nz dlpeople
</code>
\\
To move to the newly created 
collection directory:
<code>
cd $GSDL3HOME/sites/localsite/collect/dlpeople
</code>
\\
You can list the contents of this directory by typing ''ls''. In the collection's //etc// directory there is a file called //[[en:user:configuration files#collection configuration files|collect.cfg]]//. 
You can open and edit this using your favorite text editor — emacs is a popular editor on Linux. 


To copy the contents of the ''/home/documents/dldocuments'' 
directory into the ''GSDL3HOME/sites/localsite/collect/dlpeople/import'' directory. To do this, type the command
<code>
cp —r /home/documents/dldocuments/*   import/
</code>
\\


To “import” the collection:
<code>
import.pl -site localsite dlpeople
</code>
\\

Next, “build” the collection: 
<code>
buildcol.pl -site localsite dlpeople
</code>
\\

Finally, make the collection “live” by putting 
all the material that has just been put in the collection's 
//building// directory into the //index// directory. First, 
remove the old index:
<code>
rm —r index/*
</code>
//(assuming you are in the ''dlpeople'' directory)//

\\
And move the building directory to index:
<code>
mv building/* index/
</code>

\\
In summary then, the commands typed to produced the //dlpeople// collection are:
<code>
cd /home/jsmith/Greenstone3 # assuming default Greenstone in user directory
source ./gs3-setup.bash 
mkcol.pl —creator me@cs.waikato.ac.nz dlpeople
cd $GSDL3HOME/collect/dlpeople
cp —r /home/documents/dldocuments/*   import/
import.pl -site localsite dlpeople
buildcol.pl -site localsite dlpeople
rm -r index/*
mv building/* index
</code>



===== Additional Resources =====

While this page only goes through the basics of building collections, there
are many other scripts that can be run from the command line 
(like [[en:user_advanced:command_line_download|downloading]] documents). You can take a look at
 the [[script_options|scripts and their options]] to get an idea of what else is available.
</TAB>
<!-- #############################################################################################
#############################################################################################
#############################################################################################-->
<TAB>

===== Windows =====
==== Open a terminal ====
On Windows, there are several different ways to open a DOS terminal (a black console screen known as the DOS Prompt). Do one of the following:
  * ''Start -> All Programs -> Accessories -> Command Prompt''
  * Under the Start menu, type ''cmd'' into the search box and press Enter
  * Hold down your keyboard's Windows key and press the key for letter r. (The Windows key is located between the Ctrl and Alt keys on your keyboard.) In the Run dialog that appears, type ''cmd'' in the textfield and press the OK button. 
  * In any Windows Explorer, hold down Shift and right click in an empty area in the window. Select ''Open command window here'' from the menu.


==== Setup the Environment ====

In order to build collections in Greenstone (or run any other Greenstone scripts from the 
command line), you must first setup the terminal's environment for Greenstone. To do this, first
change into the directory where Greenstone has been installed. 
Assuming Greenstone was installed in its default location (and your username is "jsmith"), you can move there by typing:

<code>
cd C:\Users\jsmith\Greenstone
</code>
//**Note** if the path to your Greenstone installation includes spaces (e.g. Program Files), you **must**
put quotations around the path. For example: ''cd "C:\Program Files\Greenstone"'' and ''cd "%GSDLHOME%\collect\dlpeople"''.//

Next, at the prompt type:

<code>
setup
</code>

This batch file (which you can read if you like) tells the system where to look for Greenstone programs.

//Note: On Windows 95/98 systems running ''setup.bat'' may fail with an **Out of environment space** error. If this happens, you should edit your system's ''config.sys'' file (normally found at ''C:\config.sys'') and add the line ''shell=C:\command.com /e:4096 /p'' (where ''C:'' is your system drive letter) to expand the size of the environment table. You'll need to reboot for this change to take effect, and then repeat the steps above for Greenstone.//

If, later on in your interactive session at the DOS prompt, 
you wish to return to the top level Greenstone directory you can accomplish this by typing 
''cd %GSDLHOME%''. 

**//If you close your DOS window and start another one, you will need to invoke ''setup.bat'' again.//**

<!-- id:33 -->Now you are in a position to make, build and rebuild collections. 

==== Create a collection ====

The first program we will look at is the Perl program ''mkcol.pl'', 
whose name stands for “make a collection.” Typing ''perl —S mkcol.pl'' will provide 
the full list of options, which you can also view [[script_options#mkcol.pl|here]]. 

//(If your Windows environment is set up to associate the Perl application with 
files ending in ''.pl'', you can leave off the ''perl -S'' for all of these scripts.)//

To create a new collection:
<code>
perl -S mkcol.pl [options] collection-name
</code>

<!-- id:34 -->For example, to create a collection named //dlpeople// 
with the creator's email address of //me@cs.waikato.ac.nz//, type

<code>
perl —S mkcol.pl —creator me@cs.waikato.ac.nz dlpeople
</code>
\\
//Please substitute your email address for mine!//

<!-- id:36 -->To view the newly created files, move to the newly created 
collection directory by typing

<code>
cd %GSDLHOME%\collect\dlpeople
</code>


<!-- id:38 -->You can list the contents of this directory by typing //dir//.
 There should be six subdirectories: 
  * //etc//
  * //images//
  * //import//
  * //macros//
  * //script//
  * //style//

==== Add documents ====

<!-- id:39 -->Now we must populate the collection with sample documents. To do this, 
we copy documents into the collections ''import'' folder. Assuming your documents are in the folder
''C:\Users\jsmith\dldocuments'', you can either:

<!-- id:40 -->select the contents of the ''dldocuments'' directory
and drag them into the ''dlpeople'' collection's ''import'' directory.

<!-- id:41 -->Or, you can type the command

<code>
xcopy /s C:\Users\jsmith\dldocuments\* import
</code>

==== Edit the Config file ====

<!-- id:42 -->In the collection's ''etc'' directory there is a file called ''collect.cfg''. 
Open it using your favorite text editor, e.g. Notepad or Wordpad. Any modifications that you 
can make in the GLI, can also be achieved by manually editing this
collection configuration file. Simply open it using your favorite text editor, 
e.g. Notepad or Wordpad, make changes and save it.
You can learn more about the Collection configuration file [[configuration files#Collection configuration files|here]].

==== Build the collection ====

Building a collection consists of two main stages, importing and building. Importing is the process of bringing the documents into the Greenstone system, 
standardizing the document format, the way that metadata is specified, 
and the file structure in which the documents are stored. 
The building stage generates the indexes, databases and other auxiliary files that are needed to make the collection work in Greenstone.

These processes can be run separately, or, in later Greenstone versions, a single script can be run which invokes both processes (see [[#build_the_collection_in_one_easy_step | below]]).

=== Importing ===
<!-- id:43 -->
Type //perl —S import.pl// at the prompt to get a list of all the options for the import program, 
or view them [[script_options#import.pl|here]].

<code>
perl —S import.pl dlpeople
</code>

<!-- id:44 -->Don't worry about all the text that scrolls past—it's just reporting 
the progress of the import. Note that you do not have to be in either the 
//collect// or //dlpeople// directories when this command is entered;
 because ''%GSDLHOME%'' is already set, the Greenstone software can work out where 
the necessary files are.

=== Building ===

<!-- id:49 -->
Type ''perl —S buildcol.pl'' at the command prompt for a list of
 collection-building options, which are also listed [[script_options#buildcol.pl|here]]. 
For now, stick to the defaults by typing:

<code>
perl —S buildcol.pl dlpeople
</code>

<!-- id:50 -->Again, don't worry about the “progress report” text that scrolls past.

=== Make the collection live ===

Finally, we need to make the collection "live" by replacing the collection's old ''index'' folder
with the contents of the ''building'' folder. We can do this in two ways:

In an explorer window (i.e. outside of the terminal) simply select 
the contents of the //dlpeople// collection's ''building''
directory and drag them into the ''index'' directory.

<!-- id:53 -->Alternatively, you can remove the ''index'' directory 
(and all its contents) by typing the command
<code>
rd /s index            # on Windows NT/2000
deltree /Y index       # on Windows 95/98
</code>

<!-- id:54 -->and then change the name of the ''building'' directory to ''index'' with

<code>
ren building index
</code>

It is important that these commands are issued from the correct directory 
(unlike the Greenstone commands ''mkcol.pl'', ''import.pl'' and ''buildcol.pl'').
 If the current working directory is not //dlpeople//, type 
''cd %GSDLHOME%\collect\dlpeople'' before going through the 
''rd'', ''ren'' and ''mkdir'' sequence above.

<!-- id:57 -->If your Greenstone server is already running, you should be able to access the newly built collection 
from your Greenstone homepage. You will have to reload the page 
if you already had it open in your browser, or perhaps even close 
the browser and restart it (to prevent caching problems). Alternatively,
 if you are using the “local library” version of Greenstone you 
will have to restart the library program. 

==== Build the collection in one easy step ====

An alternative to running import, then build, then deleting the old index and renaming building to index, is to run a single command, full-rebuild.pl.

<code>
perl -S full-rebuild.pl dlpeople
</code>

This will run import.pl, buildcol.pl and then remove the old indexes and copy the new ones into the index folder.

Import or buildcol options can be passed to full-rebuild. If the option is shared between import.pl and buildcol.pl then it can appear as is, such as -verbosity 5.  This value will be passed to both programs. If an option is specific to one of the programs in particular, then prefix it with 'import:' or 'buildcol:' respectively, as in '-import:OIDtype hash_on_full_filename'
       
Remember, you can run 'perl -S import.pl' or 'perl -S buildcol.pl' from the command line with no arguments to see the specific options they take.

==== Summary ====
<!-- id:59 -->In summary then, the commands typed to produce the //dlpeople// collection are:

To set up the collection:
<code>
cd C:\Users\jsmith\Greenstone # assuming default location
setup.bat
perl —S mkcol.pl —creator me@cs.waikato.ac.nz dlpeople
cd %GSDLHOME%\collect\dlpeople
xcopy   /s   d:\collect\dlpeople\*   import # assuming D drive
</code>

To build the collection:
<code>
perl -S full-rebuild.pl dlpeople
</code>
or
<code>
perl —S import.pl dlpeople
perl —S buildcol.pl dlpeople
rd /s index           # on Windows NT/2000
deltree /Y index      # on Windows 95/98
ren building index
</code>

=====MacOSX/Linux =====

Running Greenstone from the command line on MacOSX and Linux is very similar to doing it
on a Windows. Some of the commands are just a bit different. Please read through the Windows section for more information about the steps mentioned here.

First change into the directory where Greenstone has been installed. 
For example, if Greenstone is installed under its default name 
at the top level of your user account you can move there by typing

<code>
cd /home/jsmith/Greenstone
</code>
\\
To set up the Greenstone environment:
<code>
source ./setup.bash 
</code>
//If you are unsure of the shell type you are using, enter ''echo $0'' at your 
command-line prompt —it will print out the sought information. 
If you are using a different shell contact your system administrator for advice.//

\\
To create a collection:
<code>
mkcol.pl —creator me@cs.waikato.ac.nz dlpeople
</code>
\\
To move to the newly created 
collection directory:
<code>
cd $GSDLHOME/collect/dlpeople
</code>
\\
You can list the contents of this directory by typing ''ls''. In the collection's //etc// directory there is a file called //[[configuration files#collection configuration files|collect.cfg]]//. 
You can open and edit this using your favorite text editor — emacs is a popular editor on Linux. 


To copy the contents of the ''/home/documents/dldocuments'' 
directory into the ''GSDLHOME/collect/dlpeople/import'' directory. To do this, type the command
<code>
cp —r /home/documents/dldocuments/*   import/
</code>
\\

To build the collection in one step:
<code>
full-rebuild.pl dlpeople
</code>

Or, to build it step by step manually:

To “import” the collection:
<code>
import.pl dlpeople
</code>
\\

Next, “build” the collection: 
<code>
buildcol.pl dlpeople
</code>
\\

Finally, make the collection “live” by putting 
all the material that has just been put in the collection's 
//building// directory into the //index// directory. First, 
remove the old index:
<code>
rm —r index/*
</code>
//(assuming you are in the ''dlpeople'' directory)//

\\
And move the building directory to index:
<code>
mv building/* index/
</code>

\\
In summary then, the commands typed to produced the //dlpeople// collection are:
<code>
cd /home/jsmith/Greenstone # assuming default Greenstone in user directory
source ./setup.bash 
mkcol.pl —creator me@cs.waikato.ac.nz dlpeople
cd $GSDLHOME/collect/dlpeople
cp —r /home/documents/dldocuments/*   import/
</code>
To build the collection:
<code>
full-rebuild.pl dlpeople
</code>
or
<code>
import.pl dlpeople
buildcol.pl dlpeople
rm -r index/*
mv building/* index
</code>


===== Incremental Building =====

Incremental building is where you only process the new or changed documents each time you build, thereby speeding up the build process.

**Incremental importing**: New documents will be imported. Modified documents will be re-imported. Deleted documents will be removed from the collection. If metadata has changed, then documents will be reimported.

Important note for collection design: Greenstone can notice that metadata in a folder has been added/changed, but it is not smart enough to tell which documents in the folder the changed metadata
belongs to. Therefore, if metadata in a folder has changed (including new metadata being added), then all documents in that folder will be reimported. This means that if you have all your documents in the top level import folder, adding new metadata or changing any metadata for any document will result in all documents being reimported. If you intend to do incremental import, then please organize your documents into subfolders. That way modifying metadata for some documents won't result in all other documents being reimported.

**Incremental indexing**: Currently only the Lucene indexer (and Solr indexer included with Greenstone 3) can do incremental indexing. If you are using MG/MGPP then a full buildcol pass will be done, even if incremental-buildcol.pl is used.

If collection design has changed, then you will need to do a full rebuild. Changes to plugin options, and some import options will necessitate a full import. Changes to search indexes, partition indexes, browsing classifiers will necessitate a full buildcol. 

If you are doing incremental building, a full rebuild every now and then can be a good idea, in case something hasn't gone quite right in the incremental process. Once we've finished retesting incremental building, this shouldn't be a problem any more. In the meantime, if you notice anything weird after an incremental build, then a full rebuild is a good idea then too.

On the command line, you can run building/importing incrementally by using the scripts incremental-rebuild.pl, incremental-import.pl and incremental-buildcol.pl instead of full-rebuild.pl, import.pl and buildcol.pl, respectively.

Note that running incremental-buildcol.pl when you are not using Lucene for your indexer will be the same as running buildcol.pl. Without any -builddir option, incremental-buildcol.pl will do the indexing into the existing index directory, so you don't need to rename building to index.

===== Additional Resources =====

While this page only goes through the basics of building collections, there
are many other scripts that can be run from the command line 
(like [[en:user_advanced:command_line_download|downloading]] documents). You can take a look at
 the [[script_options|scripts and their options]] to get an idea of what else is available.
</TAB>
</TABAREA>