Greenstone 3.06 Release Notes

Release Name: 3.06

Release Date: 5 November 2014.

Released:

• Greenstone 3.06: 5 Nov 2014.
  Binaries for Windows, GNU/Linux 32 and 64 bit machines, Mac Leopard and Mountain Lion. (Mac versions generated on Leopard/10.5.8 and Mountain Lion/10.8.5. The latter has been spot-tested on Mac Lion but should run on Mac Mavericks.)
  The binaries have only been spot-tested. Source distributions and source components compile on the various OS, with only Snow Leopard untested.
  svn tag page trac tag page. Code revision up to 29417. Tag revision: 29420.

Release Candidate History

• Greenstone 3.06 rc1: Release Candidate 1. 10 Oct 2014.
  Binaries for Windows, GNU/Linux 32 and 64 bit machines, Mac Leopard and Mountain Lion. (Mac versions generated on Leopard/10.5.8 and Mountain Lion/10.8.5. The former has been tested on Snow Leopard/10.6, the latter on Maverick/10.9 and briefly on Mac Lion.)
  svn tag page trac tag page. Code revision up to 29361. Tag revision: 29364.

• Download the appropriate release for your operating system and run it.
• For Linux, you will need to set the file to be executable before running it. (e.g. chmod a+x Greenstone-3.06-linux)
  • The installer initially unpacks into a temporary directory (/tmp on linux). If you wish to change this, set the TMPDIR environment variable.

Note that in some cases, the following doesn't work

 export TMPDIR=/something/else
 ./Greenstone-3.06-linux

Use the following instead

 TMPDIR=/something/else ./Greenstone-3.06-linux

During the installation process you will be presented with several options. For many, the default settings will be sufficient. Some important options are

• Folder where you want greenstone3 to be installed.
• Choosing which packages to install.
  • Greenstone3 will install the Apache Tomcat webserver by default. You can choose not to install it, but then you will need to set up your own version of Tomcat to serve Greenstone. We recommend using Greenstone's Tomcat, at least initially while you get everything set up.
  • ImageMagick is bundled with Greenstone for binary web releases for all platforms, and includes JPEG2000 support. You can choose not to install it if you already have ImageMagick previously installed.
  • Ghostscript is now bundled with Greenstone for binary web releases for Windows and Mac. You can choose not to install it if you already have Ghostscript previously installed.
• Choosing a password for the administration pages. These pages allow the admin user to inspect and manage the list of registered Greenstone users. You can add new users, and change group settings for existing users. Greenstone user registration is needed if you want to use remote GLI login to the Greenstone server, or if you want to make collections/documents only accessible by certain groups of users. (If the password is not set during installation, the default password for the 'admin' user is 'admin'. You can change the password any time after installation, by running the Greenstone 3 server and visiting the Administration pages. See below).

Once you have successfully installed Greenstone3, you can start up the server by choosing Grenstone3 Digital Library from the Start menu (Windows) or running gs3-server.sh/bat. This launches a small server program which starts up Tomcat and launches a browser. A small window pops up which allows you to change some settings for your library and restart the Tomcat server. Closing this program will stop Tomcat running. By default, your library will be available at localhost:8383/greenstone3/library. File→Settings in the Greenstone Server window gives you options to change the port number and which browser it uses by default. More notes about running Greenstone can be found in the README.txt file in the top level Greenstone folder.

To build collections, run GLI from the Start menu (Windows) or by running gli/gli.sh/bat in the top level Greenstone3 folder. Tutorial exercises about building collections in Greenstone 3 can be found here. Make sure you select the Greenstone3 tab at the top if it is not already selected.

• Windows: Source Component instructions
• Linux/Mac: Source Component instructions

• Windows: Source Distribution instructions
• Linux/Mac: Source Distribution instructions

In Greenstone 3, collections should be available over OAI by default. Their collectionConfig.xml files already specify that each collection is OAI enabled, through use of an OAIPMH serviceRack element. If you want to disable a collection from being accessibile over OAI, comment out the OAIPMH serviceRack element in that collection's collectionConfig.xml. You would do so by embedding the entire element in comment markers:

  <!--
    <serviceRack name="OAIPMH">
        <ListMetadataFormats>
           ...
        </ListMetadataFormats>
    </serviceRack>
  -->

If you wish to validate the Greenstone 3 OAIServer, edit resources/oai/OAIConfig.xml to add in the adminEmail property to contain the email to where test results should be sent. Also set the repositoryId field to a ID name you want (e.g. to greenstone), beware that there are some naming conventions that govern valid values for repositoryID. If testing the behaviour of the resumptionToken, set the resumeAfter element to a low value like 5. Then restart the Greenstone server.

To validate your OAI server, visit http://www.openarchives.org/Register/ValidateSite. Your server must be available over the internet to do this. The machine on which you're running the Greenstone 3 server will have to have its firewall and virtual server (port-forwarding) settings set up such that the Greenstone server can be made accessible to the outside world.

Setting up your Greenstone 3 OAI Server is covered in further detail in the tutorial http://wiki.greenstone.org/gsdoc/tutorial/gs3-current/en/GS_OAI_server.htm

For further information on your Greenstone OAI Server, please read through OAI support.

This will allow remote client-GLI applications to connect to your Greenstone server, to remotely create and upload new collections to be built and hosted by your server machine.

Remote Greenstone 3 Server

To install the server-side functionality:

1. If you're on Windows, you will need to teach Greenstone where the perl executable is.

You can do this either manually, by editing a couple of Greenstone config files as explained just below, or you could run the Greenstone server once and press Enter Library button to visit your library home page. Doing so will automatically set up the perl path in various Greenstone files.

To do this manually on Windows,

a. Open Tomcat's conf/web.xml file (found in greenstone3/packages/tomcat/conf folder) for editing, as you may need to specify the full path of the Perl library for the parameter "executable" of CGIServlet. This takes the form:

<init-param>
	<param-name>executable</param-name>
        <param-value>C:\Program Files\greenstone3\gs2build\bin\windows\perl\bin\perl.exe</param-value>
</init-param>

b. Edit the first line of the greenstone3/web/WEB-INF/cgi/gliserver.pl file and specify the full path of the perl binary. On Windows this will be (if installed in the default location):

#!C:\Program Files\greenstone3\gs2build\bin\windows\perl\bin\perl -w

2. If you're on Linux and working with a binary release, make sure that the webserver user has executable permissions on gliserver.pl:

cd greenstone3/web/WEB-INF/cgi
chmod a+rx gliserver.pl

3. Make the Greenstone "collect" directory, located in web/sites/localsite, writeable by the webserver user.

On Unix, use chmod.

On Windows, run in a DOS prompt:

cacls "C:\Program Files\Greenstone3\web\sites\localsite\collect" /P Everyone:F

4. Open up the file build.properties located in your greenstone installation folder. Edit the tomcat.server property's value to refer to your server machine's hostname instead of leaving it at the default value of "localhost":

# tomcat info
tomcat.server=your-server-computer-name

Once the server is started up, this will update the same property in greenstone's web/WEB-INF/classes/global.properties file. Then images viewed from a browser on the client side will refer to the correct location on the remote machine.

(If you don't know what your machine's host name is and you're on Windows, then open a DOS prompt and type:

ipconfig /all

Scroll to the top of the output that gets printed to the screen and note what it mentions for HOST NAME. Also note the DNS Suffix Search List.

Put these two together with a period mark to separate them and use this as the value for your tomcat.server property.)

5. Start up your Greenstone 3 server:

ant restart

6. Check that Tomcat and Greenstone3 are working correctly by visiting

http://<your-machine-name>:<port>/greenstone3/library

7. Add some user accounts by visiting the Greenstone 3 home page (http://YOURHOST:YOURPORT/greenstone3/library), clicking the admin link at the top right and logging in. The username is admin. By default, the password is admin too, unless you already set this during the installation process or if you changed this afterwards.

Once logged in, go to the Administration page. You can access this via the link on the home page. (Or you can click the admin link, choose Account Settings, and then click the Administration Page link on the top left. )

Add a new user by providing a new username, setting a password for the user that's a minimum of 3 characters long, and by using the drop-down provided for the Groups field to one or more of the available options, such as personal-collections-editor.

Even if only the admin user wishes to use the client-gli, they will still need to log in to the Administration Page once after installation in order for the user database to be set up.

8. Finally, visit the following page in the web browser to test that your remote Greenstone server is set up properly:

http://<your-machine-name>:<port>/greenstone3/cgi-bin/gliserver.pl?cmd=check-installation

You should get a message saying "Java found" and "Installation OK!" Important: You cannot continue until this is successful, as the Remote Greenstone 3 server will not work without it!

If you get a message saying "Java failed"

• check that the Java run-time is installed and on the webserver's path. If you get a "500 Internal Server Error", check the error log of your webserver for the cause (greenstone3/web/logs/greenstone.log).
• consult the more detailed instructions at Remote Greenstone for further steps that may be necessary

Client-GLI

Assuming that the remote Greenstone server is accessible to the outside world and you're not behind a firewall, you can access the remote Greenstone server from a client-gli application installed on any other machine. To do so,

1. Run client-gli quite as you would GLI. It's accessible from the Windows Start menu, otherwise you can run the client-gli script (located at the toplevel of a Greenstone installation) from a terminal.

2. You'll be asked for the gliserver.pl URL of the remote Greenstone 3 server that you wish to connect to. This is of the form

http://<remote-gs3-server-machine-name>:<remote-port>/greenstone3/cgi-bin/gliserver.pl

It's the same URL as in Step 10 of setting up the remote GS3 server above.

The new Format Conversion Wizard to convert GS2 format statements to GS3 format statements (see this page) only appears when you're working with GLI, not client-GLI. The client-GLI for GS3 will only perform the most basic initial step in the conversion process, which is to preserve the GS2 format statements in inactive XML tags in the new collection's collectionConfig.xml.

However, if you have a local Greenstone 3 installed, you can still manage to convert a remote collection's collect.cfg file to its GS3 equivalent. See here for details.

• The most noticeable important change is that highlighting works properly again when searching
• Moved from JRE 6 to include JRE7. The Solr extenstion needs Java 7 to function properly, especially on Windows. Without it there are issues with the solr server still running in the background and holding on to indexes when Greenstone wants to use them.
• Moved from wget 1.13.4 to wget 1.15
• The Mac .app shortcuts now do logging, which may be helpful in debugging if users experience problems running Greenstone applications from these shortcuts
• Contains several bugfixes after RC1: Perl 5.18 traverses hashmaps randomly now, for which Greenstone has been modified to make sure that searching still returns the correct documents once more. Casefolding works again after building with and without accentfolding an doesn't require a server restart.

• Greenstone 3 now includes Greenbug, an online XSL template editor. See this page for details.
• PDF-box extension now included in a binary release.
• Panorama support ???
• Solr support added. Greenstone 3 can now use Solr as its indexer. See the solr page for more details. This is an extension, but is included in a binary release and checked out automatically when getting Greenstone 3 from SVN.
• Lucene search results sort by and sort order params, see the lucene page.
• Lucene and solr both upgraded to 4.7.2 (was 3.3.0). When building solr collections, this allows using the Kuromoji Japanese analyzer as suggested by Mr Yamaguchi Gaku and using the Snowball or Hunspell analyzers for Spanish and other European languages as suggested by Mr Diego Spano. (See the solr page.)
• OAI server upgrade. Super sets have been introduced (see the OAI page). Resumption tokens have been improved. You can use gs.OAIDateStamp metadata for a document to manually set the date stamp for a record.
• Support for running Greenstone3 off a read-only medium such as DVD
• depositor??
• new format elements: <gsf:rank/>, <gsf:OID/>. Output a document's rank and OID, respectively.
• gsf:option disableZoom - disable the screen image zooming
• the ability to set the default values for search parameters in the collection's collectionConfig.xml file. Add lines like <paramDefault name="case" value="0"/> to the <search> element.
• button to extract currently visible docs to their own document??
• new options for gsf:metadata format element:
  • <gsf:metadata name="dc.Subject" pos="classifiedBy"/>. When a document has multiple values for a metadata field that is being classified on, it will get classified into several positions in the list. Using pos='classifiedBy' allows for the particular metadata value that was used at each position to be output.
  • allow comma separated list of metadata in name field. This is useful when using pos="classifiedBy" when outputting documnets in a classifier, if the classifier was built on a list of metadata.
• tidying up document content editing
• RSS support now added to Greenstone 3. See the RSS page for more details.

• Support for compiling with gnome-lib
• Support for cross compilation, and for compiling for android
• ImageConverter aspectpad options to help with images that will feed into 3D panorama
• RSS support added, see this page.
• The Demo collections now use dc.Title, dc.Subject, dc.Language in place of the equivalent dls fields.
• New option for List classifier: standardize_capitalization. Metadata values are lowercased for sorting into bookshelves. With this option set, then the lowercase value will be used as the bookshelf Title. This can then be changed to title case/upper case etc using CSS. Without the option set, then the majority case variant will be used. For example, with three values (snail, Snail, Snail) then Snail will be used.
• sortfields for lucene are now defined separately to indexfields, see this page.
• A new XSL transformation file (in gli/classes/xml/xsd-to-mds.xsl) that can be used as a helpful template when needing to convert a metadata schema that is in XSD format to Greenstone's metadata set format.

• TabSeparatedPlugin, for tab-separated metadata files
• export.pl options tidied up. plugout specific options are gone. Instead, there is a new saveas_options option. The value of this will be passed to the plugout as it's options. There is also a new option include_auxiliary_database_files. If this is set then the archives database files are also output. Otherwise, only the document files will be output.
• Accented characters are now allowed in custom metadata names.
• A new List option: -metadata_selection_mode firstvalue|firstvalidmetadata|allvalues. This determines which values to classify on if using a list of metadata, eg dc.Date,Date. Replicates -firstvalueonly and -allvalues of AZCompactList, but can be specified for each level in the Hierarchy.
• PDFPlugin has now has the -use_realistic_book option - like HTML Plugin
• Lucene upgraded from 3.3.0 to 4.7.2
• wget upgraded from 1.13.4 to 1.15

• When you open a Greenstone 2 collection in a Greenstone 3 GLI, a new Format Conversion Wizard helps you to convert the format statements. See this page.
• New CDWA Lite metadata set for GLI, generated by applying xsd-to-mds.xsl to CDWALite's XSD (http://www.getty.edu/CDWA/CDWALite/CDWALite-xsd-public-v1-1.xsd).

When you've built a collection of documents, you may discover that there appears to be a copy of all these documents in the collection's import, archives and index subfolders and wonder whether Greenstone could really be so inefficient with space as to keep 3 copies to everything. As it happens though, Greenstone uses hard-links both on Linux and Windows, in order to keep just one set of your documents. Then it simply hardlinks to these, instead of making copies. By default, Windows doesn't show you when files on your filesystem are hard-linked. If you choose to install the Windows extension program Link Shell Extension (LSE), it will put red arrows on files that are hard linked.

Currently, even if you don't tick the 'enable admin access' button in the installer, you still get admin access, but with a default password of 'admin'. If you end up in this situation, please change the admin password. Login to the administration page, 'edit' the admin account, and click 'change password'. Alternatively, you can login as admin via the login button at the top right of each page. Once you are logged in, this button will change to say 'admin. Click this button and select 'account settings'. From there, you can select 'change password'.

See bugzilla report.

If attempting to view a java applet (like Collage or Phind phrase classifiers) crashes Firefox, then make sure you have the Java Applet plugin installed. If it is installed and Firefox is still crashing, then open firefox and visit the page

about:config

Scroll down to the property:

dom.ipc.plugins.java.enabled

Set it to true (rightclick and choose toggle).

If you've configured a PDFPlugin to convert PDFs to images, increase the verbosity in Import Options and Build Options to 5 in GLI's Create panel.

When rebuilding the collection, check to see if you encounter the following error message mentioning that 'memory allocation failed', a 'corrupt image' at 'ReadPNGImage' and 'PostScript delegate failed':

import.pl> Converting pdf05-notext.pdf to pagedimg_jpg format
import.pl> calling cmd "/usr/bin/perl" -S gsConvert.pl -verbose 5 -pdf_zoom 2 -errlog "/research/myfolder/gs3-svn-12Sep2013/web/sites/localsite/collect/Enhanced-PDF/tmp/1378957949/err.log" -output pagedimg_jpg "/research/myfolder/gs3-svn-12Sep2013/web/sites/localsite/collect/Enhanced-PDF/tmp/1378957949/pdf05-notext.pdf"
import.pl> Error executing pdfpstoimg.pl
import.pl> pdfpstoimg error log:
import.pl> convert: memory allocation failed `/tmp/magick-31829ofGIFuaNZ1xy1' @ error/png.c/ReadOnePNGImage/2160.
import.pl> convert: corrupt image `/tmp/magick-31829ofGIFuaNZ1xy1' @ error/png.c/ReadPNGImage/3794.
import.pl> convert: Postscript delegate failed `/research/myfolder/gs3-svn-12Sep2013/web/sites/localsite/collect/Enhanced-PDF/tmp/1378957949/pdf05-notext.pdf': No such file or directory @ error/pdf.c/ReadPDFImage/681.
import.pl> convert: no images defined `/research/myfolder/gs3-svn-12Sep2013/web/sites/localsite/collect/Enhanced-PDF/tmp/1378957949/pdf05-notext/pdf05-notext.jpg' @ error/convert.c/ConvertImageCommand/3068.
import.pl> Convert error for /research/myfolder/gs3-svn-12Sep2013/web/sites/localsite/collect/Enhanced-PDF/tmp/1378957949/pdf05-notext.pdf 
import.pl> Could not convert pdf05-notext.pdf to pagedimg_jpg format
import.pl> convert: memory allocation failed `/tmp/magick-31829ofGIFuaNZ1xy1' @ error/png.c/ReadOnePNGImage/2160.
import.pl> convert: corrupt image `/tmp/magick-31829ofGIFuaNZ1xy1' @ error/png.c/ReadPNGImage/3794.
import.pl> convert: Postscript delegate failed `/research/myfolder/gs3-svn-12Sep2013/web/sites/localsite/collect/Enhanced-PDF/tmp/1378957949/pdf05-notext.pdf': No such file or directory @ error/pdf.c/ReadPDFImage/681.
import.pl> convert: no images defined `/research/myfolder/gs3-svn-12Sep2013/web/sites/localsite/collect/Enhanced-PDF/tmp/1378957949/pdf05-notext/pdf05-notext.jpg' @ error/convert.c/ConvertImageCommand/3068.
import.pl> Convert error for /research/myfolder/gs3-svn-12Sep2013/web/sites/localsite/collect/Enhanced-PDF/tmp/1378957949/pdf05-notext.pdf 
import.pl> Converting pdf05-notext.pdf to html format

If you see the above error message, then:

1. Use a text editor to open your Greenstone 3's gs2build/ext/imagemagick/linux/etc/ImageMagick/delegates.xml

2. Find the line that would say:

<delegate decode="ps:alpha" stealth="True" command="&quot;gs&quot; -q -dQUIET -dSAFER -dBATCH -dNOPAUSE -dNOPROMPT -dMaxBitmap=500000000 -dAlignToPixels=0 -dGridFitTT=2 &quot;-sDEVICE=pngalpha&quot; -dTextAlphaBits=%u -dGraphicsAlphaBits=%u &quot;-r%s&quot; %s &quot;-sOutputFile=%s&quot; &quot;-f%s&quot; &quot;-f%s&quot;"/>

The above specifies the PostScript delegate for PNG images. It has the sDEVICE set to pngalpha.

3. Change the line to:

<delegate decode="ps:alpha" stealth="True" command="&quot;gs&quot; -q -dQUIET -dSAFER -dBATCH -dNOPAUSE -dNOPROMPT -dMaxBitmap=500000000 -dAlignToPixels=0 -dGridFitTT=2 &quot;-sDEVICE=**pnmraw**&quot; -dTextAlphaBits=%u -dGraphicsAlphaBits=%u &quot;-r%s&quot; %s &quot;-sOutputFile=%s&quot; &quot;-f%s&quot; &quot;-f%s&quot;"/>

The above changes the sDevice to pnmraw.

4. Save the file and re-run the build now.

Thanks to the following people for new and updated translations since 3.05:

• Antonin Benoit Diouf for French translations
• John Rose, Julie Verleyen, Yvan Arnaud, Sandraghassen S. Pillai and team for French translations
• Diego Spano for Spanish translations
• Sergey Karpov and Kalima Tuenbaeva for Russian and Kazakh translations. And to Zhanat Kulenov for contributions in 2009
• Kamal Salih Mustafa Khalafala for Arabic translations
• Te Taka Keegan for Māori translations
• Tomáš Fiala for Slovak translations
• Tigran Zargaryan for Armenian translations
• Gerhard Riesthuis for Dutch translations
• Lavji Zala for Gujarati translations