Greenstone 3.07 Release Notes

Release Name: 3.07

Release Date: 9 Sep 2015.

Released:

• Binaries for Windows, GNU/Linux 32 and 64 bit machines, Mac Leopard/10.5.8 and Mountain Lion. The two Mac binaries are generated on Leopard/10.5.8 and Mountain Lion/10.8.5 respectively.
  The Mountain Lion installer should work on Mac Yosemite/10.10 out of the box. It may therefore install on Maverick too, but this is untested. The Mountain Lion binary however includes tools that, having been built the Mountain Lion system, may not work on Yosemite (or Maverick). For instance, the pre-built wvware does not work on Yosemite.
  The binaries have only been spot-tested. Source distributions and source components compile on the various OS.
  svn tag page trac tag page. Code revision up to 30240. Tag revision: 30244.

Release Candidate History

• Greenstone 3.07 rc1: Release Candidate 1. Released 3 Aug 2015.
  Binaries for Windows, GNU/Linux 32 and 64 bit machines, Mac Leopard and Mountain Lion. (Mac versions generated on Leopard/10.5.8 and Mountain Lion/10.8.5. The latter works on Mac Yosemite, but with some adjustments as explained further in these release notes.)
  Download from the snapshots page
  svn tag page trac tag page. Code revision up to 30089. Tag revision: 30092.

• Greenstone 3.07 rc2: Release Candidate 2. Released 26 Aug 2015.
  Binaries for Windows, GNU/Linux 32 and 64 bit machines, Mac Leopard and Mountain Lion. The Mac binaries are generated on Leopard/10.5.8 and Mountain Lion/10.8.5. The latter works on Mac Yosemite out of the box since it now bundles a JRE.
  Download from the snapshots page
  svn tag page trac tag page. Code revision up to 30138. Tag revision: 30141.

• Download the appropriate release for your operating system and run it.

• Note to Mac users: The security settings in newer Mac OS versions have been altered to by default disallow users from casually opening and running .dmg executables that are not from Apple itself. When attempting to open the Greenstone binary .dmg file, if it pops up an error warning about security, you will need to set up the Security on your Mac to allow you to run .dmg files downloaded from the internet. Otherwise the Greenstone mac binary will not run. To do this, Go to your Mac's System Preferences. Under "Personal", select Security & Privacy. In the General tab, tick "Allow apps downloaded from Anywhere", then confirm that you want to "Allow From Anywhere". You'll need to be admin to do this, otherwise click on the padlock at the bottom left of the Security & Privacy tab and log in as admin.

• For Linux, you will need to set the file to be executable before running it. (e.g. chmod a+x Greenstone-3.07-linux)
  • The installer initially unpacks into a temporary directory (/tmp on linux). If you wish to change this, set the TMPDIR environment variable.

Note that in some cases, the following doesn't work

 export TMPDIR=/something/else
 ./Greenstone-3.07-linux

Use the following instead

 TMPDIR=/something/else ./Greenstone-3.07-linux

During the installation process you will be presented with several options. For many, the default settings will be sufficient. Some important options are

• Folder where you want greenstone3 to be installed.
• Choosing which packages to install.
  • Greenstone3 will install the Apache Tomcat webserver by default. You can choose not to install it, but then you will need to set up your own version of Tomcat to serve Greenstone. We recommend using Greenstone's Tomcat, at least initially while you get everything set up.
  • ImageMagick is bundled with Greenstone for binary web releases for all platforms, and includes JPEG2000 support. You can choose not to install it if you already have ImageMagick previously installed.
  • Ghostscript is now bundled with Greenstone for binary web releases for Windows and Mac. You can choose not to install it if you already have Ghostscript previously installed.
• Choosing a password for the administration pages. These pages allow the admin user to inspect and manage the list of registered Greenstone users. You can add new users, and change group settings for existing users. Greenstone user registration is needed if you want to use remote GLI login to the Greenstone server, or if you want to make collections/documents only accessible by certain groups of users. (If the password is not set during installation, the default password for the 'admin' user is 'admin'. You can change the password any time after installation, by running the Greenstone 3 server and visiting the Administration pages. See below).

Once you have successfully installed Greenstone3, you can start up the server by choosing Greenstone3 Digital Library from the Start menu (Windows) or running gs3-server.sh/bat. This launches a small server program which starts up Tomcat and launches a browser. A small window pops up which allows you to change some settings for your library and restart the Tomcat server. Closing this program will stop Tomcat running. By default, your library will be available at localhost:8383/greenstone3/library. File→Settings in the Greenstone Server window gives you options to change the port number and which browser it uses by default. More notes about running Greenstone can be found in the README.txt file in the top level Greenstone folder.

To build collections, run GLI from the Start menu (Windows) or by running gli/gli.sh/bat in the top level Greenstone3 folder. Tutorial exercises about building collections in Greenstone 3 can be found here. Make sure you select the Greenstone3 tab at the top if it is not already selected.

From GS3.07rc2 onwards, we're including a JRE with Mac Mountain Lion binaries, so that the 3.07rc2 Mountain Lion binaries should work on Maverick and Yosemite machines out of the box. (At present the source code still does not compile up on these newer Mac Operating Systems.)

However, the GS3.07rc1 Mac Mountain Lion binary still requires you to additionally follow the instructions below, if you wish to run it on a Mac Maverick or Mac Yosemite.

You will need to ensure that Java is installed for GS3.07rc1.

Installation on Maverick/Yosemite

In order to get the 3.07 rc1 Mac Mountain Lion binary to work on the Mac Maverick and Yosemite:

If the dmg file opens alright (see notes further above), but double-clicking on the Greenstone 3 installer icon does not launch the installer, you can run the installer manually as follows:

1. The installer file that is inside the dmg may end with a .app extension. Copy that installer file to a different location on your file system. E.g. /Users/me/Greenstone-3.07rc1.app

2. Make sure to have JAVA_HOME set on your path.

If it's not set, then first determine JAVA_HOME by running the Terminal application on your mac and typing:

ls -la /usr/bin/java

Hit enter. It may display something like

lrwxr-xr-x  1 root  wheel  74 Aug 12  2011 /usr/bin/java -> /System/Library/Frameworks/JavaVM.framework/Versions/Current/Commands/java

Now copy the text after the arrow (→) symbol and paste it in the terminal, suffixing "_home" to it.

/System/Library/Frameworks/JavaVM.framework/Versions/Current/Commands/java_home

Then hit enter. In our case, it displayed: /System/Library/Java/JavaVirtualMachines/1.6.0.jdk/Contents/Home

Copy the value that's displayed in your case: it will be your JAVA_HOME.

Next, create the JAVA_HOME environment variable using what you copied as follows. Note that the following example shows what we had to do in our case. Make sure to paste the JAVA_HOME value you found after the equals sign:

export JAVA_HOME=/System/Library/Java/JavaVirtualMachines/1.6.0.jdk/Contents/Home

3. Having set up JAVA_HOME, you can now at last run the Greenstone .app installer file. First use the terminal to change directory, cd, into the folder containing the Greenstone .app installer which you had copied in step 1 above.

cd /<here/type/the/full/path/to/the/folder/containing/Greenstone3.07rc1.app>

Hit enter. Finally, type:

java -jar Greenstone3.07rc1.app/Contents/Resources/Java/Greenstone-3.07rc1-MacOS-intel.jar

The above will run the Greenstone 3.07 rc1 installer.

Running GLI on Maverick/Yosemite

Now that you've installed Greenstone 3.07 rc1, you will still need to carry out a few more steps before you an run GLI.

1. Go to your Greenstone 3.07 rc1 installation folder's "lib/darwin" subfolder and move the file "libsqlite3.dylib" up one level (into the "lib" folder). This essentially moves the file out of the way.

2. Now try launching the GLI application by double-clicking on its icon.

That should work, but if it doesn't, then open an instance of your Mac Terminal application, go into your GS3.07rc1/gli folder. Then type: ./gli.sh

* Refer to Running the installer in text-only mode

• Windows: Source Component instructions
• Linux/Mac: Source Component instructions

• Windows: Source Distribution instructions
• Linux/Mac: Source Distribution instructions

In Greenstone 3, collections should be available over OAI by default. Their collectionConfig.xml files already specify that each collection is OAI enabled, through use of an OAIPMH serviceRack element. If you want to disable a collection from being accessibile over OAI, comment out the OAIPMH serviceRack element in that collection's collectionConfig.xml. You would do so by embedding the entire element in comment markers:

  <!--
    <serviceRack name="OAIPMH">
        ...
    </serviceRack>
  -->

If you wish to validate the Greenstone 3 OAIServer, edit resources/oai/OAIConfig.xml to add in the adminEmail property to contain the email to where test results should be sent. Also set the repositoryId field to a ID name you want (e.g. to greenstone), beware that there are some naming conventions that govern valid values for repositoryID. If testing the behaviour of the resumptionToken, set the resumeAfter element to a low value like 5. Then restart the Greenstone server.

To validate your OAI server, visit http://www.openarchives.org/Register/ValidateSite. Your server must be available over the internet to do this. The machine on which you're running the Greenstone 3 server will have to have its firewall and virtual server (port-forwarding) settings set up such that the Greenstone server can be made accessible to the outside world.

Setting up your Greenstone 3 OAI Server is covered in further detail in the tutorial http://wiki.greenstone.org/gsdoc/tutorial/gs3-current/en/GS_OAI_server.htm

For further information on your Greenstone OAI Server, please read through OAI support.

This will allow remote client-GLI applications to connect to your Greenstone server, to remotely create and upload new collections to be built and hosted by your server machine.

Remote Greenstone 3 Server

To install the server-side functionality:

1. If you're on Windows, you will need to teach Greenstone where the perl executable is.

You can do this either manually, by editing a couple of Greenstone config files as explained just below, or you could run the Greenstone server once and press Enter Library button to visit your library home page. Doing so will automatically set up the perl path in various Greenstone files.

To do this manually on Windows,

a. Open Tomcat's conf/web.xml file (found in greenstone3/packages/tomcat/conf folder) for editing, as you may need to specify the full path of the Perl library for the parameter "executable" of CGIServlet. This takes the form:

<init-param>
	<param-name>executable</param-name>
        <param-value>C:\Program Files\greenstone3\gs2build\bin\windows\perl\bin\perl.exe</param-value>
</init-param>

b. Edit the first line of the greenstone3/web/WEB-INF/cgi/gliserver.pl file and specify the full path of the perl binary. On Windows this will be (if installed in the default location):

#!C:\Program Files\greenstone3\gs2build\bin\windows\perl\bin\perl -w

2. Make the Greenstone "collect" directory, located in web/sites/localsite, writeable by the webserver user.

On Unix, use chmod.

On Windows, run in a DOS prompt:

cacls "C:\Program Files\Greenstone3\web\sites\localsite\collect" /P Everyone:F

3. Open up the file build.properties located in your greenstone installation folder. Edit the tomcat.server property's value to refer to your server machine's hostname instead of leaving it at the default value of "localhost":

# tomcat info
tomcat.server=your-server-computer-name

Once the server is started up, this will update the same property in greenstone's web/WEB-INF/classes/global.properties file. Then images viewed from a browser on the client side will refer to the correct location on the remote machine.

(If you don't know what your machine's host name is and you're on Windows, then open a DOS prompt and type:

ipconfig /all

Scroll to the top of the output that gets printed to the screen and note what it mentions for HOST NAME. Also note the DNS Suffix Search List.

Put these two together with a period mark to separate them and use this as the value for your tomcat.server property.)

4. Start up your Greenstone 3 server:

ant restart

5. Check that Tomcat and Greenstone3 are working correctly by visiting

http://<your-machine-name>:<port>/greenstone3/library

6. Add some user accounts by visiting the Greenstone 3 home page (http://YOURHOST:YOURPORT/greenstone3/library), clicking the admin link at the top right and logging in. The username is admin. By default, the password is admin too, unless you already set this during the installation process or if you changed this afterwards.

Once logged in, go to the Administration page. You can access this via the link on the home page. (Or you can click the admin link, choose Account Settings, and then click the Administration Page link on the top left. )

Add a new user by providing a new username, setting a password for the user that's a minimum of 3 characters long, and by using the drop-down provided for the Groups field to one or more of the available options, such as personal-collections-editor.

Even if only the admin user wishes to use the client-gli, they will still need to log in to the Administration Page once after installation in order for the user database to be set up.

7. Finally, visit the following page in the web browser to test that your remote Greenstone server is set up properly:

http://<your-machine-name>:<port>/greenstone3/cgi-bin/gliserver.pl?cmd=check-installation

You should get a message saying "Java found" and "Installation OK!" Important: You cannot continue until this is successful, as the Remote Greenstone 3 server will not work without it!

If you get a message saying "Java failed"

• check that the Java run-time is installed and on the webserver's path. If you get a "500 Internal Server Error", check the error log of your webserver for the cause (greenstone3/web/logs/greenstone.log).
• consult the more detailed instructions at Remote Greenstone for further steps that may be necessary

Client-GLI

Assuming that the remote Greenstone server is accessible to the outside world and you're not behind a firewall, you can access the remote Greenstone server from a client-gli application installed on any other machine. To do so,

1. Run client-gli quite as you would GLI. It's accessible from the Windows Start menu, otherwise you can run the client-gli script (located at the toplevel of a Greenstone installation) from a terminal.

2. You'll be asked for the gliserver.pl URL of the remote Greenstone 3 server that you wish to connect to. This is of the form

http://<remote-gs3-server-machine-name>:<remote-port>/greenstone3/cgi-bin/gliserver.pl

It's the same URL as in Step 10 of setting up the remote GS3 server above.

The new Format Conversion Wizard to convert GS2 format statements to GS3 format statements (see this page) only appears when you're working with GLI, not client-GLI. The client-GLI for GS3 will only perform the most basic initial step in the conversion process, which is to preserve the GS2 format statements in inactive XML tags in the new collection's collectionConfig.xml.

However, if you have a local Greenstone 3 installed, you can still manage to convert a remote collection's collect.cfg file to its GS3 equivalent. See here for details.

Release Candidate 2 (GS3.07rc2):

• The Mac Mountain Lion binary, both its greenstone installer and greenstone applications once installed, can now run out of the box on Yosemite (and we think the same to be the case with Maverick too, but this is untested). This has been achieved by now bundling a JRE with the Greenstone Mountain Lion binary, since Mac Yosemite's no longer come pre-installed with Java.
• Bugfix to search snippets by Georgy Litvinov
• The Edit Structure and Document Basket features, which are advanced and largely undocumented Greenstone 3 specific bleeding edge features, have been disabled for the present as these are not yet working properly.

Release Candidate 1 (GS3.07rc1):

Georgy Litvinov contributed the following enhancements (as well as numerous bugfixes):

• Russian morpology analyzer for solr collections
• Improved document content editor: ckeditor replaces direct-edit/seaweed
• Better search highlighting in solr collections, using the Solr Servlet
• For solr collections, search snippets are now provided for search results

Other changes:

• We've moved to a Derby Network Server (db-derby-10.1.2.1-bin) from Embedded Derby for managing the user database.
• Solr changes
  • Moved from using an EmbeddedSolrServer to an HttpSolrServer.
  • Greenstone 3's solr no longer uses its own jetty server but shares Greenstone 3's tomcat server. Now http://localhost:8383/solr hosts the solr server RESTful pages. Access is limited to localhost/host machine IP.
  • getTerms() works again after all these changes to allow search highlighting. Its functionality was previously implemented for the EmbeddedSolrServer, and has now been re-implemented for HttpSolrServer.
• The Greenstone Installer now supports as many languages as there are (near) complete translations of the installer for. Previously it only supported a limited number of languages.
• Improvement to the way OAI metadata mappings are specified in collectionConfig.xml
• Moved from tomcat 7.0.26 to tomcat 7.0.57. Tomcat 7.0.57 is necessary for the running server to work with JDK 8. Tomcat 7.0.57 still seems to work with both JDK 6 and 7 on linux (not tested with JDK 5).
• For the purpose of creating package managers greenstone for GS3, servlets previously defined in web.xml have now been moved into a separate file, servlets.xml. servlets.xml is now included in web.xml as an XML entity. As a result, customisation of greenstone library servlets is also made easier.
• Modified the gs3-server dialog to have an Allow External Connections checkbox to match with GS2.
• Introduction of a toplevel gs3-devel.bat for Windows, equivalent to the existing linux gs3-devel.sh
• A separate library_name parameter added to activate.pl so that collections in a different servlet can be activated

• Introduction of new template to help with including empty 'div' elements in the GS3 XSLT processing pipeline
• The user no longer needs to be an administrator to use greenbug, but can be an all-collections-editor or collname-collection-editor instead.
• Getting sibling and children metadata for a node id implemented. i.e. siblings:Subject, or children:Subject. Relates to <gsf:metadata name="Subject" select="children"/> Descendents still to be implemented. (See this page.)
• Lucene 'sort order' param changed to 'reverse sort' param. Matches better what lucene actually does, and removes the issue of needing a different default sort order when sorting by rank vs other field. Note that SOLR still uses sort order ascending/descending.
• Removed search prefs from prefs page. They now go into the list of params on the query page
• More accurate reporting of number of docs matched in mgpp collections (gs2mgppdemo)
• More dynamic processing of downloaders in GLI.
• OAI Validation is up to date by further changes
• Added ability to link to a static page to a collection, using a=p&sa=html&c=collname&url=xxx, which will embed xxx page into collection collname.

Bugfixes and other fixes

• Fix to removeprefix and removesuffix fields issues with \ and ". For instance, don't need to escape the \ by writing
  instead anymore. Thanks to Georgy Litvinov.
• Remove jsessionid from url by using cookies (r29866). Thanks to Georgy Litvinov.
• Some more changes to make adding metadata to filenames with non latin1 characters work.
• Bugfixes to rebuilding a solr collection
• Bugfix rebuilding a collection using the online editor on Windows. Resolves the file lock problem.
• We can no longer accidentally run ant start repeatedly and thereby run two or more tomcat instances on the same host and port.
• The user must now be authenticated when setting metadata using the GS3 online metadata editor
• Improvements and fixes to paging search results
• Incremental building now works again for solr since the upgrade from lucene/solr 3.3 to 4.7.2.
• The wvware config file has been updated to convert justified text in input word documents to justified text in the html output files generated. Previously the wvware settings would convert justified text in input documents to left-aligned html text.

For ease of access this section has been brought across from the 3.06 Release Notes, but not all of it may be relevant to the 3.07 release.

When you've built a collection of documents, you may discover that there appears to be a copy of all these documents in the collection's import, archives and index subfolders and wonder whether Greenstone could really be so inefficient with space as to keep 3 copies to everything. As it happens though, Greenstone uses hard-links both on Linux and Windows, in order to keep just one set of your documents. Then it simply hardlinks to these, instead of making copies. By default, Windows doesn't show you when files on your filesystem are hard-linked. If you choose to install the Windows extension program Link Shell Extension (LSE), it will put red arrows on files that are hard linked.

Currently, even if you don't tick the 'enable admin access' button in the installer, you still get admin access, but with a default password of 'admin'. If you end up in this situation, please change the admin password. Login to the administration page, 'edit' the admin account, and click 'change password'. Alternatively, you can login as admin via the login button at the top right of each page. Once you are logged in, this button will change to say 'admin. Click this button and select 'account settings'. From there, you can select 'change password'.

See bugzilla report.

If attempting to view a java applet (like Collage or Phind phrase classifiers) crashes Firefox, then make sure you have the Java Applet plugin installed. If it is installed and Firefox is still crashing, then open firefox and visit the page

about:config

Scroll down to the property:

dom.ipc.plugins.java.enabled

Set it to true (rightclick and choose toggle).

If you've configured a PDFPlugin to convert PDFs to images, increase the verbosity in Import Options and Build Options to 5 in GLI's Create panel.

When rebuilding the collection, check to see if you encounter the following error message mentioning that 'memory allocation failed', a 'corrupt image' at 'ReadPNGImage' and 'PostScript delegate failed':

import.pl> Converting pdf05-notext.pdf to pagedimg_jpg format
import.pl> calling cmd "/usr/bin/perl" -S gsConvert.pl -verbose 5 -pdf_zoom 2 -errlog "/research/myfolder/gs3-svn-12Sep2013/web/sites/localsite/collect/Enhanced-PDF/tmp/1378957949/err.log" -output pagedimg_jpg "/research/myfolder/gs3-svn-12Sep2013/web/sites/localsite/collect/Enhanced-PDF/tmp/1378957949/pdf05-notext.pdf"
import.pl> Error executing pdfpstoimg.pl
import.pl> pdfpstoimg error log:
import.pl> convert: memory allocation failed `/tmp/magick-31829ofGIFuaNZ1xy1' @ error/png.c/ReadOnePNGImage/2160.
import.pl> convert: corrupt image `/tmp/magick-31829ofGIFuaNZ1xy1' @ error/png.c/ReadPNGImage/3794.
import.pl> convert: Postscript delegate failed `/research/myfolder/gs3-svn-12Sep2013/web/sites/localsite/collect/Enhanced-PDF/tmp/1378957949/pdf05-notext.pdf': No such file or directory @ error/pdf.c/ReadPDFImage/681.
import.pl> convert: no images defined `/research/myfolder/gs3-svn-12Sep2013/web/sites/localsite/collect/Enhanced-PDF/tmp/1378957949/pdf05-notext/pdf05-notext.jpg' @ error/convert.c/ConvertImageCommand/3068.
import.pl> Convert error for /research/myfolder/gs3-svn-12Sep2013/web/sites/localsite/collect/Enhanced-PDF/tmp/1378957949/pdf05-notext.pdf 
import.pl> Could not convert pdf05-notext.pdf to pagedimg_jpg format
import.pl> convert: memory allocation failed `/tmp/magick-31829ofGIFuaNZ1xy1' @ error/png.c/ReadOnePNGImage/2160.
import.pl> convert: corrupt image `/tmp/magick-31829ofGIFuaNZ1xy1' @ error/png.c/ReadPNGImage/3794.
import.pl> convert: Postscript delegate failed `/research/myfolder/gs3-svn-12Sep2013/web/sites/localsite/collect/Enhanced-PDF/tmp/1378957949/pdf05-notext.pdf': No such file or directory @ error/pdf.c/ReadPDFImage/681.
import.pl> convert: no images defined `/research/myfolder/gs3-svn-12Sep2013/web/sites/localsite/collect/Enhanced-PDF/tmp/1378957949/pdf05-notext/pdf05-notext.jpg' @ error/convert.c/ConvertImageCommand/3068.
import.pl> Convert error for /research/myfolder/gs3-svn-12Sep2013/web/sites/localsite/collect/Enhanced-PDF/tmp/1378957949/pdf05-notext.pdf 
import.pl> Converting pdf05-notext.pdf to html format

If you see the above error message, then:

1. Use a text editor to open your Greenstone 3's gs2build/ext/imagemagick/linux/etc/ImageMagick/delegates.xml

2. Find the line that would say:

<delegate decode="ps:alpha" stealth="True" command="&quot;gs&quot; -q -dQUIET -dSAFER -dBATCH -dNOPAUSE -dNOPROMPT -dMaxBitmap=500000000 -dAlignToPixels=0 -dGridFitTT=2 &quot;-sDEVICE=pngalpha&quot; -dTextAlphaBits=%u -dGraphicsAlphaBits=%u &quot;-r%s&quot; %s &quot;-sOutputFile=%s&quot; &quot;-f%s&quot; &quot;-f%s&quot;"/>

The above specifies the PostScript delegate for PNG images. It has the sDEVICE set to pngalpha.

3. Change the line to:

<delegate decode="ps:alpha" stealth="True" command="&quot;gs&quot; -q -dQUIET -dSAFER -dBATCH -dNOPAUSE -dNOPROMPT -dMaxBitmap=500000000 -dAlignToPixels=0 -dGridFitTT=2 &quot;-sDEVICE=**pnmraw**&quot; -dTextAlphaBits=%u -dGraphicsAlphaBits=%u &quot;-r%s&quot; %s &quot;-sOutputFile=%s&quot; &quot;-f%s&quot; &quot;-f%s&quot;"/>

The above changes the sDevice to pnmraw.

4. Save the file and re-run the build now.

Thanks to the following people for new and updated translations since 3.06:

• Tomáš Fiala for Slovak translations
• Kamal Salih Mustafa Khalafala for Arabic translations
• Reza Monajjemi for Farsi translations
• John Rose for French translations
• Diego Spano for Spanish translations
• Gaku Yamaguchi for Japanese translations
• Lavji Zala for Gujarati translations
• Tigran Zargaryan for Armenian translations