Greenstone Wiki

This page is read only. You can view the source, but not change it. Ask your administrator if you think this is wrong.
====== Greenstone 3.09 Release Notes ======

**Release Name:** 3.09

**Release Date:** unknown

  
===== Installation Instructions=====

==== Installing and running the binary release==== 

  * Download the appropriate binary release for your operating system and run it. 

  * **Note to Mac users:** The security settings in newer Mac OS versions have been altered to by default disallow users from casually opening and running .dmg executables that are not from Apple itself. When attempting to open the Greenstone binary .dmg file, if it pops up an error warning about security, you will need to set up the Security on your Mac to allow you to run .dmg files downloaded from the internet. Otherwise the Greenstone mac binary will not run. To do this, Go to your Mac's System Preferences. Under "Personal", select Security & Privacy. In the General tab, tick "Allow apps downloaded from Anywhere", then confirm that you want to "Allow From Anywhere". You'll need to be admin to do this, otherwise click on the padlock at the bottom left of the Security & Privacy tab and log in as admin.

  * **For Linux**, you will need to set the file to be executable before running it. (e.g. chmod a+x Greenstone-3.09-linux)
    * The installer initially unpacks into a temporary directory (/tmp on linux). If you wish to change this, set the TMPDIR environment variable. 
Note that in some cases, the following **doesn't** work
<code> export TMPDIR=/something/else
 ./Greenstone-3.09-linux</code>

Use the following instead
<code> TMPDIR=/something/else ./Greenstone-3.09-linux</code>

During the installation process you will be presented with several options. For many, the default settings will be sufficient. Some important options are
  * Folder where you want greenstone3 to be installed. 
  * Choosing which packages to install.
    * Greenstone3 will install the Apache Tomcat webserver by default. You can choose not to install it, but then you will need to set up your own version of Tomcat to serve Greenstone. We recommend using Greenstone's Tomcat, at least initially while you get everything set up.
    * ImageMagick is bundled with Greenstone for binary web releases for all platforms, and includes JPEG2000 support. You can choose not to install it if you already have ImageMagick previously installed.
    * Ghostscript is now bundled with Greenstone for binary web releases for Windows and Mac. You can choose not to install it if you already have Ghostscript previously installed.
  * Choosing a password for the administration pages. These pages allow the admin user to inspect and manage the list of registered Greenstone users. You can add new users, and change group settings for existing users. Greenstone user registration is needed if you want to use remote GLI login to the Greenstone server, or if you want to make collections/documents only accessible by certain groups of users. (//If the password is not set during installation, the default password for the 'admin' user is 'admin'. You can change the password any time after installation, by running the Greenstone 3 server and visiting the Administration pages. See [[#disabling_admin_access_in_installer| below]]//). 

Once you have successfully installed Greenstone3, you can start up the server by choosing Grenstone3 Digital Library from the Start menu (Windows) or running gs3-server.sh/bat. This launches a small server program which starts up Tomcat and launches a browser. A small window pops up which allows you to change some settings for your library and restart the Tomcat server. Closing this program will stop Tomcat running. By default, your library will be available at localhost:8383/greenstone3/library. File->Settings in the Greenstone Server window gives you options to change the port number and which browser it uses by default.
More notes about running Greenstone can be found in the README.txt file in the top level Greenstone folder.

To build collections, run GLI from the Start menu (Windows) or by running gli/gli.sh/bat in the top level Greenstone3 folder. Tutorial exercises about building collections in Greenstone 3 can be found [[en:tutorials | here]]. Make sure you select the Greenstone3 tab at the top if it is not already selected.

<!--
=== Installing and running GS3.09 on Mac Maverick and Yosemite machines ===
From GS3.07rc**2** onwards, we're including a JRE with Mac Mountain Lion binaries, so that the **3.09 Mountain Lion binaries should work on Maverick and Yosemite machines out of the box.** (At present the source code still does not compile up on these newer Mac Operating Systems.)
-->

==== Installing in text-only mode ==== 
* Refer to [[http://wiki.greenstone.org/doku.php?id=en:user_advanced:installation#running_the_installer_in_text-only_mode|Running the installer in text-only mode]]


==== Adding source code to a binary release ==== 

  * [[http://wiki.greenstone.org/doku.php?id=en:user_advanced:installation#source_component|Windows: Source Component instructions]]
  * [[http://wiki.greenstone.org/doku.php?id=en:user_advanced:installation#source_component1|Linux/Mac: Source Component instructions]]
==== Installing a source release ==== 

  * [[http://wiki.greenstone.org/doku.php?id=en:user_advanced:installation#source_distribution|Windows: Source Distribution instructions]]
  * [[http://wiki.greenstone.org/doku.php?id=en:user_advanced:installation#source_distribution1|Linux/Mac: Source Distribution instructions]]


===== Further instructions =====
==== Setting up your Greenstone to run over https ====
The more secure https protocol is increasingly required by browsers and gradually superseding http. Given that you meet the following requirements and configure your GS3 as below, Greenstone 3 has now been automated to obtain an https certificate for you from the free Certification Authority "Let's Encrypt".

Requirements: because we need to temporarily run a server on port 80 to get a certificate issued and because port 80 has some access restrictions surrounding it on most machines,
  * on unix (linux and mac) systems you need to have sudo permissions
  * on windows, you probably need admin rights
  * ensure nothing is running on port 80 when you're ready to set up https certification your GS3 

Steps:
  - Edit build.properties as follows:
     * set ''tomcat.server'' to the //primary// hostname/domain name that you want your Greenstone3 to run as and which is to be registered in your certificate. This would be the host name of your machine.
     * set a value for ''keystore.pass''.\\ This will be the password on your final certificate used by tomcat.
     * Ensure ''server.protocols'' contains ''https''.\\ The ''server.protocols'' property is a comma-separated list that indicates which protocols are to be supported by your Greenstone 3 server. This property can be set to one of ''http'', ''https'', ''http, https'' or ''https, http''. The first in the list becomes the default protocol used for previewing with the GS3 server application, ''gs3-server''.
     * By default ''tomcat.port.https'' is set to 8443. Ensure this port is not already in use, otherwise change it to a port value that's not in use.
  - Make sure you have read and agree with the [[https://letsencrypt.org/documents/LE-SA-v1.2-November-15-2017.pdf|Let's Encrypt Subscriber Agreement]]
  - Use a terminal to go into your GS3 installation folder, run ''gs3-setup.bat'' on windows and ''source ./gs3-setup.sh'' on linux and mac to set up the GS3 environment, then run the ''ant setup-https-cert'' target. For example on Linux,\\ <code>cd /path/to/GS3
source ./gs3-setup.sh
ant setup-https-cert
</code>\\ You'll be asked for an **email** that Let's Encrypt can optionally communicate to you on, as well as **any additional domain names** you want in the //same// certificate (additional domains are **untested**), and whether you **agree** with the Let's Encrypt Subscriber Agreement.\\ On linux or mac, you may be asked to provide your sudo password to run a server on port 80. (On Mac and windows, GS3 uses [[https://zerossl.com/usage.html|ZeroSSL]] to get Let's Encrypt to issue certificates, which will result in GS3's own tomcat server to be run on port 80 during certificate issuance. On Linux, we use //Let's Encrypt//'s own certbot-auto script for the certification process, and have it set to run a standalone temporary server on port 80.)
  - Once the setup-https-cert ant target has finished, you can start your web GS3 server by either running the gs3-server application or by running "ant start" from the terminal.
  - If you ran the gs3-server application, press the Enter Library button to open your DL home page. If you ran ''ant start'' from the command line, then open a browser manually. Point your browser to ''https:%%//%%<tomcat.server>:<tomcat.port.https>/greenstone3/library'', adjusting the tomcat.server and tomcat.port.https values as per what you set for thse properties in your GS3 installation folder's toplevel ''build.properties'' file.
  - Once your https home page has loaded, confirm that your certificate is properly installed by looking for a green padlock next to the address bar. (Depending on your browser, you can click the padlock to get more information on the certificate issuer.)

There are 2 more https-related automated ant targets you can run from the command line:
  * ''ant remove-https-cert'': to revoke your https certificate
  * ''ant renew-existing-https-cert'': This is to renew a certificate that you'd already earlier obtained with ''ant setup-https-cert'' explained above. A Let's Encrypt certificate needs renewing every 90 days, at which point your certificate will need to be reinstalled. For renewal, you will once more need to ensure all the same conditions as for issuance (the same conditions as when you ran ''ant setup-https-cert''), such as nothing running on port 80. Since renewal reinstall your certificate, you will need to stop your GS3 server first before running the ''ant renew-existing-https-cert'' target, then after the target has finished, run your GS3 server once more. Renewal will not take place despite running the ''ant renew-existing-https-cert'' target unless the approximate time for expiry has been reached (+/- 10 days on Windows/Mac).

**Important:** Beware that if you've configured your GS3 to support http and https, by setting the ''server.protocols'' property to include both http and https, then switching between the two protocols when you visit your GS3 pages in your browser could result in the //http// variants of GS3 web pages not remembering you when you log in to them. For the solution and for further details, consult the section [[en:release:3.09_release_notes#troubleshooting|Troubleshooting > Your browser doesn't remember you being logged into greenstone]]. 


==== PDF plugin restructuring and the NEW PDFv2Plugin ====

From GS3.09 onward, the GS3 binaries will henceforth include additional tools for converting from PDF to various text/html/image/image+text formats. (For GS2, only the nightly binaries at http://www.greenstone.org/caveat-emptor/?latest=latest will contain these changes.)

We're deprecating the old "PDFPlugin". And in its place there will be 2 plugins to handle PDFs:
  * "//PDFv1Plugin//" which is the same as the old PDFPlugin but minus the PDFBox_conversion option. It returns to using the old ''pdftohtml'' tool to do the conversions, and is limited to older versions of PDFs.
  * **the recommended "//PDFv2Plugin//"**, which will contain the new functionality and should handle a greater range of PDF versions, including the newer ones that the old ''pdftohtml'' (now used by PDFv1Plugin) can't handle. The "PDFBox conversion" facility has been moved to the new PDFv2Plugin, but is now invisible: it will be triggered automatically depending on the "convert_to" format that you select when you Configure the PDFv2Plugin. PDFv2Plugin also uses additional conversion tools in the background to support the additional output formats.

For the eventual 3.09 release, the old PDFPlugin that you're familiar with, the one which has the ''pdfbox_conversion'' flag but also makes use of the old ''pdftohtml'' tool behind the scenes, will hang around with a deprecated warning, to allow people to port over their collections and keep rebuilding with the old settings or to rebuild their collection with one of the 2 new PDF plugins. However, **new collections will have the //PDFv2Plugin// in the Document Plugins pipeline by default, for GS3, and PDFv1Plugin by default for GS2, since GS2 doesn't come with the PDFbox extension out of the box.** So GS2 users will have to manually add in PDFv2Plugin in place of PDFv1Plugin for new collections, after setting up the pdfbox extension. But then it should work as usual.


The "convert_to" options/output formats of the new PDFv2Plugin are:
  * ''text'': a single stream of text;
  * ''html'': a single stream of basic html from just the extracted text, no images;
  * ''pretty_html'': each page is now an HTML page consisting of extracted text overlaid on top of a screenshot of the rest of the PDF page;
  * ''paged_pretty_html'' (also the default when convert_to is set to auto): ''pretty_html'', but each page is a section;
  * ''pagedimg_<png|jpg>'': every PDF page as an image, sectionalised by page. Not searchable, since there's only images;
  * ''pagedimgtxt_<png|jpg>'': every PDF page as an image plus that page's extracted text, sectionalised by page.

As always, text is only extracted from a PDF where extractable. This depends on user permissions for a PDF, whether the PDF contains actual extractable text and not just images of text, whether the PDF is undamaged, and any other such factors.

There may be further adjustments made, including to display strings, but so far, we've decided on the above output formats and they seem to work on my regular PDF test documents. 
====Changing the admin password====
Login to the administration page, 'edit' the admin account, and click 'change password'. Alternatively, you can login as admin via the login button at the top right of each page. Once you are logged in, this button will change to say 'admin'. Click this button and select 'account settings'. From there, you can select 'change password'.

==== Setting up your Greenstone OAI Server and using GLI to download over OAI from a Greenstone server ==== 
In Greenstone 3, collections should be available over OAI by default. Their collectionConfig.xml files already specify that each collection is OAI enabled, through use of an ''OAIPMH serviceRack'' element. If you want to disable a collection from being accessibile over OAI, comment out the ''OAIPMH serviceRack'' element in that collection's collectionConfig.xml. You would do so by embedding the entire element in ''<!--'' and ''-->'' comment markers:
<code>
  <!--
    <serviceRack name="OAIPMH">
        ...
    </serviceRack>
  -->
</code>

If you wish to validate the Greenstone 3 OAIServer, edit **resources/oai/OAIConfig.xml** to add in the **adminEmail** property to contain the email to where test results should be sent. Also set the **repositoryId** field to a ID name you want (e.g. to ''greenstone''), beware that there are some naming conventions that govern valid values for repositoryID. If testing the behaviour of the resumptionToken, set the **resumeAfter** element to a low value like 5. Then restart the Greenstone server.

To validate your OAI server, visit http://www.openarchives.org/Register/ValidateSite. Your server must be available over the internet to do this. The machine on which you're running the Greenstone 3 server will have to have its firewall and virtual server (port-forwarding) settings set up such that the Greenstone server can be made accessible to the outside world. 

Setting up your Greenstone 3 OAI Server is covered in further detail in the tutorial http://wiki.greenstone.org/gsdoc/tutorial/gs3-current/en/GS_OAI_server.htm

For further information on your Greenstone OAI Server, please read through [[en:user_advanced:oai|OAI support]].


==== Setting up a remote Greenstone 3 server ==== 
This will allow remote client-GLI applications to connect to your Greenstone server, to remotely create and upload new collections to be built and hosted by your server machine.


**Remote Greenstone 3 Server**

To install the server-side functionality: 

1. If you're on Windows, you will need to teach Greenstone where the perl executable is.

You can do this either manually, by editing a couple of Greenstone config files as explained just below, or you could run the Greenstone server once and press ''Enter Library'' button to visit your library home page. Doing so will automatically set up the perl path in various Greenstone files.

To do this manually on Windows, 

a. Open Tomcat's conf/web.xml file for editing (found in greenstone3/packages/tomcat/conf folder, if installed in the default location), as you may need to specify the full path of the Perl library for the parameter "executable" of **CGIServlet**. This takes the form:

<code>
<init-param>
	<param-name>executable</param-name>
        <param-value>C:\Program Files\greenstone3\gs2build\bin\windows\perl\bin\perl.exe</param-value>
</init-param>
</code>

b. Edit the first line of the greenstone3/web/WEB-INF/cgi/gliserver.pl file and specify the full path of the perl binary.
On Windows this will be (if installed in the default location):

<code>
#!C:\Program Files\greenstone3\gs2build\bin\windows\perl\bin\perl -w
</code>

<!-- 2. If you're on Linux and working with a binary release, make sure that the webserver user has executable permissions on gliserver.pl:
<code>
cd greenstone3/web/WEB-INF/cgi
chmod a+rx gliserver.pl
</code> -->

2. Make the Greenstone "collect" directory, located in web/sites/localsite, writeable by the webserver user.

On Unix, use //chmod//.

On Windows, run in a DOS prompt:

<code>
cacls "C:\Program Files\Greenstone3\web\sites\localsite\collect" /P Everyone:F
</code>

3. Open up the file build.properties located in your greenstone installation folder. Edit the //tomcat.server// property's value to refer to your server machine's hostname instead of leaving it at the default value of "localhost":
<code>
# tomcat info
tomcat.server=your-server-computer-name
</code>

Once the server is started up, this will update the same property in greenstone's web/WEB-INF/classes/global.properties file. Then images viewed from a browser on the client side will refer to the correct location on the remote machine.


(If you don't know what your machine's host name is and you're on Windows, then open a DOS prompt and type:
<code>ipconfig /all</code>
Scroll to the top of the output that gets printed to the screen and note what it mentions for //HOST NAME//. Also note the //DNS Suffix Search List//.

Put these two together with a period mark to separate them and use this as the value for your tomcat.server property.)

4. Set up your Greenstone environment if you've not already done so by running ''gs3-setup.bat'' in your Greenstone 3 installation folder (''source ./gs3-setup.bash'' on Linux and Mac machines). And then start up your Greenstone 3 server with:
<code>
ant start
</code>
(or ''ant restart'')

5. Check that Tomcat and Greenstone3 are working correctly by visiting
<code>http://<your-machine-name>:<port>/greenstone3/library
</code>

6. Add some user accounts by visiting the Greenstone 3 home page (http://YOURHOST:YOURPORT/greenstone3/library), clicking the ''admin'' link at the top right and logging in. The username is ''admin''. By default, the password is ''admin'' too, unless you already set this during the installation process or if you changed this afterwards. 

Once logged in, go to the Administration page. You can access this via the link on the home page. (Or you can click the admin link, choose ''Account Settings'', and then click the ''Administration Page'' link on the top left. )

Add a new user by providing a new username, setting a password for the user that's a minimum of 3 characters long, and by using the drop-down provided for the ''Groups'' field to one or more of the available options, such as ''personal-collections-editor''.

Even if only the ''admin'' user wishes to use the client-gli, they will still need to log in to the Administration Page once after installation in order for the user database to be set up. 

7. Finally, visit the following page in the web browser to test that your remote Greenstone server is set up properly:
<code>http://<your-machine-name>:<port>/greenstone3/cgi-bin/gliserver.pl?cmd=check-installation
</code>
You should get a message saying "Java found" and "Installation OK!" Important: You cannot continue until this is successful, as the Remote Greenstone 3 server will not work without it!

If you get a message saying "Java failed" 
  * check that the Java run-time is installed and on the webserver's path. If you get a "500 Internal Server Error", check the error log of your webserver for the cause (greenstone3/web/logs/greenstone.log).
  * consult the more detailed instructions on the [[en:user_advanced:remote_greenstone|Remote Greenstone page]] for further steps that may be necessary to carry out.


**Client-GLI**

Assuming that the remote Greenstone server is accessible to the outside world and you're not behind a firewall, you can access the remote Greenstone server from a client-gli application installed on any other machine. To do so, 

1. Run client-gli quite as you would GLI. It's accessible from the Windows Start menu, otherwise you can run the client-gli script (located at the toplevel of a Greenstone installation) from a terminal.

2. You'll be asked for the gliserver.pl URL of the remote Greenstone 3 server that you wish to connect to. This is of the form 
<code>http://<remote-gs3-server-machine-name>:<remote-port>/greenstone3/cgi-bin/gliserver.pl</code>
It's the same URL as in Step 10 of setting up the remote GS3 server above.


===GLI Java Web Start application (replacement for GLI Applet) - Additional Steps===
Many browsers have stopped supporting Java applets, while Microsoft's Internet Explorer, and perhaps Microsoft Edge, still support it. For this reason, 3.09's GLI is now no longer provided as an applet, but has been converted into a Java Web Start application.

Instructions for using the GLI Java Web Start, which works over the JNLP protocol, are below.

  - First follow the instructions above for [[http://wiki.greenstone.org/doku.php?id=en:release:3.09_release_notes#setting_up_a_remote_greenstone_3_server | setting up the Remote GS Server]]
  - Next, generate the SignedGatherer.jar:
      * Use a terminal to go into the Greenstone "gli" directory, then run <code>    keytool -genkey -alias privateKey -keystore appletstore -storepass greenstone</code> Enter the appropriate details for your organization. When it asks to enter the key password for <privateKey>, choose your own password or hit Enter to use "greenstone".
      * Next, run <code>    jarsigner -keystore appletstore -signedjar SignedGatherer.jar GLI.jar privateKey</code>
When it prompts, enter the password you used above.
  - Move the created ''SignedGatherer.jar'' file from the ''gli'' directory into GS3's ''web/applet'' subdirectory.
  - You need to associate JNLP files with Java Web Start (''jre/bin/javaws'').
      * On Windows, create the association in the usual way: when you first access the GLI Web Start application through Greenstone, a JNLP file called "GLIappWebStart.jnlp" will be offered for launching or download. If JavaWS is not already the default application to open JNLP files with, rightclick on the downloaded GLIappWebStart.jnlp file and choose ''Launch/Open with ...''. Browse to your Greenstone3's ''packages/jre/bin/javaws.exe'' or any installed Java's ''jre/bin/javaws.exe'' to use Java's Web Start application as the launcher.
      * For Linux, create a file with ''.desktop'' extension (e.g. "javawebstart.desktop") containing the following and save this file into ''~/.local/share/applications'':\\ <code># This file makes Ubuntu associate .jnlp files with Java Web Start (javaws)
# This file should be adjusted and then copied into ~/.local/share/applications
# as a file with .desktop extension, e.g. javawebstart.desktop
# https://askubuntu.com/questions/235861/how-to-associate-jnlp-file-with-javaws
[Desktop Entry]
Encoding=UTF-8
Name=Java 7 Web Start
Comment=Java 7 Web Start
Exec=/home/greenstone/Desktop/linux/jre/bin/javaws %u
Terminal=false
Type=Application
Icon=javaws
Categories=Application;Network;
MimeType=application/x-java-jnlp-file;
</code>
  - Launch the Java Control Panel by running ''jre/bin/javacpl.exe'' on Windows or ''jre/bin/ControlPanel'' on Linux. (GS3 binaries now include a JRE in the ''packages'' folder if you want to use the bundled JRE.) In the Java Control Panel, go to the ''Security'' tab, set Security level to ''High'' if not already set. Click ''Edit Site List'', and then press ''Add'' to add the //host:port// that the GS3 will run on. Remember, to be accessible to the outside world, the host can't be "localhost", but would be the hostname of your machine or public IP.
  - Once the GS3 code is compiled up, start up the GS3 web server and visit your DL library home page, ''http://[hostname]:8383/greenstone3/library''.
  - Since you have set up the JNLP file association in a previous step, you can now click on the "The Librarian Interface" link and your browser should offer to save or launch a file called ''GLIapplet.jnlp''. If the browser is able to successfully launch it, Java Web Start will be used to run the GLI application indicated by the JNLP file. If launching through the browser is not possible, then choose to save the JNLP file. It will download the file to a temporary user area (like C:\Users\<user>\AppData\Local\Temp on windows). And then you can rightclick on the downloaded GLIappWebStart.jnlp file, to launch it with the Java Web Start program you already associated with this file type.
  - After authorising the GLI to run, the JNLP version of the GLI Applet will eventually run and behave like the usual client-GLI (and like the old GLI applet) from this point onward. 


=== Converting a GS2 collection to GS3 when working with a remote GS3 server ===
The new Format Conversion Wizard to convert GS2 format statements to GS3 format statements (see [[en:user:gs2_to_gs3 |this page]]) only appears when you're working with GLI, not client-GLI. The client-GLI for GS3 will only perform the most basic initial step in the conversion process, which is to preserve the GS2 format statements in inactive XML tags in the new collection's collectionConfig.xml.

However, if you have a local Greenstone 3 installed, you can still manage to convert a remote collection's ''collect.cfg'' file to its GS3 equivalent. See [[en:user:gs2_to_gs3#using_a_remote_greenstone_server |here]] for details.


===== Important Changes and Bug Fixes =====
  * **HTTPS support:** Greenstone will obtain a certificate from the Certification Authority Let's Encrypt to run your GS3 tomcat over https. However, on unix systems (macs and linux), you will need to have sudo permissions. And on Windows you will probably need admin rights. For instructions on usage, see [[#setting_up_your_greenstone_to_run_over_https|Setting up your Greenstone to run over https]]
  * **GreenstoneSQLPlugin/-out:** used in place of GreenstoneXMLPlugin/-out to write metadata and/or fulltext into a MySQL database instead of Greenstone doc.xml files. You can then use SQL statements to mass-edit metadata/fulltext and rebuild your collection with the modified metadata/fulltext. See the wiki page on [[http://wiki.greenstone.org/doku.php?id=en:user_advanced:greenstonesqlplugs|Using the GreenstoneSQLPlugout with GreenstoneSQLPlugin]].
  * **The UnknownConverterPlugin:** if you have a command line tool installed that can convert from a document format to text or html (or png/jpg/gif images) and which you're able to successfully run from the command line to do such a conversion, then you can configure the new UnknownConverterPlugin to launch that command line tool and run the conversion automatically. This will allows document formats unrecognised by other Greenstone plugins to have their full text extracted and made searchable in Greenstone. There is a tutorial for Greenstone 3 that covers how to use the UnknownConverterPlugin.
  * **User comments** are now supported in GS3 as well. Refer to [[http://wiki.greenstone.org/doku.php?id=en:user:user_comments|Enabling user comments]]
  * OAI deletion policy
  * Better way to run processes from GLI will avoid some occasional and unexpected errors when GLI runs perl scripts
  * Bug fixes to file locking issues on Windows when using Lucene as indexer
  * Patch to SOLR extension to circumvent SIGPIPE errors on large collections
  * Patches to perl code upgrading perl syntax to work with newer versions of perl
===== IMPORTANT information =====

For ease of access this section has been brought across from the [[en:release:3.08_release_notes|3.08 Release Notes]], but not all of it may be relevant to the 3.09 release.

==== Troubleshooting ====

=== Content Encoding Error when visiting the local solr servlet page === 
If you see a Content Encoding Error when opening your GS3's solr servlet page at ''http://127.0.0.1:8383/solr'' or ''http://localhost:8383/solr'' in your browser, then this may have to do with the version of Java you have installed on your machine. From GS3.09 onward, if your machine has its own Java installed, then assuming that its version is sufficient and its bit-architecture (32 or 64 bit) matches, Greenstone will use your Java in preference to the bundled Java Runtime (JRE) that Greenstone ships with. We found that a recent version of Java (version 1.8.0_161 was problematic for us), caused the Content Encoding Error when visiting the solr servlet, whereas the bundled JRE and slightly earlier and much newer versions of Java such as 1.8.0_144 and 1.8.0_191 did not have these issues.

**Solution:** if you have a problematic version of Java installed, 
- either unset JAVA_HOME and remove this Java's ''bin'' folder from the PATH environment variable too, thus helping Greenstone 3 use its bundled JRE instead
- install a newer version of Java on your system. We found that the current latest one, 1.8.0_191 worked successfully for this purpose.

=== SIGPIPE errors when building a collection ===
We've added a work around to one kind of SIGPIPE errors which could occur with large collections when using ''solr'' as indexer. However, a couple of people on the mailing list encountered SIGPIPE errors on occasions when solr was not the indexer. **If your collection is using ''jdbm'' as the database type** and the error messages surrounding the SIGPIPE mention issues with "transaction commit", then Mariana Pichinini on the mailing list found that the following helped:
  * change the database type from ''jdbm'' to ''gdbm''
  * or leave the database type at ''jdbm'' and move your GS3's bundled JRE (the GS3's ''packages/jre'' subfolder) outside your GS3 installation. Next install a newer Java on your system so that GS3.08 can find that. If on Linux, ensure you open a new terminal before running GLI or command line building your collection.
=== Your browser doesn't remember you being logged into greenstone ===
**The issue:**
The following scenario can occur if you set up GS3 with https, and your server.protocols property in build.properties contains both ''http'' and ''https'' (i.e. you have ''server.protocols=http,https'' or ''server.protocols=https,http'').

Switching between visiting your Greenstone 3 digital library (DL) using http and https URLs can result in the http version of the pages not remembering your login details despite you logging in. This can happen if you ever started off with the https version of the URL to a Greenstone3 DL page and then moved to using the http version of your GS3 URL, or if you ever logged in to your GS3 over https and then attempt to log in later using http.

**The solution:**
The solution is to either start a private window if you want to access your GS3 DL pages over http, or to first clear your browser cookies related to your GS3 DL before swapping from https to http.

**The cause:**
Using https causes session cookies to have the secure flag set to true. When a session cookie has the secure flag thus set, non http URLs cannot return that cookie in their subsequent requests to the server. Only https URLs can. See https://developer.mozilla.org/en-US/docs/Web/HTTP/Cookies section "Secure and HttpOnly cookies" which states "A secure cookie is only sent to the server with an encrypted request over the HTTPS protocol" and https://stackoverflow.com/questions/2321224/cookie-across-http-and-https-in-php\\ It further seems that in http mode, the browser does not want to overwrite secure cookies created in https mode with new cookies sent by the server when using http mode. Thus after using https and acquiring secure session cookies, the server can no longer track a user's session when they switch to http until the cookies are cleared either explicitly or through opening a private window.

<!--
If you're on firefox and you just logged in to to a running Greenstone 3 digital library (DL), but visiting subsequent pages in the DL shows you that it has forgotten you're logged in, then you're probably encountering a restriction that your firefox browser has. 

**NOTE:** We've encountered this problem on Firefox and only for http URLs that use the domain/machine host name for the tomcat.server property, whereas https URLs worked fine in remembering Greenstone logins, as did http URLs using localhost/127.0.0.1.

To confirm that it only happens on firefox, try another browser first. For instance, Chrome. If your being logged in is being remembered on Chrome, then you've confirmed that the phenomenon is firefox-specific.

Things to try:
  * Close all tabs in firefox and restart firefox in the usual way. Try logging in and visiting other pages to see if it remembers your login now.
  *  If that made no difference, try launching a Private Window (Ctrl + Shift + P on firefox), visit your GS3 digital library and login again, then check whether it's being remembered across Greenstone pages now.
  * If that also made no difference then the most likely cause is Firefox plugins or extensions or addons, or maybe its hardware acceleration feature needs to be switched off. To find out if any of this is the case, quit firefox once more by first close all tabs in firefox and quit it. Then try re-launching Firefox in Safe Mode as explained at https://support.mozilla.org/en-US/questions/1213229

> [[https://support.mozilla.org/en-US/kb/troubleshoot-firefox-issues-using-safe-mode|Start Firefox in Safe Mode]] by holding down the <Shift> (Mac=Options) key, and then starting Firefox.
> A small dialog should appear. Click Start In Safe Mode (not Refresh). Is the problem still there? 

(If that doesn't work on Linux, firefox can be launched in safe mode from a terminal by typing ''firefox -safe-mode''.)

After relaunching Firefox in Safe Mode, test whether your login details are being remembered this time. If it works now, it could indeed be an addon/extension/plugin or the hardware acceleration feature. Follow the suggestions and instructions at https://support.mozilla.org/en-US/questions/1213229 and https://support.mozilla.org/en-US/kb/forum-response-disable-hardware-acceleration to narrow down which of these it is.
-->

=== Mac Installer fails ===
If running the Mac installer results in a message about having failed to copy to ''/var/folders/zz/'' ''permission denied''), then restarting the Mac followed by running the installer once again worked for us, resulting in a successful installation.

=== Using the command line to reset the admin password when it isn't recognised ===
If at any stage, your admin password is no longer recognised from the Greenstone Reader interface (the Greenstone pages that you view through the web browser), then you can try to reset your admin password through the command line.

1. On Linux and Mac, open a terminal and use it to navigate to your Greenstone 3 installation folder.

On Windows, you can either [[http://wiki.greenstone.org/doku.php?id=old:miscellaneous_questions&s[]=dos#how_do_i_open_a_terminal_called_dos_console_in_windows | open a DOS terminal the usual way]] and then likewise use it to navigate to your Greenstone 3 installation folder (using the ''cd'' command). Or, you could open a Windows File Explorer first and use it to navigate to your Greenstone 3 installation. Then you could easily get a DOS prompt at that File Explorer location, as explained at [[https://stackoverflow.com/questions/378319/windows-explorer-command-prompt-here/379804 | stackoverflow]]:
> Hold Shift while Right-Clicking a blank space in the desired folder to bring up a more verbose context menu. One of the options is //Open Command Window Here//. This works in Windows Vista, 7, 8, and 10.

2. Now that your terminal is at the GS3 installation folder, you can type the following command in the terminal:
<code>ant config-admin</code>

4. It will ask for a new admin password. Type the new password and hit enter.

5. Once you restart the server, try out your new password.

==== Useful information ====
When you've built a collection of documents, you may discover that there appears to be a copy of all these documents in the collection's //import//, //archives// and //index// subfolders and wonder whether Greenstone could really be so  inefficient with space as to keep 3 copies to everything. As it happens though, Greenstone uses **hard-links** both on Linux and **Windows**, in order to keep just one set of your documents. Then it simply hardlinks to these, instead of making copies.
By default, Windows doesn't show you when files on your filesystem are hard-linked. If you choose to install the Windows extension program [[http://schinagl.priv.at/nt/hardlinkshellext/hardlinkshellext.html|Link Shell Extension (LSE)]], it will put red arrows on files that are hard linked.

====Known Issues====

=== Changes to Tomcat port affects Solr collections ===

Currently when you change your tomcat port (either in build.properties, or using File->Settings in the Server program) the changes won't propagate to Solr which will still try to use the old port number.
If you make changes to Tomcat port, please shutdown and restart the server from the Start Menu.
If you are starting from within a terminal, you will need to shut down Greenstone and restart it from a fresh terminal.

=== Web document editor requires perl CGI module ===

Editing documents through the web interface requires your system perl to have CGI installed. Without CGI, you can edit metadata and save, but your changes won't have been applied. You can tell this is the case by looking at the build log files in the collection, e.g. //greenstone3/web/sites/localsite/collect/<collname>/log/build_log.1480553047299.txt//. The message might say something like

<code>
Document Editor Build 
Command = /bin/perl -S /greenstone/pei-jones/web/WEB-INF/cgi/metadata-server.pl 
Content-type:text/plain

ERROR: Can't locate CGI.pm in @INC (@INC contains: /usr/local/lib64/perl5 /usr/local/share/perl5 /usr/lib64/perl5/vendor_perl /usr/share/perl5/vendor_perl /usr/lib64/perl5 /usr/share/perl5 .) at ./gsdlCGI.pm line 9.
BEGIN failed--compilation aborted at ./gsdlCGI.pm line 9.
Compilation failed in require at (eval 1) line 1.
</code>

If you get this error, then please install the CGI module for your system perl.
====Work Arounds====
===Filenames in collections should not contain &, < and >===
In order to have better handling of different file and filename encodings, a [[http://trac.greenstone.org/changeset/29793|sacrifice]] [[http://trac.greenstone.org/timeline?from=19.03.2015&daysback=30&ticket=on&changeset=on&milestone=on&wiki=on&update=Update|was made]] of being unable to support &, < and > characters in filenames. Filenames with such characters will make metadata.xml invalid, as a result of which, GLI won't be able to reload the metadata therein.

A workaround is to use "and" in place of "&" in filenames. And either use different characters for < and >, such as [ and ] or ( and ), or leave them out from filenames.
===Greenstone applets (Phind, Collage) crash Firefox===
See [[https://bugzilla.redhat.com/show_bug.cgi?id=789959|bugzilla report]].

If attempting to view a java applet (like Collage or Phind phrase classifiers) crashes Firefox, then make sure you have the Java Applet plugin installed. If it is installed and Firefox is still crashing, then open firefox and visit the page
<code>about:config</code>
Scroll down to the property:
<code>dom.ipc.plugins.java.enabled</code>
Set it to true (rightclick and choose //toggle//).

===PDF to image conversion error on Linux===
If you've configured a PDFPlugin to convert PDFs to images, increase the verbosity in Import Options and Build Options to 5 in GLI's Create panel.

When rebuilding the collection, check to see if you encounter the following error message mentioning that 'memory allocation failed', a 'corrupt image' at 'ReadPNGImage' and 'PostScript delegate failed':
<code>import.pl> Converting pdf05-notext.pdf to pagedimg_jpg format
import.pl> calling cmd "/usr/bin/perl" -S gsConvert.pl -verbose 5 -pdf_zoom 2 -errlog "/research/myfolder/gs3-svn-12Sep2013/web/sites/localsite/collect/Enhanced-PDF/tmp/1378957949/err.log" -output pagedimg_jpg "/research/myfolder/gs3-svn-12Sep2013/web/sites/localsite/collect/Enhanced-PDF/tmp/1378957949/pdf05-notext.pdf"
import.pl> Error executing pdfpstoimg.pl
import.pl> pdfpstoimg error log:
import.pl> convert: memory allocation failed `/tmp/magick-31829ofGIFuaNZ1xy1' @ error/png.c/ReadOnePNGImage/2160.
import.pl> convert: corrupt image `/tmp/magick-31829ofGIFuaNZ1xy1' @ error/png.c/ReadPNGImage/3794.
import.pl> convert: Postscript delegate failed `/research/myfolder/gs3-svn-12Sep2013/web/sites/localsite/collect/Enhanced-PDF/tmp/1378957949/pdf05-notext.pdf': No such file or directory @ error/pdf.c/ReadPDFImage/681.
import.pl> convert: no images defined `/research/myfolder/gs3-svn-12Sep2013/web/sites/localsite/collect/Enhanced-PDF/tmp/1378957949/pdf05-notext/pdf05-notext.jpg' @ error/convert.c/ConvertImageCommand/3068.
import.pl> Convert error for /research/myfolder/gs3-svn-12Sep2013/web/sites/localsite/collect/Enhanced-PDF/tmp/1378957949/pdf05-notext.pdf 
import.pl> Could not convert pdf05-notext.pdf to pagedimg_jpg format
import.pl> convert: memory allocation failed `/tmp/magick-31829ofGIFuaNZ1xy1' @ error/png.c/ReadOnePNGImage/2160.
import.pl> convert: corrupt image `/tmp/magick-31829ofGIFuaNZ1xy1' @ error/png.c/ReadPNGImage/3794.
import.pl> convert: Postscript delegate failed `/research/myfolder/gs3-svn-12Sep2013/web/sites/localsite/collect/Enhanced-PDF/tmp/1378957949/pdf05-notext.pdf': No such file or directory @ error/pdf.c/ReadPDFImage/681.
import.pl> convert: no images defined `/research/myfolder/gs3-svn-12Sep2013/web/sites/localsite/collect/Enhanced-PDF/tmp/1378957949/pdf05-notext/pdf05-notext.jpg' @ error/convert.c/ConvertImageCommand/3068.
import.pl> Convert error for /research/myfolder/gs3-svn-12Sep2013/web/sites/localsite/collect/Enhanced-PDF/tmp/1378957949/pdf05-notext.pdf 
import.pl> Converting pdf05-notext.pdf to html format
</code>

If you see the above error message, then:

1. Use a text editor to open your Greenstone 3's gs2build/ext/imagemagick/linux/etc/ImageMagick/delegates.xml

2. Find the line that would say: 
<code><delegate decode="ps:alpha" stealth="True" command="&quot;gs&quot; -q -dQUIET -dSAFER -dBATCH -dNOPAUSE -dNOPROMPT -dMaxBitmap=500000000 -dAlignToPixels=0 -dGridFitTT=2 &quot;-sDEVICE=pngalpha&quot; -dTextAlphaBits=%u -dGraphicsAlphaBits=%u &quot;-r%s&quot; %s &quot;-sOutputFile=%s&quot; &quot;-f%s&quot; &quot;-f%s&quot;"/></code>

The above specifies the PostScript delegate for PNG images. It has the sDEVICE set to **pngalpha**.

3. Change the line to:
<code><delegate decode="ps:alpha" stealth="True" command="&quot;gs&quot; -q -dQUIET -dSAFER -dBATCH -dNOPAUSE -dNOPROMPT -dMaxBitmap=500000000 -dAlignToPixels=0 -dGridFitTT=2 &quot;-sDEVICE=**pnmraw**&quot; -dTextAlphaBits=%u -dGraphicsAlphaBits=%u &quot;-r%s&quot; %s &quot;-sOutputFile=%s&quot; &quot;-f%s&quot; &quot;-f%s&quot;"/></code>

The above changes the sDevice to **pnmraw**.

4. Save the file and re-run the build now.

===== Updated Translations =====

Thanks to the following people for new and updated translations since 3.09: