Z3950 support old

An updated version for Greenstone 2.82 and above is available here.

Z39.50 is an international client/server protocol for searching bibliographic data. It can use the Internet Protocol (TCP/IP), which makes the databases on a server available from almost anywhere around the globe. It is widely used, for example, in on-line library catalogues. It allows a user to search one or more databases and retrieve the results of the query.

Greenstone has support for z3950, both as a client and a server. This support is not enabled by default, and recompilation is needed to enable it. To do this, you need the source code. If you haven't already got the source, rerun the installer, select custom, and then choose only the source code.

z3950 support in Greenstone is based around the YAZ toolkit, written by IndexData. We are currently using YAZ version 2.1.4. This is included in the Greenstone distribution without modification. Greenstone links against the libyaz.a library. We have also written yaz_zclient.h/cpp, which is based on the sample client.h/cpp. This is found in greenstone/src/colservr.

Greenstone can download records from a specified Z39.50 server from GLI. Start the Greenstone Librarian Interface. On the left-hand side of the Librarian Interface's Download panel, select Z39.50, and then specify the arguments on the right-hand side of the panel. There are five arguments:

• host: name of the Z39.50 server, for example z3950.loc.gov
• prot: port number of the server, for example 7090
• database: name of the database to which the query will be send
• find: query

The YAZ client supports the type-1 RPN query model. Both simple and advanced queries are supported, like single term query, boolean search and fielded search, etc. This page describes in detail the query syntax of YAZ. For exmaple, the following query finds records that have the phrase car history in title field @attr 1=4 "car history"

• max_records: select to limit the size of result set, by default 500 resultes will be retrieved

You can view the downloaded marc files on the Gather panel. On the left-hand side of the panel, double click the Downloaded Files folder to expand its content. The subfolders are named by the Z39.50 server url. The marc files are named as the combination of database name, query, and max_records if max_records is specified. These marc files are physically stored in a temporary cache directory.

You can build a collection using these downloaded marc files by dragging them across to the Collection section on the right-hand side of the Gather panel. MARCPlug must be included in the collection plugin list. MARCPlug has three options:

• metadata_mapping: this is the name of file that includes mapping details from MARC values to Greenstone metadata names. marctodc.txt, which reside in the site's etc directory, is used by default.
• process_exp: specifies a perl regular expression to match against filenames.
• split_exp: specifies a perl regular expression to split marc files into segments, and each segment becomes a record in subsequent processing.

SRW (Search/Retrieve Web service) is an alternative way that uses web service to search and retrieval Z39.50 repositories. It replaces the Z39.50 communications protocol with HTTP and SOAP, but still supports the Z39.50 query syntax. Search results from SRW are in XML format.

Greenstone also support download from a Z39.50 server through SRW. Go to the Download panel in Greenstone Librarian Interface and select SRW. On the right-hand side of the panel, there are five parameters, which are the same with Z39.50 download. But different host and port values should be used here. For example, to connect to the Library of Congress Z39.50 server through SRW, the following host and port should be specified:

host: http://z3950.loc.gov

port: 7090/voyager?

The downloaded records are in XML format. Here is a sample record. You can view the downloaded records on the Gather panel. They are in the same subfolder as downloading from Z39.50 above.

To build a collection using the downloaded XML files, MARCXMLPlug must be added to the collection plugin list. MARCXMLPlug has one option metadata_mapping, which is the same with MARCPlug and defaults to marctodc.txt as well.

• Linux/MacOS: in the top level greenstone directory, run

 ./configure --enable-z3950
 make
 make install

• For MacOS, you may need to add –with-gdbm=<path to gdbm> to the configure command if gdbm is not installed in a standard place
• For Greenstone, YAZ is set up to not use iconv (for character set conversion) or xml2 (for SRW and SRU). If you want to use these features, you need to add -lxml2 and/or -liconv to the link command in src/recpt (z3950server). Also, edit greenstone/packages/configure and remove the –with-xml2=no and/or –with-iconv=no bits from the yaz configure line.

• Windows:
  • needs Visual Studio 6
  • It only works with the web library, not the local library
  • In the packages\yaz directory, extract the files from yaz-2.1.4.tar.gz. You'll need to run the extraction twice, once for gzip, once for tar. You should end up with a yaz-2.1.4 directory here.
  • Edit the yaz-2.1.4/include/yaz/oid.h file and change "list" on line 256 to "greenstone_list".
  • Edit the src\recpt\z3950proto.cpp file and delete "extern" from lines 38 and 39 (yyin and yyout).
  • in the greenstone directory, run

 nmake /f win32.mak USE_Z3950=1

• To enable iconv or xml2, edit greenstone\win32.mak, and remove HAVE_ICONV=0 and/or HAVE_LIBXML2=0 from the make command for Yaz. You'll need to install these libraries.

Once Greenstone has z3950 support compiled in, it can act as a client to multiple z3950 servers. The file greenstone/etc/packages/z3950.cfg specifies a list of servers to connect to. By default, no servers are set up, although the config file comes with two (commented out) example Z39.50 servers, both for servers of the United States' Library of Congress.

Each entry consists of:

• A unique "short name" for internal use by Greenstone.
• The internet name or address of the server, and optionally the port that the server is running on if not the default port 210.
• The name of the database to search on that server.
• A string that provides a meaningful name for the "collection".
• An optional "About" string, providing some information about the database and/or server.
• Optional icon fields, which are displayed instead of the text on the front page.

The entries need only be separated by whitespace, but for the purposes of clarity the sample entry uses newlines and tabs.

There is a list at the Library Of Congress website containing some servers publicly available for testing.

• The z3950server program is installed into bin/linux (or bin/darwin, bin/windows, depending on the operating system). It can be run from there, or copied to somewhere else.
• The gsdlsite.cfg file needs to be copied from the cgi-bin directory to the directory you are running z3950server from. Check that the 'gsdlhome' entry is valid.
• 'setup' (Windows) or 'source setup.bash' (Linux/MacOS) needs to be run in the top level greenstone directory before running the server.

• By default, the server listens on localhost:9999. To change the port or address, run

z3950server tcp:server-name:port-num ** For example, 'z3950server tcp:localhost:8080' or 'z3950server tcp:kanuka.cs.waikato.ac.nz:7070'

• On windows, the yaz.dll file (packages/yaz/yaz-2.1.4/bin/yaz.dll) needs to be on your Path, or put in the same directory as the z3950server.exe.
• You can run 'z3950server -h' to see the list of options for the server.

You can test the z3950 server by connecting to it using the Greenstone z3950 client. In the greenstone/etc/packages/z3950.cfg file, add a server entry similer to the following:

 Greenstone
        kanuka.cs.waikato.ac.nz:9999
        demo
        "The demo collection via z3950"
        About "This collection contains a few records from the Humanity Development Library"

• The database name (line 3) specifies which Greenstone collection to search.

• Because of the open nature of the standard, our client may not work with some servers.
• Because of the large number of Marc fields, only the most frequently used fields have been given explicit names in the results. Furthermore, these are currently hard-coded to correspond to the USMARC field names.
• The z39.50 client will not work when fast-cgi is used!. Currently, the use of the fast-cgi package (off by default) will disable the use of the z39.50 client code for Greenstone.
• When viewing a z3950 "collection" in Greenstone, sometimes the browser may insert a large amount of space between the collection title and the navigation bar. Reload or Shift Reload should fix this problem.