Z39.50

Z39.50 is an international client/server protocol for searching bibliographic data. It can use the Internet Protocol (TCP/IP), which makes the databases on a server available from almost anywhere around the globe. It is widely used, for example, in on-line library catalogues. It allows a user to search one or more databases and retrieve the results of the query.

Greenstone 2 has support for Z39.50, both as a client and a server. GLI (in Greenstone 2 and 3) can download MARC records using Z39.50 and MARCXML records using SRU. These records can then be included in a collection. This support is enabled by default.

The Greenstone 2 run-time can act as a client to multiple Z39.50 servers, and a Z39.50 server program is also available. This support is not enabled by default, and recompilation is needed to enable it. To do this, you need the source code. If you don't already have the source, download the version of the source code component that matches the distribution you are using.

Z39.50 support in Greenstone is based around the YAZ toolkit, written by IndexData. We are currently using YAZ version 2.1.4. This is included in the Greenstone 2 distribution without modification. Greenstone links against the libyaz.a library. We have also written yaz_zclient.h/cpp, which is based on the sample client.h/cpp. This is found in greenstone/runtime-src/src/z3950.

See the download page for general information on downloading records through Greenstone.

Greenstone (2 and 3) can download records from a specified Z39.50 server from the GLI (from the Download panel) or the command line (using the downloadfrom.pl script). Either way, you are presented with five options:

If downloading via the GLI, you can view the downloaded MARC files on the Gather panel. On the left-hand side of the panel, double click the Downloaded Files folder to expand its content. The subfolders are named by the Z39.50 server url. The MARC files are named as the combination of database name, query, and max_records if max_records is specified. These MARC files are physically stored in a temporary cache directory. You can build a collection using these downloaded MARC files by dragging them across to the Collection section on the right-hand side of the Gather panel.

If downloading via the command line, a folder containing the files will be downloaded into the directory from which you run the script (unless the -cache_dir <full-path-to-folder> option is set).

To see the script options, run

downloadfrom.pl -h   (shows the general downloading options)
downloadfrom.pl -download_mode Z3950 -h  (shows the specific options for Z3950 mode)

A sample command might be

downloadfrom.pl -download_mode Z3950 -host lx2.loc.gov -port 210 -database LCDB_MARC8 -find "snails" -max_records 10

This will make a folder called lx2.loc.gov in the folder that you ran the command from, containing a MARC file with your result records.

As the files downloaded via Z39.50 are in MARC format, they can be processed by MARCPlugin, which must be included in the collection plugin list.

https://www.loc.gov/z3950/lcserver.html#addr shows a list of the Library of Congress Z39.50 catalogs.

The YAZ client supports the type-1 RPN query model. Both simple and advanced queries are supported, like single term query, boolean search and fielded search, etc. This page describes in detail the query syntax of YAZ. For example, the following query finds records that have the phrase car history in title field:

 @attr 1=4 "car history"

SRU (Search/Retrieve via URL) is an alternative method that uses a web service to search and retrieve records from Z39.50 repositories. It replaces the Z39.50 communications protocol with HTTP, but still supports the Z39.50 query syntax. Search results from SRU are in XML format. (One method of SRU, SRU VIA HTTP SOAP, was previously referred to as SRW. This name is now deprecated, however some Greenstone scripts have not yet been updated and the name SRW is still used.)

Greenstone also supports downloading from a Z39.50 server through SRU. If using the GLI, select SRU on the Download panel. The options for SRU are the same as for Z39.50 download, but different host and port values should be used here. For example, to connect to the Library of Congress Z39.50 server through SRU, the following host and port should be specified:

 host: http://lx2.loc.gov
 port: 210/LCDB?

If downloading via the GLI, you can view the downloaded records on the Gather panel. They are in the same subfolder as downloading from Z39.50 above.

If downloading on the command line using downloadfrom.pl, the -download_mode is SRW, and a folder containing the downloaded files will be created in the current directory.

The downloaded records are in XML format. Here is a sample record.

To build a collection using the downloaded XML files, MARCXMLPlugin must be added to the collection plugin list.

• Linux/MacOS: in the top level greenstone directory, run

 ./configure --enable-z3950 (--enable-apache-httpd)
 make
 make install

• Use the enable-apache-httpd if you have been using or want to use the apache built in to Greenstone.

• Windows:
  • needs Visual Studio 6
  • It only works with the web library, not the local library
  • In the runtime-src\packages\yaz directory, extract the files from yaz-2.1.4.tar.gz. You'll need to run the extraction twice, once for gzip, once for tar. You should end up with a yaz-2.1.4 directory here.
  • Edit the yaz-2.1.4/include/yaz/oid.h file and change "list" on line 256 to "greenstone_list".
  • Edit the runtime-src\src\z3950\z3950proto.cpp file and delete "extern" from lines 38 and 39 (yyin and yyout).
  • in the greenstone directory, run

 nmake /f win32.mak USE_Z3950=1

• To enable iconv or xml2, edit greenstone\runtime-src\win32.mak, and remove HAVE_ICONV=0 and/or HAVE_LIBXML2=0 from the make command for Yaz. You'll need to install these libraries.

Once Greenstone 2 has Z39.50 support compiled in, it can act as a client to multiple z39.50 servers. The file Greenstone/etc/packages/z3950/z3950.cfg specifies a list of servers to connect to. By default, no servers are set up, although the config file comes with one (commented out) example Z39.50 server, for the United States' Library of Congress.

Each entry consists of:

• A unique "short name" for internal use by Greenstone. (Note, this should not be the same as any local collection short name.)
• The internet name or address of the server, and optionally the port that the server is running on if not the default port 210.
• The name of the database to search on that server.
• A string that provides a meaningful name for the "collection".
• An optional "About" string, providing some information about the database and/or server.
• Optional icon fields, which are displayed instead of the text on the front page.

The entries need only be separated by whitespace, but for the purposes of clarity the sample entry uses newlines and tabs.

There is a list at the Library Of Congress website containing some servers publicly available for testing.

Greenstone will display a new "collection" for each server listed.

• The z3950server program is installed into bin/linux (or bin/darwin, bin/windows, depending on the operating system). It can be run from there, or copied to somewhere else.
• The gsdlsite.cfg file needs to be copied from the cgi-bin directory to the directory you are running z3950server from. Check that the 'gsdlhome' entry is valid. The other entries don't matter.
• Before running the server, set up your Greenstone environment in the terminal by running the following in your top level Greenstone directory:

Linux/Mac

source setup.bash

Windows

setup.bat

• By default, the server listens on localhost:9999. To change the port or address, run

 z3950server tcp:server-name:port-num

• For example, z3950server tcp:localhost:8080 or z3950server tcp:kanuka.cs.waikato.ac.nz:7070

• On Windows, the yaz.dll file (runtime-src/packages/yaz/yaz-2.1.4/bin/yaz.dll) needs to be on your Path, or put in the same directory as the z3950server.exe.
• You can run z3950server -h to see the list of options for the server.

You can test the z3950 server by connecting to it using the Greenstone z3950 client. In the greenstone/etc/packages/z3950/z3950.cfg file, add a server entry similar to the following:

 zdemo
        kanuka.cs.waikato.ac.nz:9999
        demo
        "The demo collection via z3950"
        About "This collection contains a few records from the Humanity Development Library"

• The database name (line 3) specifies which Greenstone collection to search.
• This works with MG, MGPP and Lucene collections.

• Because of the open nature of the standard, our client may not work with some servers.
• Because of the large number of MARC fields, only the most frequently used fields have been given explicit names in the results. Furthermore, these are currently hard-coded to correspond to the USMARC field names.
• The Z39.50 client will not work when fast-cgi is used. Currently, the use of the fast-cgi package (off by default) will disable the use of the Z39.50 client code for Greenstone.
• When viewing a Z39.50 "collection" in Greenstone, sometimes the browser may insert a large amount of space between the collection title and the navigation bar. Reload or Shift Reload should fix this problem.

Sometimes, the best explanation is the simplest one. Z39.50 is a network protocol that provides access to information systems.

Z39.50 was originally developed to meet the needs of libraries to integrate their online catalog systems. It has since been adopted by various other domains, and it remains arguably the most flexible, versatile and efficient information retrieval around. It is also open; the specification can be downloaded freely from the net.

It has been considered complex by some, but its flexibility is mostly derived from the fact that it is a fairly complete model of the task if information retrieval – and from that follows a certain measure of complexity.

We still consider Z39.50 to be an important building block in the construction of information architectures, and much of our work in the past has been about maximising the usefulness and minimising the complexity of Z39.50 by providing well-designed toolkits for programmers wishing to implement the protocol. You'll find much more information about our tools under the software page.

YAZ stands for Yet Another Z39.50 toolkit.

YAZ is a compact toolkit that provides access to the Z39.50 and SRW/SRU protocols, as well as a set of higher-level tools for implementing the server and client roles, respectively. The documentation can be used on its own, or as a reference when looking at the example applications provided with the package.

http://www.indexdata.dk/yaz/doc/installation.unix.tkl#id2528482

This section describes how to compile - and link your own applications using the YAZ toolkit. If you're used to Makefiles this shouldn't be hard. As for other libraries you have used before, you have to set a proper include path for your C/C++ compiler and specify the location of YAZ libraries. You can do it by hand, but generally we suggest you use the yaz-config that is generated by configure. This is especially important if you're using the threaded version of YAZ which requires you to pass more options to your linker/compiler.

The yaz-config script accepts command line options that makes the yaz-config script print options that you should use in your make process. The most important ones are: –cflags, –libs which prints C compiler flags, and linker flags respectively.

A small and complete Makefile for a C application consisting of one source file, myprog.c, may look like this:

YAZCONFIG=/usr/local/bin/yaz-config CFLAGS=`$(YAZCONFIG) –cflags` LIBS=`$(YAZCONFIG) –libs` myprog: myprog.o $ (CC) $(CFLAGS) -o myprog myprog.o $(LIBS)

The CFLAGS variable consists of a C compiler directive that will set the include path to the parent directory of yaz. That is, if YAZ header files were installed in /usr/local/include/yaz, then include path is set to /usr/local/include. Therefore, in your applications you should use # include <yaz/proto.h> and not # include <proto.h>

For Libtool users, the yaz-config script provides a different variant of option –libs, called –lalibs that returns the name of the Libtool acrhive(s) for YAZ rather than the ordinary ones.

For applications using the threaded version of YAZ, specify threads after the other options. When threads is given, more flags and linker flags will be printed by yaz-config. If our previous example was using threads, you'd have to modify the lines that set CFLAGS and LIBS as follows: CFLAGS=`$(YAZCONFIG) –cflags threads` LIBS=`$(YAZCONFIG) –libs threads`

There is no need specify POSIX thread libraries in your Makefile. The LIBS variable includes that as well.