Kraken Community Installation
If you want to use yt on the Kraken compute nods, we now have a community installation available. For more information, see the KrakenCommunityInstallation wiki page.
Cray XT5 Installation Instructions
The Cray XT5, such as Kraken at NICS, use a specialized lightweight Linux kernel on the compute nodes that doesn't allow shared libraries to be loaded by executables. This means to run yt on the compute nodes, Python needs to be built with all the needed modules built into Python statically. yt runs 'out of the box' on the login nodes of the XT5, so if the parallel capabilities of yt are not needed, or lengthy batch queue-style runs won't be run, the standard installation instructions can be followed. It may be advantageous to build two versions of yt, a standard version and a statically-linked version for the compute nodes. Indeed, if yt is built the standard way first, it will retrieve several of the sources needed for the statically-linked version.
Environment Settings
The first step is to make the environment amenable. In particular, PrgEnv-gnu needs to be loaded, which may require 'swap'-ing out a different PrgEnv. The module iobuf is known to have caused problems, so remove it if it's loaded. Below is an example set of modules which are known to work for this install:
> module list Currently Loaded Modulefiles: 1) modules/3.1.6 14) xt-asyncpe/2.0 2) MySQL/5.0.45 15) xt-pe/2.1.50HD 3) torque/2.3.5 16) xt-mpt/3.1.0 4) moab/5.2.5.s12070 17) xt-libsci/10.3.1 5) xtpe-quadcore 18) fftw/3.1.1 6) xt-service/2.1.50HD 19) xt-totalview/8.4.1b 7) xt-libc/2.1.50HD 20) totalview-support/1.0.5 8) xt-os/2.1.50HD 21) gcc/4.2.0.quadcore 9) xt-boot/2.1.50HD 22) subversion/1.4.6 10) xt-lustre-ss/2.1.50HD_1.6.5 23) imagemagick/6.4.2 11) xtpe-target-cnl 24) java/jdk1.6.0_05 12) Base-opts/2.1.50HD 25) globus/4.0.8 13) PrgEnv-gnu/2.1.50HD
Also set the environment variables export ARCH_PATH=/path/to/new/yt/install/base, which is the location of the new yt, and export CC=cc;export CXX=CC which are the Cray compiler wrappers.
HDF5
In the HDF5 source directory (hdf5-1.8.2 is known to work):
> ./configure --prefix=$ARCH_PATH --disable-shared > make -j8 > make install
zlib
In the zlib source directory (zlib-1.2.3 is known to work):
> ./configure --prefix=$ARCH_PATH > make > make install
The ./configure step will give an error, but it's OK. Just make sure that after the make install step that libz.a is in the $ARCH_PATH/lib directory.
expat
expat is not included in the standard yt, and expat-2.0.1 is known to work. In the source directory:
> ./configure --disable-shared --prefix=$ARCH_PATH > make -j8 > make install
Python
This method to build statically-linked Python is based on instructions from the GPAW project.
Starting from the Python source directory, in the directory Python, create a file named dynload_cnl.c:
/* This module provides the simulation of dynamic loading in Kraken */
#include "Python.h"
#include "importdl.h"
const struct filedescr _PyImport_DynLoadFiletab[] = {
{".a", "rb", C_EXTENSION},
{0, 0}
};
extern struct _inittab _PyImport_Inittab[];
dl_funcptr _PyImport_GetDynLoadFunc(const char *fqname, const char *shortname,
const char *pathname, FILE *fp)
{
struct _inittab *tab = _PyImport_Inittab;
while (tab->name && strcmp(shortname, tab->name)) tab++;
return tab->initfunc;
}
In the file Lib/distutils/unixccompiler.py, add this to the end making sure it's a member of class UnixCCompiler(CCompiler):
def link_shared_object (self,
objects,
output_filename,
output_dir=None,
libraries=None,
library_dirs=None,
runtime_library_dirs=None,
export_symbols=None,
debug=0,
extra_preargs=None,
extra_postargs=None,
build_temp=None,
target_lang=None):
if output_dir is None:
(output_dir, output_filename) = os.path.split(output_filename)
output_fullname = os.path.join(output_dir, output_filename)
linkline = "%s %s" % (output_filename[:-2], output_fullname)
for l in library_dirs:
linkline += " -L" + l
for l in libraries:
linkline += " -l" + l
old_fmt = self.static_lib_format
self.static_lib_format = "%s%.0s"
self.create_static_lib(objects,
output_filename,
output_dir,
debug,
target_lang)
self.static_lib_format = old_fmt
print "Append to Setup: ", linkline
In the file Lib/locale.py, change the third definition of getpreferredencoding() under resetlocale() such that for if do_setlocale: returns None:
def getpreferredencoding(do_setlocale = True):
"""Return the charset that the user is likely using,
according to the system configuration."""
if do_setlocale:
return None
oldloc = setlocale(LC_CTYPE)
setlocale(LC_CTYPE, "")
result = nl_langinfo(CODESET)
setlocale(LC_CTYPE, oldloc)
else:
return nl_langinfo(CODESET)
Then in the top level of the Python source directory invoke
./configure --prefix=$ARCH_PATH SO=.a DYNLOADFILE=dynload_cnl.o MACHDEP=cnl --host=x86_64-unknown-linux-gnu --disable-sockets --disable-ssl --enable-static --disable-shared
Modify the file Modules/Setup to include references to modules which will be needed. There is an example at the bottom of this page. Make sure all the lines at the bottom are commented out. Next invoke these commands in the top level Python source directory:
> make -j8 > make install > export PATH=/path/to/new/yt/install/bin:$PATH > export PYTHONPATH=/path/to/new/yt/install/lib/python2.6/site-packages/ > which python /path/to/new/yt/install/bin/python
Setting the PATH and PYTHONPATH variables makes things simpler in the future steps.
numpy
In the numpy source directory (numpy-1.2.1 is known to work), invoke
python setup.py install >& numpy.install.log
Inspect numpy.install.log, looking for lines that start with 'Append to Setup:' Make sure that the lines for numpy in Modules/Setup above are in numpy.install.log.
Python
Back in the Python source, modify Modules/Setup by commenting out the numpy lines, and making sure CNL_PATH and NUMPY are set and match the install location. Then invoke
make -j8; make install
in the Python source.
Cython
In the Cython source directory, invoke
python setup.py install >& cython.install.log
As above, make sure that the Cython lines are mirrored in Modules/Setup. Cython-0.10.3 is known to work.
Python
Comment out the Cython lines in Modules/Setup and make sure CYTHON is correct. Then invoke
make -j8; make install
mpi4py
Obtain mpi4py. As of 24 April, 2009, the latest version of trunk mpi4py gave one user problems. Revision 279 is known to work, which can be downloaded with the flag shown below:
svn checkout http://mpi4py.googlecode.com/svn/trunk/ mpi4py -r279
and inside of that directory, edit setup.cfg to use the cc/CC wrappers:
[config] mpicc = cc mpicxx = CC [build] debug = 0 #compiler=mingw32 [sdist] force_manifest=1 [install] optimize = 1 [bdist_rpm] release = 1 group = Libraries/Python vendor = CIMEC <http://www.cimec.org.ar> packager = Lisandro Dalcin <dalcinl@gmail.com> doc_files = LICENSE.txt README.txt THANKS.txt docs/html
Before installing mpi4py, it needs to be 'cythonized':
python ./conf/cythonize.py python setup.py install >& mpi4py.install.log
Again, inspect mpi4py.install.log for 'Append to Setup:' and compare to the lines in Modules/Setup.
Python
Comment out the mpi4py lines in Modules/Setup, and then
make -j8; make install
tables
Tables is now optional, its functionality has been replaced by h5py.
tables-2.1 is known to work. In the tables source directory, edit the function getLibrary() in src/utils.c to look like this. Comment out dlopen and dlclose, and add return 0 above dlopen. This prevents tables from loading the dynamic HDF5 libraries:
/* Routine to detect the existance of shared libraries in UNIX. This
has to be checked in MacOSX. However, this is not used right now in
utilsExtension.pyx because UNIX does not complain when trying to
load an extension library that depends on a shared library that it
is not in the system (python raises just the ImportError). */
int getLibrary(char *libname) {
void *hinstLib;
/* Load the dynamic library */
return 0;
/* hinstLib = dlopen(libname, RTLD_LAZY); */
if (hinstLib != NULL) {
/* Free the dynamic library */
/* dlclose(hinstLib); */
return 0;
}
else {
return -1;
}
}
Then, as before, but also referencing the yt install base for the HDF5 files,
python setup.py --hdf5=$ARCH_PATH install >& tables.install.log
and make sure the 'Append to Setup:' lines match with Modules/Setup.
Python
Comment out the tables lines, making sure TABLES is correct, and then invoke
make -j8; make install
h5py
h5py 1.1.0 is known to work. Inside the directory
python setup.py configure --hdf5=$ARCH_PATH --api=16 python setup.py cython python setup.py install >& h5py.install.log
and make sure the 'Append to Setup:' lines match with Modules/Setup
Python
Comment out the h5py lines, making sure H5PY is correct, and then invoke
make -j8; make install
matplotlib
matplotlib is quite a bit more involved, and requires some changes before it will work. matplotlib-0.98.5.2 and 0.98.3 are known to work.
In the matplotlib source directory, change setupext.py:
diff -ur orig-mpl-0.98.3/setupext.py new-mpl-0.98.3/setupext.py
--- orig-mpl-0.98.3/setupext.py 2008-08-03 11:15:22.000000000 -0700
+++ new-mpl-0.98.3/setupext.py 2008-11-13 20:30:38.000000000 -0800
@@ -59,6 +59,7 @@
'gnukfreebsd5' : ['/usr/local', '/usr'],
'gnukfreebsd6' : ['/usr/local', '/usr'],
'aix5' : ['/usr/local'],
+ 'cnl' : ['/usr/'],
}
import sys, os, stat
src/_backend_agg.cpp:
diff -ur orig-mpl-0.98.3/src/_backend_agg.cpp new-mpl-0.98.3/src/_backend_agg.cpp --- orig-mpl-0.98.3/src/_backend_agg.cpp 2008-08-03 11:14:18.000000000 -0700 +++ new-mpl-0.98.3/src/_backend_agg.cpp 2008-11-13 21:34:51.000000000 -0800 @@ -38,6 +38,8 @@ #define PY_ARRAY_TYPES_PREFIX NumPy #include "numpy/arrayobject.h" +#define py_to_agg_transformation_matrix bagg_py_to_agg_transformation_matrix +#define py_convert_bbox bagg_py_convert_bbox #include "agg_py_transforms.h" #ifndef M_PI
src/_backend_agg.h:
diff -ur orig-mpl-0.98.3/src/_backend_agg.h new-mpl-0.98.3/src/_backend_agg.h --- orig-mpl-0.98.3/src/_backend_agg.h 2008-08-03 11:14:18.000000000 -0700 +++ new-mpl-0.98.3/src/_backend_agg.h 2008-11-13 21:38:21.000000000 -0800 @@ -38,6 +38,7 @@ #include "agg_scanline_p.h" #include "agg_vcgen_markers_term.h" +#define code_map bagg_code_map #include "agg_py_path_iterator.h" // These are copied directly from path.py, and must be kept in sync
src/_path.cpp:
diff -ur orig-mpl-0.98.3/src/_path.cpp new-mpl-0.98.3/src/_path.cpp --- orig-mpl-0.98.3/src/_path.cpp 2008-08-03 11:14:18.000000000 -0700 +++ new-mpl-0.98.3/src/_path.cpp 2008-11-13 21:38:34.000000000 -0800 @@ -1,4 +1,7 @@ +#define code_map path_code_map #include "agg_py_path_iterator.h" +#define py_to_agg_transformation_matrix path_py_to_agg_transformation_matrix +#define py_convert_bbox path_py_convert_bbox #include "agg_py_transforms.h" #include <limits>
As before, invoke
python setup.py install >& matplotlib.install.log
making sure the 'Append to Setup' lines in matplotlib.install.log match Modules/Setup.
Python
Back in Python, uncomment the matplotlib lines in Modules/Setup, making sure MATPLOTLIB is correct, and then invoke
make -j8
It will fail because matplotlib has some C++ code, and the compile line it fails on needs to be called with CC and not cc. Go into the Makefile in the Python source directory, and change the usage of LINKCC to CXX, e.g. change this:
# Build the interpreter $(BUILDPYTHON): Modules/python.o $(LIBRARY) $(LDLIBRARY) $(LINKCC) $(LDFLAGS) $(LINKFORSHARED) -o $@ \ Modules/python.o \ $(BLDLIBRARY) $(LIBS) $(MODLIBS) $(SYSLIBS) $(LDLAST)
to this:
# Build the interpreter $(BUILDPYTHON): Modules/python.o $(LIBRARY) $(LDLIBRARY) $(CXX) $(LDFLAGS) $(LINKFORSHARED) -o $@ \ Modules/python.o \ $(BLDLIBRARY) $(LIBS) $(MODLIBS) $(SYSLIBS) $(LDLAST)
Then:
make -j8; make install
which should be successful.
yt
Make sure the trunk version of yt is being used, which can be obtained:
svn co http://svn.enzotools.org/yt/trunk yt-trunk-svn
in yt-trunk-svn create or correct the file hdf5.cfg to point to the yt install, which is the same as the value of ARCH_PATH above. Then,
python setup.py install >& yt.install.log
Like before, make sure the 'Append to Setup:' lines are in Modules/Setup.
Python
Uncomment the yt lines in Modules/Setup, making sure YT is correct, and invoke:
make
As above for matplotlib, it will fail. Then re-open the Python Makefile (which has changed since the matplotlib steps above), and change LINKCC to CXX, and then:
make -j8; make install
Success?
If this was successful, python should be invokable without error (besides readline):
> python
Python 2.6.1 (r261:67515, Feb 23 2009, 21:20:19)
[GCC 4.2.0 20070514 (quadcore:rpm:113)] on linux2
Type "help", "copyright", "credits" or "license" for more information.
Traceback (most recent call last):
File "/etc/pythonstart", line 7, in <module>
import readline
ImportError: No module named readline
>>>
And then within python, this should be the result:
>>> from yt.mods import * yt.lagos WARNING 2009-02-24 14:00:06,092 No HDF4 support
Running on a compute node
Debug queue
To run on the compute nodes in the debug queue, follow these steps. On Kraken the compute nodes can only see the /lustre/scratch disk, so the whole yt-cnl hierarchy (src isn't needed on scratch, but it won't harm anything) will have to be copied onto the scratch disk.
> cp -r /path/to/the/new/yt-cnl /lustre/scratch/username/
First request a debug job, and set a couple environmental settings:
> qsub -I -V -q debug -l size=8;limit=1:00:00 > export MPLCONFIGDIR=/lustre/scratch/username/ > export PYTHONHOME=/lustre/scratch/username/yt-cnl/
Once the job has started, cd to the scratch disk, and this command will show a successful build of yt-cnl:
> aprun -n1 /lustre/scratch/username/yt-cnl/bin/python -c "from yt.mods import *" yt.lagos WARNING 2009-03-07 22:40:04,355 No HDF4 support Application 27219 resources: utime 0, stime 0
This command shows that parallel python works:
> aprun -n8 /lustre/scratch/username/yt-cnl/bin/python -c "from mpi4py import MPI; print 'hello from processor ' + str(MPI.COMM_WORLD.rank)" hello from processor 0 hello from processor 2 hello from processor 4 hello from processor 6 hello from processor 5 hello from processor 3 hello from processor 7 hello from processor 1 Application 27223 resources: utime 0, stime 0
Batch Queue
Here is an example batch queue script. Note the --parallel after the script name which tells yt that OD.py should be run in parallel. Also note the two environment variables. This job will run 128 threads (-n128) with one thread per socket (-S1) and two threads per node (-N2) for twelve hours. Please see man qsub page for more details.
#PBS -N jobname #PBS -l walltime=12:00:00,size=512 #PBS -e jobname.err #PBS -o jobname.out #PBS -m be #PBS -M user@domain.edu #PBS -S /bin/tcsh #PBS -V cd /lustre/scratch/username/dataset MPLCONFIGDIR=/lustre/scratch/username/ PYTHONHOME=/luster/scratch/username/yt-cnl aprun -S1 -n128 -N2 /lustre/scratch/username/yt-cnl/bin/python ./OD.py --parallel > ./OD.log
Submit the job with the command:
> qsub Script.txt
Updating yt
To update yt follow these simple steps. It is very similar to what is described above. Make sure the same modules are loaded as above.
yt
At the top level directory of yt, type:
> svn up
and a number of files will be updated. As long as there are no conflicts, the yt directory is now updated. Next in the same directory:
> python setup.py install
making sure the python being invoked is the CNL version built previously.
Python
In the Python source directory, open up Modules/Setup and change it in a meaningless way (say with a new blank line), or touch Modules/Setup. This tells the Python make system to rebuild Makefile. Then, as before, in the Python source directory:
> make -j8
When that errors, modify LINKCC to CXX in Makefile, and then:
> make -j8 > make install
Once that is finished update the copy of yt-cnl on the scratch disk with the new version.
Using rsync
The easiest way to keep the home disk and scratch disk versions of yt-cnl in sync is to use rsync. Use a command like this below, and after the first time it will significantly speed up the procedure. Also, this makes it easy to make sure that the copy on scratch hasn't been eviscerated by the auto-purge.
> rsync --verbose --progress --stats --recursive --times --perms --executability --links --delete /path/to/home/install/of/yt-cnl /lustre/scratch/username
this will put the contents of /path/to/home/install/of/yt-cnl in /lustre/scratch/username/yt-cnl.
