Using MPI
We want to use MPI because we want to be able to communicate picklable objects back and forth between instances of wrapped Enzo.
We can utilize ctypes to initialize the MPI library, and then use SWIG to actually call the Enzo routines. I believe this will work, however, I am not certain because I have not tested it, nor am I positive how the processor information is kept in memory.
What I do know is that this works:
from ctypes import *
print "Loading"
mpi = cdll.LoadLibrary('/afs/slac/package/OpenMPI/lib/libmpi.so')
print "Getting"
f = pythonapi.Py_GetArgcArgv
argc = c_int()
argv = POINTER(c_char_p)()
f(byref(argc), byref(argv))
print "MPI_Init"
mpi.MPI_Init(byref(argc), byref(argv))
print "Finalize"
mpi.MPI_Finalize()
print "Done"
It returns the following output:
mturk@orange $ mpirun -np 1 $ARCH_PATH/bin/python2.5 testmpi.py Loading Getting MPI_Init Finalize Done
I will look into the SWIG-ification later. However, this seems quite promising. (If only ctypes supported C++. Sigh.)
IPython
After having played a bit with ctypes and MPI (and successfully getting information like current processor number, total number, etc) it is becoming more and more obvious that the real method we should use to communicate between processors is not something I want much to do with, if we are approaching from an interactive standpoint. (Now, were we to want to do it from a fully-automated standpoint, I'd be up for it. But I'm not sure what the utility of that would be.)
With IPython, we would initialize the ipengines via mpirun, and then initialize the interactor on the command line. We would leave the MPI interaction to enzo (and if we really needed it, the proc numbers, etc, could be made accessible.)
The real trick would be to properly insert the throwing of exceptions. It seems to me that we would want to #define a generic FAILURE macro (or even a FAILURE() function) that we could have return an exception; were this done, we'd have to have the interactor node catch the exception, and then have it cease iterating the others at the next valid stopping point.
(Also, this brings up the point that any convenience functions we need -- such as output initiation -- could be put into the PyEvolveLevel? routine.)
