Re: Assignment from a list is slow in Numarray

Francesc Alted <falted@xxxxxxxxxxxx> writes:
> Well, if you are pondering using parallel reading because of speed,

I was actually pondering using parallel _writing_ because of speed.
Parallel reading is easy: each process just opens the file and reads
independently.  But merely switching to NetCDF gave a decent speed
improvement even with sequential writing.

> Are you sure? Here you have a couple of OpenDX data importers for HDF5:

I was aware of dxhdf5 but I don't think it handles irregular meshes.
It seems that the Cactus one doesn't either.

> Before doing that, talk with Konrad. I know that Scientific Python supports
> MPI and BSPlib right-out-of-the-box, so maybe there is a shorter path to do
> what you want.

Unfortunately I was not able to use Konrad's MPI bindings.  Petsc has
its own initialization routine that needs to be called early on.  I
had to create another special version of the Python interpreter,
different from Konrad's.  I also needed more functionality than
Konrad's bindings have - I even use MPI_Alltoallv at one point.
Fortunately creating my own MPI bindings with Swig and Numarray was
fairly easy.

> So, perhaps spending your time writing Python bindings for
> Parallel-HDF5 would be a better bet for future applications.

Perhaps, but first I'll have to concentrate on the actual
number crunching code to get some data to write.  Then I'll see
whether I really need parallel writing.

Thanks to everybody for helpful suggestions.

-- 
        Timo Korvola            <URL:http://www.iki.fi/tkorvola>


-------------------------------------------------------
This SF.Net email is sponsored by: YOU BE THE JUDGE. Be one of 170
Project Admins to receive an Apple iPod Mini FREE for your judgement on
who ports your project to Linux PPC the best. Sponsored by IBM.
Deadline: Sept. 24. Go here: http://sf.net/ppc_contest.php