Dynamic MPI support

In the context of NEURON wheel integration of CoreNEURON we need to implement dynamic MPI capabilities.

Before diving into implementation details, a few ideas to bare in mind:

would it be possible to capitalize on the libnrnmpi library?
if yes, for the short term we can detect is CoreNEURON is a submodule and use libnrnmpi; on the longer term, we can then see about implementing dynamic MPI in CoreNEURON or maybe extract libnrnmpi in another repo ?

would it be possible to capitalize on the libnrnmpi library?

I think the answer is yes. But it is not clear to me how independent the data part has to be when NEURON and CoreNEURON coexist in direct mode. In direct mode, I think NEURON can initialize MPI and CoreNEURON just use that instance. Ie. the

MPI_Comm nrnmpi_comm;

Something to note is that a script, mkdynam.sh, transforms the nrnmpidec.h list of nrnmpi wrappers for the desired MPI functions
into the helper files, nrnmpi_dynam_wrappers.inc, nrnmpi_dynam.h, nrnmpi_dynam_cinc. And nrnmpi_dynam.cpp is responsible
for dlopening the correct libmpi and then, based on the libmpi implementation, dlopening the correct libnrnmpi.

@alexsavulescu

@alexsavulescu : this ticket would require some analysis of current code usage in CoreNEURON. We can have discussion about this in next month / after summer break.

@sergiorg-hpc

Tagging here @sergiorg-hpc as well as he has recent experience with dynamic library loading. Next week can discuss some of the technical details.

Refactor/re-organising MPI related code

I pushed a sketch of how I think this could be handled to https://bbpgitlab.epfl.ch/hpc/user/coreneuron-dynamic-mpi-sandbox (https://godbolt.org/z/erY4Y6f9K is a single-file no-dependency version of the same).

Refactor/re-organising MPI related code (#625)

@pramodk

Some pointers/next steps:

The nrnmpi_... functions that are used in the CoreNEURON code are declared here and defined here.

In CoreNEURON these should be replaced with instances of the mpi_function<...> helper in the coreneuron-dynamic-mpi-sandbox example above. These should be callable exactly as before, i.e. no changes should be needed at the call sites:

CoreNeuron/coreneuron/utils/memory_utils.cpp

Lines 82 to 84 in db8477c

    
           mem_avg = nrnmpi_dbl_allreduce(cur_mem, 1) / nrnmpi_numprocs; 
        
           mem_max = nrnmpi_dbl_allreduce(cur_mem, 2); 
        
           mem_min = nrnmpi_dbl_allreduce(cur_mem, 3);

With dynamic MPI disabled in CoreNEURON (i.e. taking the 2nd branch of https://bbpgitlab.epfl.ch/hpc/user/coreneuron-dynamic-mpi-sandbox/-/blob/aab81a461c3a1d69ac4aaa760ee1eec04068ab23/include/coreneuron/mpi_utils.h#L49-58), nothing should break. Implementations like this are compiled/linked into other CoreNEURON code, -lmpi is used.
With dynamic MPI enabled in CoreNEURON, we need to compile a separate shared library containing the implementations like this. This shared library is linked with -lmpi, but CoreNEURON/special/... do not link against it. It should be dynamically loaded if --mpi is passed, the instances of mpi_function<...> will use it to initialise their function pointers to the implementations.

For this ticket, we can ignore the possibility of having multiple MPI implementations. Later, when CoreNEURON and NEURON are more tightly integrated, we can inherit that logic from NEURON.

cc: @pramodk @alexsavulescu @alkino

Refactor/re-organising MPI related code (BlueBrain/CoreNeuron#625)

Support for dynamic MPI library loading (BlueBrain/CoreNeuron#641)

Navigation Menu

Repository navigation

Dynamic MPI support #600

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

	mem_avg = nrnmpi_dbl_allreduce(cur_mem, 1) / nrnmpi_numprocs;
	mem_max = nrnmpi_dbl_allreduce(cur_mem, 2);
	mem_min = nrnmpi_dbl_allreduce(cur_mem, 3);

Dynamic MPI support #600

Description

Activity

nrnhines commented on Jul 23, 2021

pramodk commented on Jul 26, 2021

pramodk commented on Aug 19, 2021

olupton commented on Aug 31, 2021

olupton commented on Sep 15, 2021

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions