In a267ce8, network topology info was used to optimally distribute mesh partitions among MPI ranks. This was immediately rolled back in 57a6721, for the following reasons:
- Calling
MPI_Dist_graph_create (without even using the created communicator in any way) caused an order of magnitude slowdown in Tpetra library code related to algebraic system initialization. The cause of this is unclear, especially since the created communicator was never passed to Tpetra and freed shortly after creation. The only feasible reason seems to be some sort of interaction with global MPI state, which I am unwilling and unable to investigate at this time.
- There did not seem to be meaningful performance gains from using the optimal mesh distribution on the test cluster. Admittedly, it is rather small, larger and more complex architectures may benefit from this.
- Given that algebraic system initialization is currently painfully slow, it does not seem wise to hamper it even further in the pursuit of hypothetical performance gains.
Despite this, it may prove worthwhile to pursue optimal distribution in the future. This issue is here as a reminder.
In a267ce8, network topology info was used to optimally distribute mesh partitions among MPI ranks. This was immediately rolled back in 57a6721, for the following reasons:
MPI_Dist_graph_create(without even using the created communicator in any way) caused an order of magnitude slowdown in Tpetra library code related to algebraic system initialization. The cause of this is unclear, especially since the created communicator was never passed to Tpetra and freed shortly after creation. The only feasible reason seems to be some sort of interaction with global MPI state, which I am unwilling and unable to investigate at this time.Despite this, it may prove worthwhile to pursue optimal distribution in the future. This issue is here as a reminder.