v0.8.23
Public Member Functions | Public Attributes | List of all members
MoFEM::CommInterface Struct Reference

Managing BitRefLevels. More...

#include <src/interfaces/CommInterface.hpp>

Inheritance diagram for MoFEM::CommInterface:
[legend]
Collaboration diagram for MoFEM::CommInterface:
[legend]

Public Member Functions

MoFEMErrorCode query_interface (const MOFEMuuid &uuid, UnknownInterface **iface) const
 
 CommInterface (const MoFEM::Core &core)
 
 ~CommInterface ()
 Destructor. More...
 
- Public Member Functions inherited from MoFEM::UnknownInterface
template<class IFACE >
MoFEMErrorCode registerInterface (const MOFEMuuid &uuid, bool error_if_registration_failed=true)
 Register interface. More...
 
template<class IFACE , bool VERIFY = false>
MoFEMErrorCode getInterface (const MOFEMuuid &uuid, IFACE *&iface) const
 Get interface by uuid and return reference to pointer of interface. More...
 
template<class IFACE >
MoFEMErrorCode getInterface (IFACE *&iface) const
 Get interface refernce to pointer of interface. More...
 
template<class IFACE >
MoFEMErrorCode getInterface (IFACE **const iface) const
 Get interface pointer to pointer of interface. More...
 
template<class IFACE , typename boost::enable_if< boost::is_pointer< IFACE >, int >::type = 0>
IFACE getInterface () const
 Get interface pointer to pointer of interface. More...
 
template<class IFACE , typename boost::enable_if< boost::is_reference< IFACE >, int >::type = 0>
IFACE getInterface () const
 Get reference to interface. More...
 
template<class IFACE >
IFACE * getInterface () const
 Function returning pointer to interface. More...
 
virtual ~UnknownInterface ()
 
virtual MoFEMErrorCode getLibVersion (Version &version) const
 Get library version. More...
 
virtual const MoFEMErrorCode getFileVersion (moab::Interface &moab, Version &version) const
 Get database major version. More...
 
virtual MoFEMErrorCode getInterfaceVersion (Version &version) const
 Get database major version. More...
 
template<>
MoFEMErrorCode getInterface (const MOFEMuuid &uuid, UnknownInterface *&iface) const
 

Public Attributes

MoFEM::CorecOre
 
bool dEbug
 

Make elemnts multishared

MoFEMErrorCode resolveSharedFiniteElements (const Problem *problem_ptr, const std::string &fe_name, int verb=DEFAULT_VERBOSITY)
 resolve shared entities for finite elements in the problem More...
 
MoFEMErrorCode resolveSharedFiniteElements (const std::string &name, const std::string &fe_name, int verb=DEFAULT_VERBOSITY)
 resolve shared entities for finite elements in the problem More...
 

Make entities multishared

MoFEMErrorCode makeEntitiesMultishared (const EntityHandle *entities, const int num_entities, const int owner_proc=0, int verb=DEFAULT_VERBOSITY)
 make entities from proc 0 shared on all proc More...
 
MoFEMErrorCode makeEntitiesMultishared (Range &entities, const int owner_proc=0, int verb=DEFAULT_VERBOSITY)
 make entities from proc 0 shared on all proc More...
 
MoFEMErrorCode makeFieldEntitiesMultishared (const std::string field_name, const int owner_proc=0, int verb=DEFAULT_VERBOSITY)
 make field entities multi shared More...
 
MoFEMErrorCode exchangeFieldData (const std::string field_name, int verb=DEFAULT_VERBOSITY)
 Exchange field data. More...
 

Synchronize entities (Following functions in future will be

deprecated)

MoFEMErrorCode synchroniseEntities (Range &ent, int verb=DEFAULT_VERBOSITY)
 
MoFEMErrorCode synchroniseFieldEntities (const std::string name, int verb=DEFAULT_VERBOSITY)
 

Additional Inherited Members

- Protected Member Functions inherited from MoFEM::UnknownInterface
boost::typeindex::type_index getClassIdx (const MOFEMuuid &uid) const
 Get type name for interface Id. More...
 
MOFEMuuid getUId (const boost::typeindex::type_index &class_idx) const
 Get interface Id for class name. More...
 

Detailed Description

Managing BitRefLevels.

Examples
dm_partitioned_no_field.cpp.

Definition at line 33 of file CommInterface.hpp.

Constructor & Destructor Documentation

◆ CommInterface()

MoFEM::CommInterface::CommInterface ( const MoFEM::Core core)

Definition at line 34 of file CommInterface.cpp.

35  : cOre(const_cast<MoFEM::Core &>(core)), dEbug(false) {}

◆ ~CommInterface()

MoFEM::CommInterface::~CommInterface ( )

Destructor.

Definition at line 36 of file CommInterface.cpp.

36 {}

Member Function Documentation

◆ exchangeFieldData()

MoFEMErrorCode MoFEM::CommInterface::exchangeFieldData ( const std::string  field_name,
int  verb = DEFAULT_VERBOSITY 
)

Exchange field data.

Exchange field for all shared and ghosted entities. This function should be called collectively over the communicator for this ParallelComm. If the entities vector is empty, all shared entities participate in the exchange. If a proc has no owned entities this function must still be called since it is collective.

Note
collective - need tu be run on all processors in communicator
Todo:
It is not working if field has entities diffrent than vertices.
Parameters
verb
field_name
Returns
MoFEMErrorCode

Definition at line 532 of file CommInterface.cpp.

533  {
534  MoFEM::Interface &m_field = cOre;
536  if (m_field.get_comm_size() > 1) {
537 
538  const FieldEntity_multiIndex *field_ents;
539  CHKERR m_field.get_field_ents(&field_ents);
540 
541  Range exchange_ents_data_verts, exchange_ents_data;
542 
543  for (auto it = field_ents->get<FieldName_mi_tag>().lower_bound(field_name);
544  it != field_ents->get<FieldName_mi_tag>().upper_bound(field_name);
545  ++it)
546  if (
547 
548  ((*it)->getPStatus()) &&
549 
550  (*it)->getNbDofsOnEnt()
551 
552  ) {
553  if ((*it)->getEntType() == MBVERTEX)
554  exchange_ents_data_verts.insert((*it)->getRefEnt());
555  else
556  exchange_ents_data.insert((*it)->getRefEnt());
557  }
558 
559  auto field_ptr = m_field.get_field_structure(field_name);
560  ParallelComm *pcomm = ParallelComm::get_pcomm(
561  &m_field.get_moab(), m_field.get_basic_entity_data_ptr()->pcommID);
562 
563  auto exchange = [&](const Range &ents, Tag th) {
565  if (!ents.empty()) {
566  std::vector<Tag> tags;
567  tags.push_back(th);
568  CHKERR pcomm->exchange_tags(tags, tags, ents);
569  }
571  };
572 
573  CHKERR exchange(exchange_ents_data_verts, field_ptr->th_FieldDataVerts);
574  CHKERR exchange(exchange_ents_data, field_ptr->th_FieldData);
575  }
577 }
virtual moab::Interface & get_moab()=0
virtual boost::shared_ptr< BasicEntityData > & get_basic_entity_data_ptr()=0
Get pointer to basic entity data.
virtual int get_comm_size() const =0
multi_index_container< boost::shared_ptr< FieldEntity >, indexed_by< ordered_unique< tag< Unique_mi_tag >, member< FieldEntity, UId, &FieldEntity::globalUId > >, ordered_non_unique< tag< FieldName_mi_tag >, const_mem_fun< FieldEntity::interface_type_Field, boost::string_ref, &FieldEntity::getNameRef > >, ordered_non_unique< tag< Ent_mi_tag >, const_mem_fun< FieldEntity, EntityHandle, &FieldEntity::getEnt > >, ordered_non_unique< tag< Composite_Name_And_Ent_mi_tag >, composite_key< FieldEntity, const_mem_fun< FieldEntity::interface_type_Field, boost::string_ref, &FieldEntity::getNameRef >, const_mem_fun< FieldEntity, EntityHandle, &FieldEntity::getEnt > > > > > FieldEntity_multiIndex
MultiIndex container keeps FieldEntity.
#define MoFEMFunctionReturn(a)
Last executable line of each PETSc function used for error handling. Replaces return()
Definition: definitions.h:477
#define CHKERR
Inline error check.
Definition: definitions.h:596
virtual MoFEMErrorCode get_field_ents(const FieldEntity_multiIndex **field_ents) const =0
Get field multi index.
#define MoFEMFunctionBegin
First executable line of each MoFEM function, used for error handling. Final line of MoFEM functions ...
Definition: definitions.h:407
virtual const Field * get_field_structure(const std::string &name)=0
get field structure

◆ makeEntitiesMultishared() [1/2]

MoFEMErrorCode MoFEM::CommInterface::makeEntitiesMultishared ( const EntityHandle entities,
const int  num_entities,
const int  owner_proc = 0,
int  verb = DEFAULT_VERBOSITY 
)

make entities from proc 0 shared on all proc

Note
collective - need tu be run on all processors in communicator
Parameters
entities
num_entities
my_procdefault proc id to share from
verb
Returns
MoFEMErrorCode

Definition at line 319 of file CommInterface.cpp.

321  {
322  MoFEM::Interface &m_field = cOre;
324 
325  if (m_field.get_comm_size() > 1) {
326 
327  ParallelComm *pcomm = ParallelComm::get_pcomm(
328  &m_field.get_moab(), m_field.get_basic_entity_data_ptr()->pcommID);
329  const EntityHandle *ent = entities;
330  const EntityHandle *const end = entities + num_entities;
331  std::vector<EntityHandle> all_ents_vec(ent, end);
332 
333  auto print_vec = [&](auto name, auto &vec) {
334  std::ostringstream ss;
335  ss << "Proc " << m_field.get_comm_rank() << " ";
336  ss << "vector " << name << " [ ";
337  for (auto v : vec)
338  ss << v << " ";
339  ss << "]" << endl;
340  PetscSynchronizedPrintf(m_field.get_comm(), "%s", ss.str().c_str());
341  PetscSynchronizedFlush(m_field.get_comm(), PETSC_STDOUT);
342  };
343 
344  if (verb >= NOISY)
345  print_vec("all_ents_vec", all_ents_vec);
346 
347  const size_t block_size = sizeof(EntityHandle) / sizeof(int);
348 
349  int tag;
350  CHKERR PetscCommGetNewTag(m_field.get_comm(), &tag);
351 
352  std::vector<EntityHandle> recv_ents_vec[m_field.get_comm_size()];
353  std::vector<MPI_Request> r_waits(m_field.get_comm_size());
354  for (int proc = 0, kk = 0; proc < m_field.get_comm_size(); ++proc) {
355  if (proc != m_field.get_comm_rank()) {
356  recv_ents_vec[proc].resize(all_ents_vec.size());
357  CHKERR MPI_Irecv(&*recv_ents_vec[proc].begin(), // buffer to receive
358  recv_ents_vec[proc].size() *
359  block_size, // message length
360  MPIU_INT, proc, // to proc
361  tag, m_field.get_comm(), &r_waits[kk]);
362  ++kk;
363  }
364  }
365 
366  std::vector<MPI_Request> s_waits(
367  m_field.get_comm_size()); // status of send messages
368  for (int proc = 0, kk = 0; proc < m_field.get_comm_size(); ++proc) {
369  if (proc != m_field.get_comm_rank()) {
370  CHKERR MPI_Isend(&*all_ents_vec.begin(), // buffer to send
371  all_ents_vec.size() * block_size, // message length
372  MPIU_INT, proc, // to proc
373  tag, m_field.get_comm(), &s_waits[kk]);
374  ++kk;
375  }
376  }
377 
378  std::vector<MPI_Status> status(m_field.get_comm_size());
379 
380  // Wait for received
381  CHKERR MPI_Waitall(m_field.get_comm_size() - 1, &*r_waits.begin(),
382  &*status.begin());
383 
384  // Wait for send messages
385  CHKERR MPI_Waitall(m_field.get_comm_size() - 1, &*s_waits.begin(),
386  &*status.begin());
387 
388  if (verb >= NOISY) {
389  for (int proc = 0; proc < m_field.get_comm_size(); ++proc) {
390  if (proc != m_field.get_comm_rank()) {
391  print_vec("recv_ents_vec received from " +
392  boost::lexical_cast<std::string>(proc),
393  recv_ents_vec[proc]);
394  }
395  }
396  }
397 
398  unsigned char pstatus = 0;
399 
400  if (m_field.get_comm_rank() != owner_proc)
401  pstatus = PSTATUS_NOT_OWNED;
402 
403  pstatus |= PSTATUS_SHARED;
404  if (m_field.get_comm_size() > 2)
405  pstatus |= PSTATUS_MULTISHARED;
406 
407  if (pcomm->size() != m_field.get_comm_size())
408  SETERRQ(PETSC_COMM_WORLD, MOFEM_NOT_IMPLEMENTED,
409  "Implementation that size of PComm and MoFEM are diffrent not "
410  "implemented");
411 
412  std::vector<int> shprocs(MAX_SHARING_PROCS, 0);
413  std::vector<EntityHandle> shhandles(MAX_SHARING_PROCS, 0);
414 
415  for (int e = 0; e != all_ents_vec.size(); ++e) {
416 
417  size_t rrr = 0;
418  if (m_field.get_comm_rank() != owner_proc) {
419  shhandles[rrr] = (recv_ents_vec[owner_proc])[e];
420  shprocs[rrr] = owner_proc;
421  ++rrr;
422  }
423 
424  for (size_t rr = 0; rr < m_field.get_comm_size(); ++rr) {
425  if (rr != owner_proc && rr != m_field.get_comm_rank()) {
426  shhandles[rrr] = (recv_ents_vec[rr])[e];
427  shprocs[rrr] = rr;
428  ++rrr;
429  }
430  }
431 
432  for (; rrr != MAX_SHARING_PROCS; ++rrr)
433  shprocs[rrr] = -1;
434 
435  CHKERR m_field.get_moab().tag_set_data(pcomm->sharedp_tag(),
436  &all_ents_vec[e], 1, &shprocs[0]);
437  CHKERR m_field.get_moab().tag_set_data(
438  pcomm->sharedh_tag(), &all_ents_vec[e], 1, &shhandles[0]);
439  if (pstatus & PSTATUS_MULTISHARED) {
440  CHKERR m_field.get_moab().tag_set_data(
441  pcomm->sharedps_tag(), &all_ents_vec[e], 1, &shprocs[0]);
442  CHKERR m_field.get_moab().tag_set_data(
443  pcomm->sharedhs_tag(), &all_ents_vec[e], 1, &shhandles[0]);
444  }
445 
446  CHKERR m_field.get_moab().tag_set_data(pcomm->pstatus_tag(),
447  &all_ents_vec[e], 1, &pstatus);
448 
449  auto print_owner = [&]() {
451  int moab_owner_proc;
452  EntityHandle moab_owner_handle;
453  CHKERR pcomm->get_owner_handle(all_ents_vec[e], moab_owner_proc,
454  moab_owner_handle);
455 
456  std::ostringstream ss;
457 
458  ss << "Rank " << m_field.get_comm_rank() << " ";
459  if (!(pstatus & PSTATUS_NOT_OWNED))
460  ss << "OWNER ";
461  if (pstatus & PSTATUS_SHARED)
462  ss << "PSTATUS_SHARED ";
463  else if (pstatus & PSTATUS_MULTISHARED)
464  ss << "PSTATUS_MULTISHARED ";
465 
466  ss << "owner " << moab_owner_proc << " (" << owner_proc << ") ";
467 
468  ss << std::endl;
469  PetscSynchronizedPrintf(m_field.get_comm(), "%s", ss.str().c_str());
470  PetscSynchronizedFlush(m_field.get_comm(), PETSC_STDOUT);
471 
473  };
474 
475  if (verb >= NOISY)
476  CHKERR print_owner();
477 
478  auto check_owner = [&]() {
480  int moab_owner_proc;
481  EntityHandle moab_owner_handle;
482  CHKERR pcomm->get_owner_handle(all_ents_vec[e], moab_owner_proc,
483  moab_owner_handle);
484  if (owner_proc != moab_owner_proc)
485  SETERRQ2(PETSC_COMM_SELF, MOFEM_DATA_INCONSISTENCY, "%d != %d",
486  owner_proc, moab_owner_proc);
487  if (m_field.get_comm_rank() != owner_proc)
488  if (recv_ents_vec[owner_proc][e] != moab_owner_handle)
489  SETERRQ2(PETSC_COMM_SELF, MOFEM_DATA_INCONSISTENCY, "%lu != %lu",
490  recv_ents_vec[owner_proc][e], moab_owner_handle);
492  };
493 
494  CHKERR check_owner();
495  }
496  }
497 
499 }
virtual moab::Interface & get_moab()=0
virtual boost::shared_ptr< BasicEntityData > & get_basic_entity_data_ptr()=0
Get pointer to basic entity data.
virtual int get_comm_size() const =0
#define MoFEMFunctionReturn(a)
Last executable line of each PETSc function used for error handling. Replaces return()
Definition: definitions.h:477
virtual int get_comm_rank() const =0
#define CHKERR
Inline error check.
Definition: definitions.h:596
#define MoFEMFunctionBegin
First executable line of each MoFEM function, used for error handling. Final line of MoFEM functions ...
Definition: definitions.h:407
virtual MPI_Comm & get_comm() const =0

◆ makeEntitiesMultishared() [2/2]

MoFEMErrorCode MoFEM::CommInterface::makeEntitiesMultishared ( Range &  entities,
const int  owner_proc = 0,
int  verb = DEFAULT_VERBOSITY 
)

make entities from proc 0 shared on all proc

Note
collective - need tu be run on all processors in communicator
Parameters
entities
my_procdefault proc id to share from
verb
Returns
MoFEMErrorCode

Definition at line 501 of file CommInterface.cpp.

503  {
504  MoFEM::Interface &m_field = cOre;
506  if (m_field.get_comm_size() > 1) {
507  const int num_ents = entities.size();
508  std::vector<EntityHandle> vec_ents(num_ents);
509  std::copy(entities.begin(), entities.end(), vec_ents.begin());
510  CHKERR makeEntitiesMultishared(&*vec_ents.begin(), num_ents, owner_proc,
511  verb);
512  }
514 }
virtual int get_comm_size() const =0
#define MoFEMFunctionReturn(a)
Last executable line of each PETSc function used for error handling. Replaces return()
Definition: definitions.h:477
MoFEMErrorCode makeEntitiesMultishared(const EntityHandle *entities, const int num_entities, const int owner_proc=0, int verb=DEFAULT_VERBOSITY)
make entities from proc 0 shared on all proc
#define CHKERR
Inline error check.
Definition: definitions.h:596
#define MoFEMFunctionBegin
First executable line of each MoFEM function, used for error handling. Final line of MoFEM functions ...
Definition: definitions.h:407

◆ makeFieldEntitiesMultishared()

MoFEMErrorCode MoFEM::CommInterface::makeFieldEntitiesMultishared ( const std::string  field_name,
const int  owner_proc = 0,
int  verb = DEFAULT_VERBOSITY 
)

make field entities multi shared

Note
collective - need tu be run on all processors in communicator
Parameters
field_name
owner_proc
verb
Returns
MoFEMErrorCode

Definition at line 517 of file CommInterface.cpp.

518  {
519  MoFEM::Interface &m_field = cOre;
521  if (m_field.get_comm_size() > 1) {
522  EntityHandle field_meshset = m_field.get_field_meshset(field_name);
523  std::vector<EntityHandle> field_ents;
524  CHKERR m_field.get_moab().get_entities_by_handle(field_meshset, field_ents,
525  true);
526  CHKERR makeEntitiesMultishared(&*field_ents.begin(), field_ents.size(),
527  owner_proc, verb);
528  }
530 }
virtual moab::Interface & get_moab()=0
virtual int get_comm_size() const =0
#define MoFEMFunctionReturn(a)
Last executable line of each PETSc function used for error handling. Replaces return()
Definition: definitions.h:477
virtual EntityHandle get_field_meshset(const std::string &name) const =0
get field meshset
MoFEMErrorCode makeEntitiesMultishared(const EntityHandle *entities, const int num_entities, const int owner_proc=0, int verb=DEFAULT_VERBOSITY)
make entities from proc 0 shared on all proc
#define CHKERR
Inline error check.
Definition: definitions.h:596
#define MoFEMFunctionBegin
First executable line of each MoFEM function, used for error handling. Final line of MoFEM functions ...
Definition: definitions.h:407

◆ query_interface()

MoFEMErrorCode MoFEM::CommInterface::query_interface ( const MOFEMuuid uuid,
UnknownInterface **  iface 
) const
virtual

Implements MoFEM::UnknownInterface.

Definition at line 22 of file CommInterface.cpp.

23  {
25  *iface = NULL;
26  if (uuid == IDD_MOFEMComm) {
27  *iface = const_cast<CommInterface *>(this);
29  }
30  SETERRQ(PETSC_COMM_SELF, MOFEM_DATA_INCONSISTENCY, "unknown interface");
32 }
#define MoFEMFunctionBeginHot
First executable line of each MoFEM function, used for error handling. Final line of MoFEM functions ...
Definition: definitions.h:501
#define MoFEMFunctionReturnHot(a)
Last executable line of each PETSc function used for error handling. Replaces return()
Definition: definitions.h:508
static const MOFEMuuid IDD_MOFEMComm

◆ synchroniseEntities()

MoFEMErrorCode MoFEM::CommInterface::synchroniseEntities ( Range &  ent,
int  verb = DEFAULT_VERBOSITY 
)

synchronize entity range on processors (collective)

Note
collective - need tu be run on all processors in communicator

Definition at line 38 of file CommInterface.cpp.

38  {
39  MoFEM::Interface &m_field = cOre;
40  const RefEntity_multiIndex *ref_ents_ptr;
42  CHKERR m_field.get_ref_ents(&ref_ents_ptr);
43 
44  // make a buffer
45  std::vector<std::vector<EntityHandle>> sbuffer(m_field.get_comm_size());
46 
47  Range::iterator eit = ents.begin();
48  for (; eit != ents.end(); eit++) {
49 
50  auto meit = ref_ents_ptr->get<Ent_mi_tag>().find(*eit);
51  if (meit == ref_ents_ptr->get<Ent_mi_tag>().end()) {
52  continue;
53  SETERRQ2(PETSC_COMM_SELF, MOFEM_DATA_INCONSISTENCY,
54  "rank %d entity %lu not exist on database, local entity can not "
55  "be found for this owner",
56  m_field.get_comm_rank(), *eit);
57  }
58 
59  unsigned char pstatus = (*meit)->getPStatus();
60 
61  if (pstatus == 0)
62  continue;
63 
64  if (verb >= NOISY) {
65  std::ostringstream zz;
66  zz << "pstatus " << std::bitset<8>(pstatus) << " ";
67  PetscSynchronizedPrintf(m_field.get_comm(), "%s", zz.str().c_str());
68  }
69 
70  for (int proc = 0;
71  proc < MAX_SHARING_PROCS && -1 != (*meit)->getSharingProcsPtr()[proc];
72  proc++) {
73  if ((*meit)->getSharingProcsPtr()[proc] == -1)
74  SETERRQ(PETSC_COMM_SELF, MOFEM_IMPOSIBLE_CASE,
75  "sharing processor not set");
76 
77  if ((*meit)->getSharingProcsPtr()[proc] == m_field.get_comm_rank())
78  continue;
79 
80  EntityHandle handle_on_sharing_proc =
81  (*meit)->getSharingHandlersPtr()[proc];
82  sbuffer[(*meit)->getSharingProcsPtr()[proc]].push_back(
83  handle_on_sharing_proc);
84  if (verb >= NOISY)
85  PetscSynchronizedPrintf(
86  m_field.get_comm(), "send %lu (%lu) to %d at %d\n",
87  (*meit)->getRefEnt(), handle_on_sharing_proc,
88  (*meit)->getSharingProcsPtr()[proc], m_field.get_comm_rank());
89 
90  if (!(pstatus & PSTATUS_MULTISHARED))
91  break;
92  }
93  }
94 
95  int nsends = 0; // number of messages to send
96  std::vector<int> sbuffer_lengths(
97  m_field.get_comm_size()); // length of the message to proc
98  const size_t block_size = sizeof(EntityHandle) / sizeof(int);
99  for (int proc = 0; proc < m_field.get_comm_size(); proc++) {
100 
101  if (!sbuffer[proc].empty()) {
102 
103  sbuffer_lengths[proc] = sbuffer[proc].size() * block_size;
104  nsends++;
105 
106  } else {
107 
108  sbuffer_lengths[proc] = 0;
109  }
110  }
111 
112  // Make sure it is a PETSc m_field.get_comm()
113  CHKERR PetscCommDuplicate(m_field.get_comm(), &m_field.get_comm(), NULL);
114 
115  std::vector<MPI_Status> status(m_field.get_comm_size());
116 
117  // Computes the number of messages a node expects to receive
118  int nrecvs; // number of messages received
119  CHKERR PetscGatherNumberOfMessages(m_field.get_comm(), NULL,
120  &sbuffer_lengths[0], &nrecvs);
121 
122  // Computes info about messages that a MPI-node will receive, including
123  // (from-id,length) pairs for each message.
124  int *onodes; // list of node-ids from which messages are expected
125  int *olengths; // corresponding message lengths
126  CHKERR PetscGatherMessageLengths(m_field.get_comm(), nsends, nrecvs,
127  &sbuffer_lengths[0], &onodes, &olengths);
128 
129  // Gets a unique new tag from a PETSc communicator. All processors that share
130  // the communicator MUST call this routine EXACTLY the same number of times.
131  // This tag should only be used with the current objects communicator; do NOT
132  // use it with any other MPI communicator.
133  int tag;
134  CHKERR PetscCommGetNewTag(m_field.get_comm(), &tag);
135 
136  // Allocate a buffer sufficient to hold messages of size specified in
137  // olengths. And post Irecvs on these buffers using node info from onodes
138  int **rbuf; // must bee freed by user
139  MPI_Request *r_waits; // must bee freed by user
140  // rbuf has a pointers to messages. It has size of of nrecvs (number of
141  // messages) +1. In the first index a block is allocated,
142  // such that rbuf[i] = rbuf[i-1]+olengths[i-1].
143  CHKERR PetscPostIrecvInt(m_field.get_comm(), tag, nrecvs, onodes, olengths,
144  &rbuf, &r_waits);
145 
146  MPI_Request *s_waits; // status of sens messages
147  CHKERR PetscMalloc1(nsends, &s_waits);
148 
149  // Send messages
150  for (int proc = 0, kk = 0; proc < m_field.get_comm_size(); proc++) {
151  if (!sbuffer_lengths[proc])
152  continue; // no message to send to this proc
153  CHKERR MPI_Isend(&(sbuffer[proc])[0], // buffer to send
154  sbuffer_lengths[proc], // message length
155  MPIU_INT, proc, // to proc
156  tag, m_field.get_comm(), s_waits + kk);
157  kk++;
158  }
159 
160  // Wait for received
161  if (nrecvs)
162  CHKERR MPI_Waitall(nrecvs, r_waits, &status[0]);
163 
164  // Wait for send messages
165  if (nsends)
166  CHKERR MPI_Waitall(nsends, s_waits, &status[0]);
167 
168  if (verb >= VERY_VERBOSE) {
169  PetscSynchronizedPrintf(m_field.get_comm(), "Rank %d nb. before ents %u\n",
170  m_field.get_comm_rank(), ents.size());
171  }
172 
173  // synchronise range
174  for (int kk = 0; kk < nrecvs; kk++) {
175 
176  int len = olengths[kk];
177  int *data_from_proc = rbuf[kk];
178 
179  for (int ee = 0; ee < len; ee += block_size) {
180 
181  EntityHandle ent;
182  bcopy(&data_from_proc[ee], &ent, sizeof(EntityHandle));
183  auto meit = ref_ents_ptr->get<Ent_mi_tag>().find(ent);
184  if (meit == ref_ents_ptr->get<Ent_mi_tag>().end())
185  SETERRQ2(PETSC_COMM_SELF, MOFEM_DATA_INCONSISTENCY,
186  "rank %d entity %lu not exist on database, local entity can "
187  "not be found for this owner",
188  m_field.get_comm_rank(), ent);
189 
190  if (verb >= VERY_VERBOSE)
191  PetscSynchronizedPrintf(
192  m_field.get_comm(), "received %ul (%ul) from %d at %d\n",
193  (*meit)->getRefEnt(), ent, onodes[kk], m_field.get_comm_rank());
194 
195  ents.insert((*meit)->getRefEnt());
196  }
197  }
198 
199  if (verb >= VERBOSE)
200  PetscSynchronizedPrintf(m_field.get_comm(), "Rank %d nb. after ents %u\n",
201  m_field.get_comm_rank(), ents.size());
202 
203  // Cleaning
204  CHKERR PetscFree(s_waits);
205  CHKERR PetscFree(rbuf[0]);
206  CHKERR PetscFree(rbuf);
207  CHKERR PetscFree(r_waits);
208  CHKERR PetscFree(onodes);
209  CHKERR PetscFree(olengths);
210 
211  if (verb >= VERBOSE)
212  PetscSynchronizedFlush(m_field.get_comm(), PETSC_STDOUT);
213 
215 }
virtual MoFEMErrorCode get_ref_ents(const RefEntity_multiIndex **refined_ents_ptr) const =0
Get ref entities multi-index from database.
virtual int get_comm_size() const =0
#define MoFEMFunctionReturn(a)
Last executable line of each PETSc function used for error handling. Replaces return()
Definition: definitions.h:477
virtual int get_comm_rank() const =0
#define CHKERR
Inline error check.
Definition: definitions.h:596
multi_index_container< boost::shared_ptr< RefEntity >, indexed_by< ordered_unique< tag< Ent_mi_tag >, member< RefEntity::BasicEntity, EntityHandle, &RefEntity::ent > >, ordered_non_unique< tag< Ent_Ent_mi_tag >, const_mem_fun< RefEntity, EntityHandle, &RefEntity::getParentEnt > >, ordered_non_unique< tag< EntType_mi_tag >, const_mem_fun< RefEntity::BasicEntity, EntityType, &RefEntity::getEntType > >, ordered_non_unique< tag< ParentEntType_mi_tag >, const_mem_fun< RefEntity, EntityType, &RefEntity::getParentEntType > >, ordered_non_unique< tag< Composite_EntType_and_ParentEntType_mi_tag >, composite_key< RefEntity, const_mem_fun< RefEntity::BasicEntity, EntityType, &RefEntity::getEntType >, const_mem_fun< RefEntity, EntityType, &RefEntity::getParentEntType > > >, ordered_non_unique< tag< Composite_ParentEnt_And_EntType_mi_tag >, composite_key< RefEntity, const_mem_fun< RefEntity::BasicEntity, EntityType, &RefEntity::getEntType >, const_mem_fun< RefEntity, EntityHandle, &RefEntity::getParentEnt > > > > > RefEntity_multiIndex
#define MoFEMFunctionBegin
First executable line of each MoFEM function, used for error handling. Final line of MoFEM functions ...
Definition: definitions.h:407
virtual MPI_Comm & get_comm() const =0

Member Data Documentation

◆ cOre

MoFEM::Core& MoFEM::CommInterface::cOre

Definition at line 38 of file CommInterface.hpp.

◆ dEbug

bool MoFEM::CommInterface::dEbug

Definition at line 39 of file CommInterface.hpp.


The documentation for this struct was generated from the following files: