v0.9.1
Public Member Functions | Public Attributes | List of all members
MoFEM::CommInterface Struct Reference

Managing BitRefLevels. More...

#include <src/interfaces/CommInterface.hpp>

Inheritance diagram for MoFEM::CommInterface:
[legend]
Collaboration diagram for MoFEM::CommInterface:
[legend]

Public Member Functions

MoFEMErrorCode query_interface (const MOFEMuuid &uuid, UnknownInterface **iface) const
 
 CommInterface (const MoFEM::Core &core)
 
 ~CommInterface ()
 Destructor. More...
 
- Public Member Functions inherited from MoFEM::UnknownInterface
template<class IFACE >
MoFEMErrorCode registerInterface (const MOFEMuuid &uuid, bool error_if_registration_failed=true)
 Register interface. More...
 
template<class IFACE , bool VERIFY = false>
MoFEMErrorCode getInterface (const MOFEMuuid &uuid, IFACE *&iface) const
 Get interface by uuid and return reference to pointer of interface. More...
 
template<class IFACE >
MoFEMErrorCode getInterface (IFACE *&iface) const
 Get interface refernce to pointer of interface. More...
 
template<class IFACE >
MoFEMErrorCode getInterface (IFACE **const iface) const
 Get interface pointer to pointer of interface. More...
 
template<class IFACE , typename boost::enable_if< boost::is_pointer< IFACE >, int >::type = 0>
IFACE getInterface () const
 Get interface pointer to pointer of interface. More...
 
template<class IFACE , typename boost::enable_if< boost::is_reference< IFACE >, int >::type = 0>
IFACE getInterface () const
 Get reference to interface. More...
 
template<class IFACE >
IFACE * getInterface () const
 Function returning pointer to interface. More...
 
virtual ~UnknownInterface ()=default
 
virtual MoFEMErrorCode getLibVersion (Version &version) const
 Get library version. More...
 
virtual const MoFEMErrorCode getFileVersion (moab::Interface &moab, Version &version) const
 Get database major version. More...
 
virtual MoFEMErrorCode getInterfaceVersion (Version &version) const
 Get database major version. More...
 
template<>
MoFEMErrorCode getInterface (const MOFEMuuid &uuid, UnknownInterface *&iface) const
 

Public Attributes

MoFEM::CorecOre
 
bool dEbug
 

Make elemnts multishared

MoFEMErrorCode resolveSharedFiniteElements (const Problem *problem_ptr, const std::string &fe_name, int verb=DEFAULT_VERBOSITY)
 resolve shared entities for finite elements in the problem More...
 
MoFEMErrorCode resolveSharedFiniteElements (const std::string &name, const std::string &fe_name, int verb=DEFAULT_VERBOSITY)
 resolve shared entities for finite elements in the problem More...
 

Make entities multishared

MoFEMErrorCode makeEntitiesMultishared (const EntityHandle *entities, const int num_entities, const int owner_proc=0, int verb=DEFAULT_VERBOSITY)
 make entities from proc 0 shared on all proc More...
 
MoFEMErrorCode makeEntitiesMultishared (Range &entities, const int owner_proc=0, int verb=DEFAULT_VERBOSITY)
 make entities from proc 0 shared on all proc More...
 
MoFEMErrorCode makeFieldEntitiesMultishared (const std::string field_name, const int owner_proc=0, int verb=DEFAULT_VERBOSITY)
 make field entities multi shared More...
 
MoFEMErrorCode exchangeFieldData (const std::string field_name, int verb=DEFAULT_VERBOSITY)
 Exchange field data. More...
 

Synchronize entities (Following functions in future will be

deprecated)

MoFEMErrorCode synchroniseEntities (Range &ent, int verb=DEFAULT_VERBOSITY)
 
MoFEMErrorCode synchroniseFieldEntities (const std::string name, int verb=DEFAULT_VERBOSITY)
 

Additional Inherited Members

- Protected Member Functions inherited from MoFEM::UnknownInterface
boost::typeindex::type_index getClassIdx (const MOFEMuuid &uid) const
 Get type name for interface Id. More...
 
MOFEMuuid getUId (const boost::typeindex::type_index &class_idx) const
 Get interface Id for class name. More...
 

Detailed Description

Managing BitRefLevels.

Examples
dm_partitioned_no_field.cpp, elasticity.cpp, and mesh_smoothing.cpp.

Definition at line 33 of file CommInterface.hpp.

Constructor & Destructor Documentation

◆ CommInterface()

MoFEM::CommInterface::CommInterface ( const MoFEM::Core core)

Definition at line 34 of file CommInterface.cpp.

35  : cOre(const_cast<MoFEM::Core &>(core)), dEbug(false) {}

◆ ~CommInterface()

MoFEM::CommInterface::~CommInterface ( )

Destructor.

Definition at line 36 of file CommInterface.cpp.

36 {}

Member Function Documentation

◆ exchangeFieldData()

MoFEMErrorCode MoFEM::CommInterface::exchangeFieldData ( const std::string  field_name,
int  verb = DEFAULT_VERBOSITY 
)

Exchange field data.

Exchange field for all shared and ghosted entities. This function should be called collectively over the communicator for this ParallelComm. If the entities vector is empty, all shared entities participate in the exchange. If a proc has no owned entities this function must still be called since it is collective.

Note
collective - need tu be run on all processors in communicator
Todo:
It is not working if field has entities diffrent than vertices.
Parameters
verb
field_name
Returns
MoFEMErrorCode

Definition at line 490 of file CommInterface.cpp.

491  {
492  MoFEM::Interface &m_field = cOre;
494  if (m_field.get_comm_size() > 1) {
495 
496  const FieldEntity_multiIndex *field_ents;
497  CHKERR m_field.get_field_ents(&field_ents);
498 
499  Range exchange_ents_data_verts, exchange_ents_data;
500 
501  for (auto it = field_ents->get<FieldName_mi_tag>().lower_bound(field_name);
502  it != field_ents->get<FieldName_mi_tag>().upper_bound(field_name);
503  ++it)
504  if (
505 
506  ((*it)->getPStatus()) &&
507 
508  (*it)->getNbDofsOnEnt()
509 
510  ) {
511  if ((*it)->getEntType() == MBVERTEX)
512  exchange_ents_data_verts.insert((*it)->getRefEnt());
513  else
514  exchange_ents_data.insert((*it)->getRefEnt());
515  }
516 
517  auto field_ptr = m_field.get_field_structure(field_name);
518  ParallelComm *pcomm = ParallelComm::get_pcomm(
519  &m_field.get_moab(), m_field.get_basic_entity_data_ptr()->pcommID);
520 
521  auto exchange = [&](const Range &ents, Tag th) {
523  if (!ents.empty()) {
524  std::vector<Tag> tags;
525  tags.push_back(th);
526  CHKERR pcomm->exchange_tags(tags, tags, ents);
527  }
529  };
530 
531  CHKERR exchange(exchange_ents_data_verts, field_ptr->th_FieldDataVerts);
532  CHKERR exchange(exchange_ents_data, field_ptr->th_FieldData);
533  }
535 }
Deprecated interface functions.
virtual moab::Interface & get_moab()=0
virtual boost::shared_ptr< BasicEntityData > & get_basic_entity_data_ptr()=0
Get pointer to basic entity data.
virtual int get_comm_size() const =0
multi_index_container< boost::shared_ptr< FieldEntity >, indexed_by< ordered_unique< tag< Unique_mi_tag >, member< FieldEntity, UId, &FieldEntity::globalUId > >, ordered_non_unique< tag< FieldName_mi_tag >, const_mem_fun< FieldEntity::interface_type_Field, boost::string_ref, &FieldEntity::getNameRef > >, ordered_non_unique< tag< Ent_mi_tag >, const_mem_fun< FieldEntity, EntityHandle, &FieldEntity::getEnt > >, ordered_non_unique< tag< Composite_Name_And_Ent_mi_tag >, composite_key< FieldEntity, const_mem_fun< FieldEntity::interface_type_Field, boost::string_ref, &FieldEntity::getNameRef >, const_mem_fun< FieldEntity, EntityHandle, &FieldEntity::getEnt > > > > > FieldEntity_multiIndex
MultiIndex container keeps FieldEntity.
#define MoFEMFunctionReturn(a)
Last executable line of each PETSc function used for error handling. Replaces return()
Definition: definitions.h:482
#define CHKERR
Inline error check.
Definition: definitions.h:601
virtual MoFEMErrorCode get_field_ents(const FieldEntity_multiIndex **field_ents) const =0
Get field multi index.
#define MoFEMFunctionBegin
First executable line of each MoFEM function, used for error handling. Final line of MoFEM functions ...
Definition: definitions.h:412
virtual const Field * get_field_structure(const std::string &name)=0
get field structure

◆ makeEntitiesMultishared() [1/2]

MoFEMErrorCode MoFEM::CommInterface::makeEntitiesMultishared ( const EntityHandle entities,
const int  num_entities,
const int  owner_proc = 0,
int  verb = DEFAULT_VERBOSITY 
)

make entities from proc 0 shared on all proc

Note
collective - need tu be run on all processors in communicator
Parameters
entities
num_entities
my_procdefault proc id to share from
verb
Returns
MoFEMErrorCode

Definition at line 319 of file CommInterface.cpp.

321  {
322  MoFEM::Interface &m_field = cOre;
324 
325  if (m_field.get_comm_size() > 1) {
326 
327  ParallelComm *pcomm = ParallelComm::get_pcomm(
328  &m_field.get_moab(), m_field.get_basic_entity_data_ptr()->pcommID);
329 
330  Range all_ents_range;
331  all_ents_range.insert_list(entities, entities + num_entities);
332 
333  auto get_tag = [&]() {
334  Tag th_gid;
335  const int zero = 0;
336  CHKERR m_field.get_moab().tag_get_handle(
337  "TMP_GLOBAL_ID_TAG_NAME", 1, MB_TYPE_INTEGER, th_gid,
338  MB_TAG_SPARSE | MB_TAG_CREAT, &zero);
339  return th_gid;
340  };
341 
342  auto delete_tag = [&](auto &&th_gid) {
344  CHKERR m_field.get_moab().tag_delete(th_gid);
346  };
347 
348  auto resolve_shared_ents = [&](auto &&th_gid, auto &all_ents_range) {
349  auto set_gid = [&](auto &th_gid) {
350  std::vector<int> gids(num_entities);
351  for (size_t g = 0; g != all_ents_range.size(); ++g)
352  gids[g] = g + 1;
353  CHKERR m_field.get_moab().tag_set_data(th_gid, all_ents_range,
354  &*gids.begin());
355 
356  return &th_gid;
357  };
358 
359  auto get_skin_ents = [&](auto &all_ents_range) {
360  std::array<Range, 4> proc_ents_skin;
361  proc_ents_skin[3] = all_ents_range.subset_by_dimension(3);
362  proc_ents_skin[2] = all_ents_range.subset_by_dimension(2);
363  proc_ents_skin[1] = all_ents_range.subset_by_dimension(1);
364  proc_ents_skin[0] = all_ents_range.subset_by_dimension(0);
365  return proc_ents_skin;
366  };
367 
368  auto resolve_dim = [&](auto &all_ents_range) {
369  for (int resolve_dim = 3; resolve_dim >= 0; --resolve_dim) {
370  if (all_ents_range.num_of_dimension(resolve_dim))
371  return resolve_dim;
372  }
373  return -1;
374  };
375 
376  auto get_proc_ent = [&](auto &all_ents_range) {
377  Range proc_ent;
378  if (m_field.get_comm_rank() == owner_proc)
379  proc_ent = all_ents_range;
380  return proc_ent;
381  };
382 
383  auto resolve_shared_ents = [&](auto &&proc_ents, auto &&skin_ents) {
384  return pcomm->resolve_shared_ents(
385  0, proc_ents, resolve_dim(all_ents_range),
386  resolve_dim(all_ents_range), skin_ents.data(), set_gid(th_gid));
387  };
388 
389  CHKERR resolve_shared_ents(get_proc_ent(all_ents_range),
390  get_skin_ents(all_ents_range));
391 
392  return th_gid;
393  };
394 
395  CHKERR delete_tag(resolve_shared_ents(get_tag(), all_ents_range));
396 
397  if (verb >= NOISY) {
398 
399  auto print_owner = [&](const EntityHandle e) {
401  int moab_owner_proc;
402  EntityHandle moab_owner_handle;
403  CHKERR pcomm->get_owner_handle(e, moab_owner_proc, moab_owner_handle);
404 
405  unsigned char pstatus = 0;
406 
407  CHKERR m_field.get_moab().tag_get_data(pcomm->pstatus_tag(), &e, 1,
408  &pstatus);
409 
410  std::vector<int> shprocs(MAX_SHARING_PROCS, 0);
411  std::vector<EntityHandle> shhandles(MAX_SHARING_PROCS, 0);
412 
413  CHKERR m_field.get_moab().tag_get_data(pcomm->sharedp_tag(), &e, 1,
414  &shprocs[0]);
415  CHKERR m_field.get_moab().tag_get_data(pcomm->sharedh_tag(), &e, 1,
416  &shhandles[0]);
417  if (pstatus & PSTATUS_MULTISHARED) {
418  CHKERR m_field.get_moab().tag_get_data(pcomm->sharedps_tag(), &e, 1,
419  &shprocs[0]);
420  CHKERR m_field.get_moab().tag_get_data(pcomm->sharedhs_tag(), &e, 1,
421  &shhandles[0]);
422  }
423 
424  std::ostringstream ss;
425 
426  ss << "Rank " << m_field.get_comm_rank() << " ";
427  if (!(pstatus & PSTATUS_NOT_OWNED))
428  ss << "OWNER ";
429  if (pstatus & PSTATUS_SHARED)
430  ss << "PSTATUS_SHARED ";
431  if (pstatus & PSTATUS_MULTISHARED)
432  ss << "PSTATUS_MULTISHARED ";
433 
434  ss << "owner " << moab_owner_proc << " (" << owner_proc << ") ";
435 
436  ss << "shprocs: ";
437  for (size_t r = 0; r != m_field.get_comm_size() + 1; ++r)
438  ss << shprocs[r] << " ";
439 
440  ss << "shhandles: ";
441  for (size_t r = 0; r != m_field.get_comm_size() + 1; ++r)
442  ss << shhandles[r] << " ";
443 
444  ss << std::endl;
445  PetscSynchronizedPrintf(m_field.get_comm(), "%s", ss.str().c_str());
446  PetscSynchronizedFlush(m_field.get_comm(), PETSC_STDOUT);
447 
449  };
450 
451  for (auto e : all_ents_range)
452  CHKERR print_owner(e);
453  }
454  }
455 
457 }
Deprecated interface functions.
virtual moab::Interface & get_moab()=0
virtual boost::shared_ptr< BasicEntityData > & get_basic_entity_data_ptr()=0
Get pointer to basic entity data.
virtual int get_comm_size() const =0
#define MoFEMFunctionReturn(a)
Last executable line of each PETSc function used for error handling. Replaces return()
Definition: definitions.h:482
virtual int get_comm_rank() const =0
#define CHKERR
Inline error check.
Definition: definitions.h:601
#define MoFEMFunctionBegin
First executable line of each MoFEM function, used for error handling. Final line of MoFEM functions ...
Definition: definitions.h:412
virtual MPI_Comm & get_comm() const =0

◆ makeEntitiesMultishared() [2/2]

MoFEMErrorCode MoFEM::CommInterface::makeEntitiesMultishared ( Range &  entities,
const int  owner_proc = 0,
int  verb = DEFAULT_VERBOSITY 
)

make entities from proc 0 shared on all proc

Note
collective - need tu be run on all processors in communicator
Parameters
entities
my_procdefault proc id to share from
verb
Returns
MoFEMErrorCode

Definition at line 459 of file CommInterface.cpp.

461  {
462  MoFEM::Interface &m_field = cOre;
464  if (m_field.get_comm_size() > 1) {
465  const int num_ents = entities.size();
466  std::vector<EntityHandle> vec_ents(num_ents);
467  std::copy(entities.begin(), entities.end(), vec_ents.begin());
468  CHKERR makeEntitiesMultishared(&*vec_ents.begin(), num_ents, owner_proc,
469  verb);
470  }
472 }
Deprecated interface functions.
virtual int get_comm_size() const =0
#define MoFEMFunctionReturn(a)
Last executable line of each PETSc function used for error handling. Replaces return()
Definition: definitions.h:482
MoFEMErrorCode makeEntitiesMultishared(const EntityHandle *entities, const int num_entities, const int owner_proc=0, int verb=DEFAULT_VERBOSITY)
make entities from proc 0 shared on all proc
#define CHKERR
Inline error check.
Definition: definitions.h:601
#define MoFEMFunctionBegin
First executable line of each MoFEM function, used for error handling. Final line of MoFEM functions ...
Definition: definitions.h:412

◆ makeFieldEntitiesMultishared()

MoFEMErrorCode MoFEM::CommInterface::makeFieldEntitiesMultishared ( const std::string  field_name,
const int  owner_proc = 0,
int  verb = DEFAULT_VERBOSITY 
)

make field entities multi shared

Note
collective - need tu be run on all processors in communicator
Parameters
field_name
owner_proc
verb
Returns
MoFEMErrorCode

Definition at line 475 of file CommInterface.cpp.

476  {
477  MoFEM::Interface &m_field = cOre;
479  if (m_field.get_comm_size() > 1) {
480  EntityHandle field_meshset = m_field.get_field_meshset(field_name);
481  std::vector<EntityHandle> field_ents;
482  CHKERR m_field.get_moab().get_entities_by_handle(field_meshset, field_ents,
483  true);
484  CHKERR makeEntitiesMultishared(&*field_ents.begin(), field_ents.size(),
485  owner_proc, verb);
486  }
488 }
Deprecated interface functions.
virtual moab::Interface & get_moab()=0
virtual int get_comm_size() const =0
#define MoFEMFunctionReturn(a)
Last executable line of each PETSc function used for error handling. Replaces return()
Definition: definitions.h:482
virtual EntityHandle get_field_meshset(const std::string &name) const =0
get field meshset
MoFEMErrorCode makeEntitiesMultishared(const EntityHandle *entities, const int num_entities, const int owner_proc=0, int verb=DEFAULT_VERBOSITY)
make entities from proc 0 shared on all proc
#define CHKERR
Inline error check.
Definition: definitions.h:601
#define MoFEMFunctionBegin
First executable line of each MoFEM function, used for error handling. Final line of MoFEM functions ...
Definition: definitions.h:412

◆ query_interface()

MoFEMErrorCode MoFEM::CommInterface::query_interface ( const MOFEMuuid uuid,
UnknownInterface **  iface 
) const
virtual

Implements MoFEM::UnknownInterface.

Definition at line 22 of file CommInterface.cpp.

23  {
25  *iface = NULL;
26  if (uuid == IDD_MOFEMComm) {
27  *iface = const_cast<CommInterface *>(this);
29  }
30  SETERRQ(PETSC_COMM_SELF, MOFEM_DATA_INCONSISTENCY, "unknown interface");
32 }
#define MoFEMFunctionBeginHot
First executable line of each MoFEM function, used for error handling. Final line of MoFEM functions ...
Definition: definitions.h:506
#define MoFEMFunctionReturnHot(a)
Last executable line of each PETSc function used for error handling. Replaces return()
Definition: definitions.h:513
static const MOFEMuuid IDD_MOFEMComm

◆ synchroniseEntities()

MoFEMErrorCode MoFEM::CommInterface::synchroniseEntities ( Range &  ent,
int  verb = DEFAULT_VERBOSITY 
)

synchronize entity range on processors (collective)

Note
collective - need tu be run on all processors in communicator

Definition at line 38 of file CommInterface.cpp.

38  {
39  MoFEM::Interface &m_field = cOre;
40  const RefEntity_multiIndex *ref_ents_ptr;
42  CHKERR m_field.get_ref_ents(&ref_ents_ptr);
43 
44  // make a buffer
45  std::vector<std::vector<EntityHandle>> sbuffer(m_field.get_comm_size());
46 
47  Range::iterator eit = ents.begin();
48  for (; eit != ents.end(); eit++) {
49 
50  auto meit = ref_ents_ptr->get<Ent_mi_tag>().find(*eit);
51  if (meit == ref_ents_ptr->get<Ent_mi_tag>().end()) {
52  continue;
53  SETERRQ2(PETSC_COMM_SELF, MOFEM_DATA_INCONSISTENCY,
54  "rank %d entity %lu not exist on database, local entity can not "
55  "be found for this owner",
56  m_field.get_comm_rank(), *eit);
57  }
58 
59  unsigned char pstatus = (*meit)->getPStatus();
60 
61  if (pstatus == 0)
62  continue;
63 
64  if (verb >= NOISY) {
65  std::ostringstream zz;
66  zz << "pstatus " << std::bitset<8>(pstatus) << " ";
67  PetscSynchronizedPrintf(m_field.get_comm(), "%s", zz.str().c_str());
68  }
69 
70  for (int proc = 0;
71  proc < MAX_SHARING_PROCS && -1 != (*meit)->getSharingProcsPtr()[proc];
72  proc++) {
73  if ((*meit)->getSharingProcsPtr()[proc] == -1)
74  SETERRQ(PETSC_COMM_SELF, MOFEM_IMPOSIBLE_CASE,
75  "sharing processor not set");
76 
77  if ((*meit)->getSharingProcsPtr()[proc] == m_field.get_comm_rank())
78  continue;
79 
80  EntityHandle handle_on_sharing_proc =
81  (*meit)->getSharingHandlersPtr()[proc];
82  sbuffer[(*meit)->getSharingProcsPtr()[proc]].push_back(
83  handle_on_sharing_proc);
84  if (verb >= NOISY)
85  PetscSynchronizedPrintf(
86  m_field.get_comm(), "send %lu (%lu) to %d at %d\n",
87  (*meit)->getRefEnt(), handle_on_sharing_proc,
88  (*meit)->getSharingProcsPtr()[proc], m_field.get_comm_rank());
89 
90  if (!(pstatus & PSTATUS_MULTISHARED))
91  break;
92  }
93  }
94 
95  int nsends = 0; // number of messages to send
96  std::vector<int> sbuffer_lengths(
97  m_field.get_comm_size()); // length of the message to proc
98  const size_t block_size = sizeof(EntityHandle) / sizeof(int);
99  for (int proc = 0; proc < m_field.get_comm_size(); proc++) {
100 
101  if (!sbuffer[proc].empty()) {
102 
103  sbuffer_lengths[proc] = sbuffer[proc].size() * block_size;
104  nsends++;
105 
106  } else {
107 
108  sbuffer_lengths[proc] = 0;
109  }
110  }
111 
112  // Make sure it is a PETSc m_field.get_comm()
113  CHKERR PetscCommDuplicate(m_field.get_comm(), &m_field.get_comm(), NULL);
114 
115  std::vector<MPI_Status> status(m_field.get_comm_size());
116 
117  // Computes the number of messages a node expects to receive
118  int nrecvs; // number of messages received
119  CHKERR PetscGatherNumberOfMessages(m_field.get_comm(), NULL,
120  &sbuffer_lengths[0], &nrecvs);
121 
122  // Computes info about messages that a MPI-node will receive, including
123  // (from-id,length) pairs for each message.
124  int *onodes; // list of node-ids from which messages are expected
125  int *olengths; // corresponding message lengths
126  CHKERR PetscGatherMessageLengths(m_field.get_comm(), nsends, nrecvs,
127  &sbuffer_lengths[0], &onodes, &olengths);
128 
129  // Gets a unique new tag from a PETSc communicator. All processors that share
130  // the communicator MUST call this routine EXACTLY the same number of times.
131  // This tag should only be used with the current objects communicator; do NOT
132  // use it with any other MPI communicator.
133  int tag;
134  CHKERR PetscCommGetNewTag(m_field.get_comm(), &tag);
135 
136  // Allocate a buffer sufficient to hold messages of size specified in
137  // olengths. And post Irecvs on these buffers using node info from onodes
138  int **rbuf; // must bee freed by user
139  MPI_Request *r_waits; // must bee freed by user
140  // rbuf has a pointers to messages. It has size of of nrecvs (number of
141  // messages) +1. In the first index a block is allocated,
142  // such that rbuf[i] = rbuf[i-1]+olengths[i-1].
143  CHKERR PetscPostIrecvInt(m_field.get_comm(), tag, nrecvs, onodes, olengths,
144  &rbuf, &r_waits);
145 
146  MPI_Request *s_waits; // status of sens messages
147  CHKERR PetscMalloc1(nsends, &s_waits);
148 
149  // Send messages
150  for (int proc = 0, kk = 0; proc < m_field.get_comm_size(); proc++) {
151  if (!sbuffer_lengths[proc])
152  continue; // no message to send to this proc
153  CHKERR MPI_Isend(&(sbuffer[proc])[0], // buffer to send
154  sbuffer_lengths[proc], // message length
155  MPIU_INT, proc, // to proc
156  tag, m_field.get_comm(), s_waits + kk);
157  kk++;
158  }
159 
160  // Wait for received
161  if (nrecvs)
162  CHKERR MPI_Waitall(nrecvs, r_waits, &status[0]);
163 
164  // Wait for send messages
165  if (nsends)
166  CHKERR MPI_Waitall(nsends, s_waits, &status[0]);
167 
168  if (verb >= VERY_VERBOSE) {
169  PetscSynchronizedPrintf(m_field.get_comm(), "Rank %d nb. before ents %u\n",
170  m_field.get_comm_rank(), ents.size());
171  }
172 
173  // synchronise range
174  for (int kk = 0; kk < nrecvs; kk++) {
175 
176  int len = olengths[kk];
177  int *data_from_proc = rbuf[kk];
178 
179  for (int ee = 0; ee < len; ee += block_size) {
180 
181  EntityHandle ent;
182  bcopy(&data_from_proc[ee], &ent, sizeof(EntityHandle));
183  auto meit = ref_ents_ptr->get<Ent_mi_tag>().find(ent);
184  if (meit == ref_ents_ptr->get<Ent_mi_tag>().end())
185  SETERRQ2(PETSC_COMM_SELF, MOFEM_DATA_INCONSISTENCY,
186  "rank %d entity %lu not exist on database, local entity can "
187  "not be found for this owner",
188  m_field.get_comm_rank(), ent);
189 
190  if (verb >= VERY_VERBOSE)
191  PetscSynchronizedPrintf(
192  m_field.get_comm(), "received %ul (%ul) from %d at %d\n",
193  (*meit)->getRefEnt(), ent, onodes[kk], m_field.get_comm_rank());
194 
195  ents.insert((*meit)->getRefEnt());
196  }
197  }
198 
199  if (verb >= VERBOSE)
200  PetscSynchronizedPrintf(m_field.get_comm(), "Rank %d nb. after ents %u\n",
201  m_field.get_comm_rank(), ents.size());
202 
203  // Cleaning
204  CHKERR PetscFree(s_waits);
205  CHKERR PetscFree(rbuf[0]);
206  CHKERR PetscFree(rbuf);
207  CHKERR PetscFree(r_waits);
208  CHKERR PetscFree(onodes);
209  CHKERR PetscFree(olengths);
210 
211  if (verb >= VERBOSE)
212  PetscSynchronizedFlush(m_field.get_comm(), PETSC_STDOUT);
213 
215 }
virtual MoFEMErrorCode get_ref_ents(const RefEntity_multiIndex **refined_ents_ptr) const =0
Get ref entities multi-index from database.
Deprecated interface functions.
virtual int get_comm_size() const =0
#define MoFEMFunctionReturn(a)
Last executable line of each PETSc function used for error handling. Replaces return()
Definition: definitions.h:482
virtual int get_comm_rank() const =0
#define CHKERR
Inline error check.
Definition: definitions.h:601
multi_index_container< boost::shared_ptr< RefEntity >, indexed_by< ordered_unique< tag< Ent_mi_tag >, member< RefEntity::BasicEntity, EntityHandle, &RefEntity::ent > >, ordered_non_unique< tag< Ent_Ent_mi_tag >, const_mem_fun< RefEntity, EntityHandle, &RefEntity::getParentEnt > >, ordered_non_unique< tag< EntType_mi_tag >, const_mem_fun< RefEntity::BasicEntity, EntityType, &RefEntity::getEntType > >, ordered_non_unique< tag< ParentEntType_mi_tag >, const_mem_fun< RefEntity, EntityType, &RefEntity::getParentEntType > >, ordered_non_unique< tag< Composite_EntType_and_ParentEntType_mi_tag >, composite_key< RefEntity, const_mem_fun< RefEntity::BasicEntity, EntityType, &RefEntity::getEntType >, const_mem_fun< RefEntity, EntityType, &RefEntity::getParentEntType > > >, ordered_non_unique< tag< Composite_ParentEnt_And_EntType_mi_tag >, composite_key< RefEntity, const_mem_fun< RefEntity::BasicEntity, EntityType, &RefEntity::getEntType >, const_mem_fun< RefEntity, EntityHandle, &RefEntity::getParentEnt > > > > > RefEntity_multiIndex
#define MoFEMFunctionBegin
First executable line of each MoFEM function, used for error handling. Final line of MoFEM functions ...
Definition: definitions.h:412
virtual MPI_Comm & get_comm() const =0

Member Data Documentation

◆ cOre

MoFEM::Core& MoFEM::CommInterface::cOre

Definition at line 38 of file CommInterface.hpp.

◆ dEbug

bool MoFEM::CommInterface::dEbug

Definition at line 39 of file CommInterface.hpp.


The documentation for this struct was generated from the following files: