v0.13.2
Loading...
Searching...
No Matches
Public Member Functions | Public Attributes | List of all members
MoFEM::CommInterface Struct Reference

Managing BitRefLevels. More...

#include <src/interfaces/CommInterface.hpp>

Inheritance diagram for MoFEM::CommInterface:
[legend]
Collaboration diagram for MoFEM::CommInterface:
[legend]

Public Member Functions

MoFEMErrorCode query_interface (boost::typeindex::type_index type_index, UnknownInterface **iface) const
 
 CommInterface (const MoFEM::Core &core)
 
 ~CommInterface ()=default
 Destructor. More...
 
- Public Member Functions inherited from MoFEM::UnknownInterface
virtual MoFEMErrorCode query_interface (boost::typeindex::type_index type_index, UnknownInterface **iface) const =0
 
template<class IFACE >
MoFEMErrorCode registerInterface (bool error_if_registration_failed=true)
 Register interface. More...
 
template<class IFACE >
MoFEMErrorCode getInterface (IFACE *&iface) const
 Get interface refernce to pointer of interface. More...
 
template<class IFACE >
MoFEMErrorCode getInterface (IFACE **const iface) const
 Get interface pointer to pointer of interface. More...
 
template<class IFACE , typename boost::enable_if< boost::is_pointer< IFACE >, int >::type = 0>
IFACE getInterface () const
 Get interface pointer to pointer of interface. More...
 
template<class IFACE , typename boost::enable_if< boost::is_reference< IFACE >, int >::type = 0>
IFACE getInterface () const
 Get reference to interface. More...
 
template<class IFACE >
IFACE * getInterface () const
 Function returning pointer to interface. More...
 
virtual ~UnknownInterface ()=default
 

Public Attributes

MoFEM::CorecOre
 
bool dEbug
 

Make elemnts multishared

MoFEMErrorCode resolveSharedFiniteElements (const Problem *problem_ptr, const std::string &fe_name, int verb=DEFAULT_VERBOSITY)
 resolve shared entities for finite elements in the problem More...
 
MoFEMErrorCode resolveSharedFiniteElements (const std::string name, const std::string &fe_name, int verb=DEFAULT_VERBOSITY)
 resolve shared entities for finite elements in the problem More...
 

Make entities multishared

MoFEMErrorCode makeEntitiesMultishared (const EntityHandle *entities, const int num_entities, const int owner_proc=0, int verb=DEFAULT_VERBOSITY)
 make entities from proc 0 shared on all proc More...
 
MoFEMErrorCode makeEntitiesMultishared (Range &entities, const int owner_proc=0, int verb=DEFAULT_VERBOSITY)
 make entities from proc 0 shared on all proc More...
 
MoFEMErrorCode makeFieldEntitiesMultishared (const std::string field_name, const int owner_proc=0, int verb=DEFAULT_VERBOSITY)
 make field entities multi shared More...
 
MoFEMErrorCode exchangeFieldData (const std::string field_name, int verb=DEFAULT_VERBOSITY)
 Exchange field data. More...
 

Synchronize entities (Following functions in future will be

deprecated)

MoFEMErrorCode synchroniseEntities (Range &ent, int verb=DEFAULT_VERBOSITY)
 
MoFEMErrorCode synchroniseFieldEntities (const std::string name, int verb=DEFAULT_VERBOSITY)
 
MoFEMErrorCode resolveParentEntities (const Range &ent, int verb=DEFAULT_VERBOSITY)
 Synchronise parent entities. More...
 

Read load and boradcoast

@

MoFEMErrorCode partitionMesh (const Range &ents, const int dim, const int adj_dim, const int n_parts, Tag *th_vertex_weights=nullptr, Tag *th_edge_weights=nullptr, Tag *th_part_weights=nullptr, int verb=VERBOSE, const bool debug=false)
 Set partition tag to each finite element in the problem. More...
 

Additional Inherited Members

- Static Public Member Functions inherited from MoFEM::UnknownInterface
static MoFEMErrorCode getLibVersion (Version &version)
 Get library version. More...
 
static MoFEMErrorCode getFileVersion (moab::Interface &moab, Version &version)
 Get database major version. More...
 
static MoFEMErrorCode setFileVersion (moab::Interface &moab, Version version=Version(MoFEM_VERSION_MAJOR, MoFEM_VERSION_MINOR, MoFEM_VERSION_BUILD))
 Get database major version. More...
 
static MoFEMErrorCode getInterfaceVersion (Version &version)
 Get database major version. More...
 

Detailed Description

Managing BitRefLevels.

Examples
dm_partitioned_no_field.cpp, elasticity.cpp, level_set.cpp, mesh_smoothing.cpp, mixed_poisson.cpp, navier_stokes.cpp, and partition_mesh.cpp.

Definition at line 21 of file CommInterface.hpp.

Constructor & Destructor Documentation

◆ CommInterface()

MoFEM::CommInterface::CommInterface ( const MoFEM::Core core)

Definition at line 23 of file CommInterface.cpp.

24 : cOre(const_cast<MoFEM::Core &>(core)), dEbug(false) {}
Core (interface) class.
Definition: Core.hpp:82

◆ ~CommInterface()

MoFEM::CommInterface::~CommInterface ( )
default

Destructor.

Member Function Documentation

◆ exchangeFieldData()

MoFEMErrorCode MoFEM::CommInterface::exchangeFieldData ( const std::string  field_name,
int  verb = DEFAULT_VERBOSITY 
)

Exchange field data.

Exchange field for all shared and ghosted entities. This function should be called collectively over the communicator for this ParallelComm. If the entities vector is empty, all shared entities participate in the exchange. If a proc has no owned entities this function must still be called since it is collective.

Note
collective - need tu be run on all processors in communicator
Todo:
It is not working if field has entities diffrent than vertices.
Parameters
verb
field_name
Returns
MoFEMErrorCode

Definition at line 797 of file CommInterface.cpp.

798 {
799 MoFEM::Interface &m_field = cOre;
801 if (m_field.get_comm_size() > 1) {
802
803 Range exchange_ents_data_verts, exchange_ents_data;
804
805 auto *field_ents = m_field.get_field_ents();
806 auto field_bit_number = m_field.get_field_bit_number(field_name);
807 auto lo = field_ents->get<Unique_mi_tag>().lower_bound(
808 FieldEntity::getLoBitNumberUId(field_bit_number));
809 auto hi = field_ents->get<Unique_mi_tag>().lower_bound(
810 FieldEntity::getHiBitNumberUId(field_bit_number));
811
812 for (auto it = lo; it != hi; ++it)
813 if (
814
815 ((*it)->getPStatus()) &&
816
817 (*it)->getNbDofsOnEnt()
818
819 ) {
820 if ((*it)->getEntType() == MBVERTEX)
821 exchange_ents_data_verts.insert((*it)->getEnt());
822 else
823 exchange_ents_data.insert((*it)->getEnt());
824 }
825
826 auto field_ptr = m_field.get_field_structure(field_name);
827 ParallelComm *pcomm = ParallelComm::get_pcomm(
828 &m_field.get_moab(), m_field.get_basic_entity_data_ptr()->pcommID);
829
830 auto exchange = [&](const Range &ents, Tag th) {
832 if (!ents.empty()) {
833 std::vector<Tag> tags;
834 tags.push_back(th);
835 CHKERR pcomm->exchange_tags(tags, tags, ents);
836 }
838 };
839
840 CHKERR exchange(exchange_ents_data_verts, field_ptr->th_FieldDataVerts);
841 CHKERR exchange(exchange_ents_data, field_ptr->th_FieldData);
842 }
844}
#define MoFEMFunctionBegin
First executable line of each MoFEM function, used for error handling. Final line of MoFEM functions ...
Definition: definitions.h:346
#define MoFEMFunctionReturn(a)
Last executable line of each PETSc function used for error handling. Replaces return()
Definition: definitions.h:416
#define CHKERR
Inline error check.
Definition: definitions.h:535
virtual const FieldEntity_multiIndex * get_field_ents() const =0
Get the field ents object.
virtual const Field * get_field_structure(const std::string &name, enum MoFEMTypes bh=MF_EXIST) const =0
get field structure
constexpr auto field_name
virtual int get_comm_size() const =0
virtual FieldBitNumber get_field_bit_number(const std::string name) const =0
get field bit number
virtual moab::Interface & get_moab()=0
virtual boost::shared_ptr< BasicEntityData > & get_basic_entity_data_ptr()=0
Get pointer to basic entity data.
Deprecated interface functions.
static UId getHiBitNumberUId(const FieldBitNumber bit_number)
static UId getLoBitNumberUId(const FieldBitNumber bit_number)

◆ makeEntitiesMultishared() [1/2]

MoFEMErrorCode MoFEM::CommInterface::makeEntitiesMultishared ( const EntityHandle entities,
const int  num_entities,
const int  owner_proc = 0,
int  verb = DEFAULT_VERBOSITY 
)

make entities from proc 0 shared on all proc

Note
collective - need tu be run on all processors in communicator
Parameters
entities
num_entities
my_procdefault proc id to share from
verb
Returns
MoFEMErrorCode

Definition at line 633 of file CommInterface.cpp.

635 {
636 MoFEM::Interface &m_field = cOre;
638
639 if (m_field.get_comm_size() > 1) {
640
641 ParallelComm *pcomm = ParallelComm::get_pcomm(
642 &m_field.get_moab(), m_field.get_basic_entity_data_ptr()->pcommID);
643
644 Range all_ents_range;
645 all_ents_range.insert_list(entities, entities + num_entities);
646
647 auto get_tag = [&]() { return m_field.get_moab().globalId_tag(); };
648
649 auto delete_tag = [&](auto &&th_gid) {
651 CHKERR m_field.get_moab().tag_delete(th_gid);
653 };
654
655 auto resolve_shared_ents = [&](auto &&th_gid, auto &all_ents_range) {
656 auto set_gid = [&](auto &th_gid) {
657 std::vector<int> gids(num_entities);
658 for (size_t g = 0; g != all_ents_range.size(); ++g)
659 gids[g] = g + 1;
660 CHKERR m_field.get_moab().tag_set_data(th_gid, all_ents_range,
661 &*gids.begin());
662
663 return &th_gid;
664 };
665
666 auto get_skin_ents = [&](auto &all_ents_range) {
667 std::array<Range, 4> proc_ents_skin;
668 proc_ents_skin[3] = all_ents_range.subset_by_dimension(3);
669 proc_ents_skin[2] = all_ents_range.subset_by_dimension(2);
670 proc_ents_skin[1] = all_ents_range.subset_by_dimension(1);
671 proc_ents_skin[0] = all_ents_range.subset_by_dimension(0);
672 return proc_ents_skin;
673 };
674
675 auto resolve_dim = [&](auto &all_ents_range) {
676 for (int resolve_dim = 3; resolve_dim >= 0; --resolve_dim) {
677 if (all_ents_range.num_of_dimension(resolve_dim))
678 return resolve_dim;
679 }
680 return -1;
681 };
682
683 auto get_proc_ent = [&](auto &all_ents_range) {
684 Range proc_ent;
685 if (m_field.get_comm_rank() == owner_proc)
686 proc_ent = all_ents_range;
687 return proc_ent;
688 };
689
690 auto resolve_shared_ents = [&](auto &&proc_ents, auto &&skin_ents) {
691 return pcomm->resolve_shared_ents(
692 0, proc_ents, resolve_dim(all_ents_range),
693 resolve_dim(all_ents_range), skin_ents.data(), set_gid(th_gid));
694 };
695
696 CHKERR resolve_shared_ents(get_proc_ent(all_ents_range),
697 get_skin_ents(all_ents_range));
698
699 return th_gid;
700 };
701
702 CHKERR delete_tag(resolve_shared_ents(get_tag(), all_ents_range));
703
704 if (verb >= NOISY) {
705
706 auto print_owner = [&](const EntityHandle e) {
708 int moab_owner_proc;
709 EntityHandle moab_owner_handle;
710 CHKERR pcomm->get_owner_handle(e, moab_owner_proc, moab_owner_handle);
711
712 unsigned char pstatus = 0;
713
714 CHKERR m_field.get_moab().tag_get_data(pcomm->pstatus_tag(), &e, 1,
715 &pstatus);
716
717 std::vector<int> shprocs(MAX_SHARING_PROCS, 0);
718 std::vector<EntityHandle> shhandles(MAX_SHARING_PROCS, 0);
719
720 CHKERR m_field.get_moab().tag_get_data(pcomm->sharedp_tag(), &e, 1,
721 &shprocs[0]);
722 CHKERR m_field.get_moab().tag_get_data(pcomm->sharedh_tag(), &e, 1,
723 &shhandles[0]);
724 if (pstatus & PSTATUS_MULTISHARED) {
725 CHKERR m_field.get_moab().tag_get_data(pcomm->sharedps_tag(), &e, 1,
726 &shprocs[0]);
727 CHKERR m_field.get_moab().tag_get_data(pcomm->sharedhs_tag(), &e, 1,
728 &shhandles[0]);
729 }
730
731 std::ostringstream ss;
732
733 ss << "Rank " << m_field.get_comm_rank() << " ";
734 if (!(pstatus & PSTATUS_NOT_OWNED))
735 ss << "OWNER ";
736 if (pstatus & PSTATUS_SHARED)
737 ss << "PSTATUS_SHARED ";
738 if (pstatus & PSTATUS_MULTISHARED)
739 ss << "PSTATUS_MULTISHARED ";
740
741 ss << "owner " << moab_owner_proc << " (" << owner_proc << ") ";
742
743 ss << "shprocs: ";
744 for (size_t r = 0; r != m_field.get_comm_size() + 1; ++r)
745 ss << shprocs[r] << " ";
746
747 ss << "shhandles: ";
748 for (size_t r = 0; r != m_field.get_comm_size() + 1; ++r)
749 ss << shhandles[r] << " ";
750
751 ss << std::endl;
752 MOFEM_LOG("SYNC", Sev::noisy) << ss.str();
754
756 };
757
758 for (auto e : all_ents_range)
759 CHKERR print_owner(e);
760 }
761 }
762
764}
#define MOFEM_LOG_SYNCHRONISE(comm)
Synchronise "SYNC" channel.
Definition: LogManager.hpp:338
@ NOISY
Definition: definitions.h:211
#define MOFEM_LOG(channel, severity)
Log.
Definition: LogManager.hpp:301
const double r
rate factor
constexpr double g
virtual MPI_Comm & get_comm() const =0
virtual int get_comm_rank() const =0

◆ makeEntitiesMultishared() [2/2]

MoFEMErrorCode MoFEM::CommInterface::makeEntitiesMultishared ( Range entities,
const int  owner_proc = 0,
int  verb = DEFAULT_VERBOSITY 
)

make entities from proc 0 shared on all proc

Note
collective - need tu be run on all processors in communicator
Parameters
entities
my_procdefault proc id to share from
verb
Returns
MoFEMErrorCode

Definition at line 766 of file CommInterface.cpp.

768 {
769 MoFEM::Interface &m_field = cOre;
771 if (m_field.get_comm_size() > 1) {
772 const int num_ents = entities.size();
773 std::vector<EntityHandle> vec_ents(num_ents);
774 std::copy(entities.begin(), entities.end(), vec_ents.begin());
775 CHKERR makeEntitiesMultishared(&*vec_ents.begin(), num_ents, owner_proc,
776 verb);
777 }
779}
MoFEMErrorCode makeEntitiesMultishared(const EntityHandle *entities, const int num_entities, const int owner_proc=0, int verb=DEFAULT_VERBOSITY)
make entities from proc 0 shared on all proc

◆ makeFieldEntitiesMultishared()

MoFEMErrorCode MoFEM::CommInterface::makeFieldEntitiesMultishared ( const std::string  field_name,
const int  owner_proc = 0,
int  verb = DEFAULT_VERBOSITY 
)

make field entities multi shared

Note
collective - need tu be run on all processors in communicator
Parameters
field_name
owner_proc
verb
Returns
MoFEMErrorCode

Definition at line 782 of file CommInterface.cpp.

783 {
784 MoFEM::Interface &m_field = cOre;
786 if (m_field.get_comm_size() > 1) {
787 EntityHandle field_meshset = m_field.get_field_meshset(field_name);
788 std::vector<EntityHandle> field_ents;
789 CHKERR m_field.get_moab().get_entities_by_handle(field_meshset, field_ents,
790 true);
791 CHKERR makeEntitiesMultishared(&*field_ents.begin(), field_ents.size(),
792 owner_proc, verb);
793 }
795}
virtual EntityHandle get_field_meshset(const std::string name) const =0
get field meshset

◆ query_interface()

MoFEMErrorCode MoFEM::CommInterface::query_interface ( boost::typeindex::type_index  type_index,
UnknownInterface **  iface 
) const
virtual

Implements MoFEM::UnknownInterface.

Definition at line 16 of file CommInterface.cpp.

17 {
19 *iface = const_cast<CommInterface *>(this);
21}
#define MoFEMFunctionReturnHot(a)
Last executable line of each PETSc function used for error handling. Replaces return()
Definition: definitions.h:447
#define MoFEMFunctionBeginHot
First executable line of each MoFEM function, used for error handling. Final line of MoFEM functions ...
Definition: definitions.h:440
CommInterface(const MoFEM::Core &core)

◆ resolveParentEntities()

MoFEMErrorCode MoFEM::CommInterface::resolveParentEntities ( const Range ent,
int  verb = DEFAULT_VERBOSITY 
)

Synchronise parent entities.

Note
collective - need tu be run on all processors in communicator

Exchange parent entity tag and bitref of entity. Note thar parent handle can be different on each processor.

Parameters
ent
verb
Returns
MoFEMErrorCode

Definition at line 235 of file CommInterface.cpp.

236 {
237 MoFEM::Interface &m_field = cOre;
238 ParallelComm *pcomm = ParallelComm::get_pcomm(
239 &m_field.get_moab(), m_field.get_basic_entity_data_ptr()->pcommID);
240
242
243 Range shared = ents;
244 CHKERR pcomm->filter_pstatus(shared, PSTATUS_NOT_OWNED, PSTATUS_NOT, -1,
245 nullptr);
246 CHKERR pcomm->filter_pstatus(shared, PSTATUS_SHARED | PSTATUS_MULTISHARED,
247 PSTATUS_OR, -1, nullptr);
248
249 auto th_RefParentHandle = cOre.get_th_RefParentHandle();
250 auto th_RefBitLevel = cOre.get_th_RefBitLevel();
251
252 auto get_pstatus = [&](const auto ent) {
253 unsigned char pstatus;
254 CHK_MOAB_THROW(m_field.get_moab().tag_get_data(pcomm->pstatus_tag(), &ent,
255 1, &pstatus),
256 "can not get pstatus");
257 return pstatus;
258 };
259
260 auto get_sharing_procs = [&](const auto ent, const auto pstatus) {
261 std::vector<int> sharing_procs(MAX_SHARING_PROCS, -1);
262 if (pstatus & PSTATUS_MULTISHARED) {
263 // entity is multi shared
264 CHK_MOAB_THROW(m_field.get_moab().tag_get_data(
265 pcomm->sharedps_tag(), &ent, 1, &sharing_procs[0]),
266 "can not ger sharing_procs_ptr");
267 } else if (pstatus & PSTATUS_SHARED) {
268 // shared
269 CHK_MOAB_THROW(m_field.get_moab().tag_get_data(pcomm->sharedp_tag(), &ent,
270 1, &sharing_procs[0]),
271 "can not get sharing proc");
272 }
273 return sharing_procs;
274 };
275
276 auto get_sharing_handles = [&](const auto ent, const auto pstatus) {
277 std::vector<EntityHandle> sharing_handles(MAX_SHARING_PROCS, 0);
278 if (pstatus & PSTATUS_MULTISHARED) {
279 // entity is multi shared
280 CHK_MOAB_THROW(m_field.get_moab().tag_get_data(
281 pcomm->sharedhs_tag(), &ent, 1, &sharing_handles[0]),
282 "get shared handles");
283 } else if (pstatus & PSTATUS_SHARED) {
284 // shared
285 CHK_MOAB_THROW(m_field.get_moab().tag_get_data(pcomm->sharedh_tag(), &ent,
286 1, &sharing_handles[0]),
287 "get sharing handle");
288 }
289 return sharing_handles;
290 };
291
292 auto get_parent_and_bit = [&](const auto ent) {
293 EntityHandle parent;
295 m_field.get_moab().tag_get_data(th_RefParentHandle, &ent, 1, &parent),
296 "get parent");
299 m_field.get_moab().tag_get_data(th_RefBitLevel, &ent, 1, &bit),
300 "get parent");
301 return std::make_pair(parent, bit);
302 };
303
304 auto set_parent = [&](const auto ent, const auto parent) {
305 return m_field.get_moab().tag_set_data(th_RefParentHandle, &ent, 1,
306 &parent);
307 };
308
309 auto set_bit = [&](const auto ent, const auto bit) {
310 return m_field.get_moab().tag_set_data(th_RefBitLevel, &ent, 1, &bit);
311 };
312
313 // make a buffer
314 std::vector<std::vector<unsigned long long>> sbuffer(m_field.get_comm_size());
315
316 for (auto ent : shared) {
317
318 auto pstatus = get_pstatus(ent);
319 auto sharing_procs = get_sharing_procs(ent, pstatus);
320 auto sharing_handles = get_sharing_handles(ent, pstatus);
321 auto [parent, bit] = get_parent_and_bit(ent);
322
323 if (verb >= NOISY)
324 MOFEM_LOG("SYNC", Sev::noisy) << "pstatus " << std::bitset<8>(pstatus);
325
326 if (parent) {
327 auto pstatus_parent = get_pstatus(parent);
328 auto sharing_procs_parent = get_sharing_procs(parent, pstatus_parent);
329 auto sharing_handles_parent = get_sharing_handles(parent, pstatus_parent);
330
331 for (int proc = 0; proc < MAX_SHARING_PROCS && -1 != sharing_procs[proc];
332 proc++) {
333 if (sharing_procs[proc] == -1)
334 SETERRQ(PETSC_COMM_SELF, MOFEM_IMPOSSIBLE_CASE,
335 "sharing processor not set");
336
337 if (sharing_procs[proc] != m_field.get_comm_rank()) {
338
339 auto it = std::find(sharing_procs_parent.begin(),
340 sharing_procs_parent.end(), sharing_procs[proc]);
341 if (it == sharing_procs_parent.end()) {
342 SETERRQ1(
343 PETSC_COMM_SELF, MOFEM_DATA_INCONSISTENCY,
344 "Sharing proc for parent entity can not be found proc = %u",
345 sharing_procs[proc]);
346 }
347
348 auto handle_on_sharing_proc = sharing_handles[proc];
349 auto parent_handle_on_sharing_proc =
350 sharing_handles_parent[std::distance(sharing_procs_parent.begin(),
351 it)];
352 sbuffer[sharing_procs[proc]].push_back(handle_on_sharing_proc);
353 sbuffer[sharing_procs[proc]].push_back(parent_handle_on_sharing_proc);
354 try {
355 sbuffer[sharing_procs[proc]].push_back(bit.to_ullong());
356 } catch (std::exception &ex) {
357 MOFEM_LOG("SELF", Sev::warning) << ex.what();
358 MOFEM_LOG("SELF", Sev::warning)
359 << "On " << ent << " "
360 << moab::CN::EntityTypeName(type_from_handle(ent));
361 MOFEM_LOG("SELF", Sev::warning) << "For bit ref " << bit;
362 }
363 if (verb >= NOISY)
364 MOFEM_LOG_C("SYNC", Sev::noisy, "send %lu (%lu) to %d at %d\n", ent,
365 handle_on_sharing_proc, sharing_procs[proc],
366 m_field.get_comm_rank());
367
368 if (!(pstatus & PSTATUS_MULTISHARED))
369 break;
370 }
371 }
372 } else {
373 for (int proc = 0; proc < MAX_SHARING_PROCS && -1 != sharing_procs[proc];
374 proc++) {
375 if (sharing_procs[proc] == -1)
376 SETERRQ(PETSC_COMM_SELF, MOFEM_IMPOSSIBLE_CASE,
377 "sharing processor not set");
378
379 if (sharing_procs[proc] != m_field.get_comm_rank()) {
380 auto handle_on_sharing_proc = sharing_handles[proc];
381 sbuffer[sharing_procs[proc]].push_back(handle_on_sharing_proc);
382 sbuffer[sharing_procs[proc]].push_back(parent);
383
384 try {
385 sbuffer[sharing_procs[proc]].push_back(bit.to_ullong());
386 } catch (std::exception &ex) {
387 MOFEM_LOG("SELF", Sev::warning) << ex.what();
388 MOFEM_LOG("SELF", Sev::warning)
389 << "On " << ent << " "
390 << moab::CN::EntityTypeName(type_from_handle(ent));
391 MOFEM_LOG("SELF", Sev::warning) << "For bit ref " << bit;
392 }
393
394 if (verb >= NOISY)
395 MOFEM_LOG_C("SYNC", Sev::noisy, "send %lu (%lu) to %d at %d\n", ent,
396 handle_on_sharing_proc, sharing_procs[proc],
397 m_field.get_comm_rank());
398
399 if (!(pstatus & PSTATUS_MULTISHARED))
400 break;
401 }
402 }
403 }
404 }
405
406 int nsends = 0; // number of messages to send
407 std::vector<int> sbuffer_lengths(
408 m_field.get_comm_size()); // length of the message to proc
409
410 const size_t block_size = sizeof(unsigned long long) / sizeof(int);
411 for (int proc = 0; proc < m_field.get_comm_size(); proc++) {
412
413 if (!sbuffer[proc].empty()) {
414
415 sbuffer_lengths[proc] = sbuffer[proc].size() * block_size;
416 nsends++;
417
418 } else {
419
420 sbuffer_lengths[proc] = 0;
421 }
422 }
423
424 // Make sure it is a PETSc m_field.get_comm()
425 MPI_Comm comm;
426 CHKERR PetscCommDuplicate(m_field.get_comm(), &comm, NULL);
427
428 std::vector<MPI_Status> status(m_field.get_comm_size());
429
430 // Computes the number of messages a node expects to receive
431 int nrecvs; // number of messages received
432 CHKERR PetscGatherNumberOfMessages(comm, NULL, &sbuffer_lengths[0], &nrecvs);
433
434 // Computes info about messages that a MPI-node will receive, including
435 // (from-id,length) pairs for each message.
436 int *onodes; // list of node-ids from which messages are expected
437 int *olengths; // corresponding message lengths
438 CHKERR PetscGatherMessageLengths(comm, nsends, nrecvs, &sbuffer_lengths[0],
439 &onodes, &olengths);
440
441 // Gets a unique new tag from a PETSc communicator. All processors that share
442 // the communicator MUST call this routine EXACTLY the same number of times.
443 // This tag should only be used with the current objects communicator; do NOT
444 // use it with any other MPI communicator.
445 int tag;
446 CHKERR PetscCommGetNewTag(comm, &tag);
447
448 // Allocate a buffer sufficient to hold messages of size specified in
449 // olengths. And post Irecvs on these buffers using node info from onodes
450 int **rbuf; // must bee freed by user
451 MPI_Request *r_waits; // must bee freed by user
452 // rbuf has a pointers to messages. It has size of of nrecvs (number of
453 // messages) +1. In the first index a block is allocated,
454 // such that rbuf[i] = rbuf[i-1]+olengths[i-1].
455 CHKERR PetscPostIrecvInt(comm, tag, nrecvs, onodes, olengths, &rbuf,
456 &r_waits);
457
458 MPI_Request *s_waits; // status of sens messages
459 CHKERR PetscMalloc1(nsends, &s_waits);
460
461 // Send messages
462 for (int proc = 0, kk = 0; proc < m_field.get_comm_size(); proc++) {
463 if (!sbuffer_lengths[proc])
464 continue; // no message to send to this proc
465 CHKERR MPI_Isend(&(sbuffer[proc])[0], // buffer to send
466 sbuffer_lengths[proc], // message length
467 MPIU_INT, proc, // to proc
468 tag, comm, s_waits + kk);
469 kk++;
470 }
471
472 // Wait for received
473 if (nrecvs)
474 CHKERR MPI_Waitall(nrecvs, r_waits, &status[0]);
475
476 // Wait for send messages
477 if (nsends)
478 CHKERR MPI_Waitall(nsends, s_waits, &status[0]);
479
480 if (verb >= VERBOSE) {
481 MOFEM_LOG_C("SYNC", Sev::verbose,
482 "Rank %d nb. shared to synchronise parents ents %u\n",
483 m_field.get_comm_rank(), shared.size());
484 }
485
486 // synchronise range
487 for (int kk = 0; kk < nrecvs; kk++) {
488
489 int len = olengths[kk];
490 int *data_from_proc = rbuf[kk];
491
492 for (int ee = 0; ee < len;) {
493 EntityHandle ent;
494 EntityHandle parent;
495 unsigned long long uulong_bit;
496 bcopy(&data_from_proc[ee], &ent, sizeof(EntityHandle));
497 ee += block_size;
498 bcopy(&data_from_proc[ee], &parent, sizeof(EntityHandle));
499 ee += block_size;
500 bcopy(&data_from_proc[ee], &uulong_bit, sizeof(unsigned long long));
501 ee += block_size;
502
503 CHKERR set_parent(ent, parent);
504 CHKERR set_bit(ent, BitRefLevel(uulong_bit));
505
506 if (verb >= VERBOSE) {
507 MOFEM_LOG_C("SYNC", Sev::noisy, "received %lu (%lu) from %d at %d\n",
508 ent, parent, onodes[kk], m_field.get_comm_rank());
509 MOFEM_LOG("SYNC", Sev::noisy) << "Bit " << BitRefLevel(uulong_bit);
510 }
511 }
512 }
513
514 // Cleaning
515 CHKERR PetscFree(s_waits);
516 CHKERR PetscFree(rbuf[0]);
517 CHKERR PetscFree(rbuf);
518 CHKERR PetscFree(r_waits);
519 CHKERR PetscFree(onodes);
520 CHKERR PetscFree(olengths);
521 CHKERR PetscCommDestroy(&comm);
522
523 if (verb >= VERBOSE)
525
527}
#define MOFEM_LOG_C(channel, severity, format,...)
Definition: LogManager.hpp:304
@ VERBOSE
Definition: definitions.h:209
#define CHK_MOAB_THROW(err, msg)
Check error code of MoAB function and throw MoFEM exception.
Definition: definitions.h:576
@ MOFEM_IMPOSSIBLE_CASE
Definition: definitions.h:35
@ MOFEM_DATA_INCONSISTENCY
Definition: definitions.h:31
auto bit
set bit
std::bitset< BITREFLEVEL_SIZE > BitRefLevel
Bit structure attached to each entity identifying to what mesh entity is attached.
Definition: Types.hpp:40
auto type_from_handle(const EntityHandle h)
get type from entity handle
Definition: Templates.hpp:1594
Tag get_th_RefBitLevel() const
Definition: Core.hpp:198
Tag get_th_RefParentHandle() const
Definition: Core.hpp:197

◆ synchroniseEntities()

MoFEMErrorCode MoFEM::CommInterface::synchroniseEntities ( Range ent,
int  verb = DEFAULT_VERBOSITY 
)

synchronize entity range on processors (collective)

Note
collective - need tu be run on all processors in communicator

Definition at line 26 of file CommInterface.cpp.

26 {
27 MoFEM::Interface &m_field = cOre;
28 ParallelComm *pcomm = ParallelComm::get_pcomm(
29 &m_field.get_moab(), m_field.get_basic_entity_data_ptr()->pcommID);
31
32 auto get_pstatus = [&](const auto ent) {
33 unsigned char pstatus;
34 CHK_MOAB_THROW(m_field.get_moab().tag_get_data(pcomm->pstatus_tag(), &ent,
35 1, &pstatus),
36 "can not get pstatus");
37 return pstatus;
38 };
39
40 auto get_sharing_procs = [&](const auto ent, const auto pstatus) {
41 std::vector<int> sharing_procs(MAX_SHARING_PROCS, -1);
42 if (pstatus & PSTATUS_MULTISHARED) {
43 // entity is multi shared
44 CHK_MOAB_THROW(m_field.get_moab().tag_get_data(
45 pcomm->sharedps_tag(), &ent, 1, &sharing_procs[0]),
46 "can not ger sharing_procs_ptr");
47 } else if (pstatus & PSTATUS_SHARED) {
48 // shared
49 CHK_MOAB_THROW(m_field.get_moab().tag_get_data(pcomm->sharedp_tag(), &ent,
50 1, &sharing_procs[0]),
51 "can not get sharing proc");
52 }
53 return sharing_procs;
54 };
55
56 auto get_sharing_handles = [&](const auto ent, const auto pstatus) {
57 std::vector<EntityHandle> sharing_handles(MAX_SHARING_PROCS, 0);
58 if (pstatus & PSTATUS_MULTISHARED) {
59 // entity is multi shared
60 CHK_MOAB_THROW(m_field.get_moab().tag_get_data(
61 pcomm->sharedhs_tag(), &ent, 1, &sharing_handles[0]),
62 "get shared handles");
63 } else if (pstatus & PSTATUS_SHARED) {
64 // shared
65 CHK_MOAB_THROW(m_field.get_moab().tag_get_data(pcomm->sharedh_tag(), &ent,
66 1, &sharing_handles[0]),
67 "get sharing handle");
68 }
69 return sharing_handles;
70 };
71
72 // make a buffer
73 std::vector<std::vector<EntityHandle>> sbuffer(m_field.get_comm_size());
74
75 for (auto ent : ents) {
76
77 auto pstatus = get_pstatus(ent);
78 if (pstatus) {
79 auto sharing_procs = get_sharing_procs(ent, pstatus);
80 auto sharing_handles = get_sharing_handles(ent, pstatus);
81
82 if (verb >= NOISY) {
83 MOFEM_LOG("SYNC", Sev::noisy) << "pstatus " << std::bitset<8>(pstatus);
84 }
85
86 for (int proc = 0; proc < MAX_SHARING_PROCS && -1 != sharing_procs[proc];
87 proc++) {
88 if (sharing_procs[proc] == -1)
89 SETERRQ(PETSC_COMM_SELF, MOFEM_IMPOSSIBLE_CASE,
90 "sharing processor not set");
91
92 if (sharing_procs[proc] == m_field.get_comm_rank())
93 continue;
94
95 const auto handle_on_sharing_proc = sharing_handles[proc];
96 sbuffer[sharing_procs[proc]].push_back(handle_on_sharing_proc);
97 if (verb >= NOISY)
98 MOFEM_LOG_C("SYNC", Sev::noisy, "send %lu (%lu) to %d at %d\n", ent,
99 handle_on_sharing_proc, sharing_procs[proc],
100 m_field.get_comm_rank());
101
102 if (!(pstatus & PSTATUS_MULTISHARED))
103 break;
104 }
105 }
106 }
107
108 int nsends = 0; // number of messages to send
109 std::vector<int> sbuffer_lengths(
110 m_field.get_comm_size()); // length of the message to proc
111 const size_t block_size = sizeof(EntityHandle) / sizeof(int);
112 for (int proc = 0; proc < m_field.get_comm_size(); proc++) {
113
114 if (!sbuffer[proc].empty()) {
115
116 sbuffer_lengths[proc] = sbuffer[proc].size() * block_size;
117 nsends++;
118
119 } else {
120
121 sbuffer_lengths[proc] = 0;
122 }
123 }
124
125 // Make sure it is a PETSc m_field.get_comm()
126 MPI_Comm comm;
127 CHKERR PetscCommDuplicate(m_field.get_comm(), &comm, NULL);
128
129 std::vector<MPI_Status> status(m_field.get_comm_size());
130
131 // Computes the number of messages a node expects to receive
132 int nrecvs; // number of messages received
133 CHKERR PetscGatherNumberOfMessages(comm, NULL, &sbuffer_lengths[0], &nrecvs);
134
135 // Computes info about messages that a MPI-node will receive, including
136 // (from-id,length) pairs for each message.
137 int *onodes; // list of node-ids from which messages are expected
138 int *olengths; // corresponding message lengths
139 CHKERR PetscGatherMessageLengths(comm, nsends, nrecvs, &sbuffer_lengths[0],
140 &onodes, &olengths);
141
142 // Gets a unique new tag from a PETSc communicator. All processors that share
143 // the communicator MUST call this routine EXACTLY the same number of times.
144 // This tag should only be used with the current objects communicator; do NOT
145 // use it with any other MPI communicator.
146 int tag;
147 CHKERR PetscCommGetNewTag(comm, &tag);
148
149 // Allocate a buffer sufficient to hold messages of size specified in
150 // olengths. And post Irecvs on these buffers using node info from onodes
151 int **rbuf; // must bee freed by user
152 MPI_Request *r_waits; // must bee freed by user
153 // rbuf has a pointers to messages. It has size of of nrecvs (number of
154 // messages) +1. In the first index a block is allocated,
155 // such that rbuf[i] = rbuf[i-1]+olengths[i-1].
156 CHKERR PetscPostIrecvInt(comm, tag, nrecvs, onodes, olengths, &rbuf,
157 &r_waits);
158
159 MPI_Request *s_waits; // status of sens messages
160 CHKERR PetscMalloc1(nsends, &s_waits);
161
162 // Send messages
163 for (int proc = 0, kk = 0; proc < m_field.get_comm_size(); proc++) {
164 if (!sbuffer_lengths[proc])
165 continue; // no message to send to this proc
166 CHKERR MPI_Isend(&(sbuffer[proc])[0], // buffer to send
167 sbuffer_lengths[proc], // message length
168 MPIU_INT, proc, // to proc
169 tag, comm, s_waits + kk);
170 kk++;
171 }
172
173 // Wait for received
174 if (nrecvs)
175 CHKERR MPI_Waitall(nrecvs, r_waits, &status[0]);
176
177 // Wait for send messages
178 if (nsends)
179 CHKERR MPI_Waitall(nsends, s_waits, &status[0]);
180
181 if (verb >= VERBOSE) {
182 MOFEM_LOG_C("SYNC", Sev::verbose, "Rank %d nb. before ents %u\n",
183 m_field.get_comm_rank(), ents.size());
184 }
185
186 // synchronise range
187 for (int kk = 0; kk < nrecvs; kk++) {
188
189 int len = olengths[kk];
190 int *data_from_proc = rbuf[kk];
191
192 for (int ee = 0; ee < len;) {
193 EntityHandle ent;
194 bcopy(&data_from_proc[ee], &ent, sizeof(EntityHandle));
195 ents.insert(ent);
196 ee += block_size;
197
198 if (verb >= VERBOSE)
199 MOFEM_LOG_C("SYNC", Sev::noisy, "received %lu from %d at %d\n", ent,
200 onodes[kk], m_field.get_comm_rank());
201 }
202 }
203
204 if (verb >= VERBOSE)
205 MOFEM_LOG_C("SYNC", Sev::verbose, "Rank %d nb. after ents %u",
206 m_field.get_comm_rank(), ents.size());
207
208 // Cleaning
209 CHKERR PetscFree(s_waits);
210 CHKERR PetscFree(rbuf[0]);
211 CHKERR PetscFree(rbuf);
212 CHKERR PetscFree(r_waits);
213 CHKERR PetscFree(onodes);
214 CHKERR PetscFree(olengths);
215 CHKERR PetscCommDestroy(&comm);
216
217 if (verb >= VERBOSE)
219
221}

Member Data Documentation

◆ cOre

MoFEM::Core& MoFEM::CommInterface::cOre

Definition at line 26 of file CommInterface.hpp.

◆ dEbug

bool MoFEM::CommInterface::dEbug

Definition at line 27 of file CommInterface.hpp.


The documentation for this struct was generated from the following files: