How to cite us?

If you write a paper using results obtained with the help of MoFEM, please cite one or more of the following references:

Version 0.5.42 cite as

DOI:

L. Kaczmarczyk, Z. Ullah, K. Lewandowski, Xuan Meng, Xiao-Yi Zhou, C. Pearce. (2017). MoFEM-v0.5.42 [Data set]. Zenodo. http://doi.org/10.5281/zenodo.438712

@misc{mofem_v0.5.42,
  author       = {L. Kaczmarczyk, Z. Ullah, K. Lewandowski, Xuan Meng, Xiao-Yi Zhou, Chris Pearce},
  title        = {MoFEM-v0.5.42},
  month        = mar,
  year         = 2017,
  note         = {http://mofem.eng.gla.ac.uk/mofem/html/},
  doi          = {10.5281/zenodo.438712},
  url          = {https://doi.org/10.5281/zenodo.438712}
}

Each user module can have own DOI and reference. All module references supplements MoFEM reference. For example homogenisation module DOI as follows

How MoFEM version is changed?

The current MoFEM version can be identified by Semantic Versioning and Git Commit Id. Git Commit Id is unique and points to particular code commit. Semantic Versioning is not unique, more than one commits to git repository can have the same version.

First two lines of every executed code which is build with MoFEM library look like that

version 0.3.24 <--- (MAJOR_VERSION.MINOR_VERSION.BUILD_VERSION)

git commit id 3d06924d27973df16e3260c6e962929330cf9348

That allows to identify git commit id and human readable MoFEM version.

MoFEM is developed continuously (see How MoFEM is developed?) and any commit which introducing changes that directly have an impact on users modules implementation should result in increment build version. For example if new functionality is added, name of interface function changed or some function deprecated etc. that will result in incremented build version. Note that each users module has set minimal MoFEM version to which is compatible, see How to add user module? for details.

On the other hand changes/improvements to core library like modification of local variable name or when local documentation added, that will result in new commit however will NOT result in new build version, since implementation of users modules is not influenced by changes in main library.

Build version is in range from 0-100, at the end of the range minor version should be automatically incremented. Minor version is in the range from 0-10, at the end of the range major version should be automatically incremented. The minor or major version could be changed at any time when major/minor feature is initiated.

In addition to above rules, the general principles could apply, in short,

3.7.11 (bug fix), incremental change in build version
3.7.12 (partial new feature, hidden behind a Feature Toggle.), incremental change in build version
3.7.13 (performance improvement), incremental change in build version
3.8.0 (completed feature initiated in 3.7.12), direct change in minor version
4.0.0 (breaking changes in public API), direct change in major version

The MoFEM version can be set in CMakeList.txt located in root source directory, in lines

set(MoFEM_VERSION_MAJOR 0)
set(MoFEM_VERSION_MINOR 3)
set(MoFEM_VERSION_BUILD 24)

How MoFEM is developed?

MoFEM is developed continuously, i.e. any merged pull request to the CDashTesting branch triggers automatic testing on the development server. The code is verified during when a pull request is accepted and merged and validated when the test on the development server are passed. If tests are passed CDashBranch is merged to master branch.

How to generate this documentation on your local machine?

In you library directory execute

make doc

This create directory html. Open file html/index.html to see results in your browser.

How to partition mesh?

If problem is large, mesh can be partitioned for to save memory and improve efficiency. This can be done using native MoFEM tool,

$USER_MODULES/tools/mofem_part -my_file $USER_MODULES/basic_finite_elements/nonlinear_elasticity/examples/cylinder.cub -my_nparts 16

The partitioned mesh is saved to file out.h5m in current working directory.

For large meshes, partitioning can be in parallel, for example

mpirun -np 2 $USER_MODULES/tools/mofem_part -my_file $USER_MODULES/basic_finite_elements/nonlinear_elasticity/examples/cylinder.cub -my_nparts 16

How to multi-grid solver via approximation orders?

Code is run using direct solver, i.e. MUMPS on coarse level. Note that loaded mesh is portioned and each processor only reads part of the mesh, i.e. -my_is_partitioned.

mpirun -np 4 ./elasticity \
  -my_file dam_4parts.h5m -my_is_partitioned \
  -ksp_type gmres -ksp_max_it 1000  -ksp_atol 1e-13 -ksp_rtol 0  -ksp_monitor_lg_residualnorm  -ksp_final_residual -ksp_monitor -ksp_converged_reason
  -my_order 1  -my_block_config block_congig.in   \
  -mofem_mg_verbose 1 -mofem_mg_coarse_order 1 -mofem_mg_levels 4  \
  -pc_type mg \
  -mg_coarse_ksp_type preonly -mg_coarse_pc_type lu -mg_coarse_pc_factor_mat_solver_package mumps \
  -pc_mg_smoothup 20 -pc_mg_smoothdown 20  -pc_mg_type multiplicative

Option -my_is_partitioned is set if mesh is partitioned using mbpart
Option -mofem_mg_coarse_order 1 set coarse level is for linear approximation, i.e. order 1.
Option -mofem_mg_levels 4 set number of multi-grid level. In that case maximal approx. order for some part of mesh is 4, thus 4 multi-grid levels.
Option -pc_mg_smoothup 20 -pc_mg_smoothdown 20 set number of smoothing iterations, for more details look to PETSc manual.
In line
-mg_coarse_ksp_type preonly -mg_coarse_pc_type lu -mg_coarse_pc_factor_mat_solver_package mumps
a direct solver for coarse mesh is set.

How to update MoFEM on Live USB Stick?

MoFEM update on Live USB stick:

mofem_update.sh

mofem_build.sh

Following command run test verifying updated code:

mofem_fast_check.sh

If you run MoFEM update at University of Glasgow behind proxy server, set proxy servers as follows:

export http_proxy=http://wwwcache.gla.ac.uk:8080

export https_proxy=http://wwwcache.gla.ac.uk:8080

How to run ctest?

You can run tests and report results to MoFEM CDash web page. Form mofem user modules build directory executing run script

./bin/mofem_fast_check.sh

Results of test can be seen on http://cdash.eng.gla.ac.uk/cdash/.

Note that test tests for MoFEM library and User Modules are run independently and can be seen as a two different projects.

If you run test behind proxy server you can have to set http_proxy and http_proxy environmental variables. For example if you run mofem at Glasgow University, please do:

export http_proxy=http://wwwcache.gla.ac.uk:8080

export https_proxy=http://wwwcache.gla.ac.uk:8080

You can as well run ctest directly by simply executing command line:

ctest -V -D Experimental

where option -V sets verbose version and all test output is printed on screen and -D Experimental tels ctest to submit results to Experimental build on CDash MoFEM server.

How to make 2nd order geometry in cubit and transfer it to MoFEM?

In cubit you need to generate 10 node tetrahedral. You simply create block and set element type to TETRA10, as follows

set duplicate block elements on
block 2 volume 1
block 2 element type TETRA10

In the code, you need to create field to keep geometry

ierr = m_field.add_field("MESH_NODE_POSITIONS",H1,3,MF_ZERO); CHKERRQ(ierr);

ierr = m_field.add_ents_to_field_by_TETs(0,"MESH_NODE_POSITIONS",2); CHKERRQ(ierr);

Next set order of approximation field. If you have 10 node tetrahedrons, you need to have at least 2nd order polynomials to approximate geometry;

ierr = m_field.set_field_order(0,MBTET,"MESH_NODE_POSITIONS",2); CHKERRQ(ierr);
ierr = m_field.set_field_order(0,MBTRI,"MESH_NODE_POSITIONS",2); CHKERRQ(ierr);
ierr = m_field.set_field_order(0,MBEDGE,"MESH_NODE_POSITIONS",2); CHKERRQ(ierr);
ierr = m_field.set_field_order(0,MBVERTEX,"MESH_NODE_POSITIONS",1); CHKERRQ(ierr);

The last step is to prject information from 10 node terahedrons on hierarchical approximation field, as follows

Projection10NodeCoordsOnField ent_method_material(m_field,"MESH_NODE_POSITIONS");

ierr = m_field.loop_dofs("MESH_NODE_POSITIONS",ent_method_material); CHKERRQ(ierr);

Look at examples of use to user modules, for example elasticity.cpp.

How to add user module?

MoFEM is a core library providing functionality for implementation of user modules where applications for particular finite elements or problems are implemented. User module is an independent repository, private or public and independently managed by its owner.

User module is added to the project by cloning repository into directory $HOME/mofem-cephas/mofem/users_modules, for example, module for computational homogenisation has repository in Bitbucket and can be added by

cd $HOME/mofem-cephas/mofem/users_modules

git clone https://likask@bitbucket.org/likask/mofem_um_homogenisation.git homogenisation

Sometimes users modules depend on other modules, in that case, homogenisation module uses some old obsolete classes (which should not be used in new developments), thus in this particular case addition you have to clone also obsolete module

git clone https://likask@bitbucket.org/likask/mofem_um_obsolete.git obsolete

Once the module is added, you have to go main build directory where users modules are located and rebuild the code. So you have to do

cd $HOME/mofem_build/um
touch CMakeCache.txt
make -j 4

Note the first command is used to trigger reconfiguration of users modules with the new module.

Note, each user module consist InstalledAddModule.cmake, with beginning lines,

# Check minimum version of MoFEM with this module works
check_mofem_version(0 5 63)
add_subdirectory(${PROJECT_SOURCE_DIR}/homogenisation)

In that file minimal version of the core library is given (e.g. v0.5.63). Thus if you have too old version of core lib, it won't be added and cmake will generate an error. In that case, you need to update core library by pulling most recent version from Bitbucket repository and install core library.

User module can be independent repository, private or public and independently managed and owned form MoFEM library. If user module is part of MoFEM repository can be simply added by adding it to ModulesLists.cmake in users modules directory. However is recommended that user module is project/respository by its own, repository should be cloned to users modules directory, f.e. in mofem-cephas/mofem/users_modules

git clone https://likask@bitbucket.org/likask/mofem_um_homogenisation.git homogenisation

In user module directory directory should be file file, InstalledAddModule.cmake, with content,

# Check monimimal version of MoFEM with this module works
check_mofem_version(0 3 6)
add_subdirectory(${PROJECT_SOURCE_DIR}/homogenisation)

How to profile code using XCode tools?

To profile code in MacOS X environment, from line command you execute instruments, for example

instruments -v -t "Time Profiler" \
./elasticity -my_file LShape.h5m  -ksp_type fgmres -ksp_monitor -my_order 5 \
-pc_type mg -mofem_mg_verbose 1 -mofem_mg_coarse_order 1 \
-mofem_mg_levels 5 -mg_coarse_ksp_type preonly \
-mg_coarse_pc_type lu -mg_coarse_pc_factor_mat_solver_package mumps \
-pc_mg_smoothup 10 -pc_mg_smoothdown 10 -pc_mg_type multiplicative -log_view

This generates directory instrumentscli0.trace and for next run instrumentscli1.trace, and similarly for subsequent runs. You can see changes in execution of the code by

open instrumentscli0.trace

If you use Linux you can alternatively use Valgrind, see How to profile code with Valgrind?.

How to profile code using PETSc tools?

See PETSc documentation http://www.mcs.anl.gov/petsc/petsc-current/docs/manualpages/Profiling/PetscLogStageRegister.html and examples http://www.mcs.anl.gov/petsc/petsc-current/src/ksp/ksp/examples/tutorials/ex9.c.html

How to profile code with Valgrind?

You have to install Valgrind http://valgrind.org and graphic user interface KCachegrind http://kcachegrind.sourceforge.net/html/Home.html. If you using Linux, for example ubuntu you can do that executing following commands,

sudo apt-get install valgrind kcachegrind

If you using OS X you can use Homebrew http://brew.sh to make installation,

brew install valgrind

brew install qcachegrind --with-graphviz

If you have packages installed, follow instruction from http://kcachegrind.sourceforge.net/html/Documentation.html

How to make debugging in docekr?

You can use gdb inside docker, remembering to set cmake with debugging build type, for example

cmake -DCMAKE_BUILD_TYPE=Release -DCMAKE_CXX_FLAGS="-Wall" users_modules

It is one non trivial trick that you have to do, to make debugging possible in docker, you need to run docker container in privileged mode, for example

docker run --privileged -it --name work --hostname mofem -v $HOME:/mnt/home -v mofem_build:/my_build likask/mofem_build:v0.2 /bin/bash

Who contribuiting to MoFEM?

MoFEM is copyright by Lukasz Kaczmarczyk (Lukasz.Kaczmarczyk at glasgow.ac.uk) and has received contributions by several people. MoFEM can be used freely for educational, research or commercial purposes by other institutions.

Users modules are not part of MoFEM library and can be created privately, independently form MoFEM developers. User modules may follow different license agreement and copyrights. However, all users modules which are created at Glasgow University are under the same license agreement as main MoFEM library.

List is generated automatically at time when documentation is build based on contributions to main git repository. Note that particular module can have separate contributors list. If one author uses more than one name the same person can appear more than once.

Detailed git-statistic for core lib only http://mofem.eng.gla.ac.uk/mofem/html/git-stats

MoFEM lib contributors

  4085	Lukasz Kaczmarczyk <likask at wp.pl>
   875	likask <Lukasz.Kaczmarczyk at glasgow.ac.uk>
   211	Lukasz Kaczmarczyk <Lukasz.Kaczmarczyk at glasgow.ac.uk>
   177	Zahur Ullah <zahur.ullah at glasgow.ac.uk>
   114	Dimitrios Kourepinis <dimitrios.kourepinis at glasgow.ac.uk>
   103	Thomas Meng <m.xuan.1 at research.gla.ac.uk>
    73	Mike Cortis <mikecortis at gmail.com>
    61	Michael Cortis <m.cortis.1 at research.gla.ac.uk>
    43	Euan Muir <euan.j.muir at gmail.com>
    36	Ross Mackenzie <r.mackenzie.2 at research.gla.ac.uk>
    26	Karol Lewandowski <k.lewandowski.1 at research.gla.ac.uk>
    23	likask <lukasz at rdb-srv1.eng.gla.ac.uk>
    23	Euan Muir <e.muir.1 at research.gla.ac.uk>
    17	Hassan Al-Budairi <Hassan.Al-Budairi at glasgow.ac.uk>
    12	likask <likask at wp.pl>
    11	Zahur  Ullah <Zahur.Ullah at glasgow.ac.uk>
     7	zhou <xiaoyi.zhou at ncl.ac.uk>
     6	Michael Cortis <micgemma at gmail.com>
     6	Michael Cortis <michael at rdb-srv1.eng.gla.ac.uk>
     4	dkour <dimitrios.kourepinis at glasgow.ac.uk>
     3	mofem <Lukasz.Kaczmarczyk at glasgow.ac.uk>
     3	Euan Muir <euan at Euans-MacBook-Pro.local>
     2	Costy Kodsi <c.kodsi.1 at research.gla.ac.uk>
     2	Lukasz Kaczmarczyk <kkb14112 at archie-w.archie.cluster.local>
     2	mikecortis <m.cortis.1 at research.gla.ac.uk>
     2	rossmackenzie <r.mackenzie.2 at research.gla.ac.uk>
     1	DESKTOP-9KQ812P\erich <e.richardson.1 at research.gla.ac.uk>
     1	Chris Pearce <Chris.Pearce at glasgow.ac.uk>

Modules contributors

small_strain_plasticity
    67	Lukasz Kaczmarczyk <likask at wp.pl>
    12	Zahur Ullah <zahur.ullah at glasgow.ac.uk>
     6	Zahur  Ullah <Zahur.Ullah at glasgow.ac.uk>
     4	likask <Lukasz.Kaczmarczyk at glasgow.ac.uk>
     1	likask <lukasz at rdb-srv1.eng.gla.ac.uk>

bone_remodelling
   101	Lukasz Kaczmarczyk <likask at wp.pl>
    69	Karol Lewandowski <k.lewandowski.1 at research.gla.ac.uk>
     3	karol41 <k.lewandowski.1 at research.gla.ac.uk>
     1	John Marshall <John.F.Marshall at glasgow.ac.uk>

fracture_mechanics
    51	Lukasz Kaczmarczyk <likask at wp.pl>
     6	Zahur Ullah <zahur.ullah at glasgow.ac.uk>
     4	Chris Pearce <Chris.Pearce at glasgow.ac.uk>
     3	Lukasz Kaczmarczyk <Lukasz.Kaczmarczyk at glasgow.ac.uk>
     2	Karol Lewandowski <k.lewandowski.1 at research.gla.ac.uk>

cell_engineering
    16	Karol Lewandowski <k.lewandowski.1 at research.gla.ac.uk>
     7	Lukasz Kaczmarczyk <likask at wp.pl>

moisture_transport
     7	Lukasz Kaczmarczyk <likask at wp.pl>

obsolete
    35	Lukasz Kaczmarczyk <likask at wp.pl>

gels
   136	Euan Richardson <Euan Richardson>
    36	Lukasz Kaczmarczyk <likask at wp.pl>
    26	unknown <0800939r at erichardson-lap.eng-ad.gla.ac.uk>
     9	DESKTOP-9KQ812P\erich <e.richardson.1 at research.gla.ac.uk>
     5	Euan_R <e.richardson.1 at research.gla.ac.uk>
     1	Lukasz Kaczmarczyk <Lukasz.Kaczmarczyk at glasgow.ac.uk>
     1	likask <Lukasz.Kaczmarczyk at glasgow.ac.uk>
     1	unknown <0800939R at erichardson-lap.eng-ad.gla.ac.uk>
     1	Euan Richardson <euanrichardson at rdb-srv1.eng.gla.ac.uk>

ground_surface_temperature
    22	Lukasz Kaczmarczyk <likask at wp.pl>

helmholtz
    10	Lukasz Kaczmarczyk <likask at wp.pl>

homogenisation
   108	Lukasz Kaczmarczyk <likask at wp.pl>
    15	Zahur Ullah <zahur.ullah at glasgow.ac.uk>
     6	Zahur  Ullah <Zahur.Ullah at glasgow.ac.uk>
     5	likask <Lukasz.Kaczmarczyk at glasgow.ac.uk>
     1	Lukasz Kaczmarczyk <Lukasz.Kaczmarczyk at glasgow.ac.uk>

minimal_surface_equation
    23	Lukasz Kaczmarczyk <likask at wp.pl>

solid_shell_prism_element
   207	Lukasz Kaczmarczyk <likask at wp.pl>
    12	Zahur Ullah <zahur.ullah at glasgow.ac.uk>
     3	likask <Lukasz.Kaczmarczyk at glasgow.ac.uk>

optimal_mass_transport
     1	Lukasz Kaczmarczyk <likask at wp.pl>

List is generated with command,

git shortlog -s -e -n

Note that contributions to forked module are not on the list.