treelmesh_module.f90 Source File


This file depends on

sourcefile~~treelmesh_module.f90~~EfferentGraph sourcefile~treelmesh_module.f90 treelmesh_module.f90 sourcefile~tem_topology_module.f90 tem_topology_module.f90 sourcefile~treelmesh_module.f90->sourcefile~tem_topology_module.f90 sourcefile~tem_aux_module.f90 tem_aux_module.f90 sourcefile~treelmesh_module.f90->sourcefile~tem_aux_module.f90 sourcefile~tem_global_module.f90 tem_global_module.f90 sourcefile~treelmesh_module.f90->sourcefile~tem_global_module.f90 sourcefile~tem_sparta_module.f90 tem_sparta_module.f90 sourcefile~treelmesh_module.f90->sourcefile~tem_sparta_module.f90 sourcefile~tem_logging_module.f90 tem_logging_module.f90 sourcefile~treelmesh_module.f90->sourcefile~tem_logging_module.f90 sourcefile~tem_tools_module.f90 tem_tools_module.f90 sourcefile~treelmesh_module.f90->sourcefile~tem_tools_module.f90 sourcefile~tem_property_module.f90 tem_property_module.f90 sourcefile~treelmesh_module.f90->sourcefile~tem_property_module.f90 sourcefile~env_module.f90 env_module.f90 sourcefile~treelmesh_module.f90->sourcefile~env_module.f90 sourcefile~tem_topology_module.f90->sourcefile~env_module.f90 sourcefile~tem_aux_module.f90->sourcefile~tem_logging_module.f90 sourcefile~tem_aux_module.f90->sourcefile~tem_tools_module.f90 sourcefile~tem_aux_module.f90->sourcefile~env_module.f90 sourcefile~tem_revision_module.f90 tem_revision_module.f90 sourcefile~tem_aux_module.f90->sourcefile~tem_revision_module.f90 sourcefile~tem_comm_env_module.f90 tem_comm_env_module.f90 sourcefile~tem_aux_module.f90->sourcefile~tem_comm_env_module.f90 sourcefile~tem_lua_requires_module.f90 tem_lua_requires_module.f90 sourcefile~tem_aux_module.f90->sourcefile~tem_lua_requires_module.f90 sourcefile~tem_global_module.f90->sourcefile~tem_aux_module.f90 sourcefile~tem_global_module.f90->sourcefile~tem_logging_module.f90 sourcefile~tem_global_module.f90->sourcefile~env_module.f90 sourcefile~tem_prophead_module.f90 tem_prophead_module.f90 sourcefile~tem_global_module.f90->sourcefile~tem_prophead_module.f90 sourcefile~tem_sparta_module.f90->sourcefile~tem_aux_module.f90 sourcefile~tem_sparta_module.f90->sourcefile~tem_logging_module.f90 sourcefile~tem_sparta_module.f90->sourcefile~env_module.f90 sourcefile~tem_float_module.f90 tem_float_module.f90 sourcefile~tem_sparta_module.f90->sourcefile~tem_float_module.f90 sourcefile~tem_logging_module.f90->sourcefile~env_module.f90 sourcefile~tem_tools_module.f90->sourcefile~env_module.f90 sourcefile~tem_property_module.f90->sourcefile~env_module.f90 sourcefile~tem_property_module.f90->sourcefile~tem_prophead_module.f90 sourcefile~tem_float_module.f90->sourcefile~env_module.f90 sourcefile~tem_prophead_module.f90->sourcefile~env_module.f90 sourcefile~tem_lua_requires_module.f90->sourcefile~env_module.f90

Files dependent on this one

treelmesh_module.f90whvs_ascii_module.f90
w
whvs_output_module.f90
w
whvs_vtk_dummy.f90
w
whvs_vtk_module.f90
w
wtem_adaptation_module.f90
w
wtem_bc_prop_module.f90
w
wtem_canonical_module.f90
w
wtem_color_prop_module.f90
w
wtem_construction_module.f90
w
wtem_convergence_module.f90
w
wtem_cube_module.f90
w
wtem_face_module.f90
w
wtem_geometry_line.f90
w
wtem_geometry_module.f90
w
wtem_interpolation_module.f90
w
wtem_logical_operation_var_module.f90
w
wtem_logical_operator_test.f90
w
wtem_meshInfo_module.f90
w
wtem_operation_var_module.f90
w
wtem_parallel_singlelevel_test.f90
w
wtem_pointData_module.f90
w
wtem_reduction_spatial_module.f90
w
wtem_refining_module.f90
w
wtem_restart_module.f90
w
wtem_serial_multilevel_2_test.f90
w
wtem_serial_singlelevel_test.f90
w
wtem_shape_module.f90
w
wtem_spacetime_fun_module.f90
w
wtem_spacetime_fun_test.f90
w
wtem_spacetime_var_module.f90
w
wtem_spatial_module.f90
w
wtem_spongelayer_module.f90
w
wtem_subres_prop_module.f90
w
wtem_subTree_module.f90
w
wtem_subTree_type_module.f90
w
wtem_surfaceData_module.f90
w
wtem_tracking_module.f90
w
wtem_utestEnv_module.f90
w
wtem_variable_combine_test.f90
w
wtem_variable_evaltype_test.f90
w
wtem_variable_extract_test.f90
w
wtem_varSys_deriveVar_test.f90
w
wtem_varSys_module.f90
w
wtem_varSys_opVar_test.f90
w
wtem_varSys_stateVar_test.f90
w
wtem_varSys_stfunVar_test.f90
w
wtem_varSys_test.f90
w
wtem_vrtx_module.f90
w

Contents

Source Code


Source Code

! Copyright (c) 2011-2018,2021 Harald Klimach <harald.klimach@dlr.de>
! Copyright (c) 2011-2012 Manuel Hasert <m.hasert@grs-sim.de>
! Copyright (c) 2011-2014 Simon Zimny <s.zimny@grs-sim.de>
! Copyright (c) 2011 Metin Cakircali <m.cakircali@grs-sim.de>
! Copyright (c) 2011 Aravindh Krishnamoorthy <aravindh28.4@gmail.com>
! Copyright (c) 2011-2012 Khaled Ibrahim <k.ibrahim@grs-sim.de>
! Copyright (c) 2011, 2014-2016 Jiaxing Qi <jiaxing.qi@uni-siegen.de>
! Copyright (c) 2011-2016, 2019 Kannan Masilamani <kannan.masilamani@uni-siegen.de>
! Copyright (c) 2012 Jens Zudrop <j.zudrop@grs-sim.de>
! Copyright (c) 2013 Melven Zoellner <yameta@freenet.de>
! Copyright (c) 2013-2015 Kartik Jain <kartik.jain@uni-siegen.de>
! Copyright (c) 2014, 2017 Verena Krupp <verena.krupp@uni-siegen.de>
! Copyright (c) 2014 Monika Harlacher <monika.harlacher@uni-siegen.de>
! Copyright (c) 2016 Tobias Schneider <tobias1.schneider@student.uni-siegen.de>
! Copyright (c) 2016, 2018, 2020 Peter Vitt <peter.vitt2@uni-siegen.de>
! Copyright (c) 2016 Neda Ebrahimi Pour <neda.epour@uni-siegen.de>
! Copyright (c) 2017 Daniel PetrĂ³ <daniel.petro@student.uni-siegen.de>
! Copyright (c) 2017 Raphael Haupt <Raphael.Haupt@student.uni-siegen.de>
!
! Redistribution and use in source and binary forms, with or without
! modification, are permitted provided that the following conditions are met:
!
! 1. Redistributions of source code must retain the above copyright notice, this
! list of conditions and the following disclaimer.
!
! 2. Redistributions in binary form must reproduce the above copyright notice,
! this list of conditions and the following disclaimer in the documentation
! and/or other materials provided with the distribution.
!
! THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS"
! AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
! IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE
! DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT HOLDER OR CONTRIBUTORS BE LIABLE
! FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL
! DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR
! SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER
! CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY,
! OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE
! OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
! **************************************************************************** !
!
!> TREe based ELemental Mesh module, providing the definition and methods to
!! act on a octree based mesh.
!!
!! The mesh may either be externally generated by seeder or some simple mesh
!! configurations can also be created internally.
!! In case of an external mesh from seeder, the directory containing the mesh
!! needs to be provided in the configuration:
!!
!!```lua
!!  mesh = '/path/to/mesh/'
!!```
!!
!! Note that the string provided here is used as a prefix. Thus, if the mesh
!! is in a directory it needs to end in a path seperator.
!! The default is 'mesh/', which results in the application to look for the
!! mesh in the `mesh` subdirectory in the working directory of the application.
!!
!! Internal meshes are defined in a table, where the first option describes
!! which of the predefined meshes is to be generated, then there needs to be
!! a definition of the universe (or root) cube within which the octree lives.
!! This is given by an `origin` and then the `length` by which it extends along
!! each axis. Further options may be required by the respective predefined mesh
!! types.
!! An example for a fully periodic uniform cubical mesh:
!!
!!```lua
!!  mesh = {
!!    predefined = 'cube',
!!    origin = {0., 0., 0.},
!!    length = 10.,
!!    refinementLevel = 4
!!  }
!!```
!!
!! The following predefined meshes are available:
!!
!! * 'cube' create a fully periodic cube without boundary conditions and uniform
!!   element sizes by refining the root element (universe). For this kind of
!!   mesh you need to provide an `refinementLevel` which defines how often the
!!   encompassing cube should be bisected in each dimension leading to meshes
!!   with `8^refinementLevel` elements.
!! * 'slice' create a slice of the fully periodic cube with uniform elements.
!!   For the slice you need to provide the `refinementLevel`, as for the cube.
!!   The resulting mesh will consist of `4^refinementLevel` elements.
!! * 'line' creates a periodic line. For this kind of mesh it is possible to
!!   specify either an `refinementLevel` to create meshes with
!!   `2^refinementLevel` elements, or the number of elements in the mesh by
!!   setting `element_count`.
!! * 'line_bounded' create a line with boundary conditions in west (on the left)
!!   and east (on the right). For this mesh the length needs to be defined by
!!   `element_count`.
!!
!! The mesh is stored in [[treelmesh_type]].
!! The definition is designed to be used for parallel executions.
!! It includes a reference to [[tem_global_module:tem_global_type]]
!! and the number of elements, which were loaded on the local partition, as well
!! as the first and last tree ID of each partition in the current communicator.
!! This enables the determination of any tree ID's position among the processes,
!! independent of their actual existence.
!! This is important to identify the location of neighbor elements when creating
!! connectivity lists for the linearized element lists locally on each process
!! within the solvers.
!!
!! ![apes_schematic](|media|/apes_schematic.png)
!!
!! See also
!! ---
!!
!! - [[tem_balance_module]] "Load Balancing"
!! - [[tem_dyn_array_module]] "Dynamic Data Structures"
!!
module treelmesh_module

  ! include treelm modules
  use mpi
  use env_module, only: rk,                      &
    &                   long_k,                  &
    &                   long_k_mpi,              &
    &                   rk_mpi,                  &
    &                   PathLen,                 &
    &                   tem_create_EndianSuffix, &
    &                   globalMaxLevels,         &
    &                   io_buffer_size
  use tem_aux_module, only: tem_open, tem_abort, check_mpi_error
  use tem_global_module, only: tem_global_type, &
    &                          load_tem_global, &
    &                          dump_tem_global, &
    &                          tem_global_mesh_free
  use tem_property_module, only: tem_property_type, &
    &                            gather_property,   &
    &                            prp_hasbnd,        &
    &                            prp_sendHalo,      &
    &                            prp_hasQVal
  use tem_topology_module, only: tem_path_type,      &
    &                            tem_firstIdAtLevel, &
    &                            tem_lastIdAtLevel,  &
    &                            tem_levelOf,        &
    &                            tem_IDofCoord
  use tem_logging_module, only: logUnit
  use tem_tools_module, only: tem_horizontalSpacer
  use tem_sparta_module, only: tem_balance_sparta, &
    &                          tem_sparta_type,    &
    &                          tem_init_sparta,    &
    &                          tem_destroy_sparta, &
    &                          tem_exchange_sparta

  ! include aotus modules
  use aotus_module, only: aot_get_val,        &
    &                     flu_state,          &
    &                     aoterr_NonExistent, &
    &                     aoterr_WrongType,   &
    &                     aoterr_Fatal
  use aot_table_module, only: aot_table_open,  &
    &                         aot_table_close, &
    &                         aot_table_length

  implicit none

  private

  public :: load_tem
  public :: unload_treelmesh
  public :: treelmesh_type, load_treelmesh, dump_treelmesh
  public :: free_treelmesh
  public :: generate_treelm_cube
  public :: treelmesh_fromList
  public :: tem_load_internal
  public :: dump_MeshHeader
  public :: exchange_elements
  public :: tem_dump_weights
  public :: tem_load_weights

  !> declaration of the overall mesh on the local partition
  type treelmesh_type
    !> This entry provides some global informations on the mesh, which is needed
    !! across all partitions.
    type(tem_global_type) :: global

    ! @todo: need update after mesh changes
    integer :: nElems !< Total number of Elements on this partition
    integer(kind=long_k) :: elemOffset !< Offset of this partition

    !> List of treeIDs of the first element on each partition, has a length of
    !! global%nParts
    integer(kind=long_k), allocatable :: Part_First(:)

    !> List of treeIDs of the last element on each partition, has a length of
    !! global%nParts
    integer(kind=long_k), allocatable :: Part_Last(:)

    !> The treeIDs of each element (the array has a length of nElems)
    !! these IDs identify the element in the complete mesh, with a breadth-first
    !! numbering scheme. Allowing easy computation of parents and children.
    !! [treeid] \ref treeID
    !!
    !! # TreeID {#treeID}
    !! The coordinate description of an element has three spatial entries and a
    !! fourth entry indicating the refinement level in the octree, resulting in
    !! a tuple of four integers: \f$(x, y, z, L)\f$.
    !!
    !! This coordinate fully describes the spatial shape and position of an
    !! element and is important to determine spatial relations between elements.
    !! It can be obtained from a given treeID by the following procedure.
    !! 1. First, the \ref tem_topology_module::tem_levelof "refinement level" is
    !!    found.
    !! 2. Then a \ref tem_topology_module::tem_firstidatlevel "level-offset" is
    !!    calculated.
    !! 3. After the offset is known, the Morton index of the node in question is
    !!    obtained by subtracting the offset \f$s\f$ from the treeID of the
    !!    node.
    !! 4. Finally a simple bit interleaving rule allows the
    !!    \ref tem_topology_module::tem_coordofid "transformation from the
    !!    Morton index to the three dimensional coordinates".
    !!
    !! The spatial indices are limited by the refinement level:
    !! \f$x, y, z \in \mathbf{Z}_{\ge 0}: x, y, z < 2^L\f$.
    !! To avoid undefined coordinates, movements through the mesh by additions
    !! to indices are done in a modulo(\f$2^L\f$) safeguard resulting in a
    !! periodic universe cube.
    !! This enables the movement in the mesh in horizontal direction (e.g.
    !! for stencil neighbor identification) by index alteration and translation
    !! into a treeID again.
    !! @todo: change to dynamic array
    integer(kind=long_k), allocatable :: treeID(:)

    !> Bit field of element properties for each element (the array has a length
    !! of nElems), each bit in the 8 byte integer indicates the presence or
    !! absence of a given property, which are further described in the
    !! Property component.
    !! @todo: change to dynamic array
    integer(kind=long_k), allocatable :: ElemPropertyBits(:)

    !> declaration of additional elemental properties, the array has a length of
    !! global%nProperties, each property provides a list of local element
    !! indices of elements with this property.
    type(tem_property_type), pointer :: Property(:) => null()

    !> Path for all elements in the treeID array
    type(tem_path_type), allocatable :: pathList(:)

    !> Name of the file that contains weight information for each element to
    !! allow load balancing.
    !! If empty string, balancing will be deactivated.
    character(len=pathLen) :: weights_file = ''

    !> File to write weights to after simulation.
    !! Defaults to the same name as weights_file, if you do not want to have
    !! the input file overwritten, you can provide a different file to write
    !! to by setting write_weights accordingly in the configuration.
    character(len=pathLen) :: write_weights

    !> Level weights as provided by the solver.
    real(kind=rk) :: levelWeight(globalMaxLevels)
  end type


contains


  ! ************************************************************************ !
  !> Load the treelmesh
  !!
  !! Depending on the configuration given in conf, this will either load a
  !! a \ref tem_distributed_octree "tree mesh" from disk with the given prefix
  !! or \ref tem_load_internal "create the mesh internally".
  !! @todo: to calculate levelWeight, global has to be loaded fisrt,
  !!        maybe load_global shoule be moved out of this routine
  subroutine load_tem( me, conf, myPart, nParts, comm, levelWeight, meshDir, &
    &                  parent)
    ! -------------------------------------------------------------------- !
    !> Structure to load the mesh to
    type(treelmesh_type), intent(out) :: me
    !> Directory containing the mesh informations
    type(flu_State) :: conf
    !> Partition to use on the calling process (= MPI Rank in comm)
    integer, intent(in) :: myPart
    !> Number of partitions, the mesh is partitioned into (= Number of MPI
    !! processes in comm).
    integer, intent(in) :: nParts
    !> MPI Communicator to use
    !! now defaults to the one given in the tree%global%comm
    integer, intent(in) :: comm
    !> Balancing weight for elements on different levels.
    !! If these weights are present, the mesh is partitioned initially according
    !! to the given weights.
    !! This can not be combined with given offsets. If offset is also provided
    !! they will overwrite the levelWeight distribution.
    real(kind=rk), intent(in), optional :: levelWeight(globalMaxLevels)
    !> output Mesh directory name
    character(len=*), intent(out), optional :: meshDir
    !> optional parent handle
    integer, intent(in), optional :: parent
    ! -------------------------------------------------------------------- !
    integer :: tem_handle
    character(len=pathLen) :: dirname
    character(len=4) :: EndianSuffix
    integer :: mesh_error, weights_error
    ! The communicator which is used
    integer :: commLocal
    real(kind=rk), allocatable :: weights(:)
    integer(kind=long_k) :: chunksize, remainder
    logical :: found_weights
    type( tem_sparta_type ) :: sparta
    ! -------------------------------------------------------------------- !

    ! Use the incoming communicator
    commLocal = comm

    call tem_horizontalSpacer(fUnit = logUnit(1))

    EndianSuffix = tem_create_EndianSuffix()

    ! Make sure the dirname has some sane value.
    dirname = ''

    call aot_table_open(L       = conf,       &
      &                 parent  = parent,     &
      &                 thandle = tem_handle, &
      &                 key     = 'mesh'      )


    if (tem_handle /= 0) then ! key mesh is a table

      ! The mesh is actually given as a table, parse it and
      ! generate a mesh internally.
      write(logUnit(1),*) 'Generating an internally defined mesh'
      call tem_load_internal(me, conf, tem_handle, myPart, nParts, commLocal)

    else ! tem_handle == 0, mesh is not a table
      ! The mesh is not a table, try to interpret it as a string.
      call aot_get_val( L       = conf,       &
        &               thandle = parent,     &
        &               key     = 'mesh',     &
        &               val     = dirname,    &
        &               ErrCode = mesh_error, &
        &               default = 'mesh/'     )

      if (btest(mesh_error, aoterr_Fatal)) then
        write(logUnit(0),*) 'FATAL Error occured, while retrieving mesh folder!'
        if (btest(mesh_error, aoterr_NonExistent))then
          write(logUnit(0),*) 'Variable not existent!'
          write(logUnit(0),*) 'Using the default value mesh folder: mesh/'
        else if (btest(mesh_error, aoterr_WrongType))then
          write(logUnit(0),*) 'Variable has wrong type!'
          write(logUnit(0),*) 'STOPPING'
          call tem_abort()
        end if
      end if

      ! Load global information
      ! @todo: this routine may already be called outside
      if (associated(me%global%property)) deallocate(me%global%property)
      call load_tem_global(me%global, dirname, myPart, nParts, commLocal)

      ! Homogeneous distribution
      chunksize = me%global%nElems / nParts
      remainder = mod(me%global%nElems, int(nParts, kind=long_k))
      me%nElems = int(chunksize)
      if (myPart < int(remainder)) then
        me%nElems = me%nElems + 1
      end if
      me%elemOffset = chunksize * int(myPart, kind=long_k)     &
        &             + min(int(myPart, kind=long_k), remainder)

      call load_treelmesh( me     = me,    &
        &                  nParts = nParts )

    end if

    ! If requested, provide the directory name to the caller.
    if ( present(meshDir) ) then
      meshDir = trim( dirname )
    end if

    ! Make sure each process has at least one element!
    if (me%global%nElems/me%global%nParts == 0) then
      write(logunit(1),*) ''
      write(logunit(1),*) 'FATAL Error occured:'
      write(logunit(1),*) 'Less elements then processes, this is not'
      write(logunit(1),*) 'supported! And you should reconfigure your'
      write(logunit(1),*) 'setup.'
      write(logunit(1),*) ''
      write(logunit(1),*) 'Restart this simulation with at most ', &
        &                 me%global%nElems, ' processes!'
      call tem_abort()
    end if

    me%levelWeight = 0.0_rk
    if (present(levelWeight)) then
      me%levelWeight = levelWeight
    end if

    ! Get the name for the weights file.
    ! This defaults to the mesh prefix with weights appended.
    call aot_get_val( L       = conf,            &
      &               key     = 'weights',       &
      &               thandle = parent,          &
      &               val     = me%weights_file, &
      &               ErrCode = weights_error,   &
      &               default = ''               )
    if (.not. btest(weights_error, aoterr_NonExistent)) then
      if (myPart == 0 .and. trim(me%weights_file) /= '') then
        inquire(file=trim(me%weights_file)//EndianSuffix, exist=found_weights)
        if (.not. found_weights) then
          write(logUnit(1),*) '!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!'
          write(logUnit(1),*) 'Weight file '                         &
            &                 // trim(me%weights_file)//EndianSuffix &
            &                 // ' can NOT be found!'
          write(logUnit(1),*) 'May proceed WITHOUT BALANCING!'
          write(logUnit(1),*) '!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!'
        end if
      end if
    end if

    call aot_get_val( L       = conf,             &
      &               key     = 'write_weights',  &
      &               thandle = parent,           &
      &               val     = me%write_weights, &
      &               ErrCode = weights_error,    &
      &               default = me%weights_file   )

    if (trim(me%write_weights) /= '') then
      write(logunit(1),*) 'The computational weight of each element will be'
      write(logunit(1),*) 'written at the end of the simulation into file'
      write(logunit(1),*) trim(me%write_weights)//EndianSuffix
      write(logunit(1),*) 'If this is not desired, set write_weights explicitly'
      write(logunit(1),*) 'to an empty string.'
    end if

    ! Do sparta balancing, but only if there are actually multiple processes.
    if (me%global%nParts > 1) then
      allocate( weights(me%nElems) )
      call tem_load_weights( me      = me,           &
        &                    weights = weights,      &
        &                    success = found_weights )

      if ( found_weights ) then
        call tem_init_sparta( sparta, nParts )
        call tem_balance_Sparta(weight  = weights,      &
          &                     myPart  = myPart,       &
          &                     nParts  = nParts,       &
          &                     comm    = commLocal,    &
          &                     myElems = me%nElems,    &
          &                     sparta  = sparta,       &
          &                     offset  = me%elemOffset )
        call exchange_elements( me, sparta )
        call tem_destroy_sparta( sparta )
      end if
      deallocate( weights )
    end if

    ! Output basic mesh infos to the screen
    call dump_meshHeader( me )

    call aot_table_close(L=conf, thandle=tem_handle)

    call tem_horizontalSpacer(fUnit = logUnit(1))

  end subroutine load_tem
  ! ************************************************************************ !


  ! ************************************************************************ !
  !> Unload the treelmesh
  !!
  !! Deallocate the data within tree, while keep the global information
  !!
  subroutine unload_treelmesh( me )
    ! -------------------------------------------------------------------- !
    !> Structure to load the mesh to
    type(treelmesh_type), intent(inout) :: me
    ! -------------------------------------------------------------------- !

    write(logUnit(6),*) 'Deallocate the tree data structure'

    if (allocated(me%Part_first)) deallocate( me%Part_First )
    if (allocated(me%Part_last)) deallocate( me%Part_Last  )
    if (allocated(me%treeID)) deallocate( me%treeID  )
    if (allocated(me%ElemPropertyBits)) deallocate( me%ElemPropertyBits  )
    if (associated(me%property)) deallocate(me%property)
    if (allocated(me%pathList)) deallocate( me%pathList  )

  end subroutine unload_treelmesh
  ! ************************************************************************ !


  ! ************************************************************************ !
  !> Completely free the memory of the treelmesh again.
  subroutine free_treelmesh(me)
    !> Structure to free
    type(treelmesh_type), intent(inout) :: me

    call unload_treelmesh(me)
    call tem_global_mesh_free(me%global)
  end subroutine free_treelmesh
  ! ************************************************************************ !


  ! ************************************************************************ !
  !> Load an internally generated mesh
  !!
  !! The predefined meshes are illustrated in the following configuration
  !! snippet:
  !!
  !!```lua
  !! -- use the predefined full cube
  !! mesh = { predefined = 'cube',
  !!          -- for a single element in Z-direction use slice:
  !!          -- predefined = 'slice',
  !!
  !!          -- for just a line (single element in Y and Z) use line:
  !!          -- predefined = 'line',
  !!          -- If you set the refinementLevel to 0, a single element will
  !!          -- be created with treeID=1 on level 1, with the bounding cube
  !!          -- redefined such, that the element fills the original
  !!          -- specification for the bounding cube.
  !!          -- The line can also be generated with an arbitrary number of
  !!          -- elements instead of a level. In this case the length refers
  !!          -- to the overall length of the given elements not the extent of
  !!          -- the cube. Periodicity is assumed in X direction as well.
  !!          -- To create the line with a certain number of elements set the
  !!          -- element_count instead of the refinementLevel. If both are
  !!          -- given, the element_count overrules the refinementLevel.
  !!
  !!          -- for a line with boundary conditions line_bounded:
  !!          -- predefined = 'line_bounded'
  !!          -- Like line, but needs to have `element_count` given, not
  !!          -- `refinementLevel`. This mesh defines boundary conditions in the
  !!          -- X direction as west (on the left) and east (on the right), that
  !!          -- then need to be defined in the solver configuration.
  !! -- origin of the cube
  !! origin = {0.,0.,0.},
  !! -- length of the cube
  !! length = 10.,
  !! -- refinement level to resolve the cube
  !! refinementLevel = 4 }
  !!```
  !!
  subroutine tem_load_internal( me, conf, thandle, myPart, nParts, comm )
    ! -------------------------------------------------------------------- !
    !> Structure to load the mesh to
    type(treelmesh_type), intent(out) :: me
    !> Directory containing the mesh informations
    type(flu_State) :: conf
    !> Handle for the table to read the description
    !! of the mesh from.
    integer, intent(in) :: thandle
    !> Partition to use on the calling process (= MPI Rank in comm)
    integer, intent(in) :: myPart
    !> Number of partitions, the mesh is partitioned into (= Number of MPI
    !! processes in comm).
    integer, intent(in) :: nParts
    !> MPI Communicator to use
    integer, intent(in) :: comm
    ! -------------------------------------------------------------------- !
    character(len=40) :: meshtype
    real(kind=rk) :: origin(3)
    real(kind=rk) :: length
    integer :: elementcount
    integer :: level
    integer :: sub_handle
    integer :: iError, i
    integer :: orig_err(3)
    ! -------------------------------------------------------------------- !

    call aot_get_val( L       = conf,         &
      &               thandle = thandle,      &
      &               val     = meshtype,     &
      &               ErrCode = iError,       &
      &               key     = 'predefined', &
      &               default = 'cube'        )

    select case(trim(meshtype))
    case('cube')
      write(logUnit(1),*)'Creating a full cubical mesh without boundaries'

      ! Get the origin of the cube:
      call aot_table_open( L       = conf,       &
        &                  parent  = thandle,    &
        &                  thandle = sub_handle, &
        &                  key     = 'origin'    )
      if (aot_table_length(L = conf, thandle = sub_handle) == 3) then
        do i=1,3
          call aot_get_val( L       = conf,       &
            &               thandle = sub_handle, &
            &               val     = origin(i),  &
            &               ErrCode = iError,     &
            &               pos     = i           )
        end do
      else
        write(logUnit(1),*)'Origin specified for the mesh to generate has to ' &
          &            //'have 3 entries!'
        write(logUnit(1),*)'Default to {0., 0., 0.} '
        origin = 0.0_rk
      end if
      call aot_table_close( L=conf, thandle=sub_handle )

      ! Get the length of the cube:
      call aot_get_val( L       = conf,     &
        &               thandle = thandle,  &
        &               val     = length,   &
        &               ErrCode = iError,   &
        &               key     = 'length', &
        &               default = 1.0_rk    )

      ! Get the refinement level:
      call aot_get_val( L       = conf,             &
        &               thandle = thandle,          &
        &               val     = level,            &
        &               ErrCode = iError,           &
        &               key     = 'refinementLevel' )

      ! Now generate a mesh with the gathered configuration parameters
      if (level > 0) then
        ! Only if the level is greater than 0, actually create mesh.
        call generate_treelm_cube( me     = me,     &
          &                        origin = origin, &
          &                        length = length, &
          &                        level  = level,  &
          &                        myPart = myPart, &
          &                        nParts = nParts, &
          &                        comm   = comm    )
      else
        ! Meshes on level 0 have only one element, but we need treeids > 0,
        ! thus we create single elements on level 1 with treeid=1, and
        ! redefine the bounding cube accordingly.
        call generate_treelm_single( me     = me,       &
          &                          origin = origin,   &
          &                          length = 2*length, &
          &                          myPart = myPart,   &
          &                          nParts = nParts,   &
          &                          comm   = comm      )
      end if


    case('slice')
      write(logUnit(1),*)'Creating a slice (single element in Z) mesh' &
        &          //' without boundaries'

      ! Get the origin of the cube:
      origin = 0.0_rk
      call aot_table_open( L       = conf,       &
        &                  parent  = thandle,    &
        &                  thandle = sub_handle, &
        &                  key     = 'origin'    )
      if (aot_table_length(L = conf, thandle = sub_handle) >= 2) then
        do i=1,2
          call aot_get_val( L       = conf,       &
            &               thandle = sub_handle, &
            &               val     = origin(i),  &
            &               ErrCode = iError,     &
            &               pos     = i           )
        end do
      else
        write(logUnit(1),*)'Origin specified for the mesh to generate has to ' &
          &            // 'have 2 entries!'
        write(logUnit(1),*)'Default to {0., 0.} '
      end if
      call aot_get_val( L       = conf,       &
        &               thandle = sub_handle, &
        &               val     = origin(3),  &
        &               ErrCode = iError,     &
        &               Default = 0.0_rk,     &
        &               pos     = 3           )
      call aot_table_close( L=conf, thandle=sub_handle )

      ! Get the length of the cube:
      call aot_get_val( L       = conf,     &
        &               thandle = thandle,  &
        &               val     = length,   &
        &               ErrCode = iError,   &
        &               key     = 'length', &
        &               default = 1.0_rk    )

      ! Get the refinement level:
      call aot_get_val( L       = conf,             &
        &               thandle = thandle,          &
        &               val     = level,            &
        &               ErrCode = iError,           &
        &               key     = 'refinementLevel' )

      ! Now generate a mesh with the gathered configuration parameters
      if (level > 0) then
        ! Only if the level is greater than 0, actually create mesh.
        call generate_treelm_slice( me     = me,     &
          &                         origin = origin, &
          &                         length = length, &
          &                         level  = level,  &
          &                         myPart = myPart, &
          &                         nParts = nParts, &
          &                         comm   = comm    )
      else
        ! Meshes on level 0 have only one element, but we need treeids > 0,
        ! thus we create single elements on level 1 with treeid=1, and
        ! redefine the bounding cube accordingly.
        call generate_treelm_single( me     = me,       &
          &                          origin = origin,   &
          &                          length = 2*length, &
          &                          myPart = myPart,   &
          &                          nParts = nParts,   &
          &                          comm   = comm      )
      end if


    case('line')
      write(logUnit(1),*) 'Creating a line (single element in Y and Z) mesh' &
        &                 // ' without boundaries'

      ! Get the origin of the cube:
      origin = 0.0_rk
      call aot_table_open( L       = conf,       &
        &                  parent  = thandle,    &
        &                  thandle = sub_handle, &
        &                  key     = 'origin'    )
      if (aot_table_length(L = conf, thandle = sub_handle) >= 1) then
        call aot_get_val( L       = conf,       &
          &               thandle = sub_handle, &
          &               val     = origin(1),  &
          &               ErrCode = iError,     &
          &               pos     = 1           )
      else
        write(logUnit(1),*) 'Origin specified for the mesh to generate has' &
          &                 // ' to have 1 entry!'
        write(logUnit(1),*)'Default to {0.} '
      end if
      call aot_get_val( L       = conf,       &
        &               thandle = sub_handle, &
        &               val     = origin(2),  &
        &               ErrCode = iError,     &
        &               Default = 0.0_rk,     &
        &               pos     = 2           )
      call aot_get_val( L       = conf,       &
        &               thandle = sub_handle, &
        &               val     = origin(3),  &
        &               ErrCode = iError,     &
        &               Default = 0.0_rk,     &
        &               pos     = 3           )
      call aot_table_close( L=conf, thandle=sub_handle )

      ! Get the length of the cube:
      call aot_get_val( L       = conf,     &
        &               thandle = thandle,  &
        &               val     = length,   &
        &               ErrCode = iError,   &
        &               key     = 'length', &
        &               default = 1.0_rk    )

      ! Get the refinement level:
      call aot_get_val( L       = conf,              &
        &               thandle = thandle,           &
        &               val     = level,             &
        &               ErrCode = iError,            &
        &               key     = 'refinementLevel', &
        &               default = -1                 )

      ! Get the element count:
      call aot_get_val( L       = conf,            &
        &               thandle = thandle,         &
        &               val     = elementcount,    &
        &               ErrCode = iError,          &
        &               key     = 'element_count', &
        &               default = -1               )

      if ( (elementcount < 1) .and. (level < 0) ) then
        write(logUnit(1),*) 'For the line mesh you need to specify either'
        write(logUnit(1),*) 'the element_count or the refinementLevel!'
        write(logUnit(1),*) 'STOPPING...'
        call tem_abort()
      end if

      ! Now generate a mesh with the gathered configuration parameters
      if (elementcount > 1) then
        call generate_treelm_elements( me           = me,              &
          &                            origin       = origin,          &
          &                            length       = length,          &
          &                            elementcount = elementcount,    &
          &                            myPart       = myPart,          &
          &                            nParts       = nParts,          &
          &                            comm         = comm,            &
          &                            predefined   = trim(meshtype),  &
          &                            bclabel      = 'internal 1D BC' )
      else
        if ((level > 0) .and. (elementcount < 1)) then
          ! Only if the level is greater than 0, actually create mesh.
          call generate_treelm_line( me     = me,     &
            &                        origin = origin, &
            &                        length = length, &
            &                        level  = level,  &
            &                        myPart = myPart, &
            &                        nParts = nParts, &
            &                        comm   = comm    )
        else
          ! Meshes on level 0 have only one element, but we need treeids > 0,
          ! thus we create single elements on level 1 with treeid=1, and
          ! redefine the bounding cube accordingly.
          call generate_treelm_single( me     = me,       &
            &                          origin = origin,   &
            &                          length = 2*length, &
            &                          myPart = myPart,   &
            &                          nParts = nParts,   &
            &                          comm   = comm      )
        end if
      end if

    case('line_bounded')
      write(logUnit(1),*) 'Creating a line (single element in Y and Z) mesh' &
        &                 // ' WITH boundaries'
      write(logUnit(1),*) 'Boundaries west and east have to be provided!'

      ! Get the origin of the cube:
      origin = 0.0_rk
      call aot_table_open( L       = conf,       &
        &                  parent  = thandle,    &
        &                  thandle = sub_handle, &
        &                  key     = 'origin'    )
      if (aot_table_length(L = conf, thandle = sub_handle) >= 1) then
        call aot_get_val( L       = conf,       &
          &               thandle = sub_handle, &
          &               val     = origin(1),  &
          &               ErrCode = iError,     &
          &               pos     = 1           )
      else
        write(logUnit(1),*) 'Origin specified for the mesh to generate has' &
          &                 // ' to have 1 entry!'
        write(logUnit(1),*)'Default to {0.} '
      end if
      call aot_get_val( L       = conf,       &
        &               thandle = sub_handle, &
        &               val     = origin(2),  &
        &               ErrCode = iError,     &
        &               Default = 0.0_rk,     &
        &               pos     = 2           )
      call aot_get_val( L       = conf,       &
        &               thandle = sub_handle, &
        &               val     = origin(3),  &
        &               ErrCode = iError,     &
        &               Default = 0.0_rk,     &
        &               pos     = 3           )
      call aot_table_close( L=conf, thandle=sub_handle )

      ! Get the length of the cube:
      call aot_get_val( L       = conf,     &
        &               thandle = thandle,  &
        &               val     = length,   &
        &               ErrCode = iError,   &
        &               key     = 'length', &
        &               default = 1.0_rk    )

      ! Get the element count:
      call aot_get_val( L       = conf,            &
        &               thandle = thandle,         &
        &               val     = elementcount,    &
        &               ErrCode = iError,          &
        &               key     = 'element_count', &
        &               default = -1               )

      if ( (elementcount < 1) ) then
        write(logUnit(1),*) 'For the bounded line mesh you need to specify'
        write(logUnit(1),*) 'the element_count!'
        write(logUnit(1),*) 'STOPPING...'
        call tem_abort()
      end if

      ! Now generate a mesh with the gathered configuration parameters
      call generate_treelm_elements( me           = me,                      &
        &                            origin       = origin,                  &
        &                            length       = length,                  &
        &                            elementcount = elementcount,            &
        &                            myPart       = myPart,                  &
        &                            nParts       = nParts,                  &
        &                            comm         = comm,                    &
        &                            predefined   = trim(meshtype),          &
        &                            bclabel      = 'bounded internal 1D BC' )

    case('single')
      write(logUnit(1),*) 'Creating a single element mesh without boundaries'

      ! Get the origin of the cube:
      call aot_get_val( L       = conf,                    &
        &               thandle = thandle,                 &
        &               val     = origin,                  &
        &               ErrCode = orig_err,                &
        &               key     = 'origin',                &
        &               default = [0.0_rk, 0.0_rk, 0.0_rk] )

      ! Get the length of the cube:
      call aot_get_val( L       = conf,     &
        &               thandle = thandle,  &
        &               val     = length,   &
        &               ErrCode = iError,   &
        &               key     = 'length', &
        &               default = 1.0_rk    )

      ! Now generate a mesh with the gathered configuration parameters
      call generate_treelm_single( me     = me,     &
        &                          origin = origin, &
        &                          length = length, &
        &                          myPart = myPart, &
        &                          nParts = nParts, &
        &                          comm   = comm    )


    case('default')
      write(logUnit(1),*) 'Do not know how to generate mesh ' &
        &                 // trim(meshtype)

    end select

  end subroutine tem_load_internal
  ! ************************************************************************ !


  ! ************************************************************************ !
  !> This subroutine reads a mesh in treelm format from disk at the
  !! specified directory name.
  !!
  subroutine load_treelmesh(me, nParts )
    ! -------------------------------------------------------------------- !
    !> Structure to load the mesh to
    type(treelmesh_type), intent(inout) :: me
    !> Directory containing the mesh informations
    ! character(len=*), intent(in) :: dirname
    !> Partition to use on the calling process (= MPI Rank in comm)
    ! integer, intent(in) :: myPart
    !> Number of partitions, the mesh is partitioned into (= Number of MPI
    !! processes in comm).
    integer, intent(in) :: nParts
    ! -------------------------------------------------------------------- !
    logical :: ex
    integer :: iElem
    integer :: iProp
    integer :: fh, etype, ftype
    integer :: iostatus( MPI_STATUS_SIZE )
    integer :: iError
    integer :: file_rec_len
    integer :: typesize
    character(len=300)   :: ElemFileName
    character(len=4)     :: EndianSuffix
    integer(kind=long_k), allocatable :: buffer(:)
    integer(kind=long_k), allocatable :: globbuffer(:)
    integer(kind=MPI_OFFSET_KIND)     :: displacement
    ! -------------------------------------------------------------------- !

    EndianSuffix = tem_create_EndianSuffix()
    ElemFileName = trim(me%global%dirname)//'elemlist'//EndianSuffix
    write(logUnit(1), *) 'Load mesh from file: '//trim(ElemFileName)

    if (associated(me%property)) deallocate(me%property)
    allocate(me%Property(me%global%nProperties))
    allocate(me%Part_First(nParts))
    allocate(me%Part_Last(nParts))

    allocate(me%treeID(me%nElems))

    if (me%global%myPart == 0) then
      ! Only one process should check the file existence
      inquire(file=trim(ElemFileName), exist=ex)
      if (.not. ex) then
        write(logUnit(1),*) 'ERROR: File ' // trim(ElemFileName) &
          &                 // ' not found!'
        write(logUnit(1),*) 'Aborting.'
        call tem_abort()
      end if
    end if

    allocate(me%ElemPropertyBits(me%nElems))
    allocate(buffer(me%nElems*2))
    if (me%global%nElems*2 > io_buffer_size) then
      write(logUnit(1),*) 'using MPI to read'

      ! Create a contiguous type to describe the vector per element
      call MPI_Type_contiguous( 2, long_k_mpi, etype, iError )
      call check_mpi_error(iError,'type etype in load_treelmesh')
      call MPI_Type_commit(     etype, iError )
      call check_mpi_error(iError,'commit etype in load_treelmesh')
      call MPI_Type_size(etype, typesize, iError )
      call check_mpi_error(iError,'typesize in load_treelmesh')

      ! Create a MPI Subarray  as ftype for file view
      call MPI_Type_contiguous( me%nElems, etype, ftype, iError )
      call check_mpi_error(iError,'type ftype in load_treelmesh')
      call MPI_Type_commit(     ftype, iError )
      call check_mpi_error(iError,'commit ftype in load_treelmesh')

      ! Calculate displacement for file view
      displacement = me%elemOffset * typesize * 1_MPI_OFFSET_KIND

      ! Set the view for each process on the file above
      ! Open the binary file for MPI I/O (Write)
      call MPI_File_open( me%global%comm, trim(ElemFileName),        &
        &                 MPI_MODE_RDONLY, MPI_INFO_NULL, fh, iError )
      call check_mpi_error(iError,'file_open in load_treelmesh')

      call MPI_File_set_view( fh, displacement, etype, ftype, "native", &
        &                     MPI_INFO_NULL, iError                     )
      call check_mpi_error(iError,'set_view in load_treelmesh')
      ! Read data from the file
      call MPI_File_read_all( fh, buffer, me%nElems, etype, iostatus, iError )
      call check_mpi_error(iError,'read_all in load_treelmesh')

      !Free the MPI_Datatypes which were created and close the file
      call MPI_Type_free (etype, iError)
      call check_mpi_error(iError,'free etype in load_treelmesh')
      call MPI_Type_free (ftype, iError)
      call check_mpi_error(iError,'free ftype in load_treelmesh')
      call MPI_File_close(fh,    iError)
      call check_mpi_error(iError,'close file in load_treelmesh')

      ! END IO-part

    else

      !> The mesh is so small, it probably is better to read it on one process
      !! and distribute the resulting data via the network.
      write(logUnit(1),*) 'using one process to read and broadcast'
      allocate( globbuffer(me%global%nElems*2) )
      if (me%global%myPart == 0) then !! single proc read

        inquire(iolength = file_rec_len) globbuffer
        call tem_open( newunit = fh,                 &
          &            file    = trim(ElemFileName), &
          &            recl    = file_rec_len,       &
          &            action  = 'read',             &
          &            access  = 'direct',           &
          &            form    = 'unformatted'       )

        read(fh, rec=1) globbuffer

        close(fh)

      end if !! single proc read

      call MPI_Bcast( globbuffer, int(2*me%global%nElems), long_k_mpi, 0, &
        &             me%global%comm, iError                              )

      buffer = globbuffer(me%elemOffset*2+1:(me%ElemOffset+me%nElems)*2)
      deallocate(globbuffer)

    end if ! END IO-part

    ! Fill the tree with what was read in buffer from disk
    do iElem = 1, me%nElems
      me%treeID(iElem)           = buffer( (iElem-1)*2 + 1 )
      me%ElemPropertyBits(iElem) = buffer( (iElem-1)*2 + 2 )
    end do

    deallocate( buffer )

    ! clear the prp_sendHalo property since the distribution of elements
    ! might have changed
    do iElem=1,me%nElems
      me%ElemPropertyBits(iElem)  = ibclr( me%ElemPropertyBits(iElem), &
        &                                  prp_sendHalo                )
    end do

    call MPI_allgather( me%treeID(1), 1, long_k_mpi,  &
      &                 me%Part_First, 1, long_k_mpi, &
      &                 me%global%comm, iError        )
    call MPI_allgather( me%treeID(me%nElems), 1, long_k_mpi, &
      &                 me%Part_Last, 1, long_k_mpi,         &
      &                 me%global%comm, iError               )

    do iProp=1,me%global%nProperties
      call gather_Property( Property = me%Property(iProp),        &
        &                   Header   = me%global%Property(iProp), &
        &                   BitField = me%ElemPropertyBits,       &
        &                   comm     = me%global%comm             )
    end do

    write(logUnit(1),*) 'Done, reading the mesh!'

  end subroutine load_treelmesh
  ! ************************************************************************ !


  ! ************************************************************************ !
  !> Write a given mesh to disk. it is stored to the directory given in the
  !! [[tem_global_module:tem_global_type]].
  !!
  !! Dump treelmesh_type%global to header.lua
  !! Dump treeID and propertyBits to elemlist.lsb (little endian environments)
  !! or elemlist.msb (big endian environments)
  !!
  subroutine dump_treelmesh( me, root_only )
    ! -------------------------------------------------------------------- !
    !> Mesh to dump to disk
    type(treelmesh_type), intent(inout) :: me
    !> root dump global mesh when true and
    !! all process dump its own mesh when false
    logical, intent(in), optional       :: root_only
    ! -------------------------------------------------------------------- !
    integer(kind=long_k), allocatable   :: buffer(:)
    integer                             :: fh, etype, ftype
    integer                             :: iostatus( MPI_STATUS_SIZE )
    integer                             :: iElem
    integer                             :: iError
    integer                             :: typesize
    logical                             :: root_not_participating
    character(len=300)                  :: ElemFileName
    character(len=4)                    :: EndianSuffix
    integer(kind=long_k)                :: file_offset
    integer(kind=MPI_OFFSET_KIND)       :: displacement
    ! -------------------------------------------------------------------- !

    if(present(root_only)) then
      root_not_participating = root_only
    else
      root_not_participating = .true.
    endif

    EndianSuffix = tem_create_EndianSuffix()

    ! Dump global info to header.lua
    ! if root_not_participating = .false. that each process dumps its own mesh
    ! set global%nElems to local%nElems to dump local nElems
    ! in header file of each process
    if (.not. root_not_participating) me%global%nElems = me%nElems

    call dump_tem_global(me%global, root_only)
    ElemFileName = trim(me%global%dirname) // 'elemlist' // EndianSuffix

    file_offset = 0
    allocate( buffer(me%nElems * 2) )

    ! Get the offset for MPI File view
    if(root_not_participating) then
      call MPI_Exscan( int(me%nElems,long_k), file_offset, 1, &
        &              long_k_mpi, MPI_SUM, me%global%comm, iError)
    else
      file_offset = 0
    endif

    ! Fill the buffer with data to be written on disk
    do iElem = 1, me%nElems
      buffer((iElem-1)*2+1) = me%treeID(iElem)
      buffer((iElem-1)*2+2) = me%ElemPropertyBits(iElem)
    end do

    write(logUnit(1),*) 'using MPI to write'

    ! Open the binary file for MPI I/O (Write)
    call MPI_File_open( me%global%comm,                       &
      &                 trim(ElemFileName),                   &
      &                 ior(MPI_MODE_WRONLY,MPI_MODE_CREATE), &
      &                 MPI_INFO_NULL,                        &
      &                 fh, iError                            )

    ! Catch if there was an exception by MPI
    call check_mpi_error(iError,'file_open in dump_treelmesh')

    ! Create a contiguous type to describe the vector per element
    call MPI_Type_contiguous( 2, long_k_mpi, etype, iError )
    call check_mpi_error(iError,'type etype in dump_treelmesh')
    call MPI_Type_commit(     etype, iError )

    ! Catch MPI Exception(s)
    call check_mpi_error(iError,'commit etype in dump_treelmesh')

    !get size of etype
    call MPI_Type_size(etype, typesize, iError )
    call check_mpi_error(iError,'typesize in dump_treelmesh')

    ! Create a MPI Contiguous as ftype for file view
    call MPI_Type_contiguous( me%nElems, etype, ftype, iError )
    call check_mpi_error(iError,'type ftype in dump_treelmesh')
    call MPI_Type_commit(     ftype, iError )


    ! Catch MPI Exception(s)
    call check_mpi_error(iError,'commit ftype in dump_treelmesh')

    ! calculate dsplacement for file view
    displacement = file_offset * typesize * 1_MPI_OFFSET_KIND

    ! Set the view for each process on the file above
    call MPI_File_set_view( fh, displacement, etype, ftype, "native", &
      &                     MPI_INFO_NULL, iError                     )

    ! Catch MPI Exception(s)
    call check_mpi_error(iError,'set_view in dump_treelmesh')

    ! Write the Data to File
    call MPI_File_write_all( fh, buffer, me%nElems, etype, iostatus, &
      &                      iError                                  )

    ! Catch MPI Exception(s)
    call check_mpi_error(iError,'write_all in dump_treelmesh')

    !Free the MPI_Datatypes which were created and close the file
    call MPI_Type_free(etype, iError)
    call check_mpi_error(iError,'free etype in dump_treelmesh')
    call MPI_Type_free(ftype, iError)
    call check_mpi_error(iError,'free ftype in dump_treelmesh')
    call MPI_File_close(fh,    iError)
    call check_mpi_error(iError,'close file in dump_treelmesh')

    deallocate( buffer )

  end subroutine dump_treelmesh
  ! ************************************************************************ !

  ! ************************************************************************ !
  !> summary: Generate a predefined cube
  !! This serves as an simple grid generation for performance or scaling
  !! analysis without being obliged to use Seeder.
  !! You have to specify the generic grid parameters in the lua file insted of
  !! the mesh folder
  !!```lua
  !! mesh = { predefined='cube',
  !!          origin = {0.,0.,0.},
  !!          length = 10.,
  !!          refinementLevel = 6 }
  !!```
  !! You have to specify the shape 'cube', a bounding box origin, its length
  !! and also the refinement level, which results in different amount elements
  !! in the grid.
  !! The result of this routine is mainly the treeID list with the additional
  !! lists for saving the properties. They are all set to zero here, however.
  !! As we only have a simple cube which includes all the elements on this
  !! level, the treeID list just contains contiguously increasing integers
  !!
  subroutine generate_treelm_cube( me, origin, length, level,  myPart, nParts, &
    &                              comm )
    ! -------------------------------------------------------------------- !
    !> Mesh to generate
    type(treelmesh_type), intent(out) :: me
    !> Corner of the cube
    real(kind=rk), intent(in) :: origin(3)
    !> Length of cube
    real(kind=rk), intent(in) :: length
    !> Resolution level
    integer, intent(in) :: level
    !> Partition of the caller (starts with 0)
    integer, intent(in) :: myPart
    !> Number of partitions
    integer, intent(in) :: nParts
    !> communicator to be used
    integer, intent(in) :: comm
    ! -------------------------------------------------------------------- !
    integer(kind=long_k) :: firstID, lastID
    integer(kind=long_k) :: share
    integer :: remainder
    integer :: iPart, iElem
    ! -------------------------------------------------------------------- !

    me%global%nParts = nParts
    me%global%myPart = myPart
    me%global%comm = comm

    me%global%origin = origin
    me%global%BoundingCubeLength = length
    me%global%minLevel = level
    me%global%maxLevel = level
    me%global%label = 'Generic_Cube'
    me%global%predefined = 'cube'
    write(me%global%comment,'(a15,i7,a16,i2,a1)')                   &
      &     'Generated with ', nParts, ' Parts on Level ', level, '.'
    me%global%dirname = './'

    ! No properties in this mesh
    me%global%nProperties = 0
    if (associated(me%global%Property)) deallocate(me%global%Property)
    allocate(me%global%Property(me%global%nProperties))

    allocate(me%Part_First(nParts))
    allocate(me%Part_Last(nParts))

    ! Compute the treeIDs of the mesh:
    firstID = tem_firstIdAtLevel(level)
    lastID = tem_lastIdAtLevel(level)

    ! Total number of elements in this mesh
    me%global%nElems = lastID - firstID + 1_long_k

    share = me%global%nElems / int(nParts, kind=long_k)
    remainder = int(mod(me%global%nElems, int(nParts, kind=long_k)))

    ! The first partition starts always with the firstID
    me%Part_First(1) = firstID

    ! Up to remainder partitions have share + 1 elements
    do iPart=2,remainder+1
      me%Part_Last(iPart-1) = me%Part_First(iPart-1) + share
      me%Part_First(iPart) = me%Part_Last(iPart-1) + 1
    end do

    ! The remaining elements get exactly the share elements:
    do iPart=remainder+2,nParts
      me%Part_Last(iPart-1) = me%Part_First(iPart-1) + share - 1
      me%Part_First(iPart) = me%Part_Last(iPart-1) + 1
    end do

    ! The last partition ends always with the lastID
    me%Part_Last(nParts) = lastID

    ! Local data:
    me%nElems = int(me%Part_Last(myPart+1) - me%Part_First(myPart+1)) + 1
    me%elemOffset = me%Part_first(myPart+1) - firstID

    allocate(me%treeID(me%nElems))
    allocate(me%ElemPropertyBits(me%nElems))
    if (associated(me%property)) deallocate(me%property)
    allocate(me%Property(me%global%nProperties))

    ! No Properties:
    me%ElemPropertyBits = 0_long_k

    ! Filling the treeIDs:
    do iElem=1,me%nElems
      me%treeID(iElem) = me%Part_First(myPart+1) + int(iElem - 1, kind=long_k)
    end do

  end subroutine generate_treelm_cube
  ! ************************************************************************ !


  ! ************************************************************************ !
  function tem_CoordOf_2d_Id(id2d, level) result(coord)
    ! -------------------------------------------------------------------- !
    !> input element ID
    integer(kind=long_k), intent(in) :: Id2d
    integer, intent(in) :: level
    !> coordinate of element return value
    integer :: coord(4)
    ! -------------------------------------------------------------------- !
    integer(kind=long_k) :: fak(2)
    integer :: bitlevel
    integer :: i
    integer(kind=long_k) :: telem
    ! -------------------------------------------------------------------- !

    coord(:) = 0
    coord(4) = level
    fak(1) = 1
    fak(2) = 2
    bitlevel = 1

    tElem = id2d

    ! get x coordinate from
    !
    ! x = sum(iLevel=0, inf) { 2**iLevel * mod(tElem / (4**iLevel) ),2)   }
    ! y = sum(iLevel=0, inf) { 2**iLevel * mod(tElem /(2 * (4**iLevel) ),2)   }
    !
    do while ((tElem / fak(1)) > 0)
      do i=1,2
        coord(i) = coord(i) + bitlevel * int(mod(tElem / fak(i), 2_long_k))
      end do
      bitlevel = bitlevel * 2
      fak = fak * 4
    end do
  end function tem_CoordOf_2d_Id
  ! ************************************************************************ !


  ! ************************************************************************ !
  !> summary: Generate a predefined slice
  !! This serves as an simple grid generation for performance or scaling
  !! analysis without being obliged to use Seeder.
  !! You have to specify the generic grid parameters in the lua file instead of
  !! the mesh folder
  !!```lua
  !! mesh = { predefined='slice',
  !!          origin = {0.,0.,0.},
  !!          length = 10.,
  !!          refinementLevel = 6 }
  !!```
  !! You have to specify the shape 'slice', a bounding box origin, its length
  !! and also the refinement level, which results in different amount elements
  !! in the grid.
  !! The result of this routine is mainly the treeID list with the additional
  !! lists for saving the properties.
  !! The generated slice will be all elements in the first layer (XY plane) in
  !! Z-direction, with periodicity in Z direction.
  !!
  subroutine generate_treelm_slice( me, origin, length, level,  myPart, &
    &                               nParts, comm )
    ! -------------------------------------------------------------------- !
    !> Mesh to generate
    type(treelmesh_type), intent(out) :: me
    !> Corner of the cube
    real(kind=rk), intent(in) :: origin(3)
    !> Length of cube
    real(kind=rk), intent(in) :: length
    !> Resolution level
    integer, intent(in) :: level
    !> Partition of the caller (starts with 0)
    integer, intent(in) :: myPart
    !> Number of partitions
    integer, intent(in) :: nParts
    !> communicator to be used
    integer, intent(in) :: comm
    ! -------------------------------------------------------------------- !
    integer(kind=long_k) :: firstID, lastID
    integer(kind=long_k) :: share
    integer(kind=long_k) :: previous
    integer :: remainder
    integer :: iPart, iElem
    integer :: coord(4)
    integer :: lastcoord
    ! -------------------------------------------------------------------- !

    me%global%nParts = nParts
    me%global%myPart = myPart
    me%global%comm = comm

    me%global%origin = origin
    me%global%BoundingCubeLength = length
    me%global%minLevel = level
    me%global%maxLevel = level
    me%global%label = 'Generic_Slice'
    me%global%predefined = 'slice'
    write(me%global%comment,'(a15,i7,a16,i2,a1)')                   &
      &     'Generated with ', nParts, ' Parts on Level ', level, '.'
    me%global%dirname = './'

    ! Boundary property to define periodic boundary
    me%global%nProperties = 1
    if (associated(me%global%Property)) deallocate(me%global%property)
    if (associated(me%Property)) deallocate(me%property)
    allocate(me%global%Property(me%global%nProperties))
    allocate(me%Property(me%global%nProperties))

    allocate(me%Part_First(nParts))
    allocate(me%Part_Last(nParts))

    ! Compute the treeIDs of the mesh:
    firstID = tem_firstIdAtLevel(level)
    lastcoord = 2**level - 1
    lastID = tem_IdOfCoord( coord  = [lastcoord, lastcoord, 0, level], &
      &                     offset = firstID                           )

    ! Total number of elements in this mesh
    me%global%nElems = 4_long_k**level

    share = me%global%nElems / int(nParts, kind=long_k)
    remainder = int(mod(me%global%nElems, int(nParts, kind=long_k)))

    ! The first partition starts always with the firstID
    me%Part_First(1) = firstID

    ! Up to remainder partitions have share + 1 elements
    previous = 0_long_k
    do iPart=2,remainder+1
      previous = previous + int(share)
      coord = tem_coordof_2d_id(previous, level)
      me%Part_Last(iPart-1) = tem_idofcoord(coord, offset = firstID)
      previous = previous + 1
      coord = tem_coordof_2d_id(previous, level)
      me%Part_First(iPart) = tem_idofcoord(coord, offset = firstID)
    end do

    ! The remaining elements get exactly the share elements:
    do iPart=remainder+2,nParts
      previous = previous + int(share) - 1
      coord = tem_coordof_2d_id(previous, level)
      me%Part_Last(iPart-1) = tem_idofcoord(coord, offset = firstID)
      previous = previous + 1
      coord = tem_coordof_2d_id(previous, level)
      me%Part_First(iPart) = tem_idofcoord(coord, offset = firstID)
    end do

    ! The last partition ends always with the lastID
    me%Part_Last(nParts) = lastID

    ! Local data:
    if (myPart < remainder) then
      me%nElems = int(share+1)
      me%elemOffset = int(myPart, kind=long_k) * (share+1_long_k)
    else
      me%nElems = int(share)
      me%elemOffset = (int(myPart, kind=long_k) * share) &
        &           + int(remainder, kind=long_k)
    end if
    ! All elements have (periodic) boundaries
    me%Property(1)%nElems = me%nElems
    me%Property(1)%offset = me%elemOffset
    allocate(me%Property(1)%ElemID(me%Property(1)%nElems))
    ! Please note, that the tem_bc_prop_module will set boundary conditions
    ! based on this label accordingly!
    me%global%Property(1)%label = 'internal 2D BC'
    me%global%Property(1)%bitpos = prp_hasBnd
    me%global%Property(1)%nElems = me%Property(1)%nElems

    allocate(me%treeID(me%nElems))
    allocate(me%ElemPropertyBits(me%nElems))

    ! No Properties:
    me%ElemPropertyBits = ibset(0_long_k, prp_hasBnd)

    ! Filling the treeIDs:
    do iElem=1,me%nElems
      me%Property(1)%ElemID(iElem) = iElem
      coord = tem_coordof_2d_id(me%elemOffset + iElem - 1, level)
      me%treeID(iElem) = tem_idofcoord(coord, offset=firstID)
    end do

  end subroutine generate_treelm_slice
  ! ************************************************************************ !


  ! ************************************************************************ !
  !> summary: Generate a predefined line
  !> This serves as an simple grid generation for performance or scaling
  !! analysis without being obliged to use Seeder.
  !! You have to specify the generic grid parameters in the lua file instead of
  !! the mesh folder
  !! ~~~~~~~~~~~~~~~~~~~~~{.lua}
  !! mesh = { predefined='line',
  !!          origin = {0.,0.,0.},
  !!          length = 10.,
  !!          refinementLevel = 6 }
  !! ~~~~~~~~~~~~~~~~~~~~~
  !! \n
  !! You have to specify the shape 'line', a bounding box origin, its length
  !! and also the refinement level, which results in different amount elements
  !! in the grid.\n
  !! The result of this routine is mainly the treeID list with the additional
  !! lists for saving the properties.
  !! The generated line will be all elements along the X-Axis
  !! with periodicity in Y and Z direction.
  !!
  subroutine generate_treelm_line( me, origin, length, level,  myPart, &
    &                              nParts, comm )
    ! -------------------------------------------------------------------- !
    !> Mesh to generate
    type(treelmesh_type), intent(out) :: me
    !> Corner of the cube
    real(kind=rk), intent(in) :: origin(3)
    !> Length of cube
    real(kind=rk), intent(in) :: length
    !> Resolution level
    integer, intent(in) :: level
    !> Partition of the caller (starts with 0)
    integer, intent(in) :: myPart
    !> Number of partitions
    integer, intent(in) :: nParts
    !> communicator to be used
    integer, intent(in) :: comm
    ! -------------------------------------------------------------------- !
    integer(kind=long_k) :: firstID, lastID
    integer(kind=long_k) :: share
    integer :: remainder
    integer :: iPart, iElem
    integer :: coord(4)
    integer :: lastcoord
    ! -------------------------------------------------------------------- !

    me%global%nParts = nParts
    me%global%myPart = myPart
    me%global%comm = comm

    me%global%origin = origin
    me%global%BoundingCubeLength = length
    me%global%minLevel = level
    me%global%maxLevel = level
    me%global%label = 'Generic_Line'
    me%global%predefined = 'line'
    write(me%global%comment,'(a15,i7,a16,i2,a1)')                   &
      &     'Generated with ', nParts, ' Parts on Level ', level, '.'
    me%global%dirname = './'

    ! Boundary property to define periodic boundary
    me%global%nProperties = 1
    if (associated(me%global%property)) deallocate(me%global%property)
    if (associated(me%property)) deallocate(me%property)
    allocate(me%global%Property(me%global%nProperties))
    allocate(me%Property(me%global%nProperties))

    allocate(me%Part_First(nParts))
    allocate(me%Part_Last(nParts))

    ! Compute the treeIDs of the mesh:
    firstID = tem_firstIdAtLevel(level)
    lastcoord = 2**level - 1
    lastID = tem_IdOfCoord( coord = [lastcoord, 0, 0, level], offset = firstID )

    ! Total number of elements in this mesh
    me%global%nElems = 2_long_k**level

    share = me%global%nElems / int(nParts, kind=long_k)
    remainder = int(mod(me%global%nElems, int(nParts, kind=long_k)))

    ! The first partition starts always with the firstID
    me%Part_First(1) = firstID

    ! Up to remainder partitions have share + 1 elements
    coord = 0
    coord(4) = level
    do iPart=2,remainder+1
      coord(1) =  coord(1) + int(share)
      me%Part_Last(iPart-1) = tem_idofcoord(coord, offset = firstID)
      coord(1) = coord(1) + 1
      me%Part_First(iPart) = tem_idofcoord(coord, offset = firstID)
    end do

    ! The remaining elements get exactly the share elements:
    do iPart=remainder+2,nParts
      coord(1) = coord(1) + int(share) - 1
      me%Part_Last(iPart-1) = tem_idofcoord(coord, offset = firstID)
      coord(1) = coord(1) + 1
      me%Part_First(iPart) = tem_idofcoord(coord, offset = firstID)
    end do

    ! The last partition ends always with the lastID
    me%Part_Last(nParts) = lastID

    ! Local data:
    if (myPart < remainder) then
      me%nElems = int(share+1)
      me%elemOffset = int(myPart, kind=long_k) * (share+1_long_k)
    else
      me%nElems = int(share)
      me%elemOffset = (int(myPart, kind=long_k) * share) &
        &           + int(remainder, kind=long_k)
    end if
    ! All elements have (periodic) boundaries
    me%Property(1)%nElems = me%nElems
    me%Property(1)%offset = me%elemOffset
    allocate(me%Property(1)%ElemID(me%Property(1)%nElems))
    ! Please note, that the tem_bc_prop_module will set boundary conditions
    ! based on this label accordingly!
    me%global%Property(1)%label = 'internal 1D BC'
    me%global%Property(1)%bitpos = prp_hasBnd
    me%global%Property(1)%nElems = me%Property(1)%nElems

    allocate(me%treeID(me%nElems))
    allocate(me%ElemPropertyBits(me%nElems))

    ! Only has boundary property:
    me%ElemPropertyBits = ibset(0_long_k, prp_hasBnd)

    ! Filling the treeIDs:
    do iElem=1,me%nElems
      me%Property(1)%ElemID(iElem) = iElem
      ! We can only have 2**20 elements per dimension, so in this case where
      ! we create a line, we won't exceed the integer limit. Thus we safely can
      ! cast the long integer elemOffset to a normal integer.
      coord(1) = int(me%elemOffset) + iElem - 1
      me%treeID(iElem) = tem_idofcoord(coord, offset=firstID)
    end do

  end subroutine generate_treelm_line
  ! ************************************************************************ !


  ! ************************************************************************ !
  !> Generate a predefined line with a given number of elements
  !!
  !! This serves as an simple grid generation for performance or scaling
  !! analysis without being obliged to use Seeder.
  !! You have to specify the generic grid parameters in the lua file instead of
  !! the mesh folder
  !! ~~~~~~~~~~~~~~~~~~~~~{.lua}
  !! mesh = { predefined='line', -- or: 'line_bounded'
  !!          origin = {0.,0.,0.},
  !!          length = 10.,
  !!          elementcount = 6 }
  !! ~~~~~~~~~~~~~~~~~~~~~
  !!
  !! You have to specify the shape 'line', a bounding box origin, its length
  !! and the number of elements, which results in different amount elements
  !! in the grid.\n
  !! The result of this routine is mainly the treeID list with the additional
  !! lists for saving the properties.
  !! The generated line will be a line of elementcount elements along the
  !! X-Axis with periodicity in all directions.
  !!
  !! It is also possible to generate a mesh with boundary conditions in the X
  !! direction (west and east), by using the predefined 'line_bounded'.
  !! In this case these two boundary conditions need to be provided in the
  !! solver configuration.
  subroutine generate_treelm_elements( me, origin, length, elementcount, &
    &                                  myPart, nParts, comm, predefined, &
    &                                  bclabel )
    ! -------------------------------------------------------------------- !
    !> Mesh to generate
    type(treelmesh_type), intent(out) :: me
    !> Corner of the cube
    real(kind=rk), intent(in) :: origin(3)
    !> Length of cube
    real(kind=rk), intent(in) :: length
    !> Number of elements in the line
    integer, intent(in) :: elementcount
    !> Partition of the caller (starts with 0)
    integer, intent(in) :: myPart
    !> Number of partitions
    integer, intent(in) :: nParts
    !> communicator to be used
    integer, intent(in) :: comm
    !> Label describing the internal mesh.
    character(len=*), intent(in) :: predefined
    !> Label describing the boundary conditions to set for this mesh.
    character(len=*), intent(in) :: bclabel
    ! -------------------------------------------------------------------- !
    integer :: level
    integer(kind=long_k) :: firstID, lastID
    integer(kind=long_k) :: share
    integer :: remainder
    integer :: iPart, iElem
    integer :: coord(4)
    integer :: lastcoord
    integer :: xbound_pad
    ! -------------------------------------------------------------------- !

    me%global%nParts = nParts
    me%global%myPart = myPart
    me%global%comm = comm

    ! Face definitions make the first and last face coincide.
    ! To allow boundary definitions on both sides, we need to make sure, there
    ! is at least one additional element.
    if (predefined == 'line_bounded') then
      xbound_pad = 1
    else
      xbound_pad = 0
    end if

    ! Find an appropriate level of at least 1.
    level = max( ceiling(log(real(elementcount+xbound_pad,kind=rk)) &
      &                  / log(2.0_rk)),            1               )

    me%global%origin = origin
    me%global%BoundingCubeLength = length*(real(2**level, kind=rk) &
      &                                    /real(elementcount, kind=rk))
    me%global%minLevel = level
    me%global%maxLevel = level
    me%global%label = 'Generic_Line'
    me%global%predefined = predefined
    write(me%global%comment,'(a15,i7,a16,i2,a1)')                             &
      &               'Generated with ', nParts, ' Parts and ', elementcount, &
      &               ' elements.'
    me%global%dirname = './'

    ! Boundary property to define periodic boundary
    me%global%nProperties = 1
    if (associated(me%global%property)) deallocate(me%global%property)
    if (associated(me%property)) deallocate(me%property)
    allocate(me%global%Property(me%global%nProperties))
    allocate(me%Property(me%global%nProperties))

    allocate(me%Part_First(nParts))
    allocate(me%Part_Last(nParts))

    ! Compute the treeIDs of the mesh:
    firstID = tem_firstIdAtLevel(level)
    lastcoord = elementcount - 1
    lastID = tem_IdOfCoord(coord = [lastcoord, 0, 0, level], offset = firstID )

    ! Total number of elements in this mesh
    me%global%nElems = elementcount

    share = me%global%nElems / int(nParts, kind=long_k)
    remainder = int(mod(me%global%nElems, int(nParts, kind=long_k)))

    ! The first partition starts always with the firstID
    me%Part_First(1) = firstID

    ! Up to remainder partitions have share + 1 elements
    coord = 0
    coord(4) = level
    do iPart=2,remainder+1
      coord(1) =  coord(1) + int(share)
      me%Part_Last(iPart-1) = tem_idofcoord(coord, offset = firstID)
      coord(1) = coord(1) + 1
      me%Part_First(iPart) = tem_idofcoord(coord, offset = firstID)
    end do

    ! The remaining elements get exactly the share elements:
    do iPart=remainder+2,nParts
      coord(1) = coord(1) + int(share) - 1
      me%Part_Last(iPart-1) = tem_idofcoord(coord, offset = firstID)
      coord(1) = coord(1) + 1
      me%Part_First(iPart) = tem_idofcoord(coord, offset = firstID)
    end do

    ! The last partition ends always with the lastID
    me%Part_Last(nParts) = lastID

    ! Local data:
    if (myPart < remainder) then
      me%nElems = int(share+1)
      me%elemOffset = int(myPart, kind=long_k) * (share+1_long_k)
    else
      me%nElems = int(share)
      me%elemOffset = (int(myPart, kind=long_k) * share) &
        &           + int(remainder, kind=long_k)
    end if
    ! All elements have (periodic) boundaries
    me%Property(1)%nElems = me%nElems
    me%Property(1)%offset = me%elemOffset
    allocate(me%Property(1)%ElemID(me%Property(1)%nElems))
    ! Please note, that the tem_bc_prop_module will set boundary conditions
    ! based on this label accordingly!
    me%global%Property(1)%label = trim(bclabel)
    me%global%Property(1)%bitpos = prp_hasBnd
    me%global%Property(1)%nElems = me%Property(1)%nElems

    allocate(me%treeID(me%nElems))
    allocate(me%ElemPropertyBits(me%nElems))

    ! Only has Boundary Property:
    me%ElemPropertyBits = ibset(0_long_k, prp_hasBnd)

    ! Filling the treeIDs:
    do iElem = 1, me%nElems
      me%Property(1)%ElemID(iElem) = iElem
      ! We can only have 2**20 elements per dimension, so in this case where
      ! we create a line, we won't exceed the integer limit. Thus we safely can
      ! cast the long integer elemOffset to a normal integer.
      coord(1) = int(me%elemOffset) + iElem - 1
      me%treeID(iElem) = tem_idofcoord(coord, offset=firstID)
    end do

  end subroutine generate_treelm_elements
  ! ************************************************************************ !


  ! ************************************************************************ !
  !> summary: Generate a predefined single element
  !> This serves as an simple grid generation for performance or scaling
  !! analysis without being obliged to use Seeder.
  !! You have to specify the generic grid parameters in the lua file instead of
  !! the mesh folder
  !!```lua
  !! mesh = { predefined='single',
  !!          origin = {0.,0.,0.},
  !!          length = 10.}
  !!```
  !! You have to specify the shape 'single', a bounding box origin and its
  !! length. The generated element will have treeID 1 and is periodic in
  !! all directions.
  !! The given length will be the length of the element, and the bounding
  !! cube will be adapted accordingly, to map this to treeID 1.
  !!
  subroutine generate_treelm_single( me, origin, length, myPart, &
    &                                nParts, comm )
    ! -------------------------------------------------------------------- !
    !> Mesh to generate
    type(treelmesh_type), intent(out) :: me
    !> Corner of the cube
    real(kind=rk), intent(in) :: origin(3)
    !> Length of cube
    real(kind=rk), intent(in) :: length
    !> Partition of the caller (starts with 0)
    integer, intent(in) :: myPart
    !> Number of partitions
    integer, intent(in) :: nParts
    !> communicator to be used
    integer, intent(in) :: comm
    ! -------------------------------------------------------------------- !
    integer(kind=long_k) :: firstID, lastID
    integer :: iPart
    ! -------------------------------------------------------------------- !

    me%global%nParts = nParts
    me%global%myPart = myPart
    me%global%comm = comm

    me%global%origin = origin
    me%global%BoundingCubeLength = length
    me%global%minLevel = 1
    me%global%maxLevel = 1
    me%global%label = 'Generic_Single'
    me%global%predefined = 'single'
    write(me%global%comment,'(a15,i7,a16,i2,a1)')         &
      &     'Generated with ', nParts, ' Parts on Level 1.'
    me%global%dirname = './'

    ! Boundary property to define periodic boundary
    me%global%nProperties = 1
    if (associated(me%global%property)) deallocate(me%global%property)
    if (associated(me%property)) deallocate(me%property)
    allocate(me%global%Property(me%global%nProperties))
    allocate(me%Property(me%global%nProperties))

    allocate(me%Part_First(nParts))
    allocate(me%Part_Last(nParts))

    ! Compute the treeIDs of the mesh:
    firstID = 1_long_k
    lastID = firstID

    ! Total number of elements in this mesh
    me%global%nElems = 1_long_k

    ! The first partition starts always with the firstID
    me%Part_First(1) = firstID
    me%Part_Last(1) = lastID

    ! There is only one element, all partitions but the first one are empty.
    do iPart=2,nParts
      me%Part_First(iPart) = lastID
      me%Part_Last(iPart) = lastID - 1
    end do

    ! Local data:
    if (myPart == 0) then
      me%nElems = 1
      me%elemOffset = 0_long_k
    else
      me%nElems = 0
      me%elemOffset = 1_long_k
    end if

    ! All elements have (periodic) boundaries
    me%Property(1)%nElems = me%nElems
    me%Property(1)%offset = me%elemOffset
    allocate(me%Property(1)%ElemID(me%Property(1)%nElems))
    ! Please note, that the tem_bc_prop_module will set boundary conditions
    ! based on this label accordingly!
    me%global%Property(1)%label = 'internal 0D BC'
    me%global%Property(1)%bitpos = prp_hasBnd
    me%global%Property(1)%nElems = me%Property(1)%nElems

    allocate(me%treeID(me%nElems))
    allocate(me%ElemPropertyBits(me%nElems))

    ! No Properties:
    me%ElemPropertyBits = ibset(0_long_k, prp_hasBnd)

    ! There is only one element:
    me%treeID = firstID

  end subroutine generate_treelm_single
  ! ************************************************************************ !


  ! ************************************************************************ !
  !> This subroutine creates a simple mesh based on a treeID list
  !! requires the setting of me%global before hand
  !! Works on the communicator specified in me%global%comm
  !!
  subroutine treelmesh_fromList(me, tIDlist, comm, dirname)
    ! -------------------------------------------------------------------- !
    !> Structure to load the mesh to
    type(treelmesh_type), intent(inout) :: me
    !> treeID list to generate the global list in the mesh from
    integer(kind=long_k), intent(in) :: tIDlist(:)
    !> MPI Communicator to use, defaults to the one in me%global%comm if not
    !! specified
    integer, intent(in), optional :: comm
    !> Directory to store the mesh in. Is taken to be me%global%dirname if not
    !! specified
    character(len=*), intent(in), optional :: dirname
    ! -------------------------------------------------------------------- !
    integer :: commLoc
    integer(kind=long_k) :: offset, nElems
    integer :: iError
    integer :: nElemsList
    ! mpi variables
    integer :: commsize, rank
    ! -------------------------------------------------------------------- !

    if (present(dirname)) then
      me%global%dirname = dirname
    end if

    if (present(comm)) then
      commLoc = comm
    else
      commLoc = me%global%comm
    end if

    ! determine number of treeIDs in list
    nElemsList  = size(tIDlist)
    ! Get size of communicator and my rank in it
    call mpi_comm_size(commLoc, commsize, iError)
    call mpi_comm_rank(commLoc, rank, iError)
    me%global%nParts = commsize
    me%global%myPart = rank

    if (associated(me%property)) deallocate(me%property)
    allocate(me%Property(me%global%nProperties))
    allocate(me%treeID(nElemsList))
    allocate(me%ElemPropertyBits(nElemsList))
    allocate(me%Part_First(commsize))
    allocate(me%Part_Last(commsize))
    ! Assign treeID list to the new mesh. Sort first

    offset = 0
    nElems  = int(nElemsList, kind=long_k)
    ! Determine offset for each process (get nElems of ranks before)
    call MPI_Exscan( nElems, offset, 1, MPI_INTEGER8, MPI_SUM, commLoc, iError )
    me%elemOffset = offset
    me%nElems = nElemsList
    ! Last process offset + its nElems is total number of elements
    nElems = offset + nElems
    ! Distribute among new communicator
    call MPI_Bcast( nElems, 1, MPI_INTEGER8, commsize-1, commLoc, iError )
    ! Store in global number of elements
    me%global%nElems = nElems
    ! Copy treeID list to treeID in the mesh
    me%treeID = tIDlist

    ! Reset properties. Not needed
    me%ElemPropertyBits = 0
    ! Exchange Part_First and Part_Last
    call mpi_allgather( me%treeID(1),                   &
      &                 1, MPI_INTEGER8,                &
      &                 me%Part_First, 1, MPI_INTEGER8, &
      &                 commLoc, iError                 )

    call mpi_allgather( me%treeID(nElemsList),         &
      &                 1, MPI_INTEGER8,               &
      &                 me%Part_Last, 1, MPI_INTEGER8, &
      &                 commLoc, iError                )

  end subroutine treelmesh_fromList
  ! ************************************************************************ !


  ! ************************************************************************ !
  !> write mesh information
  subroutine dump_meshHeader( me )
    ! -------------------------------------------------------------------- !
    !> fluid tree from mesh
    type( treelmesh_type ), intent(in) :: me
    ! -------------------------------------------------------------------- !
    integer :: iProp
    ! -------------------------------------------------------------------- !
    write(logunit(1),*)  'Got a mesh with following properties:'
    write(logUnit(1), "('                 Mesh name: ',A)") &
      &               trim(me%global%label)
    if (trim(me%global%predefined) /= '') then
      write(logunit(1), "(' This is a predefined mesh: ', A)") &
        &               trim(me%global%predefined)
    end if
    write(logunit(1), "('  Total number of elements: ', I0)") me%global%nElems
    write(logunit(1), "('  Local number of elements: ', I0)") me%nElems
    write(logUnit(1), "('      Number of properties: ', I0)") &
      &               me%global%nProperties
    write(logunit(1), "('      Number of partitions: ', I0)") me%global%nParts
    write(logunit(1), "('             Minimum level: ', I0)") me%global%minlevel
    write(logunit(1), "('             Maximum level: ', I0)") me%global%maxlevel
    write(logunit(1), "('       Bounding Box Origin: ', 3F10.4 )") &
      &               me%global%origin
    write(logunit(1), "('       Bounding Box Length: ', F10.4 )") &
      &               real(me%global%BoundingCubeLength)
    write(logUnit(1), "('     Coarsest element size: ', F10.4 )") &
      &               real( me%global%BoundingCubeLength /      &
      &               real( 2**me%global%minlevel, kind=rk ) )
    write(logUnit(1), "('       Finest element size: ', F10.4 )") &
      &               real( me%global%BoundingCubeLength /      &
      &               real( 2**me%global%maxlevel, kind=rk ) )
    do iProp=1,me%global%nProperties
      if ( me%global%Property(iProp)%BitPos == prp_hasBnd ) then
        write(logUnit(1), "('  Number of boundary elements: ', I0 )") &
          &               me%Property(iProp)%nElems
      end if
      if ( me%global%Property(iProp)%BitPos == prp_hasQVal ) then
        write(logUnit(1), "('      Number of qVal elements: ', I0 )") &
          &               me%Property(iProp)%nElems
      end if
    end do
  end subroutine dump_meshHeader
  ! ************************************************************************ !


  ! ************************************************************************ !
  subroutine tem_load_weights( me, weights, success )
    ! -------------------------------------------------------------------- !
    type(treelmesh_type), intent(in) :: me
    real(kind=rk), intent(out) :: weights(me%nElems)
    logical, intent(out) :: success
    ! -------------------------------------------------------------------- !
    integer(kind=MPI_OFFSET_KIND) :: displacement
    integer(kind=MPI_OFFSET_KIND) :: filebytes
    integer :: fh, ftype, iError, typesize
    integer :: iostatus( MPI_STATUS_SIZE )
    integer :: level
    integer :: iElem
    logical :: ex
    character(len=PathLen) :: filename
    character(len=4) :: EndianSuffix
    ! -------------------------------------------------------------------- !

    EndianSuffix = tem_create_endianSuffix()
    if (trim(me%weights_file) /= '') then
      filename = trim(me%weights_file)//EndianSuffix
      ! check if a corresponding weight file exists
      if (me%global%myPart == 0) then
        inquire(file=trim(filename), exist=ex)
      end if

      call MPI_Bcast(ex, 1, MPI_LOGICAL, 0, me%global%comm, iError)
    else
      ex = .false.
    end if

    if (ex) then
      ! Found a weights file, which is used to read a weight for each
      ! element.
      write(logUnit(3),*) 'Loading Weights from file: '//trim(filename)
      ! Open the binary file for MPI I/O (Write)
      call MPI_File_open( me%global%comm, trim(filename),            &
        &                 MPI_MODE_RDONLY, MPI_INFO_NULL, fh, iError )
      call check_mpi_error(iError,'file_open in load_weights')

      ! Create a MPI Subarray  as ftype for file view
      call MPI_Type_contiguous( me%nElems, rk_mpi , ftype, iError )
      call check_mpi_error(iError,'type ftype in load_weights')
      call MPI_Type_commit( ftype, iError )
      call check_mpi_error(iError,'commit ftype in load_weights')

      !get size of etype
      call MPI_Type_size(rk_mpi, typesize, iError )
      call check_mpi_error(iError,'typesize in load_weights')

      call MPI_File_get_size(fh, filebytes, iError)

      ! Check whether the weight file has as many values as there are
      ! elements in the mesh.
      if (filebytes /= typesize*me%global%nElems) then

        write(logunit(3),*) 'Mesh file has wrong number of elements!'
        write(logunit(3),*) 'NO BALANCING!'
        success = .false.

      else

        ! File has correct number of elements matching the mesh.
        ! calculate displacement
        displacement = me%elemoffset * typesize * 1_MPI_OFFSET_KIND

        ! Set the view for each process on the file above
        call MPI_File_set_view( fh, displacement, rk_mpi, ftype, &
          &                     "native", MPI_INFO_NULL, iError  )
        call check_mpi_error(iError,'set_view in load_weights')

        ! Read data from the file
        call MPI_File_read_all( fh, weights, me%nElems, rk_mpi, iostatus, &
          &                     iError                                    )
        call check_mpi_error(iError,'read_all in load_weights')

        success = .true.

      end if

      !Free the MPI_Datatypes which were created and close the file
      call MPI_Type_free(ftype, iError)
      call check_mpi_error(iError,'free ftype in load_weights')
      call MPI_File_close(fh,    iError)
      call check_mpi_error(iError,'close file in load_weights')

    else if (maxval(me%levelweight) > 0.0_rk) then
      write(logunit(3),*) 'Using levelwise weights.'
      do iElem=1,me%nElems
        level = tem_levelOf( me%treeID(iElem) )
        weights(iElem) = me%levelWeight(level)
      end do
      success = .true.
    else
      write(logunit(3),*) 'NO BALANCING!'
      success = .false.
    end if
    write(logUnit(3),*) 'Done loading weights.'

  end subroutine tem_load_weights
  ! ************************************************************************ !

  ! ************************************************************************ !
  subroutine tem_dump_weights( me, filename, weights )
    ! -------------------------------------------------------------------- !
    type(treelmesh_type), intent(in) :: me
    !> Weights file name
    character(len=*), intent(in) :: filename
    real(kind=rk), intent(in)    :: weights(me%nElems)
    ! -------------------------------------------------------------------- !
    integer                       :: nElems
    integer                       :: comm
    integer(kind=long_k)          :: offset, globElems
    integer(kind=long_k)          :: filesize
    integer(kind=MPI_OFFSET_KIND) :: displacement
    integer                       :: fh, ftype, iError
    integer                       :: iostatus( MPI_STATUS_SIZE )
    integer                       :: typesize
    ! -------------------------------------------------------------------- !

    nElems = me%nElems
    comm = me%global%comm
    offset = me%elemOffset
    globElems = me%global%nElems

    ! Found a weights file, which is used to read a weight for each
    ! element.
    write(logUnit(1),*) 'Dumping Weights to file: '//trim(filename)
    ! Open the binary file for MPI I/O (Write)
    call MPI_File_open( comm, trim(filename),                 &
      &                 ior(MPI_MODE_WRONLY,MPI_MODE_CREATE), &
      &                 MPI_INFO_NULL, fh, iError             )
    call check_mpi_error(iError,'file_open in dump_weights')

    ! Create a MPI Subarray  as ftype for file view
    call MPI_Type_contiguous( nElems, rk_mpi , ftype, iError )
    call check_mpi_error(iError,'type ftype in dump_weights')
    call MPI_Type_commit( ftype, iError )
    call check_mpi_error(iError,'commit ftype in dump_weights')

    !get size of etype
    call MPI_Type_size(rk_mpi, typesize, iError )
    call check_mpi_error(iError,'typesize in dump_weights')

    ! calculate displacement
    displacement= offset * typesize * 1_MPI_OFFSET_KIND

    ! set filesize
    filesize = globElems * typesize
    call MPI_File_set_size( fh, filesize, iError )

    ! Set the view for each process on the file above
    call MPI_File_set_view( fh, displacement, rk_mpi, ftype, &
      &                     "native", MPI_INFO_NULL, iError  )
    call check_mpi_error(iError,'set_view in dump_weights')

    ! Read data from the file
    call MPI_File_write_all( fh, weights, nElems, rk_mpi, iostatus, iError )
    call check_mpi_error(iError,'read_all in dump_weights')

    !Free the MPI_Datatypes which were created and close the file
    call MPI_Type_free(ftype, iError)
    call check_mpi_error(iError,'free ftype in dump_weights')
    call MPI_File_close(fh,    iError)
    call check_mpi_error(iError,'close file in dump_weights')
    write(logUnit(1),*) 'Done dumping weights.'

  end subroutine tem_dump_weights
  ! ************************************************************************ !


  ! ************************************************************************ !
  !> Exchange mesh elements with other process
  !! The following data within treelmesh_type is updated in this routine:
  !!    treeID, ElemPropertyBits, Part_First, Part_Last
  !! Data pathList is deallocated.
  subroutine exchange_elements( me, sparta )
    ! -------------------------------------------------------------------- !
    !> Tree Mesh type
    !! nElems and offset should be ready before this routine
    type( treelmesh_type ), intent(inout) :: me
    !> How many elements to exchange with other processes
    type( tem_sparta_type ), intent(in) :: sparta
    ! -------------------------------------------------------------------- !
    integer :: iError, iProp
    ! -------------------------------------------------------------------- !

    ! Exchange treeID and propertyBits ---------------------------------
    call tem_exchange_sparta( me = sparta, &
      &                       val = me%treeID, &
      &                       nComponents = 1, &
      &                       comm = me%global%comm )
    call tem_exchange_sparta( me = sparta, &
      &                       val = me%ElemPropertyBits, &
      &                       nComponents = 1, &
      &                       comm = me%global%comm )
    ! Exchange treeID and propertyBits ---------------------------------

    ! Update Part_first and Part_Last ------------------------
    call MPI_allgather( me%treeID(1), 1, long_k_mpi,  &
      &                 me%Part_First, 1, long_k_mpi, &
      &                 me%global%comm, iError        )
    call MPI_allgather( me%treeID(me%nElems), 1, long_k_mpi, &
      &                 me%Part_Last, 1, long_k_mpi,         &
      &                 me%global%comm, iError               )
    ! Update Part_first and Part_Last ------------------------

    do iProp=1,me%global%nProperties
      call gather_Property( Property = me%Property(iProp),                     &
        &                   Header   = me%global%Property(iProp),              &
        &                   BitField = me%ElemPropertyBits,                    &
        &                   comm     = me%global%comm )
    end do

    if (allocated(me%pathList)) deallocate( me%pathList  )

  end subroutine exchange_elements
  ! ************************************************************************ !

end module treelmesh_module
! **************************************************************************** !

!> \page motivation Motivation for the TreElm Library
!!
!! Mesh-based algorithms build one of the most important discretization schemes
!! for the numerical modeling of field problems.
!! They are also well suited for parallel computations, as the meshes can be
!! partitioned and the smaller problems solved concurrently.
!! Their basic idea is the discretization of the computational domain by
!! covering it fully with smaller, well defined elements.
!! For complex geometries, it is in general necessary to use an unstructured
!! mesh to discretize the computational domain.
!! In unstructured meshes each element has typically a different size and form.
!! Therefore, all information of element geometry and neighbor relations have to
!! be described explicitly.
!! This fully explicit description however, poses a severe hurdle for parallel
!! processing, as potentially any part of the mesh might be required in the
!! neighborhood of any process.
!! Due to the limited memory, providing the complete mesh information locally to
!! each partition is not an option, as the ability to process larger meshes is
!! often the motivation to switch from one to many computing units in the first
!! place.
!! Although, it is possible to move this limitation out of the solver itself,
!! Tu et al. pointed out the importance to avoid such bottlenecks throughout
!! the complete chain of tools in the simulation.
!! Thus, not only the main computational loop, but the entire simulation process
!! has to be considered, including mesh generation and post-processing.
!! The usage of hierarchical subdivisions in a fixed space resulting in and
!! octree representation offers an option to overcome the hurdle for parallel
!! processing of complex geometries with mesh-based methods.
!!
!! TreElM provides a basis for massively parallel mesh-based simulations, using
!! an octree discretization in combination with a space-filling curve.
!! The usage of octree meshes for flow simulations has long been proposed, for
!! example by Flaherty et al. in 1997.
!! By using octree meshes the main bottleneck of unstructured
!! meshes is avoided, while the flexibility of local refinement and resolution
!! of arbitrary complex geometries is maintained.
!! With the known topology of the octree, the needed information about other
!! partitions can be kept at a minimum and the connectivity can be computed
!! locally.
!! Direct usage of cubic octree elements has the advantage that they can be
!! computed efficiently in most schemes, or are even required as for example
!! in the lattice Boltzmann method.
!! \image html clumps_sm.png
!! The figure above shows an example for meshes, that are obtained by the
!! octree discretization.
!! It shows the filling of a channel with many small cylindric obstacles,
!! indicated by transparent objects.
!! The complete space covered by the octree is outlined by the blue grid lines
!! and shows the refinement towards the channel with obstacles.
!! Inside the channel, there is the actual computational domain, shown by the
!! green cubical elements.
!!
!! The basic concept in TreElM is to exploit the known hierarchical
!! topology, provided by the octree.
!! Another concept that is supported by the octree and beneficial for the
!! parallel operation is the strictly elementwise view on the mesh data.
!! As each element will be uniquely associated with a certain partition during
!! the computation, such a view enables the parallel treatment straight from
!! reading or creating the mesh onwards.
!! TreElM therefore maintains an elemental view on the mesh data in the data layout
!! and the operations on the data.
!!
!! \section toolchain The Simulation Tool Chain
!!
!! The tools in the framework range from the mesh generator *Seeder*
!! over a finite volume solver for compressible and a
!! lattice Boltzmann solver for incompressible flow to the post-processing tool
!! *Harvester*, that produces visualization files from simulation data
!! attached to the mesh.
!! This interaction is sketched in the following Figure.
!! \image html apes_schematic.png
!! Since neither input nor output (I/O) can be avoided to the largest part, even
!! with an integrated workflow, data structures are designed to be suitable for
!! efficient parallel reading and writing.
!!
!! \subsection see_also See Also
!! - \ref tem_distributed_octree "The distributed Octree"
!! - \ref tem_node_identification "Node identification in the tree"
!!
!! [treeid]: @ref treelmesh_module#treelmesh_type#treeid "treeID"
!!
!! \page tem_distributed_octree Distributed Octree
!! An [octree](http://en.wikipedia.org/wiki/Octree)
!! is obtained by recursive bisection in each dimension of a
!! three-dimensional cube.
!! As such it is the generalization of a binary tree into three dimensions, the
!! two dimensional case is usually referred to as quadtree.
!! We use quadtrees and two dimensional meshes for illustrations.
!! A three-dimensional picture of the actual octree meshes, obtained by this
!! method is provided in the Figure.
!! \image html clumps_sm.png
!! This illustration shows the subdivision of coarser elements into eight
!! smaller elements and the recursive repetition of this refinement where
!! necessary.
!! The outermost cube is also addressed as the universe, as it contains the
!! complete representable space of the octree and all elements have to live
!! inside of it.
!! The basic idea in TreElM for a scalable parallel deployment of the octree
!! discretization is a simple serialized tree representation and sketched in
!! the following figure.
!! \image html tree_structure_sm.png
!! The serialization turns the hierarchical structure of the tree, where each
!! node has several links to other nodes into a serial ordered list of one node
!! after the other.
!! This is achieved by a [space-filling curve](http://en.wikipedia.org/wiki/Space-filling_curve).
!! Space filling curves are designed to fill the three dimensional space
!! completely with a one-dimensional one.
!! They thus map a one-dimensional unit interval onto a higher dimensional unit
!! interval.
!! This feature can be exploited to define a serialization of the three
!! dimensional space.
!! They are also defined recursively and complete coverage of the continuous
!! space is achieved after infinite iterations.
!!
!! Finite iterations of the space filling curve definition can be interpreted as
!! discrete curves and can be used to describe the ordering for finite intervals
!! in the higher dimensional space.
!! This recursive definition of the space filling curve ordering nicely fits the
!! recursive definition of the spatial subdivisions by the octree.
!! We can thus interpret the space filling curve as a rule how the children of a
!! node in the octree should be numbered.
!! Consecutively numbering all nodes on each refinement level according to the
!! space filling curve results in a breadth first numbering of the complete
!! tree.
!! As these numbers identify the elements in the full tree, we refer to them as
!! [treeid]s.
!! Their properties are described in the following and the relation
!! between mesh and tree is illustrated with the help of a quadtree in the Figure
!! above.
!!
!! \section partitioning Partitioning of the Tree
!! With this numbering scheme for all nodes in the full tree it is possible to
!! deduce the geometrical position of each node purely by its assigned number.
!! Just the [treeid]s of the leaf
!! nodes, that are actually part of the
!! computational domain are stored with the described ordering to represent the
!! mesh.
!! Thus, we obtain a sparse octree, that contains only the relevant leaf nodes
!! for the computational nodes.
!! In the Figure of the 3D mesh, these are highlighted as green cubes.
!! None of the other tree nodes, including the parents of the elements in the
!! computational domain, are actually stored.
!! Due to the ordering, the serialized list of elements is efficiently
!! searchable.
!! The one dimensional list of elements can be easily partitioned by splitting
!! it into individual chunks.
!! This design is sketched in the Figure above for a two
!! dimensional mesh as shown on the left side of the figure.
!! There is an obstacle in the middle of the mesh, covering the four central
!! fine elements.
!! The mesh is refined towards this obstacle.
!! A line is used to indicate the connection between consecutive elements
!! according to the ordering with the space filling curve.
!! On the right side the according quadtree is shown and the partitioning into
!! 6 parts is indicated by regions with different colors.
!! Thus, we obtain a simple ordered linear representation of the sparse octree.
!! Each actually existing element in the mesh is numbered by position in
!! addition to its [treeid] and the resulting serialized tree representation is
!! given at the bottom of the Figure in the form of a simple [treeid] list.
!!
!! \subsection see_also See also
!! - \ref treelm_fileformat Treelm File Format
!! - \ref tem_balance_module "Load Balancing"
!!
!! [treeid]: @ref treelmesh_module#treelmesh_type#treeid "treeID"
!!
!! \page treelm_fileformat Treelm File Format
!! As already introduced in the previous sections, the
!! \ref tem_distributed_octree "octree mesh" is represented by the sparse
!! storage of the leaf nodes in the computational domain.
!! The storage of the mesh on disk is basically the sequence of [treeid]s
!! building the computational domain.
!! However, this is in general insufficient to fully describe the mesh and an
!! additional elemental information is introduced to attach further data to the
!! elements.
!! TreElM uses 64-bit signed integers to encode the [treeid]s for
!! greatest portability.
!! In addition, a second 64-bit signed integer is used to build a bit-mask for
!! properties like boundary conditions, that might be attached to each element.
!! This data is written in native binary format to disk, resulting in a file
!! with 16 bytes per element.
!! Due to this simple format, the data might be accessed on an elemental basis
!! with Fortran's direct IO, it can be easily converted between big and little
!! endian representations and allows fast reading and writing.
!!
!! This binary data is accompanied by a header file with descriptive information
!! about the mesh, like the total number of elements, the physical extent and
!! origin of the universal cube and descriptions for the attached properties.
!! The data in this header is provided in the form of a [Lua](http://www.lua.org/)
!! script, to allow for a flexible handling of additional data and future evolution
!! of the format.
!! With the help of the \ref aotus_module "Aotus library" data can be easily
!! retrieved from and written to such Lua scripts.
!!
!! [treeid]: @ref treelmesh_module#treelmesh_type#treeid "treeID"
!!
!!
!! \page implementation Implementation in the APES Framework
!! In distributed computations it is important to avoid any kind of bottleneck,
!! as the local computing resources in each computing unit are
!! usually quite limited in comparison to the total computing power.
!! For large scale computations that aim to use the full system for their
!! simulation, it therefore is essential that the complete processing chain is
!! distributable.
!! Otherwise any step from the mesh generation over the simulation to the
!! post-processing might prohibit a successful simulation.
!! The *TreElM* library builds the basis for a completely
!! integrated framework that allows the distributed processing of all necessary
!! simulation steps.
!! To enable a flexible and convenient user configuration, the \ref aotus_module
!! "Aotus library"
!! is not only used in *TreElM* header files, but also exposed to the
!! calling applications.
!! The layout of the overall framework with the central *TreElM* library is
!! shown by the schematic figure in the \ref motivation "Motivation".
!!
!! The solvers read the mesh data, using *TreElM* functions and process the
!! mesh into a solver specific data structure with attached simulation data.
!! Results are written in the form of restart files, that follow the elemental
!! design of the *TreElM* mesh by writing the elemental solution following the
!! same ordering to disk.
!! With the help of tracking objects it is possible to write subsets of the mesh
!! with attached simulation data in arbitrary intervals to disk.
!! The *TreElM* library not only provides the means to identify the halos to
!! be exchanged between partitions but also provides various communication
!! pattern that can be used for the actual exchange.
!! Due to this encapsulation of the communication, the communication layout can
!! be easily exchanged.
!! It is even possible to replace the complete parallel paradigm in the
!! deployment with only few changes to the code.
!!
!! Aside from these basic functionalities the library provides several auxiliary
!! routines, that provide for example logging and debugging facilities.
!! Finally the post processing tool *Harvester* is used to visualize the
!! results of the simulation.
!! Since *Harvester* is a stand alone tool, it can be deployed on specialized
!! machines, that are more suited for visualization.
!! Additionally, the separated design ensures the independence of the solvers
!! from any third party libraries, that are typically required for visualization
!! outputs.
!!
!! [treeid]: @ref treelmesh_module#treelmesh_type#treeid "treeID"
!!
!!
!! \page tem_requirements Requirements
!!
!! Below are the most important requirements for getting, compiling and running
!! solvers developed with the TreElm framework.
!!
!! Compilation
!! -----------
!! - [Mercurial](mercurial.selenic.com): Getting sources
!! - [Python](www.python.org): the build tool waf relies on python and most of
!!           the scripts inside the package are written with Python
!! - Compiler: the vast source files are written in Fortran 95/2003.
!!            We recommend [GNU Gfortran 4.6+](http://gcc.gnu.org/wiki/GFortran)
!!            and [Intel 12.1+](http://software.intel.com/en-us/articles/
!!                              intel-compilers/)
!! - MPI 2.0 enabled environment
!!           We recommend [OpenMPI 1.5+](http://www.open-mpi.org/)
!!
!! Post-Processing and Visualization
!! ---------------------------------
!! - [Gnuplot](http://www.gnuplot.info/)
!! - [Paraview](http://www.paraview.org/)
!! - [Tecplot](http://www.tecplot.com)