DISPATCH
refine_mod Module Reference

This module handles checking max change between neighboring points. Each instance of it needs an index to the variable, whether to take the log, and a value for the max allowed change. It might be applied to log density, log pressure, log opacity, velocity amplitude, magnetic field amplitude, etc. More...

Functions/Subroutines

subroutine make_child_patch (self, tlist, parent, pos, child_size, child)
 Create a new patch using an existing "parent" patch. This procedure is used frequently during refinement.
 
integer function check_compress (self, patch)
 Refine / derefine on vorticity (velocity difference per cell)
 
integer function check_vorticity (self, patch)
 Refine / derefine on vorticity (velocity difference per cell)
 
integer function shock_and_cd_detector (self, patch)
 Refine on shocks, with threshold set by tolerance parameter tol_shock.
 
integer function gradient_detector (self, patch)
 Refine on shocks, with threshold set by tolerance parameter max_shock.
 
integer function refine_region (self, patch)
 Given a box defined by its lower and upper corners, flag a patch for refinement if it lies inside this box, up to a maximum level. This particular criteria is primarily used for testing and debugging.
 
subroutine sanity_check (self, min, max, label, reverse)
 Check that the derefinement criterion is sane.
 

Variables

type(refine_t), public refine
 

Detailed Description

This module handles checking max change between neighboring points. Each instance of it needs an index to the variable, whether to take the log, and a value for the max allowed change. It might be applied to log density, log pressure, log opacity, velocity amplitude, magnetic field amplitude, etc.

task_list_tupdate check_current refine_needed selective_refine if ... refine make_child_patch tasklistlockset init_nbors ... check_nbors ... tasklistlockunset if ... derefine remove_and_reset tasklistlockset init_nbors ... check_nbors ... tasklistlockunset download_link update ...

Support criterion

A patch should be able to load guard zone values from nbors differing at most by one AMR-level, to avoid too abrupt changes of resolution at patch boundaries. The criterion to test must be formulated as a criterion on the patch that should potentially be created, and not as a criterion on the that needs the support (to avoid having to find a sub-region of the parent of the parent – and thus have to maintain parent relationships)

On the other hand, the criterion needs to be checked at the point in time when a new, higher level patch is to be created, to check if this requires that an nbor patch should also be refined. The typical situation to look for is when creating a patch at level L+1 requires of the patch whose sub- region is being investigated requires a new patch at level L – a so far non-existing nbor of the patch at level L under investigation. The logic is, in brief:

for candidate in sub_regions_of (patch): if needed(candidate): attach_preliminary_nbor_list_to (candidate) for nbor in candidate.nbors: if nbor.patch.level==patch_candidate+2: for nbor_candidate in sub_region_of (nbor.patch): if overlap_between (candidate,nbor_candidate): create(nbor_candidate) candidate.add_nbor (nbor_candidate) update_nbor_lists

Note that this procedure does not assume a specific (e.g. side-by-side) arrangement of patches, and hence allows for moving patches. Thus, it also does not rely on the existence of a parent-child relation between patches.

OpenMP synchronization is implemented with two (nested) OMP locks; one (check_lock) responsible for synchronizing calls to list_tcheck_nbors, and one (download_lock) responsible for synchronizing filling of guard zones (as well as restrict and prolong operations).

When this module decides to refine or derefine a task, it acquires both locks, and hence blocks check_nbors and download_link() calls while the operation is on-going. Since no time advance is involved in the refine / derefine, the blocking cannot cause a hang.

Since refine/derefine is not done every timestep, the impact on running tasks from the locking is not severe, and since two different locks are used, one avoids the significant impact that would be the result of locking out check_nbors and download_link from occuring at the same time.

The impact on download_link could be entirely removed by always using the sorted nbor list, and by collecting patches to be removed on a GC list, removing them a few AMR cycles later.

The absolute minimum amount of synchronization would be to just make sure that an nbor list 1) can never change exactly when it is accessed, 2) to immediately make a copy of it, 3) to make sure no to remove any task on the nbor list while it is being used, and 4) to make sure that rotating the memory slots will not invalidate using the nbor list, as is.

An nbor list relevant for a particular task update is accessed in the following order:

1) During a link_tcheck_nbors() call, made after an nbor of the task in question has been updated, link_tcheck_ready() is called, and runs through the tasks nbor list, comparing tasktime values, finding that all nbors of the task are advanced enough to provide boundary zone data, or other data that the task needs. The task is then added to the ready queue. The comparisons of times requires only that task_ttime is updated atomically. These updates happen in task_trotate, which also manipulates the task_tiit(:) array, which holds the indices to the memory slots.

2) After a thread picks that task off the head of the ready queue, the first action (in task_list_tupdate) is to check if the task should be refined or derefined. The test itself does not use the nbor lists, but if either the current thread, or a thread working on an nbor decides to refine or derefine, then the nbor list of the task may become modified.

a) If it is refined, the data upon which the decision to use it as a source of guard zone data still exists (in the parent task), and the refined data in the child task(s) is initially just a prolongation of the parent data, so does not yet provide any improvement. Therefore, there is no problem, and the insertion of the new nbor list should be delayed until it is entirely complete, at which point the insertion can be done using a very brief locking of the link, while changeing tasklinknbor to point to the new first nbor.

b) If it is derefined, one just has to leave the nbor list and the task there until after they have been used, in guard zone download, or in check_ready().

3) The next step is download of data, which relies on data from all source tasks, one of which could possible be in the process of being derefined. To minimize the acess time needed, the download procedure should copy the nbor list, a step that is anyway needed to sort the list into level order. Then, to guarantee existence of the derefined patch, it has to either be prevented from going away by locking, or else it must be put in a garbage collector, which only removes patches when they are no longer needed.

The latter option is the easier one, since it avoid messy synchronization. A sync would require that derefinement is prevented in the interval btw the task has been put in the ready queue, and until all nbors that might possibly need it has done their downloads; a tall order.

As part of putting the task in the ready queue, a copy of the nbor list should be made – this could be exactly the sorted copy that will be needed in the download step in any case.

Keeping track of "no longer needed" can be done with an atomically updated counter in the task, which starts out with a value of one, and is incremented when a task is put on a sorted, temporary nbor list, pending use in download. When used as a source in download, the counter is decremented, endingg up on one again, unless the task has been derefined away in the meantime, in which case the counter ends on zero, and the task is then deallocated / deleted.

Updating an nbor list with list_tinit_nbors() should be done in two steps: first the head of the list is allocated, without linking it to linknbor, and the rest of the tasks are appended (or prepended). Then, in an lock protected assignment, linknbor is pointed to the head of the new nbor list. Before that, a link is saved to the head of the existing nbor list, and after the atomic reassignment, the old list is deleted.

A derefined task is immediately removed from the task list, and hence can no longer be added to re-initialized nbor lists. It thus ceases to be needed when the last task that still has it on its nbor list copy deallocates its nbor list, after using it in dowload_link().

Summary: A sorted copy of the nbor list is created in connection with the queue_by_time() call in check_ready(). At the same time a counter task_t% n_needed is incremented for all tasks in the temporary nbor list. When the task is used as a source in download_link() the counter is decremented (atomically), and if the task is derefined the counter is decremented there as well, leading to a zero cound when the task is no longer on an nbor list copy. Then, and only then, the task may be deleted.

FIXME: Care should be extended also to updates over MPI. FIXME: Use of the deprecated shared_mod should be removed

MPI specific handling: Boundary patches are always sent to vnbors, and if they are new on the recieving rank, new nbor lists are generated, so no new actions should be needed there. If a boundary task is removed, the vnbors are send a copy with the "remove" bit set, and they should react correspond- ingly, by doing essentially what remove_and_reset does.