2.8.4. Batch Conversion

For a compact user-facing recommendation on choosing the export depth, see Choosing levelmax.

Run time depends strongly on:

  • the number of overlapping DISPATCH patches

  • whether the selected box is effectively uniform at the chosen levelmax

  • whether mode='octree' must reconstruct a large mixed-level tree

Real sink-centered timings from a validated ISM snapshot (run='20z-18s180-395NIMHD', iout=440, sink=178) were:

  • mode='octree', fixed-width sink-centered crop, sizeau=100 image:

    • levelmax=18, npix=64: export about 2.5 s, image about 1.4 s

    • levelmax=19, npix=128: export about 23 s, image about 8.3 s

    • levelmax=20, npix=256: export about 200s, image about 65 s

  • For a levelmax=20, npix=128 case, the original Python path took about 40 s for cell collection, 46 s for normalization, 5 s for boundary fill, and 91 s for oct-tree insertion; in all about 180 s. After a local-box optimization the same export now takes about 26 s total on the validated real snapshot.

When exporting many snapshots, use parallel processes rather than threads. Snapshot loading and oct-tree construction are CPU-heavy, and a separate process per conversion also isolates the temporary memory footprint of each job.

Example:

from concurrent.futures import ProcessPoolExecutor, as_completed

import dispatch


def convert_one(iout):
    sn = dispatch.snapshot(iout=iout, run='20z-18s180-395NIMHD')
    out = f'radmc3d_{iout:05d}'
    return dispatch.select.export2radmc3d(
        sn,
        path=out,
        mode='octree',
        sink=178,
        levelmax=20,
        npix=128,
        density='d',
        velocity=None)


iouts = [430, 435, 440, 445]

with ProcessPoolExecutor(max_workers=4) as pool:
    futures = {pool.submit(convert_one, iout): iout for iout in iouts}
    for future in as_completed(futures):
        iout = futures[future]
        result = future.result()
        print(iout, result['path'], result['mode'])

  • prefer one worker per export directory

  • start with max_workers=2 or 4, not with all cores

  • on shared systems, monitor memory before scaling up worker count

  • if you also run RADMC-3D immediately afterward, count that cost separately