SENSEI data adaptor mesh metadata for multiple chunk rectilinear grid

SENSEI
1

Hi everyone,

I’m currently instrumenting a miniapp with rectilinear grids, multiple processes (MPI) and multiple chunks on each process. For example, I might have 64 MPI processes with 9 chunks per process.

How should I map this to a VTK data structure?

  • A vtkMultiBlockDataSet dataset containing vtkRectilinearGrid objects?
  • A vtkMultiBlockDataSet, containing vtkMultiPieceDataSet objects, containing vtkRectilinearGrid objects?
  • Something else?

How should I set up the metadata in the data adaptor?

  • Should NumBlocks be the number of MPI ranks, or the number of MPI ranks * chunks per process
  • Should NumBlocksLocal be 1 or number of chunks per process

Thanks for you help!

Jim :smile:

2

Hi @benjaminjeliot !

A little more information would help; are the 9 rectlinear grids within a process related to each other? Do they overlap? Are there 9 per rank regardless of problem size (i.e., the simulation uses them for 9 different purposes) or is this the distribution you are using on one specific combination of simulation domain and HPC machine?

If all the chunks across all the ranks form a single, large rectilinear grid, you should be able to model the data as a single grid with different extents on each chunk (no multiblock involved). The oscillator miniapp’s DataAdaptor is a good pattern to follow if this is the case.

If not, say because the grids hold different field data or overlap, then it might be better for each grid to be exposed as a different mesh by your data adaptor rather than grouping them into a multiblock.

3

Hi @dcthomp,

Thanks for your reply! :smile:

All the chunks across all the ranks form one rectilinear grid. The grid is first split into MPI ranks, then split again on each process. The number of (sub)chunks on each process is the same for all processes, but can vary from run to run. Every (sub)chunk has a ghost halo.

So are you saying that:

  • metadata MeshType should be VTK_RECTILINEAR_GRID?
  • metadata NumBlocksLocal is number of chunks on each process?

What should I return from GetMesh? Should it be a vtkMultiBlockDataSet containing n_chunks_per_process vtkRectilinearGrids?

Thanks again,
Jim

4

Yes. Where you see the oscillator return VTK_IMAGE_DATA (meshName == “mesh”) or VTK_UNSTRUCTURED_GRID (meshName=“ucdmesh”) you should return VTK_RECTILINEAR_GRID instead.

Yes.