Hello team,
We are planning to develop a tool on UnrealEngine which could assist Medical practitioners like surgeons to demonstrate a surgery on a soft tissue like heart,liver , brain ,abdomen .
For that I am planning to use the DICOM images of MRI/CT scans . Our proposal is to
From the DICOM image get a surface representation. I was suggested to use Kitwares slicer tool or VTK to achieve this
From the surface mesh create a tetra hedra
Now I plan to import this tetra hedra mesh into Unreal Engine
Now can I use IMSTK to add physics to the tetrahedral mesh ?
Are there any tools available to cut this tetrahedra and add interactions ?
Probably I feel , since IMSTK is build in C++, it can be integrated to UE
Hi Sameer, welcome to the iMSTK Discourse. A question upfront have you looked into Unreal’s Chaos Flesh ? Does that fullfill the your needs or is it not sufficient ?
We currently don’t support iMSTK and Unreal and might not provide a lot of support with the issues that you are facing. Besides that yes iMSTK can simulate volumes via PBD. Be that of bag like shells using surface meshes or solid substances via a tetrhedral representation. But getting everything into Unreal might be a lot of work. The rough points are
Create an interface between the imstk simulated mesh and the mesh that you are rendering in Unreal
Drive the iMSTK update from unreal (scene->advance), you should be running advance with fairly small timesteps i.e. < 0.01s, once you’ve updated to the “current time” read the updated data and modify your unreal objects accordingly
I’d suggest looking through the examples to get started, the documentation should also be helpful.
If you want to collide with this mesh you have two options, create all colliders in the imstk scene as well and communicate about updates between unreal and imstk, or do collision detection in unreal and convey that information into imstk (this will be the harder path).
Structurally all of these tasks are already being done by the iMSTK-Unity package, you might find some patterns from there useful as it’s solving the same problems, e.g. scene setup in Unity, mesh updates, rigid updates, etc.
What do you mean by “tools” surgical tools or an API ? Currently there is a class that can remove tetrahedra on the fly and update boundary representations under that change, but this is only for complete tetrahedra, There is another class that can split triangles in a surface mesh as well, that might be helpful, that hasn’t been touched in a while ymmv.
Hi ,
Thanks for all those pointers . I had a look at chaos flesh but its still not very compatible with what we want to achieve. I will try to work on the steps you suggested
Chaos expects you work with models created inside Unreal or volume created within unreal . If we try to import a Volume Mesh from a external entity like tetmesh …the result is not good and would need post processing
I had a quick look at the chaos flesh community tutorial (Chaos Flesh | Community tutorial). Of course it’s your decision what technology to use. But based one our experiences with integrating some of our technologies into other engines it’s never a trivial enterprise even if you are trying to target a small set of features.
I’d suggest you look at the procedures you are trying to simulate, do a task analysis (split the procedures up in their single steps), decide which of these tasks you want to actually simulate. From there you should be able to gauge what features you will need from the physics engine (e.g. deformations, grasping, cutting, …), with this you can then look at your options and make a decision based on the gaps (or non gaps). At the same time you can start experimenting with the technologies to better be able to make a decision.
Wrt to the mesh generation, you will probably have to do some pre and post processing either way. It sounds like unreal does it’s own tetrahedralization. You wouldn’t have to do it on your own.
I don’t see any “time stamps” in the algorithm if you’re talking about the “time steps” from line 1, i’d say as you are trying to solve continous cutting the path the scalpel is taking over time is interesting, meaning you at least need the previous and the current position.