# general exception crash / restarting from last good epoch

**URL:** https://discourse.kitware.com/t/general-exception-crash-restarting-from-last-good-epoch/1668
**Category:** VIAME & DIVE
**Created:** [March 9, 2026, 2:34pm UTC](https://discourse.kitware.com/t/general-exception-crash-restarting-from-last-good-epoch/1668 "2026-03-09T14:34:29Z")
**Posts on this page:** 3
**Page:** 1

<div class="post-metadata">

### Author: ![christian\_reiss](https://discourse.kitware.com/user_avatar/discourse.kitware.com/christian_reiss/32/1187_2.png) [@christian\_reiss](https://discourse.kitware.com/u/christian_reiss)
#### Post date: [March 9, 2026, 2:34pm UTC](https://discourse.kitware.com/t/general-exception-crash-restarting-from-last-good-epoch/1668/1 "2026-03-09T14:34:30Z")

</div>

Hi all-

I was training a model using the desktop viame and the the code stopped executing with this general exception fault. I am wondering whether it would be possible to restart the model at the last good epoch, rather than running the model from the beginning. Also, if anyone knows whether this exception a common fault, and there is a fix, that would be good to know.

The computer is a lenovo/legion 32GB mem, rtx 4070, running windows 11.

INFO: general exception  
harn.preferences = \<FitHarnPreferences({‘keyboard\_debug’: False, ‘snapshot\_after\_error’: True, ‘deploy\_after\_error’: True, ‘show\_prog’: True, ‘prog\_backend’: ‘progiter’, ‘ignore\_inf\_loss\_parts’: False, ‘log\_gradients’: False, ‘use\_tensorboard’: True, ‘eager\_dump\_tensorboard’: True, ‘dump\_tensorboard’: True, ‘tensorboard\_groups’: [‘loss’], ‘export\_modules’: [‘bioharn’], ‘export\_on\_init’: True, ‘large\_loss’: 1000, ‘num\_keep’: 10, ‘keep\_freq’: 5, ‘timeout’: 1209600, ‘auto\_prepare\_batch’: False, ‘verbose’: 1, ‘log\_resources’: True, ‘use\_tqdm’: None, ‘colored’: True, ‘allow\_unicode’: False}) at 0x21205e78070\>  
INFO: Attempting to checkpoint before crashing  
INFO: Saving EXPLICIT snapshot to deep\_training\fit\runs\viame-netharn-detector\hesvuwxi\explicit\_checkpoints\_epoch\_00000036\_2026-03-09T071653-5.pt  
Traceback (most recent call last):  
File “C:\Program Files\VIAME\lib\python3.10\site-packages\viame\pytorch\netharn\fit\_harn.py”, line 1594, in run  
harn.\_run\_tagged\_epochs(  
File “C:\Program Files\VIAME\lib\python3.10\site-packages\viame\pytorch\netharn\fit\_harn.py”, line 1822, in \_run\_tagged\_epochs  
harn.\_run\_epoch(train\_loader, tag=‘train’, learn=True)  
File “C:\Program Files\VIAME\lib\python3.10\site-packages\viame\pytorch\netharn\fit\_harn.py”, line 2064, in _run\_epoch  
harn.backpropogate(bx, batch, loss)  
File “C:\Program Files\VIAME\lib\python3.10\site-packages\viame\pytorch\netharn\fit\_harn.py”, line 2504, in backpropogate  
loss.backward()  
File “C:\Program Files\VIAME\lib\python3.10\site-packages\torch\_tensor.py”, line 630, in backward  
torch.autograd.backward(  
File "C:\Program Files\VIAME\lib\python3.10\site-packages\torch\autograd\_init_.py", line 364, in backward  
\_engine\_run\_backward(  
File “C:\Program Files\VIAME\lib\python3.10\site-packages\torch\autograd\graph.py”, line 865, in \_engine\_run\_backward  
return Variable.\_execution\_engine.run\_backward( # Calls into the C++ engine to run the backward pass  
RuntimeError: cuDNN error: CUDNN\_STATUS\_INTERNAL\_ERROR\_HOST\_ALLOCATION\_FAILED

During handling of the above exception, another exception occurred:

Traceback (most recent call last):  
File “C:\Program Files\VIAME\lib\python3.10\runpy.py”, line 196, in \_run\_module\_as\_main  
return \_run\_code(code, main\_globals, None,  
File “C:\Program Files\VIAME\lib\python3.10\runpy.py”, line 86, in \_run\_code  
exec(code, run\_globals)  
File “C:\Program Files\VIAME\lib\python3.10\site-packages\viame\pytorch\netharn\detect\_fit.py”, line 1405, in  
fit()  
File “C:\Program Files\VIAME\lib\python3.10\site-packages\viame\pytorch\netharn\detect\_fit.py”, line 1324, in fit  
return harn.run()  
File “C:\Program Files\VIAME\lib\python3.10\site-packages\viame\pytorch\netharn\fit\_harn.py”, line 1674, in run  
harn.save\_snapshot(explicit=True)  
File “C:\Program Files\VIAME\lib\python3.10\site-packages\viame\pytorch\netharn\fit\_harn.py”, line 1226, in save\_snapshot  
torch.save(snapshot\_state, save\_file)  
File “C:\Program Files\VIAME\lib\python3.10\site-packages\torch\serialization.py”, line 977, in save  
\_save(  
File “C:\Program Files\VIAME\lib\python3.10\site-packages\torch\serialization.py”, line 1284, in _save  
storage = storage.cpu()  
File “C:\Program Files\VIAME\lib\python3.10\site-packages\torch\storage.py”, line 264, in cpu  
return torch.UntypedStorage(self.size()).copy_(self, False)  
torch.AcceleratorError: CUDA error: out of memory  
Search for `cudaErrorMemoryAllocation' in https://docs.nvidia.com/cuda/cuda-runtime-api/group __CUDART__ TYPES.html for more information. CUDA kernel errors might be asynchronously reported at some other API call, so the stacktrace below might be incorrect. For debugging consider passing CUDA_LAUNCH_BLOCKING=1 Compile with `TORCH\_USE\_CUDA\_DSA` to enable device-side assertions.

---

<div class="post-metadata">

### Author: ![matt\_dawkins](https://discourse.kitware.com/user_avatar/discourse.kitware.com/matt_dawkins/32/523_2.png) [@matt\_dawkins](https://discourse.kitware.com/u/matt_dawkins)
#### Post date: [March 9, 2026, 6:55pm UTC](https://discourse.kitware.com/t/general-exception-crash-restarting-from-last-good-epoch/1668/2 "2026-03-09T18:55:03Z")

</div>

Which model were you training?

Your main error is: “CUDA error: out of memory” - this means your GPU is just running out of VRAM on the card. This is usually due to either:

1. You launched some other application that is using VRAM too, like a video call, firefox playing video, some CAD program, etc…

2. Your training config file / image size is just using too much VRAM. While we try to automatically set these values, it doesn’t always work. To resolve this, you would need to lower the ‘batch\_size’ parameter down in the config system of whatever model you’re using, which can be specified differently depending on the model.

As for your other question about restarting where you trained off, if using the default netharn/cfrnn model, then a script like the following can accomplish it (if not the answer is still yes and you have to go into the config system for the respective detector):

> <https://github.com/VIAME/VIAME/blob/main/configs/prj-windows/continue_training_deep_cfrnn_detector.bat>

> <https://github.com/VIAME/VIAME/blob/main/configs/prj-linux/continue_training_deep_cfrnn_detector.sh>

---

<div class="post-metadata">

### Author: ![christian\_reiss](https://discourse.kitware.com/user_avatar/discourse.kitware.com/christian_reiss/32/1187_2.png) [@christian\_reiss](https://discourse.kitware.com/u/christian_reiss)
#### Post date: [March 9, 2026, 8:38pm UTC](https://discourse.kitware.com/t/general-exception-crash-restarting-from-last-good-epoch/1668/3 "2026-03-09T20:38:07Z")

</div>

Thanks Matt! This will help me out a lot. I am certain when I opened up a matlab instance and started overworking my laptop that is when it crapped out. Lesson learned!

c