Interestingly NVIDIA says that this issue might be an error with an app:
XID 13: GR: SW Notify Error
This event is logged for general user application faults. Typically this is an out-of-bounds error where the user has walked past the end of an array, but could also be an illegal instruction, illegal register, or other case.
In rare cases, it’s possible for a hardware failure or system software bugs to materialize as XID 13.
When this event is logged, NVIDIA recommends the following:
Run the application in cuda-gdb or cuda-memcheck , or
Run the application with CUDA_DEVICE_WAITS_ON_EXCEPTION=1 and then attach later with cuda-gdb, or
File a bug if the previous two come back inconclusive to eliminate potential NVIDIA driver or hardware bug.
Note: The cuda-memcheck tool instruments the running application and reports which line of code performed the illegal read.
Also there’s this note in drivers 367.43:
5.1. Memory exhaustion can occur with vGPU profiles that have 512 Mbytes or less of frame buffer
Description
Memory exhaustion can occur with vGPU profiles that have 512 Mbytes or less of frame buffer. This issue typically occurs when multiple display heads are used with Citrix XenDesktop or VMware Horizon on a Windows 10 guest VM.
When this error occurs, the NVIDIA host driver reports Xid error 31 and Xid error 43 in the VMware vSphere log file vmware.log in the guest VM’s storage directory.
The following vGPU profiles have 512 Mbytes or less of frame buffer:
‣ Tesla M6-0B, M6-0Q
‣ Tesla M10-0B, M10-0Q
‣ Tesla M60-0B, M60-0Q
‣ GRID K100, K120Q
‣ GRID K200, K220Q
Version
Workaround
Status: Open
Last but not least when I downclocked my GPU VRAM from effective 8000MHz to 7600MHz the error disappeared (so far).
Also is too much to ask from you to create a CUDA/OpenCL memory checker? All the existing ones are dead/unsupported/don’t support GPUs with 4GB of VRAM or more.