<4>[ 2554.736152] NVRM: loading NVIDIA UNIX x86_64 Kernel Module  418.67  Sat Apr  6 03:07:24 CDT 2019
<4>[    9.815323] NVRM: loading NVIDIA UNIX x86_64 Kernel Module  418.67  Sat Apr  6 03:07:24 CDT 2019
<4>[    9.868780] NVRM: loading NVIDIA UNIX x86_64 Kernel Module  418.67  Sat Apr  6 03:07:24 CDT 2019
<4>[427273.385600] NVRM: failed to copy out ioctl data
<4>[583364.603649] NVRM: failed to copy out ioctl data
<4>[1339358.550880] NVRM: GPU at PCI:0000:05:00: GPU-50d762b3-c9fc-0f61-0ac8-844c9c4cba76
<4>[1339358.550898] NVRM: GPU Board Serial Number: 1564819002810
<4>[1339358.550902] NVRM: Xid (PCI:0000:05:00): 63, Dynamic Page Retirement: New page retired, reboot to activate (0x0000000000679694).
<4>[    9.824871] NVRM: loading NVIDIA UNIX x86_64 Kernel Module  418.67  Sat Apr  6 03:07:24 CDT 2019
<4>[602035.067573] NVRM: GPU at PCI:0000:84:00: GPU-a25bfe4b-3e44-ab1c-94da-208c78183fba
<4>[602035.067586] NVRM: GPU Board Serial Number: 1564819003353
<4>[602035.067589] NVRM: Xid (PCI:0000:84:00): 63, Dynamic Page Retirement: New page retired, reboot to activate (0x000000000006a066).
<4>[    9.835329] NVRM: loading NVIDIA UNIX x86_64 Kernel Module  418.67  Sat Apr  6 03:07:24 CDT 2019
<4>[    9.830074] NVRM: loading NVIDIA UNIX x86_64 Kernel Module  418.67  Sat Apr  6 03:07:24 CDT 2019
<4>[2042401.373214] NVRM: GPU at PCI:0000:83:00: GPU-da4dd811-8954-1f67-bd1a-2e0e2e6da387
<4>[2042401.373230] NVRM: GPU Board Serial Number: 1564819002884
<4>[2042401.373233] NVRM: Xid (PCI:0000:83:00): 13, Graphics SM Warp Exception on (GPC 3, TPC 2, SM 1): Out Of Range Address
<4>[2042401.373303] NVRM: Xid (PCI:0000:83:00): 13, Graphics SM Global Exception on (GPC 3, TPC 2, SM 1): Multiple Warp Errors
<4>[2042401.373350] NVRM: Xid (PCI:0000:83:00): 13, Graphics Exception: ESR 0x51d7b0=0x101000e 0x51d7b4=0x4 0x51d7a8=0x4c1eb72 0x51d7ac=0x174
<4>[2042401.374685] NVRM: Xid (PCI:0000:83:00): 13, Graphics Exception: ChID 0018, Class 0000c3c0, Offset 00000510, Data 00419e84
<4>[2042401.402599] NVRM: Xid (PCI:0000:83:00): 31, Ch 00000018, intr 00000000. MMU Fault: ENGINE GRAPHICS GPCCLIENT_GCC faulted @ 0x2_86d4c000. Fault is of type FAULT_PDE ACCESS_TYPE_VIRT_READ
<4>[2077540.572568] NVRM: Xid (PCI:0000:83:00): 13, Graphics SM Warp Exception on (GPC 3, TPC 2, SM 1): Out Of Range Address
<4>[2077540.572622] NVRM: Xid (PCI:0000:83:00): 13, Graphics Exception: ESR 0x51d7b0=0x101000e 0x51d7b4=0x0 0x51d7a8=0x4c1eb72 0x51d7ac=0x174
<4>[2077540.576646] NVRM: Xid (PCI:0000:83:00): 13, Graphics Exception: ChID 0018, Class 0000c3c0, Offset 00000510, Data 00419e84
<4>[2077540.602722] NVRM: Xid (PCI:0000:83:00): 31, Ch 00000018, intr 00000000. MMU Fault: ENGINE GRAPHICS GPCCLIENT_GCC faulted @ 0x3197_fed4c000. Fault is of type FAULT_PDE ACCES
<4>
<4>[2077865.277969] NVRM: Xid (PCI:0000:83:00): 31, Ch 00000018, intr 00000000. MMU Fault: ENGINE GRAPHICS GPCCLIENT_T1_5 faulted @ 0x7ffe_e5aa3000. Fault is of type FAULT_PDE ACCE
<4>
<4>[2109578.094810] NVRM: Xid (PCI:0000:83:00): 31, Ch 00000018, intr 00000000. MMU Fault: ENGINE GRAPHICS GPCCLIENT_T1_5 faulted @ 0x7fe3_48ad3000. Fault is of type FAULT_PDE ACCESS_TYPE_VIRT_READ
<4>[2110315.458916] NVRM: Xid (PCI:0000:83:00): 31, Ch 00000018, intr 00000000. MMU Fault: ENGINE GRAPHICS GPCCLIENT_T1_5 faulted @ 0x7fd6_e0418000. Fault is of type FAULT_PDE ACCESS_TYPE_VIRT_READ
<4>[2112908.025769] NVRM: Xid (PCI:0000:83:00): 31, Ch 00000018, intr 00000000. MMU Fault: ENGINE GRAPHICS GPCCLIENT_T1_5 faulted @ 0x7fbd_48c2c000. Fault is of type FAULT_PDE ACCESS_TYPE_VIRT_READ
<4>[2113263.754149] NVRM: Xid (PCI:0000:83:00): 31, Ch 00000018, intr 00000000. MMU Fault: ENGINE GRAPHICS GPCCLIENT_T1_5 faulted @ 0x7f2d_d476d000. Fault is of type FAULT_PDE ACCESS_TYPE_VIRT_READ
<4>[2113646.165150] NVRM: Xid (PCI:0000:83:00): 31, Ch 00000018, intr 00000000. MMU Fault: ENGINE GRAPHICS GPCCLIENT_T1_5 faulted @ 0x7fe3_a55e4000. Fault is of type FAULT_PDE ACCE
<4>
<4>[2113972.777074] NVRM: Xid (PCI:0000:83:00): Jul  1 11:20:53 vla2-8840 kernel: [2114412.733779] pid 471305 (print-crit-chec), uid 29066: exited on signal 0
<4>[2114587.300105] NVRM: Xid (PCI:0000:83:00): 31, Ch 00000018, intr 00000000. MMU Fault: ENGINE GRAPHICS GPCCLIENT_T1_5 faulted @ 0x7fe6_d23a4000. Fault is of type FAULT_PDE ACCE
<4>
<4>[2115495.573803] NVRM: Xid (PCI:0000:83:00): 31, Ch 00000018, intr 00000000. MMU Fault: ENGINE GRAPHICS GPCCLIENT_T1_5 faulted @ 0x7fca_d2f54000. Fault is of type FAULT_PDE ACCE
<4>
<4>[2203342.749040] NVRM: Xid (PCI:0000:83:00): 31, Ch 00000018, intr 00000000. MMU Fault: ENGINE GRAPHICS GPCCLIENT_T1_5 faulted @ 0x7ff1_f3dc0000. Fault is of type FAULT_PDE ACCE
<4>
<4>[2203432.747556] NVRM: Xid (PCI:0000:83:00): 31, Ch 00000018, intr 00000000. MMU Fault: ENGINE GRAPHICS GPCCLIENT_T1_5 faulted @ 0x7f62_91050000. Fault is of type FAULT_PDE ACCE
<4>SS_TYPE_VIRT_READ
