Challenges in Decompilation and Reverse Engineering of CUDA-based Kernels

Parallel computing architectures -- such as CUDA -- are taking on a central role in areas such as artificial intelligence, graphics, image processing, machine learning, and deep learning, with virtually unlimited applications. Unlike desktop and embedded platforms, these solutions have unique characteristics that call into question many of the assumptions underlying the main decompilers.

Research on decompilers has historically focused on desktop and embedded architectures; in the parallel context, however, new challenges emerge: opaque (black-box) hardware models, parallel execution semantics, and complex type systems (tensors, matrices).

In this work, we address decompilation for CUDA-based kernels, exploring concepts such as divergence, synchronization, and barriers, analyzing memory hierarchies, and discussing how to integrate them into a coherent framework for a proof-of-concept decompiler. We explore the file formats provided by NVIDIA compilers, the languages that have been designed and we consider how to address many of the challenges provided by newer parallel architectures.

We analyze the role of intermediate representations such as PTX and SASS in enabling or constraining the reconstruction of high-level semantics from compiled CUDA kernels. The roadmap includes stages such as IR lifting from PTX, reconstruction of control-flow and synchronization semantics, and recovery of high-level data abstractions.

The goal is to outline the differences introduced by CUDA and draw up a roadmap for designing a decompiler targeted at this architecture. Such a tool would enable applications ranging from software verification to malware analysis, from low-level optimization to compiler troubleshooting.