Enterprise Organization of National Program of Excellence in Software

Eom Jiyong, a PhD student in SecAI Lab (Advisor: Professor Hyungjoon Koo, https://secai.skku.edu/) and Omar Abusabha, a PhD student in SoftSec Lab (Advisor: Professor Sungjae Hwang, https://softsec.skku.edu/), have co-authored a paper titled "A Deep Dive into Function Inlining and its Security Implications for ML-based Binary Analysis", which has been accepted for the premier security conference, The Network and Distributed System Security Symposium 2026 (NDSS '26), and is scheduled to be presented in February 2026.

Function inlining optimization is a representative technique used by compilers to improve program performance. Instead of making a function call, the compiler directly inserts the function's code at the call site, reducing the overhead associated with the function call. Function inlining is applied extensively throughout the compilation process, and some inlining occurs even when optimization options are disabled (-O0). Recently, machine learning models that assist in binary reverse engineering rely heavily on various static features of functions. However, in-depth analyses of how normal inlining optimizations affect the performance of these models have not yet been sufficiently conducted.

The study shows that function inlining can significantly distort the static features used by ML models, leading to performance degradation, and that attackers can intentionally exploit this using only the compiler's default flags, without employing complex techniques such as obfuscation. To investigate this, the authors first analyzed the inlining optimization mechanisms of the LLVM compiler toolchain and systematically organized the compiler options affecting inlining. They then derived combinations of options that induce higher inlining than typical optimization levels. Subsequently, experiments were conducted across five ML-based tasks—including binary reverse engineering and malware detection—using a total of 20 ML models.

Abstract: A function inlining optimization is a widely used transformation in modern compilers, which replaces a call site with the callee's body in need. While this transformation improves performance, it significantly impacts static features such as machine instructions and control flow graphs, which are crucial to binary analysis. Yet, despite its broad impact, the security impact of function inlining remains underexplored to date. In this paper, we present the first comprehensive study of function inlining through the lens of machine learning-based binary analysis. To this end, we dissect the inlining decision pipeline within the LLVM's cost model and explore the combinations of the compiler options that aggressively promote the function inlining ratio beyond standard optimization levels, which we term extreme inlining. We focus on five ML-assisted binary analysis tasks for security, using 20 unique models to systematically evaluate their robustness under extreme inlining scenarios. Our extensive experiments reveal several significant findings: i) function inlining, though a benign transformation in intent, can (in)directly affect ML model behaviors, being potentially exploited by evading discriminative or generative ML models; ii) ML models relying on static features can be highly sensitive to inlining; iii) subtle compiler settings can be leveraged to deliberately craft evasive binary variants; and iv) inlining ratios vary substantially across applications and build configurations, undermining assumptions of consistency in training and evaluation of ML models.

| Professor Hyungjoon Koo | kevin.koo@skku.edu, kevinkoo001.github.io

| SecAI Lab | secai.skku.edu/

| Professor Sungjae Hwang | sungjaeh@skku.edu

| SoftSec Lab | softsec.skku.edu/