Analysis of ChatGPT's capabilities for solving problems of reverse-engineering of software

Authors

  • Volodymyr Sokolov Institute of special communication and information protection of National technical university of Ukraine “Igor Sikorsky Kyiv polytechnic institute”, Kyiv, Ukraine https://orcid.org/0000-0002-5779-7167

DOI:

https://doi.org/10.20535/2411-1031.2023.11.2.293778

Keywords:

ChatGPT, reverse engineering, software

Abstract

The article presents the results of the analysis and evaluation of the capabilities of the open version of the chatbot with artificial intelligence ChatGPT 3.5 for solving typical problems of software reverse engineering. Three classes of reverse engineering tasks were selected for analysis: source code analysis, binary code analysis, and data models analysis. In each class of tasks, the most typical tasks were selected taking into account the limitations of ChatGPT regarding the processing of graphical models and the amount of input and output data, and sets of test tasks were developed for each task. As an approach to assessment, an approach similar to the assessment of competencies of higher education applicants after studying the relevant discipline was chosen. The following were considered as criteria for evaluating answers to test tasks: correctness (rightness, coincidence with expectations); completeness (obtaining the final result); accuracy (solving the task without additional questions); reasonableness (availability of explanations and answers to questions). The following scale was used and described for evaluations: excellent, very good, good, satisfactory, sufficient, unsatisfactory. During the testing for each test task, tasks statement and all necessary data were entered through the ChatGPT interface. As a result of the analysis, it was determined that ChatGPT better solves the problems of source code analysis (excellent and very good grades were obtained for semantic and structural analysis, restoration of mathematical support, quality assessment, security audit and refactoring, as well as for conversion to another programming language), tasks of decompilation of IDA pseudocode into complete C source code, tasks of reverse engineering of relational databases, and tasks of generating YARA rules for recognizing file formats. Unsatisfactory grades were obtained for dynamic analysis of assembly code and determination of binary file structures of non-standard formats. ChatGPT solves the rest of the problems well and satisfactorily, but requires checking the results, clarifying queries and prompts, as well as manual error correction in some cases. ChatGPT errors were observed when analyzing binary data represented by hexadecimal symbols, as well as errors in generated scripts for programming in IDA. On the basis of the set grades, conclusions were made regarding the expediency, possibility, or impracticality of using ChatGPT to solve each type of software reverse engineering problems, and appropriate recommendations were also provided. Prospects for further research include testing new versions of ChatGPT and other similar artificial intelligence systems regarding the capabilities of analyzing and synthesizing graphical models of software.

Author Biography

Volodymyr Sokolov, Institute of special communication and information protection of National technical university of Ukraine “Igor Sikorsky Kyiv polytechnic institute”, Kyiv

candidate of technical sciences, associate professor, associate professor at the cybersecurity and application of information systems and technologies academic department

References

X. Xu et al., “LmPa: Improving Decompilation by Synergy of Large Language Model and Program Analysis”, arXiv preprint arXiv:2306.02546, 2023. doi: https://doi.org/10.48550/arXiv.2306.02546.

S. Frieder et al. “Mathematical capabilities of chatgpt”, arXiv preprint arXiv:2301.13867, 2023. doi: https://doi.org/10.48550/arXiv.2301.13867.

Z. Wang, Z. Lan, and L. Peng, “ChatGPT for Software Security: Exploring the Strengths and Limitations of ChatGPT in the Security Applications”, arXiv preprint arXiv:2307.12488, 2023. doi: https://doi.org/10.48550/arXiv.2307.12488.

W. Tann et al., “Using Large Language Models for Cybersecurity Capture-The-Flag Challenges and Certification Questions”, arXiv preprint arXiv:2308.10443, 2023. doi: https://doi.org/10.48550/arXiv.2308.10443.

M. Gupta, C. kiri, K. Aryal, E. Parker, and L. Praharaj, “From ChatGPT to ThreatGPT: Impact of Generative AI in Cybersecurity and Privacy”, arXiv preprint arXiv:2307.00691, 2023. doi: https://doi.org/10.48550/arXiv.2307.00691.

A. Qammar et al., “Chatbots to ChatGPT in a Cybersecurity Space: Evolution, Vulnerabilities, Attacks, Challenges, and Future Recommendations”, arXiv preprint arXiv:2306.09255, 2023. doi: https://doi.org/10.48550/arXiv.2306.09255.

B. Yetiştiren et al., “Evaluating the Code Quality of AI-Assisted Code Generation Tools: An Empirical Study on GitHub Copilot, Amazon CodeWhisperer, and ChatGPT”, arXiv preprint arXiv:2304.10778, 2023. doi: https://doi.org/10.48550/arXiv.2304.10778.

A. Sadik et al., “Analysis of ChatGPT on Source Code”, arXiv preprint arXiv:2306.00597, 2023. doi: https://doi.org/10.48550/arXiv.2306.00597.

J. White et al., “Chatgpt prompt patterns for improving code quality, refactoring, requirements elicitation, and software design”, arXiv preprint arXiv:2303.07839, 2023. doi: https://doi.org/10.48550/arXiv.2303.07839.

A. Buscemi, “A Comparative Study of Code Generation using ChatGPT 3.5 across 10 Programming Languages”, arXiv preprint arXiv:2308.04477, 2023. doi: https://doi.org/10.48550/arXiv.2308.04477.

Md. M. Rahman, and W. Yutaka, “ChatGPT for Education and Research: Opportunities, Threats, and Strategies”, Applied Sciences, vol. 13 (9), art. 5783, 2023. doi: https://doi.org/10.3390/app13095783.

Downloads

Published

2023-12-28

How to Cite

Sokolov, V. (2023). Analysis of ChatGPT’s capabilities for solving problems of reverse-engineering of software. Collection "Information Technology and Security", 11(2), 193–205. https://doi.org/10.20535/2411-1031.2023.11.2.293778

Issue

Vol. 11 No. 2 (2023)

Section

ARTIFICIAL INTELLIGENCE IN THE CYBERSECURITY FIELD

License

Copyright (c) 2023 Collection "Information Technology and Security"

Creative Commons License

This work is licensed under a Creative Commons Attribution 4.0 International License.

The authors that are published in this collection, agree to the following terms:

  1. The authors reserve the right to authorship of their work and pass the collection right of first publication this work is licensed under the Creative Commons Attribution License, which allows others to freely distribute the published work with the obligatory reference to the authors of the original work and the first publication of the work in this collection.
     
  2. The authors have the right to conclude an agreement on exclusive distribution of the work in the form in which it was published this anthology (for example, to place the work in a digital repository institution or to publish in the structure of the monograph), provided that references to the first publication of the work in this collection.
     
  3. Policy of the journal allows and encourages the placement of authors on the Internet (for example, in storage facilities or on personal web sites) the manuscript of the work, prior to the submission of the manuscript to the editor, and during its editorial processing, as it contributes to productive scientific discussion and positive effect on the efficiency and dynamics of citations of published work (see The Effect of Open Access).