Avaliação de modelos de linguagem na resolução de problemas de programação

Abstract

The adoption of artificial intelligence-based tools in software development is an increasing trend. Accordingly, this work aimed to investigate the applicability of language models such as GPT-3.5, GPT-4, and the OpenAI Codex in solving programming challenges. To conduct this research, tools such as ChatGPT 3.5, ChatGPT 4, and Copilot were used, each integrating the aforementioned language models. The challenges were selected from the BeeCrowd platform, known for its diverse programming problems. This platform categorizes problems and difficulty levels, creating a conducive environment for research. It is important to note that problems containing images were excluded from selection due to the limitations of the tools used, which focus primarily on text. The problems were then categorized based on the platform’s existing difficulty levels and categories, grouped into two distinct sets: one comprising problems with the highest number of correct submissions and another highlighting those with the fewest correct submissions. The mentioned tools were then employed to develop solutions for the programming problems in question, whose resolutions were submitted to the BeeCrowd platform for evaluation. The results demonstrated the substantial potential of language models in source code generation, particularly in scenarios involving less complex programming problems, that is, those with a high number of correct submissions within the platform. The success rate varied among the different tools, with ChatGPT-4 standing out with the highest efficacy (76.47%), followed by Copilot (64.7%) and ChatGPT-3.5 (58.82%). However, for more complex problems, characterized by a reduced number of correct submissions, all models still showed room for improvement, with ChatGPT-4 leading with a success rate of 38.23%, followed by ChatGPT-3.5 (17.64%) and Copilot (16.17%). These results contribute to a better understanding of the role played by these language models in the context of programming problem resolution. They provide insight into how these technologies can be comparable to human abilities in code creation, serving as facilitators in the activities performed by software developers.