:::info
This paper is available on arxiv under CC 4.0 license.
Authors:
(1) Anh Nguyen-Duc, University of South Eastern Norway, BøI Telemark, Norway3800 and Norwegian University of Science and Technology, Trondheim, Norway7012;
(2) Beatriz Cabrero-Daniel, University of Gothenburg, Gothenburg, Sweden;
(3) Adam Przybylek, Gdansk University of Technology, Gdansk, Poland;
(4) Chetan Arora, Monash University, Melbourne, Australia;
(5) Dron Khanna, Free University of Bozen-Bolzano, Bolzano, Italy;
(6) Tomas Herda, Austrian Post – Konzern IT, Vienna, Austria;
(7) Usman Rafiq, Free University of Bozen-Bolzano, Bolzano, Italy;
(8) Jorge Melegati, Free University of Bozen-Bolzano, Bolzano, Italy;
(9) Eduardo Guerra, Free University of Bozen-Bolzano, Bolzano, Italy;
(10) Kai-Kristian Kemell, University of Helsinki, Helsinki, Finland;
(11) Mika Saari, Tampere University, Tampere, Finland;
(12) Zheying Zhang, Tampere University, Tampere, Finland;
(13) Huy Le, Vietnam National University Ho Chi Minh City, Hochiminh City, Vietnam and Ho Chi Minh City University of Technology, Hochiminh City, Vietnam;
(14) Tho Quan, Vietnam National University Ho Chi Minh City, Hochiminh City, Vietnam and Ho Chi Minh City University of Technology, Hochiminh City, Vietnam;
(15) Pekka Abrahamsson, Tampere University, Tampere, Finland.
:::
Table of Links
Abstract and Introduction
Background
Research Approach
Research Agenda
Outlook and Conclusions
References
Abstract
Context Generative Artificial Intelligence (GenAI) tools have become increasingly prevalent in software development, offering assistance to various managerial and technical project activities. Notable examples of these tools include OpenAI’s ChatGPT, GitHub Copilot, and Amazon CodeWhisperer.
Objective. Although many recent publications have explored and evaluated the application of GenAI, a comprehensive understanding of the current development, applications, limitations, and open challenges remains unclear to many. Particularly, we do not have an overall picture of the current state of GenAI technology in practical software engineering usage scenarios.
Method. We conducted a literature review and focus groups for a duration of five months to develop a research agenda on GenAI for Software Engineering.
Results. We identified 78 open Research Questions (RQs) in 11 areas of Software Engineering. Our results show that it is possible to explore the adoption of GenAI in partial automation and support decision-making in all software development activities. While the current literature is skewed toward software implementation, quality assurance and software maintenance, other areas, such as requirements engineering, software design, and software engineering education, would need further research attention. Common considerations when implementing GenAI include industry level assessment, dependability and accuracy, data accessibility, transparency, and sustainability aspects associated with the technology.
Conclusions. GenAI is bringing significant changes to the field of software engineering. Nevertheless, the state of research on the topic still remains immature. We believe that this research agenda holds significance and practical value for informing both researchers and practitioners about current applications and guiding future research.
Keywords: Generative Artificial Intelligence, GenAI, ChatGPT, CoPilot, Software Engineering, Software Project, Software Development, Focus Group, Literature review, Structured review, Literature survey, Research Agenda, Research Roadmap
1. Introduction
In recent years, Generative Artificial Intelligence (GenAI) has attracted significant attention and interest from Software Engineering (SE) research and practice. GenAI tools such as GitHub Copilot1 and ChatGPT2 have been rapidly explored in various professional contexts given their remarkable performance in producing human-like content. The emerging adoption of these tools reintroduces a host of classic concerns about productivity and quality when adopting new technologies. Clearly, code generation and test case optimization are SE tasks that directly benefit from recent large language models (LLMs) [1, 2, 3, 4]. Beyond traditional research on applied. Machine Learning (ML), GenAI tools introduced usability and accessibility, leveraging AI-generated content to a broader range of professionals, requiring less technical competence to work and integrate them into existing work environment. GenAI tools are also showing their potential in non-coding tasks, such as assisting requirements engineering, software processes and project management.
At the moment, GenAI is an active research areas with several challenges and open questions. To do well on specific tasks, LLMs require fine-tuning or training. Research has much focused so far on achieving reliable and scalable GenAI output for different SE tasks. GenAI models are inherently
nondeterministic: the same prompt produces different answers on different inference executions [5]. Moreover, GenAI’s output can be very sensitive to the input parameters or settings [6, 7]. Hallucination is another common concern with AI-generated content, as they can be imaginary or fictional [8]. The promise of AI automation might be closer than ever in SE [9], when these open challenges are sufficiently addressed.
Research agenda is a popular type of work for guiding and organizing research efforts in certain research areas [10, 11, 12, 13, 14, 15]. It often includes a review of existing work, and a presentation of directions, visions and priorities, enabling researchers to make meaningful contributions to their respective fields and to address pressing challenges. This research agenda is driven by past and current work on GenAI for SE. Based on focus groups and a literature review, we portray the current research on GenAI in SE and present the research agenda with open challenges for future research. While we do not emphasize the comprehensiveness of the literature review due to the fast-moving nature of the topic, the focus groups present practical expectations of the future roles of GenAI in software development. The research agenda is organized into 11 areas of concern, which are: (1) Requirements Engineering, (2) Software Design,(3) Sofware Implementation, (4) Quality Assurance, (5) Software Maintenance and Evolution, (6) Software Processes and Tools, (7) Engineering Management, (8) Professional Competencies, (9) Software Engineering Education, (10) Macro Aspects and (11) Fundamental concerns of GenAI.
Two international events have provided a ground for this work. As a part of the first international workshop on AI-assisted Agile Software Development3 , we ran our first focus group to explore the benefits and challenges of AI-assisted Agile software development. At the Requirements Engineering (RE) conference, we organized the second international workshop on Requirements Engineering for Software Startups and Emerging Technologies (RESET) 4 with a special theme on GenAI and Requirements Engineering.
Consequently, the identified open challenges reflect not just the existing gaps in literature but also draw from the practical, value-driven insights of the co-authors.
The paper is structured as follows: we clarify the meaning of GenAI and what we know about GenAI in SE in Section 2, and we will present our research approach in Section 3. Section 4 presents our Research Agenda. Finally, Section 5 is Outlook and Conclusions.