Data di Pubblicazione:
2023
Citazione:
CombTransformers: Statement-Wise Transformers for Statement-Wise Representations / F. Bertolotti, W. Cazzola. - In: IEEE TRANSACTIONS ON SOFTWARE ENGINEERING. - ISSN 0098-5589. - 49:10(2023), pp. 4677-4690. [10.1109/TSE.2023.3310793]
Abstract:
This study presents a novel category of Transformer architectures known as comb transformers, which effectively reduce the space complexity of the self-attention layer from a quadratic to a subquadratic level. This is achieved by processing sequence segments independently and incorporating X -word embeddings to merge cross-segment information. The reduction in attention memory requirements enables the deployment of deeper architectures, potentially leading to more competitive outcomes. Furthermore, we design an abstract syntax tree (AST)-based code representation to effectively exploit comb transformer properties. To explore the potential of our approach, we develop nine specific instances based on three popular architectural concepts: funnel, hourglass, and encoder-decoder. These architectures are subsequently trained on three code-related tasks: method name generation, code search, and code summarization. These tasks encompass a range of capabilities: short/long sequence generation and classification. In addition to the proposed comb transformers, we also evaluate several baseline architectures for comparative analysis. Our findings demonstrate that the comb transformers match the performance of the baselines and frequently perform better.
Tipologia IRIS:
01 - Articolo su periodico
Keywords:
Codes, Transformers, Task analysis, Computer architecture, Artificial neural networks, Documentation, Training
Elenco autori:
F. Bertolotti, W. Cazzola
Link alla scheda completa:
Link al Full Text: