Browsing by Subject "Graph neural networks (GNNs)"

Now showing 1 - 2 of 2

Open Access
Addressing encoder-only transformer limitations with graph neural networks for text classification
(2025-01) Aras, Arda Can
Recent advancements in NLP have been primarily driven by transformer-based models, which capture contextual information within sequences, revolutionizing tasks such as text classification and natural language understanding. In parallel, GNNs have emerged as powerful tools for modeling structured data, leveraging graph representations to capture global relationships across entities. However, significant challenges persist at the intersection of these fields, limiting the efficacy and scalability of existing models. These challenges include the inability to seamlessly integrate contextual and structural information, computational inefficiencies associated with static graph construction and transductive learning, and the underperformance of models in low-labeled data scenarios. This thesis explores and addresses these challenges by developing novel methodologies that unify transformers and GNNs, leveraging their complementary strengths. The first contribution, GRTE, introduces an architecture that combines pre-trained transformer models with heterogeneous and homogeneous graph representations to enhance text classification in both inductive and transductive settings. Compared to state-of-the-art models, GRTE achieves significant computational efficiency, reducing training overhead by up to 100 times. The second contribution, Text-RGNN, proposes a relational modeling framework for heterogeneous text graphs, enabling the nuanced representation of diverse interactions between nodes and demonstrating substantial accuracy improvements of up to 10.61% over existing models, particularly in low-labeled data settings. Finally, the third contribution, VISPool, introduces a scalable architecture that dynamically constructs vector visibility graphs from transformer outputs, enabling seamless integration of graph-based reasoning into transformer pipelines while improving performance on NLP benchmarks such as GLUE, with performance improvements of up to 13% in specific tasks. Through comprehensive experimentation and benchmarking against state-of-the-art models, this thesis establishes the efficacy of these proposed methodologies. The results demonstrate the potential for improved performance, scalability, and the ability to address long-standing challenges in NLP and GNN integration. These contributions lay a robust foundation for future research and applications at the intersection of graph-based and transformer-based approaches, advancing the state of the art in text representation and classification.
Open Access
Text-RGNNs: Relational modeling for heterogeneous text graphs
(IEEE, 2024) Aras, Arda Can; Alikaşifoğlu, Tuna; Koç, Aykut
Text-graph convolutional Network (TextGCN) is the fundamental work representing corpus with heterogeneous text graphs. Its innovative application of GCNs for text classification has garnered widespread recognition. However, GCNs are inherently designed to operate within homogeneous graphs, potentially limiting their performance. To address this limitation, we present Text Relational Graph Neural Networks (Text-RGNNs), which offer a novel methodology by assigning dedicated weight matrices to each relation within the graph by using heterogeneous GNNs. This approach leverages RGNNs, enabling nuanced and compelling modeling of relationships inherent in the heterogeneous text graphs, ultimately resulting in performance enhancements. We present a theoretical framework for the relational modeling of GNNs for text classification within the context of document classification and demonstrate its effectiveness through extensive experimentation on benchmark datasets. Conducted experiments reveal that Text-RGNNs outperform the existing state-of-the-art in scenarios with complete labeled nodes and minimal labeled training data proportions by incorporating relational modeling into heterogeneous text graphs. Text-RGNNs outperform the second-best models by up to 10.61% for the corresponding evaluation metric.