Browsing by Subject "BERT"

Filter results by typing the first few letters
Now showing 1 - 4 of 4
  • Thumbnail Image
    ItemOpen Access
    Graph receptive transformer encoder for text classification
    (IEEE, 2024) Aras, Arda Can; Alikaşifoğlu, Tuna; Koç, Aykut
    By employing attention mechanisms, transformers have made great improvements in nearly all NLP tasks, including text classification. However, the context of the transformer's attention mechanism is limited to single sequences, and their fine-tuning stage can utilize only inductive learning. Focusing on broader contexts by representing texts as graphs, previous works have generalized transformer models to graph domains to employ attention mechanisms beyond single sequences. However, these approaches either require exhaustive pre-training stages, learn only transductively, or can learn inductively without utilizing pre-trained models. To address these problems simultaneously, we propose the Graph Receptive Transformer Encoder (GRTE), which combines graph neural networks (GNNs) with large-scale pre-trained models for text classification in both inductive and transductive fashions. By constructing heterogeneous and homogeneous graphs over given corpora and not requiring a pre-training stage, GRTE can utilize information from both large-scale pre-trained models and graph-structured relations. Our proposed method retrieves global and contextual information in documents and generates word embeddings as a by-product of inductive inference. We compared the proposed GRTE with a wide range of baseline models through comprehensive experiments. Compared to the state-of-the-art, we demonstrated that GRTE improves model performances and offers computational savings up to ˜100×.
  • Thumbnail Image
    ItemOpen Access
    Measuring and mitigating gender bias in legal contextualized language models
    (Association for Computing Machinery, 2024-02-13) Bozdağ, Mustafa; Sevim, Nurullah; Koç, Aykut
    Transformer-based contextualized language models constitute the state-of-the-art in several natural language processing (NLP) tasks and applications. Despite their utility, contextualized models can contain human-like social biases, as their training corpora generally consist of human-generated text. Evaluating and removing social biases in NLP models has been a major research endeavor. In parallel, NLP approaches in the legal domain, namely, legal NLP or computational law, have also been increasing. Eliminating unwanted bias in legal NLP is crucial, since the law has the utmost importance and effect on people. In this work, we focus on the gender bias encoded in BERT-based models. We propose a new template-based bias measurement method with a new bias evaluation corpus using crime words from the FBI database. This method quantifies the gender bias present in BERT-based models for legal applications. Furthermore, we propose a new fine-tuning-based debiasing method using the European Court of Human Rights (ECtHR) corpus to debias legal pre-trained models. We test the debiased models’ language understanding performance on the LexGLUE benchmark to confirm that the underlying semantic vector space is not perturbed during the debiasing process. Finally, we propose a bias penalty for the performance scores to emphasize the effect of gender bias on model performance.
  • Thumbnail Image
    ItemOpen Access
    Transformer-based bug/feature classification
    (IEEE - Institute of Electrical and Electronics Engineers, 2023-08-28) Öztürk, Ceyhun Emre; Yılmaz, E. H.; Köksal, Ö.
    Automatic classification of a software bug report as a 'bug' or 'feature' is essential to accelerate closed-source software development. In this work, we focus on automating the bug/feature classification task with artificial intelligence using a newly constructed dataset of Turkish software bug reports collected from a commercial project. We train and test support vector machine (SVM), k-nearest neighbors (KNN), convolutional neural network (CNN), transformer-based models, and similar artificial intelligence models on the collected reports. Results of the experiments show that transformer-based BERTurk is the best-performing model for the bug/feature classification task.
  • Thumbnail Image
    ItemOpen Access
    Türkçe mikroblog metinlerinde derin öğrenme dil modeli tabanlı konu tespiti
    (IEEE, 2021-07-19) Şahinuç, Furkan; Toraman, Çağrı; Koç, Aykut
    Mikroblog yazıları insanların sosyal medyada görü şlerini ifade ettikleri ve herhangi bir düzene bağlı olmayan kısa metinlerdir. Sosyal medya mikroblog metinlerinin konularına göre sınıflandırılması anlamsal bir altyapı oluştururken birçok uygulamaya da yardımcı olur. Bu çalışmada, mikroblog metinlerinde konu tespiti problemi için geleneksel kelime torbası ve derin öğrenme tabanlı modellerin karşılaştırıldığı bir analiz sunulmaktadır. Veri kümesinin oluşturulması için Türkiye’de yaşanan güncel olaylarla ilgili mikroblog metinleri içeren Türkçe "tweet"ler toplanmıştır. Oluşturulan veri kümesindeki "tweet"ler içerdikleri "hashtag" ifadelerine göre etiketlenmiştir. Son haline getirilen veri kümesinde bir adet geleneksel kelime torbası (TFIDF tabanlı SVM) ve iki adet güncel derin öğrenme yöntemi (BERT ve BERTurk) ile eğitim yapılmıştır. Modellerin başarısı ağırlıklı F1 skoru ile ölçülmüştür. TF-IDF tabanlı SVM 0,807, BERT 0,831 ve BERTurk 0,854 F1 skoru elde etmiştir.