Leveraging Open-Source Large Language Models for Native Language Identification

Yee Man Ng; Ilia Markov

Leveraging Open-Source Large Language Models for Native Language Identification

Research output: Working paper / Preprint › Preprint › Academic

21 Downloads (Pure)

Abstract

Native Language Identification (NLI) - the task of identifying the native language (L1) of a person based on their writing in the second language (L2) - has applications in forensics, marketing, and second language acquisition. Historically, conventional machine learning approaches that heavily rely on extensive feature engineering have outperformed transformer-based language models on this task. Recently, closed-source generative large language models (LLMs), e.g., GPT-4, have demonstrated remarkable performance on NLI in a zero-shot setting, including promising results in open-set classification. However, closed-source LLMs have many disadvantages, such as high costs and undisclosed nature of training data. This study explores the potential of using open-source LLMs for NLI. Our results indicate that open-source LLMs do not reach the accuracy levels of closed-source LLMs when used out-of-the-box. However, when fine-tuned on labeled training data, open-source LLMs can achieve performance comparable to that of commercial LLMs.

Original language	Undefined/Unknown
Publication status	Published - 15 Sept 2024

Keywords

cs.CL

Access to Document

2409.09659v1

Persistent URL (handle)

Persistent link

1 Conference contribution

Leveraging Open-Source Large Language Models for Native Language Identification
Ng, Y. M. & Markov, I., 2025, VarDial 2025 - 12th Workshop on NLP for Similar Languages, Varieties and Dialects, Proceedings of the Workshop. Scherrer, Y., Jauhiainen, T., Ljubesic, N., Nakov, P., Tiedemann, J. & Zampieri, M. (eds.). Association for Computational Linguistics (ACL), p. 20-28 9 p. (VarDial 2025 - 12th Workshop on NLP for Similar Languages, Varieties and Dialects, Proceedings of the Workshop).
Research output: Chapter in Book / Report / Conference proceeding › Conference contribution › Academic › peer-review

Cite this

@techreport{b4cf46ce349445faa2497de26ac4e286,

title = "Leveraging Open-Source Large Language Models for Native Language Identification",

abstract = "Native Language Identification (NLI) - the task of identifying the native language (L1) of a person based on their writing in the second language (L2) - has applications in forensics, marketing, and second language acquisition. Historically, conventional machine learning approaches that heavily rely on extensive feature engineering have outperformed transformer-based language models on this task. Recently, closed-source generative large language models (LLMs), e.g., GPT-4, have demonstrated remarkable performance on NLI in a zero-shot setting, including promising results in open-set classification. However, closed-source LLMs have many disadvantages, such as high costs and undisclosed nature of training data. This study explores the potential of using open-source LLMs for NLI. Our results indicate that open-source LLMs do not reach the accuracy levels of closed-source LLMs when used out-of-the-box. However, when fine-tuned on labeled training data, open-source LLMs can achieve performance comparable to that of commercial LLMs.",

keywords = "cs.CL",

author = "Ng, {Yee Man} and Ilia Markov",

year = "2024",

month = sep,

day = "15",

language = "Undefined/Unknown",

type = "WorkingPaper",

}

TY - UNPB

T1 - Leveraging Open-Source Large Language Models for Native Language Identification

AU - Ng, Yee Man

AU - Markov, Ilia

PY - 2024/9/15

Y1 - 2024/9/15

N2 - Native Language Identification (NLI) - the task of identifying the native language (L1) of a person based on their writing in the second language (L2) - has applications in forensics, marketing, and second language acquisition. Historically, conventional machine learning approaches that heavily rely on extensive feature engineering have outperformed transformer-based language models on this task. Recently, closed-source generative large language models (LLMs), e.g., GPT-4, have demonstrated remarkable performance on NLI in a zero-shot setting, including promising results in open-set classification. However, closed-source LLMs have many disadvantages, such as high costs and undisclosed nature of training data. This study explores the potential of using open-source LLMs for NLI. Our results indicate that open-source LLMs do not reach the accuracy levels of closed-source LLMs when used out-of-the-box. However, when fine-tuned on labeled training data, open-source LLMs can achieve performance comparable to that of commercial LLMs.

AB - Native Language Identification (NLI) - the task of identifying the native language (L1) of a person based on their writing in the second language (L2) - has applications in forensics, marketing, and second language acquisition. Historically, conventional machine learning approaches that heavily rely on extensive feature engineering have outperformed transformer-based language models on this task. Recently, closed-source generative large language models (LLMs), e.g., GPT-4, have demonstrated remarkable performance on NLI in a zero-shot setting, including promising results in open-set classification. However, closed-source LLMs have many disadvantages, such as high costs and undisclosed nature of training data. This study explores the potential of using open-source LLMs for NLI. Our results indicate that open-source LLMs do not reach the accuracy levels of closed-source LLMs when used out-of-the-box. However, when fine-tuned on labeled training data, open-source LLMs can achieve performance comparable to that of commercial LLMs.

KW - cs.CL

M3 - Preprint

BT - Leveraging Open-Source Large Language Models for Native Language Identification

ER -