Health & Environmental Research Online (HERO)


Print Feedback Export to File
7204235 
Journal Article 
Incorporating Dialectal Variability for Socially Equitable Language Identification 
Jurgens, D; Tsvetkov, Y; Jurafsky, Dan; , 
2017 
ASSOC COMPUTATIONAL LINGUISTICS-ACL 
STROUDSBURG 
51-57 
WOS:000493992300009 
Language identification (LID) is a critical first step for processing multilingual text. Yet most LID systems are not designed to handle the linguistic diversity of global platforms like Twitter, where local dialects and rampant code-switching lead language classifiers to systematically miss minority dialect speakers and multilingual speakers. We propose a new dataset and a character-based sequence-to-sequence model for LID designed to support dialectal and multilingual language varieties. Our model achieves state-of-the-art performance on multiple LID benchmarks. Furthermore, in a case study using Twitter for health tracking, our method substantially increases the availability of texts written by underrepresented populations, enabling the development of "socially inclusive" NLP tools. 
Barzilay, R; Kan, MY; 
55th Annual Meeting of the Association-for-Computational-Linguistics (ACL) 
Vancouver, CANADA