Seminars
Abstract
Large Language Models (LLMs) have become ubiquitous across various domains, transforming the way we interact with information and conduct research. While the proliferation of LLMs has enhanced numerous applications, a significant number of high-performing models remain proprietary, impeding the progress of scientific exploration. LLMs are also susceptible to hallucinations, generating seemingly credible yet factually inaccurate information that can impact their broad acceptance and integration. In this seminar, I will commence by introducing one of our open-sourced XGen LLMs. I will delve into its pre-training process and present its results on standard benchmarks. Subsequently, I will discuss our work involving reasoning with LLMs, democratizing them for low-resource languages, and distilling knowledge from a larger (175B) proprietary LLM to a smaller (7B) model in a personalized manner. Finally, I will conclude by addressing some limitations of LLMs, emphasizing that scaling alone might not suffice as a solution and that new innovations are needed to tackle these challenges.
Bio
Dr. Shafiq Joty (https://raihanjoty.github.io/) is currently a Research Director at Salesforce Research (Palo Alto, USA), where he oversees the NLP group’s efforts in large language modeling (LLM) and generative AI. He also holds the position of a tenured Associate Professor (currently on leave) in the School of Computer Science and Engineering (SCSE) at NTU, Singapore. He was a founding manager of the Salesforce Research Asia (Singapore) lab. His research has contributed to over 30+ patents and 140+ papers in top-tier NLP and ML conferences and journals. He has served as the Program Chair of SIGDIAL-2023, as a member of the best paper award committees for ICLR-23 and NAACL-22, and in the capacity of a (senior) area chair for many of the leading NLP and ML conferences.
Abstract
Large language models (LMs) are pretrained on diverse data sources—news, discussion forums, books, online encyclopedias. A significant portion of this data includes facts and opinions which, on one hand, celebrate democracy and diversity of ideas, and on the other hand are inherently socially biased. In this talk. I’ll present our recent work proposing new methods to (1) measure media biases in LMs trained on such corpora, along the social and economic axes, and (2) measure the fairness of downstream NLP models trained on top of politically biased LMs. In this study, we find that pretrained LMs do have political leanings which reinforce the polarization present in pretraining corpora, propagating social biases into social-oriented tasks such as hate speech and misinformation detection. In the second part of my talk, I’ll discuss ideas on mitigating LMs’ unfairness. Rather than debiasing models—which, our work shows, is impossible—we propose to understand, calibrate, and better control for their social impacts using modular methods in which diverse LMs collaborate at inference time.
Bio
Yulia Tsvetkov is an associate professor at the Paul G. Allen School of Computer Science & Engineering at University of Washington. Her research group works on fundamental advancements to large language models, multilingual NLP, and AI ethics. This research is motivated by a unified goal: to extend the capabilities of human language technology beyond individual populations and across language boundaries, thereby making NLP tools available to all users. Prior to joining UW, Yulia was an assistant professor at Carnegie Mellon University and a postdoc at Stanford. Yulia is a recipient of NSF CAREER, Sloan Fellowship, Okawa Research award, and several paper awards and runner-ups at NLP, ML, and CSS conferences.
