Dr. Vijaya Kolachalama
Associate Professor
Department of Medicine & Department of Computing and Data Science
Boston University
Email: vkola@bu.edu
Website: https://vkola-lab.github.io
Dr. Ebrahim Bagheri
Associate Professor
Canada Research Chair in Software and Semantic Computing
NSERC/WL Industrial Research Chair in Social Media Analytics
Ryerson University, Canada
Email: bagheri@ryerson.ca
Website:https://www.torontomu.ca
Title:Neural Information Retrieval: A Doubtful Advantage
Abstract:
There are >5 billion users on the Internet, 98% of whom collectively submit >10 billion queries per day to search engines – powered by Information Retrieval (IR) methods. As such, IR methods adopted by search engines play a major role in how people think, perceive and act, and as a result have the potential to shape each individual’s personal beliefs and consequently impact the social fabric. The development of robust and effective IR methods is likely to have immediate impact on individuals and broader populations with important socioeconomic outcomes. As Large Language Models (LLMs) grow in complexity, many IR tasks, which were considered stubborn problems, have now become increasingly softer to address. The tangible evidence of the impact of LLMs can be observed on the standard MSMARCO passage retrieval benchmark, which consists of over 10M queries, passages, and relevance labels where the mean reciprocal rank measure has, as a result, increased from ~0.19 to >0.45 – a 2.3x increase in performance. However, the substantial increase in performance is due to a strong positive impact on a subset of the query space, where such queries receive a near perfect treatment (i.e., reciprocal rank of ~1), but a notable subset of the query space remains completely unsatisfied. For example, on the MSMARCO development set, at least 30% of queries receive a reciprocal rank of 0 by state-of-the-art rankers (completely unaddressed) and the reciprocal rank of one of the best available rankers, namely TCT-ColBERT, is only 0.04 on 50% of the queries in this set. These observations can be explained by the preferential bias of LLMs towards certain subspaces. The objective of this talk will be to explore how LLM-based retrieval methods can potentially break-free from already encoded preferential biases in order to allow them to show robust and effective retrieval performance across the whole spectrum of user queries.
Dr. Mohammad Sabokrou
Okinawa Institute of Science and Technology Graduate University
Email: mohammad.sabokrou@oist.jp
Website:https://sabokrou
Title: Trustworthy Machine Learning: Exploring Reliability in AI
