Inferential Machine Learning: Towards Human-collaborative Foundation Models

Neural network driven applications like ChatGPT suffer from hallucinations where they confidently provide inaccurate information. A funda- mental reason for this inaccuracy is the feed-forward nature of inductive decisions taken by neural networks. Such decisions are a result of training schemes that do not allow networks to deviate from and creatively abduce reasons at inference. With the advent of foundation models that are adapted across applications and data, humans can directly intervene and prompt vision-language foundation models. However, without understanding the operational limits of the underlying networks, human interventions often lead to unfair, inaccurate, hallucinated and unintelligible outputs. These outputs undermine the trust in foundation models, thereby causing roadblocks to their adoption in everyday lives. In this talk, we review systematic ways to analyze and understand human interventions in neural network functionality at inference. Specifically, we show that a human-AI collaborative environment via inferential machine learning techniques is a promising endeavor.