In plain words: Tested Gemini's everyday common sense on 12 reasoning tasks, mostly text-only plus one with images, alongside four text-only and two vision models. Gemini matched the GPT models, contradicting an earlier test that used just one task.
Abstract
The burgeoning interest in Multimodal Large Language Models (MLLMs), such as OpenAI's GPT-4V(ision), has significantly impacted both academic and industrial realms. These models enhance Large Language Models (LLMs) with advanced visual understanding capabilities, facilitating their application in a variety of multimodal tasks. Recently, Google introduced Gemini, a cutting-edge MLLM designed specifically for multimodal integration. Despite its advancements, preliminary benchmarks indicate that Gemini lags behind GPT models in commonsense reasoning tasks. However, this assessment, based on a limited dataset (i.e., HellaSWAG), does not fully capture Gemini's authentic commonsense reasoning potential. To address this gap, our study undertakes a thorough evaluation of Gemini's performance in complex reasoning tasks that necessitate the integration of commonsense knowledge across modalities. We carry out a comprehensive analysis of 12 commonsense reasoning datasets, ranging from general to domain-specific tasks. This includes 11 datasets focused solely on language, as well as one that incorporates multimodal elements. Our experiments across four LLMs and two MLLMs demonstrate Gemini's competitive commonsense reasoning capabilities. Additionally, we identify common challenges faced by current LLMs and MLLMs in addressing commonsense problems, underscoring the need for further advancements in enhancing the commonsense reasoning abilities of these models.
Yuqing Wang, Yun Zhao
arXiv:2312.17661 · cs.CL, cs.AI, cs.CV · submitted Dec 29, 2023
abstract · pdf · html · Data and results are available at: https://github.com/EternityYW/Gemini-Commonsense-Evaluation/
To get to this, I asked him to predict the return value of some simple python function (for """def func(a,b): return a*3 + 2*b)""" what's func(2)).
without requiring the explanation, the answers were not only wrong but also changed between attempts for the same input.
Asking the model to provide an explanation, however, resulted in it giving the correct answer each time.
This method still allows for improvement in the quality of answers in more complex problems, although at a bit lower correctness rate.