Benno Krojer

I'm a 3rd-year PhD student in AI/NLP at MILA & McGill in Montreal with Siva Reddy, supported by the Vanier Scholarship. For the summer of 2024, I am also a research intern at FAIR, Meta's fundamental AI research department. While I am curious about all forms of language grounding (for examples, see Experience Grounds Language paper), I have worked on vision-and-language reasoning in the past two years: How can models reason with pragmatic language and visual nuances? Can we repurpose Stable Diffusion for challenging discriminative tasks? I also want to incorporate questions around ethics and society into my research.
Pronouns: he/him

My research interests as of Dec 2023 with papers (sometimes mine) to make it more concrete:

Vision-and-Language (VL) reasoning and compositionality
(Winoground, ImageCoDe, Iterated Learning)
Text-to-image models (Stable Diffusion): leveraging and evaluating their representations
(DiffusionITM, DSG, T2I-CompBench)
Relationship between generation and understanding?
(DiffusionITM, The Generative AI Paradox: What It Can Create, It May Not Understand,)
Creatively stress-testing models
(Winoground, ImageCoDe, Do Androids Laugh at Electric Sheep?)
Video + Language
(ImageCoDe, MERLOT Reserve)
VL + ethics/society
(Stable Bias, MaRVL)
VL + creativity/humor/storytelling
(ViPE: Visualize Pretty-much Everything, Do Androids Laugh at Electric Sheep?)

Aside from research, I have organized the NewInML workshop at NeurIPS 2023 and founded a reading group on language grounding at Mila. In my undergrad, I also founded a philosophy society which is still thriving to this day! I strongly believe these kinds of activities next to publishing papers are crucial to a healthy and vibrant research community.

Before coming to Mila, I graduated from LMU Munich in computational linguistics where I worked on relational reasoning and machine translation. I also had great experiences at early-stage startups during and after my undergrad!

In my free time, I play Ultimate Frisbee competetively and have coached juniors and a mixed-gender team. With the PhD, I don't play on the highest level anymore but look back on great memories from U24 Worlds with Team Germany and several European Club Championships.

Let's chat about research and life: benno.krojer@gmail.com

Are Diffusion Models Vision-And-Language Reasoners?
Benno Krojer, Elinor Poole-Dayan, Vikram Voleti, Christopher Pal, Siva Reddy
NeurIPS'23 | Conference on Neural Information Processing Systems

Image Retrieval from Contextual Descriptions
Benno Krojer, Vaibhav Adlakha, Vibhav Vineet, Yash Goyal, Edoardo Ponti, Siva Reddy
ACL'22 | Association for Computational Linguistics

Improving Automatic VQA Evaluation Using Large Language Models
Oscar Manas, Benno Krojer, Aishwarya Agrawal
submitted to AAAI'23 | AAAI Conference on Artificial Intelligence

Pragmatic Inference with a CLIP Listener for Contrastive Captioning
Jiefu Ou, Benno Krojer, Daniel Fried
ACL Findings'23 | Findings of the Association for Computational Linguistics

Are Pretrained Language Models Symbolic Reasoners Over Knowledge?
Nora Kassner*, Benno Krojer*, Hinrich Schütze
CoNLL'20 | Conference on Computational Natural Language Learning
(* = Equal Contribution)

ContraCAT: Contrastive Coreference Analytical Templates for Machine Translation
Dario Stojanovski*, Benno Krojer*, Denis Peskov*, Alexander Fraser
COLING'20 | Conference on Computational Linguistics
(* = Equal Contribution)

Things I like
Jan 2nd, 2024

Elevator Pitch & ELI5 of my first PhD project: Image Retrieval from Contextual Descriptions
May 30th, 2022
Explaining my research to someone outside the field.

What made me do a PhD far away?
April 28th, 2021
Sometimes people ask me why I left Europe when there is good opportunities there, too.

[DRAFT]: Defining the most basic concepts in Language Grounding
Jan 22nd, 2021
We rarely think about what the most basic words mean.

Attending my first conference! ACL 2020 from an undergrad's perspective
Jul 2nd, 2020
Networking, learning about language grounding and mentally prepping for the PhD.

Why the current Corona situation makes memories blurry
May 8th, 2020
Applying knowledge from reading about neuroscience

Moral Pluralism through the lense of optimization
Apr 29th, 2020
Some reflections after a philosophy society meeting

Thank you to Sebastian Santy for the awesome website template!