Contents Science Lab

Nagoya University Graduate School of Informatics
Back to project list

Project: Image captioning considering imageability

In this research, we aim to generate image captions tailored to the actual usage of image captions. To achieve this, we consider psycholinguistic measurements during the generation of captions.

For example, in order for visually impaired people to understand the image content, captions that describe the image content in as much detail as possible are preferred. On the other hand, for images in news articles, captions that include the content of the news article are preferred to the description of the image content.

There are various situations in image captions used in the real world, and the desired properties differ depending on each application. Aiming to generate captions according to these, Ide Laboratory is working on image captioning that freely adjusts the details of the explanation of the image contents.

Therefore, consider “Imageability”, which is a measurement of showing the ease of imagining a word’s content. Including this into caption generation, it is possible to tailor the caption to an intended degree of visualness. In the resulting model, if you input an image into the caption model and specify a low value, a concise caption will be generated. In contrast, if you specify a high value, a caption that describes the image content in detail will be generated.

Featured image

Project members

Avatar image
Kazuki Umemura

M2 student

Avatar image
Dr. Marc A. Kastner

Cooperative Research Fellow (National Institute of Informatics)

Recent publications

    Tell as you imagine: Sentence imageability-aware image captioning
    Kazuki Umemura, Marc A. Kastner, Ichiro Ide, Yasutomo Kawanishi, Takatsugu Hirayama, Keisuke Doman, Daisuke Deguchi, Hiroshi Murase
    27th Int. Conf. on MultiMedia Modeling (MMM2021), Prague, Czech Republic, To be published in January 2021.
    TBA (in Japanese)
    Kazuki Umemura, Marc Aurel Kastner, Ichiro Ide, Yasutomo Kawanishi, Takatsugu Hirayama, Keisuke Doman, Daisuke Deguchi, Hiroshi Murase
    Meeting on Image Recognition and Understanding (MIRU) 2020, no.IS3-2-1, Online, August 2020.
    A study on image captioning considering its imageability (in Japanese)
    Kazuki Umemura, Marc Aurel Kastner, Ichiro Ide, Yasutomo Kawanishi, Takatsugu Hirayama, Keisuke Doman, Daisuke Deguchi, Hiroshi Murase
    IEICE Tech. Rep. Media Experience and Virtual Environment, MVE2019-69; MVE Award, Online, March 2020.
    Estimating the imageability of a sentence for image caption evaluation
    Kazuki Umemura, Marc Aurel Kastner, Ichiro Ide, Yasutomo Kawanishi, Daisuke Deguchi, Hiroshi Murase
    Japan-Taiwan Joint Workshop on Multimedia and HCI, National Cheng-Kung University (Tainan, Taiwan), April 2019.
    TBA (in Japanese)
    Kazuki Umemura, Marc Aurel Kastner, Ichiro Ide, Yasutomo Kawanishi, Takatsugu Hirayama, Keisuke Doman, Daisuke Deguchi, Hiroshi Murase
    Proc. 25th ANLP Annual Meeting, no.A4-9; pp.755-758, Nagoya Univ, March 2019.


Last updated: 2020-11-27 13:48:05.686858303 +0900 JST m=+0.984544686.