Contents Science Lab

Nagoya University Graduate School of Informatics
Back to project list

Project: Mimetic words

In Japanese, human gait (walking appearance) is often expressed by using mimetic words (or “onomatopoeia”) such as “sutasuta” and “yoroyoro”, and many more.

In our research, we aim to build a framework that describes gaits in mimetic words and conversely generates gaits from them. It is desirable that AI which interacts with and assists humans can input and output information that is easy for humans to understand. For that purpose, human senses, textures, etc. must be better analyzed and understood by the machine. It is known that human senses have a co-occurrence relationship between multiple modalities, as represented by the Bouba-Kiki effect (the relationship between phonemes and shapes of objects). We try to model such a relationship for human gaits in order to better describe them like a human would do.

Data-driven modeling of these relationships between modality using an informatics approach not only helps to elucidate the cross-modality of human senses, but also helps computers understand human senses.

We published a dataset for this research: http://www.murase.is.i.nagoya-u.ac.jp/~katoh/hoyo.html

Featured image
Featured image

Project members

Avatar image
Hirotaka Kato

D3 student

Recent publications

    More-natural mimetic words generation for fine-grained gait description
    Hirotaka Kato, Takatsugu Hirayama, Ichiro Ide, Keisuke Doman, Yasutomo Kawanishi, Daisuke Deguchi, Hiroshi Murase
    MultiMedia Modeling -26th Int. Conf., MMM 2020, Daejeon, South Korea, Jan. 5-8, 2020, Procs., Part II, W.-H. Cheng, J. Kim, W.-T. Chu, P. Cui, J.-W. Choi, M.-C. Hu, W. De Neve eds., Lecture Notes in Computer Science, vol.11962, pp.214-225, Daejeon Convention Center(大韓民国大田広域市), January 2020.
    HOYO: A gait dataset annotated with mimetic words (in Japanese)
    Hirotaka Kato, Takatsugu Hirayama, Keisuke Doman, Yasutomo Kawanishi, Ichiro Ide, Daisuke Deguchi, Hiroshi Murase
    IEICE Tech. Rep. Media Experience and Virtual Environment, MVE2018-21, Osaka Inst. of Tech., Umeda campus, September 2018.
    TBA (in Japanese)
    Hirotaka Kato, Takatsugu Hirayama, Keisuke Doman, Yasutomo Kawanishi, Ichiro Ide, Daisuke Deguchi, Hiroshi Murase
    Meeting on Image Recognition and Understanding (MIRU) 2018, no.PS3-44, Sapporo Convention Center, August 2018.
    Describing gaits by onomatopoeias with sound symbolism (in Japanese)
    Hirotaka Kato, Takatsugu Hirayama, Keisuke Doman, Ichiro Ide, Yasutomo Kawanishi, Daisuke Deguchi, Hiroshi Murase
    Trans. JSAI, vol.33, no.4, pp.B-HC2_1-9, July 2018.
    Toward describing human gaits by onomatopoeias
    Hirotaka Kato, Takatsugu Hirayama, Yasutomo Kawanishi, Keisuke Doman, Ichiro Ide, Daisuke Deguchi, Hiroshi Murase
    Proc. 7th IEEE Int. Workshop on Analysis and Modeling of Faces and Gestures (AMFG2017) in Conjunction with ICCV2017, pp.1573-1580, Palazzo del Cinema -Venice Convention Center (Venezia, Veneto, Italy), November 2017.
    A study of zero-shot gaits description by onomatopoeia based on the relationship between phoneme and body-parts movement (in Japanese)
    Hirotaka Kato, Takatsugu Hirayama, Yasutomo Kawanishi, Ichiro Ide, Daisuke Deguchi, Hiroshi Murase
    Proc. 2017 Electric / Electronic / Information Engineering Related Society Tokai Sectors Joint Convention, B2-6, Nagoya Univ., September 2017.
    Toward description of gaits by onomatopoeia based on the relationship between phoneme and body-parts movement (in Japanese)
    Hirotaka Kato, Takatsugu Hirayama, Yasutomo Kawanishi, Keisuke Doman, Ichiro Ide, Daisuke Deguchi, Hiroshi Murase
    IEICE Tech. Rep. Pattern Recognition and Media Understanding, PRMU2017-25, Tohoku University Katahira Campus, June 2017.


Last updated: 2020-11-27 13:48:05.739916527 +0900 JST m=+1.037602912.