Contents Science Lab

Project: Mimetic words

In Japanese, human gait (walking appearance) is often expressed by using mimetic words (or “onomatopoeia”) such as “sutasuta” and “yoroyoro”, and many more.

In our research, we aim to build a framework that describes gaits in mimetic words and conversely generates gaits from them. It is desirable that AI which interacts with and assists humans can input and output information that is easy for humans to understand. For that purpose, human senses, textures, etc. must be better analyzed and understood by the machine. It is known that human senses have a co-occurrence relationship between multiple modalities, as represented by the Bouba-Kiki effect (the relationship between phonemes and shapes of objects). We try to model such a relationship for human gaits in order to better describe them like a human would do.

Data-driven modeling of these relationships between modality using an informatics approach not only helps to elucidate the cross-modality of human senses, but also helps computers understand human senses.

We published a dataset for this research: http://www.cs.is.i.nagoya-u.ac.jp/opensource/hoyo/

Main project members

Dr. Hirotaka Kato

Post-doctor Researcher (Institutes of Innovation for Future Society)

Recent publications

Gaits generation from a mimetic word based on sound symbolism (in Japanese)

Hirotaka Kato, Takatsugu Hirayama, Keisuke Doman, Ichiro Ide, Yasutomo Kawanishi, Daisuke Deguchi, Hiroshi Murase
Trans. JSAI, vol.36, no.5, pp.D-KC7_1-10, September 2021.

Introduction to the Contents Science Lab

Ichiro Ide, Keisuke Doman, Takahiro Komamizu, Marc A. Kastner, Hirotaka Kato, Chihaya Matsuhira
Joint Int. Workshop on Smart Systems, Online, July 2021.

More-natural mimetic words generation for fine-grained gait description

Hirotaka Kato, Takatsugu Hirayama, Ichiro Ide, Keisuke Doman, Yasutomo Kawanishi, Daisuke Deguchi, Hiroshi Murase
MultiMedia Modeling -26th Int. Conf., MMM 2020, Daejeon, South Korea, Jan. 5-8, 2020, Procs., Part II, W.-H. Cheng, J. Kim, W.-T. Chu, P. Cui, J.-W. Choi, M.-C. Hu, W. De Neve eds., Lecture Notes in Computer Science, vol.11962, pp.214-225, Daejeon Convention Center（大韓民国大田広域市）, January 2020.

HOYO: A gait dataset annotated with mimetic words (in Japanese)

Hirotaka Kato, Takatsugu Hirayama, Keisuke Doman, Yasutomo Kawanishi, Ichiro Ide, Daisuke Deguchi, Hiroshi Murase
IEICE Tech. Rep. Media Experience and Virtual Environment, MVE2018-21, Osaka Inst. of Tech., Umeda campus, September 2018.

TBA (in Japanese)

Hirotaka Kato, Takatsugu Hirayama, Keisuke Doman, Yasutomo Kawanishi, Ichiro Ide, Daisuke Deguchi, Hiroshi Murase
Meeting on Image Recognition and Understanding (MIRU) 2018, no.PS3-44, Sapporo Convention Center, August 2018.

Describing gaits by onomatopoeias with sound symbolism (in Japanese)

Hirotaka Kato, Takatsugu Hirayama, Keisuke Doman, Ichiro Ide, Yasutomo Kawanishi, Daisuke Deguchi, Hiroshi Murase
Trans. JSAI, vol.33, no.4, pp.B-HC2_1-9, July 2018.

Toward describing human gaits by onomatopoeias

Hirotaka Kato, Takatsugu Hirayama, Yasutomo Kawanishi, Keisuke Doman, Ichiro Ide, Daisuke Deguchi, Hiroshi Murase
Proc. 7th IEEE Int. Workshop on Analysis and Modeling of Faces and Gestures (AMFG2017) in Conjunction with ICCV2017, pp.1573-1580, Palazzo del Cinema -Venice Convention Center (Venezia, Veneto, Italy), November 2017.

A study of zero-shot gaits description by onomatopoeia based on the relationship between phoneme and body-parts movement (in Japanese)

Hirotaka Kato, Takatsugu Hirayama, Yasutomo Kawanishi, Ichiro Ide, Daisuke Deguchi, Hiroshi Murase
Proc. 2017 Electric / Electronic / Information Engineering Related Society Tokai Sectors Joint Convention, B2-6, Nagoya Univ., September 2017.

Toward description of gaits by onomatopoeia based on the relationship between phoneme and body-parts movement (in Japanese)

Hirotaka Kato, Takatsugu Hirayama, Yasutomo Kawanishi, Keisuke Doman, Ichiro Ide, Daisuke Deguchi, Hiroshi Murase
IEICE Tech. Rep. Pattern Recognition and Media Understanding, PRMU2017-25, Tohoku University Katahira Campus, June 2017.