CV — Yuto Shibata

Research Interests

Computer Vision • Physics-Based Human Motion Modeling • Multi-Agent Reinforcement Learning • Human-Robot Interaction • Active Human Sensing

Education

Keio University

Sep 2024 – Present

Ph.D. in Integrated Design Engineering

Tokyo, Japan

Advisor: Prof. Yoshimitsu Aoki & Dr. Mariko Isogawa
Research: Physics-grounded human motion modeling, active human sensing (CVPR 2023, BMVC 2024)

Carnegie Mellon University — Visiting Researcher

Jun 2025 – Jun 2026

Machine Learning Department, School of Computer Science

Pittsburgh, PA, USA

Host: Prof. Katerina Fragkiadaki
Conducted research on physics-based human-human interaction control via multi-agent reinforcement learning
Developed AssistMimic, a framework for learning assistive human behavior through physics simulation, accepted at CVPR 2026

Keio University

Apr 2023 – Sep 2024

M.Eng. in Integrated Design Engineering

Tokyo, Japan

Completed master's program in 1.5 years (accelerated to PhD)

University of Edinburgh — Exchange Program

Sep 2021 – Jun 2022

School of Informatics, Computer Science

Edinburgh, UK

Keio University

Apr 2018 – Mar 2023

B.Eng. in Electronics and Electrical Engineering

Tokyo, Japan

Selected Publications

[1] Y. Shibata*, K. Amaya*, K. Yamazaki, L. Jayanti, Y. Aoki, M. Isogawa, K. Fragkiadaki. "VLM-Guided Contact Retargeting and Generative Partner Modeling for Humanoid-Human Interaction." CoRL 2026 *Equal contribution

Project Code

[2] Y. Shibata, K. Yamazaki, L. Jayanti, Y. Aoki, M. Isogawa, K. Fragkiadaki. "Learning to Assist: Physics-Grounded Human-Human Control via Multi-Agent Reinforcement Learning." CVPR 2026

Project arXiv Code

[3] Y. Oumi, Y. Shibata, G. Irie, A. Kimura, Y. Aoki, M. Isogawa. "Sound-based Multi-Person 3D Pose Estimation." ECCV 2026

Project arXiv Code Dataset

[4] Y. Shibata, Y. Oumi, G. Irie, A. Kimura, Y. Aoki, M. Isogawa. "BGM2Pose: Active 3D Human Pose Estimation with Non-Stationary Sounds." IEEE Open Journal of Signal Processing 2026

Project Paper arXiv Code

[5] T. Manabe, Y. Shibata, S. Takamichi, Y. Aoki. "Sign-to-Speech Prosody Transfer via Sign Reconstruction-based GAN." ICPR 2026 — Best Scientific Paper Award

Project arXiv Code

[6] Y. Shibata, M. Ishige, H. Murakami, Y. Kawahara. "Occlusion-Resilient UWB Localization Via Ceiling Reflection With Floor Anchor Installation." ACM SenSys 2025 (Poster)

Paper

[7] Y. Shibata, K. Tanaka, Y. Bando, H. Kataoka, Y. Aoki. "Formula-Supervised Sound Event Detection: Pre-Training Without Real Data." IEEE ICASSP 2025

Project Paper Code

[8] Y. Shibata, Y. Kudo, Y. Sugawara. "Anatomical Three-Dimensional Style Transfer Enabling Efficient Federated Learning with Extremely Low Communication Costs." NeurIPS 2024 Workshop (AIM-FM)

arXiv

[9] Y. Oumi, Y. Shibata, G. Irie, A. Kimura, Y. Aoki, M. Isogawa. "Acoustic-based 3D Human Pose Estimation Robust to Human Position." BMVC 2024

arXiv

[10] Y. Shibata, Y. Kawashima, M. Isogawa, G. Irie, A. Kimura, Y. Aoki. "Listening Human Behavior: 3D Human Pose Estimation With Acoustic Signals." CVPR 2023

Project Paper Code

Honors & Awards

Best Scientific Paper Award — ICPR 2026

2026

MIRU Excellent Paper Award — MIRU 2025

Jul 2025

Ranked 2nd out of 726 presentations. "Universal Human Pose Representation for Multi-Modal Active Sensing"

MIRU Interactive Presentation Award — MIRU 2024

Aug 2024

Awarded to the author of a paper with an excellent interactive presentation.

MIRU Student Encouragement Award — MIRU 2024

Aug 2024

Awarded to presenters of outstanding oral research presentations by student first authors.

IPSJ Yamashita SIG Research Award

Jul 2023

Awarded to presenters of particularly outstanding papers from workshops and symposia.

Frontier Award — MIRU 2022

Jul 2022

Awarded to the most pioneering initiative in research with an outstanding presentation.

Research Experience

Carnegie Mellon University

Jun 2025 – Jun 2026

Visiting Researcher, Machine Learning Department

Pittsburgh, PA, USA

Host: Prof. Katerina Fragkiadaki
Research on physics-based human-human interaction control via multi-agent reinforcement learning
Developed AssistMimic for physics-based assistive human–human interaction, accepted at CVPR 2026
Developed REACT for contact-rich humanoid–human interaction, accepted at CoRL 2026

Aoki Media Laboratory, Keio University

2021 – Present

Graduate Student

Tokyo, Japan

Pioneered acoustic-based 3D human pose estimation using reflected chirp signals — accepted at CVPR 2023
Developed active sensing methods combining audio and multimodal signals for human understanding

National Institute of Advanced Industrial Science and Technology (AIST)

Jul 2023 – Sep 2024

Research Assistant

Tsukuba, Japan

Conducted research on formula-supervised sound event detection pre-training
Work accepted at IEEE ICASSP 2025

Industry Experience

CyberAgent, Inc.

Feb 2025 – Jun 2025

Research Intern / Part-time Researcher

Tokyo, Japan

Conducted research on UWB localization resilient to occlusion via ceiling reflection
Work accepted as a poster at ACM SenSys 2025

Preferred Networks, Inc.

Aug 2023 – Mar 2024

Research Intern

Tokyo, Japan

Developed anatomical 3D style transfer for communication-efficient medical federated learning
Work accepted at NeurIPS 2024 Workshop (AIM-FM)

Recruit Co., Ltd.

Feb 2021 – Mar 2021, Aug 2022 – Mar 2023

ML Engineer / Research Intern

Tokyo, Japan

Applied transformer architectures for large-scale recommendation systems

Tenchijin, Inc.

Aug 2022 – Present

Data Scientist

Tokyo, Japan

Analyzed geospatial (GIS) data for satellite-based applications

Invited Talks

MIRU 2023 (Jul 2023) — "Listening Human Behavior: 3D Human Pose Estimation With Acoustic Signals"

Technical Skills

Programming

Python

ML / DL

PyTorch, Isaac Gym / IsaacLab, Hugging Face, scikit-learn

Research

Reinforcement Learning, Physics Simulation, Computer Vision, Human Pose Estimation, Physics-Based Human(oid) Tracking, Signal Processing

Tools

Git, Docker, Linux, LaTeX, Weights & Biases

Languages

Japanese (native), English (fluent — exchange at Univ. of Edinburgh, research at CMU)

Competitions

Kaggle — Multiple silver medals across diverse domains [Profile]

OSIC Pulmonary Fibrosis Progression — 20th / 2,097 teams (Silver)
Cassava Leaf Disease Classification — 54th / 3,900 teams (Silver)
CommonLit Readability Prize — 139th / 3,633 teams (Silver)