Curieux.JY
  • JungYeon Lee
  • Post
  • ๐Ÿ•ธ๏ธ Graph
  • Lecture
  • Note

On this page

  • ๐Ÿ” Ping Review
  • ๐Ÿ”” Ring Review
    • ํ•œ ์ค„๋กœ ์‹œ์ž‘ํ•˜๋ฉด
    • ๋ฐฐ๊ฒฝ: ์™œ ์‚ฌ๋žŒ ์‹œ์—ฐ์ด ๋ฐ”๋กœ ๋กœ๋ด‡ ๋ฐ์ดํ„ฐ๊ฐ€ ๋˜์ง€ ์•Š๋Š”๊ฐ€ (ยง1, ยง2)
    • ๋ฐฉ๋ฒ• ์ƒ์„ธ (ยง3)
      • 3.1 Motion capture์™€ ๋ฐ์ดํ„ฐ ์ฒ˜๋ฆฌ
      • 3.2 Physics refinement: kinematic ๊ธฐ๋ก์„ ์ˆ˜์„ ํ•˜๋Š” policy
      • 3.3 Contact-anchored retargeting
      • 3.4 Residual policy learning (๋กœ๋ด‡ ์ชฝ)
      • 3.5 ํ•˜๋“œ์›จ์–ด ๋ฐฐํฌ
      • ๋…ผ๋ฌธ์ด ์ˆซ์ž๋กœ ๋ฐํžŒ ์„ค์ •๊ณผ ๋ฐํžˆ์ง€ ์•Š์€ ์„ค์ •
    • ์ง๊ด€: ์™œ ์ ‘์ด‰ ๊ตฌ์กฐ๊ฐ€ ์˜ฎ๊ฒจ์ง€๊ณ , ์™œ policy ํ•˜๋‚˜๋กœ ์ถฉ๋ถ„ํ•œ๊ฐ€
    • ์‹คํ—˜ (ยง4)
      • 4.1 ๊ทœ๋ชจ์—์„œ์˜ ๋ฌผ๋ฆฌ ์ผ๊ด€์„ฑ
      • 4.2 ์† ๋ชจ์–‘์„ ๋„˜๋Š” ์ ‘์ด‰ ๊ธฐ๋ฐ˜ ์ „์ด
      • 4.3 ๋ฐ์ดํ„ฐ์—์„œ ๋ฐฐํฌ๊นŒ์ง€
    • ์ธก์ •๋œ ๊ฒƒ๊ณผ ์„ ์–ธ๋œ ๊ฒƒ
    • ๋น„ํŒ์ ์œผ๋กœ ๋ณด๋ฉด
      • ๊ฐ•์ 
      • ์•ฝ์ ยทํ•œ๊ณ„
      • ๊ณต๊ฐœ ์ƒํƒœ
    • ๊ด€๋ จ ์—ฐ๊ตฌ์™€์˜ ์ž๋ฆฌ ๋งค๊น€
    • ์š”์•ฝ
    • ์ด ๊ธ€์— ๋Œ€ํ•ด

๐Ÿ“ƒContact-Anchored Retargeting ๋ฆฌ๋ทฐ

dexterity
retargeting
residual
cross-embodiment
real2sim
Dexterous Robot Manipulation from Human Demonstrations via Contact-Anchored Retargeting and Residual Policy Learning
Published

November 10, 2026

  • Paper Link (arXiv:2609.24093v2)

  • Code Link (GitHub): README์™€ ๋ฐ๋ชจ ์˜์ƒ๋งŒ ์žˆ๊ณ  ์ฝ”๋“œ๋Š” ์•„์ง ์—†๋‹ค(โ€œCode release coming soon.โ€).

  • Zihao Yang*, Chengyuan Liu*, Yu Zhou, Runze Lv, Tianyu Cui, Sheng Yi, Haohua Zhu, Irvine Luโ€ , JieQ Sunโ€  (*๊ณต๋™ 1์ €์ž, โ€ ๊ต์‹ ์ €์ž)

  • DexGEM Lab ยท Shanghai Jiao Tong University ยท Tongji University ยท DexRobot Co. Ltd.

  • arXiv preprint, 2026 (v1 2026-09-21, v2 2026-11-03, ํ•™ํšŒ ์ฑ„ํƒ ์ •๋ณด ์—†์Œ)

  1. ๐Ÿ’ก ์‚ฌ๋žŒ ์† mocap์—๋Š” grasp์˜ ์„ฑํŒจ๋ฅผ ๊ฐ€๋ฅด๋Š” ์ ‘์ด‰๋ ฅ์ด ์—†๊ณ  ์‚ฌ๋žŒ ์†๊ณผ ๋กœ๋ด‡ ์†์€ ๋ชจ์–‘์ด ๋‹ค๋ฅด๋‹ค. ์ด ๋…ผ๋ฌธ์€ ์†์„ ๋ฐ”๊ฟ”๋„ ๋‚จ๋Š” ๊ฒƒ์€ ๊ด€์ ˆ ๊ถค์ ์ด ์•„๋‹ˆ๋ผ ์ ‘์ด‰ ๊ตฌ์กฐ(์–ด๋А ์†๊ฐ€๋ฝ ๋ถ€์œ„๊ฐ€ ๋ฌผ์ฒด์˜ ์–ด๋””์— ์–ด๋–ค ์ˆœ์„œ๋กœ ๋‹ฟ๋Š”๊ฐ€)๋ผ๊ณ  ๋ณด๊ณ , ๊ทธ๊ฒƒ์„ ์˜ฎ๊ธฐ๋Š” ํŒŒ์ดํ”„๋ผ์ธ์„ ๋งŒ๋“ ๋‹ค.
  2. โš™๏ธ ์‹œ๋ฎฌ๋ ˆ์ด์…˜ MANO ์†์„ ํ•˜๋‚˜์˜ residual RL policy๋กœ ๋‹ค์‹œ ์‹คํ–‰ํ•ด ์ ‘์ด‰๊ณผ ํž˜์„ ๋ณต์›ํ•˜๊ณ (physics refinement), ๋กœ๋ด‡ ์† ํ‘œ๋ฉด์ ์„ ๊ทธ ์ ‘์ด‰ ์œ„์น˜๋กœ ๋Œ์–ด๋‹น๊ธฐ๋Š” ์† ๋ชจ์–‘ ๋…๋ฆฝ์ ์ธ ์ตœ์ ํ™”๋กœ retargetingํ•œ ๋’ค, ๊ฐ™์€ residual ํ˜•์‹์˜ ๋‘ ๋ฒˆ์งธ policy๋กœ ๋กœ๋ด‡ ๊ตฌ๋™์— ๋งž๊ฒŒ ๋ณด์ •ํ•œ๋‹ค.
  3. ๐ŸŽฏ ํ•œ ์† ๊ถค์  25,454๊ฐœ์˜ ์„ฑ๊ณต๋ฅ ์ด 7.26% โ†’ 59.31%, ์–‘์† 25๊ณผ์ œ๊ฐ€ 16.0% โ†’ 62.4%(task-level)๋กœ ์˜ค๋ฅด๊ณ , ๊ฐ™์€ ์‚ฌ๋žŒ ๋ฐ์ดํ„ฐ๊ฐ€ ๋กœ๋ด‡ ์† 4์ข…์—์„œ ํ‰๊ท  17.1% โ†’ 79.5%๋กœ ์žฌ๊ตฌ์„ฑ๋˜๋ฉฐ, ์‹ค๊ธฐ ์–‘ํŒ” 4๊ณผ์ œ๋Š” 7/10ยท5/10ยท6/10ยท3/10์ด๋‹ค.

๐Ÿ” Ping Review

๐Ÿ” Ping โ€” A light tap on the surface. Get the gist in seconds.

์‚ฌ๋žŒ ์‹œ์—ฐ์œผ๋กœ ์†์žฌ์ฃผ๋ฅผ ๊ฐ€๋ฅด์น˜๋ ค๋ฉด ๋‘ ๊ฐ„๊ทน์„ ๋„˜์–ด์•ผ ํ•œ๋‹ค. embodiment gap์€ โ€œ๋ฌด์—‡์„ ์˜ฎ๊ธธ ๊ฒƒ์ธ๊ฐ€โ€์˜ ๋ฌธ์ œ๋‹ค. ๊ด€์ ˆ๊ฐ์ด๋‚˜ ์†๋ ์œ„์น˜๋ฅผ ๊ทธ๋Œ€๋กœ ๋งž์ถ”๋ฉด ์‚ฌ๋žŒ ์†๊ณผ ๋กœ๋ด‡ ์†์„ ํฌ๊ธฐ๋งŒ ๋‹ค๋ฅธ ๋ณต์‚ฌ๋ณธ์œผ๋กœ ์ทจ๊ธ‰ํ•˜๊ฒŒ ๋˜๊ณ , ์†๊ฐ€๋ฝ ์ˆ˜ยท๋น„์œจยท๊ด€์ ˆ ๋ฒ”์œ„๊ฐ€ ๋‹ค๋ฅด๋ฉด ๋ฌด๋„ˆ์ง„๋‹ค. physics gap์€ โ€œ๋ฌด์—‡์ด ๊ทธ ๋™์ž‘์„ ๋ฌผ๋ฆฌ์ ์œผ๋กœ ๊ฐ€๋Šฅํ•˜๊ฒŒ ํ•˜๋Š”๊ฐ€โ€์˜ ๋ฌธ์ œ๋‹ค. mocap ๊ธฐ๋ก์€ kinematic์ด๋ผ ํž˜์ด ์—†์œผ๋ฏ€๋กœ ์‹œ๋ฎฌ๋ ˆ์ด์…˜ ์žฌ๊ตฌ์„ฑ์€ ํ”ผํ•  ์ˆ˜ ์—†๋Š”๋ฐ, ๊ธฐ์กด real2sim2real ํŒŒ์ดํ”„๋ผ์ธ(DexMachina, DexMV, DemoBot ๋“ฑ)์€ ๊ณผ์ œ๋งˆ๋‹ค policyยท๋ณด์ƒ curriculumยทํ•™์Šต์„ ๋”ฐ๋กœ ๋งŒ๋“ ๋‹ค.

์ด ๋…ผ๋ฌธ์˜ ๋‹ต์€ ๋‘ ๊ด€์ฐฐ์ด๋‹ค. ์ฒซ์งธ, ์†์„ ๋ฐ”๊ฟ”๋„ ์‚ด์•„๋‚จ๋Š” ๊ฒƒ์€ ์ ‘์ด‰ ๊ตฌ์กฐ๋‹ค. ๊ทธ๋ž˜์„œ retargeting์˜ ๋ชฉ์ ํ•จ์ˆ˜๋ฅผ ๋กœ๋ด‡ ์† ํ‘œ๋ฉด์ ๊ณผ ์‚ฌ๋žŒ์ด ๋‚จ๊ธด ๋ฌผ์ฒด ํ‘œ๋ฉด ์ ‘์ด‰ ์œ„์น˜ ์‚ฌ์ด์˜ ๊ฑฐ๋ฆฌ๋กœ๋งŒ ์ •์˜ํ•˜๋ฉด ์† ๋ชจ์–‘๊ณผ ๋ฌด๊ด€ํ•œ ์‹์ด ๋œ๋‹ค. ๋‘˜์งธ, ๋ฌผ๋ฆฌ์  ์ผ๊ด€์„ฑ์€ ๊ณผ์ œ๋งˆ๋‹ค ์„ค๊ณ„ํ•  ํ•„์š”๊ฐ€ ์—†๋‹ค. ๋ชจ๋“  ์‹œ์—ฐ์— ๊ฑธ์ณ ํ•œ ๋ฒˆ ํ•™์Šตํ•œ residual RL policy ํ•˜๋‚˜๊ฐ€ kinematic ๊ธฐ๋ก์„ โ€œ์ˆ˜์„ โ€ํ•ด ์ ‘์ด‰์ด ์ฃผ์„๋œ ๊ถค์ ์œผ๋กœ ๋ฐ”๊ฟ€ ์ˆ˜ ์žˆ๊ณ , ๊ฐ™์€ residual ํ˜•์‹์ด retargeting ๋’ค์˜ ๋™์—ญํ•™ ๋ณด์ •์—๋„ ์“ฐ์ธ๋‹ค.


Fig. 1: ํŒŒ์ดํ”„๋ผ์ธ ๊ฐœ์š”. ์™ผ์ชฝ๋ถ€ํ„ฐ ์‚ฌ๋žŒ mocap(์‹ค์ œ) โ†’ MANO physics refinement(์‹œ๋ฎฌ๋ ˆ์ด์…˜, residual policy๊ฐ€ ์ ‘์ด‰๊ณผ ํž˜์„ ๋ณต์›) โ†’ contact-anchored retargeting(MANO ์ ‘์ด‰ ์œ„์น˜์— ๋กœ๋ด‡ ์† ํ‘œ๋ฉด์ ์„ ๋งž์ถ”๋Š” ์ตœ์ ํ™”) โ†’ robot residual learning(์‹œ๋ฎฌ๋ ˆ์ด์…˜) โ†’ ์‹ค๊ธฐ ๊ฒ€์ฆ(policy ๊ณ ์ •, ์‹ค๋กœ๋ด‡ ํ•™์Šต ๋ฐ์ดํ„ฐ ์—†์Œ). ๊ฐ€์šด๋ฐ ์•„๋ž˜ โ€œShared residual controlโ€์ด ๋‘ residual policy๊ฐ€ ๊ณต์œ ํ•˜๋Š” ์ œ์–ด ํ˜•์‹์ด๋‹ค.
๊ณต์‹ ๋ ˆํฌ README์˜ ๋ฐ๋ชจ ์˜์ƒ 4๊ฐœ(ํ•ด์ƒ๋„๋ฅผ ์ค„์—ฌ ์‹ค์—ˆ๋‹ค). ์™ผ์ชฝ ์œ„๋ถ€ํ„ฐ ์‚ฌ๋žŒ mocap, MANO physics replay, ์‹œ๋ฎฌ๋ ˆ์ด์…˜ ์† DexHand(contact-anchored retargeting), ์‹ค๊ธฐ ์–‘ํŒ” ์‹คํ–‰. README๊ฐ€ "bimanual bottle stacking"์ด๋ผ ๋ถ€๋ฅด๋Š” ๋งˆ์š”๋„ค์ฆˆ ๋ณ‘ ๊ณผ์ œ๋กœ, Fig. 1์— ๋‚˜์˜ค๋Š” ์žฅ๋ฉด๊ณผ ๊ฐ™์€ ๊ณ„์—ด์ด์ง€๋งŒ ๋…ผ๋ฌธ์ด ์„ฑ๊ณต๋ฅ ์„ ๋ณด๊ณ ํ•œ ์‹ค๊ธฐ 4๊ณผ์ œ(Fig. 5)์™€ ์ด๋ฆ„์ด ์ผ์น˜ํ•˜๋Š” ๊ณผ์ œ๋Š” ์•„๋‹ˆ๋‹ค.

ํ•ต์‹ฌ ๋ฐฉ๋ฒ•๋ก :

๋‘ residual policy๋Š” ๊ฐ™์€ ์ œ์–ด ํ˜•์‹์„ ์“ด๋‹ค(Eq. 1). reference ์ž์„ธ q^{\mathrm{ref}}_t์— policy action a_t๋ฅผ ์ฐจ์›๋ณ„ scale s๋กœ ๋ˆ„์ ํ•œ ๋ณด์ • \Delta q_t๋ฅผ ๋”ํ•œ๋‹ค.

\Delta q_t=\gamma\,\Delta q_{t-1}+s\odot a_t,\qquad q^{\mathrm{cmd}}_t=q^{\mathrm{ref}}_t+\Delta q_t

\gamma๋Š” decay ๊ณ„์ˆ˜์ด๊ณ  \Delta q_t๋Š” ์ฐจ์›๋ณ„๋กœ clip๋˜์–ด ๋ช…๋ น์ด reference ์ฃผ๋ณ€์˜ bounded envelope ์•ˆ์— ๋จธ๋ฌธ๋‹ค. retargeting์˜ ์ฃผ ๋ชฉ์ ํ•ญ์€ contact anchor๋‹ค(Eq. 3). ํ”„๋ ˆ์ž„ t์˜ ์ ‘์ด‰ ์ง‘ํ•ฉ \mathcal{C}_t์— ๋Œ€ํ•ด ๋กœ๋ด‡ ์† ํ‘œ๋ฉด์  c_i(u_t)๊ฐ€ ๊ธฐ๋ก๋œ ๋ฌผ์ฒด ํ‘œ๋ฉด ์ ‘์ด‰ ์œ„์น˜ p_i์— 1 cm ํ—ˆ์šฉ์˜ค์ฐจ \delta ์•ˆ์œผ๋กœ ๋“ค์–ด์˜ค๊ฒŒ ํ•œ๋‹ค.

E_{\mathrm{tgt}}=\sum_{i\in\mathcal{C}_t}\Bigl[\max\bigl(\lVert c_i(u_t)-p_i\rVert-\delta,\;0\bigr)\Bigr]^2

u_t๋Š” ์†๋ชฉ \mathrm{SE}(3)์™€ ๊ทธ ์†์˜ ๋Šฅ๋™ ์†๊ฐ€๋ฝ ์ œ์–ด๋ฅผ ๋ฌถ์€ ๋ฒกํ„ฐ๋‹ค. ์†์ด ๋ฐ”๋€Œ๋ฉด u_t์˜ ์ฐจ์›๊ณผ ๊ธฐํ•˜ ์งˆ์˜๋งŒ ๋ฐ”๋€Œ๊ณ  ์‹์€ ๊ทธ๋Œ€๋กœ๋‹ค.

์ฃผ์š” ๊ฒฐ๊ณผ:

  • Physics refinement(Table 1, Fig. 3): ํ•œ ์† 25,454 ๊ถค์ (๋ฌผ์ฒด 14๊ฐœ, ๋ฌผ์ฒด-์ œ์Šค์ฒ˜ 85์Œ)์—์„œ replay 7.26% โ†’ residual 59.31%(๋ฌผ์ฒด๋ณ„ ๋น„๊ฐ€์ค‘ ํ‰๊ท ). replay๊ฐ€ 0%์˜€๋˜ ๋ฌผ์ฒด 4๊ฐœ๋„ 41.5~70.8%. ์–‘์† 25๊ณผ์ œ 250 ๊ถค์ ์€ task-level 16.0% โ†’ 62.4%, attempt-level 8.4% โ†’ 43.9%.
  • ์ฒ˜์Œ ๋ณด๋Š” ๋ฌผ์ฒด(Table 2): ํ•™์Šต์— ์—†๋˜ YCB ๋ฌผ์ฒด 6๊ฐœ์— fine-tuning ์—†์ด pooled 72.4%(7,683 trial, ๋ฌผ์ฒด๋ณ„ 46.5~88.7%).
  • Retargeting ๋น„๊ต(Table 5, MANO 100 ๊ถค์  โ†’ DexHand 021): fidelity๋Š” Vector 10.6 mm๊ฐ€ ์ตœ๊ณ , ์ด ๋…ผ๋ฌธ 14.7 mm. ๊ด€์ ˆ ํ•œ๊ณ„ ์œ„๋ฐ˜ 3.2%, jitter 0.53 rad/s. ๋Œ€์‹  ์†Œ์š” ์‹œ๊ฐ„ 8,513 s๋กœ ๋น„๊ต ๋Œ€์ƒ ์ค‘ ๊ฐ€์žฅ ๋А๋ฆฌ๋‹ค.
  • Cross-embodiment(Table 3): ๊ฐ™์€ ์„ค์ •์œผ๋กœ DexHand 021ยท021ProยทShadowยทAllegro์— 200 ๊ถค์ ์”ฉ, replay ํ‰๊ท  17.1% โ†’ residual 79.5%(+62.4 pp), ๋„ค ์† ๋ชจ๋‘ 76.5~82.5%.
  • Downstream(Table 4): Diffusion Policy์— ๋ณต์›ํ•œ ์ ‘์ด‰๋ ฅ 16์ฐจ์›์„ ๋”ํ•˜๋ฉด 34.7% โ†’ 46.5%(+11.8 pp).
  • ์‹ค๊ธฐ(Fig. 5): ์–‘ํŒ” JAKA Mini 2 + DexHand 021, 4๊ณผ์ œ ๊ฐ 10ํšŒ. ์ข…์ด์ปต ์Œ“๊ธฐ 7/10, ๋ณ‘ ๊ฑด๋„ค๊ณ  ์„ธ์šฐ๊ธฐ 5/10, ๋ฌผ ๋ถ“๊ธฐ 6/10, ์† ์•ˆ ๋ณ‘ ์žฌ๋ฐฐ์น˜ 3/10.

๊ฒฐ๋ก :

์ƒˆ๋กœ์šด ์ ์€ โ€œ์ ‘์ด‰ ๊ตฌ์กฐ๋ฅผ ์˜ฎ๊ธด๋‹คโ€๋Š” retargeting ๊ด€์ ๊ณผ โ€œphysics refinement๋ฅผ ๊ณผ์ œ๋ณ„ ํ•™์Šต์ด ์•„๋‹ˆ๋ผ amortized repair operator๋กœ ๋ณธ๋‹คโ€๋Š” ๊ด€์ ์„ ํ•œ ํŒŒ์ดํ”„๋ผ์ธ์œผ๋กœ ๋ฌถ์€ ๊ฒƒ์ด๋‹ค. ๋‹ค๋งŒ ์ฆ๊ฑฐ์˜ ๋ชจ์–‘์„ ๋ณด๋ฉด ์ธก์ •๋œ ๋น„๊ต๋Š” ๊ฑฐ์˜ ๋ชจ๋‘ ์ž๊ธฐ ๋ฐฉ๋ฒ•์˜ replay ๋Œ€๋น„ residual์ด๋‹ค. ๋‹ค๋ฅธ retargeter๋กœ ๋งŒ๋“  reference๋ฅผ ๊ฐ™์€ residual ๋‹จ๊ณ„์— ๋„ฃ์–ด ๋น„๊ตํ•˜๋Š” ์‹คํ—˜, ๊ณผ์ œ๋ณ„ policy ๋Œ€๋น„ ๋‹จ์ผ policy์˜ ์†์‹ค์„ ์žฌ๋Š” ์‹คํ—˜, ๋‹ค๋ฅธ physics refinement ๋ฐฉ๋ฒ•๊ณผ์˜ ์„ฑ๊ณต๋ฅ  ๋น„๊ต๊ฐ€ ์—†๋‹ค. ๊ทธ๋ž˜์„œ โ€œ์ ‘์ด‰ ๊ตฌ์กฐ๊ฐ€ embodiment gap์„ ๋” ์ž˜ ๋‹ซ๋Š”๋‹คโ€๋Š” ํ•ต์‹ฌ ์ฃผ์žฅ์€ kinematic ์ง€ํ‘œ(Table 5)๋กœ๋งŒ ๋’ท๋ฐ›์นจ๋˜๊ณ , ๊ทธ ํ‘œ์—์„œ๋„ ์ด ๋ฐฉ๋ฒ•์€ ๊ฐ€์žฅ ๋А๋ฆฌ๋‹ค. ์ฝ”๋“œยท๋ฐ์ดํ„ฐยทํ•˜์ดํผํŒŒ๋ผ๋ฏธํ„ฐ๋Š” ๊ณต๊ฐœ๋˜์ง€ ์•Š์•˜๋‹ค.


๐Ÿ”” Ring Review

๐Ÿ”” Ring โ€” An idea that echoes. Grasp the core and its value.

ํ•œ ์ค„๋กœ ์‹œ์ž‘ํ•˜๋ฉด

์ด ๋…ผ๋ฌธ์€ โ€œ์‚ฌ๋žŒ ์‹œ์—ฐ์—์„œ ๋กœ๋ด‡์ด ๊ฐ€์ ธ๊ฐ€์•ผ ํ•  ๊ฒƒ์€ ์†์˜ ์›€์ง์ž„์ด ์•„๋‹ˆ๋ผ ์ ‘์ด‰์˜ ๋ฐฐ์น˜๋‹คโ€๋ผ๋Š” ๋ฌธ์žฅ์—์„œ ์ถœ๋ฐœํ•œ๋‹ค(ยง1: โ€œWhat actually survives the change of embodiment is the contact structureโ€). ์ ‘์ด‰ ๋ฐฐ์น˜๋ฅผ ์˜ฎ๊ธฐ๋ ค๋ฉด ๋จผ์ € ์ ‘์ด‰์ด ์žˆ์–ด์•ผ ํ•˜๋Š”๋ฐ mocap์—๋Š” ์ ‘์ด‰๋„ ํž˜๋„ ์—†๋‹ค. ๊ทธ๋ž˜์„œ ์ˆœ์„œ๊ฐ€ ์ •ํ•ด์ง„๋‹ค. ์‹œ๋ฎฌ๋ ˆ์ด์…˜์—์„œ ์‚ฌ๋žŒ ์† ๋ชจ๋ธ(MANO)๋กœ ์‹œ์—ฐ์„ ๋‹ค์‹œ ์‹คํ–‰ํ•ด ์ ‘์ด‰์„ ๋งŒ๋“ค์–ด ๋‚ด๊ณ , ๊ทธ ์ ‘์ด‰ ์œ„์น˜๋ฅผ ๋กœ๋ด‡ ์† ํ‘œ๋ฉด์ ์˜ ๋ชฉํ‘œ๋กœ ์‚ผ์•„ retargetingํ•˜๊ณ , ๊ทธ ๊ฒฐ๊ณผ๊ฐ€ ์—ฌ์ „ํžˆ kinematic์ด๋ฏ€๋กœ ๋กœ๋ด‡ ์†์—์„œ ํ•œ ๋ฒˆ ๋” residual๋กœ ๋™์—ญํ•™์„ ๋งž์ถ˜๋‹ค. ๋‘ ๋ฒˆ์˜ ์‹œ๋ฎฌ๋ ˆ์ด์…˜ ๋‹จ๊ณ„ ๋ชจ๋‘ ๊ณผ์ œ๋ณ„ ํ•™์Šต์ด ์•„๋‹ˆ๋ผ ์„ค์ •๋‹น policy ํ•˜๋‚˜๋กœ ์ฒ˜๋ฆฌํ•œ๋‹ค๋Š” ๊ฒƒ์ด ํ™•์žฅ์„ฑ ์ฃผ์žฅ์˜ ๊ทผ๊ฑฐ๋‹ค.

๋ฐฐ๊ฒฝ: ์™œ ์‚ฌ๋žŒ ์‹œ์—ฐ์ด ๋ฐ”๋กœ ๋กœ๋ด‡ ๋ฐ์ดํ„ฐ๊ฐ€ ๋˜์ง€ ์•Š๋Š”๊ฐ€ (ยง1, ยง2)

๋…ผ๋ฌธ์€ ๋ฐ์ดํ„ฐ ์ถœ์ฒ˜๋ฅผ ์„ธ ๊ฐˆ๋ž˜๋กœ ๋‚˜๋ˆ  ๊ฐ๊ฐ์˜ ํ•œ๊ณ„๋ฅผ ์งš๋Š”๋‹ค.

์ถœ์ฒ˜ ์žฅ์  ๋…ผ๋ฌธ์ด ์งš๋Š” ํ•œ๊ณ„
๋กœ๋ด‡ teleop ๋ช…๋ น ๊ณต๊ฐ„ = ํ–‰๋™ ๊ณต๊ฐ„์ด๋ฉด ๊ทธ๋Œ€๋กœ ๋ฐ์ดํ„ฐ๊ฐ€ ๋œ๋‹ค(ํŒ” + ํ‰ํ–‰ ๊ทธ๋ฆฌํผ) ๋‹ค์ง€ ์†์—์„œ๋Š” โ‘  ์‚ฌ๋žŒ ์†์ด synergy๋กœ ์›€์ง์—ฌ 20๊ฐœ ์ด์ƒ ๊ฒฐํ•ฉ ์ž์œ ๋„๋ฅผ ์˜์‹์ ์œผ๋กœ ์กฐ์ข…ํ•˜๊ธฐ ์–ด๋ ต๊ณ  โ‘ก ์ง€์—ฐ์ด ์กฐ์ž‘ ํ’ˆ์งˆ์„ ๋–จ์–ด๋œจ๋ฆฌ๋ฉฐ โ‘ข ๊ฐ€๋ ค์ง„ ์ ‘์ด‰์˜ haptic feedback์ด ์—†๋‹ค. ํž˜ ํ”ผ๋“œ๋ฐฑ์„ ๋นผ๋ฉด contact-rich teleop ์„ฑ๊ณต๋ฅ ์ด 78%์—์„œ 48%๋กœ ๋–จ์–ด์ง„๋‹ค๋Š” Glovity(Gao et al., 2025) ๊ฒฐ๊ณผ๋ฅผ ์ธ์šฉํ•œ๋‹ค
์˜์ƒ ๊ฐ€์žฅ ์‹ธ๊ณ , egocentric ์˜์ƒ์œผ๋กœ ๊ทœ๋ชจ๋ฅผ ํ‚ค์šธ ์ˆ˜ ์žˆ๋‹ค(EgoScale) ์†-๋ฌผ์ฒด ์ž์„ธ๋„ ์ ‘์ด‰๋ ฅ๋„ ์ •๋ฐ€ํ•˜๊ฒŒ ๋ณต์›ํ•˜์ง€ ๋ชปํ•œ๋‹ค. EgoScale์˜ action label๋„ ํ”ฝ์…€์ด ์•„๋‹ˆ๋ผ ๊ณ„์ธก ์žฅ๊ฐ‘์—์„œ ์˜จ๋‹ค๊ณ  ์ง€์ ํ•œ๋‹ค
Motion capture ๋กœ๋ด‡ ์—†์ด mm ์ •๋ฐ€๋„์˜ ์†ยท๋ฌผ์ฒด kinematics ์ˆœ์ˆ˜ kinematic์ด๋ผ ์ ‘์ด‰๋ ฅ์ด ์—†๊ณ , ์† ๋ชจ์–‘์ด ๋‹ฌ๋ผ ๋กœ๋ด‡์—์„œ ๊ทธ๋Œ€๋กœ replayํ•  ์ˆ˜ ์—†๋‹ค

์ด ์œ„์—์„œ ๊ธฐ์กด ์ ‘๊ทผ์„ ๋‘ ์ถ•์œผ๋กœ ์ •๋ฆฌํ•œ๋‹ค(ยง2). retargeting ์ถ•์—์„œ๋Š” DexPilotยทAnyTeleop(dex-retargeting์˜ PositionยทVector)์ฒ˜๋Ÿผ ๊ธฐํ•˜ ๋Œ€์‘์„ ๋งž์ถ”๋Š” ๋ฐฉ๋ฒ•์ด kinematic topology๊ฐ€ ๋น„์Šทํ•˜๋‹ค๊ณ  ๊ฐ€์ •ํ•˜๊ณ , DexGraspNet๋ฅ˜ ์ ‘์ด‰ยทforce-closure ํ•ฉ์„ฑ์€ ์ •์  grasp์— ๋จธ๋ฌผ๋ฉฐ, SPIDER๋Š” ๋ฌผ๋ฆฌ ๊ธฐ๋ฐ˜ ์ƒ˜ํ”Œ๋ง์œผ๋กœ ๋™์—ญํ•™์ ์œผ๋กœ ๊ฐ€๋Šฅํ•œ ๊ถค์ ์„ ๋งŒ๋“ค์ง€๋งŒ ๊ถค์ ๋‹น ๋น„์šฉ์ด ํฌ๋‹ค๊ณ  ์“ด๋‹ค. real2sim2real ์ถ•์—์„œ๋Š” DexMV(๊ณผ์ œ๋ณ„ RL), DexMachina(๊ธฐ์ˆ ๋ณ„ curriculum ๋ณด์ƒ), DexMimicGen(๊ณผ์ œ๋ณ„ imitation policy), ๊ทธ๋ฆฌ๊ณ  ๊ฐ€์žฅ ๊ฐ€๊นŒ์šด DemoBot(์‚ฌ๋žŒ ์˜์ƒ motion prior ์œ„์˜ residual RL, ๋‹จ ๊ณผ์ œ๋ณ„ ์‹œ๊ฐ„ ๋ถ„ํ• ยทevent ๊ธฐ๋ฐ˜ ๋ณด์ƒ curriculumยท์„ฑ๊ณต ์กฐ๊ฑด๋ถ€ reset์ด ํ•„์š”)์„ ๋“ ๋‹ค. ๊ณตํ†ต ๋ฌธ์ œ๋Š” ๊ณผ์ œ๋ณ„ ํ•™์Šต๊ณผ ๊ณผ์ œ ๋ณต์žก๋„์˜ ํ•œ๊ณ„(in-hand ํšŒ์ „, ๊ทธ๋ฆฌํผ pick-and-place, grasp ์ˆ˜์ค€)์ด๊ณ , ๋…ผ๋ฌธ์€ ๋‹ค์ง€ ์–‘์† contact-rich ์กฐ์ž‘์„ real2sim2real๋กœ ๋‹ค๋ฃฌ ์‚ฌ๋ก€๊ฐ€ ์•„์ง ์—†๋‹ค๊ณ  ์ฃผ์žฅํ•œ๋‹ค.

์ด ์ •๋ฆฌ์—์„œ ๋ˆˆ์— ๋„๋Š” ๋นˆ์นธ์ด ํ•˜๋‚˜ ์žˆ๋‹ค. ๊ฐ™์€ ๋ฌธ์ œ(MANO ์‹œ์—ฐ โ†’ ๋กœ๋ด‡ ์†, residual RL๋กœ ๋ฌผ๋ฆฌ ๋ณด์ •)๋ฅผ ๋‹ค๋ฃฌ ManipTrans์™€ DexTrack์€ ๊ด€๋ จ ์—ฐ๊ตฌ์— ์—†๋‹ค. ๊ทธ๋ฆฌ๊ณ  ๊ฐ™์€ ๊ทธ๋ฃน์˜ ๋ฐ์ดํ„ฐ์…‹ ๋…ผ๋ฌธ DexCanvas๋Š” ๋ฌผ์ฒดยท์ œ์Šค์ฒ˜ ์ •์˜์˜ ์ถœ์ฒ˜๋กœ๋งŒ ์ธ์šฉ๋œ๋‹ค. ์ด ๊ด€๊ณ„๋Š” ๋’ค์˜ ์ž๋ฆฌ ๋งค๊น€ ์ ˆ์—์„œ ๋‹ค๋ฃฌ๋‹ค.

๋ฐฉ๋ฒ• ์ƒ์„ธ (ยง3)

์‹คํ—˜์šฉ ๋กœ๋ด‡ ์†์€ DexHand 021์ด๋‹ค. ๋Šฅ๋™ ์ œ์–ด 12๊ฐœ๊ฐ€ tendon์œผ๋กœ ๊ฒฐํ•ฉ๋ผ ๋ฌผ๋ฆฌ ์†๊ฐ€๋ฝ ๊ด€์ ˆ 20๊ฐœ๋ฅผ ์›€์ง์ด๋Š” 5์ง€ ์†์ด๊ณ , ์ €์ž ์†Œ์† ์ค‘ ํ•˜๋‚˜์ธ DexRobot Co. Ltd.์˜ ์†์ด๋‹ค(์ธ์šฉ Yuan et al., 2025). ๋…ผ๋ฌธ์€ ๋ฐฉ๋ฒ•์ด ์ด ์†์— ํŠนํ™”๋˜์ง€ ์•Š์•˜๊ณ  ์†์„ ๋ฐ”๊พธ๋ ค๋ฉด embodiment profile ๊ฐฑ์‹ ๊ณผ ๋กœ๋ด‡ ์ชฝ residual policy ์žฌํ•™์Šต๋งŒ ํ•„์š”ํ•˜๋‹ค๊ณ  ์“ด๋‹ค(ยง3 ์„œ๋‘).

3.1 Motion capture์™€ ๋ฐ์ดํ„ฐ ์ฒ˜๋ฆฌ


Fig. 2: (a) ์ ์™ธ์„  ์นด๋ฉ”๋ผ 22๋Œ€์™€ ๋™๊ธฐํ™”๋œ RGB-D ์นด๋ฉ”๋ผ๋กœ ์ด๋ค„์ง„ ๋ฐฉ ๊ทœ๋ชจ ๊ด‘ํ•™ mocap ์‹œ์Šคํ…œ. (b) ๋งˆ์ปค๋ฅผ MANO ์ขŒํ‘œ๊ณ„๋กœ ์ „์ฒ˜๋ฆฌํ•ด MANO ์† ๋ชจ๋ธ์— ๋งž์ถ”๊ณ , ๋ฌผ๋ฆฌ ์‹œ๋ฎฌ๋ ˆ์ด์…˜์„ ๊ฑฐ์ณ ์ ‘์ด‰๋ ฅ ์ฃผ์„์„ ์–ป๋Š” ์ฒ˜๋ฆฌ ํ๋ฆ„.
ํ•ญ๋ชฉ ๊ฐ’(ยง3.1)
์นด๋ฉ”๋ผ ์ ์™ธ์„  22๋Œ€(mm ์ •๋ฐ€๋„ ๊ด‘ํ•™ ์ถ”์ ) + ๋™๊ธฐํ™” RGB-D 2๋Œ€
์† ๋งˆ์ปค ํ•ด๋ถ€ํ•™์  ๋žœ๋“œ๋งˆํฌ์— ๋ฐ˜์‚ฌ ๋งˆ์ปค 14๊ฐœ
๋ฌผ์ฒด ๋งˆ์ปค 3D ํ”„๋ฆฐํŒ… ๋งˆ์šดํŠธ์— ๋ถ€์ฐฉ. ๋งˆ์šดํŠธ ์œ„์น˜๊ฐ€ ์‹œ๋ฎฌ๋ ˆ์ด์…˜ URDF์™€ ์ •ํ™•ํžˆ ์ผ์น˜ํ•˜๋„๋ก ์„ค๊ณ„
๊ธฐ๋ก ์† ์ž์„ธ(6-DoF ์†๋ชฉ + ์†๊ฐ€๋ฝ ๊ด€์ ˆ)์™€ ๋ฌผ์ฒด ๊ถค์ , 120 Hz
์† ํ˜•์ƒ \beta\in\mathbb{R}^{10} ๋ณด์ • ์‹œํ€€์Šค์˜ RGB ํ”„๋ ˆ์ž„์— HaMeR๋ฅผ ์ ์šฉํ•ด ํ”ผํ—˜์ž๋ณ„ ์ถ”์ •
ํ”„๋ ˆ์ž„๋ณ„ ์ž์„ธ ํ•ด๋ถ€ํ•™์  ์ œ์•ฝ ์•„๋ž˜ ๋งˆ์ปค-ํ‘œ๋ฉด ๊ฑฐ๋ฆฌ๋ฅผ ์ตœ์†Œํ™”ํ•˜๋„๋ก ์†๋ชฉ \mathrm{SE}(3)์™€ ์†๊ฐ€๋ฝ ์ž์„ธ ์ตœ์ ํ™”

๋ฌผ์ฒด ๋งˆ์šดํŠธ๋ฅผ URDF์™€ ๋งž์ถฐ ์„ค๊ณ„ํ•œ ๊ฒƒ์€ ์‚ฌ์†Œํ•ด ๋ณด์ด์ง€๋งŒ ์ค‘์š”ํ•œ ์„ ํƒ์ด๋‹ค. ๋’ค์˜ ๋ชจ๋“  ๋‹จ๊ณ„๊ฐ€ โ€œ๋ฌผ์ฒด ํ‘œ๋ฉด์˜ ์–ด๋А ์ โ€์„ ๋ชฉํ‘œ๋กœ ์‚ผ๊ธฐ ๋•Œ๋ฌธ์— ๋ฌผ์ฒด ์ขŒํ‘œ๊ณ„ ์˜ค์ฐจ๋Š” ๊ณง ์ ‘์ด‰ ๋ชฉํ‘œ์˜ ์˜ค์ฐจ๊ฐ€ ๋œ๋‹ค.

3.2 Physics refinement: kinematic ๊ธฐ๋ก์„ ์ˆ˜์„ ํ•˜๋Š” policy

๊ฐ ์‹œ์—ฐ์„ Isaac Gym์—์„œ ๊ตฌ๋™๋˜๋Š” MANO ์†์œผ๋กœ ๋‹ค์‹œ ์‹คํ–‰ํ•œ๋‹ค. MANO๋Š” parametric mesh์ด๋ฏ€๋กœ ์†๊ฐ€๋ฝ ๊ด€์ ˆ 22๊ฐœ(๋Šฅ๋™) + 6-DoF ์†๋ชฉ์„ ๊ฐ€์ง„ ๊ด€์ ˆ ๊ฐ•์ฒด ๋ชจ๋ธ๋กœ ๋ฐ”๊พธ๊ณ  PD ์ œ์–ด๊ธฐ๋กœ ๊ตฌ๋™ํ•œ๋‹ค. residual policy๊ฐ€ Eq. 1๋กœ reference์— ๋ณด์ •์„ ๋”ํ•˜๊ณ , ์‹œ๋ฎฌ๋ ˆ์ดํ„ฐ์˜ ์ ‘์ด‰ ๋ชจ๋ธ์ด ๋ฌผ๋ฆฌ์ ์œผ๋กœ ์ผ๊ด€๋œ ํž˜์„ ๋‚ธ๋‹ค.

  • ๋ณด์ƒ: ์‹œ๋ฎฌ๋ ˆ์ด์…˜ ๋ฌผ์ฒด๊ฐ€ ๊ธฐ๋ก๋œ ๊ถค์ ์—์„œ ๋ฒ—์–ด๋‚˜๋Š” ๊ฒƒ์„ ๋ฒŒํ•˜๋Š” ๋ฌผ์ฒด ๊ถค์  ์ถ”์  ํ•ญ + ์ง€์ •๋œ ์†๊ฐ€๋ฝ์ด ๋ฌผ์ฒด ํ‘œ๋ฉด์— ๋‹ฟ๋„๋ก ํ•˜๋Š” contact-achievement ํ•ญ. ๊ฐ€์ค‘์น˜์™€ ์ •ํ™•ํ•œ ํ˜•ํƒœ๋Š” ๋…ผ๋ฌธ์— ์—†๋‹ค.
  • ์„ค์ •๋‹น policy ํ•˜๋‚˜: ํ•œ ์† ์„ค์ •๊ณผ ์–‘์† ์„ค์ • ๊ฐ๊ฐ์—์„œ ๋ชจ๋“  ๊ถค์ ์„ ํ•˜๋‚˜์˜ policy๋กœ ํ•™์Šตํ•œ๋‹ค. ์กฐ๊ฑด์€ ๊ณผ์ œ ์‹๋ณ„์ž, ๋ฌผ์ฒด ๊ธฐํ•˜, ํ˜„์žฌ์™€ ๋ฏธ๋ž˜์˜ reference ๋ฌผ์ฒด ์šด๋™์ด๋‹ค. ๋…ผ๋ฌธ์˜ ๋…ผ๋ฆฌ๋Š” ์ด policy๊ฐ€ ์Šค์Šค๋กœ ๊ณ„ํšํ•˜๋Š” ๊ฒƒ์ด ์•„๋‹ˆ๋ผ reference๋ฅผ ์ˆ˜์„ ํ•˜๋ฏ€๋กœ ์ด ๊ด€์ธก์ด ํ•™์Šต๊ณผ ์ถ”๋ก  ์–‘์ชฝ์—์„œ ์ฃผ์–ด์ง„๋‹ค๋Š” ๊ฒƒ์ด๋‹ค(ยง3.2). ๊ทธ๋ž˜์„œ ์ƒˆ ์‹œ์—ฐ์€ ๊ถค์ ๋ณ„ ์žฌํ•™์Šต ์—†์ด ์ฒ˜๋ฆฌ๋œ๋‹ค.
  • Synthesis: ํ•™์Šต๋œ policy๋ฅผ ๋ฌผ์ฒด ์ดˆ๊ธฐ ์ž์„ธ ์„ญ๋™, ๋งˆ์ฐฐยท์งˆ๋Ÿ‰ randomization, ๋ฌด์ž‘์œ„ ์™ธ๋ ฅ ์•„๋ž˜์—์„œ rolloutํ•ด ์‹œ์—ฐ๋งˆ๋‹ค ๋ฌผ๋ฆฌ์ ์œผ๋กœ ์ผ๊ด€๋œ ๋ณ€ํ˜•์„ ๋” ๋งŒ๋“ ๋‹ค. ๊ฐ ๋ณ€ํ˜•์€ ๊ณผ์ œ์˜ ์ ‘์ด‰ ๊ตฌ์กฐ๋ฅผ ์œ ์ง€ํ•œ ์ฑ„ ์ด์›ƒํ•œ ๋™์—ญํ•™์ ์œผ๋กœ ๊ฐ€๋Šฅํ•œ ๊ถค์ ์„ ํƒ์ƒ‰ํ•œ๋‹ค๊ณ  ์“ด๋‹ค.
  • ์ถœ๋ ฅ: MANO ์† ์šด๋™๊ณผ ๋ฌผ์ฒด ์šด๋™, ๊ทธ๋ฆฌ๊ณ  ํ”„๋ ˆ์ž„๋ณ„ ์ ‘์ด‰ ๋ผ๋ฒจ(์–ด๋А ์†๊ฐ€๋ฝ ๋ถ€์œ„๊ฐ€ ๋‹ฟ์•˜๋‚˜), ๋ฌผ์ฒด ํ‘œ๋ฉด ์ ‘์ด‰ ์œ„์น˜(๋ฌผ์ฒด์˜ ์–ด๋””์— ๋‹ฟ์•˜๋‚˜), ์žฌ๊ตฌ์„ฑํ•œ ํž˜ ๋ฒกํ„ฐ. ์•ž์˜ ๋‘˜์ด retargeting์˜ ์ฃผ ์ž…๋ ฅ์ด๋‹ค.

3.3 Contact-anchored retargeting

physics-refined ๊ถค์ ์˜ ํ”„๋ ˆ์ž„๋งˆ๋‹ค ๋กœ๋ด‡ ์† configuration u_t\in\mathbb{R}^D(์†๋ชฉ \mathrm{SE}(3) + ๋Šฅ๋™ ์†๊ฐ€๋ฝ ์ œ์–ด)๋ฅผ ์ตœ์ ํ™”ํ•œ๋‹ค. ํ”„๋ ˆ์ž„์€ ์ ‘์ด‰ ํŒจํ„ด(๋™์‹œ์— ๋‹ฟ์•„ ์žˆ๋Š” ์†๊ฐ€๋ฝ ๋ถ€์œ„์˜ ์ง‘ํ•ฉ)์œผ๋กœ ๋ถ„ํ• ํ•˜๊ณ , ์ ‘์ด‰์ด ๋งŽ์€ ํ”„๋ ˆ์ž„๊ณผ ์ ์€ ํ”„๋ ˆ์ž„์— ๋‹ค๋ฅธ ๋ชฉ์ ํ•จ์ˆ˜๋ฅผ ์“ด๋‹ค.

๋‹ค์ค‘ ์ ‘์ด‰ ํ”„๋ ˆ์ž„์€ Eq. 2๋ฅผ ์ตœ์†Œํ™”ํ•œ๋‹ค.

E_{\mathrm{ct}}=w_{\mathrm{tgt}}E_{\mathrm{tgt}}+w_{\mathrm{pen}}E_{\mathrm{pen}}+w_{\mathrm{h2o}}E_{\mathrm{h2o}}+w_{\mathrm{spen}}E_{\mathrm{spen}}+w_jE_{\mathrm{joint}}+w_gE_{\mathrm{ground}}+w_bE^{\mathrm{base}}_{\mathrm{sm}}+w_cE^{\mathrm{ctrl}}_{\mathrm{sm}}+w_rE^{\mathrm{rot}}_{\mathrm{sm}}

ํ•ญ ์—ญํ• 
E_{\mathrm{tgt}} (Eq. 3) ์ฃผ ๋ชฉ์ . ๋กœ๋ด‡ ํ‘œ๋ฉด์ ์„ ๊ธฐ๋ก๋œ ๋ฌผ์ฒด ํ‘œ๋ฉด ์ ‘์ด‰ ์œ„์น˜๋กœ, 1 cm ํ—ˆ์šฉ์˜ค์ฐจ ๋ฐ–๋งŒ ๋ฒŒํ•จ
E_{\mathrm{pen}}, E_{\mathrm{spen}} ๋ฌผ์ฒด ๊ด€ํ†ต, ์ž๊ธฐ ๊ด€ํ†ต
E_{\mathrm{h2o}} ์†-๋ฌผ์ฒด ๊ฑฐ๋ฆฌ๊ฐ€ ์ง€๋‚˜์น˜๊ฒŒ ํฐ ๊ฒƒ
E_{\mathrm{joint}} ๊ด€์ ˆ ํ•œ๊ณ„ ์œ„๋ฐ˜
E_{\mathrm{ground}} ์ง€๋ฉด ๊ด€ํ†ต
E^{\mathrm{base}}_{\mathrm{sm}}, E^{\mathrm{ctrl}}_{\mathrm{sm}}, E^{\mathrm{rot}}_{\mathrm{sm}} ์†๋ชฉ ์œ„์น˜ยท์†๊ฐ€๋ฝ ์ œ์–ดยท์†๋ชฉ ํšŒ์ „์˜ ์‹œ๊ฐ„์  jitter

๊ฐ€์ค‘์น˜ w_\ast์˜ ๊ฐ’์€ ๋…ผ๋ฌธ์— ์—†๋‹ค.

ํ‘œ๋ฉด์  ์„ ํƒ. ์ ‘์ด‰ ๋ผ๋ฒจ์€ ํ•ด๋ถ€ํ•™์  ๋ถ€์œ„(โ€œindex fingertipโ€, โ€œthumb padโ€)๋ฅผ ๊ฐ€๋ฆฌํ‚ค๊ณ , ๋กœ๋ด‡ ์†์—์„œ ๊ทธ ๋ถ€์œ„๋Š” mesh ํ‘œ๋ฉด์  ์ง‘ํ•ฉ์— ๋Œ€์‘ํ•œ๋‹ค. ์ ‘์ด‰๋งˆ๋‹ค ๋งํฌ ์ค‘์‹ฌ์—์„œ ๋ณธ ๋ฐฉํ–ฅ์ด ํ•ด๋‹น MANO ์ ‘์ด‰์˜ ๋ฐฉํ–ฅ๊ณผ ๊ฐ€์žฅ ์ž˜ ๋งž๋Š” ํ‘œ๋ฉด์ ์„ ๊ณ ๋ฅด๊ณ , ๊ฑฐ๋ฆฌ๋ฅผ tiebreaker๋กœ ์“ด๋‹ค. ์ฆ‰ โ€œ๋กœ๋ด‡ ์†๊ฐ€๋ฝ์˜ ์–ด๋А ๋ฉด์œผ๋กœ ๋‹ฟ์„์ง€โ€๋ฅผ ์‚ฌ๋žŒ ์†๊ฐ€๋ฝ์ด ๋‹ฟ์€ ๋ฉด์˜ ๋ฐฉํ–ฅ์œผ๋กœ ์ •ํ•œ๋‹ค.

์ ‘์ด‰์ด ์ ์€ ํ”„๋ ˆ์ž„(์ ‘์ด‰ 0๊ฐœ ๋˜๋Š” 1๊ฐœ)์€ ์†๋ชฉ์„ ๊ณ ์ •ํ•˜๊ณ  feasibility ํ•ญ๋งŒ์œผ๋กœ ์†๊ฐ€๋ฝ ์ œ์–ด๋ฅผ ์ตœ์ ํ™”ํ•œ๋‹ค.

์ดˆ๊ธฐํ™”. ํ”„๋ ˆ์ž„๋งˆ๋‹ค ๋Œ€์‘ํ•˜๋Š” MANO configuration์—์„œ ์‹œ์ž‘ํ•œ๋‹ค. MANO ์†๊ฐ€๋ฝ ๊ด€์ ˆ 22๊ฐœ๋ฅผ ๊ทผ์‚ฌ์  kinematic ๋Œ€์‘์œผ๋กœ ๋กœ๋ด‡์˜ ๋Šฅ๋™ ์ œ์–ด์— ์‚ฌ์ƒํ•ด ๊ด€์ ˆ ํ•œ๊ณ„๋กœ clipํ•˜๊ณ , ์†๋ชฉ ๋ฐฉํ–ฅ์€ ๋ฌผ์ฒด์— ๋Œ€ํ•œ MANO ์†๋ชฉ ๋ฐฉํ–ฅ์„, ์†๋ชฉ ์œ„์น˜๋Š” ๋กœ๋ด‡๊ณผ MANO ์†๋ ์œ„์น˜์˜ ํ‰๊ท  offset์œผ๋กœ ์žก๋Š”๋‹ค.

์† ๋ชจ์–‘ ์ผ๋ฐ˜์„ฑ. ์ตœ์ ํ™”๋Š” forward kinematics ํ‘œ๋ฉด์ , embodiment ๊ธฐํ•˜, ๊ด€์ ˆ ํ•œ๊ณ„, ๋Šฅ๋™ ์ œ์–ด ๋ฒกํ„ฐ๋งŒ ๋‹ค๋ฃฌ๋‹ค. ์†์ด ๋ฐ”๋€Œ๋ฉด ์ตœ์ ํ™” ์ฐจ์›๊ณผ ๊ธฐํ•˜ ์งˆ์˜๊ฐ€ ๋ฐ”๋€” ๋ฟ ์†์‹ค ์‹์€ ๊ฐ™๋‹ค. ์†๋งˆ๋‹ค ํ•„์š”ํ•œ ์ž…๋ ฅ์€ embodiment profile(kinematic ๋ชจ๋ธ, ์ œ์–ด-๊ด€์ ˆ ๊ฒฐํ•ฉ, ๊ด€์ ˆ ํ•œ๊ณ„, ์ ‘์ด‰ ๋ถ€์œ„๋ณ„ ํ›„๋ณด ํ‘œ๋ฉด์ ) ํ•˜๋‚˜๋ผ๊ณ  ์“ด๋‹ค.

์—ฌ๊ธฐ์„œ ์งš์–ด ๋‘˜ ์ ์ด ์žˆ๋‹ค. โ€œ๋ชฉ์ ํ•จ์ˆ˜๊ฐ€ ์† ๋ชจ์–‘๊ณผ ๋…๋ฆฝโ€์ด๋ผ๋Š” ๋ง์€ ์‹์˜ ํ˜•ํƒœ์— ๋Œ€ํ•œ ์ง„์ˆ ์ด๋‹ค. ์†์— ๋”ฐ๋ผ ๋‹ฌ๋ผ์ง€๋Š” ํŒ๋‹จ, ์ฆ‰ โ€œMANO์˜ thumb pad๊ฐ€ Allegro์˜ ์–ด๋А ํ‘œ๋ฉด์  ์ง‘ํ•ฉ์— ๋Œ€์‘ํ•˜๋Š”๊ฐ€โ€์™€ โ€œMANO ๊ด€์ ˆ 22๊ฐœ๋ฅผ ์ด ์†์˜ ๋Šฅ๋™ ์ œ์–ด๋กœ ์–ด๋–ป๊ฒŒ ๊ทผ์‚ฌ ์‚ฌ์ƒํ•˜๋Š”๊ฐ€โ€๋Š” embodiment profile๊ณผ ์ดˆ๊ธฐํ™”๋กœ ์˜ฎ๊ฒจ ๊ฐ”์„ ๋ฟ ์‚ฌ๋ผ์ง€์ง€ ์•Š์•˜๋‹ค. 4์ง€ Allegro์—์„œ ์ƒˆ๋ผ์†๊ฐ€๋ฝ ์ ‘์ด‰์„ ์–ด๋–ป๊ฒŒ ์ฒ˜๋ฆฌํ•˜๋Š”์ง€๋„ ๋…ผ๋ฌธ์— ์—†๋‹ค.

3.4 Residual policy learning (๋กœ๋ด‡ ์ชฝ)

retargeting ๊ฒฐ๊ณผ๋Š” ์†๊ฐ€๋ฝ์„ ์˜ฌ๋ฐ”๋ฅธ ์œ„์น˜์— ์˜ฌ๋ฐ”๋ฅธ ์ˆœ์„œ๋กœ ๋†“์ง€๋งŒ kinematic ํ•ด๋‹ค. ๊ตฌ๋™๊ธฐ ํ† ํฌ ํ•œ๊ณ„, tendon ๊ฒฐํ•ฉ, ๋กœ๋ด‡ ์ ‘์ด‰ ๊ธฐํ•˜์—์„œ์˜ ๋ฌผ์ฒด ๋™์—ญํ•™์„ ๋ชจ๋ฅธ๋‹ค. ๋‘ ๋ฒˆ์งธ residual policy(Eq. 1๊ณผ ๊ฐ™์€ ํ˜•์‹)๊ฐ€ ์†๋ชฉ ์œ„์น˜์™€ ์†๊ฐ€๋ฝ ์ œ์–ด์— ์ž‘์€ ๋ณด์ •์„ ํ•™์Šตํ•˜๊ณ , ์†๋ชฉ ๋ฐฉํ–ฅ์€ reference๋ฅผ ๊ทธ๋Œ€๋กœ ๋”ฐ๋ฅธ๋‹ค. DexHand 021์—์„œ๋Š” policy์˜ ๋Šฅ๋™ ์†๊ฐ€๋ฝ ์ œ์–ด 12๊ฐœ๊ฐ€ ๊ฒฐ์ •์ ์ธ tendon ๊ฒฐํ•ฉ์„ ๊ฑฐ์ณ ๋ฌผ๋ฆฌ ๊ด€์ ˆ 20๊ฐœ๋ฅผ ์›€์ง์ธ๋‹ค.

์„ค๊ณ„์ƒ ํฅ๋ฏธ๋กœ์šด ๋””ํ…Œ์ผ์ด ํ•˜๋‚˜ ์žˆ๋‹ค. ๋งˆ์ง€๋ง‰ ์‹œ์—ฐ ์ ‘์ด‰ ์ดํ›„์—๋Š” ์†๊ฐ€๋ฝ ์ฐจ์›์˜ \gamma๋ฅผ 1๋กœ ๋‘”๋‹ค(decay ์—†์Œ). ๋ˆ„์ ๋œ ์†๊ฐ€๋ฝ ๋ณด์ •์ด ์‚ฌ๋ผ์ง€์ง€ ์•Š์•„ ์ด๋ฏธ ๋งŒ๋“  grasp๊ฐ€ ์œ ์ง€๋˜๊ณ , ์†๋ชฉ ์œ„์น˜ ์ฐจ์›์€ ์›๋ž˜ decay ์ผ์ •์„ ๋”ฐ๋ฅธ๋‹ค. ํ•™์Šต์€ ์„ค์ •๋‹น policy ํ•˜๋‚˜, Isaac Gym, PPO, ์กฐ๊ฑด์€ ๊ณผ์ œ ์‹๋ณ„์žยท๋ฌผ์ฒด ๊ธฐํ•˜ยทreference ์šด๋™์ด๋‹ค. domain randomization ์•„๋ž˜ synthesis rollout์œผ๋กœ retargeted reference๋งˆ๋‹ค ๋™์—ญํ•™์ ์œผ๋กœ ๊ฐ€๋Šฅํ•œ ๋กœ๋ด‡ ๊ถค์ ์„ ๋” ๋งŒ๋“ ๋‹ค.

3.5 ํ•˜๋“œ์›จ์–ด ๋ฐฐํฌ

๋งค ์ œ์–ด ์ฃผ๊ธฐ๋งˆ๋‹ค policy๋Š” ํ•˜๋“œ์›จ์–ด ์„ผ์„œ์˜ ํ˜„์žฌ ๊ด€์ ˆ ์œ„์น˜์™€ ๋ฌผ์ฒด ์ž์„ธ๋ฅผ ํ•™์Šต ๋•Œ์™€ ๊ฐ™์€ ๊ด€์ธก schema๋กœ ๋ฐ›์•„ residual์„ ๊ณ„์‚ฐํ•˜๊ณ , retargeted reference์˜ ํ˜„์žฌ ํ”„๋ ˆ์ž„์— ๋”ํ•ด ๊ด€์ ˆ ๋ช…๋ น์„ ๋‚ธ๋‹ค. ๊ฐ€์ค‘์น˜๋Š” ๊ณ ์ •์ด๋‹ค. ์—ํ”ผ์†Œ๋“œ ์ „๋งˆ๋‹ค ์ •์  ๊ธฐํ•˜ ๋ณด์ •์œผ๋กœ ํ•˜๋“œ์›จ์–ด ์ขŒํ‘œ๊ณ„๋ฅผ ์‹œ๋ฎฌ๋ ˆ์ด์…˜ ์ขŒํ‘œ๊ณ„์— ๋งž์ถ”๊ณ , dynamics identificationยทfine-tuningยทonline adaptation์€ ํ•˜์ง€ ์•Š๋Š”๋‹ค. ๋ฌผ์ฒด ์ž์„ธ๋Š” YOLOv5 โ†’ MobileSAM โ†’ FoundationPose ์ธ์ง€ ์Šคํƒ์ด ์‹ค์‹œ๊ฐ„์œผ๋กœ ์ค€๋‹ค(ยง4.3).

๋…ผ๋ฌธ์ด ์“ฐ์ง€ ์•Š์€ ๊ฒƒ๋„ ์ ์–ด ๋‘”๋‹ค. policy๊ฐ€ ๋‚ด๋Š” ์†๋ชฉ \mathrm{SE}(3) ๋ช…๋ น์„ 6-DoF JAKA ํŒ” ๊ด€์ ˆ๋กœ ๋ฐ”๊พธ๋Š” ๋ฐฉ๋ฒ•(IK ๋“ฑ), ์ œ์–ด ์ฃผํŒŒ์ˆ˜, ์‹ค๊ธฐ์—์„œ ๋ฌผ์ฒด ์ดˆ๊ธฐ ์œ„์น˜๋ฅผ ์‹œ์—ฐ๊ณผ ์–ผ๋งˆ๋‚˜ ๋‹ค๋ฅด๊ฒŒ ๋‘์—ˆ๋Š”์ง€๋Š” ๋‚˜์˜ค์ง€ ์•Š๋Š”๋‹ค. policy๋Š” ํŠน์ • ์‹œ์—ฐ์˜ reference๋ฅผ ์‹œ๊ฐ„ ์ˆœ์„œ๋Œ€๋กœ ๋”ฐ๋ผ๊ฐ€๋ฉฐ ์ˆ˜์„ ํ•˜๋Š” ๊ตฌ์กฐ์ด๋ฏ€๋กœ, ์ดˆ๊ธฐ ๋ฌผ์ฒด ๋ฐฐ์น˜์˜ ํ—ˆ์šฉ ๋ฒ”์œ„๊ฐ€ ์‹ค๊ธฐ ๊ฒฐ๊ณผ์˜ ์˜๋ฏธ๋ฅผ ํฌ๊ฒŒ ์ขŒ์šฐํ•œ๋‹ค.

๋…ผ๋ฌธ์ด ์ˆซ์ž๋กœ ๋ฐํžŒ ์„ค์ •๊ณผ ๋ฐํžˆ์ง€ ์•Š์€ ์„ค์ •

๋ถ€๋ก์ด ์—†๋Š” 16์ชฝ ๋…ผ๋ฌธ์ด๋ผ ์žฌํ˜„์— ํ•„์š”ํ•œ ๊ฐ’์˜ ์ƒ๋‹น์ˆ˜๊ฐ€ ๋น ์ ธ ์žˆ๋‹ค.

๊ตฌ๋ถ„ ํ•ญ๋ชฉ
์ˆซ์ž๋กœ ๋ฐํž˜ mocap 120 Hz, ์นด๋ฉ”๋ผ 22 + 2๋Œ€, ๋งˆ์ปค 14๊ฐœ, \beta\in\mathbb{R}^{10}, MANO ๊ด€์ ˆ 22 + ์†๋ชฉ 6-DoF, \delta=1 cm, retargeting n_{\mathrm{iter}}=3000, DexHand 021 ๋Šฅ๋™ 12 / ๊ด€์ ˆ 20, ํ‰๊ฐ€ ์„ญ๋™ ๋ฌผ์ฒด ํฌ๊ธฐ์˜ ์ตœ๋Œ€ 10%, ๊ถค์ ๋‹น 50ํšŒ ์‹œ๋„, ์„ฑ๊ณต ๊ธฐ์ค€ 3 cm / 30ยฐ, Diffusion Policy ์ž…๋ ฅ 64 / 80์ฐจ์›, ํ•˜์ค‘ 0.5 N
๋ฐํžˆ์ง€ ์•Š์Œ \gamma ๊ฐ’(๋งˆ์ง€๋ง‰ ์ ‘์ด‰ ์ดํ›„ ์†๊ฐ€๋ฝ \gamma=1๋งŒ ๋ช…์‹œ), scale s, clip ๋ฒ”์œ„, ๋‘ policy์˜ ๋„คํŠธ์›Œํฌ ๊ตฌ์กฐ, PPO ํ•˜์ดํผํŒŒ๋ผ๋ฏธํ„ฐ, ๋ณด์ƒ ๊ฐ€์ค‘์น˜, Eq. 2์˜ w_\ast, synthesis์˜ randomization ๋ฒ”์œ„, sparse/multi-contact ํŒ์ • ์™ธ์˜ ๋ถ„ํ•  ๊ธฐ์ค€, Diffusion Policy ํ•™์Šต ์„ค์ •, ํ•™์Šต ์‹œ๊ฐ„ยทGPU, seed ์ˆ˜, ์‹ค๊ธฐ ์ œ์–ด ์ฃผํŒŒ์ˆ˜

์ง๊ด€: ์™œ ์ ‘์ด‰ ๊ตฌ์กฐ๊ฐ€ ์˜ฎ๊ฒจ์ง€๊ณ , ์™œ policy ํ•˜๋‚˜๋กœ ์ถฉ๋ถ„ํ•œ๊ฐ€

์ ‘์ด‰์€ ๋ฌผ์ฒด ์ขŒํ‘œ๊ณ„์— ์‚ฐ๋‹ค. ๊ด€์ ˆ๊ฐ์€ ์†์˜ ์ขŒํ‘œ๊ณ„์— ์‚ฌ๋Š” ์–‘์ด๋ผ ์†์ด ๋ฐ”๋€Œ๋ฉด ์˜๋ฏธ๊ฐ€ ๋ฐ”๋€๋‹ค. ๋ฐ˜๋ฉด โ€œ์—„์ง€ ๋์ด ๋ณ‘๋šœ๊ป‘ ์™ผ์ชฝ 2 cm ์ง€์ ์— ๋‹ฟ๋Š”๋‹คโ€๋Š” ๋ฌผ์ฒด ์ขŒํ‘œ๊ณ„์˜ ์‚ฌ์‹ค์ด๊ณ , ์–ด๋–ค ์†์œผ๋กœ ์žก๋“  ๊ฐ™์€ ๋ฌผ์ฒด๋ฅผ ๊ฐ™์€ ๋ฐฉ์‹์œผ๋กœ ๋‹ค๋ฃจ๋ ค๋ฉด ์ง€์ผœ์•ผ ํ•˜๋Š” ์กฐ๊ฑด์ด๋‹ค. Eq. 3์ด ์† ๋ชจ์–‘๊ณผ ๋…๋ฆฝ์ธ ์ด์œ ๊ฐ€ ์—ฌ๊ธฐ ์žˆ๋‹ค. ๋ชฉํ‘œ p_i๋Š” ๋ฌผ์ฒด ์œ„์˜ ์ ์ด๊ณ , ์†์€ ๊ทธ ์ ์— ๋‹ฟ์„ ํ‘œ๋ฉด์  c_i(u_t)๋งŒ ๋‚ด๋†“์œผ๋ฉด ๋œ๋‹ค.

1 cm ํ—ˆ์šฉ์˜ค์ฐจ๋Š” โ€œ๋Œ€๋žต ๊ฑฐ๊ธฐโ€๊นŒ์ง€๋งŒ ๋Œ๊ณ  ๊ฐ„๋‹ค. E_{\mathrm{tgt}}๋Š” ๊ฑฐ๋ฆฌ๊ฐ€ \delta ์•ˆ์ด๋ฉด 0์ด๋‹ค. ๊ทธ๋ž˜์„œ ์ตœ์ ํ™”๋Š” ํ‘œ๋ฉด์ ์„ ๋ชฉํ‘œ 1 cm ์•ˆ์œผ๋กœ ๋„ฃ๋Š” ์ˆœ๊ฐ„ ๊ทธ ์ด์ƒ ๋‹น๊ธฐ์ง€ ์•Š๊ณ , ๋‚˜๋จธ์ง€ ํ•ญ(๊ด€ํ†ตยทjitterยท๊ด€์ ˆ ํ•œ๊ณ„)์„ ์ค„์ด๋Š” ๋ฐ ์—ฌ์œ ๋ฅผ ์“ด๋‹ค. ์ด๋ ‡๊ฒŒ ๋ณด๋ฉด Table 5์˜ ๋ชจ์–‘์ด ์„ค๋ช…๋œ๋‹ค(ํ•„์ž์˜ ํ•ด์„). ์ด ๋ฐฉ๋ฒ•์˜ ํ‰๊ท  fidelity๊ฐ€ 14.7 mm๋กœ Vector(10.6 mm)๋ณด๋‹ค ๋‚˜์˜๊ณ  5 mm ์ด๋‚ด retention์ด 13.9%์— ๊ทธ์น˜๋Š” ๊ฒƒ์€ ๋ชฉ์ ํ•จ์ˆ˜๊ฐ€ ์• ์ดˆ์— 1 cm๋ณด๋‹ค ๊ฐ€๊น๊ฒŒ ๋ถ™์ด๋„๋ก ์„ค๊ณ„๋˜์ง€ ์•Š์•˜๊ธฐ ๋•Œ๋ฌธ์ด๊ณ , ๋Œ€์‹  jitter์™€ ๊ด€์ ˆ ํ•œ๊ณ„ ์œ„๋ฐ˜์ด ๊ฐ€์žฅ ๋‚ฎ์€ ์ถ•์— ๋“ ๋‹ค. ๋งˆ์ง€๋ง‰ ๋ช‡ mm์™€ ์ ‘์ด‰๋ ฅ์€ ๋’ค์˜ residual policy๊ฐ€ ์‹œ๋ฎฌ๋ ˆ์ด์…˜ ์•ˆ์—์„œ ๋งž์ถ˜๋‹ค.

policy ํ•˜๋‚˜๋กœ ์ถฉ๋ถ„ํ•œ ์ด์œ ๋Š” ๋ฌธ์ œ๊ฐ€ โ€œ๊ณ„ํšโ€์ด ์•„๋‹ˆ๋ผ โ€œ์ถ”์ โ€์ด๊ธฐ ๋•Œ๋ฌธ์ด๋‹ค. physics refinement policy๋Š” ๋ฏธ๋ž˜์˜ reference ๋ฌผ์ฒด ์šด๋™์„ ๊ด€์ธก์œผ๋กœ ๋ฐ›๋Š”๋‹ค. ๋ฌด์—‡์„ ํ• ์ง€๋Š” reference๊ฐ€ ์ด๋ฏธ ์ •ํ–ˆ๊ณ , policy๋Š” ๊ทธ๊ฒƒ์„ ๋ฌผ๋ฆฌ์ ์œผ๋กœ ์„ฑ๋ฆฝ์‹œํ‚ค๋Š” ์ž‘์€ ๋ณด์ •๋งŒ ๋‚ธ๋‹ค. ๊ณผ์ œ๊ฐ€ ๋‹ฌ๋ผ๋„ โ€œ๋ฌผ์ฒด๋ฅผ ๊ธฐ๋ก๋œ ๊ถค์ ์— ๋ถ™์—ฌ ๋‘๊ธฐ ์œ„ํ•ด ์†๊ฐ€๋ฝ์„ ์กฐ๊ธˆ ๋” ์˜ค๋ฏ€๋ฆฐ๋‹คโ€ ๊ฐ™์€ ๋ณด์ •์˜ ์ข…๋ฅ˜๋Š” ๋น„์Šทํ•˜๋ฏ€๋กœ ๊ณผ์ œ ๊ฐ„ ๊ณต์œ ๊ฐ€ ๊ฐ€๋Šฅํ•˜๋‹ค๋Š” ๊ฒƒ์ด ๋…ผ๋ฌธ์˜ ๋…ผ๋ฆฌ๋‹ค(ยง3.2). Eq. 1์˜ decay \gamma<1์€ ๋ณด์ •์ด ์Œ“์—ฌ reference์—์„œ ๋ฉ€๋ฆฌ ๋– ๋‚˜๋Š” ๊ฒƒ์„ ๋ง‰๋Š” ์žฅ์น˜์ด๊ณ , ๋งˆ์ง€๋ง‰ ์ ‘์ด‰ ๋’ค์— ์†๊ฐ€๋ฝ๋งŒ \gamma=1๋กœ ๋ฐ”๊พธ๋Š” ๊ฒƒ์€ โ€œgrasp๋ฅผ ๋งŒ๋“  ๋ณด์ •์€ ์œ ์ง€ํ•ด์•ผ ํ•œ๋‹คโ€๋Š” ์˜ˆ์™ธ ์ฒ˜๋ฆฌ๋‹ค.

์ด ์ง๊ด€์˜ ํ•œ๊ณ„๋„ ๊ฐ™์€ ๊ณณ์—์„œ ๋‚˜์˜จ๋‹ค. policy๊ฐ€ reference๋ฅผ ๋”ฐ๋ผ๊ฐ€๋Š” ๊ตฌ์กฐ๋ผ๋ฉด reference ์ž์ฒด๊ฐ€ ํ‹€๋ฆฐ ๊ฒฝ์šฐ, ์˜ˆ์ปจ๋Œ€ retargeting์ด ๋ฌผ๋ฆฌ์ ์œผ๋กœ ๋ถˆ๊ฐ€๋Šฅํ•œ grasp๋ฅผ ๋งŒ๋“  ๊ฒฝ์šฐ์—๋Š” bounded envelope ์•ˆ์˜ ๋ณด์ •์œผ๋กœ ๋ณต๊ตฌํ•  ์ˆ˜ ์—†๋‹ค. ์ด ๋ฐฉ๋ฒ•์ด โ€œretargeting์ด ์ข‹์•„์„œโ€ ์„ฑ๊ณตํ•˜๋Š”์ง€ โ€œresidual์ด ๊ฐ•ํ•ด์„œโ€ ์„ฑ๊ณตํ•˜๋Š”์ง€๋ฅผ ๊ฐ€๋ฅด๋ ค๋ฉด ๋‹ค๋ฅธ retargeter์˜ reference๋ฅผ ๊ฐ™์€ residual ๋‹จ๊ณ„์— ๋„ฃ์–ด ๋ด์•ผ ํ•˜๋Š”๋ฐ, ๊ทธ ์‹คํ—˜์ด ์—†๋‹ค(์•„๋ž˜ ์‹คํ—˜ ์ ˆ๊ณผ ์•ฝ์  ์ ˆ).

์‹คํ—˜ (ยง4)

๋…ผ๋ฌธ์€ ์‹คํ—˜์„ contribution๋งˆ๋‹ค ํ•˜๋‚˜์”ฉ, ์„ธ ์งˆ๋ฌธ์œผ๋กœ ์ง ๋‹ค. (1) ๋‹จ์ผ amortized policy๊ฐ€ ๊ทœ๋ชจ์—์„œ physics gap์„ ๋‹ซ๊ณ  ํ•™์Šต ๋ถ„ํฌ ๋ฐ–์œผ๋กœ ์ผ๋ฐ˜ํ™”ํ•˜๋Š”๊ฐ€(ยง4.1). (2) ์ ‘์ด‰ ๊ธฐ๋ฐ˜ ์ „์ด๊ฐ€ ๊ธฐํ•˜ retargeting๋ณด๋‹ค embodiment gap์„ ์ž˜ ๋‹ซ๋Š”๊ฐ€, ๋‹จ๋…์œผ๋กœ ๊ทธ๋ฆฌ๊ณ  ์† ๋ชจ์–‘์„ ๋ฐ”๊ฟ” ๊ฐ€๋ฉฐ(ยง4.2). (3) ๊ฒฐ๊ณผ ๋ฐ์ดํ„ฐ๊ฐ€ downstream ํ•™์Šต์„ ๋•๊ณ  ์ „์ฒด ํŒŒ์ดํ”„๋ผ์ธ์ด ์‹ค๋กœ๋ด‡ ํ•™์Šต ๋ฐ์ดํ„ฐ ์—†์ด ํ•˜๋“œ์›จ์–ด์—์„œ ๋„๋Š”๊ฐ€(ยง4.3). ์‹œ๋ฎฌ๋ ˆ์ด์…˜ ์„ฑ๊ณต ํŒ์ •์€ ํ•˜๋‚˜๋กœ ํ†ต์ผํ•œ๋‹ค: ์‹คํ–‰ ๋‚ด๋‚ด ์–ด๋–ค ๋ฌผ์ฒด๋„ mocap reference์—์„œ 3 cm ๋„˜๊ฒŒ ๋ฒ—์–ด๋‚˜์ง€ ์•Š๊ณ , ๋ฐฉํ–ฅ ์ œ์•ฝ ๊ณผ์ œ๋Š” ๋งˆ์ง€๋ง‰ ํ”„๋ ˆ์ž„ ์˜ค์ฐจ 30ยฐ ์ดํ•˜.

4.1 ๊ทœ๋ชจ์—์„œ์˜ ๋ฌผ๋ฆฌ ์ผ๊ด€์„ฑ

์…‹์—…. ํ•œ ์† ๋ฐ์ดํ„ฐ์…‹์€ mocap ๊ถค์  25,454๊ฐœ(๋ฌผ์ฒด 14๊ฐœ, ๋ฌผ์ฒด-์ œ์Šค์ฒ˜ 85์Œ, ๋ฌผ์ฒด์™€ ์ œ์Šค์ฒ˜์˜ ์ •์˜๋Š” DexCanvas๋ฅผ ๋”ฐ๋ฅธ๋‹ค), ์–‘์† ๋ฐ์ดํ„ฐ์…‹์€ 250 ๊ถค์ (25๊ณผ์ œ, ๋ฌผ์ฒด 12๊ฐœ)์ด๋‹ค. ํ‰๊ฐ€์—์„œ ๋ฌผ์ฒด-์ œ์Šค์ฒ˜ ์Œ๋งˆ๋‹ค ๋ฌผ์ฒด ํฌ๊ธฐ์˜ ์ตœ๋Œ€ 10%๋ฅผ ์„ญ๋™ํ•˜๊ณ  ๊ถค์ ๋งˆ๋‹ค 50ํšŒ ์‹œ๋„ํ•œ๋‹ค. ์‚ฐ์ˆ ๋กœ๋Š” ํ•œ ์† ์„ค์ •๋งŒ ์•ฝ 127๋งŒ rollout์ด๋‹ค(ํ•„์ž ๊ณ„์‚ฐ).

Table 1: ํ•œ ์† ์„ฑ๊ณต๋ฅ (ํ‰๊ท ์€ ๋ฌผ์ฒด 14๊ฐœ์˜ ๋น„๊ฐ€์ค‘ ํ‰๊ท )

๋ฌผ์ฒด Replay With residual ์Œ ์ˆ˜
Large clamp 23.99% 62.80% 5
Cylinder1 16.49% 68.20% 8
Cube1 15.44% 77.20% 7
Mayonnaise 10.70% 56.40% 5
Banana 9.86% 59.40% 7
Cube2 9.12% 81.10% 8
Cylinder2 8.88% 66.40% 7
Power drill 5.85% 33.50% 6
Cylinder3 0.66% 35.50% 9
iPhone 0.61% 48.80% 8
Bowl 0.00% 41.50% 3
Cuboid1 0.00% 60.00% 5
Cylinder4 0.00% 68.80% 3
Sphere1 0.00% 70.80% 4
ํ‰๊ท  7.26% 59.31% 85

์นธ ๊ฐ’์œผ๋กœ ํ‰๊ท ์„ ๋‹ค์‹œ ๋‚ด๋ฉด 7.257%์™€ 59.314%๋กœ ํ‘œ์™€ ๋งž๊ณ , ์Œ ์ˆ˜ ํ•ฉ๋„ 85๋‹ค. ์Œ ์ˆ˜๋กœ ๊ฐ€์ค‘ํ•˜๋ฉด 7.81%์™€ 59.25%๋กœ ๊ฒฐ๋ก ์€ ๊ฐ™๋‹ค(ํ•„์ž ๊ณ„์‚ฐ). ๋ชจ๋“  ๋ฌผ์ฒด๊ฐ€ ๊ฐœ์„ ๋˜๊ณ  replay๊ฐ€ ํ•œ ๋ฒˆ๋„ ์„ฑ๊ณตํ•˜์ง€ ๋ชปํ•œ 4๊ฐœ ๋ฌผ์ฒด๋„ 41.5~70.8%์— ์ด๋ฅธ๋‹ค. ๋‹ค๋งŒ ์ด ํ‘œ์—์„œ ๋น„๊ต ๋Œ€์ƒ์ธ โ€œReplayโ€๋Š” kinematic reference๋ฅผ PD๋กœ ๊ทธ๋Œ€๋กœ ์‹คํ–‰ํ•œ ๊ฒƒ์ด๋‹ค. ๋ฌผ๋ฆฌ ๋ณด์ •์„ ์ „ํ˜€ ํ•˜์ง€ ์•Š๋Š” ํ•˜ํ•œ์ด๋ฏ€๋กœ, ์ด ํ‘œ๊ฐ€ ๋ณด์—ฌ ์ฃผ๋Š” ๊ฒƒ์€ โ€œresidual ๋ณด์ •์ด ํ•„์š”ํ•˜๊ณ  ํšจ๊ณผ๊ฐ€ ์žˆ๋‹คโ€์ด์ง€ โ€œ๋‹ค๋ฅธ physics refinement ๋ฐฉ๋ฒ•๋ณด๋‹ค ๋‚ซ๋‹คโ€๊ฐ€ ์•„๋‹ˆ๋‹ค. ์ €์ž๋Š” ์ด ์„ฑ๊ณต๋ฅ ์ด data synthesis์˜ ํ’ˆ์งˆ๋„ ๋ฐ˜์˜ํ•œ๋‹ค๊ณ  ์“ด๋‹ค. ํ•™์Šต๋œ policy๊ฐ€ ๋ฌผ์ฒด์™€ ์†์˜ ์ดˆ๊ธฐ ์ž์„ธ๋ฅผ ๋ฒ”์œ„ ์•ˆ์—์„œ ์„ญ๋™ํ•ด ๊ถค์ ์„ ํ•ฉ์„ฑํ•˜๊ณ  ๊ฐ™์€ ์„ฑ๊ณต ๊ธฐ์ค€์œผ๋กœ ๊ฑธ๋Ÿฌ ๋‚ด๊ธฐ ๋•Œ๋ฌธ์ด๋‹ค.


Fig. 3: ์–‘์† ํ‰๊ฐ€๋ฅผ ๊ณผ์ œ ์œ ํ˜•๋ณ„๋กœ ๋ฌถ์€ ๊ฒฐ๊ณผ(25๊ณผ์ œ, 250 ๊ถค์ , ๋ฌผ์ฒด ํฌ๊ธฐ ยฑ10% ์„ญ๋™). ํšŒ์ƒ‰ ์›์€ residual ์—†์Œ, ํŒŒ๋ž€ ์ ์€ residual ์žˆ์Œ. ์™ผ์ชฝ์€ task-level, ์˜ค๋ฅธ์ชฝ์€ attempt-level ์„ฑ๊ณต๋ฅ .

๋ณธ๋ฌธ์ด ์ˆซ์ž๋กœ ์ ์€ ๊ฒƒ์€ Overall๋ฟ์ด๋‹ค(task-level 16.0% โ†’ 62.4%, attempt-level 8.4% โ†’ 43.9%). ์œ ํ˜•๋ณ„ ๊ฐ’์„ ๊ทธ๋ฆผ์—์„œ ์ฝ์œผ๋ฉด ๋‹ค์Œ๊ณผ ๊ฐ™๋‹ค(ํ•„์ž ํŒ๋…, ์•ฝ ยฑ2%p).

์œ ํ˜• Task-level (์—†์Œ โ†’ ์žˆ์Œ) Attempt-level (์—†์Œ โ†’ ์žˆ์Œ)
Align โ‰ˆ10 โ†’ โ‰ˆ80 โ‰ˆ1 โ†’ โ‰ˆ75
Balance โ‰ˆ0 โ†’ โ‰ˆ70 โ‰ˆ0 โ†’ โ‰ˆ50
Stand โ‰ˆ16 โ†’ โ‰ˆ67 โ‰ˆ10 โ†’ โ‰ˆ45
Center โ‰ˆ22 โ†’ โ‰ˆ63 โ‰ˆ10 โ†’ โ‰ˆ46
Lay โ‰ˆ10 โ†’ โ‰ˆ33 โ‰ˆ8 โ†’ โ‰ˆ18
Overall 16.0 โ†’ 62.4 8.4 โ†’ 43.9

โ€œ์–ด๋–ค ์œ ํ˜•์—์„œ๋„ ํ‡ด๋ณด๊ฐ€ ์—†๋‹คโ€๋Š” ๋ณธ๋ฌธ ์„œ์ˆ ์€ ๊ทธ๋ฆผ๊ณผ ๋งž๋Š”๋‹ค. ๊ทธ๋Ÿฌ๋‚˜ task-level๊ณผ attempt-level์˜ ์ •์˜๊ฐ€ ๋…ผ๋ฌธ ์–ด๋””์—๋„ ์—†๋‹ค. task-level์ด attempt-level๋ณด๋‹ค ์ผ๊ด€๋˜๊ฒŒ ๋†’์œผ๋ฏ€๋กœ task-level์ด ๋” ๊ด€๋Œ€ํ•œ ์ง‘๊ณ„๋ผ๋Š” ๊ฒƒ๋งŒ ์ถ”์ •ํ•  ์ˆ˜ ์žˆ๋‹ค. ์ดˆ๋ก์ด ์ธ์šฉํ•˜๋Š” 16.0% โ†’ 62.4%๋Š” ์ด ์ •์˜ ์—†๋Š” task-level ๊ฐ’์ด๋‹ค. Lay ์œ ํ˜•์€ residual์„ ์จ๋„ attempt-level ์•ฝ 18%๋กœ, ์–‘์† ๊ณผ์ œ ์ค‘ ์ผ๋ถ€๋Š” ์—ฌ์ „ํžˆ ๋Œ€๋ถ€๋ถ„ ์‹คํŒจํ•œ๋‹ค.

Table 2: ์ฒ˜์Œ ๋ณด๋Š” YCB ๋ฌผ์ฒด(fine-tuning ์—†์Œ)

YCB ๋ฌผ์ฒด Trials ์„ฑ๊ณต๋ฅ 
Tomato soup can 1,289 46.5%
Tuna fish can 1,287 88.7%
Pudding box 1,287 63.6%
Gelatin box 1,295 83.5%
Potted meat can 1,238 67.8%
Mug 1,287 84.6%
Pooled 7,683 72.4%

์ด ํ‘œ์—๋Š” ์„ธ ๊ฐ€์ง€ ๋‹จ์„œ๊ฐ€ ํ•„์š”ํ•˜๋‹ค.

  • ์ฒ˜์Œ ๋ณด๋Š” ๋ฌผ์ฒด๊ฐ€ ํ•™์Šต ๋ฌผ์ฒด๋ณด๋‹ค ์ž˜ ๋œ๋‹ค. pooled 72.4%๋Š” ํ•™์Šต ๋ฌผ์ฒด ํ‰๊ท  59.31%(Table 1)๋ณด๋‹ค 13 %p ๋†’๋‹ค. ์ผ๋ฐ˜ํ™” ์‹คํ—˜์—์„œ ๋ถ„ํฌ ๋ฐ–์ด ๋ถ„ํฌ ์•ˆ๋ณด๋‹ค ๋†’์€ ์—ญ์ „์ธ๋ฐ ๋…ผ๋ฌธ์€ ์ด๋ฅผ ์„ค๋ช…ํ•˜์ง€ ์•Š๋Š”๋‹ค. ์บ”ยท์ƒ์žยท๋จธ๊ทธ์ฒ˜๋Ÿผ ํ˜•์ƒ์ด ๋‹จ์ˆœํ•œ ๋ฌผ์ฒด๋ฅผ ๊ณจ๋ž์„ ์ˆ˜ ์žˆ๊ณ , ํ‰๊ฐ€ ํ”„๋กœํ† ์ฝœ์ด Table 1๊ณผ ๋‹ค๋ฅผ ์ˆ˜๋„ ์žˆ๋‹ค.
  • ํ”„๋กœํ† ์ฝœ์ด ์„œ์ˆ ๋˜์ง€ ์•Š์•˜๋‹ค. ์ฒ˜์Œ ๋ณด๋Š” ๋ฌผ์ฒด์—๋Š” mocap ์‹œ์—ฐ์ด ์—†์„ ํ…๋ฐ ์–ด๋–ค reference(์–ด๋А ์ œ์Šค์ฒ˜์˜ ์–ด๋А ๊ถค์ )๋ฅผ ์ผ๋Š”์ง€๊ฐ€ ์—†๋‹ค. trial ์ˆ˜(1,238~1,295)๋Š” โ€œ๊ถค์ ๋‹น 50ํšŒโ€์˜ ๋ฐฐ์ˆ˜๊ฐ€ ์•„๋‹ˆ๋ผ์„œ Table 1๊ณผ ๊ฐ™์€ ๋ฐฉ์‹์ด ์•„๋‹˜์„ ์‹œ์‚ฌํ•œ๋‹ค.
  • ๊ฐ™์€ ๊ทธ๋ฃน์˜ ๋‹ค๋ฅธ ๋…ผ๋ฌธ๊ณผ ํ‘œ๊ฐ€ ๊ฐ™๋‹ค. DexCanvas(arXiv:2510.15786) v3์˜ Table 2๊ฐ€ ๊ฐ™์€ ๋ฌผ์ฒด 6๊ฐœ, ๊ฐ™์€ trial ์ˆ˜, ๊ฐ™์€ ๋ฌผ์ฒด๋ณ„ ์„ฑ๊ณต๋ฅ ์„ ์‹ฃ๊ณ  ์„ฑ๊ณต ํšŸ์ˆ˜(ํ•ฉ 5,567)๊นŒ์ง€ ์ ๋Š”๋‹ค. 5,567/7,683์€ 72.46%๋ผ ๋ฐ˜์˜ฌ๋ฆผํ•˜๋ฉด DexCanvas๊ฐ€ ์“ด 72.5%์ด๊ณ , ์ด ๋…ผ๋ฌธ์˜ 72.4%๋Š” ์ ˆ์‚ฌ๊ฐ’์œผ๋กœ ๋ณด์ธ๋‹ค(ํ•„์ž ๊ณ„์‚ฐ). ๊ฐ™์€ ์‹คํ—˜ ๊ฒฐ๊ณผ๊ฐ€ ๋‘ ๋…ผ๋ฌธ์— ํ•จ๊ป˜ ์‹ค๋ ธ๋‹ค๋Š” ์‚ฌ์‹ค์€ ์ด ๋…ผ๋ฌธ ๋ณธ๋ฌธ์— ๋‚˜์˜ค์ง€ ์•Š๋Š”๋‹ค.

์ €์ž๋Š” ์ด ๊ฒฐ๊ณผ๋ฅผ โ€œdistinguishes a learned repair operator from a per-dataset artifactโ€๋กœ ํ•ด์„ํ•œ๋‹ค. ์ฒ˜์Œ ๋ณด๋Š” ๋ฌผ์ฒด์—์„œ ์ˆ˜์„ ์ด ๋œ๋‹ค๋Š” ๊ฒƒ ์ž์ฒด๋Š” ์˜๋ฏธ ์žˆ๋Š” ์ฆ๊ฑฐ๋‹ค. ๋‹ค๋งŒ ์œ„ ๋‹จ์„œ ๋•Œ๋ฌธ์— โ€œํ•™์Šต ๋ถ„ํฌ ๋ฐ–์—์„œ๋„ ํ•™์Šต ๋ถ„ํฌ๋งŒํผ ๋œ๋‹คโ€๋Š” ๊ฐ•ํ•œ ํ˜•ํƒœ๋กœ ์ฝ์œผ๋ฉด ์•ˆ ๋œ๋‹ค.

4.2 ์† ๋ชจ์–‘์„ ๋„˜๋Š” ์ ‘์ด‰ ๊ธฐ๋ฐ˜ ์ „์ด

๋จผ์ € retargeting ๋‹จ๊ณ„๋งŒ ๋–ผ์–ด ๋น„๊ตํ•œ๋‹ค. MANO ๊ถค์  100๊ฐœ๋ฅผ DexHand 021๋กœ ์˜ฎ๊ธฐ๋ฉฐ DexPilot, dex-retargeting(Position, Vector), Spider์™€ ๋น„๊ตํ•œ๋‹ค. Spider์˜ M์€ ๋‚ด๋ถ€ ๊ถค์  ์ตœ์ ํ™”๊ธฐ์˜ ๋ฐ˜๋ณต ์ƒํ•œ์ด๋‹ค.

Table 5: Retargeting ๋น„๊ต(MANO 100 ๊ถค์  โ†’ DexHand 021)

๋ฐฉ๋ฒ• Time (s) Tput. (f/s) Fidelity (mm)โ†“ Ret. โ‰ค5 mmโ†‘ Ret. โ‰ค10 mmโ†‘ Jt. viol.โ†“ Jitter (rad/s)โ†“
DexPilot 186.8 126.2 15.8 2.0% 35.1% 21.4% 2.94
Position 21.8 1081.7 15.7 0.3% 26.9% 10.9% 1.63
Vector 169.2 139.4 10.6 32.0% 53.5% 29.9% 3.46
Spider M=16 3742.0 6.3 25.5 0.6% 5.4% 2.66% 3.03
Spider M=32 3320.8 7.1 25.3 0.8% 6.8% 2.68% 3.15
Spider default 1079.3 27.5 49.9 0.2% 1.4% 3.27% 0.73
Ours (n_{\mathrm{iter}}=3000) 8513.2 5.5 14.7 13.9% 44.6% 3.2% 0.53

๊ตต์€ ๊ธ€์”จ๋Š” ์—ด๋ณ„ ์ตœ๊ณ ๊ฐ’์„ ํ•„์ž๊ฐ€ ๋‹ค์‹œ ํ‘œ์‹œํ•œ ๊ฒƒ์ด๋‹ค. ๋ณธ๋ฌธ์˜ ๊ฒฐ๋ก ์€ โ€œthe only method that jointly achieves high fidelity, high retention, low violations, and low jitterโ€๋‹ค. ๋„ค ์ง€ํ‘œ๋ฅผ ๋™์‹œ์— ์ƒ์œ„๊ถŒ์— ๋‘๋Š” ๋ฐฉ๋ฒ•์ด ์ด๊ฒƒ๋ฟ์ด๋ผ๋Š” ๋ง์€ ํ‘œ์™€ ๋งž๋Š”๋‹ค. ํ•˜์ง€๋งŒ ํ‘œ๋ฅผ ์—ด๋ณ„๋กœ ์ฝ์œผ๋ฉด ์ด ๋ฌธ์žฅ์ด ๊ฐ€๋ฆฌ๋Š” ๊ฒƒ์ด ์…‹ ์žˆ๋‹ค.

  • ์–ด๋А ์—ด์—์„œ๋„ ๋‹จ๋… 1์œ„๋Š” jitter ํ•˜๋‚˜๋‹ค. fidelityยทretention์€ Vector๊ฐ€, ๊ด€์ ˆ ํ•œ๊ณ„ ์œ„๋ฐ˜์€ Spider M=16(2.66%)ยทM=32(2.68%)๊ฐ€ ๋” ๋‚ซ๋‹ค. ๊ด€์ ˆ ํ•œ๊ณ„ ์œ„๋ฐ˜์€ 3.2%๋กœ Spider์™€ ๋น„์Šทํ•œ ์ˆ˜์ค€์ด๋‹ค.
  • ๊ฐ€์žฅ ๋А๋ฆฌ๋‹ค. Time 8,513 s๋Š” Spider default(1,079 s)์˜ 7.9๋ฐฐ, Spider M=16(3,742 s)์˜ 2.3๋ฐฐ, Position(21.8 s)์˜ 390๋ฐฐ๋‹ค(ํ•„์ž ๊ณ„์‚ฐ). ๋ณธ๋ฌธ์€ TimeยทTput ์—ด์„ ํ•œ ๋ฒˆ๋„ ์–ธ๊ธ‰ํ•˜์ง€ ์•Š๋Š”๋‹ค. ์„œ๋ก ๊ณผ ๊ด€๋ จ ์—ฐ๊ตฌ๋Š” SPIDER๋ฅผ โ€œat high per-trajectory costโ€๋กœ ๋น„ํŒํ•˜๋Š”๋ฐ, ๊ฐ™์€ ๋…ผ๋ฌธ์˜ ํ‘œ์—์„œ๋Š” ์ด ๋ฐฉ๋ฒ•์ด SPIDER๋ณด๋‹ค ๋น„์‹ธ๋‹ค. ๊ฒŒ๋‹ค๊ฐ€ SPIDER์˜ ์ถœ๋ ฅ์€ ์ด๋ฏธ ๋™์—ญํ•™์ ์œผ๋กœ ๊ฐ€๋Šฅํ•œ ๊ถค์ ์ด๊ณ , ์ด ๋ฐฉ๋ฒ•์˜ ์ถœ๋ ฅ์€ ์•„์ง residual RL ํ•™์Šต์„ ๊ฑฐ์ณ์•ผ ํ•˜๋Š” kinematic reference๋‹ค. ๋น„์šฉ ๋น„๊ต๋Š” ์ด ๋ฐฉ๋ฒ•์— ๋” ๋ถˆ๋ฆฌํ•˜๊ฒŒ ๊ธฐ์šธ์–ด ์žˆ๋‹ค.
  • Time๊ณผ Tput์ด ์„œ๋กœ ๋งž์ง€ ์•Š๋Š” ํ–‰์ด ์žˆ๋‹ค. Time ร— Tput์€ ์ฒ˜๋ฆฌํ•œ frame ์ˆ˜์—ฌ์•ผ ํ•œ๋‹ค. DexPilotยทPositionยทVectorยทSpider M=16ยทM=32์˜ ๋‹ค์„ฏ ํ–‰์€ ๋ชจ๋‘ ์•ฝ 23,580 frame์œผ๋กœ ์ผ์น˜ํ•˜๋Š”๋ฐ, Ours๋Š” 46,823, Spider default๋Š” 29,681์ด๋‹ค(ํ•„์ž ๊ณ„์‚ฐ). ๋‘ ์—ด์˜ ์ •์˜๊ฐ€ ๋…ผ๋ฌธ์— ์—†์–ด ์–ด๋А ์ชฝ์ด ๋งž๋Š”์ง€ ํŒ๋‹จํ•  ์ˆ˜ ์—†๋‹ค.

์ง€ํ‘œ ์„ค๊ณ„์—๋„ ๊ธฐ์šธ๊ธฐ๊ฐ€ ์žˆ๋‹ค. fidelity์™€ retention์€ MANO ์ ‘์ด‰ ์œ„์น˜๊นŒ์ง€์˜ ๊ฑฐ๋ฆฌ๋กœ ์ •์˜๋œ๋‹ค. ์ด ๋ฐฉ๋ฒ•์€ ๊ทธ ๊ฑฐ๋ฆฌ๋ฅผ ์ง์ ‘ ์ตœ์†Œํ™”ํ•˜๊ณ (Eq. 3), DexPilotยทPositionยทVector๋Š” ์†๋ ์œ„์น˜ยท๋ฒกํ„ฐ๋ฅผ ๋งž์ถœ ๋ฟ ๋ฌผ์ฒด ์ ‘์ด‰ ์œ„์น˜๋ฅผ ๋ณด์ง€ ์•Š๋Š”๋‹ค. ์ง€ํ‘œ๊ฐ€ ์ด ๋ฐฉ๋ฒ•์˜ ๋ชฉ์ ํ•จ์ˆ˜์™€ ๊ฐ™์€ ๊ณต๊ฐ„์— ์žˆ๋Š”๋ฐ๋„ Vector๊ฐ€ fidelity์—์„œ ์ด๊ธด๋‹ค๋Š” ๊ฒƒ์€, ์•ž์˜ ์ง๊ด€ ์ ˆ์—์„œ ๋งํ•œ 1 cm ํ—ˆ์šฉ์˜ค์ฐจ์˜ ์˜ํ–ฅ์œผ๋กœ ์ฝํžŒ๋‹ค. SPIDER๋Š” ๋ฌผ๋ฆฌ ์‹œ๋ฎฌ๋ ˆ์ด์…˜ ์•ˆ์—์„œ ๊ถค์ ์„ ์ฐพ์œผ๋ฏ€๋กœ kinematic ๋ชฉํ‘œ์—์„œ ๋ฉ€์–ด์ง€๋Š” ๊ฒƒ์ด ์ •์ƒ์ด๊ณ , ๊ทธ๋ž˜์„œ kinematic ์ง€ํ‘œ๋งŒ์œผ๋กœ SPIDER๋ฅผ ํ‰๊ฐ€ํ•˜๋ฉด SPIDER๊ฐ€ ์ž˜ํ•˜๋Š” ๊ฒƒ(๋™์—ญํ•™์  ์‹คํ˜„ ๊ฐ€๋Šฅ์„ฑ)์ด ํ‘œ์—์„œ ๋น ์ง„๋‹ค.

๋…ผ๋ฌธ๋„ โ€œComponent metrics alone do not imply usable referencesโ€๋ผ๊ณ  ์ธ์ •ํ•˜๊ณ  ํ๋ฃจํ”„ ์žฌ๊ตฌ์„ฑ์œผ๋กœ ๋„˜์–ด๊ฐ„๋‹ค.

Table 3: Cross-embodiment ์žฌ๊ตฌ์„ฑ(์†๋‹น 200 ์—ํ”ผ์†Œ๋“œ)

์† Replay Residual ฮ” (pp) ๋ฐฐ์œจ
DexHand 021 0.5% 79.0% +78.5 158ร—
DexHand 021Pro 3.5% 80.0% +76.5 23ร—
Shadow Hand 14.5% 82.5% +68.0 5.7ร—
Allegro Hand 50.0% 76.5% +26.5 1.5ร—
Overall 17.1% 79.5% +62.4 4.6ร—

ํ•œ ์† ๊ถค์  200๊ฐœ(๋ฌผ์ฒด 3๊ฐœ, ์ œ์Šค์ฒ˜ 7๊ฐœ, 14์Œ)๋ฅผ ๋„ค ์†์— ๊ฐ™์€ ํ•™์Šต ์„ค์ •์œผ๋กœ, ์†๋ณ„ ํŠœ๋‹ ์—†์ด ์˜ฎ๊ธด๋‹ค. ์„ฑ๊ณต์€ ๋ฌผ์ฒด๋ฅผ ์žก๊ณ  ๋“ค์–ด ์•ˆ์ •์ ์œผ๋กœ ๋‚ด๋ ค๋†“๋˜ ๊ฐ™์€ 3 cm / 30ยฐ ๊ธฐ์ค€์„ ์ง€ํ‚ค๋Š” ๊ฒƒ์ด๋‹ค. ํ‰๊ท ๊ณผ ๋ฐฐ์œจ์€ ์นธ ๊ฐ’๊ณผ ๋งž๋Š”๋‹ค(17.125%, 79.5%, 79.5/17.1 โ‰ˆ 4.6). ์ดˆ๋ก์˜ โ€œ+62.4 ppโ€๋Š” ์ด ํ‘œ์˜ ๊ฐ’์ด๊ณ , ์ดˆ๋ก์˜ ์–‘์† ๊ฒฐ๊ณผ โ€œ16.0%โ†’62.4%โ€์˜ 62.4์™€๋Š” ์šฐ์—ฐํžˆ ๊ฐ™์€ ์ˆซ์ž์ผ ๋ฟ ๋‹ค๋ฅธ ์–‘์ด๋‹ค.


Fig. 6: ๋„ค ๋กœ๋ด‡ ์†(DexHand 021, 021Pro, Shadow, Allegro)์ด ๊ฐ™์€ retargeted grasp-and-lift reference๋ฅผ residual policy๋กœ ์‹คํ–‰ํ•˜๋Š” ์žฅ๋ฉด. ๊ทธ๋ฆผ์˜ ์—ด์€ ApproachยทContactยทGraspยทLiftยทRelease์ธ๋ฐ, ์บก์…˜์€ ๋‹ค์„ฏ ๋‹จ๊ณ„๋ฅผ โ€œStart, Approach, Grasp, Lift, Releaseโ€๋กœ ์ ์–ด ๊ทธ๋ฆผ๊ณผ ๋‹ค๋ฅด๋‹ค.

์ด ํ‘œ๋Š” โ€œ๋‹จ์ผ ํ•™์Šต ๋ ˆ์‹œํ”ผ๊ฐ€ ์† ๋ชจ์–‘์„ ๋„˜์–ด ์ผ๋ฐ˜ํ™”ํ•œ๋‹คโ€๋Š” ๋ฐ๋Š” ์ข‹์€ ์ฆ๊ฑฐ๋‹ค. ๊ทธ๋Ÿฌ๋‚˜ ยง4 ์„œ๋‘์˜ ์งˆ๋ฌธ (2), ์ฆ‰ ์ ‘์ด‰ ๊ธฐ๋ฐ˜ ์ „์ด๊ฐ€ ๊ธฐํ•˜ retargeting๋ณด๋‹ค embodiment gap์„ ๋” ์ž˜ ๋‹ซ๋Š”๊ฐ€์—๋Š” ๋‹ตํ•˜์ง€ ์•Š๋Š”๋‹ค. ๋น„๊ต ๋Œ€์ƒ์ด ์ž๊ธฐ ๋ฐฉ๋ฒ•์˜ replay๋ฟ์ด๊ณ , DexPilot์ด๋‚˜ Vector๋กœ ๋งŒ๋“  reference์— ๊ฐ™์€ residual์„ ๋ถ™์ธ ์กฐ๊ฑด์ด ์—†๊ธฐ ๋•Œ๋ฌธ์ด๋‹ค. ์˜คํžˆ๋ ค residual ๋’ค์— ๋„ค ์†์ด 76.5~82.5%๋กœ ์ˆ˜๋ ดํ•˜๊ณ  replay๊ฐ€ 50%์˜€๋˜ Allegro์™€ 0.5%์˜€๋˜ DexHand 021์ด ๊ฑฐ์˜ ๊ฐ™์€ ๊ณณ์— ๋„๋‹ฌํ•œ๋‹ค๋Š” ์‚ฌ์‹ค์€, ์ตœ์ข… ์„ฑ๊ณต๋ฅ ์„ ์ •ํ•˜๋Š” ๊ฒƒ์ด retargeting๋ณด๋‹ค residual ๋‹จ๊ณ„์ผ ์ˆ˜ ์žˆ๋‹ค๋Š” ํ•ด์„๋„ ํ—ˆ์šฉํ•œ๋‹ค. ์†๊ฐ€๋ฝ์ด ๋„ท๋ฟ์ธ Allegro์—์„œ replay๊ฐ€ ๊ฐ€์žฅ ๋†’์€(50%) ์ด์œ ๋„ ๋…ผ๋ฌธ์€ ๋งํ•˜์ง€ ์•Š๋Š”๋‹ค.

4.3 ๋ฐ์ดํ„ฐ์—์„œ ๋ฐฐํฌ๊นŒ์ง€

Table 4: Diffusion Policy, ์ ‘์ด‰๋ ฅ ์œ ๋ฌด(0.5 N ํ•˜์ค‘, 3 cm / 30ยฐ ๊ธฐ์ค€)

๋ฌผ์ฒด(์ด ํ‘œ ์•ˆ์˜ ๋ฒˆํ˜ธ) w/o ct. (64-d) w/ ct. (80-d) ฮ”
Cube1 35.2% 42.4% +7.2
Cylinder1 43.6% 60.4% +16.8
Cube2 30.8% 50.0% +19.2
Cylinder2 29.2% 33.2% +4.0
ํ‰๊ท  34.7% 46.5% +11.8

64์ฐจ์› kinematic ์ž…๋ ฅ์˜ ๊ธฐ์ค€์„ ๊ณผ, ํ”„๋ ˆ์ž„๋ณ„ ์ ‘์ด‰๋ ฅ 16์ฐจ์›์„ ๋ง๋ถ™์ธ 80์ฐจ์› ๋ณ€ํ˜•์„ ๊ฐ™์€ ๊ตฌ์กฐยท๋ฐ์ดํ„ฐยทํ•˜์ดํผํŒŒ๋ผ๋ฏธํ„ฐ๋กœ ํ•™์Šตํ•œ๋‹ค. ๋ฌผ์ฒด 4๊ฐœ ร— grasp ์œ ํ˜• 4๊ฐœ(๊ฐ 50 ๊ถค์ )๋ฅผ 50ํšŒ์”ฉ ์‹œํ—˜ํ•œ๋‹ค. ํ‰๊ท ์€ ์นธ ๊ฐ’๊ณผ ๋งž๋Š”๋‹ค. ๋ชจ๋“  ๋ฌผ์ฒด์—์„œ ๊ฐœ์„ ๋˜๊ณ  Discussion์€ ์ด๊ฒƒ์„ โ€œ์ ‘์ด‰์ด ๋น ์ง„ supervisionโ€์ด๋ผ๋Š” ๊ฒฐ๋ก ์˜ ๋‘ ๊ทผ๊ฑฐ ์ค‘ ํ•˜๋‚˜๋กœ ์“ด๋‹ค.

๋‹จ์„œ๊ฐ€ ์…‹ ์žˆ๋‹ค. ์ฒซ์งธ, ํ‰๊ฐ€ ์‹œ ์ ‘์ด‰๋ ฅ 16์ฐจ์›์ด ์–ด๋””์„œ ์˜ค๋Š”์ง€ ๋…ผ๋ฌธ์ด ๋ฐํžˆ์ง€ ์•Š๋Š”๋‹ค. ์‹œ๋ฎฌ๋ ˆ์ด์…˜ ํ‰๊ฐ€๋ผ๋ฉด ์‹œ๋ฎฌ๋ ˆ์ดํ„ฐ์˜ ์ •๋‹ต ์ ‘์ด‰๋ ฅ์ผ ๊ฐ€๋Šฅ์„ฑ์ด ๋†’์€๋ฐ, ๊ทธ๋ ‡๋‹ค๋ฉด ์ด ๊ฒฐ๊ณผ๋Š” โ€œ์ ‘์ด‰ ์ •๋ณด๋ฅผ ์•Œ๋ฉด ๋‚ซ๋‹คโ€๋Š” ์ •๋ณด ๊ฐ€์น˜์˜ ์ƒํ•œ์ด์ง€, DexHand 021์ฒ˜๋Ÿผ ํž˜ ์„ผ์„œ๊ฐ€ ์—†๋Š” ์†์—์„œ ๋ฐฐํฌ ๊ฐ€๋Šฅํ•œ ๊ฐœ์„ ์ด ์•„๋‹ˆ๋‹ค. ์‹ค๊ธฐ ์‹คํ—˜์—๋Š” ์ด Diffusion Policy๊ฐ€ ์“ฐ์ด์ง€ ์•Š๋Š”๋‹ค. ๋‘˜์งธ, โ€œ0.5 N loadโ€๊ฐ€ ๋ฌด์—‡์— ๊ฑธ๋ฆฌ๋Š” ํ•˜์ค‘์ธ์ง€ ์ •์˜๊ฐ€ ์—†๋‹ค. ์…‹์งธ, ๋ฌผ์ฒด๋งˆ๋‹ค policy๋ฅผ ๋”ฐ๋กœ ํ•™์Šตํ–ˆ๋Š”์ง€ ํ•˜๋‚˜๋กœ ํ•™์Šตํ–ˆ๋Š”์ง€, seed ์ˆ˜์™€ ๋ถ„์‚ฐ์ด ์—†๋‹ค. ๊ฐ™์€ ๊ทธ๋ฃน์˜ DexCanvas๋Š” ๋‹ค๋ฅธ ์„ค์ •์—์„œ ์ ‘์ด‰ ํŠน์ง•์ด 67.5% โ†’ 82.2%(18,522 trial)๋ฅผ ๋งŒ๋“ ๋‹ค๊ณ  ๋ณด๊ณ ํ•˜๋Š”๋ฐ, ๋‘ ์‹คํ—˜์˜ ๊ด€๊ณ„๋„ ์ด ๋…ผ๋ฌธ์—๋Š” ๋‚˜์˜ค์ง€ ์•Š๋Š”๋‹ค.

์‹ค๊ธฐ ์…‹์—….


Fig. 4: ์‹ค๊ธฐ ํ•˜๋“œ์›จ์–ด. 6-DoF JAKA Mini 2 ํŒ” 2๋Œ€์— ๊ฐ๊ฐ DexHand 021, RealSense D435 ์นด๋ฉ”๋ผ 2๋Œ€.

๊ณผ์ œ๋Š” ์ข…์ด์ปต ์Œ“๊ธฐ, ๋ณ‘ ๊ฑด๋„ค๊ณ  ์„ธ์šฐ๊ธฐ, ๋ฌผ ๋ถ“๊ธฐ, ์† ์•ˆ ๋ณ‘ ์žฌ๋ฐฐ์น˜์ด๊ณ  ๊ฐ 10ํšŒ๋‹ค. ์„ฑ๊ณต์€ ๋…นํ™”๋œ RGB ์˜์ƒ์œผ๋กœ ์ด์ง„ ํŒ์ •ํ•œ๋‹ค: ๋‘ ๋ฒˆ์งธ ์ปต์ด ์ฒซ ์ปต ์•ˆ์— ๋“ค์–ด๊ฐ€๊ณ  ๋„˜์–ด์ง€์ง€ ์•Š์Œ / ๋ณ‘์„ ์™ผ์†์—์„œ ์˜ค๋ฅธ์†์œผ๋กœ ๋„˜๊ธด ๋’ค ํ…Œ์ด๋ธ”์— ์„ธ์›Œ ๋„˜์–ด์ง€์ง€ ์•Š์Œ / ์›๋ž˜ ์ปต์—์„œ ๋ชฉํ‘œ ์ข…์ด์ปต์œผ๋กœ ๋ฐ–์— ํ˜๋ฆฌ์ง€ ์•Š๊ณ  ๋ฌผ์„ ๋ถ€์Œ / ๋ณ‘์„ ๋“ค์–ด ์† ์•ˆ์—์„œ ๋’ค์ง‘์€ ๋’ค ํ…Œ์ด๋ธ”์— ๋˜‘๋ฐ”๋กœ ์„ธ์›€.


Fig. 5: ์‹ค๊ธฐ 4๊ณผ์ œ. ๊ณผ์ œ๋งˆ๋‹ค ์œ—์ค„์ด ์‚ฌ๋žŒ ์‹œ์—ฐ(์ดˆ๋ก ๋ฐฐ๊ฒฝ), ์•„๋žซ์ค„์ด ๋กœ๋ด‡ ์‹คํ–‰์ด๊ณ  ์‹œ๊ฐ„์€ ์™ผ์ชฝ์—์„œ ์˜ค๋ฅธ์ชฝ์œผ๋กœ ํ๋ฅธ๋‹ค. ์œ„์—์„œ๋ถ€ํ„ฐ ์ข…์ด์ปต ์Œ“๊ธฐ(7/10), ๋ณ‘ ๊ฑด๋„ค๊ณ  ์„ธ์šฐ๊ธฐ(5/10), ๋ฌผ ๋ถ“๊ธฐ(6/10), ์† ์•ˆ ๋ณ‘ ์žฌ๋ฐฐ์น˜(3/10).
๊ณผ์ œ ์„ฑ๊ณต Wilson 95% ๊ตฌ๊ฐ„(ํ•„์ž ๊ณ„์‚ฐ)
์ข…์ด์ปต ์Œ“๊ธฐ 7/10 39.7~89.2%
๋ณ‘ ๊ฑด๋„ค๊ณ  ์„ธ์šฐ๊ธฐ 5/10 23.7~76.3%
๋ฌผ ๋ถ“๊ธฐ 6/10 31.3~83.2%
์† ์•ˆ ๋ณ‘ ์žฌ๋ฐฐ์น˜ 3/10 10.8~60.3%
ํ•ฉ๊ณ„ 21/40 37.5~67.1%

๋ณธ๋ฌธ์˜ โ€œoverall success rate above 50%โ€๋Š” 21/40 = 52.5%๋‹ค. 10ํšŒ ์‹œํ–‰์˜ ๊ตฌ๊ฐ„์€ ์„œ๋กœ ํฌ๊ฒŒ ๊ฒน์ณ์„œ, ๊ณผ์ œ ๊ฐ„ ์ˆœ์œ„(โ€œin-hand ์žฌ๋ฐฐ์น˜๊ฐ€ ๊ฐ€์žฅ ์–ด๋ ต๋‹คโ€)๋Š” 3/10๊ณผ 7/10 ์‚ฌ์ด์—์„œ๋งŒ ๊ตฌ๊ฐ„์ด ์ผ๋ถ€ ๊ฐˆ๋ฆฌ๋Š” ์ •๋„๋‹ค. ์ €์ž๋Š” ์žฌ๋ฐฐ์น˜์˜ ์‹คํŒจ ์›์ธ์„ โ€œsmall timing errors compoundโ€๋กœ ์„ค๋ช…ํ•˜์ง€๋งŒ ์‹คํŒจ ์œ ํ˜• ๋ถ„์„์€ ์—†๋‹ค.

์ด ์‹คํ—˜์ด ๋ณด์—ฌ ์ฃผ๋Š” ๊ฒƒ์€ ์‹คํ–‰ ๊ฐ€๋Šฅ์„ฑ์ด๋‹ค. ์‹ค๋กœ๋ด‡ ํ•™์Šต ๋ฐ์ดํ„ฐ ์—†์ด, ๊ณ ์ •๋œ policy๋กœ, ์–‘์† ๋‹ค์ง€ contact-rich ๊ณผ์ œ๊ฐ€ ์ ˆ๋ฐ˜ ์ •๋„ ์„ฑ๊ณตํ•œ๋‹ค๋Š” ๊ฒƒ์€ ์˜๋ฏธ ์žˆ๋Š” ์‹œ์—ฐ์ด๋‹ค. ๋ฐ˜๋ฉด ๋น„๊ต ๋Œ€์ƒ์ด ํ•˜๋‚˜๋„ ์—†๋‹ค. residual ์—†์ด retargeted reference๋งŒ ์žฌ์ƒํ•œ ์กฐ๊ฑด, teleop ๊ธฐ์ค€์„ , sim์—์„œ์˜ ๊ฐ™์€ ๊ณผ์ œ ์„ฑ๊ณต๋ฅ ์ด ์—†์–ด์„œ ๊ฐ ๋‹จ๊ณ„๊ฐ€ ์‹ค๊ธฐ ์„ฑ๊ณต์— ์–ผ๋งˆ๋‚˜ ๊ธฐ์—ฌํ–ˆ๋Š”์ง€ ์•Œ ์ˆ˜ ์—†๋‹ค. ์ €์ž๋Š” ์ด 4๊ณผ์ œ๋ฅผ โ€œpose particular challenges for teleoperation systemsโ€๋ผ์„œ ๊ณจ๋ž๋‹ค๊ณ  ์“ฐ์ง€๋งŒ teleop์œผ๋กœ ์‹ค์ œ ์‹œ๋„ํ•œ ๊ฒฐ๊ณผ๋Š” ์—†๋‹ค. ๋ฌผ ๋ถ“๊ธฐ๋Š” ์œ ์ฒด๋ฅผ ๋‹ค๋ฃจ๋Š”๋ฐ Isaac Gym์—์„œ ๋ฌผ์„ ์–ด๋–ป๊ฒŒ ๋‹ค๋ค˜๋Š”์ง€(๋ฌผ ์—†์ด ์ปต ์šด๋™๋งŒ ์ถ”์ ํ–ˆ๋Š”์ง€)๋„ ๋‚˜์˜ค์ง€ ์•Š๋Š”๋‹ค. ๊ธฐ์—ฌ ๋ชฉ๋ก์€ ์‹ค๊ธฐ๋ฅผ โ€œa single residual policy shared across tasksโ€๋กœ ์“ฐ์ง€๋งŒ ยง4.3์€ โ€œthe deployed policiesโ€๋ผ๊ณ  ๋ณต์ˆ˜๋กœ ์“ฐ๊ณ , ์ด policy๊ฐ€ ์–‘์† 25๊ณผ์ œ ์„ค์ •์—์„œ ํ•™์Šต๋œ ๊ฒƒ์ธ์ง€, ์‹ค๊ธฐ 4๊ณผ์ œ๊ฐ€ ๊ทธ 25๊ณผ์ œ์— ํฌํ•จ๋˜๋Š”์ง€๋„ ๋ฐํžˆ์ง€ ์•Š๋Š”๋‹ค.

์ธก์ •๋œ ๊ฒƒ๊ณผ ์„ ์–ธ๋œ ๊ฒƒ

์ฃผ์žฅ ๊ทผ๊ฑฐ์˜ ์ข…๋ฅ˜ ๊ทผ๊ฑฐ
residual ๋ณด์ •์ด kinematic replay๋ฅผ ํฌ๊ฒŒ ๊ฐœ์„ ํ•œ๋‹ค ์ธก์ •(๋Œ€์กฐ๊ตฐ์€ replay) Table 1, Fig. 3, Table 3
๋‹จ์ผ policy๊ฐ€ ๊ณผ์ œ๋ณ„ ํ•™์Šต์„ ๋Œ€์ฒดํ•œ๋‹ค ์„ค๊ณ„ ๋…ผ์ฆ(๊ณผ์ œ๋ณ„ policy ๋Œ€๋น„ ๋น„๊ต ์—†์Œ) ยง3.2
์ฒ˜์Œ ๋ณด๋Š” ๋ฌผ์ฒด์— ์ผ๋ฐ˜ํ™”ํ•œ๋‹ค ์ธก์ •, ํ”„๋กœํ† ์ฝœ ๋ฏธ์„œ์ˆ  Table 2(DexCanvas v3 Table 2์™€ ๋™์ผ ๊ฒฐ๊ณผ)
์ ‘์ด‰ ๊ธฐ๋ฐ˜ retargeting์ด ๊ธฐํ•˜ retargeting๋ณด๋‹ค ๋‚ซ๋‹ค ์ธก์ •(kinematic ์ง€ํ‘œ๋งŒ) Table 5. ํ๋ฃจํ”„ ๋น„๊ต ์—†์Œ
๋ชฉ์ ํ•จ์ˆ˜๊ฐ€ ์† ๋ชจ์–‘๊ณผ ๋…๋ฆฝ์ด๋‹ค ์„ค๊ณ„ ๋…ผ์ฆ(์‹์˜ ํ˜•ํƒœ) + ์ธก์ •(4์ข… ์†) ยง3.3, Table 3
๋ณต์›ํ•œ ์ ‘์ด‰๋ ฅ์ด downstream ํ•™์Šต์„ ๋•๋Š”๋‹ค ์ธก์ •, ํ‰๊ฐ€ ์‹œ ํž˜ ์ž…๋ ฅ ์ถœ์ฒ˜ ๋ฏธ์„œ์ˆ  Table 4
์‹ค๋กœ๋ด‡ ํ•™์Šต ๋ฐ์ดํ„ฐ ์—†์ด ํ•˜๋“œ์›จ์–ด์—์„œ ์‹คํ–‰๋œ๋‹ค ์ธก์ •(4๊ณผ์ œ ร— 10ํšŒ, ๋น„๊ต๊ตฐ ์—†์Œ) Fig. 5
retargeting์ด SPIDER๋ณด๋‹ค ๋น„์šฉ ํšจ์œจ์ ์ด๋‹ค ์„ ์–ธ(์„œ๋ก ), ํ‘œ๋Š” ๋ฐ˜๋Œ€ Table 5 TimeยทTput
์ƒ์„ฑ ๋ฐ์ดํ„ฐ๊ฐ€ in-context policy ํ•™์Šต์— ๋งž๋Š”๋‹ค ์ „๋ง ยง5

๋น„ํŒ์ ์œผ๋กœ ๋ณด๋ฉด

๊ฐ•์ 

  • โ€œ๋ฌด์—‡์„ ์˜ฎ๊ธธ ๊ฒƒ์ธ๊ฐ€โ€์— ๋Œ€ํ•œ ๋‹ต์ด ๋ช…๋ฃŒํ•˜๋‹ค. ์ ‘์ด‰ ๋ผ๋ฒจ๊ณผ ๋ฌผ์ฒด ํ‘œ๋ฉด ์ ‘์ด‰ ์œ„์น˜๋ฅผ embodiment ๋ถˆ๋ณ€๋Ÿ‰์œผ๋กœ ์ง€์ •ํ•˜๊ณ , retargeting ๋ชฉ์ ํ•จ์ˆ˜๋ฅผ ๊ทธ ์œ„์น˜๊นŒ์ง€์˜ ๊ฑฐ๋ฆฌ๋กœ๋งŒ ์ •์˜ํ•œ ๊ฒƒ์€ ๋‹จ์ˆœํ•˜๊ณ  ์„ค๋ช…๋ ฅ์ด ์žˆ๋‹ค. ์†์„ ์ถ”๊ฐ€ํ•  ๋•Œ ๋ฐ”๊ฟ€ ๊ฒƒ์„ embodiment profile ํ•˜๋‚˜๋กœ ๋ชจ์€ ์ธํ„ฐํŽ˜์ด์Šค ์„ค๊ณ„๋„ ์‹ค์šฉ์ ์ด๋‹ค.
  • residual ํ˜•์‹์˜ ์žฌ์‚ฌ์šฉ์ด ๊น”๋”ํ•˜๋‹ค. physics refinement์™€ ๋กœ๋ด‡ ์ชฝ ๋ณด์ •์ด ๊ฐ™์€ Eq. 1์„ ์“ฐ๊ณ , decay๋กœ envelope๋ฅผ ๋ฌถ๊ณ , ๋งˆ์ง€๋ง‰ ์ ‘์ด‰ ์ดํ›„ ์†๊ฐ€๋ฝ๋งŒ decay๋ฅผ ๋„๋Š” ์ฒ˜๋ฆฌ๋Š” ๊ฐ๊ฐ ์ด์œ ๊ฐ€ ๋ถ„๋ช…ํ•˜๋‹ค. ์ˆ˜์„ ๊ธฐ(repair operator)๋ฅผ ๊ณผ์ œ๋ณ„๋กœ ๋‹ค์‹œ ๋งŒ๋“ค์ง€ ์•Š๋Š”๋‹ค๋Š” ๋ฐฉํ–ฅ์€ real2sim2real์˜ ๋ณ‘๋ชฉ์„ ์ •ํ™•ํžˆ ๊ฒจ๋ƒฅํ•œ๋‹ค.
  • ๊ทœ๋ชจ์™€ ๋ฒ”์œ„๊ฐ€ ๋„“๋‹ค. ํ•œ ์† 2.5๋งŒ ๊ถค์ , ์–‘์† 25๊ณผ์ œ, ๋กœ๋ด‡ ์† 4์ข…, ์‹ค๊ธฐ ์–‘ํŒ” 4๊ณผ์ œ๊นŒ์ง€ ํ•œ ๋…ผ๋ฌธ ์•ˆ์—์„œ ์ด์–ด์ง„๋‹ค. ํŠนํžˆ ๊ฐ™์€ ์‚ฌ๋žŒ ๋ฐ์ดํ„ฐ๋ฅผ ShadowยทAllegro๊นŒ์ง€ ๊ฐ™์€ ์„ค์ •์œผ๋กœ ์˜ฎ๊ฒจ ๋„ค ์† ๋ชจ๋‘ 76.5~82.5%์— ๋„๋‹ฌํ•œ Table 3์€ โ€œ๋ ˆ์‹œํ”ผ์˜ ์ด์‹์„ฑโ€์— ๋Œ€ํ•œ ์ข‹์€ ์ฆ๊ฑฐ๋‹ค.
  • ์‹ค๊ธฐ ๊ณผ์ œ๊ฐ€ ์–ด๋ ต๋‹ค. ์–‘์† ํ•ธ๋“œ์˜ค๋ฒ„ ํ›„ ์„ธ์šฐ๊ธฐ, ๋ฌผ ๋ถ“๊ธฐ, ์† ์•ˆ ์žฌ๋ฐฐ์น˜์ฒ˜๋Ÿผ ๊ทธ๋ฆฌํผ๋กœ๋Š” ์–ด๋ ต๊ณ  teleop์œผ๋กœ๋„ ๊นŒ๋‹ค๋กœ์šด ๊ณผ์ œ๋ฅผ ์‹ค๋กœ๋ด‡ ํ•™์Šต ๋ฐ์ดํ„ฐ ์—†์ด ์‹œ๋„ํ–ˆ๋‹ค. ์„ฑ๊ณต ๊ธฐ์ค€์„ ๊ณผ์ œ๋งˆ๋‹ค ๋ฌธ์žฅ์œผ๋กœ ์ •์˜ํ•œ ๊ฒƒ๋„ ์ข‹๋‹ค.
  • ์ž๊ธฐ ๋ถˆ๋ฆฌํ•œ ์ˆ˜์น˜๋ฅผ ํ‘œ์— ๋‚จ๊ฒผ๋‹ค. Table 5์˜ Time ์—ด๊ณผ Vector์˜ fidelity ์šฐ์œ„, Fig. 3์˜ Lay ์œ ํ˜• ์ €์„ฑ๋Šฅ, ์‹ค๊ธฐ ์žฌ๋ฐฐ์น˜ 3/10์ด ๊ทธ๋Œ€๋กœ ์‹ค๋ ค ์žˆ๋‹ค. ๋ณธ๋ฌธ์ด ์–ธ๊ธ‰ํ•˜์ง€ ์•Š์„ ๋ฟ ๋…์ž๊ฐ€ ํ™•์ธํ•  ์ˆ˜ ์žˆ๋‹ค.

์•ฝ์ ยทํ•œ๊ณ„

  • ํ•ต์‹ฌ ๋น„๊ต๊ฐ€ ๋น„์–ด ์žˆ๋‹ค. ์„ธ ๊ธฐ์—ฌ ๊ฐ๊ฐ์— ๋งž๋Š” ๋Œ€์กฐ๊ตฐ์ด ์—†๋‹ค. โ‘  ๋‹จ์ผ amortized policy์˜ ๋น„์šฉ(๊ณผ์ œ๋ณ„ policy ๋Œ€๋น„ ์„ฑ๊ณต๋ฅ  ์†์‹ค)์„ ์žฐ ์‹คํ—˜์ด ์—†๋‹ค. โ‘ก โ€œ์ ‘์ด‰ ๊ธฐ๋ฐ˜ ์ „์ด๊ฐ€ embodiment gap์„ ๋” ์ž˜ ๋‹ซ๋Š”๋‹คโ€๋Š” ์ฃผ์žฅ(ยง4 ์งˆ๋ฌธ 2)์„ ํ๋ฃจํ”„์—์„œ ๊ฒ€์ฆํ•˜๋ ค๋ฉด DexPilotยทVectorยทSPIDER์˜ reference์— ๊ฐ™์€ residual ๋‹จ๊ณ„๋ฅผ ๋ถ™์—ฌ์•ผ ํ•˜๋Š”๋ฐ, Table 3์˜ ๋น„๊ต๋Š” ์ž๊ธฐ ๋ฐฉ๋ฒ•์˜ replay๋ฟ์ด๋‹ค. โ‘ข physics refinement์˜ ์„ฑ๊ณต๋ฅ ์„ ManipTransยทDexMachinaยทSPIDER ๊ฐ™์€ dynamic retargeting ๋ฐฉ๋ฒ•๊ณผ ๋น„๊ตํ•˜์ง€ ์•Š๋Š”๋‹ค. ๊ทธ๋ž˜์„œ ์ธก์ •๋œ ๊ฒƒ์€ โ€œresidual์ด replay๋ณด๋‹ค ๋‚ซ๋‹คโ€์ด๊ณ , ์ด๋Š” ์ด ๊ณ„์—ด์—์„œ ์ด๋ฏธ ์•Œ๋ ค์ง„ ์‚ฌ์‹ค์— ๊ฐ€๊น๋‹ค.
  • retargeting ๋น„๊ตํ‘œ๊ฐ€ ๋ณธ๋ฌธ ๊ฒฐ๋ก ๋ณด๋‹ค ๋ถˆ๋ฆฌํ•˜๋‹ค. Table 5์—์„œ ์ด ๋ฐฉ๋ฒ•์ด ๋‹จ๋… 1์œ„์ธ ์—ด์€ jitter๋ฟ์ด๊ณ , ์‹œ๊ฐ„์€ Spider default์˜ 7.9๋ฐฐ๋กœ ๊ฐ€์žฅ ๋А๋ฆฌ๋‹ค. ์„œ๋ก ์ด SPIDER์˜ ๊ถค์ ๋‹น ๋น„์šฉ์„ ๋น„ํŒํ•˜๋ฉด์„œ ์ž๊ธฐ ๋น„์šฉ์€ ๋ณธ๋ฌธ์—์„œ ์–ธ๊ธ‰ํ•˜์ง€ ์•Š๋Š”๋‹ค. Time๊ณผ Tput์˜ ๊ณฑ์ด ๋‹ค๋ฅธ ํ–‰๊ณผ 2๋ฐฐ ๊ฐ€๊นŒ์ด ์–ด๊ธ‹๋‚˜๋Š” ๊ฒƒ๋„ ์—ด ์ •์˜๊ฐ€ ์—†์–ด ํ•ด์†Œ๋˜์ง€ ์•Š๋Š”๋‹ค. fidelityยทretention ์ง€ํ‘œ๋Š” ์ด ๋ฐฉ๋ฒ•์˜ ๋ชฉ์ ํ•จ์ˆ˜์™€ ๊ฐ™์€ ๊ณต๊ฐ„์—์„œ ์ •์˜๋˜๊ณ , SPIDER์˜ ๊ฐ•์ (๋™์—ญํ•™์  ์‹คํ˜„ ๊ฐ€๋Šฅ์„ฑ)์€ ํ‘œ์— ๋“ค์–ด๊ฐ€์ง€ ์•Š๋Š”๋‹ค.
  • ์ •์˜๋˜์ง€ ์•Š์€ ์ง€ํ‘œ๊ฐ€ ํ—ค๋“œ๋ผ์ธ์— ์žˆ๋‹ค. ์ดˆ๋ก์˜ ์–‘์† ๊ฒฐ๊ณผ 16.0% โ†’ 62.4%๋Š” ์ •์˜๊ฐ€ ์—†๋Š” task-level ๊ฐ’์ด๋‹ค. attempt-level(8.4% โ†’ 43.9%)์ด ๋” ์—„๊ฒฉํ•œ ์ชฝ์œผ๋กœ ๋ณด์ธ๋‹ค.
  • ์ผ๋ฐ˜ํ™” ์‹คํ—˜์˜ ํ•ด์„์ด ์–ด๋ ต๋‹ค. ์ฒ˜์Œ ๋ณด๋Š” YCB ๋ฌผ์ฒด(72.4%)๊ฐ€ ํ•™์Šต ๋ฌผ์ฒด(59.31%)๋ณด๋‹ค ์ž˜ ๋˜๋Š” ์—ญ์ „์ด ์„ค๋ช…๋˜์ง€ ์•Š๊ณ , reference๋ฅผ ์–ด๋–ป๊ฒŒ ์ •ํ–ˆ๋Š”์ง€์™€ trial ๊ตฌ์„ฑ(50์˜ ๋ฐฐ์ˆ˜๊ฐ€ ์•„๋‹˜)์ด ์„œ์ˆ ๋˜์ง€ ์•Š์•˜๋‹ค.
  • ๊ฐ™์€ ๊ทธ๋ฃน ์„ ํ–‰ ๋…ผ๋ฌธ๊ณผ์˜ ๊ด€๊ณ„๊ฐ€ ๋“œ๋Ÿฌ๋‚˜์ง€ ์•Š๋Š”๋‹ค. DexCanvas๋Š” v1(2025-10)์—์„œ MANO residual RL๋กœ ์‹œ์—ฐ์„ ์žฌ๊ตฌ์„ฑํ•ด ์ ‘์ด‰๋ ฅ์„ ์–ป๋Š” ํŒŒ์ดํ”„๋ผ์ธ์„ ๋ƒˆ๊ณ (๋‹น์‹œ๋Š” ๋ฌผ์ฒด-์กฐ์ž‘ ์Œ๋ณ„ policy), v3(2026-11-03, ์ด ๋…ผ๋ฌธ v2์™€ ๊ฐ™์€ ๋‚ )์—์„œ ๋‹จ์ผ residual policy, ๋™์ผํ•œ YCB Table 2, DexHand 021/021Pro๋กœ์˜ ์ ‘์ด‰ ๊ธฐ๋ฐ˜ retargeting์„ ์‹ฃ๊ณ  ์ด ๋…ผ๋ฌธ์„ ํ™•์žฅ์œผ๋กœ ์ธ์šฉํ•œ๋‹ค. ์ด ๋…ผ๋ฌธ์€ DexCanvas๋ฅผ ๋ฌผ์ฒดยท์ œ์Šค์ฒ˜ ์ •์˜์˜ ์ถœ์ฒ˜๋กœ๋งŒ ์ธ์šฉํ•œ๋‹ค. ๊ทธ๋ž˜์„œ ๊ธฐ์—ฌ 2(โ€œphysics consistency as an amortized, learned operatorโ€)์˜ ์–ด๋””๊นŒ์ง€๊ฐ€ ์ด ๋…ผ๋ฌธ์—์„œ ์ƒˆ๋กœ ๋‚˜์˜จ ๊ฒƒ์ธ์ง€ ๋ณธ๋ฌธ๋งŒ์œผ๋กœ๋Š” ์•Œ ์ˆ˜ ์—†๊ณ , Table 2๊ฐ€ ๋‘ ๋…ผ๋ฌธ์— ๊ณต์œ ๋œ ๊ฒฐ๊ณผ๋ผ๋Š” ์‚ฌ์‹ค๋„ ๋…์ž๊ฐ€ ๋”ฐ๋กœ ์ฐพ์•„์•ผ ํ•œ๋‹ค.
  • Downstream ๊ฒฐ๊ณผ๋Š” ๋ฐฐํฌ ๊ฐ€๋Šฅํ•œ ๊ฐœ์„ ์ด ์•„๋‹ ์ˆ˜ ์žˆ๋‹ค. Table 4์˜ +11.8 pp๋Š” ํ‰๊ฐ€ ์‹œ ์ ‘์ด‰๋ ฅ ์ž…๋ ฅ์˜ ์ถœ์ฒ˜๊ฐ€ ๋ฐํ˜€์ง€์ง€ ์•Š์•˜๊ณ , ์‹œ๋ฎฌ๋ ˆ์ดํ„ฐ ์ •๋‹ต ํž˜์ด๋ผ๋ฉด ํž˜ ์„ผ์„œ ์—†๋Š” ์‹ค๊ธฐ์—์„œ๋Š” ์“ธ ์ˆ˜ ์—†๋Š” ์ •๋ณด๋‹ค. ํ•˜์ค‘ ์ •์˜, seed, ๋ถ„์‚ฐ๋„ ์—†๋‹ค.
  • ์‹ค๊ธฐ๋Š” ๋น„๊ต๊ตฐ ์—†๋Š” ์‹คํ–‰ ๊ฐ€๋Šฅ์„ฑ ์‹œ์—ฐ์ด๋‹ค. ๊ณผ์ œ๋‹น 10ํšŒ๋ผ ๊ตฌ๊ฐ„์ด ๋„“๊ณ (ํ•ฉ๊ณ„ Wilson 37.5~67.1%), residual ์—†๋Š” ์žฌ์ƒยทteleopยทsim ์„ฑ๊ณต๋ฅ ์ด ์—†์–ด ๋‹จ๊ณ„๋ณ„ ๊ธฐ์—ฌ๋ฅผ ์•Œ ์ˆ˜ ์—†๋‹ค. ์ดˆ๊ธฐ ๋ฌผ์ฒด ๋ฐฐ์น˜์˜ ๋ณ€ํ™” ํญ, ์†๋ชฉ ๋ช…๋ น์„ ํŒ”๋กœ ์˜ฎ๊ธฐ๋Š” ๋ฐฉ๋ฒ•, ์ œ์–ด ์ฃผํŒŒ์ˆ˜, ๋ฌผ ๋ถ“๊ธฐ์˜ ์‹œ๋ฎฌ๋ ˆ์ด์…˜ ์ฒ˜๋ฆฌ, ์‹ค๊ธฐ์— ์“ด policy๊ฐ€ ์–ด๋А ์„ค์ •์—์„œ ํ•™์Šต๋๋Š”์ง€๊ฐ€ ๋น ์ ธ ์žˆ๋‹ค.
  • ์žฌํ˜„ ์ •๋ณด๊ฐ€ ๊ฑฐ์˜ ์—†๋‹ค. ๋ถ€๋ก์ด ์—†๊ณ  \gammaยทsยทclipยท๋ณด์ƒ ๊ฐ€์ค‘์น˜ยทEq. 2 ๊ฐ€์ค‘์น˜ยท๋„คํŠธ์›ŒํฌยทPPO ์„ค์ •ยทseedยทํ•™์Šต ๋น„์šฉ์ด ๋ชจ๋‘ ๋น ์ ธ ์žˆ๋‹ค. ๋…ผ๋ฌธ ์ฒซ ์ชฝ์— ์ฝ”๋“œ ๋งํฌ๊ฐ€ ์žˆ์ง€๋งŒ ๋ ˆํฌ์—๋Š” ์ฝ”๋“œ๊ฐ€ ์—†๊ณ , ๋ฐ์ดํ„ฐยท์ž์‚ฐยท์ฒดํฌํฌ์ธํŠธ ๊ณต๊ฐœ ๊ณ„ํš๋„ ๋ณธ๋ฌธ์— ์—†๋‹ค(์•„๋ž˜ ๊ณต๊ฐœ ์ƒํƒœ).
  • ์‚ฌ์†Œํ•œ ํ‘œ๊ธฐ ๋ถˆ์ผ์น˜. Fig. 6 ์บก์…˜์˜ ๋‹จ๊ณ„ ์ด๋ฆ„์ด ๊ทธ๋ฆผ ์—ด๊ณผ ๋‹ค๋ฅด๊ณ , Table 2์˜ pooled 72.4%๋Š” ์„ฑ๊ณต ํšŸ์ˆ˜๋กœ ๊ณ„์‚ฐํ•˜๋ฉด 72.46%๋‹ค. ์‹ค๊ธฐ ๊ณผ์ œ๋ช…์€ ๊ทธ๋ฆผ์—์„œ โ€œPassing Bottleโ€, ๋ณธ๋ฌธ์—์„œ โ€œpassing & standing a bottleโ€์ด๋‹ค.

๊ณต๊ฐœ ์ƒํƒœ

๋…ผ๋ฌธ ์ฒซ ์ชฝ๊ณผ arXiv ๋ฉ”ํƒ€๋ฐ์ดํ„ฐ๋Š” DexGEM-Lab/real2sim2real(Apache-2.0)์„ ์ฝ”๋“œ ๋งํฌ๋กœ ์ ๋Š”๋‹ค. ๊ทธ ๋ ˆํฌ์—๋Š” LICENSE, 22์ค„์งœ๋ฆฌ README, ๋ฐ๋ชจ ์˜์ƒ 5์Œ(gif + mp4)๋งŒ ์žˆ๊ณ  README ๋งˆ์ง€๋ง‰ ์ค„์ด โ€œCode release coming soon.โ€์ด๋‹ค. ์ปค๋ฐ‹์€ ๋‘ ๊ฐœ์ด๊ณ  ๋งˆ์ง€๋ง‰ push๋Š” 2026-09-17๋กœ, ๋…ผ๋ฌธ v2(2026-11-03) ์ดํ›„์—๋„ ๋ฐ”๋€Œ์ง€ ์•Š์•˜๋‹ค. ์„ธ ๋‹จ๊ณ„์˜ ์ฝ”๋“œ, 25,454๊ฐœ ํ•œ ์† ๊ถค์ ๊ณผ ์–‘์† 250 ๊ถค์ , ๋ฌผ์ฒด URDFยท๋ฉ”์‹œ, ๋„ค ์†์˜ embodiment profile, ์ฒดํฌํฌ์ธํŠธ๊ฐ€ ๋ชจ๋‘ ๊ณต๊ฐœ๋˜์ง€ ์•Š์•˜๋‹ค. README ๋ฐ๋ชจ๋Š” ๋งˆ์š”๋„ค์ฆˆ ๋ณ‘์„ ๋ธ”๋ก์— ์˜ฌ๋ฆฌ๋Š” ์–‘์† ๊ณผ์ œ๋กœ, ๋…ผ๋ฌธ์ด ์ˆ˜์น˜๋ฅผ ๋ณด๊ณ ํ•œ ์‹ค๊ธฐ ๊ณผ์ œ์™€ ์ด๋ฆ„์ด ์ผ์น˜ํ•˜์ง€ ์•Š๋Š”๋‹ค. ๊ทธ๋ž˜์„œ ์œ„ ํ‘œ์—์„œ โ€œ๋ฐํžˆ์ง€ ์•Š์Œโ€์œผ๋กœ ๋ถ„๋ฅ˜ํ•œ ์„ค์ •์€ ํ˜„์žฌ๋กœ์„œ๋Š” ์–ด๋””์—์„œ๋„ ํ™•์ธํ•  ์ˆ˜ ์—†๋‹ค. ๋ฐ์ดํ„ฐ ์ชฝ์€ ๊ฐ™์€ ๊ทธ๋ฃน์˜ DexCanvas๊ฐ€ Hugging Face์— force-annotated ๊ถค์ ์„ ๊ณต๊ฐœํ•˜๊ณ  ์žˆ์ง€๋งŒ, ์ด ๋…ผ๋ฌธ์˜ ๋ฐ์ดํ„ฐ์…‹์ด ๊ทธ ์ผ๋ถ€์ธ์ง€๋Š” ๋ณธ๋ฌธ์— ์—†๋‹ค.

๊ด€๋ จ ์—ฐ๊ตฌ์™€์˜ ์ž๋ฆฌ ๋งค๊น€

์ด ๋…ผ๋ฌธ์€ ์‚ฌ๋žŒ ์†-๋ฌผ์ฒด ์‹œ์—ฐ์„ ๋กœ๋ด‡ ์† ๋ฐ์ดํ„ฐ๋กœ ๋ฐ”๊พธ๋Š” ๊ณ„๋ณด์—์„œ kinematic retargeting๊ณผ dynamic refinement๋ฅผ ๋‘˜ ๋‹ค ํ•˜๋‚˜์˜ ํ”„๋ ˆ์ž„์œผ๋กœ ๋‹ค์‹œ ์ง  ์‹œ์Šคํ…œ ๋…ผ๋ฌธ์ด๋‹ค.

๊ณ„์—ด ๋Œ€ํ‘œ ์ด ๋…ผ๋ฌธ๊ณผ์˜ ๊ด€๊ณ„
๊ธฐํ•˜ ๋Œ€์‘ kinematic retargeting DexPilot, AnyTeleop Table 5์˜ ๋ฒ ์ด์Šค๋ผ์ธ. ์ด ๋…ผ๋ฌธ์€ ์†๋ ๋Œ€์‹  ๋ฌผ์ฒด ์ ‘์ด‰ ์œ„์น˜๋ฅผ ๋ชฉํ‘œ๋กœ ๋ฐ”๊พผ๋‹ค
๋ฌผ์ฒด ์ธ์ง€ kinematic retargeting DexOAK, Regrind, OmniRetarget ์†-๋ฌผ์ฒด ์ƒ๋Œ€ ๋ฐฐ์น˜๋ฅผ ๋ณด์กดํ•œ๋‹ค๋Š” ๊ฐ™์€ ๋ฌธ์ œ์˜์‹. DexOAKยทRegrind๋Š” interaction graph์˜ Laplacian ์ขŒํ‘œ๋ฅผ, ์ด ๋…ผ๋ฌธ์€ ์ ‘์ด‰ ๋ผ๋ฒจ + ํ‘œ๋ฉด์  ๊ฑฐ๋ฆฌ๋ฅผ ์“ด๋‹ค. ์ด ๋…ผ๋ฌธ์€ ์ด๋“ค์„ ์ธ์šฉํ•˜์ง€ ์•Š๋Š”๋‹ค
๋ฌผ๋ฆฌ ๊ธฐ๋ฐ˜ dynamic retargeting SPIDER, ManipTrans, DexMachina, DexTrack ์ด ๋…ผ๋ฌธ์˜ residual ๋‹จ๊ณ„์™€ ๊ฐ™์€ ์ž๋ฆฌ. SPIDER๋งŒ kinematic ์ง€ํ‘œ๋กœ ๋น„๊ต๋˜๊ณ , ํ๋ฃจํ”„ ์„ฑ๊ณต๋ฅ  ๋น„๊ต๋Š” ์–ด๋А ๊ฒƒ๊ณผ๋„ ์—†๋‹ค
์ ‘์ด‰ ์ •๋ณด๋กœ ์ •์ฑ… ํ•™์Šต CHORD ์‚ฌ๋žŒ ์‹œ์—ฐ์˜ contact wrench๋ฅผ guidance๋กœ ์“ฐ๋Š” ์ชฝ. ์ด ๋…ผ๋ฌธ์€ ์ ‘์ด‰๋ ฅ์„ ์‹œ๋ฎฌ๋ ˆ์ด์…˜์—์„œ ๋ณต์›ํ•ด Diffusion Policy ์ž…๋ ฅ์œผ๋กœ ์“ด๋‹ค
์‚ฌ๋žŒ ๋ฐ์ดํ„ฐ ๊ทœ๋ชจํ™” EgoScale, DexMimicGen, Human2Sim2Robot ์˜์ƒ ๊ทœ๋ชจํ™” ๋˜๋Š” ์ƒ์„ฑ. ์ด ๋…ผ๋ฌธ์€ ์ •๋ฐ€ mocap + ์‹œ๋ฎฌ๋ ˆ์ด์…˜ ์ฆ๊ฐ• ์ชฝ์— ์„ ๋‹ค

๊ฐ€์žฅ ๊ฐ€๊นŒ์šด ์ด์›ƒ DexOAK์™€ ๋‚˜๋ž€ํžˆ ๋†“์œผ๋ฉด ์ฐจ์ด๊ฐ€ ์„ ๋ช…ํ•˜๋‹ค. ๋‘˜ ๋‹ค โ€œkinematic ๋‹จ๊ณ„์—์„œ ๋ฌผ์ฒด๋ฅผ ๋ด์•ผ RL์ด dynamics์— ์ง‘์ค‘ํ•œ๋‹คโ€๋Š” ์ง„๋‹จ์„ ๊ณต์œ ํ•œ๋‹ค. DexOAK๋Š” reference๋งŒ ๋ฐ”๊พธ๊ณ  RL backend๋ฅผ ๊ณ ์ •ํ•œ ํ†ต์ œ ์‹คํ—˜(๊ฐ™์€ ManipTrans residual RL์—์„œ 20.6% โ†’ 48.2%)์œผ๋กœ ๊ทธ ์ง„๋‹จ์„ ์ธก์ •ํ–ˆ๋‹ค. ์ด ๋…ผ๋ฌธ์€ ๊ฐ™์€ ์ง„๋‹จ์„ ๋” ํฐ ๊ทœ๋ชจ(์† 4์ข…, ์‹ค๊ธฐ ์–‘ํŒ”)๋กœ ๋ฐ€์—ˆ์ง€๋งŒ reference๋งŒ ๋ฐ”๊พผ ํ†ต์ œ ๋น„๊ต๋Š” ์—†๋‹ค. ๋‘ ๋…ผ๋ฌธ์„ ํ•ฉ์ณ ์ฝ์œผ๋ฉด โ€œ๋ฌผ์ฒด ์ธ์ง€ reference๊ฐ€ downstream RL ์„ฑ๊ณต์„ ์˜ฌ๋ฆฐ๋‹คโ€๋Š” ๋ช…์ œ๋Š” DexOAK ์ชฝ ์ฆ๊ฑฐ๋กœ, โ€œ๊ทธ ์œ„์˜ residual ๋ ˆ์‹œํ”ผ๊ฐ€ ์† 4์ข…์— ์ด์‹๋œ๋‹คโ€๋Š” ๋ช…์ œ๋Š” ์ด ๋…ผ๋ฌธ ์ชฝ ์ฆ๊ฑฐ๋กœ ๋’ท๋ฐ›์นจ๋œ๋‹ค.

๊ทธ๋ฆฌ๊ณ  ๊ฐ™์€ ๊ทธ๋ฃน์˜ DexCanvas(arXiv:2510.15786)๊ฐ€ ์ด ๋…ผ๋ฌธ์˜ 1๋‹จ๊ณ„์— ํ•ด๋‹นํ•œ๋‹ค. DexCanvas๋Š” 70์‹œ๊ฐ„ mocap์„ MANO residual RL๋กœ ์žฌ๊ตฌ์„ฑํ•ด 7,000์‹œ๊ฐ„ ๋ถ„๋Ÿ‰์˜ force-annotated ๋ฐ์ดํ„ฐ์…‹์œผ๋กœ ํ‚ค์šด ๋ฐ์ดํ„ฐ์…‹ ๋…ผ๋ฌธ์ด๊ณ , v3์—์„œ๋Š” ๋‹จ์ผ policyยท์ฒ˜์Œ ๋ณด๋Š” YCB ๋ฌผ์ฒดยทDexHand 021/021Pro retargeting๊นŒ์ง€ ํฌํ•จํ•œ๋‹ค. ๋‘ ๋…ผ๋ฌธ ์‚ฌ์ด์—์„œ ์ด ๋…ผ๋ฌธ๋งŒ์˜ ๋ชซ์€ ์† 4์ข…์œผ๋กœ์˜ ํ™•์žฅ, retargeting ๋ฒ ์ด์Šค๋ผ์ธ ๋น„๊ต, ์–‘์† 25๊ณผ์ œ, ์‹ค๊ธฐ ์–‘ํŒ” ๋ฐฐํฌ๋กœ ์ฝ๋Š” ๊ฒƒ์ด ์ •ํ™•ํ•ด ๋ณด์ธ๋‹ค(ํ•„์ž์˜ ํ•ด์„). ๋…ผ๋ฌธ ๋ณธ๋ฌธ์€ ์ด ๊ฒฝ๊ณ„๋ฅผ ๊ทธ๋ฆฌ์ง€ ์•Š๋Š”๋‹ค.

์š”์•ฝ

์ด ๋…ผ๋ฌธ์€ ์‚ฌ๋žŒ mocap์„ ๋‹ค์ง€ ๋กœ๋ด‡ ์† policy๋กœ ๋ฐ”๊พธ๋Š” ํŒŒ์ดํ”„๋ผ์ธ์„ ์„ธ ๋‹จ๊ณ„๋กœ ๊ตฌ์„ฑํ•œ๋‹ค. ์‹œ๋ฎฌ๋ ˆ์ด์…˜ MANO ์†์„ ๋‹จ์ผ residual RL policy๋กœ ๋‹ค์‹œ ์‹คํ–‰ํ•ด ์ ‘์ด‰๊ณผ ํž˜์„ ๋ณต์›ํ•˜๊ณ , ๊ทธ ์ ‘์ด‰ ์œ„์น˜์— ๋กœ๋ด‡ ์† ํ‘œ๋ฉด์ ์„ ๋งž์ถ”๋Š” ์† ๋ชจ์–‘ ๋…๋ฆฝ์  ๋ชฉ์ ํ•จ์ˆ˜๋กœ retargetingํ•˜๊ณ , ๊ฐ™์€ residual ํ˜•์‹์˜ ๋‘ ๋ฒˆ์งธ policy๋กœ ๋กœ๋ด‡ ๊ตฌ๋™์— ๋งž์ถฐ ๋ณด์ •ํ•œ๋‹ค. ๊ฒฐ๊ณผ๋Š” ๋„“๋‹ค. ํ•œ ์† 2.5๋งŒ ๊ถค์ ์—์„œ 7.26% โ†’ 59.31%, ์† 4์ข…์—์„œ 17.1% โ†’ 79.5%, ์‹ค๊ธฐ ์–‘ํŒ” 4๊ณผ์ œ์—์„œ 21/40์ด ์‹ค๋กœ๋ด‡ ํ•™์Šต ๋ฐ์ดํ„ฐ ์—†์ด ๋‚˜์˜จ๋‹ค.

์ •์งํ•˜๊ฒŒ ๋‚˜๋ˆ„๋ฉด, ์ธก์ •๋œ ๊ฒƒ์€ ๊ฑฐ์˜ ๋ชจ๋‘ โ€œ์ž๊ธฐ ๋ฐฉ๋ฒ•์˜ replay ๋Œ€๋น„ residualโ€๊ณผ โ€œkinematic ์ง€ํ‘œ์—์„œ์˜ retargeting ๋น„๊ตโ€๋‹ค. ์ ‘์ด‰ ๊ตฌ์กฐ๊ฐ€ ๊ธฐํ•˜ ๋Œ€์‘๋ณด๋‹ค embodiment gap์„ ๋” ์ž˜ ๋‹ซ๋Š”๋‹ค๋Š” ํ•ต์‹ฌ ์ฃผ์žฅ์€ ํ๋ฃจํ”„ ๋น„๊ต ์—†์ด kinematic ์ง€ํ‘œ๋กœ๋งŒ ๋’ท๋ฐ›์นจ๋˜๊ณ , ๊ทธ ํ‘œ์—์„œ ์ด ๋ฐฉ๋ฒ•์€ ๊ฐ€์žฅ ๋А๋ฆฌ๋‹ค. ๋‹จ์ผ policy๊ฐ€ ๊ณผ์ œ๋ณ„ policy๋ฅผ ๋Œ€์ฒดํ•œ๋‹ค๋Š” ์ฃผ์žฅ๋„ ๋น„๊ต ์‹คํ—˜ ์—†์ด ์„ค๊ณ„ ๋…ผ์ฆ์œผ๋กœ ๋‚จ์•„ ์žˆ๋‹ค. ์ฒ˜์Œ ๋ณด๋Š” ๋ฌผ์ฒด ๊ฒฐ๊ณผ๋Š” ๊ฐ™์€ ๊ทธ๋ฃน DexCanvas์™€ ๊ณต์œ ๋œ ํ‘œ์ด๊ณ  ํ”„๋กœํ† ์ฝœ์ด ์„œ์ˆ ๋˜์ง€ ์•Š์•˜๋‹ค. ์ฝ”๋“œยท๋ฐ์ดํ„ฐยทํ•˜์ดํผํŒŒ๋ผ๋ฏธํ„ฐ๊ฐ€ ๊ณต๊ฐœ๋˜์ง€ ์•Š์•„ ์ง€๊ธˆ์€ ์ด ๊ฐ„๊ทน๋“ค์„ ๋…์ž๊ฐ€ ์ง์ ‘ ๋ฉ”์šธ ๋ฐฉ๋ฒ•๋„ ์—†๋‹ค. ์ ‘์ด‰ ๊ตฌ์กฐ๋ฅผ embodiment ๋ถˆ๋ณ€๋Ÿ‰์œผ๋กœ ์‚ผ๋Š” ๊ด€์ ๊ณผ residual ์ˆ˜์„ ๊ธฐ๋ฅผ amortizeํ•œ๋‹ค๋Š” ๋ฐฉํ–ฅ์€ ์„ค๋“๋ ฅ์ด ์žˆ์œผ๋ฏ€๋กœ, ๋‹ค๋ฅธ retargeter์™€์˜ ํ๋ฃจํ”„ ๋น„๊ต์™€ ์ฝ”๋“œ ๊ณต๊ฐœ๊ฐ€ ๋‚˜์˜ค๋ฉด ์ด ๋…ผ๋ฌธ์˜ ์œ„์น˜๊ฐ€ ๋ถ„๋ช…ํ•ด์งˆ ๊ฒƒ์ด๋‹ค.

์ด ๊ธ€์— ๋Œ€ํ•ด

  • ์ฝ์€ ๋ฒ”์œ„: arXiv v2(2026-11-03)์˜ PDF์™€ HTML์„ ํ•จ๊ป˜ ์ฝ์—ˆ๋‹ค. v1(2026-09-21)๊ณผ v2๋Š” ๋ณธ๋ฌธ ํ…์ŠคํŠธ๊ฐ€ ๊ฐ™๊ณ  ์กฐํŒ๋งŒ ๋‹ค๋ฅด๋‹ค. ์ด ๋…ผ๋ฌธ์—๋Š” ๋ถ€๋ก์ด ์—†๋‹ค. ๊ทธ๋ฆผ 2ยท4ยท5ยท6์€ arXiv HTML์˜ ์›๋ณธ ๊ทธ๋ฆผ์ด๊ณ , Fig. 1๊ณผ Fig. 3์€ PDF์—์„œ ์ž˜๋ผ ์‹ค์—ˆ๋‹ค. ๋…ผ๋ฌธ์˜ ๊ทธ๋ฆผ์€ ์ด 6์žฅ์ด ์ „๋ถ€๋‹ค.
  • ์˜์ƒ: Ping์˜ ์˜์ƒ 4๊ฐœ๋Š” ๊ณต์‹ GitHub ๋ ˆํฌ README์˜ ๋ฐ๋ชจ ์˜์ƒ(Apache-2.0)์„ ํ•ด์ƒ๋„๋งŒ ์ค„์—ฌ ์‹ค์€ ๊ฒƒ์ด๋‹ค.
  • ๊ณต๊ฐœ ๋ฒ”์œ„: ์ฝ”๋“œยท๋ฐ์ดํ„ฐยท์ฒดํฌํฌ์ธํŠธ๋Š” ๊ณต๊ฐœ๋˜์ง€ ์•Š์•˜๋‹ค(๋ณธ๋ฌธ โ€œ๊ณต๊ฐœ ์ƒํƒœโ€ ์ ˆ). ๊ทธ๋ž˜์„œ ๊ตฌํ˜„ ์„ธ๋ถ€๋Š” ๋…ผ๋ฌธ ์„œ์ˆ  ๋ฒ”์œ„๋กœ ํ•œ์ •ํ–ˆ๋‹ค.
  • ๊ด€๋ จ ๋…ผ๋ฌธ ๋Œ€์กฐ: DexCanvas(arXiv:2510.15786)๋Š” v1๊ณผ v3๋ฅผ ์ฝ์–ด ์ด ๊ธ€์˜ ๋น„๊ต์— ์ผ๋‹ค.
  • ํ‘œ๊ธฐ: โ€œํ•„์ž ๊ณ„์‚ฐโ€์€ ๋…ผ๋ฌธ ํ‘œ์˜ ์นธ ๊ฐ’์œผ๋กœ ๋‹ค์‹œ ๊ณ„์‚ฐํ•œ ๊ฐ’์ด๊ณ , โ€œํ•„์ž ํŒ๋…โ€์€ ๊ทธ๋ฆผ์—์„œ ์ฝ์€ ๊ทผ์‚ฌ๊ฐ’์ด๋ฉฐ, โ€œํ•„์ž์˜ ํ•ด์„โ€์€ ๋…ผ๋ฌธ์— ์—†๋Š” ํ•ด์„์ด๋‹ค. ๊ทธ ๋ฐ–์˜ ์ˆ˜์น˜๋Š” ๋ชจ๋‘ ๋…ผ๋ฌธ์— ์ ํžŒ ๊ฐ’์ด๋‹ค.

Copyright 2026, JungYeon Lee