Curieux.JY
  • JungYeon Lee
  • Post
  • ๐Ÿ•ธ๏ธ Graph
  • Lecture
  • Note

On this page

  • ๐Ÿ” Ping Review
  • ๐Ÿ”” Ring Review
    • ํ•œ ์ค„๋กœ ์‹œ์ž‘ํ•˜๋ฉด
    • ๋ฐฐ๊ฒฝ: ์ด‰๊ฐ์€ ์™œ generalist policy์—์„œ ๋น ์ ธ ์žˆ์—ˆ๋‚˜ (ยง1, ๋ถ€๋ก A)
    • ๋ฐฉ๋ฒ• ์ƒ์„ธ 1: ์ „์ฒด ๊ตฌ์กฐ (ยง2, Fig. 2)
    • ๋ฐฉ๋ฒ• ์ƒ์„ธ 2: Morphology-Aware Tactile Token Space (ยง2.1, Fig. 3)
    • ๋ฐฉ๋ฒ• ์ƒ์„ธ 3: ์ด์งˆ์  ์ด‰๊ฐ encoder (ยง2.2, ๋ถ€๋ก B.2)
    • ๋ฐฉ๋ฒ• ์ƒ์„ธ 4: Unified Action Space์™€ FAAS (๋ถ€๋ก B.1)
    • ๋ฐฉ๋ฒ• ์ƒ์„ธ 5: ์‚ฌ์ „ํ•™์Šต ๋ฐ์ดํ„ฐ (ยง2.4, ๋ถ€๋ก C, Fig. 4)
    • ์ง๊ด€: ์™œ โ€œ์–ด๋””์— ๋‹ฟ์•˜๋‚˜โ€๊ฐ€ ๊ณตํ†ต ์–ธ์–ด๊ฐ€ ๋˜๋‚˜
    • ์‹คํ—˜ ์„ค์ •: ๋‹ค์„ฏ ์…‹์—…, ์„ธ ๋ฒ ์ด์Šค๋ผ์ธ (ยง3.1ยทยง3.2, ๋ถ€๋ก DยทE)
    • ์‹คํ—˜ 1: ์‚ฌ์ „ํ•™์Šต์— ์žˆ๋˜ ์„ผ์„œ (ยง3.3, Table 1ยท2)
      • UniVTAC ์‹œ๋ฎฌ๋ ˆ์ด์…˜ (Table 1)
      • ์‹ค๊ธฐ (Table 2)
    • ์‹คํ—˜ 2: ์‚ฌ์ „ํ•™์Šต์— ์—†๋˜ ์„ผ์„œ (ยง4.1, Table 3)
    • ์‹คํ—˜ 3: ์ด๋“์€ ์ด‰๊ฐ ์‚ฌ์ „ํ•™์Šต์—์„œ ์˜ค๋Š”๊ฐ€ (ยง4.2, Fig. 7, Table 10)
      • ๊ฐ™์€ ๊ถค์ ์— ๋…ธ์ถœ๋œ VLA์™€์˜ ๋น„๊ต (ยง4.3, ๋ถ€๋ก F.1, Table 4)
      • ์‚ฌ์ „ํ•™์Šต ๊ทœ๋ชจ (ยง4.3, ๋ถ€๋ก F.2, Table 11)
      • ๊ณ ์ • ์ด‰๊ฐ feature probing (ยง4.3, ๋ถ€๋ก F.3, Table 12)
    • ์ •์„ฑ ๊ฒฐ๊ณผ์™€ ์˜์ƒ (๋ถ€๋ก E.3ยทE.5, ํ”„๋กœ์ ํŠธ ํŽ˜์ด์ง€)
    • ๊ณต๊ฐœ ์ฝ”๋“œยท์ฒดํฌํฌ์ธํŠธ์™€ ๋Œ€์กฐ (์ •์  ํ™•์ธ)
    • ๋น„ํŒ์ ์œผ๋กœ ๋ณด๋ฉด
      • ๊ฐ•์ 
      • ์•ฝ์ ยทํ•œ๊ณ„
    • ๊ด€๋ จ ์—ฐ๊ตฌ์™€์˜ ์ž๋ฆฌ ๋งค๊น€
    • ์š”์•ฝ
    • ์ด ๊ธ€์— ๋Œ€ํ•ด

๐Ÿ“ƒFTP-1 ๋ฆฌ๋ทฐ

tactile
vla
foundation-model
cross-embodiment
pretraining
FTP-1: A Generalist Foundation Tactile Policy Across Tactile Sensors for Contact-Rich Manipulation
Published

November 11, 2026

  • Paper Link (arXiv:2606.13102)

  • Code Link (github.com/michaelyuancb/ftp1-policy)

  • Project

  • Model (Hugging Face MJJJJ1064/ftp1_v0426_50kstep)

  • Dataset (ModelScope Eureka1064/FTP-1-Dataset)

  • Chengbo Yuan, Zicheng Zhang, Mingjie Zhou, Wendi Chen, Yi Wang, Zhuoyang Liu, Dantong Niu, Shuo Wang, Hui Zhang, Wenkang Zhang, Yingdong Hu, Yuanqing Gong, Wanli Xing, Chuan Wen, Cewu Lu, Kaifeng Zhang, Yang Gao (Tsinghua University ยท Shanghai Qi Zhi Institute ยท Sharpa ยท Shanghai Jiao Tong University ยท UC Berkeley ยท ETH Zurich ยท Fudan University ยท Shanghai Innovation Institute)

  • arXiv preprint, 2026

  1. ๐Ÿ’ก ์ด‰๊ฐ policy๊ฐ€ ์„ผ์„œ ํ•˜๋‚˜์— ๋ฌถ์—ฌ ์žˆ๋Š” ์ด์œ ๋ฅผ ์„ผ์„œ ๊ฐ„ ์‹ ํ˜ธ์˜ ์ด์งˆ์„ฑ(imageยทarrayยทstate, ํ•ด์ƒ๋„, ํ˜•ํƒœ)์œผ๋กœ ๋ณด๊ณ , ์†์˜ ๊ธฐ๋Šฅ ์˜์—ญ 24์นธ์œผ๋กœ ๋œ ๊ณตํ†ต ํ† ํฐ ๊ณต๊ฐ„(MTTS)๊ณผ ๋ณ„๋„์˜ tactile expert๋ฅผ \pi_{0.5} ์œ„์— ์–น์–ด ์—ฌ๋Ÿฌ ์„ผ์„œ๋ฅผ ํ•˜๋‚˜์˜ policy๋กœ ์‚ฌ์ „ํ•™์Šตํ•˜๋Š” FTP-1์„ ์ œ์•ˆํ•œ๋‹ค.
  2. โš™๏ธ ์„ผ์„œ๋งˆ๋‹ค ์ž‘์€ encoder(์ด๋ฏธ์ง€๋Š” ViT + ๊ณต์œ  T3 trunk, array๋Š” CNN, state๋Š” Fourier MLP)๊ฐ€ ๊ธฐ๋Šฅ ์˜์—ญ๋ณ„ ํ† ํฐ 1๊ฐœ์”ฉ์„ ๋งŒ๋“ค๊ณ , ์—ฌ๊ธฐ์— ์„ผ์„œ ๊ณตํ†ต์˜ functional-area embedding์„ ๋”ํ•ด tactile expert๊ฐ€ ์ฒ˜๋ฆฌํ•œ๋‹ค. action expert๋Š” vision-language ํ† ํฐ๊ณผ tactile ํ† ํฐ์„ ๋ชจ๋‘ attendํ•ด flow matching์œผ๋กœ action chunk๋ฅผ ๋‚ธ๋‹ค. 26๊ฐœ ์ถœ์ฒ˜ยท21๊ฐœ ์„ผ์„œยท์•ฝ 3,000์‹œ๊ฐ„ ๋ฐ์ดํ„ฐ๋กœ ์‚ฌ์ „ํ•™์Šตํ•œ๋‹ค.
  3. ๐ŸŽฏ ์‚ฌ์ „ํ•™์Šต์— ์žˆ๋˜ ์„ผ์„œ์˜ ์„ธ ์…‹์—…์—์„œ UniVTAC ํ‰๊ท  66.66%(์ฐจ์ˆœ์œ„ 49.16%), ์‹ค๊ธฐ ํ‰๊ท  62.5%(์ฐจ์ˆœ์œ„ \pi_{0.5} 45.3%)๋ฅผ, ์‚ฌ์ „ํ•™์Šต์— ์—†๋˜ XenseยทContactile ๋‘ ์…‹์—…์—์„œ 46.6%(FTP-\pi_{0.5}ยท\pi_{0.5} 15.0%)๋ฅผ ๋ณด๊ณ ํ•œ๋‹ค. ์‹ค๊ธฐ๋Š” ๊ณผ์ œ๋‹น 20ํšŒ, ํ•™์Šต 1ํšŒ ๊ธฐ์ค€์ด๋‹ค.

๐Ÿ” Ping Review

๐Ÿ” Ping โ€” A light tap on the surface. Get the gist in seconds.

์‹œ๊ฐ ๊ธฐ๋ฐ˜ generalist policy(\pi_{0.5}, GR00T ๋“ฑ)๋Š” ํฐ ์ด์งˆ์  ๋ฐ์ดํ„ฐ๋กœ ์‚ฌ์ „ํ•™์Šตํ•œ ์ฒดํฌํฌ์ธํŠธ๋ฅผ ์ƒˆ ๋กœ๋ด‡์— ๋ฏธ์„ธ์กฐ์ •ํ•˜๋Š” ํ๋ฆ„์„ ๋งŒ๋“ค์—ˆ๋‹ค. ์ด‰๊ฐ ์ชฝ์€ ๊ทธ๋ ‡์ง€ ๋ชปํ–ˆ๋‹ค. GelSight ๋ฅ˜ ๊ด‘ํ•™ ์„ผ์„œ๋Š” ์ด๋ฏธ์ง€๋ฅผ, ํ•€ ๋ฐฐ์—ด ์„ผ์„œ๋Š” H\times W\times D ๋ฐฐ์—ด์„, ํž˜/ํ† ํฌ ์„ผ์„œ๋Š” ๋ช‡ ์ฐจ์› ๋ฒกํ„ฐ๋ฅผ ๋‚ธ๋‹ค. ์„ผ์„œ๋งˆ๋‹ค ์‹ ํ˜ธ ๋ชจ์–‘์ด ๋‹ฌ๋ผ์„œ ํ•œ policy๊ฐ€ ์—ฌ๋Ÿฌ ์„ผ์„œ์˜ ๋ฐ์ดํ„ฐ๋ฅผ ํ•จ๊ป˜ ๋จน๊ธฐ ์–ด๋ ต๊ณ , ๊ทธ๋ž˜์„œ ๊ธฐ์กด ์ด‰๊ฐ policy๋‚˜ tactile VLA๋Š” ๋Œ€๊ฐœ ์„ผ์„œ ํ•˜๋‚˜์™€ ๋กœ๋ด‡ ํ•˜๋‚˜์— ๋ฌถ์—ฌ ์žˆ๋‹ค.

FTP-1์€ ์ด ์ด์งˆ์„ฑ์„ โ€œ์„ผ์„œ๊ฐ€ ์†์˜ ์–ด๋””์— ๋ถ™์–ด ์žˆ๋Š”๊ฐ€โ€๋ผ๋Š” ๊ณตํ†ต ์ขŒํ‘œ๋กœ ํก์ˆ˜ํ•œ๋‹ค. ์‹ ํ˜ธ ๋ชจ์–‘์€ ์„ผ์„œ๋ณ„ encoder๊ฐ€ ๋งก๊ณ , ๊ทธ ์ถœ๋ ฅ์€ ์†์˜ ๊ธฐ๋Šฅ ์˜์—ญ(์†๋, ์†๊ฐ€๋ฝ ๋งˆ๋””, ์†๋ฐ”๋‹ฅ, ์†๋ชฉ ํ† ํฌ ๋“ฑ) 24์นธ ์ค‘ ํ•˜๋‚˜์˜ ํ† ํฐ์ด ๋œ๋‹ค. ๊ฐ™์€ ์นธ์—๋Š” ์„ผ์„œ๊ฐ€ ๋‹ฌ๋ผ๋„ ๊ฐ™์€ learnable embedding์ด ๋”ํ•ด์ง„๋‹ค. ๊ทธ ์œ„์˜ tactile expert์™€ action expert๋Š” ์„ผ์„œ๋ฅผ ๋ชจ๋ฅธ๋‹ค. ์ƒˆ ์„ผ์„œ๊ฐ€ ์˜ค๋ฉด encoder๋งŒ ์ƒˆ๋กœ ๋งŒ๋“ค๊ณ  ๋‚˜๋จธ์ง€๋Š” ์‚ฌ์ „ํ•™์Šต ๊ฐ€์ค‘์น˜์—์„œ ์‹œ์ž‘ํ•œ๋‹ค.


Fig. 1. 26๊ฐœ ์ถœ์ฒ˜์˜ ์ด์งˆ์  ์ด‰๊ฐ ๋ฐ์ดํ„ฐ๋กœ ์‚ฌ์ „ํ•™์Šตํ•œ FTP-1์„ 5๊ฐœ ์…‹์—…์—์„œ ๋ฏธ์„ธ์กฐ์ •ํ•ด ํ‰๊ฐ€ํ•œ๋‹ค. ๋ง‰๋Œ€๋Š” ์‚ฌ์ „ํ•™์Šต์— ์žˆ๋˜ ์„ผ์„œ์˜ ์‹ค๊ธฐ ํ‰๊ท (45.3 โ†’ 62.5%)๊ณผ ์—†๋˜ ์„ผ์„œ์˜ ํ‰๊ท (15.0 โ†’ 46.6%)์ด๋‹ค.

ํ•ต์‹ฌ ๋ฐฉ๋ฒ•๋ก :

์ž…๋ ฅ์€ ์–ธ์–ด ์ง€์‹œ \ell, (๋‹ค์‹œ์ ) RGB \mathcal I_t, proprioception \mathbf s_t, ์ด‰๊ฐ ๊ด€์ธก \mathcal X_t์ด๊ณ  ์ถœ๋ ฅ์€ ๊ธธ์ด H์˜ action chunk๋‹ค.

\hat{\mathbf{A}}_{t:t+H-1}=\pi_\theta(\ell,\mathcal{I}_t,\mathbf{s}_t,\mathcal{X}_t),\qquad \hat{\mathbf{A}}_{t:t+H-1}\in\mathbb{R}^{H\times D}

D๋Š” ํŒ”ยท์†ยท๋จธ๋ฆฌ ์ œ์–ด๋ฅผ ๊ณ ์ • ๊ธธ์ด ๋ฒกํ„ฐ์— ์นธ์œผ๋กœ ๋ฐฐ์น˜ํ•œ Unified Action Space(UAS)์˜ ์ฐจ์›์ด๋‹ค. ์ด‰๊ฐ ํ† ํฐ์€ ๊ธฐ๋Šฅ ์˜์—ญ k๋งˆ๋‹ค

\mathbf z_k=\mathrm{MLP}\big(\mathrm{LN}(E_{\text{sensor},\,\text{type}}(x_k))+\mathbf e^{\text{area}}_k\big)

๋กœ ๋งŒ๋“ค์–ด์ง„๋‹ค(E๋Š” ์„ผ์„œยท๊ด€์ธก ์œ ํ˜•๋ณ„ encoder, \mathbf e^{\text{area}}_k๋Š” ์„ผ์„œ ๊ณตํ†ต functional-area embedding, ๋ถ€๋ก B.2์˜ ์„œ์ˆ ์„ ์‹์œผ๋กœ ์˜ฎ๊ธด ๊ฒƒ). ์„ธ expert(vision-language, tactile, action)๋Š” ์ธต๋งˆ๋‹ค QยทKยทV๋ฅผ ์ด์–ด ๋ถ™์—ฌ ํ•˜๋‚˜์˜ masked attention์„ ์“ฐ๋Š”๋ฐ, ๋งˆ์Šคํฌ๋Š” ํ–‰=query, ์—ด=key ์ˆœ์„œ(VL, tactile, action)๋กœ

\mathbf M_{\text{expert}}=\begin{bmatrix} 1 & 0 & 0 \\ 0 & 1 & 0 \\ 1 & 1 & 1 \end{bmatrix}

์ด๋‹ค. ๋‘ ์ง€๊ฐ expert๋Š” ์„œ๋กœ๋ฅผ ๋ณด์ง€ ์•Š๊ณ , action ํ† ํฐ๋งŒ ๋‘˜ ๋‹ค ๋ณธ๋‹ค.

์ฃผ์š” ๊ฒฐ๊ณผ:

  • UniVTAC ์‹œ๋ฎฌ๋ ˆ์ด์…˜(์‚ฌ์ „ํ•™์Šต์— ์žˆ๋˜ GelSight-Mini, ๊ณผ์ œ๋‹น 100ํšŒ): 6๊ณผ์ œ ํ‰๊ท  66.66%, ๋‘ lift ๊ณผ์ œ๋ฅผ ๋บ€ ํ‰๊ท  59.5%. ์ฐจ์ˆœ์œ„๋Š” ๊ฐ๊ฐ \pi_{0.5} 49.16%, FTP-\pi_{0.5} 42.0%(Table 1).
  • ์‹ค๊ธฐ, ์‚ฌ์ „ํ•™์Šต์— ์žˆ๋˜ Sharpa DTC(๊ณผ์ œ๋‹น 20ํšŒ): ํ‰๊ท  62.5%, \pi_{0.5} 45.3%, FTP-\pi_{0.5} 41.6%, Tactile-VLA 35.8%(Table 2). ํ‘œ์˜ ๋‹ค๋ฅธ ์นธ์œผ๋กœ ๋‹ค์‹œ ๊ณ„์‚ฐํ•˜๋ฉด \pi_{0.5} ํ‰๊ท ์€ 45.83%๋‹ค(์•„๋ž˜ ์‹คํ—˜ ์ ˆ).
  • ์‹ค๊ธฐ, ์‚ฌ์ „ํ•™์Šต์— ์—†๋˜ XenseยทContactile(๊ณผ์ œ๋‹น 20ํšŒ): ํ‰๊ท  46.6%, \pi_{0.5}ยทFTP-\pi_{0.5} 15.0%, Tactile-VLA 8.3%(Table 3).
  • ์ด‰๊ฐ ์‚ฌ์ „ํ•™์Šต ์ ˆ์ œ: ๊ฐ™์€ ๋ฐ์ดํ„ฐ๋กœ ์ด‰๊ฐ๋งŒ ๋นผ๊ณ  ์‚ฌ์ „ํ•™์Šตํ•œ NTP-1์€ FlexivXense์—์„œ 5.0%, FTP-1์€ 42.5%(Fig. 7). ๊ฐ™์€ ๋กœ๋ด‡ ๊ถค์ ์œผ๋กœ \pi_{0.5}๋ฅผ ์ด์–ด ํ•™์Šตํ•œ ๋Œ€์กฐ๊ตฐ์€ UniVTAC 57.0% / lift ์ œ์™ธ 44.0%(Table 4).

๊ฒฐ๋ก :

FTP-1์˜ ์ƒˆ๋กœ์›€์€ ์ด‰๊ฐ์„ VLM์— adapter๋กœ ๋ผ์›Œ ๋„ฃ๋Š” ๋Œ€์‹  ์„ผ์„œ ๋ฌด๊ด€ํ•œ tactile expert ํ•˜๋‚˜๋ฅผ ์‚ฌ์ „ํ•™์Šต ๋Œ€์ƒ์œผ๋กœ ๋งŒ๋“  ๊ฒƒ, ๊ทธ๋ฆฌ๊ณ  ๊ทธ expert๊ฐ€ ์„ผ์„œ๋งˆ๋‹ค ๋‹ค๋ฅธ ์ž…๋ ฅ์„ โ€œ์†์˜ ์–ด๋А ์˜์—ญ์ธ๊ฐ€โ€๋กœ ์ •๋ ฌ๋œ ํ† ํฐ์œผ๋กœ ๋ฐ›๊ฒŒ ํ•œ ๊ฒƒ์ด๋‹ค. ์‹คํ—˜์€ ๋‹ค์„ฏ ๊ธฐ๊ด€์˜ ๋‹ค์„ฏ ์…‹์—…์— ์ฒดํฌํฌ์ธํŠธ๋ฅผ ๋ฐฐํฌํ•ด ๋ฏธ์„ธ์กฐ์ •ํ•˜๋Š” ํ˜•ํƒœ๋กœ, ์ด‰๊ฐ foundation policy๋ฅผ ์ฒ˜์Œ์œผ๋กœ ์ด ์ •๋„ ํญ์—์„œ ํ‰๊ฐ€ํ–ˆ๋‹ค. ๋‹ค๋งŒ ์‹ค๊ธฐ ๋น„๊ต๋Š” ๊ณผ์ œ๋‹น 20ํšŒยทํ•™์Šต 1ํšŒ๋ผ ๊ณผ์ œ ๋‹จ์œ„๋กœ๋Š” ๋Œ€๋ถ€๋ถ„ ํ†ต๊ณ„์ ์œผ๋กœ ๊ตฌ๋ถ„๋˜์ง€ ์•Š๊ณ , ์ด‰๊ฐ ์‚ฌ์ „ํ•™์Šต์˜ ์ˆœ์ˆ˜ ํšจ๊ณผ๋ฅผ ๋ถ„๋ฆฌํ•œ ํ†ต์ œ ์‹คํ—˜์€ UniVTAC์™€ FlexivXense ๋‘ ๊ณณ์—๋งŒ ์žˆ๋‹ค. ๊ณต๊ฐœ ์ฒดํฌํฌ์ธํŠธ์˜ tactile expert ํฌ๊ธฐ๊ฐ€ ๋…ผ๋ฌธ ์„œ์ˆ (300M)๊ณผ ๋‹ค๋ฅด๋‹ค๋Š” ์ ๋„ ํ™•์ธํ•ด์•ผ ํ•  ์ง€์ ์ด๋‹ค.


๐Ÿ”” Ring Review

๐Ÿ”” Ring โ€” An idea that echoes. Grasp the core and its value.

ํ•œ ์ค„๋กœ ์‹œ์ž‘ํ•˜๋ฉด

์ด‰๊ฐ ์‹ ํ˜ธ๋ฅผ ํ•˜๋‚˜์˜ ํฌ๋งท์œผ๋กœ ํ†ต์ผํ•˜๋Š” ๋Œ€์‹  โ€œ์†์˜ ์–ด๋А ๋ถ€๋ถ„์ด ๋ฌด์—‡์„ ๋А๋ผ๋Š”๊ฐ€โ€๋ฅผ ํ†ต์ผํ•˜๋ฉด, ์„ผ์„œ๊ฐ€ ๋‹ฌ๋ผ๋„ ๊ฐ™์€ tactile expert๊ฐ€ ๊ฐ™์€ ์˜๋ฏธ์˜ ํ† ํฐ์„ ๋ฐ›๋Š”๋‹ค. FTP-1์€ ์ด ์•„์ด๋””์–ด๋ฅผ \pi_{0.5}์˜ multi-expert ๊ตฌ์กฐ ์•ˆ์— tactile expert ํ•˜๋‚˜๋ฅผ ์ถ”๊ฐ€ํ•˜๋Š” ๊ฐ€์žฅ ๋‹จ์ˆœํ•œ ํ˜•ํƒœ๋กœ ๊ตฌํ˜„ํ•˜๊ณ , ์‚ฌ์ „ํ•™์Šต ๋ฐ์ดํ„ฐ๋ฅผ 26๊ฐœ ์ถœ์ฒ˜์—์„œ ๊ธ์–ด๋ชจ์•„ โ€œ์„ผ์„œ๊ฐ€ ๋ฐ”๋€Œ์–ด๋„ ๋ฏธ์„ธ์กฐ์ • ์ถœ๋ฐœ์ ์œผ๋กœ ์“ธ ์ˆ˜ ์žˆ๋Š”๊ฐ€โ€๋ฅผ ๋ฌป๋Š”๋‹ค. ๋‹ต์€ โ€œ๊ทธ๋ ‡๋‹ค, ํŠนํžˆ ์ƒˆ ์„ผ์„œ์—์„œโ€์ธ๋ฐ, ๊ทธ ๊ทผ๊ฑฐ์˜ ๊ฐ•๋„๋Š” ์‹คํ—˜๋งˆ๋‹ค ๊ฝค ๋‹ค๋ฅด๋‹ค.

๋ฐฐ๊ฒฝ: ์ด‰๊ฐ์€ ์™œ generalist policy์—์„œ ๋น ์ ธ ์žˆ์—ˆ๋‚˜ (ยง1, ๋ถ€๋ก A)

์‹œ๊ฐ generalist policy๋Š” RGB๋ผ๋Š” ๊ณตํ†ต ์ž…๋ ฅ์„ ๊ฐ€์ง„๋‹ค. ์นด๋ฉ”๋ผ๊ฐ€ ๋‹ฌ๋ผ๋„ ์ด๋ฏธ์ง€๋Š” ์ด๋ฏธ์ง€๋‹ค. ์ด‰๊ฐ์€ ๊ทธ๋ ‡์ง€ ์•Š๋‹ค. ๋…ผ๋ฌธ์ด ์ •๋ฆฌํ•œ ์ด์งˆ์„ฑ์€ ๋„ค ์ถ•์ด๋‹ค: modality(๊ด‘ํ•™ ์ด๋ฏธ์ง€ / ์ „๊ทนยท์••์ €ํ•ญ ๋ฐฐ์—ด / ํž˜ยทํ† ํฌ ๋ฒกํ„ฐ), resolution, morphology(์†๋์— ๋ถ™์—ˆ๋‚˜ ์†๋ฐ”๋‹ฅ์— ๋ถ™์—ˆ๋‚˜, ๊ทธ๋ฆฌํผ ํŒจ๋“œ์ธ๊ฐ€), contact response. ๊ฐ™์€ โ€œ์ ‘์ด‰โ€์ด GelSight-Mini์—์„œ๋Š” 224ร—224 ์ด๋ฏธ์ง€์˜ ๋ณ€ํ˜•์œผ๋กœ, Contactile์—์„œ๋Š” 3ร—3 ํ•€์˜ 3์ถ• ํž˜์œผ๋กœ, ์†๋ชฉ F/T ์„ผ์„œ์—์„œ๋Š” 6์ฐจ์› ๋ฒกํ„ฐ๋กœ ๋‚˜ํƒ€๋‚œ๋‹ค.

์ด ๋•Œ๋ฌธ์— ๊ธฐ์กด ํ๋ฆ„์€ ๋‘˜๋กœ ๋‚˜๋‰˜์—ˆ๋‹ค(๋ถ€๋ก A.2ยทA.3).

  • ์ด‰๊ฐ ํ‘œํ˜„ ์‚ฌ์ „ํ•™์Šต: T3(FoTa, 13๊ฐœ ์„ผ์„œ), Sparsh, AnyTouch ๋“ฑ์€ ์—ฌ๋Ÿฌ ์„ผ์„œ๋ฅผ ํ•˜๋‚˜์˜ ํ‘œํ˜„์œผ๋กœ ๋ฌถ์—ˆ์ง€๋งŒ action์„ ๋‚ด์ง€ ์•Š๋Š”๋‹ค. downstream policy head๋Š” ๋”ฐ๋กœ ํ•™์Šตํ•œ๋‹ค. ๋ธ”๋กœ๊ทธ์˜ Sparsh ๋ฆฌ๋ทฐ, UniTouch ๋ฆฌ๋ทฐ๊ฐ€ ์ด ๊ณ„์—ด์ด๋‹ค.
  • ์ด‰๊ฐ policy / VTLA: Tactile-VLA, OmniVTLA, OmniVTA ๋“ฑ์€ action์„ ๋‚ด์ง€๋งŒ ํŠน์ • ์„ผ์„œยท๋กœ๋ด‡์— ๋ฌถ์—ฌ ์žˆ๋‹ค. ๋Œ€๊ฐœ VLM expert์— ์ด‰๊ฐ์„ adapter๋กœ ์ฃผ์ž…ํ•œ๋‹ค. TacFiLM ๋ฆฌ๋ทฐ์˜ FiLM ์กฐ๊ฑดํ™”๋„ ์ด ์ชฝ์˜ ๊ฒฝ๋Ÿ‰ ์œตํ•ฉ์ด๋‹ค.

FTP-1์ด ๊ฒจ๋ƒฅํ•˜๋Š” ์นธ์€ ๋‘ ๊ณ„์—ด์˜ ๊ต์ง‘ํ•ฉ์ด๋‹ค: action์„ ์ง์ ‘ ๋‚ด๋Š” policy์ด๋ฉด์„œ, ์—ฌ๋Ÿฌ ์„ผ์„œ๋กœ ์‚ฌ์ „ํ•™์Šต๋˜๊ณ , ์‚ฌ์ „ํ•™์Šต์— ์—†๋˜ ์„ผ์„œ๋กœ ๋ฏธ์„ธ์กฐ์ •ํ–ˆ์„ ๋•Œ ์ด๋“์ด ์žˆ๋Š”๊ฐ€. ๋ถ€๋ก Table 5ยท6์ด ์ด ์นธ์„ ๊ธฐ์กด ๋ชจ๋ธ๊ณผ ๋Œ€์กฐํ•˜๋Š”๋ฐ, ๊ทธ ํ‘œ์—์„œ โ€œunseen sensor์—์„œ์˜ policy ๋ฏธ์„ธ์กฐ์ •โ€์„ ๋ณด๊ณ ํ•œ ๋ชจ๋ธ์€ FTP-1๋ฟ์ด๋‹ค. ํ‘œ์˜ โ€œNRโ€์€ โ€œ๊ทธ ๋…ผ๋ฌธ์ด ๋ณด๊ณ ํ•˜์ง€ ์•Š์•˜๋‹คโ€๋Š” ๋œป์ด์ง€ โ€œํ•  ์ˆ˜ ์—†๋‹คโ€๋Š” ๋œป์ด ์•„๋‹ˆ๋ผ๊ณ  ์ €์ž๋“ค์ด ๋ช…์‹œํ•œ๋‹ค.

๋ฐฉ๋ฒ• ์ƒ์„ธ 1: ์ „์ฒด ๊ตฌ์กฐ (ยง2, Fig. 2)


Fig. 2. FTP-1 ๊ตฌ์กฐ. ์„ผ์„œ๋ณ„ encoder(Stateโ†’MLP, Arrayโ†’CNN, Imageโ†’ViT)๊ฐ€ ๋งŒ๋“  ํ† ํฐ์— functional-area embedding์„ ๋”ํ•ด tactile expert๋กœ ๋ณด๋‚ด๊ณ , VL expert(SigLIP + PaliGemma)์™€ action expert๊ฐ€ ํ•จ๊ป˜ action chunk๋ฅผ ์ƒ์„ฑํ•œ๋‹ค. proprioception์€ AdaRMS๋กœ action expert์— ๋“ค์–ด๊ฐ„๋‹ค.

FTP-1์€ \pi_{0.5}(๋ธ”๋กœ๊ทธ์˜ ฯ€0 ๋ฆฌ๋ทฐ๊ฐ€ ๋‹ค๋ฃฌ multi-expert flow-matching ๊ตฌ์กฐ์˜ ํ›„์†)๋ฅผ ๊ทธ๋Œ€๋กœ ๊ณจ๊ฒฉ์œผ๋กœ ์“ด๋‹ค. ๋ฐ”๋€ ๊ฒƒ์€ ์„ธ ๊ฐ€์ง€๋‹ค.

  1. tactile expert ์ถ”๊ฐ€: VL expert, action expert ์˜†์— ์„ธ ๋ฒˆ์งธ Transformer expert๋ฅผ ๋‘”๋‹ค. ๊ฐ expert๋Š” attention projection๊ณผ FFN์„ ๋”ฐ๋กœ ๊ฐ€์ง€์ง€๋งŒ, ์ธต๋งˆ๋‹ค ์„ธ expert์˜ QยทKยทV๋ฅผ ์ด์–ด ํ•˜๋‚˜์˜ masked attention์„ ๊ณ„์‚ฐํ•œ๋‹ค(๋ถ€๋ก B.4). ์œ„์˜ \mathbf M_{\text{expert}} ๋•Œ๋ฌธ์— tactile expert๋Š” ์ด‰๊ฐ ํ† ํฐ๋ผ๋ฆฌ๋งŒ, VL expert๋Š” VL ํ† ํฐ๋ผ๋ฆฌ๋งŒ ๋ณด๊ณ , action ํ† ํฐ๋งŒ ์„ธ ํ๋ฆ„์„ ๋ชจ๋‘ ๋ณธ๋‹ค.
  2. proprioception ์ฃผ์ž… ๋ฐฉ์‹: proprioception์„ Fourier feature โ†’ 3์ธต MLP โ†’ LayerNorm์œผ๋กœ ๋งŒ๋“  ๋’ค, flow-matching timestep feature์™€ ์ด์–ด ๋ถ™์—ฌ adaptive RMSNorm์œผ๋กœ attention block์„ ์กฐ๊ฑดํ™”ํ•œ๋‹ค(๋ถ€๋ก B.3). ๋…๋ฆฝ proprioception ํ† ํฐ๋ณด๋‹ค ์ผ๋ฐ˜ํ™”ยท๊ฐ•๊ฑด์„ฑ์ด ์ข‹์•˜๋‹ค๊ณ  ์ ๋Š”๋‹ค. ๊ทธ ๋น„๊ต์˜ ์ˆ˜์น˜๋Š” ์—†๋‹ค.
  3. Unified Action Space: ๋กœ๋ด‡๋งˆ๋‹ค ๋‹ค๋ฅธ ์ œ์–ด ์‹ ํ˜ธ๋ฅผ ๊ณ ์ • ๊ธธ์ด ํฌ์†Œ ๋ฒกํ„ฐ์— ์นธ์œผ๋กœ ๋ฐฐ์น˜ํ•˜๊ณ , ์—†๋Š” ์นธ์€ loss mask๋กœ ์ง€์šด๋‹ค(๋ถ€๋ก B.1, ์•„๋ž˜ ๋ฐฉ๋ฒ• ์ƒ์„ธ 4).

์ €์ž๋“ค์ด tactile expert๋ฅผ ๋”ฐ๋กœ ๋‘” ์ด์œ ๋Š” ์„ธ ๊ฐ€์ง€๋กœ ์ ํ˜€ ์žˆ๋‹ค(ยง2.3). โ‘  ์ƒˆ ์„ผ์„œ์—์„œ ์‚ฌ์ „ํ•™์Šต๋œ tactile expert๋ฅผ ๊ทธ๋Œ€๋กœ ์žฌ์‚ฌ์šฉํ•  ์ˆ˜ ์žˆ๋‹ค. โ‘ก ์‚ฌ์ „ํ•™์Šต๋œ vision-language ์ง€์‹์„ ์ด‰๊ฐ ํ† ํฐ์ด ํ”๋“ค์ง€ ์•Š๋Š”๋‹ค. โ‘ข ์ด‰๊ฐ ์ฒ˜๋ฆฌ ํšจ์œจ๊ณผ ์„ฑ๋Šฅ์ด adapter ๋ฐฉ์‹(Tactile-VLA)๋ณด๋‹ค ๋‚ซ๋‹ค. โ‘ข์€ Table 1~3์˜ Tactile-VLA ๋น„๊ต๋กœ ๋’ท๋ฐ›์นจ๋˜๊ณ , โ‘ ์€ ยง4์˜ unseen sensor ์‹คํ—˜์ด ๊ทธ ์ž์ฒด๋‹ค. โ‘ก๋Š” ์ง์ ‘ ์ธก์ •ํ•œ ์‹คํ—˜์ด ์—†๋‹ค. MoE ๊ธฐ๋ฐ˜ ์œตํ•ฉ๋„ ์‹œ๋„ํ–ˆ์ง€๋งŒ ์ผ๊ด€๋œ ์ด๋“์ด ์—†์–ด ๊ฐ€์žฅ ๋‹จ์ˆœํ•œ multi-expert๋ฅผ ํƒํ–ˆ๋‹ค๊ณ  ์“ด๋‹ค(์ˆ˜์น˜ ์—†์Œ).

๋ฐฉ๋ฒ• ์ƒ์„ธ 2: Morphology-Aware Tactile Token Space (ยง2.1, Fig. 3)


Fig. 3. MTTS์˜ ๊ธฐ๋Šฅ ์˜์—ญ ์ •์˜. ์† ์˜์—ญ ์ด‰๊ฐ ์นธ(0~14), ์†๋ชฉ ํ† ํฌ(15), ์†๊ฐ€๋ฝ ๊ด€์ ˆ ํ† ํฌ(16~20), ๋ฏธ์‚ฌ์šฉ(21~23). ํ‰ํ–‰ ๊ทธ๋ฆฌํผ์˜ ๋‘ ํŒจ๋“œ๋Š” ์—„์ง€ ๋(0)๊ณผ ๊ฒ€์ง€ ๋(1) ์นธ์„ ๊ณต์œ ํ•œ๋‹ค.

MTTS๋Š” ์ด‰๊ฐ์„ 24๊ฐœ ๊ธฐ๋Šฅ ์˜์—ญ ์Šฌ๋กฏ์œผ๋กœ ์ •๋ฆฌํ•œ๋‹ค. ๋ณธ๋ฌธ ์ •์˜๋Š” ์ด๋ ‡๋‹ค.

์Šฌ๋กฏ ์˜๋ฏธ ๋น„๊ณ 
0~14 ์†์˜ ์ด‰๊ฐ ์˜์—ญ(์†๋ยท์†๊ฐ€๋ฝ ๋งˆ๋””ยท์†๋ฐ”๋‹ฅ ๋“ฑ) ๊ทธ๋ฆฌํผ ๋‘ ํŒจ๋“œ๋Š” 0(์—„์ง€ ๋)ยท1(๊ฒ€์ง€ ๋)๋กœ ๋งคํ•‘
15~20 ์†๋ชฉยท์†๊ฐ€๋ฝ์˜ ํž˜/ํ† ํฌ ์‹ ํ˜ธ Fig. 3์€ 15๋ฅผ wrist torque, 16~20์„ finger joint torque๋กœ ํ‘œ๊ธฐ
21~23 ์˜ˆ์•ฝ(๋ฏธ์‚ฌ์šฉ)

์Šฌ๋กฏ๋งˆ๋‹ค ํ† ํฐ์€ ์ •ํ™•ํžˆ ํ•˜๋‚˜์ด๊ณ , ์Šฌ๋กฏ ๋ฒˆํ˜ธ์— ํ•ด๋‹นํ•˜๋Š” learnable functional-area embedding์ด ๋ชจ๋“  ์„ผ์„œ์— ๊ณต์œ ๋œ๋‹ค. ์™ผ์†๊ณผ ์˜ค๋ฅธ์†์€ ๋ณ„๋„ embedding์„ ์“ด๋‹ค. ๊ณต๊ฐœ ์ฒดํฌํฌ์ธํŠธ config์—๋„ single_hand_num_tactile_areas: 24๊ฐ€ ์ ํ˜€ ์žˆ๋‹ค. ๊ทธ๋Ÿฌ๋‹ˆ ํ•œ ์‹œ์ ์˜ ์ด‰๊ฐ ์ž…๋ ฅ์€ ์ตœ๋Œ€ 48๊ฐœ ํ† ํฐ(์–‘์† ร— 24, ํ•„์ž ๊ณ„์‚ฐ)์ด๊ณ , ์„ผ์„œ๊ฐ€ ๋ฎ์ง€ ์•Š๋Š” ์นธ์€ padding mask๋กœ ๋น ์ง„๋‹ค.

์ด ์„ค๊ณ„๊ฐ€ ๋…ธ๋ฆฌ๋Š” ๊ฒƒ์€ ์„ผ์„œ ๊ฐ„ ์˜๋ฏธ์˜ ์ •๋ ฌ์„ ๋ฐ์ดํ„ฐ๊ฐ€ ์•„๋‹ˆ๋ผ ์ธํ„ฐํŽ˜์ด์Šค์—์„œ ์ฃผ๋Š” ๊ฒƒ์ด๋‹ค. GelSight๋ฅผ ๋‹จ ๊ทธ๋ฆฌํผ์˜ ์™ผ์ชฝ ํŒจ๋“œ์™€, ์‚ฌ๋žŒ ์žฅ๊ฐ‘์˜ ์—„์ง€ ๋ ์••๋ ฅ ๋ฐฐ์—ด๊ณผ, Sharpa ์†์˜ ์—„์ง€ ๋ ๊ด‘ํ•™ ์„ผ์„œ๊ฐ€ ๋ชจ๋‘ ๊ฐ™์€ slot 0 embedding์„ ๋ฐ›๋Š”๋‹ค. tactile expert ์ž…์žฅ์—์„œ๋Š” โ€œ์—„์ง€ ์ชฝ์— ์ด๋Ÿฐ ์ ‘์ด‰์ด ์™”๋‹คโ€๋Š” ๊ฐ™์€ ์œ„์น˜ ์ •๋ณด ์œ„์—์„œ ์„ผ์„œ๋ณ„ ๋‚ด์šฉ์ด ๋‹ฌ๋ผ์งˆ ๋ฟ์ด๋‹ค. ๊ทธ๋ฆฌํผ๋ฅผ ์—„์ง€ยท๊ฒ€์ง€๋กœ ๋ณด๋Š” ๋งคํ•‘์€ ๋‘ ์†๊ฐ€๋ฝ ์ง‘๊ธฐ๋ผ๋Š” ๊ธฐ๋Šฅ์  ์œ ๋น„์—์„œ ๋‚˜์˜จ ์„ ํƒ์ด๋‹ค.

์ •์˜์ƒ ๋ช‡ ๊ฐ€์ง€๊ฐ€ ๋‚จ๋Š”๋‹ค. ์ฒซ์งธ, ํ•œ ์Šฌ๋กฏ์— ํ•œ ํ† ํฐ์ด๋ฏ€๋กœ ์†๋ ํ•˜๋‚˜๋ฅผ ๋ฎ๋Š” ๊ณ ํ•ด์ƒ๋„ ๊ด‘ํ•™ ์„ผ์„œ๋„ [CLS] ํ† ํฐ ํ•˜๋‚˜๋กœ ์••์ถ•๋œ๋‹ค. ์ ‘์ด‰ ํŒจํ„ด์˜ ๊ณต๊ฐ„ ์ •๋ณด๋Š” encoder ์•ˆ์—์„œ๋งŒ ์‚ด์•„๋‚จ๋Š”๋‹ค. ๋‘˜์งธ, Fig. 3์˜ ๋ฒ”๋ก€๋Š” โ€œ1-14: Tactileโ€์ด๋ผ ์ ํ˜€ ์žˆ๊ณ  ๋ณธ๋ฌธ์€ โ€œslots 0-14โ€๋ผ ์ ๋Š”๋‹ค. ๊ทธ๋ฆผ์— 0๋ฒˆ ์นธ์ด ๊ทธ๋ ค์ ธ ์žˆ์œผ๋ฏ€๋กœ ๋ฒ”๋ก€ ์ชฝ์˜ ํ‘œ๊ธฐ ์ฐจ์ด๋กœ ๋ณด์ธ๋‹ค. ์…‹์งธ, 15~20์€ ์—„๋ฐ€ํžˆ ์ด‰๊ฐ์ด ์•„๋‹ˆ๋ผ ํž˜/ํ† ํฌ๋‹ค. ์ด ์ ์€ โ€œ21๊ฐœ ์„ผ์„œโ€๋ฅผ ์…€ ๋•Œ ๋‹ค์‹œ ์ค‘์š”ํ•ด์ง„๋‹ค(๋ฐ์ดํ„ฐ ์ ˆ).

๋ฐฉ๋ฒ• ์ƒ์„ธ 3: ์ด์งˆ์  ์ด‰๊ฐ encoder (ยง2.2, ๋ถ€๋ก B.2)

๊ฐ ์„ผ์„œ์˜ ์‹ ํ˜ธ๋ฅผ ๋จผ์ € MTTS ์˜์—ญ๋ณ„๋กœ ๋ฌถ๊ณ , ๊ฐ ๋ฌถ์Œ์„ image / array / state ์…‹ ์ค‘ ํ•˜๋‚˜๋กœ ๋ถ„๋ฅ˜ํ•ด encoder๋ฅผ ๊ณ ๋ฅธ๋‹ค.

์œ ํ˜• ์ž…๋ ฅ ๋ชจ์–‘ encoder ์ถœ๋ ฅ
image (์˜ˆ: GelSight) (H,W,C) โ†’ 224ร—224 ์„ผ์„œ๋ณ„ Transformer(depth 3, width 768, 12 heads) โ†’ ๊ณต์œ  Transformer trunk(depth 9, width 768, 12 heads, T3๋กœ ์ดˆ๊ธฐํ™”) ๋งˆ์ง€๋ง‰ [CLS] ํ† ํฐ
array (์˜ˆ: Contactile) (H,W,D) ์‹ ํ˜ธ ์ฐจ์›์— Fourier encoding ํ›„ ์› ์ž…๋ ฅ๊ณผ concat โ†’ 3์ธต CNN โ†’ 2์ธต ReLU MLP ์˜์—ญ๋‹น ํ† ํฐ 1๊ฐœ
state (์˜ˆ: F/T) (D,) Fourier encoding ํ›„ ์› ์ƒํƒœ์™€ concat โ†’ 3์ธต ReLU MLP ํ† ํฐ 1๊ฐœ

๋…ผ๋ฌธ์— ๋”ฐ๋ฅด๋ฉด ๋ชจ๋“  ํ† ํฐ์€ LayerNorm โ†’ functional-area embedding ๋ง์…ˆ โ†’ 2์ธต GELU MLP๋กœ tactile expert ์ž…๋ ฅ ํญ์— ๋งž์ถ˜๋‹ค(๊ณต๊ฐœ ์ฝ”๋“œ์˜ ์ˆœ์„œ๋Š” ์กฐ๊ธˆ ๋‹ค๋ฅด๋‹ค. ์•„๋ž˜ ๋Œ€์กฐ ์ ˆ). ํ•œ ์„ผ์„œ๊ฐ€ ๊ฐ™์€ ๋ชจ์–‘์˜ ์˜์—ญ์„ ์—ฌ๋Ÿฌ ๊ฐœ ๊ฐ€์ง€๋ฉด encoder๋ฅผ ์˜์—ญ ๊ฐ„์— ๊ณต์œ ํ•œ๋‹ค. encoder๋Š” ๋ฐ์ดํ„ฐ ์ถœ์ฒ˜๊ฐ€ ์•„๋‹ˆ๋ผ ์„ผ์„œ ๋‹จ์œ„๋กœ ์กฐ์ง๋œ๋‹ค. ๋‹ค๋ฅธ ๋ฐ์ดํ„ฐ์…‹์ด๋ผ๋„ ๊ฐ™์€ GelSight-Mini๋ฉด ๊ฐ™์€ encoder๋ฅผ ์“ด๋‹ค(๋ถ€๋ก C).

์ด ํ‘œ์—์„œ ์ด๋ฏธ์ง€ ๊ฒฝ๋กœ๋งŒ ํŠน๋ณ„ํ•˜๋‹ค. image ๊ณ„์—ด ์„ผ์„œ๋Š” ์„ผ์„œ๋ณ„ 3์ธต ์œ„์— ์„ผ์„œ ๊ณตํ†ต์˜ 9์ธต trunk๋ฅผ ๊ณต์œ ํ•˜๊ณ  ๊ทธ trunk๊ฐ€ T3 ์‚ฌ์ „ํ•™์Šต ๊ฐ€์ค‘์น˜์—์„œ ์ถœ๋ฐœํ•œ๋‹ค. ์ฆ‰ ์ด๋ฏธ์ง€ํ˜• ์ด‰๊ฐ์—๋Š” ์‚ฌ์ „ํ•™์Šต๋œ ๊ณต์œ  ์ง€๊ฐ ๋ชจ๋“ˆ์ด ๋‘ ๊ฒน(T3 trunk, tactile expert) ์žˆ๊ณ , arrayยทstateํ˜•์—๋Š” tactile expert ํ•œ ๊ฒน๋ฟ์ด๋‹ค. ๊ณต๊ฐœ ์ฒดํฌํฌ์ธํŠธ์˜ hpt_tokenizer/ ํด๋”์—๋„ ์„ผ์„œ๋ณ„ encoder ํŒŒ์ผ๊ณผ shared_image_chunk_encoder๊ฐ€ ๋”ฐ๋กœ ์ €์žฅ๋ผ ์žˆ๊ณ , ๊ณต์œ  trunk๋Š” ์•ฝ 63.8M ํŒŒ๋ผ๋ฏธํ„ฐ๋‹ค(์ฒดํฌํฌ์ธํŠธ ํ—ค๋”์—์„œ ๊ณ„์‚ฐ, 9์ธต ViT-B ๊ทœ๋ชจ์™€ ๋งž๋Š”๋‹ค).

๋ฐฉ๋ฒ• ์ƒ์„ธ 4: Unified Action Space์™€ FAAS (๋ถ€๋ก B.1)

์—ฌ๋Ÿฌ ๋กœ๋ด‡์˜ action์„ ํ•œ policy๋กœ ํ•™์Šตํ•˜๋ ค๋ฉด action ๊ณต๊ฐ„๋„ ๋งž์ถฐ์•ผ ํ•œ๋‹ค. FTP-1์€ ํ•œ ์Šคํ…์˜ action์„

\mathbf a=[\mathbf a^{L},\mathbf a^{R},\mathbf a^{ego},\mathbf a^{sup}],\qquad \mathbf a^{b}=[\mathbf t_w^{b}\in\mathbb R^3,\ \mathbf r_w^{b}\in\mathbb R^6,\ \mathbf q_{\text{arm}}^{b}\in\mathbb R^7,\ \mathbf q_{\text{hand}}^{b}\in\mathbb R^{32}]

๋กœ ๋‘”๋‹ค. ์†๋ชฉ ์œ„์น˜ยท6D ํšŒ์ „ยทํŒ” ๊ด€์ ˆ 7๊ฐœยท์† ๊ด€์ ˆ 32์นธ์ด ํŒ”๋งˆ๋‹ค ์žˆ๊ณ , \mathbf a^{ego}๋Š” ๋จธ๋ฆฌ ์ž์„ธ(์œ„์น˜ 3 + ํšŒ์ „ 6), \mathbf a^{sup}๋Š” ์ด๋™ยทํ—ˆ๋ฆฌ ๊ฐ™์€ ๋ณด์กฐ ์นธ์ด๋‹ค. ์† 32์นธ์€ UniDex ๋ฆฌ๋ทฐ์—์„œ ์ •๋ฆฌํ•œ FAAS(Functionโ€“Actuatorโ€“Aligned Space)๋ฅผ ๊ทธ๋Œ€๋กœ ๊ฐ€์ ธ์˜จ๋‹ค. ๊ธฐ๋Šฅ์ด ๋น„์Šทํ•œ ๊ด€์ ˆ์„ ๊ฐ™์€ ์นธ์— ๋„ฃ๋Š” ๋ฐฉ์‹์ด๋‹ค. ํ‰ํ–‰ ๊ทธ๋ฆฌํผ๋Š” ๋…๋ฆฝ ์นธ 28์„ ์“ด๋‹ค. ์ง€์›๋˜์ง€ ์•Š๋Š” ์นธ์€ mask \mathbf M\in\{0,1\}^D๋กœ loss์—์„œ ๋บ€๋‹ค. ์† action ๊ณต๊ฐ„ ํ†ต์ผ์ด๋ผ๋Š” ๊ฐ™์€ ๋ฌธ์ œ๋ฅผ ๋‹ค๋ฃฌ ๋‹ค๋ฅธ ์ ‘๊ทผ์€ UHAS ๋ฆฌ๋ทฐ์— ์žˆ๋‹ค.


Fig. 8. FAAS๊ฐ€ ์ •์˜๋œ ๋กœ๋ด‡ ์†๋“ค(์™ผ์ชฝ๋ถ€ํ„ฐ Ability, Allegro, Inspire, Leap, Oymotion, Shadow, Wuji, Xhand)์˜ ๊ด€์ ˆ-์นธ ๋งคํ•‘. UniDex ๋…ผ๋ฌธ์˜ ๊ทธ๋ฆผ์„ ์˜ฎ๊ธด ๊ฒƒ์ด๋‹ค.


Fig. 9. ์‚ฌ๋žŒ ์†์˜ FAAS ์ •์˜.


Fig. 10. Sharpa Wave ์˜ค๋ฅธ์†์˜ FAAS ์นธ ๋ฒˆํ˜ธ. (a) ๊ตฝํž˜/ํ„, (b) ๋ฒŒ๋ฆผ/๋ชจ์Œ๊ณผ ์ƒˆ๋ผ์†๊ฐ€๋ฝ CMC.

ํ‰๊ฐ€ ๋กœ๋ด‡ ์ค‘ ๋‘ ์…‹์—…์ด ์“ฐ๋Š” Sharpa Wave ์†์€ ์†๋‹น 22๊ฐœ ๊ตฌ๋™ ๊ด€์ ˆ์„ ๊ฐ€์ง€๋ฉฐ, native ์ˆœ์„œ๋Œ€๋กœ FAAS ์นธ \mathbf s=[1,2,3,26,4,\ 7,6,8,9,\ 12,11,13,14,\ 17,16,18,19,\ 25,22,21,23,24]์— ๋„ฃ๋Š”๋‹ค(Table 7). ๋‚จ๋Š” 10์นธ \{0,5,10,15,20,27,\dots,31\}์€ 0์œผ๋กœ ์ฑ„์šฐ๊ณ  loss์—์„œ ๋บ€๋‹ค. 22 + 10 = 32๋กœ ๋งž์•„๋–จ์–ด์ง„๋‹ค. ๊ฐ™์€ ๋งคํ•‘์ด proprioception ์ž…๋ ฅ๊ณผ action target์— ์“ฐ์ธ๋‹ค.

๋…ผ๋ฌธ์€ H์™€ D์˜ ๊ฐ’์„ ์ ์ง€ ์•Š๋Š”๋‹ค. ๊ณต๊ฐœ ์ฒดํฌํฌ์ธํŠธ์˜ model_config.json์—๋Š” action_dim: 120, action_horizon: 32๊ฐ€ ์žˆ๋‹ค. ์œ„ ์ •์˜๋Œ€๋กœ ์„ธ๋ฉด ์–‘ํŒ” 2\times(3+6+7+32)=96, ๋จธ๋ฆฌ 9๋ฅผ ๋”ํ•ด 105์ด๋ฏ€๋กœ \mathbf a^{sup}๋Š” 15์นธ์ด๋ผ๋Š” ๊ณ„์‚ฐ์ด ๋œ๋‹ค(ํ•„์ž ๊ณ„์‚ฐ, ๋…ผ๋ฌธ์€ \mathbf a^{sup} ํฌ๊ธฐ๋ฅผ ๋ฐํžˆ์ง€ ์•Š๋Š”๋‹ค).

๋ฐฉ๋ฒ• ์ƒ์„ธ 5: ์‚ฌ์ „ํ•™์Šต ๋ฐ์ดํ„ฐ (ยง2.4, ๋ถ€๋ก C, Fig. 4)


Fig. 4. FTP-1-Dataset ๊ฐœ์š”. 26๊ฐœ ์ถœ์ฒ˜, ์•ฝ 3,000์‹œ๊ฐ„, ์žฌ์ƒ˜ํ”Œ๋ง ํ›„ ์‚ฌ๋žŒ 20% ยท ๋‹ค์ง€์† 30% ยท ๊ทธ๋ฆฌํผ 50%, 21๊ฐœ ์„ผ์„œ(state 9 ยท image 7 ยท array 5).

๊ธฐ์กด ๊ณต๊ฐœ ๋ฐ์ดํ„ฐ์™€ ํ˜‘๋ ฅ์‚ฌ ์ œ๊ณต ๋ฐ์ดํ„ฐ๋ฅผ ๋ชจ์œผ๊ณ , ์ €์ž๋“ค์ด Sharpa North ํ”Œ๋žซํผ์œผ๋กœ ์ง์ ‘ ์ˆ˜์ง‘ํ•œ Sharpa North-FTP-1(๊ธด horizon ๋‹ค์ง€์† ์‹œ์—ฐ ์•ฝ 4,000๊ฐœ)์„ ๋”ํ–ˆ๋‹ค. ์ถœ์ฒ˜๋ณ„ ๊ทœ๋ชจ ์ฐจ์ด๋ฅผ ๋งž์ถ”๋ ค๊ณ  ์žฌ์ƒ˜ํ”Œ๋ง ๋น„์œจ์„ ๋‘”๋‹ค. ๋ถ€๋ก Table 8์„ ์˜์—ญ๋ณ„๋กœ ์ •๋ฆฌํ•˜๋ฉด ๋‹ค์Œ๊ณผ ๊ฐ™๋‹ค(Final-Percentage๋Š” ์žฌ์ƒ˜ํ”Œ๋ง ํ›„ ํ•™์Šต ํ˜ผํ•ฉ์—์„œ์˜ ๋น„์ค‘).

์˜์—ญ ์ถœ์ฒ˜ (์„ผ์„œ ์œ ํ˜•) ์žฌ์ƒ˜ํ”Œ๋ง ํ›„ ๋น„์ค‘
์‚ฌ๋žŒ ์† (20%) EgoTac-Open 8.2, EgoTac-Kitchen 5.5, EgoTac-Hotel 3.8 (AetherGlove, array) ยท OpenTouch 1.5 (array) ยท OmniSharingDB 1.0 (PaxiniGlove, state) 20.0%
๋‹ค์ง€์† (30%) Sharpa North-FTP-1 9.0 (Sharpa DTC, image) ยท HumanoidEveryday 6.0 (Unitree Dex3, state) ยท OpenLETDex 6.0 (LinkerHand L6 Touch, array) ยท MotionTrans 3.6 (Inspire, state) ยท DexUMI-Inspire 3.0 ยท DexUMI-XHand 2.4 (state) 30.0%
๊ทธ๋ฆฌํผ/UMI (50%) VTouch-D-WHEEL 5.7 ยท FreeTacMan 4.4 ยท TouchInTheWild 4.0 (3D-ViTac, array) ยท VTouch-QINGLONG 3.9 ยท exUMI 3.8 ยท RH20T-Cfg6 3.8 (ATI F/T, state) ยท ViTaMIn 3.4 ยท RH20T-Cfg7 3.4 (uSkin + ATI) ยท RH20T-Cfg5 3.3 (Franka ๊ด€์ ˆ ํ† ํฌ) ยท REASSEMBLE 3.2 (F/T) ยท Unit-Bimanual 2.6 ยท RDP 2.4 ยท Unit 2.2 ยท VLA-Touch 2.0 ยท RDP-Bimanual 1.9 50.0%

์˜์—ญ๋ณ„ ๋น„์ค‘์˜ ํ•ฉ์€ ๊ฐ๊ฐ 20.0 / 30.0 / 50.0%๋กœ ํ‘œ์™€ ๋งž๋Š”๋‹ค(ํ•„์ž ๊ฒ€์‚ฐ). ์–ธ์–ด ์ง€์‹œ๋Š” GPT-4o๋กœ ๋‹ค์–‘ํ™”ํ–ˆ๊ณ , ์ •๊ทœํ™” ํ†ต๊ณ„๋Š” ๋ฐ์ดํ„ฐ์…‹๋งˆ๋‹ค ๋”ฐ๋กœ ๊ณ„์‚ฐํ•œ๋‹ค.

์„ธ ๊ฐ€์ง€๋ฅผ ์งš์–ด ๋‘”๋‹ค.

  • โ€œ21๊ฐœ ์ด‰๊ฐ ์„ผ์„œโ€์˜ ๊ตฌ์„ฑ: image 7(Sharpa DTC, ViTai-GF, FreeTacMan, exUMI, ViTaMIn, GelSight-Mini, MCTac), array 5(AetherGlove, OpenTouchGlove, LinkerHand L6 Touch, 3D-ViTac, uSkin), state 9(PaxiniGlove, Unitree Dex3, Inspire, DexUMI-Inspire, DexUMI-XHand, ATI Axia80-M20, Franka ๊ด€์ ˆ ํ† ํฌ, AIDIN F/T, Flexiv ๊ทธ๋ฆฌํผ ํ† ํฌ). state 9๊ฐœ ์ค‘์—๋Š” ์†๋ชฉ F/T ์„ผ์„œ์™€ ๋กœ๋ด‡ ๊ด€์ ˆ ํ† ํฌยท๊ทธ๋ฆฌํผ ํ† ํฌ๊ฐ€ ๋“ค์–ด ์žˆ๋‹ค. ์ด‰๊ฐ ํ”ผ๋ถ€๋ผ๋Š” ์ข์€ ์˜๋ฏธ๋กœ ์„ธ๋ฉด ์ˆซ์ž๊ฐ€ ์ค„์–ด๋“ ๋‹ค. ๋…ผ๋ฌธ์€ ์ด๋ฅผ โ€œtactile / force sensorโ€๋กœ ๋ถ€๋ฅด๊ณ  MTTS์—์„œ ๋ณ„๋„ ์นธ(15~20)์„ ์ฃผ๋ฏ€๋กœ ์ •์˜ ์œ„๋ฐ˜์€ ์•„๋‹ˆ์ง€๋งŒ, ํ—ค๋“œ๋ผ์ธ์˜ โ€œ21 sensorsโ€๋ฅผ ์ฝ์„ ๋•Œ ์•Œ์•„ ๋‘˜ ๊ตฌ์„ฑ์ด๋‹ค.
  • โ€œ์•ฝ 3,000์‹œ๊ฐ„โ€์˜ ๋ถ„ํ•ด๊ฐ€ ์—†๋‹ค: Table 8์—๋Š” Sample-Scale๊ณผ Final-Percentage๋งŒ ์žˆ๊ณ  ์ถœ์ฒ˜๋ณ„ ์‹œ๊ฐ„ยท์—ํ”ผ์†Œ๋“œ ์ˆ˜๊ฐ€ ์—†๋‹ค. ๋…ผ๋ฌธ์€ Sample-Scale์˜ ๋œป๋„ ์ •์˜ํ•˜์ง€ ์•Š๋Š”๋‹ค. ๊ณต๊ฐœ ์ฝ”๋“œ์™€ ์ฒดํฌํฌ์ธํŠธ์˜ ๋ฐ์ดํ„ฐ ์„ค์ •์„ ๋ณด๋ฉด ์ด๊ฒƒ์€ ์ถœ์ฒ˜๋ณ„ ๋ณต์ œ ๋ฐฐ์ˆ˜๋‹ค. ๊ฐ ์ถœ์ฒ˜๋ฅผ Sample-Scale์˜ ์˜ฌ๋ฆผ ํšŸ์ˆ˜๋งŒํผ ๋ณต์ œํ•ด ๋„ฃ๋Š”๋‹ค(RDP 89.76 โ†’ 90๊ฐœ, VLA-Touch 170.36 โ†’ 171๊ฐœ). ๊ทธ๋ ‡๋‹ค๋ฉด ์žฌ์ƒ˜ํ”Œ๋ง ์ „์˜ ์›๋ž˜ ๋น„์ค‘์€ Final-Percentage รท Sample-Scale์— ๋น„๋ก€ํ•œ๋‹ค. ์ด๋ ‡๊ฒŒ ์—ญ์‚ฐํ•˜๋ฉด ์žฌ์ƒ˜ํ”Œ๋ง ์ „ ์ƒ˜ํ”Œ์˜ ์•ฝ 63%๊ฐ€ ์‚ฌ๋žŒ ์† ๋ฐ์ดํ„ฐ์ด๊ณ , EgoTac ์„ธ ์ถœ์ฒ˜๋งŒ ์•ฝ 62%, Sharpa North-FTP-1์ด ์•ฝ 12%๋‹ค(ํ•„์ž ์ถ”์ •, ๋ณต์ œ ๋ฐฐ์ˆ˜ ํ•ด์„๊ณผ ํ‘œ์˜ ๋ฐ˜์˜ฌ๋ฆผ์— ์˜์กด). ์‚ฌ๋žŒ ์žฅ๊ฐ‘ ๋ฐ์ดํ„ฐ๊ฐ€ ์›๋ž˜ ๊ทœ๋ชจ์˜ ๋Œ€๋ถ€๋ถ„์ด๊ณ , ์žฌ์ƒ˜ํ”Œ๋ง์ด ๊ทธ๊ฒƒ์„ 20%๋กœ ํฌ๊ฒŒ ๋ˆŒ๋ €๋‹ค๋Š” ๋œป์ด ๋œ๋‹ค. 3,000์‹œ๊ฐ„์„ ์ถœ์ฒ˜๋ณ„ ์‹œ๊ฐ„์œผ๋กœ ๊ฒ€์‚ฐํ•  ๋ฐฉ๋ฒ•์€ ์—ฌ์ „ํžˆ ์—†๋‹ค.
  • ํ‰๊ฐ€ ํ”Œ๋žซํผ๊ณผ ๊ฐ™์€ ๋กœ๋ด‡์˜ ๋ฐ์ดํ„ฐ๊ฐ€ ์‚ฌ์ „ํ•™์Šต์— ์žˆ๋‹ค: Sharpa North-FTP-1์€ ํ‰๊ฐ€ ์…‹์—… Sharpa North์™€ ๊ฐ™์€ ํ•˜๋“œ์›จ์–ด์ด๊ณ  ํ˜ผํ•ฉ์˜ 9.0%๋กœ ๋‹จ์ผ ์ถœ์ฒ˜ ์ค‘ ๊ฐ€์žฅ ํฌ๋‹ค. ํ‰๊ฐ€ ๊ณผ์ œ(Draw Balloon, Fix Hand, Twist Cap)๊ฐ€ ์ด 4,000๊ฐœ ์‹œ์—ฐ์— ํฌํ•จ๋˜๋Š”์ง€๋Š” ๋…ผ๋ฌธ์— ์ ํ˜€ ์žˆ์ง€ ์•Š๋‹ค. ์ด ์ ์€ ์‹คํ—˜ ์ ˆ์˜ ๋น„๊ต ํ•ด์„์— ๋‹ค์‹œ ์“ฐ์ธ๋‹ค.

ํ•™์Šต์€ \pi_{0.5} ์ฝ”๋“œ๋ฒ ์ด์Šค ์œ„์—์„œ ํ•œ๋‹ค(๋ถ€๋ก D). ์‹œ๊ฐ encoderยท์–ธ์–ด tokenizerยทVL expertยทaction expert๋Š” \pi_{0.5}, ์ด๋ฏธ์ง€ ๊ณต์œ  trunk๋Š” T3์—์„œ ์ดˆ๊ธฐํ™”ํ•˜๊ณ , ์„ผ์„œ๋ณ„ encoderยทtactile expertยทfunctional-area embeddingยทAdaRMS injectorยทaction projector๋Š” scratch๋‹ค. ๊ณต๊ฐœ ์ฝ”๋“œ๋กœ ๋ณด๋ฉด ์ด ๋ชฉ๋ก์€ ์„ค๊ณ„ ์„ ํƒ์ด๋ผ๊ธฐ๋ณด๋‹ค ๋กœ๋”ฉ ๊ทœ์น™์˜ ๊ฒฐ๊ณผ๋‹ค. \pi_{0.5} ๊ฐ€์ค‘์น˜๋Š” ์ด๋ฆ„๊ณผ shape๊ฐ€ ๋ชจ๋‘ ๊ฐ™์€ ํ…์„œ๋งŒ ๋ณต์‚ฌํ•˜๋ฏ€๋กœ, UAS ๋•Œ๋ฌธ์— ์ฐจ์›์ด ๋ฐ”๋€ action ์ž…์ถœ๋ ฅ projection๊ณผ ํญ์ด ๋ฐ”๋€ AdaRMS ์กฐ๊ฑด ๊ฒฝ๋กœ๋Š” ์ž๋™์œผ๋กœ ๋น ์ง„๋‹ค. H20 GPU 48์žฅ์—์„œ 50k step, global batch 768, ํ•™์Šต๋ฅ  10^{-4}\to5\times10^{-5}. 50k step ์ดํ›„ ์„ฑ๋Šฅ์ด ํฌํ™”ํ–ˆ๋‹ค๊ณ  ์ ๋Š”๋‹ค. 50k ร— 768 = 3,840๋งŒ ์ƒ˜ํ”Œ์ด ํ•™์Šต์—์„œ ๋ณด์ธ ์ด ์ƒ˜ํ”Œ ์ˆ˜๋‹ค(ํ•„์ž ๊ณ„์‚ฐ). ๋ฐ์ดํ„ฐ์˜ ์ œ์–ด ์ฃผํŒŒ์ˆ˜๊ฐ€ ์ ํ˜€ ์žˆ์ง€ ์•Š์•„ ์ด๊ฒƒ์ด 3,000์‹œ๊ฐ„์˜ ๋ช‡ epoch์ธ์ง€๋Š” ๊ณ„์‚ฐํ•  ์ˆ˜ ์—†๋‹ค. ๋„๋ฉ”์ธ๋ณ„๋กœ GPU๋ฅผ ๋‚˜๋ˆ  ๋ฐฐ์น˜ ์•ˆ ํฌ๋งท์„ ๋งž์ถ”๊ณ , ๋„๋ฉ”์ธ ์ „์šฉ ๋ชจ๋“ˆ๊ณผ ๊ณต์œ  ๋ชจ๋“ˆ์˜ gradient๋ฅผ ๋”ฐ๋กœ ๋‹ค๋ฃจ๋Š” ํ•™์Šต ์ธํ”„๋ผ๋„ ์ƒˆ๋กœ ๋งŒ๋“ค์—ˆ๋‹ค.

๋ถ€๋ก B.5์—๋Š” ์‹ค๋ฌด ์„ ํƒ ๋‘ ๊ฐ€์ง€๊ฐ€ ๋” ์žˆ๋‹ค. ์ƒํƒœยทaction ์ •๊ทœํ™”๋Š” quantile๋ณด๋‹ค z-score๊ฐ€ ์ ‘์ด‰ ๊ณผ์ œ์˜ ๋ฏธ์„ธํ•œ action์—์„œ ๋‚˜์•˜๊ณ , Muon optimizer๋Š” ์ˆ˜๋ ด๊ณผ offline action MSE๋Š” ํฌ๊ฒŒ ๊ฐœ์„ ํ–ˆ์ง€๋งŒ ์‹ค๊ธฐ rollout์˜ ์ผ๋ฐ˜ํ™”ยท๊ฐ•๊ฑด์„ฑ์„ ๋–จ์–ด๋œจ๋ ค ์ตœ์ข… ๋ชจ๋ธ์€ AdamW๋ฅผ ์ผ๋‹ค. offline MSE๊ฐ€ ์‹ค๊ธฐ ์„ฑ๋Šฅ์˜ ๋Œ€๋ฆฌ ์ง€ํ‘œ๊ฐ€ ๋˜์ง€ ๋ชปํ•œ๋‹ค๋Š” ์‚ฌ๋ก€๋กœ ์ฝ์„ ๋งŒํ•˜๋‹ค(์ˆ˜์น˜๋Š” ์—†๋‹ค).

์ง๊ด€: ์™œ โ€œ์–ด๋””์— ๋‹ฟ์•˜๋‚˜โ€๊ฐ€ ๊ณตํ†ต ์–ธ์–ด๊ฐ€ ๋˜๋‚˜

์„ผ์„œ๋งˆ๋‹ค ์‹ ํ˜ธ์˜ ๋‹จ์œ„์™€ ๋ชจ์–‘์€ ๋‹ค๋ฅด์ง€๋งŒ, policy๊ฐ€ ์ด‰๊ฐ์—์„œ ์›ํ•˜๋Š” ์ •๋ณด๋Š” ๋Œ€๊ฐœ ๊ฐ™๋‹ค. ์–ด๋А ์†๊ฐ€๋ฝ์ด ์–ธ์ œ ๋‹ฟ์•˜๊ณ , ์–ผ๋งˆ๋‚˜ ์„ธ๊ฒŒ ๋ˆŒ๋ฆฌ๊ณ , ๋ฏธ๋„๋Ÿฌ์ง€๋Š”๊ฐ€. ์ด ์ค‘ โ€œ์–ด๋””โ€๋Š” ์„ผ์„œ์™€ ๋ฌด๊ด€ํ•œ ์ •๋ณด์ด๊ณ , ์‚ฌ๋žŒ ์† ํ•ด๋ถ€ํ•™์œผ๋กœ ๋ฏธ๋ฆฌ ์ •์˜ํ•  ์ˆ˜ ์žˆ๋‹ค. FTP-1์€ ๊ทธ โ€œ์–ด๋””โ€๋ฅผ embedding์œผ๋กœ ๊ณ ์ •ํ•ด ๋‘๊ณ  โ€œ์–ผ๋งˆ๋‚˜ยท์–ด๋–ป๊ฒŒโ€๋งŒ ์„ผ์„œ๋ณ„ encoder์— ๋งก๊ธด๋‹ค. ๊ทธ๋ž˜์„œ ์ƒˆ ์„ผ์„œ๊ฐ€ ์˜ค๋ฉด โ€œ์–ผ๋งˆ๋‚˜ยท์–ด๋–ป๊ฒŒโ€๋ฅผ ๋ฒˆ์—ญํ•˜๋Š” ์ž‘์€ ๋ชจ๋“ˆ๋งŒ ์ƒˆ๋กœ ๋ฐฐ์šฐ๊ณ , โ€œ์—„์ง€ ๋ ์ ‘์ด‰์ด ๊ฐ•ํ•ด์ง€๋ฉด ๊ฐ์†ํ•œ๋‹คโ€ ๊ฐ™์€ ์ƒ์œ„ ๊ทœ์น™์€ tactile expert์™€ action expert์— ๋‚จ์•„ ์žˆ๋Š” ๊ฒƒ์„ ์“ฐ๊ฒŒ ๋œ๋‹ค.

์ด ์ง๊ด€์ด ์„ฑ๋ฆฝํ•˜๋ ค๋ฉด ์ƒˆ ์„ผ์„œ์˜ encoder๊ฐ€ ์‚ฌ์ „ํ•™์Šต ๋•Œ์™€ ๋น„์Šทํ•œ ๋ถ„ํฌ์˜ ํ† ํฐ์„ ๋‚ด๋„๋ก ๋ฏธ์„ธ์กฐ์ • ๋ฐ์ดํ„ฐ๋งŒ์œผ๋กœ ๋ฐฐ์›Œ์ ธ์•ผ ํ•œ๋‹ค. ๋…ผ๋ฌธ์€ unseen ์„ผ์„œ๋กœ Xense(๊ด‘ํ•™ imageํ˜•)์™€ Contactile(ํ•€ ๋ฐฐ์—ด arrayํ˜•)์„ ๊ณ ๋ฅด๊ณ , ๊ฐ์ž์—๊ฒŒ ๊ฐ€์žฅ ๋น„์Šทํ•œ ์‚ฌ์ „ํ•™์Šต ์„ผ์„œ๋ฅผ GelSight-Mini์™€ AetherGlove๋กœ ์ œ์‹œํ•œ๋‹ค(๋ถ€๋ก E.4, Fig. 11). ์ฆ‰ ์ด๋ฒˆ ์‹คํ—˜์˜ โ€œunseenโ€์€ ์ด๋ฏธ ๋ณธ modality ๊ณ„์—ด ์•ˆ์˜ ์ƒˆ ํ•˜๋“œ์›จ์–ด๋‹ค. ์ €์ž๋“ค๋„ ยง4์—์„œ ์ด ๋ฒ”์œ„๋ฅผ ๋ช…์‹œํ•œ๋‹ค(โ€œnew hardware within the image- and array-based modality families covered by pretrainingโ€). Xense๋Š” ๊ณต์œ  T3 trunk๊นŒ์ง€ ์žฌ์‚ฌ์šฉํ•˜๋ฏ€๋กœ ์‚ฌ์‹ค์ƒ ์„ผ์„œ๋ณ„ 3์ธต encoder๋งŒ ์ƒˆ๋กœ ๋ฐฐ์šฐ๊ณ , Contactile์€ CNN encoder ์ „์ฒด๋ฅผ ์ƒˆ๋กœ ๋ฐฐ์šด๋‹ค.


Fig. 11. unseen ์„ผ์„œ์™€ ๊ฐ€์žฅ ๋น„์Šทํ•œ ์‚ฌ์ „ํ•™์Šต ์„ผ์„œ. Xense โ†”๏ธŽ GelSight-Mini(image), Contactile โ†”๏ธŽ AetherGlove(array).

์‹คํ—˜ ์„ค์ •: ๋‹ค์„ฏ ์…‹์—…, ์„ธ ๋ฒ ์ด์Šค๋ผ์ธ (ยง3.1ยทยง3.2, ๋ถ€๋ก DยทE)

์ฒดํฌํฌ์ธํŠธ๋ฅผ ๋‹ค์„ฏ ๊ธฐ๊ด€์— ๋ฐฐํฌํ•˜๊ณ  ๊ฐ์ž์˜ ์…‹์—…์—์„œ ๋ฏธ์„ธ์กฐ์ •ํ•œ๋‹ค. ๋ฏธ์„ธ์กฐ์ •์€ A800 8์žฅ, ๋ฐ์ดํ„ฐ์…‹๋‹น 20k step, batch 64, ํ•™์Šต๋ฅ  5\times10^{-5}\to5\times10^{-6}.

์…‹์—… ๋„๋ฉ”์ธ ๋กœ๋ด‡ / end-effector ์ด‰๊ฐ ์„ผ์„œ (์œ ํ˜•) ์‚ฌ์ „ํ•™์Šต ํฌํ•จ ์—ฌ๋ถ€ ๊ณผ์ œ (๋ฏธ์„ธ์กฐ์ • ์‹œ์—ฐ ์ˆ˜) ํ‰๊ฐ€ ํšŸ์ˆ˜
UniVTAC ์‹œ๋ฎฌ, ๊ทธ๋ฆฌํผ Franka GelSight-Mini (image) ์„ผ์„œ seen, ์‹œ๋ฎฌ ๋„๋ฉ”์ธ unseen 6๊ณผ์ œ (๊ณผ์ œ๋‹น 50, ๊ณต์‹ ๋ฐ์ดํ„ฐ) ๊ณผ์ œ๋‹น 100
Sharpa North ์‹ค๊ธฐ, ๋‹ค์ง€์† Sharpa North / Sharpa Wave Sharpa DTC (image) seen Draw Balloon ยท Fix Hand ยท Twist Cap (์„ธ ๊ณผ์ œ ํ•ฉ 250~400) ๊ณผ์ œ๋‹น 20
Sharpa&Dexmate ์‹ค๊ธฐ, ๋‹ค์ง€์† Dexmate(unseen) / Sharpa Wave Sharpa DTC (image) ์„ผ์„œ seen, ๋กœ๋ด‡ unseen Flip Book ยท Wipe Dish (๊ณผ์ œ๋‹น 100) ๊ณผ์ œ๋‹น 20
FlexivXense ์‹ค๊ธฐ, ๊ทธ๋ฆฌํผ Flexiv / Robotiq Xense (image) unseen Insert Hanoi ยท Insert USB (๊ณผ์ œ๋‹น 100) ๊ณผ์ œ๋‹น 20
TactileUMI ์‹ค๊ธฐ, ๊ทธ๋ฆฌํผ Franka / 3D ํ”„๋ฆฐํŠธ ๊ทธ๋ฆฌํผ(unseen) Contactile (array) unseen Wipe Board (50) ๊ณผ์ œ๋‹น 20

Fig. 5. ์‚ฌ์ „ํ•™์Šต์— ์žˆ๋˜ ์„ผ์„œ์˜ ์„ธ ์…‹์—…: Sharpa North, Sharpa&Dexmate, UniVTAC.

Fig. 6. ์‚ฌ์ „ํ•™์Šต์— ์—†๋˜ ์„ผ์„œ์˜ ๋‘ ์…‹์—…: FlexivXense(Xense, image)์™€ TactileUMI(Contactile, array).

๋ฒ ์ด์Šค๋ผ์ธ์€ ์„ธ ๊ฐœ์ด๊ณ , ๋ฌด์—‡์„ ๋ถ„๋ฆฌํ•˜๋ ค๋Š”์ง€๊ฐ€ ๋ถ„๋ช…ํ•˜๋‹ค.

๋ฒ ์ด์Šค๋ผ์ธ ์ด‰๊ฐ ์ž…๋ ฅ tactile expert FTP-1 ์‚ฌ์ „ํ•™์Šต ๋ถ„๋ฆฌํ•˜๋ ค๋Š” ์š”์ธ
\pi_{0.5} ์—†์Œ ์—†์Œ ์—†์Œ ์ด‰๊ฐ ์ž์ฒด์˜ ํšจ๊ณผ
Tactile-VLA ์žˆ์Œ (VLM expert์— ํ† ํฐ ์ฃผ์ž…) ์—†์Œ ์—†์Œ ์œตํ•ฉ ๊ตฌ์กฐ์˜ ํšจ๊ณผ
FTP-\pi_{0.5} ์žˆ์Œ ์žˆ์Œ ์—†์Œ (\pi_{0.5} ๊ฐ€์ค‘์น˜๋กœ ๋ถ€๋ถ„ ์ดˆ๊ธฐํ™”) ๋Œ€๊ทœ๋ชจ ์ด‰๊ฐ ์‚ฌ์ „ํ•™์Šต์˜ ํšจ๊ณผ
FTP-1 ์žˆ์Œ ์žˆ์Œ ์žˆ์Œ

๋ชจ๋“  ๋ฐฉ๋ฒ•์€ \pi_{0.5} ์ฝ”๋“œ๋ฒ ์ด์Šค๋กœ ๊ตฌํ˜„๋˜๊ณ  โ€œcomparable model scale, finetuning data, and training protocolโ€์„ ์“ด๋‹ค๊ณ  ์ ๋Š”๋‹ค. Tactile-VLA๋Š” Huang et al.์˜ ๋ฐฉ๋ฒ•์„ ์ด ์ฝ”๋“œ๋ฒ ์ด์Šค๋กœ ์˜ฎ๊ธด ๊ฒƒ์ด๊ณ , ์› ๋…ผ๋ฌธ ์ €์ž ์ƒ๋‹น์ˆ˜๊ฐ€ FTP-1 ์ €์ž์™€ ๊ฒน์นœ๋‹ค. TactileUMI ํ•˜๋“œ์›จ์–ด๋„ Tactile-VLA ๋…ผ๋ฌธ์˜ UMI ๊ธฐ๋ฐ˜ ์ˆ˜์ง‘ ์žฅ์น˜์—์„œ ์™”๋‹ค(Table 9).

UniVTAC 8๊ณผ์ œ ์ค‘ ๋‘ ๊ฐœ๋Š” ๋บ๋‹ค. Grasp Classify๋Š” ๊ธฐ์กด ๋ฒ ์ด์Šค๋ผ์ธ์ด ์ด๋ฏธ ์™„๋ฒฝํ•˜๊ฒŒ ํ’€์—ˆ๊ณ (UniVTAC ์› ๋…ผ๋ฌธ์—์„œ VITaL 100%), Insert HDMI๋Š” motion planning์œผ๋กœ ๋งŒ๋“  ์‹œ์—ฐ์ด ๋งค๋ฒˆ ์™„๋ฒฝํ•˜๊ฒŒ ๊ฝ‚ํ˜€ ์ด‰๊ฐ ํ”ผ๋“œ๋ฐฑ์„ ํ‰๊ฐ€ํ•˜๊ธฐ์— ์ ‘์ด‰ ์ •๋ณด๊ฐ€ ๋ถ€์กฑํ•˜๋‹ค๋Š” ์ด์œ ๋‹ค(๋ถ€๋ก E.2). ์•ž์˜ ์ด์œ ๋Š” ๋‚ฉ๋“๋˜์ง€๋งŒ ๋’ค์˜ ์ด์œ ๋Š” UniVTAC ๋ฆฌ๋ทฐ์— ์ •๋ฆฌํ•œ ์› ๋…ผ๋ฌธ Table I๊ณผ ๋‚˜๋ž€ํžˆ ๋†“์œผ๋ฉด ๋œ ๊น”๋”ํ•˜๋‹ค. ๊ทธ ํ‘œ์—์„œ Insert HDMI๋Š” ACT 15.0% โ†’ ACT + UniVTAC ์ด‰๊ฐ encoder 28.0%๋กœ ์ด‰๊ฐ์„ ๋„ฃ์—ˆ์„ ๋•Œ ๋‘ ๋ฐฐ ๊ฐ€๊นŒ์ด ์˜ฌ๋ž๋‹ค. ๊ทธ ๊ณผ์ œ๊ฐ€ ์ด‰๊ฐ policy ํ‰๊ฐ€์— ๋ถ€์ ํ•ฉํ•˜๋‹ค๋Š” ํŒ๋‹จ์€ FTP-1 ์ €์ž๋“ค์˜ ํŒ๋‹จ์ด๊ณ , ์› ๋ฒค์น˜๋งˆํฌ์˜ ๊ฒฐ๊ณผ์™€๋Š” ์–ด๊ธ‹๋‚œ๋‹ค.

์‹คํ—˜ 1: ์‚ฌ์ „ํ•™์Šต์— ์žˆ๋˜ ์„ผ์„œ (ยง3.3, Table 1ยท2)

UniVTAC ์‹œ๋ฎฌ๋ ˆ์ด์…˜ (Table 1)

๋ฐฉ๋ฒ• Lift Bottle Pull-out Key Lift Can Put Bottle Insert Hole Insert Tube Avg. Avg. w/o Lift
VITaL* 72 47 8 32 25 34 36.33 34.5
UniVTAC-ACT* 71 46 29 31 25 56 43.00 39.5
\pi_{0.5} 97 38 72 16 31 41 49.16 31.5
Tactile-VLA 97 32 15 10 41 56 41.83 34.75
FTP-\pi_{0.5} 77 30 26 19 47 72 45.16 42.0
FTP-1 97 48 65 47 64 79 66.66 59.5

*๋Š” UniVTAC ์› ๋…ผ๋ฌธ์˜ ๊ฐ’. ๊ณผ์ œ๋‹น 100ํšŒ.

  • ์ €์ž๋“ค์ด ๋จผ์ € ์งš๋“ฏ์ด ๋‘ lift ๊ณผ์ œ๋Š” ์ด‰๊ฐ ์—†์ด๋„ ๊ฑฐ์˜ ํ’€๋ฆฐ๋‹ค(\pi_{0.5} 97%, 72%). ๊ทธ๋ž˜์„œ โ€œAvg. w/o Liftโ€๋ฅผ ํ•จ๊ป˜ ๋ณด๊ณ ํ•œ๋‹ค. ์ด ์ง€ํ‘œ์—์„œ \pi_{0.5}๋Š” 31.5%๋กœ ์˜คํžˆ๋ ค ์ตœํ•˜์œ„๊ฐ€ ๋˜๊ณ , FTP-1์€ ๋‘ ์ง€ํ‘œ ๋ชจ๋‘์—์„œ ์ฐจ์ˆœ์œ„๋ณด๋‹ค 17.5%p ๋†’๋‹ค(์ „์ฒด ํ‰๊ท ์€ \pi_{0.5} ๋Œ€๋น„, lift ์ œ์™ธ๋Š” FTP-\pi_{0.5} ๋Œ€๋น„).
  • ํ‰๊ท  ์—ด์€ ๊ฐ ํ–‰์˜ ์นธ ๊ฐ’๊ณผ ๋ชจ๋‘ ๋งž๋Š”๋‹ค(ํ•„์ž ๊ฒ€์‚ฐ, 66.66 ๊ฐ™์€ ๊ฐ’์€ ๋ฐ˜์˜ฌ๋ฆผ์ด ์•„๋‹ˆ๋ผ ์†Œ์ˆ˜ ๋‘˜์งธ ์ž๋ฆฌ ์ ˆ์‚ฌ). ๋‹จ ํ•œ ์นธ์ด ์› ์ถœ์ฒ˜์™€ ๋‹ค๋ฅด๋‹ค. UniVTAC-ACT*์˜ Insert Hole์€ ์› ๋…ผ๋ฌธ Table I์—์„œ 24.0์ธ๋ฐ ์ด ํ‘œ๋Š” 25๋กœ ์˜ฎ๊ฒผ๋‹ค. 24๋กœ ๋‹ค์‹œ ๊ณ„์‚ฐํ•˜๋ฉด ๊ทธ ํ–‰์˜ ํ‰๊ท ์€ 42.83 / 39.25๊ฐ€ ๋œ๋‹ค. ๊ฒฐ๋ก ์— ์˜ํ–ฅ์€ ์—†๋Š” ์ „์‚ฌ ์˜ค๋ฅ˜๋‹ค.
  • *๊ฐ€ ๋ถ™์€ ๋‘ ํ–‰์€ ACT ๊ธฐ๋ฐ˜์˜ ์ž‘์€ policy์ด๊ณ  4,000 step ํ•™์Šต์ด๋‹ค. ๋‚˜๋จธ์ง€ ๋„ค ํ–‰์€ \pi_{0.5} ๊ทœ๋ชจ์˜ VLA๋‹ค. ๋‘ ๋ฌถ์Œ์„ ๊ฐ™์€ ํ‘œ์—์„œ ๋น„๊ตํ•  ๋•Œ๋Š” ์ด ๊ทœ๋ชจ ์ฐจ์ด๋ฅผ ๊ฐ์•ˆํ•ด์•ผ ํ•œ๋‹ค.
  • ๊ณผ์ œ๋‹น 100ํšŒ์ด๋ฏ€๋กœ lift ์ œ์™ธ 4๊ณผ์ œ ํ•ฉ์‚ฐ์œผ๋กœ FTP-1 238/400 vs FTP-\pi_{0.5} 168/400์˜ Fisher ์ •ํ™•๊ฒ€์ • p<10^{-4}์ด๋‹ค(ํ•„์ž ๊ณ„์‚ฐ, ํ•™์Šต 1ํšŒ์˜ rollout ๋ถ„์‚ฐ๋งŒ ๋ฐ˜์˜).

์‹ค๊ธฐ (Table 2)

๋ฐฉ๋ฒ• Draw Balloon Fix Hand (Tear) Fix Hand (Finish) Twist Cap Flip Book Wipe Dish Average (ํ‘œ๊ธฐ) Average (ํ•„์ž ์žฌ๊ณ„์‚ฐ)
\pi_{0.5} 35 70 35 40 65 30 45.3 45.83
Tactile-VLA 20 80 25 10 45 35 35.8 35.83
FTP-\pi_{0.5} 25 65 25 20 70 45 41.6 41.67
FTP-1 45 80 40 65 85 60 62.5 62.50

์•ž ๋„ค ์—ด์€ Sharpa North, ๋’ค ๋‘ ์—ด์€ Sharpa&Dexmate. ๊ณผ์ œ๋‹น 20ํšŒ.

  • ํ—ค๋“œ๋ผ์ธ +17.2%๋Š” ํ‘œ๊ธฐ ์˜ค๋ฅ˜ ์œ„์— ์žˆ๋‹ค. \pi_{0.5} ํ–‰์˜ ์—ฌ์„ฏ ์นธ ํ•ฉ์€ 275์ด๊ณ  ํ‰๊ท ์€ 45.83%๋‹ค. ๋‹ค๋ฅธ ์„ธ ํ–‰์€ ํ‘œ๊ธฐ ํ‰๊ท ์ด ์นธ ํ‰๊ท ์˜ ์ ˆ์‚ฌ๊ฐ’๊ณผ ๋งž๋Š”๋ฐ ์ด ํ–‰๋งŒ 45.3์œผ๋กœ ์ ํ˜€ ์žˆ๋‹ค(PDF์™€ HTML, ํ”„๋กœ์ ํŠธ ํŽ˜์ด์ง€ ํ‘œ ๋ชจ๋‘ 45.3). ์ดˆ๋กยท์„œ๋ก ยทFig. 1์˜ โ€œ+17.2%โ€๋Š” 62.5 โˆ’ 45.3์ด๋‹ค. ์นธ ๊ฐ’์œผ๋กœ ๊ณ„์‚ฐํ•˜๋ฉด ์ฐจ์ด๋Š” +16.7%p๋‹ค. ๊ฒฐ๋ก ์„ ๋ฐ”๊พธ๋Š” ํฌ๊ธฐ๋Š” ์•„๋‹ˆ์ง€๋งŒ ์ดˆ๋ก ์ˆซ์ž๊ฐ€ ์ด ์˜ค๊ธฐ์—์„œ ๋‚˜์™”๋‹ค.
  • Fix Hand๊ฐ€ ๋‘ ๋ฒˆ ์„ธ์–ด์ง„๋‹ค. Tear์™€ Finish๋Š” ๊ฐ™์€ ์—ํ”ผ์†Œ๋“œ์˜ ๋‘ ๋‹จ๊ณ„๋‹ค(๋ถ€๋ก E.2). 6์—ด ํ‰๊ท ์€ Fix Hand์— ๋‹ค๋ฅธ ๊ณผ์ œ์˜ ๋‘ ๋ฐฐ ๊ฐ€์ค‘์น˜๋ฅผ ์ค€๋‹ค. Finish ๋‹จ๊ณ„๋ฅผ ๋นผ๊ณ  ๊ณผ์ œ 5๊ฐœ๋กœ ๊ณ„์‚ฐํ•˜๋ฉด FTP-1 67.0%, \pi_{0.5} 48.0%๋กœ ์ฐจ์ด๋Š” 19.0%p(ํ•„์ž ๊ณ„์‚ฐ)๋ผ ๋ฐฉํ–ฅ์€ ๊ฐ™๋‹ค.
  • ๊ณผ์ œ ๋‹จ์œ„๋กœ๋Š” ๋Œ€๋ถ€๋ถ„ ๊ตฌ๋ถ„๋˜์ง€ ์•Š๋Š”๋‹ค. ๊ณผ์ œ๋‹น 20ํšŒ์—์„œ FTP-1 vs \pi_{0.5}์˜ Fisher p๋Š” Twist Cap 13/20 vs 8/20์ด 0.20, Wipe Dish 12/20 vs 6/20์ด 0.11, Flip Book 17/20 vs 13/20์ด 0.27, Draw Balloon 9/20 vs 7/20์ด 0.75๋‹ค(ํ•„์ž ๊ณ„์‚ฐ). ๋‹ค์„ฏ ๊ณผ์ œ๋ฅผ ํ•ฉ์น˜๋ฉด 67/100 vs 48/100, p\approx0.01์ด๋‹ค. ์ด ํ•ฉ์‚ฐ๋„ ํ•™์Šต 1ํšŒ์˜ rollout ๋ถ„์‚ฐ๋งŒ ๋ฐ˜์˜ํ•˜๋ฉฐ, ํ•™์Šต seed ๊ฐ„ ๋ถ„์‚ฐ์€ ๋…ผ๋ฌธ์— ์—†๋‹ค.
  • ์ €์ž๋“ค์ด ๊ฐ•์กฐํ•˜๋Š” ๊ด€์ฐฐ์€ \pi_{0.5}๊ฐ€ 2์œ„๋ผ๋Š” ์ ์ด๋‹ค. ์ด‰๊ฐ์„ ๋„ฃ์€ ๋‘ ๋ฒ ์ด์Šค๋ผ์ธ(Tactile-VLA, FTP-\pi_{0.5})์ด ์ด‰๊ฐ ์—†๋Š” \pi_{0.5}๋ณด๋‹ค ๋‚ฎ๋‹ค. ์ €์ž ํ•ด์„์€ โ€œ์ ์ ˆํ•œ ์œตํ•ฉ ์—†์ด ๋„ฃ์€ ์ด‰๊ฐ์ด vision-language ์ง€๊ฐ์„ ๋ฐฉํ•ดํ•  ์ˆ˜ ์žˆ๋‹คโ€์ด๊ณ , ์ด๋Š” ๊ฐ™์€ ๊ฒฐ๋ก ์„ ๋‚ธ ์„ ํ–‰ ์—ฐ๊ตฌ(Niu et al., touch dreaming)์™€ ์ผ์น˜ํ•œ๋‹ค๊ณ  ์ ๋Š”๋‹ค. FTP-\pi_{0.5}๋Š” FTP-1๊ณผ ๊ตฌ์กฐ๊ฐ€ ๊ฐ™๊ณ  ์‚ฌ์ „ํ•™์Šต๋งŒ ์—†๋‹ค. ๋”ฐ๋ผ์„œ ์ด ๊ด€์ฐฐ์€ โ€œ๊ตฌ์กฐโ€๋ณด๋‹ค โ€œ์‚ฌ์ „ํ•™์Šต ์—†๋Š” ์ด‰๊ฐ branchโ€๊ฐ€ ํ•ด๋กญ๋‹ค๋Š” ์ชฝ์œผ๋กœ ์ฝํžŒ๋‹ค.

์ •์„ฑ์ ์œผ๋กœ๋Š” Tactile-VLA์™€ FTP-\pi_{0.5}๊ฐ€ ์ ‘์ด‰ ์กฐ๊ฑด์ด ๋ฐ”๋€” ๋•Œ ๋ถˆ์•ˆ์ •ํ•œ action์„ ๋‚ด๊ณ , \pi_{0.5}๋Š” Sharpa&Dexmate ๋‘ ๊ณผ์ œ์—์„œ ๋ˆ„๋ฅด๋Š” ํž˜์„ ์ผ์ •ํ•˜๊ฒŒ ์œ ์ง€ํ•˜์ง€ ๋ชปํ•˜๋ฉฐ ๋ณ‘๋šœ๊ป‘์„ ๋ฐ˜์‘ ์—†์ด ๋ฐ€์–ด ๋ถ™์ธ๋‹ค๊ณ  ์ ๋Š”๋‹ค.

์ด ๋น„๊ต์—์„œ ๋ฌด์—‡์ด ํ†ต์ œ๋˜์ง€ ์•Š์•˜๋‚˜. Sharpa ๋‘ ์…‹์—…์—์„œ FTP-1์€ ์‚ฌ์ „ํ•™์Šต ๋•Œ ๊ฐ™์€ Sharpa Wave ์†ยทSharpa DTC ์„ผ์„œ์˜ ๋ฐ์ดํ„ฐ๋ฅผ ๋ณด์•˜๊ณ , ํŠนํžˆ Sharpa North๋Š” ๊ฐ™์€ ๋กœ๋ด‡์˜ ์‹œ์—ฐ 4,000๊ฐœ๋ฅผ ๋ณด์•˜๋‹ค. \pi_{0.5}์™€ FTP-\pi_{0.5}๋Š” ๊ทธ ๋ฐ์ดํ„ฐ๋ฅผ ๋ณด์ง€ ์•Š์•˜๋‹ค. ๊ทธ๋Ÿฌ๋‹ˆ Table 2์˜ ๊ฒฉ์ฐจ์—๋Š” โ€œ์ด‰๊ฐ ์‚ฌ์ „ํ•™์Šตโ€๊ณผ โ€œ๊ฐ™์€ ๋กœ๋ด‡ยท๊ฐ™์€ ์†์˜ action ๋ฐ์ดํ„ฐ ๋…ธ์ถœโ€์ด ์„ž์—ฌ ์žˆ๋‹ค. ์ด๋ฅผ ๊ฐ€๋ฅด๋Š” ๋Œ€์กฐ๊ตฐ(๊ฐ™์€ ๊ถค์ ์œผ๋กœ ์ด์–ด ํ•™์Šตํ•œ \pi_{0.5})์€ ยง4.3์— ์žˆ์ง€๋งŒ UniVTAC์—์„œ๋งŒ ๋Œ๋ ธ๋‹ค. ์‹ค๊ธฐ Sharpa ์…‹์—…์—์„œ ๊ทธ ํšจ๊ณผ๋ฅผ ๋‚˜๋ˆŒ ๊ทผ๊ฑฐ๋Š” ๋…ผ๋ฌธ ์•ˆ์— ์—†๋‹ค.

์‹คํ—˜ 2: ์‚ฌ์ „ํ•™์Šต์— ์—†๋˜ ์„ผ์„œ (ยง4.1, Table 3)

๋ฐฉ๋ฒ• Insert Hanoi (FlexivXense) Insert USB (FlexivXense) Wipe Board (TactileUMI) Avg.
\pi_{0.5} 25 0 20 15.0
Tactile-VLA 0 10 15 8.3
FTP-\pi_{0.5} 5 10 30 15.0
FTP-1 55 30 55 46.6

๊ณผ์ œ๋‹น 20ํšŒ. ๋ฏธ์„ธ์กฐ์ • ์‹œ์—ฐ์€ FlexivXense ๊ณผ์ œ๋‹น 100๊ฐœ, TactileUMI 50๊ฐœ.

FTP-1์€ ์ƒˆ ์„ผ์„œ์šฉ encoder๋ฅผ ์ฒ˜์Œ๋ถ€ํ„ฐ ํ•™์Šตํ•˜๊ณ , tactile expertยทfunctional-area embeddingยท(Xense์˜ ๊ฒฝ์šฐ) ๊ณต์œ  T3 trunk, VLยทaction expert๋Š” FTP-1 ์ฒดํฌํฌ์ธํŠธ์—์„œ ์‹œ์ž‘ํ•œ๋‹ค. ๊ณต๊ฐœ ํ•™์Šต ์ฝ”๋“œ๋Š” ๊ธฐ๋ณธ์ ์œผ๋กœ ์ด ๋ชจ๋“ˆ๋“ค์„ ์–ผ๋ฆฌ์ง€ ์•Š๊ณ  ํ•จ๊ป˜ ๋ฏธ์„ธ์กฐ์ •ํ•œ๋‹ค(๊ณต์œ  trunk๋งŒ ์„ ํƒ์ ์œผ๋กœ ์–ผ๋ฆฌ๋Š” ์˜ต์…˜์ด ์žˆ๋‹ค). ํ”„๋กœ์ ํŠธ ํŽ˜์ด์ง€๋Š” ์ด ์‹คํ—˜์„ โ€œlearning only the sensor-specific encoderโ€๋ผ๊ณ  ์†Œ๊ฐœํ•˜๋Š”๋ฐ, ๋…ผ๋ฌธ์˜ โ€œreusingโ€์€ ์ดˆ๊ธฐํ™”๋กœ ์žฌ์‚ฌ์šฉํ•œ๋‹ค๋Š” ๋œป์ด๊ณ  ํ•™์Šต ๋Œ€์ƒ์„ encoder๋กœ ํ•œ์ •ํ•˜์ง€ ์•Š๋Š”๋‹ค.

  • ํ‰๊ท  46.6%๋Š” 15.0% ๋Œ€๋น„ +31.6%p๋‹ค. ์ดˆ๋ก์€ ์ด๋ฅผ โ€œ+31%โ€๋กœ ์ ๋Š”๋‹ค. ๋น„๊ต ๊ธฐ์ค€์€ FTP-\pi_{0.5}์ด๊ณ , \pi_{0.5}๋„ ๊ฐ™์€ 15.0%๋‹ค.
  • ์„ธ ๊ณผ์ œ๋ฅผ ํ•ฉ์น˜๋ฉด 28/60 vs 9/60, Fisher p\approx0.0003์ด๋‹ค(ํ•„์ž ๊ณ„์‚ฐ). ๊ณผ์ œ๋ณ„๋กœ๋Š” Insert Hanoi 11/20 vs 5/20(\pi_{0.5}) p\approx0.11, Insert USB 6/20 vs 2/20 p\approx0.24, Wipe Board 11/20 vs 6/20 p\approx0.20์œผ๋กœ ๋‹จ๋…์œผ๋กœ๋Š” ๊ตฌ๋ถ„๋˜์ง€ ์•Š๋Š”๋‹ค. ํ•ฉ์‚ฐ ๊ฒฉ์ฐจ๋Š” ์‹ค๊ธฐ ๊ฒฐ๊ณผ ์ค‘ ๊ฐ€์žฅ ๋‹จ๋‹จํ•˜์ง€๋งŒ, ์—ฌ์ „ํžˆ ํ•™์Šต 1ํšŒ ๊ธฐ์ค€์ด๋‹ค.
  • ๋ฒ ์ด์Šค๋ผ์ธ ์ชฝ์—์„œ ๋ˆˆ์— ๋„๋Š” ์นธ์€ \pi_{0.5}์˜ Insert Hanoi 25%๋‹ค. ์ด‰๊ฐ ์—†๋Š” \pi_{0.5}๊ฐ€ ์ด‰๊ฐ์„ ๋„ฃ์€ FTP-\pi_{0.5}(5%)์™€ Tactile-VLA(0%)๋ณด๋‹ค ๋†’๋‹ค. Table 2์—์„œ ๋ณธ โ€œ์‚ฌ์ „ํ•™์Šต ์—†๋Š” ์ด‰๊ฐ branch๋Š” ํ•ด๊ฐ€ ๋  ์ˆ˜ ์žˆ๋‹คโ€๋Š” ํŒจํ„ด์ด ์—ฌ๊ธฐ์„œ๋„ ๋ฐ˜๋ณต๋œ๋‹ค. ์ €์ž๋“ค์€ ๋Œ€์‹  โ€œ\pi_{0.5}๊ฐ€ FTP-\pi_{0.5}์™€ ๋น„์Šทํ•˜๋‹คโ€๋Š” ์ชฝ์œผ๋กœ ์„œ์ˆ ํ•œ๋‹ค.

์ •์„ฑ ๊ด€์ฐฐ์€ ์ด๋ ‡๋‹ค. Insert Hanoi์—์„œ FTP-1๊ณผ \pi_{0.5}๋Š” recovery ํ–‰๋™์„ ๋ณด์ด์ง€๋งŒ ๋‚˜๋จธ์ง€๋Š” ๋ณด์ด์ง€ ์•Š๋Š”๋‹ค. FTP-1์€ ๊ณ ๋ฆฌ๊ฐ€ ๊ธฐ๋‘ฅ๊ณผ ์–ด๊ธ‹๋‚ฌ์„ ๋•Œ ์ด‰๊ฐ์„ ๋ณด๊ณ  ์‚ฝ์ž…์„ ๋Šฆ์ถ”๊ณ  \pi_{0.5}๋Š” ๊ทธ๋Ÿฌ์ง€ ์•Š๋Š”๋‹ค. Insert USB๋Š” ์‹œ์—ฐ 100๊ฐœ๋กœ๋Š” ์ตํžˆ๊ธฐ ์–ด๋ ค์šด ๊ณผ์ œ์—ฌ์„œ ๋ฐ์ดํ„ฐ ํšจ์œจ์˜ ์‹œํ—˜๋Œ€๋ผ๊ณ  ์ ๋Š”๋‹ค. Wipe Board์—์„œ๋Š” ๋‹ค๋ฅธ ๋ชจ๋ธ๋“ค์ด ๋ˆ„๋ฅด๋Š” ํž˜์„ ์œ ์ง€ํ•˜์ง€ ๋ชปํ•˜๊ณ  ์ ‘์ด‰์„ ์žƒ๋Š”๋‹ค. ์ด ๊ด€์ฐฐ๋“ค์€ ํ”„๋กœ์ ํŠธ ํŽ˜์ด์ง€ ์˜์ƒ๊ณผ Fig. 15ยท16์˜ ๋Œ€ํ‘œ rollout์œผ๋กœ๋งŒ ์ œ์‹œ๋˜๊ณ  ์ •๋Ÿ‰ํ™”๋˜์ง€ ์•Š์•˜๋‹ค(์˜ˆ: ์‚ฝ์ž… ์†๋„๋‚˜ ์ ‘์ด‰๋ ฅ์˜ ์‹œ๊ณ„์—ด ๋น„๊ต๋Š” ์—†๋‹ค).

์‹คํ—˜ 3: ์ด๋“์€ ์ด‰๊ฐ ์‚ฌ์ „ํ•™์Šต์—์„œ ์˜ค๋Š”๊ฐ€ (ยง4.2, Fig. 7, Table 10)

์ €์ž๋“ค์€ ๋‘ ๊ฐ€์„ค์„ ์„ธ์šด๋‹ค. H1์€ โ€œ์‚ฌ์ „ํ•™์Šต ๋ฐ์ดํ„ฐ ๋ถ„ํฌ๊ฐ€ downstream๊ณผ ๊ฐ€๊นŒ์›Œ์„œ ๋ฏธ์„ธ์กฐ์ •์ด ์‰ฌ์›Œ์กŒ์„ ๋ฟโ€์ด๊ณ , H2๋Š” โ€œ์ „์ด ๊ฐ€๋Šฅํ•œ ์ด‰๊ฐ ์กฐ์ž‘ ์ง€์‹์„ ๋ฐฐ์› ๋‹คโ€์ด๋‹ค. ์ด๋ฅผ ๊ฐ€๋ฅด๋ ค๊ณ  NTP๋ฅผ ๋งŒ๋“ ๋‹ค. FTP-1๊ณผ ๊ฐ™์€ ๋ฐ์ดํ„ฐยท๊ฐ™์€ ์ตœ์ ํ™”๋กœ ์‚ฌ์ „ํ•™์Šตํ•˜๋˜ ์ด‰๊ฐ ์ž…๋ ฅ๊ณผ ์ด‰๊ฐ ๊ด€๋ จ ๊ตฌ์กฐ๋ฅผ ๋บ€ ์ฒดํฌํฌ์ธํŠธ๋‹ค. ๋ฏธ์„ธ์กฐ์ • ๋•Œ FTP-1๊ณผ ๊ฐ™์€ ์ด‰๊ฐ ๊ตฌ์กฐ๋ฅผ ๋ถ™์ธ ๊ฒƒ์„ NTP-1์ด๋ผ ๋ถ€๋ฅธ๋‹ค. ์„ธ ๋ชจ๋ธ์€ ๋ฏธ์„ธ์กฐ์ • ์‹œ์ ์˜ ๊ตฌ์กฐ๊ฐ€ ๊ฐ™๊ณ  ์ฒดํฌํฌ์ธํŠธ๋งŒ ๋‹ค๋ฅด๋‹ค.


Fig. 7. FTP-\pi_{0.5} ยท NTP-1 ยท FTP-1์˜ ํ‰๊ท  ์„ฑ๊ณต๋ฅ . UniVTAC 41.8 / 50.0 / 66.6, FlexivXense 7.5 / 5.0 / 42.5.

๋ถ€๋ก Table 10์ด UniVTAC ๊ณผ์ œ๋ณ„ ๊ฐ’์„ ์ค€๋‹ค.

๋ฐฉ๋ฒ• Lift Bottle Pull-out Key Lift Can Put Bottle Insert Hole Insert Tube Avg. Avg. w/o Lift
FTP-\pi_{0.5} 77 30 26 19 47 72 45.16 42.0
NTP-1 88 38 66 32 31 45 50.00 36.5
FTP-1 97 48 65 47 64 79 66.66 59.5

์—ฌ๊ธฐ์„œ ์„ธ ๊ฐ€์ง€๊ฐ€ ๋ณด์ธ๋‹ค.

  1. Fig. 7์˜ FTP-\pi_{0.5} UniVTAC ๋ง‰๋Œ€(41.8)๋Š” ํ‘œ์™€ ๋‹ค๋ฅด๋‹ค. Table 1๊ณผ Table 10์€ ๋ชจ๋‘ 45.16์ด๋‹ค. 41.8์€ Table 1์—์„œ Tactile-VLA์˜ ํ‰๊ท (41.83)์ด๋‹ค. ๊ทธ๋ฆผ์— ๋‹ค๋ฅธ ํ–‰์˜ ๊ฐ’์ด ๋“ค์–ด๊ฐ„ ๊ฒƒ์œผ๋กœ ๋ณด์ธ๋‹ค. FlexivXense์˜ 7.5๋Š” Table 3์˜ FTP-\pi_{0.5} ๋‘ ๊ณผ์ œ ํ‰๊ท ((5+10)/2)๊ณผ ๋งž๋Š”๋‹ค.
  2. โ€œNTP-1์ด FTP-\pi_{0.5}๋ณด๋‹ค ๋‚ซ๋‹คโ€๋Š” lift ๊ณผ์ œ์—์„œ๋งŒ ์„ฑ๋ฆฝํ•œ๋‹ค. ์ €์ž๋“ค์€ ์ „์ฒด ํ‰๊ท (50.0 vs 45.16)์„ ๊ทผ๊ฑฐ๋กœ โ€œFTP-1-Dataset์˜ ๋ถ„ํฌ๊ฐ€ UniVTAC์— ๋” ๊ฐ€๊นŒ์šธ ์ˆ˜ ์žˆ๋‹คโ€๊ณ  ์“ด๋‹ค(H1์˜ ๋ถ€๋ถ„ ์ง€์ง€). ๊ทธ๋Ÿฌ๋‚˜ ์ด‰๊ฐ์ด ํ•„์š”ํ•œ lift ์ œ์™ธ 4๊ณผ์ œ์—์„œ๋Š” NTP-1์ด 36.5%๋กœ ์‚ฌ์ „ํ•™์Šต์ด ์—†๋Š” FTP-\pi_{0.5}(42.0%)๋ณด๋‹ค ๋‚ฎ๋‹ค. ํŠนํžˆ Insert Hole 31 vs 47, Insert Tube 45 vs 72๋กœ ์‚ฝ์ž… ๊ณผ์ œ์—์„œ ํฌ๊ฒŒ ๋’ค์ง„๋‹ค. ๊ฐ™์€ ๋ฐ์ดํ„ฐ๋กœ ์‹œ๊ฐ๋งŒ ์‚ฌ์ „ํ•™์Šตํ•œ ํšจ๊ณผ๋Š” ์‹œ๊ฐ์œผ๋กœ ํ’€๋ฆฌ๋Š” lift ๊ณผ์ œ์— ๋ชฐ๋ ค ์žˆ๊ณ , ์ด‰๊ฐ ๊ณผ์ œ์—๋Š” ์˜คํžˆ๋ ค ์†ํ•ด๋‹ค. ์ด ๋ถ„ํ•ด๋Š” H2๋ฅผ ์ €์ž๋“ค์˜ ์„œ์ˆ ๋ณด๋‹ค ๋” ๊ฐ•ํ•˜๊ฒŒ ์ง€์ง€ํ•œ๋‹ค. lift ์ œ์™ธ ์ง€ํ‘œ์—์„œ FTP-1 59.5 vs NTP-1 36.5๋กœ ์ฐจ์ด๊ฐ€ 23.0%p๋‹ค.
  3. FlexivXense์—์„œ์˜ ์ฐจ์ด๋Š” ํฌ์ง€๋งŒ ํ‘œ๋ณธ์ด ์ž‘๋‹ค. 42.5 vs 5.0(+37.5%p)์€ ๊ณผ์ œ 2๊ฐœ ร— 20ํšŒ๋กœ 17/40 vs 2/40, Fisher p\approx10^{-4}์ด๋‹ค(ํ•„์ž ๊ณ„์‚ฐ). NTP-1(5.0)์ด ์‚ฌ์ „ํ•™์Šต ์—†๋Š” FTP-\pi_{0.5}(7.5)๋ณด๋‹ค๋„ ๋‚ฎ๋‹ค. ๊ณผ์ œ๋ณ„ NTP-1 ๊ฐ’์€ ๋…ผ๋ฌธ์— ์—†๋‹ค.

๊ฐ™์€ ๊ถค์ ์— ๋…ธ์ถœ๋œ VLA์™€์˜ ๋น„๊ต (ยง4.3, ๋ถ€๋ก F.1, Table 4)

FTP-1 ์‚ฌ์ „ํ•™์Šต์— ์“ด ๋กœ๋ด‡ ๊ถค์ ์œผ๋กœ \pi_{0.5}๋ฅผ ๊ฐ™์€ ์˜ˆ์‚ฐ๋งŒํผ ์ด์–ด ํ•™์Šตํ•˜๊ณ (์ด‰๊ฐ ์—†์ด) UniVTAC์—์„œ ๋ฏธ์„ธ์กฐ์ •ํ–ˆ๋‹ค.

๋ฐฉ๋ฒ• Avg. Avg. w/o Lift
\pi_{0.5} + ๊ฐ™์€ ๊ถค์ ยท๊ฐ™์€ ์˜ˆ์‚ฐ์œผ๋กœ ์ด์–ด ํ•™์Šต 57.0 44.0
FTP-1 66.7 59.5

์ฐจ์ด๋Š” 9.7 / 15.5%p๋‹ค. ์ด ๋Œ€์กฐ๊ตฐ์˜ lift ์ œ์™ธ 44.0%๋Š” NTP-1(36.5%)๋ณด๋‹ค ๋†’๋‹ค. ์ •์˜์ƒ ๋‘ ๋Œ€์กฐ๊ตฐ์€ ๋ชจ๋‘ โ€œ์ด‰๊ฐ ์—†์ด ์‚ฌ์ „ํ•™์Šตโ€์ธ๋ฐ, ๋ฏธ์„ธ์กฐ์ • ๋•Œ scratch ์ด‰๊ฐ branch๋ฅผ ๋ถ™์ธ ์ชฝ(NTP-1)์ด ์ด‰๊ฐ ์—†์ด ๋๊นŒ์ง€ ๊ฐ„ ์ชฝ๋ณด๋‹ค ์ด‰๊ฐ ๊ณผ์ œ์—์„œ ๋‚ฎ๋‹ค. ์‹คํ—˜ 1ยท2์˜ โ€œ์‚ฌ์ „ํ•™์Šต ์—†๋Š” ์ด‰๊ฐ branch๋Š” ํ•ด๊ฐ€ ๋  ์ˆ˜ ์žˆ๋‹คโ€์™€ ๊ฐ™์€ ๋ฐฉํ–ฅ์ด๋‹ค. ๋‹ค๋งŒ ๋‘ ๋Œ€์กฐ๊ตฐ์˜ ๋ฐ์ดํ„ฐ๊ฐ€ ์ •ํ™•ํžˆ ๊ฐ™์€์ง€(NTP๋Š” โ€œ๊ฐ™์€ ์‚ฌ์ „ํ•™์Šต ๋ฐ์ดํ„ฐโ€, F.1์€ โ€œ๊ฐ™์€ ๋กœ๋ด‡ ๊ถค์ โ€์ด๋ผ ์ ์–ด ์‚ฌ๋žŒ ์† ๋ฐ์ดํ„ฐ ํฌํ•จ ์—ฌ๋ถ€๊ฐ€ ๋‹ค๋ฅผ ์ˆ˜ ์žˆ๋‹ค)๋Š” ๋…ผ๋ฌธ์ด ๋ช…์‹œํ•˜์ง€ ์•Š๋Š”๋‹ค.

์ด ๋Œ€์กฐ๊ตฐ์€ ์‹คํ—˜ 1์—์„œ ์ง€์ ํ•œ ํ˜ผ์žฌ(์ด‰๊ฐ ์‚ฌ์ „ํ•™์Šต vs ๊ฐ™์€ ๋กœ๋ด‡ ๋ฐ์ดํ„ฐ ๋…ธ์ถœ)๋ฅผ ๊ฐ€๋ฅด๋Š” ๋ฐ”๋กœ ๊ทธ ์‹คํ—˜์ด๋‹ค. ๊ฒฐ๊ณผ๋Š” FTP-1์— ์œ ๋ฆฌํ•˜๊ฒŒ ๋‚˜์™”์ง€๋งŒ, GelSight-Mini ๊ทธ๋ฆฌํผ ์‹œ๋ฎฌ๋ ˆ์ด์…˜ ํ•œ ๊ณณ์—์„œ๋งŒ ๋Œ๋ ธ๋‹ค. Sharpa North์ฒ˜๋Ÿผ ๊ฐ™์€ ๋กœ๋ด‡ ๋ฐ์ดํ„ฐ๊ฐ€ ์‚ฌ์ „ํ•™์Šต์— ์ง์ ‘ ๋“ค์–ด๊ฐ„ ์‹ค๊ธฐ ์…‹์—…์—์„œ๋Š” ์ด ๋Œ€์กฐ๊ฐ€ ์—†๋‹ค.

์‚ฌ์ „ํ•™์Šต ๊ทœ๋ชจ (ยง4.3, ๋ถ€๋ก F.2, Table 11)

์„ค์ • ๋ฐ์ดํ„ฐ ๋น„์œจ ํ•™์Šต step Avg. Avg. w/o Lift
0% (FTP-\pi_{0.5}) 0% 0 45.2 42.0
50% 50% 25,000 59.0 46.0
100% (FTP-1) 100% 50,000 66.7 59.5

๋ฐ์ดํ„ฐ์™€ step์„ ํ•จ๊ป˜ ํ‚ค์› ์œผ๋ฏ€๋กœ ๋‘˜ ์ค‘ ๋ฌด์—‡์ด ํšจ๊ณผ๋ฅผ ๋ƒˆ๋Š”์ง€๋Š” ๋‚˜๋‰˜์ง€ ์•Š๋Š”๋‹ค. ์ ์€ ์„ธ ๊ฐœ๋‹ค. lift ์ œ์™ธ ์ง€ํ‘œ๋Š” 42.0 โ†’ 46.0 โ†’ 59.5๋กœ 0โ†’50% ๊ตฌ๊ฐ„์˜ ์ด๋“(+4.0)๋ณด๋‹ค 50โ†’100% ๊ตฌ๊ฐ„(+13.5)์ด ํฌ๋‹ค. 50% ์„ค์ •์˜ lift ๋‘ ๊ณผ์ œ ํ‰๊ท ์€ ํ‘œ์˜ ๋‘ ํ‰๊ท ์—์„œ (6\times59.0-4\times46.0)/2=85.0\%๋กœ ์—ญ์‚ฐ๋œ๋‹ค(ํ•„์ž ๊ณ„์‚ฐ). ์ ˆ๋ฐ˜ ์‚ฌ์ „ํ•™์Šต์˜ ์ด๋“์€ ์ฃผ๋กœ lift ๊ณผ์ œ์—์„œ ๋‚˜์˜จ ์…ˆ์ด๊ณ , ์ด๋Š” ์œ„ NTP-1์˜ ํŒจํ„ด๊ณผ ๋‹ฎ์•˜๋‹ค. ๋ถ€๋ก D๋Š” 50k step ์ดํ›„ ์„ฑ๋Šฅ์ด ํฌํ™”ํ–ˆ๋‹ค๊ณ  ์ ๋Š”๋ฐ, ์ด ํฌํ™”์˜ ๊ทผ๊ฑฐ ์ˆ˜์น˜๋Š” ์ œ์‹œ๋˜์ง€ ์•Š๋Š”๋‹ค.

๊ณ ์ • ์ด‰๊ฐ feature probing (ยง4.3, ๋ถ€๋ก F.3, Table 12)

GelSight Mini encoder์™€ ๊ณต์œ  trunk๋ฅผ ์–ผ๋ฆฐ ์ฑ„ ํž˜ ์ถ”์ •(์„ ํ˜• probe, MAE)๊ณผ ๋ฏธ๋„๋Ÿฌ์ง ๊ฒ€์ถœ(attentive probe, F1)์„ ํ–ˆ๋‹ค. ์„ธ probe seed ํ‰๊ท ์ด๋‹ค.

๋ฒค์น˜๋งˆํฌ ยท ์ง€ํ‘œ T3 (์ดˆ๊ธฐ๊ฐ’) FTP-1 trunk FTP-1 tokenizer ์ฐธ๊ณ  ๋ชจ๋ธ
TacBench T1 ํž˜, ์„ ํ˜• MAE โ†“ (N) 0.106 0.107 0.116 โ€”
TacBench T2 ๋ฏธ๋„๋Ÿฌ์ง, attentive F1 โ†‘ 0.725 0.780 0.786 0.922 (Sparsh-MAE)
FeelAnyForce ํž˜, ์„ ํ˜• MAE โ†“ (N) 0.273 0.314 0.424 0.041 (ํž˜ยท๊นŠ์ด ์ง€๋„ํ•™์Šต backbone)

ยง4.3 ๋ณธ๋ฌธ์€ โ€œslip F1์ด 0.725 โ†’ 0.780/0.786์œผ๋กœ ์˜ค๋ฅด๊ณ , force probe๋„ ์ •๋Ÿ‰์  ํž˜ ์ •๋ณด๋ฅผ ๋ณต์›ํ•œ๋‹คโ€๊ณ  ์š”์•ฝํ•œ๋‹ค. ํ‘œ์˜ ์„ธ ํ–‰์„ ๋‹ค ๋†“์œผ๋ฉด ๊ทธ๋ฆผ์ด ๋‹ค๋ฅด๋‹ค. ๋ฏธ๋„๋Ÿฌ์ง์€ ๊ฐœ์„ ๋์ง€๋งŒ ํž˜ ์ถ”์ •์€ ๋‘ ๋ฒค์น˜๋งˆํฌ ๋ชจ๋‘์—์„œ T3 ์ดˆ๊ธฐ๊ฐ’๋ณด๋‹ค ๊ฐ™๊ฑฐ๋‚˜ ๋‚˜์˜๋‹ค. FeelAnyForce์—์„œ๋Š” trunk๊ฐ€ 15%, tokenizer๊ฐ€ 55% ๋‚˜์˜๋‹ค(ํ•„์ž ๊ณ„์‚ฐ). ๋ถ€๋ก F.3์˜ ๊ฒฐ๊ณผ ๋ฌธ๋‹จ๋„ T1๊ณผ T2๋งŒ ์„œ์ˆ ํ•˜๊ณ  FeelAnyForce ํ–‰์„ ์–ธ๊ธ‰ํ•˜์ง€ ์•Š๋Š”๋‹ค. ์ •ํ™•ํ•œ ์š”์•ฝ์€ โ€œpolicy ์‚ฌ์ „ํ•™์Šต์ด ๋ฏธ๋„๋Ÿฌ์ง ์ •๋ณด๋ฅผ ๋Š˜๋ฆฌ๊ณ , ํž˜ ์ •๋ณด๋Š” ๋Œ€์ฒด๋กœ ๋ณด์กดํ•˜๊ฑฐ๋‚˜ ์ผ๋ถ€ ์žƒ๋Š”๋‹คโ€์ด๋‹ค. ๋ฏธ๋„๋Ÿฌ์ง๋„ Sparsh ๊ณ„์—ด ์ฐธ๊ณ ๊ฐ’(0.922)๊ณผ๋Š” ๊ฑฐ๋ฆฌ๊ฐ€ ํฌ๋‹ค. T2 ๋ถ„ํ• ์€ ๊ณต๊ฐœ ๋ฐ์ดํ„ฐ๋ฅผ ์ €์ž๋“ค์ด ์žฌ๊ตฌ์„ฑํ•œ ๊ฒƒ์ด๋‹ค. TacBench ์ž์ฒด๋Š” ๋ธ”๋กœ๊ทธ์˜ Sparsh ๋ฆฌ๋ทฐ์— ์ •๋ฆฌ๋ผ ์žˆ๋‹ค.

์ •์„ฑ ๊ฒฐ๊ณผ์™€ ์˜์ƒ (๋ถ€๋ก E.3ยทE.5, ํ”„๋กœ์ ํŠธ ํŽ˜์ด์ง€)

์•„๋ž˜ ์˜์ƒ์€ ํ”„๋กœ์ ํŠธ ํŽ˜์ด์ง€์˜ FTP-1 rollout์ด๋ฉฐ, ํ”„๋กœ์ ํŠธ ํŽ˜์ด์ง€๊ฐ€ ๊ณ ๋ฅธ ๋Œ€ํ‘œ ์—ํ”ผ์†Œ๋“œ๋‹ค.


Draw Balloon (Sharpa North)

Twist Cap (Sharpa North)

Fix Hand (Sharpa North)

Flip Book (Sharpa&Dexmate)

Wipe Dish (Sharpa&Dexmate)

Insert Hole (UniVTAC)

Insert Hanoi (FlexivXense, unseen)

Insert USB (FlexivXense, unseen)

Wipe Board (TactileUMI, unseen)

๋…ผ๋ฌธ ๋ถ€๋ก์˜ ๋Œ€ํ‘œ rollout ๊ทธ๋ฆผ์€ ๊ณผ์ œ๋งˆ๋‹ค ์„ฑ๊ณต rollout ํ•˜๋‚˜์˜ ๋‹ค์„ฏ key frame์ด๋‹ค.


Fig. 12. UniVTAC 6๊ณผ์ œ์˜ ๋Œ€ํ‘œ rollout.

Fig. 13. Sharpa North: Draw Balloon, Fix Hand (Tear), Fix Hand (Finish), Twist Cap.

Fig. 14. Sharpa&Dexmate: Flip Book, Wipe Dish.

Fig. 15. FlexivXense: Insert Hanoi, Insert USB.

Fig. 16. TactileUMI: Wipe Board.

๊ณต๊ฐœ ์ฝ”๋“œยท์ฒดํฌํฌ์ธํŠธ์™€ ๋Œ€์กฐ (์ •์  ํ™•์ธ)

์•„๋ž˜๋Š” ๊ณต๊ฐœ ๋ ˆํฌ ํŒŒ์ผ๊ณผ Hugging Face ์ฒดํฌํฌ์ธํŠธ์˜ configยทsafetensors ํ—ค๋”(ํ…์„œ ์ด๋ฆ„๊ณผ shape)๋ฅผ ์ฝ์€ ์ •์  ํ™•์ธ์ด๋‹ค. ๋ชจ๋ธ์„ ์‹คํ–‰ํ•ด ์„ฑ๋Šฅ์„ ํ™•์ธํ•œ ๊ฒƒ์ด ์•„๋‹ˆ๋‹ค. ๊ธฐ์ค€์€ GitHub main @ 89fa681.

ํ•ญ๋ชฉ ๋…ผ๋ฌธ ๊ณต๊ฐœ ์ฝ”๋“œ / ์ฒดํฌํฌ์ธํŠธ
tactile expert ํฌ๊ธฐ โ€œ300M-parameter Transformerโ€, width 1024 ยท depth 18 ยท MLP 4096 ยท 8 heads ยท head dim 256 (๋ถ€๋ก B.5) config tactile_expert_variant: gemma_small = width 512 ยท depth 18 ยท MLP 2048 ยท 8 heads ยท head dim 256. ์‚ฌ์ „ํ•™์Šต ์ฒดํฌํฌ์ธํŠธ์˜ tactile expert ๊ฐ€์ค‘์น˜ ํ•ฉ ์•ฝ 99.1M. UniVTAC ๋ฏธ์„ธ์กฐ์ • ์ฒดํฌํฌ์ธํŠธ ์ด๋ฆ„๋„ ..._expert_gsmall_ftp1
๊ณต์œ  ์ด๋ฏธ์ง€ trunk depth 9 ยท width 768, T3 ์ดˆ๊ธฐํ™” ์ฒดํฌํฌ์ธํŠธ์— ์•ฝ 63.8M, load_t3_pretrained_checkpoint: true
attention mask ํ–‰ VL/tactile/action, [[1,0,0],[0,1,0],[1,1,1]] ftp1_attention_masks.py๊ฐ€ ๊ฐ™์€ ๋ธ”๋ก ๊ตฌ์กฐ๋ฅผ ๋งŒ๋“ ๋‹ค(VLยทtactile ๊ฐ์ž ์•ˆ์—์„œ๋งŒ, action์€ ์…‹ ๋‹ค)
action ๊ณต๊ฐ„ D, horizon H ๊ฐ’ ๋ฏธ๊ธฐ์žฌ action_dim: 120, action_horizon: 32
๋ฏธ์„ธ์กฐ์ • ์„ค์ • 20k step, batch 64, 5\times10^{-5}\to5\times10^{-6} UniVTAC ์˜ˆ์‹œ ์Šคํฌ๋ฆฝํŠธ ๊ฐ’์ด ๊ฐ™๋‹ค(๊ทธ ๋ฐ–์— warmup 500, optimizer ฮฒโ‚‚ 0.95, grad clip 1.0)
๋ฏธ์„ธ์กฐ์ • ์‹œ ๋™๊ฒฐ โ€œreuseโ€๋ผ๊ณ ๋งŒ ์„œ์ˆ  ๊ธฐ๋ณธ์€ ์ „์ฒด ๋ฏธ์„ธ์กฐ์ •, ๊ณต์œ  ์ด๋ฏธ์ง€ trunk๋งŒ ์„ ํƒ์  ๋™๊ฒฐ ์˜ต์…˜
์‚ฌ์ „ํ•™์Šต ํ•™์Šต๋ฅ ยท๋Ÿฐ ์„ค์ • H20 48์žฅ, 50k step, 10^{-4}\to5\times10^{-5} (๋ถ€๋ก D) ๊ณต๊ฐœ ์ฒดํฌํฌ์ธํŠธ์˜ train_config.json: ์ด 400k step ์˜ˆ์ •, warmup 5,000, 180k step์— ๊ฑธ์ณ 10^{-4}\to10^{-5} cosine. ์ด ์‹์œผ๋กœ 50k ์‹œ์  ํ•™์Šต๋ฅ ์€ ์•ฝ 8.6\times10^{-5}, ๋ถ€๋ก F.2์˜ 50% ์กฐ๊ฑด์ธ 25k ์‹œ์ ์€ ์•ฝ 9.7\times10^{-5}(ํ•„์ž ๊ณ„์‚ฐ). ์‹คํ—˜ ์ด๋ฆ„์— โ€œ16H20โ€์ด ๋“ค์–ด ์žˆ๋‹ค. ๋ ˆํฌ์˜ ๊ณต๊ฐœ ์‚ฌ์ „ํ•™์Šต ๋Ÿฐ์ฒ˜(pretrain_small)๋Š” 150k step, 5\times10^{-5}\to3\times10^{-6}์œผ๋กœ ๋˜ ๋‹ค๋ฅด๋‹ค
์žฌ์ƒ˜ํ”Œ๋ง โ€œdataset-specific sampling ratiosโ€ (Table 8) ์ถœ์ฒ˜๋ณ„ ๋ณต์ œ ํšŸ์ˆ˜ = Sample-Scale์˜ ์˜ฌ๋ฆผ. 26๊ฐœ ์ถœ์ฒ˜๊ฐ€ ๋ชจ๋‘ ์„ค์ •์— ๋“ค์–ด ์žˆ๋‹ค
encoder ์„ธ๋ถ€ state encoder โ€œ3-layer ReLU MLPโ€, LN โ†’ area embedding ๋ง์…ˆ โ†’ MLP state MLP ํ™œ์„ฑํ™”๋Š” GELU. area embedding์„ ๋จผ์ € ๋”ํ•˜๊ณ  ๊ทธ ๋’ค LN โ†’ Linear โ†’ GELU โ†’ Linear(์ฒดํฌํฌ์ธํŠธ์˜ ํ•ด๋‹น LN ๊ฐ€์ค‘์น˜๋กœ๋„ ํ™•์ธ). LN์˜ ์•ž๋’ค ์œ„์น˜๋งŒ ๋‹ค๋ฅธ ๊ฒฝ๋ฏธํ•œ ์ฐจ์ด๋‹ค
๋ฒ ์ด์Šค๋ผ์ธยทprobing Tactile-VLA, NTP-1, ๊ถค์  ๋…ธ์ถœ \pi_{0.5}, ๋ถ€๋ก F.3 probe ๋ ˆํฌ์— ํ•ด๋‹น ๊ตฌํ˜„ยท์Šคํฌ๋ฆฝํŠธ๊ฐ€ ์—†๋‹ค
๋„๋ฉ”์ธ๋ณ„ gradient โ€œ๋„๋ฉ”์ธ ์ „์šฉ ๋ชจ๋“ˆ์€ ๋…๋ฆฝ ๊ฐฑ์‹ , ๊ณต์œ  ๋ชจ๋“ˆ์€ ๋ณ‘ํ•ฉ ํ›„ ๊ฐฑ์‹ โ€ (๋ถ€๋ก D) ์ „์šฉ ๊ตฌํ˜„ ์—†์ด rank๋ณ„๋กœ ํ•œ ๋„๋ฉ”์ธ ๋ฐฐ์น˜๋ฅผ ์ฃผ๊ณ  ํ‘œ์ค€ DDP(find_unused_parameters=True)๋กœ ํ‰๊ท ํ•œ๋‹ค. ๊ทธ ๋„๋ฉ”์ธ์„ ์•ˆ ์“ด rank์˜ 0 gradient๋„ ํ‰๊ท ์— ๋“ค์–ด๊ฐ€๋ฏ€๋กœ, ์„ผ์„œ๋ณ„ encoder์˜ ์‹คํšจ gradient๋Š” ๊ทธ ๋„๋ฉ”์ธ์„ ๋งก์€ rank ๋น„์œจ๋งŒํผ ์ค„์–ด๋“ ๋‹ค(ํ•„์ž ํ•ด์„)
EMA ์„œ์ˆ  ์—†์Œ ํ•™์Šต ์„ค์ •์— ema_decay 0.99๊ฐ€ ์žˆ์ง€๋งŒ PyTorch ํ•™์Šต ๊ฒฝ๋กœ๋Š” EMA๋ฅผ ์ง€์›ํ•˜์ง€ ์•Š๋Š”๋‹ค๋Š” ๋กœ๊ทธ๋งŒ ๋‚จ๊ธด๋‹ค. ๊ณต๊ฐœ ์ฒดํฌํฌ์ธํŠธ๋Š” EMA ๊ฐ€์ค‘์น˜๊ฐ€ ์•„๋‹Œ ๊ฒƒ์œผ๋กœ ์ฝํžŒ๋‹ค

๊ฐ€์žฅ ํฐ ๋ถˆ์ผ์น˜๋Š” tactile expert ํฌ๊ธฐ๋‹ค. ๋…ผ๋ฌธ์ด ์ ์€ ์‚ฌ์–‘(width 1024, MLP 4096)์€ openpi์˜ gemma_300m ์‚ฌ์–‘๊ณผ ๊ฐ™๊ณ , ์ธต๋‹น ์•ฝ 17.3M ร— 18์ธต โ‰ˆ 311M์œผ๋กœ โ€œ300Mโ€๊ณผ ๋งž๋Š”๋‹ค. ๊ณต๊ฐœ๋œ gemma_small์€ ์ธต๋‹น ์•ฝ 5.5M ร— 18์ธต โ‰ˆ 99M์ด๋‹ค(ํ•„์ž ๊ณ„์‚ฐ, ์ฒดํฌํฌ์ธํŠธ ํ—ค๋”์˜ 99.1M๊ณผ ์ผ์น˜). ๋…ผ๋ฌธ ์•ˆ์—์„œ๋„ ๋‘ ํฌ๊ธฐ๊ฐ€ ์„ž์—ฌ ์žˆ๋‹ค. ๋ถ€๋ก F.3์€ probing์— ์“ด โ€œFTP-1 tokenizer featureโ€๋ฅผ 512์ฐจ์›์ด๋ผ ์ ๋Š”๋ฐ, ์ฝ”๋“œ์—์„œ tokenizer ์ถœ๋ ฅ ํญ์€ tactile expert์˜ width๋ฅผ ๊ทธ๋Œ€๋กœ ๋”ฐ๋ฅธ๋‹ค. ์ฆ‰ F.3ยท๊ณต๊ฐœ ์ฒดํฌํฌ์ธํŠธยท์ฝ”๋“œ๋Š” width 512๋กœ ์ผ๊ด€๋˜๊ณ , 1024๋ฅผ ๋งํ•˜๋Š” ๊ณณ์€ ยง2.3๊ณผ ๋ถ€๋ก B.5๋ฟ์ด๋‹ค. ์ฝ”๋“œ์— ๋‚จ์€ ์“ฐ์ด์ง€ ์•Š๋Š” ํด๋ž˜์Šค์—๋„ token_dim = 1024 # follow gemma_300m config๋ผ๋Š” ์˜› ์„ค์ •์˜ ํ”์ ์ด ์žˆ๋‹ค. ๊ทธ๋Ÿฌ๋‹ˆ ๋…ผ๋ฌธ ์‹คํ—˜์ด 512 ํญ ๋ชจ๋ธ๋กœ ๋Œ์•˜๊ณ  ยง2.3ยทB.5๊ฐ€ ์˜› ์‚ฌ์–‘์„ ์ ์—ˆ์„ ๊ฐ€๋Šฅ์„ฑ์ด ๋†’์•„ ๋ณด์ด์ง€๋งŒ, ํ™•์ •ํ•  ๊ทผ๊ฑฐ๋Š” ์—†๋‹ค. ํ•™์Šต ์„ค์ •๋„ ๊ฐ™์€ ๋ฐฉํ–ฅ์„ ๊ฐ€๋ฆฌํ‚จ๋‹ค. ๊ณต๊ฐœ ์ฒดํฌํฌ์ธํŠธ๋Š” 400k step์œผ๋กœ ๊ณ„ํš๋œ ๋Ÿฐ์˜ 50k ์‹œ์  ์Šค๋ƒ…์ƒท์ด๊ณ , ๊ทธ ์‹œ์ ์˜ ํ•™์Šต๋ฅ ์€ ๋…ผ๋ฌธ์ด ์ ์€ โ€œ5\times10^{-5}๊นŒ์ง€ ๊ฐ์‡ โ€๊ฐ€ ์•„๋‹ˆ๋ผ ์•ฝ 8.6\times10^{-5}๋‹ค. ๊ณต๊ฐœ ์ฒดํฌํฌ์ธํŠธ๊ฐ€ ๋…ผ๋ฌธ ํ‘œ์˜ ๋ชจ๋ธ๊ณผ ๊ฐ™์€ ๋Ÿฐ์ธ์ง€, ๋…ผ๋ฌธ ์„œ์ˆ ์ด ๋‹ค๋ฅธ ๋Ÿฐ์„ ์„ค๋ช…ํ•˜๋Š”์ง€๋Š” ๊ณต๊ฐœ ์ž๋ฃŒ๋งŒ์œผ๋กœ ์ •ํ•  ์ˆ˜ ์—†๋‹ค. ์ฒดํฌํฌ์ธํŠธ ์ด๋ฆ„์˜ v0426, ์‹คํ—˜ ์ด๋ฆ„์˜ โ€œ16H20โ€(๋…ผ๋ฌธ์€ H20 48์žฅ)๋„ ์ •๋ฆฌ๊ฐ€ ํ•„์š”ํ•œ ์ง€์ ์ด๋‹ค. ์žฌํ˜„ํ•˜๋ ค๋Š” ์‚ฌ๋žŒ์—๊ฒŒ ์‹ค์šฉ์ ์ธ ๊ฒฐ๋ก ์€ ์ด๋ ‡๋‹ค. ๊ณต๊ฐœ ์ฒดํฌํฌ์ธํŠธ๋กœ ์–ป๋Š” ๊ฒฐ๊ณผ๋Š” โ€œ๋…ผ๋ฌธ์ด ์„œ์ˆ ํ•œ 300M tactile expert ๋ชจ๋ธโ€์˜ ๊ฒฐ๊ณผ๊ฐ€ ์•„๋‹ ์ˆ˜ ์žˆ๋‹ค.

์žฌ์ƒ˜ํ”Œ๋ง์€ ๋ฐ˜๋Œ€๋กœ ๋…ผ๋ฌธ๊ณผ ์ •ํ™•ํžˆ ๋งž๋Š”๋‹ค. ์ฒดํฌํฌ์ธํŠธ์˜ ๋ฐ์ดํ„ฐ ์„ค์ •์— Table 8์˜ 26๊ฐœ ์ถœ์ฒ˜๊ฐ€ ๋ชจ๋‘ ์žˆ๊ณ , ๋ณต์ œ ํšŸ์ˆ˜๊ฐ€ Sample-Scale์˜ ์˜ฌ๋ฆผ๊ณผ ์ผ์น˜ํ•œ๋‹ค. ์‹คํ—˜ ์ด๋ฆ„์— โ€œno_REASSEMBLEโ€์ด ๋“ค์–ด ์žˆ์ง€๋งŒ ์„ค์ •์—๋Š” REASSEMBLE๋„ ๋“ค์–ด ์žˆ๋‹ค.

์ž‘์€ ๊ด€์ฐฐ ํ•˜๋‚˜๋ฅผ ๋ง๋ถ™์ธ๋‹ค. ์ฒดํฌํฌ์ธํŠธ์˜ ์„ผ์„œ๋ณ„ encoder ๋ชฉ๋ก์—๋Š” LinkerHandL6Torque_state_6์ฒ˜๋Ÿผ Table 8์˜ ์„ผ์„œ ๋ชฉ๋ก(LinkerHand L6 Touch, array)์— ์—†๋Š” state encoder๊ฐ€ ์žˆ๋‹ค. ์ด๋ฆ„ ๊ทœ์•ฝ ์ฐจ์ด์ผ ์ˆ˜ ์žˆ์–ด ์—ฌ๊ธฐ์„œ๋Š” ๊ด€์ฐฐ๋กœ๋งŒ ๋‚จ๊ธด๋‹ค.

๋น„ํŒ์ ์œผ๋กœ ๋ณด๋ฉด

๊ฐ•์ 

  • ๋ฌธ์ œ ์„ค์ •์ด ์ •ํ™•ํ•˜๋‹ค. ์ด‰๊ฐ์ด generalist policy์—์„œ ๋น ์ง„ ์ด์œ ๋ฅผ โ€œ๋ฐ์ดํ„ฐ๊ฐ€ ์ ์–ด์„œโ€๊ฐ€ ์•„๋‹ˆ๋ผ โ€œ์„ผ์„œ ๊ฐ„ ์ธํ„ฐํŽ˜์ด์Šค๊ฐ€ ์—†์–ด์„œโ€๋กœ ์งš๊ณ , ๊ทธ ์ธํ„ฐํŽ˜์ด์Šค๋ฅผ ์† ํ•ด๋ถ€ํ•™ ๊ธฐ๋ฐ˜์˜ ๊ณ ์ • ์Šฌ๋กฏ์œผ๋กœ ์ค€ ๊ฒƒ์€ ๋‹จ์ˆœํ•˜๋ฉด์„œ ํ™•์žฅ ๊ฐ€๋Šฅํ•˜๋‹ค. ์ƒˆ ์„ผ์„œ๋ฅผ ๋ถ™์ด๋Š” ๋น„์šฉ์ด โ€œ์–ด๋А ์Šฌ๋กฏ์— ์–ด๋–ค encoderโ€ ์ •์˜๋กœ ์ค„์–ด๋“ ๋‹ค. FAAS๋กœ action ์ชฝ์„ ํ†ต์ผํ•œ ๊ฒƒ๊ณผ ๋Œ€์นญ์„ ์ด๋ฃจ๋Š” ์„ค๊ณ„๋‹ค.
  • ๋ฒ ์ด์Šค๋ผ์ธ์ด ์š”์ธ ๋ถ„๋ฆฌ๋ฅผ ์œ„ํ•ด ์งœ์—ฌ ์žˆ๋‹ค. \pi_{0.5}(์ด‰๊ฐ ์œ ๋ฌด), Tactile-VLA(์œตํ•ฉ ๊ตฌ์กฐ), FTP-\pi_{0.5}(์‚ฌ์ „ํ•™์Šต ์œ ๋ฌด)๊ฐ€ ๊ฐ๊ฐ ํ•œ ์š”์ธ์”ฉ ๋‹ค๋ฅด๊ณ , NTP-1๊ณผ ๊ถค์  ๋…ธ์ถœ ๋Œ€์กฐ๊ตฐ์ด โ€œ๋ฐ์ดํ„ฐ ๋ถ„ํฌ vs ์ด‰๊ฐ ์ง€์‹โ€ ๊ฐ€์„ค์„ ์ง์ ‘ ๊ฒจ๋ƒฅํ•œ๋‹ค. ์ด‰๊ฐ VLA ๋…ผ๋ฌธ์—์„œ ์ด ์ •๋„๋กœ ๋Œ€์กฐ๊ตฐ์„ ์ง  ์˜ˆ๋Š” ๋งŽ์ง€ ์•Š๋‹ค.
  • unseen ์„ผ์„œ ์‹คํ—˜์ด ์ด ๋ถ„์•ผ์—์„œ ์ƒˆ๋กญ๋‹ค. Table 5๊ฐ€ ๋ณด์—ฌ ์ฃผ๋“ฏ ๊ธฐ์กด ์ด‰๊ฐ ๋ชจ๋ธ์€ unseen ์„ผ์„œ์—์„œ์˜ policy ๋ฏธ์„ธ์กฐ์ •์„ ๋ณด๊ณ ํ•˜์ง€ ์•Š์•˜๋‹ค. ๋ฒ”์œ„๋Š” ์ข์ง€๋งŒ(๊ฐ™์€ modality ๊ณ„์—ด, ์…‹์—… 2๊ฐœ) ์ฒ˜์Œ์œผ๋กœ ๊ทธ ์งˆ๋ฌธ์„ policy ์ˆ˜์ค€์—์„œ ๋˜์กŒ๋‹ค.
  • ์ž๊ธฐ ๋ถˆ๋ฆฌํ•œ ๊ด€์ฐฐ์„ ์ˆจ๊ธฐ์ง€ ์•Š๋Š”๋‹ค. \pi_{0.5}๊ฐ€ ์ด‰๊ฐ ๋ฒ ์ด์Šค๋ผ์ธ๋“ค์„ ์ด๊ธด๋‹ค๋Š” ๊ฒƒ, lift ๊ณผ์ œ๋Š” ์ด‰๊ฐ์ด ํ•„์š” ์—†๋‹ค๋Š” ๊ฒƒ, MoE ์œตํ•ฉ๊ณผ Muon์ด ์‹ค๊ธฐ์—์„œ ๋“์ด ์•ˆ ๋๋‹ค๋Š” ๊ฒƒ์„ ์ง์ ‘ ์ ๋Š”๋‹ค. โ€œAvg. w/o Liftโ€๋ฅผ ๋ง๋ถ™์ธ ๊ฒƒ๋„ ๊ทธ๋Ÿฐ ํƒœ๋„๋‹ค.
  • ๊ณต๊ฐœ ๋ฒ”์œ„๊ฐ€ ๋„“๋‹ค. ์‚ฌ์ „ํ•™์Šต ์ฒดํฌํฌ์ธํŠธ, UniVTAC 6๊ณผ์ œ ๋ฏธ์„ธ์กฐ์ • ์ฒดํฌํฌ์ธํŠธ, ์‚ฌ์ „ํ•™์Šตยท๋ฏธ์„ธ์กฐ์ • ์ฝ”๋“œ, ๋ฐ์ดํ„ฐ ์ผ๋ถ€๊ฐ€ ๊ณต๊ฐœ๋๋‹ค. ์ด‰๊ฐ foundation policy๋ฅผ ๋‹ค๋ฅธ ๊ทธ๋ฃน์ด ์ถœ๋ฐœ์ ์œผ๋กœ ์“ธ ์ˆ˜ ์žˆ๋Š” ์ƒํƒœ๋‹ค.

์•ฝ์ ยทํ•œ๊ณ„

  • ์‹ค๊ธฐ ํ†ต๊ณ„๊ฐ€ ์–‡๋‹ค. ์‹ค๊ธฐ๋Š” ๋ชจ๋‘ ๊ณผ์ œ๋‹น 20ํšŒ, ํ•™์Šต 1ํšŒ๋‹ค. ๊ณผ์ œ ๋‹จ์œ„ ๋น„๊ต๋Š” ๋Œ€๋ถ€๋ถ„ p>0.1์ด๊ณ  ํ•™์Šต seed ๊ฐ„ ๋ถ„์‚ฐ์€ ๋ณด๊ณ ๋˜์ง€ ์•Š์•˜๋‹ค. ํ—ค๋“œ๋ผ์ธ์˜ +17.2%์™€ +31.6%๋Š” ๊ณผ์ œ ํ•ฉ์‚ฐ์—์„œ๋งŒ ์„ฑ๋ฆฝํ•˜๋Š” ์ฐจ์ด๋กœ ์ฝ์–ด์•ผ ํ•œ๋‹ค. ์‹œ๋ฎฌ๋ ˆ์ด์…˜(๊ณผ์ œ๋‹น 100ํšŒ)์—์„œ๋งŒ ๊ณผ์ œ ์ˆ˜์ค€์˜ ์ฐจ์ด๊ฐ€ ๋‹จ๋‹จํ•˜๋‹ค.
  • ํ—ค๋“œ๋ผ์ธ ์ˆซ์ž์™€ ๊ทธ๋ฆผ์— ์˜ค๊ธฐ๊ฐ€ ์žˆ๋‹ค. ์‹ค๊ธฐ \pi_{0.5} ํ‰๊ท ์€ ์นธ ๊ฐ’์œผ๋กœ 45.83%์ธ๋ฐ 45.3%๋กœ ์ ํ˜€ ์ดˆ๋ก์˜ +17.2%๊ฐ€ ๋‚˜์™”๋‹ค(์นธ ๊ธฐ์ค€ +16.7%p). Fig. 7์˜ FTP-\pi_{0.5} UniVTAC ๋ง‰๋Œ€(41.8)๋Š” Tactile-VLA ๊ฐ’์ด๋‹ค. UniVTAC-ACT* Insert Hole์€ ์› ๋…ผ๋ฌธ 24.0์ด 25๋กœ ์˜ฎ๊ฒจ์กŒ๋‹ค. ๊ฐ๊ฐ์€ ๊ฒฐ๋ก ์„ ๋ฐ”๊พธ์ง€ ์•Š์ง€๋งŒ, ๊ฐ™์€ ๋…ผ๋ฌธ ์•ˆ์—์„œ ์„ธ ๊ฑด์ด ๋‚˜์˜จ๋‹ค.
  • ์‹ค๊ธฐ seen ์…‹์—…์—์„œ โ€œ์ด‰๊ฐ ์‚ฌ์ „ํ•™์Šตโ€๊ณผ โ€œ๊ฐ™์€ ๋กœ๋ด‡ ๋ฐ์ดํ„ฐ ๋…ธ์ถœโ€์ด ๋ถ„๋ฆฌ๋˜์ง€ ์•Š๋Š”๋‹ค. Sharpa North-FTP-1(ํ˜ผํ•ฉ์˜ 9.0%)์€ ํ‰๊ฐ€ ๋กœ๋ด‡๊ณผ ๊ฐ™์€ ํ•˜๋“œ์›จ์–ด์—์„œ ์ˆ˜์ง‘๋๊ณ , ํ‰๊ฐ€ ๊ณผ์ œ์™€์˜ ์ค‘๋ณต ์—ฌ๋ถ€๋„ ์ ํ˜€ ์žˆ์ง€ ์•Š๋‹ค. ์ด๋ฅผ ๊ฐ€๋ฅด๋Š” ๊ถค์  ๋…ธ์ถœ ๋Œ€์กฐ๊ตฐ์€ UniVTAC์—์„œ๋งŒ ์žˆ๋‹ค. Table 2์˜ ์ด๋“ ์ค‘ ์–ด๋””๊นŒ์ง€๊ฐ€ ์ด‰๊ฐ ๋•๋ถ„์ธ์ง€๋Š” ์—ด๋ฆฐ ๋ฌธ์ œ๋‹ค.
  • โ€œunseenโ€์˜ ๋ฒ”์œ„๊ฐ€ ์ข๋‹ค. ๋‘ unseen ์„ผ์„œ๋Š” ์‚ฌ์ „ํ•™์Šต์— ์žˆ๋˜ modality ๊ณ„์—ด(image, array) ์•ˆ์˜ ์ƒˆ ํ•˜๋“œ์›จ์–ด๋‹ค. Xense๋Š” GelSight๋ฅ˜ ๊ด‘ํ•™ ์„ผ์„œ์—ฌ์„œ ๊ณต์œ  T3 trunk๊นŒ์ง€ ๊ทธ๋Œ€๋กœ ์“ด๋‹ค. ์ด๋ฒˆ ๊ฒฐ๊ณผ๊ฐ€ ๋งํ•˜๋Š” ๊ฒƒ์€ โ€œ๊ฐ™์€ ๊ณ„์—ด์˜ ์ƒˆ ์„ผ์„œโ€๋กœ์˜ ์ „์ด์ด๊ณ , ์ƒˆ modality(์˜ˆ: ์Œํ–ฅยท์˜จ๋„ยท๊ทผ์ ‘)๋‚˜ ์ƒˆ ์† ํ˜•ํƒœ๋กœ์˜ ์ „์ด๋Š” ๋‹ค๋ฃจ์ง€ ์•Š์•˜๋‹ค. ์ €์ž๋„ ยง4์—์„œ ์ด ๋ฒ”์œ„๋ฅผ ์ ๋Š”๋‹ค.
  • ํ‘œํ˜„ ์ˆ˜์ค€ ์ฆ๊ฑฐ๊ฐ€ ๋ณธ๋ฌธ ์š”์•ฝ๋ณด๋‹ค ์•ฝํ•˜๋‹ค. probing์—์„œ ํž˜ ์ถ”์ •์€ ๋‘ ๋ฒค์น˜๋งˆํฌ ๋ชจ๋‘ T3๋ณด๋‹ค ๊ฐ™๊ฑฐ๋‚˜ ๋‚˜์˜๊ณ (FeelAnyForce trunk +15%, tokenizer +55% MAE), ๋ณธ๋ฌธยท๋ถ€๋ก ์„œ์ˆ ์€ ๊ทธ ํ–‰์„ ์–ธ๊ธ‰ํ•˜์ง€ ์•Š๋Š”๋‹ค. ๊ทธ๋Ÿฐ๋ฐ ๊ฒฐ๊ณผ ์ ˆ์˜ ์ •์„ฑ ์„œ์ˆ (๋ˆ„๋ฅด๋Š” ํž˜ ์œ ์ง€, ์ ‘์ด‰ ์‹œ ๊ฐ์†)์€ ํž˜ ์ •๋ณด๋ฅผ ์“ฐ๋Š” ํ–‰๋™์„ ์ฃผ์žฅํ•œ๋‹ค. ๋‘ ์„œ์ˆ  ์‚ฌ์ด์˜ ๊ฐ„๊ทน์€ ํ•ด๋ช…๋˜์ง€ ์•Š์•˜๋‹ค.
  • โ€œ21๊ฐœ ์„ผ์„œโ€์™€ โ€œ3,000์‹œ๊ฐ„โ€์„ ๊ฒ€์‚ฐํ•  ์ˆ˜ ์—†๋‹ค. ์„ผ์„œ 9๊ฐœ๋Š” F/T์™€ ๊ด€์ ˆยท๊ทธ๋ฆฌํผ ํ† ํฌ๋ฅผ ํฌํ•จํ•œ stateํ˜•์ด๊ณ , ์ถœ์ฒ˜๋ณ„ ์‹œ๊ฐ„์ด ํ‘œ์— ์—†๋‹ค. Sample-Scale์˜ ๋œป์€ ๋…ผ๋ฌธ์— ์—†๊ณ  ์ฝ”๋“œ์—์„œ์•ผ ๋ณต์ œ ๋ฐฐ์ˆ˜์ž„์„ ์•Œ ์ˆ˜ ์žˆ๋‹ค. ๊ทธ๋ ‡๊ฒŒ ์—ญ์‚ฐํ•˜๋ฉด ์›๋ž˜ ๊ทœ๋ชจ์˜ ์•ฝ 63%๊ฐ€ ์‚ฌ๋žŒ ์žฅ๊ฐ‘ ๋ฐ์ดํ„ฐ์ธ๋ฐ(ํ•„์ž ์ถ”์ •), ๊ทธ ์‚ฌ๋žŒ ๋ฐ์ดํ„ฐ 5๊ฐœ ์ถœ์ฒ˜๋Š” ๊ณต๊ฐœ ๋ฐ์ดํ„ฐ์…‹์— ํ•˜๋‚˜๋„ ์—†๋‹ค. ๋ฐ์ดํ„ฐ ๊ทœ๋ชจ๊ฐ€ ์ด ๋…ผ๋ฌธ์˜ ๊ธฐ์—ฌ ์ค‘ ํ•˜๋‚˜์ธ๋ฐ ๊ทธ ๊ทœ๋ชจ๋ฅผ ๊ตฌ์„ฑ ์š”์†Œ๋กœ ๋˜์งš์„ ์ˆ˜ ์—†๋‹ค.
  • ๊ณต๊ฐœ ์ฒดํฌํฌ์ธํŠธ์™€ ๋…ผ๋ฌธ ์‚ฌ์–‘์ด ๋‹ค๋ฅด๋‹ค. tactile expert๊ฐ€ ๋…ผ๋ฌธ์€ 300M(width 1024), ๊ณต๊ฐœ ์ฒดํฌํฌ์ธํŠธ๋Š” ์•ฝ 99M(width 512)์ด๊ณ , ํ•™์Šต๋ฅ  ์Šค์ผ€์ค„๋„ ๋…ผ๋ฌธ ์„œ์ˆ ๊ณผ ๋‹ค๋ฅด๋‹ค. ์–ด๋А ์ชฝ์ด ๋…ผ๋ฌธ ํ‘œ์˜ ๋ชจ๋ธ์ธ์ง€ ๊ณต๊ฐœ ์ž๋ฃŒ๋งŒ์œผ๋กœ ์ •ํ•  ์ˆ˜ ์—†๋‹ค(์œ„ ๋Œ€์กฐ ์ ˆ). ๋ฒ ์ด์Šค๋ผ์ธ ์„ธ ์ข…๊ณผ probing ์ฝ”๋“œ๋„ ๋ ˆํฌ์— ์—†์–ด, ๋…ผ๋ฌธ์˜ ๋น„๊ต ์‹คํ—˜์€ ๊ณต๊ฐœ ์ž์‚ฐ์œผ๋กœ ๋‹ค์‹œ ๋Œ๋ฆด ์ˆ˜ ์—†๋‹ค.
  • UniVTAC ๊ณผ์ œ ์„ ํƒ์˜ ๊ทผ๊ฑฐ ํ•˜๋‚˜๊ฐ€ ์› ๋ฒค์น˜๋งˆํฌ์™€ ์–ด๊ธ‹๋‚œ๋‹ค. Insert HDMI๋ฅผ โ€œ์ด‰๊ฐ ํ”ผ๋“œ๋ฐฑ ํ‰๊ฐ€์— ๋ถ€์ ํ•ฉโ€ํ•˜๋‹ค๋ฉฐ ๋บ์ง€๋งŒ, ์› ๋…ผ๋ฌธ์—์„œ๋Š” ๊ทธ ๊ณผ์ œ์—์„œ ์ด‰๊ฐ encoder๊ฐ€ ACT๋ฅผ 15% โ†’ 28%๋กœ ์˜ฌ๋ ธ๋‹ค. ๋น ์ง„ ๊ณผ์ œ์˜ ๊ฒฐ๊ณผ๊ฐ€ ์žˆ์—ˆ๋‹ค๋ฉด ํ‘œ์˜ ํ•ด์„์— ๋„์›€์ด ๋์„ ๊ฒƒ์ด๋‹ค.
  • ์ €์ž๋“ค์ด ์ ์€ ํ•œ๊ณ„: FTP-1์€ ์ผ๋ฐ˜์ ์ธ ์ด‰๊ฐ ์ง€๊ฐ์— ์ดˆ์ ์„ ๋‘๊ณ  ์ด‰๊ฐยทํž˜ ๊ธฐ๋ฐ˜ servoing์ด๋‚˜ ์ œ์–ด๋Š” ๋‹ค๋ฃจ์ง€ ์•Š์œผ๋ฉฐ, ๋ฏธ๋ž˜ ์ด‰๊ฐ ์˜ˆ์ธก์ด๋‚˜ ์˜ˆ์ธก ๊ธฐ๋ฐ˜ ์ €์ˆ˜์ค€ ์ œ์–ด๋กœ์˜ ํ™•์žฅ, ๋” ํฌ๊ณ  ๋‹ค์–‘ํ•œ ๋ฐ์ดํ„ฐ ์ง‘์ ์ด ํ–ฅํ›„ ๊ณผ์ œ๋ผ๊ณ  ์“ด๋‹ค(ยง6). 50k step ์ดํ›„ ํฌํ™”๋„ ๋ฐ์ดํ„ฐ ๋‹ค์–‘์„ฑ ๋ถ€์กฑ๊ณผ \pi_{0.5} ์ง€์‹ ๋ณด์กด ์‚ฌ์ด์˜ trade-off๋กœ ์„ค๋ช…ํ•œ๋‹ค(๋ถ€๋ก D).

๊ด€๋ จ ์—ฐ๊ตฌ์™€์˜ ์ž๋ฆฌ ๋งค๊น€

์ด‰๊ฐ ํ‘œํ˜„ ์‚ฌ์ „ํ•™์Šต ์ชฝ์—์„œ ๋ณด๋ฉด, FTP-1์€ T3์˜ โ€œ์—ฌ๋Ÿฌ ์„ผ์„œ, ๊ณต์œ  trunkโ€ ์•„์ด๋””์–ด๋ฅผ ์ด๋ฏธ์ง€ ๊ฒฝ๋กœ์— ๊ทธ๋Œ€๋กœ ๋“ค์ด๊ณ  ๊ทธ ์œ„์— action์„ ๋‚ด๋Š” policy ์‚ฌ์ „ํ•™์Šต์„ ์–น์—ˆ๋‹ค. Sparsh ๋ฆฌ๋ทฐยทUniTouch ๋ฆฌ๋ทฐ๊ฐ€ ๋‹ค๋ฃฌ ํ‘œํ˜„ ํ•™์Šต์ด โ€œ์–ด๋–ค feature๊ฐ€ ์ข‹์€๊ฐ€โ€๋ฅผ probe๋กœ ์Ÿ€๋‹ค๋ฉด, FTP-1์€ โ€œ๋ฏธ์„ธ์กฐ์ • ์ถœ๋ฐœ์ ์œผ๋กœ์„œ policy ์ „์ฒด๊ฐ€ ์ข‹์€๊ฐ€โ€๋ฅผ ์„ฑ๊ณต๋ฅ ๋กœ ์žฐ๋‹ค. ๋‹ค๋งŒ probe๋กœ ๋‹ค์‹œ ์žฌ ๋ณด๋ฉด(Table 12) policy ์‚ฌ์ „ํ•™์Šต์ด ํž˜ ์ •๋ณด๋ฅผ ๋” ์ข‹๊ฒŒ ๋งŒ๋“ค์ง€๋Š” ์•Š์•˜๋‹ค. ์ด‰๊ฐ ์ด์งˆ์„ฑ์„ ๊ทธ๋ž˜ํ”„ ๊ตฌ์กฐ๋กœ ๋‹ค๋ฃฌ ์ด์งˆ์  ์ด‰๊ฐ GNN ๋ฆฌ๋ทฐ์™€ ๋น„๊ตํ•˜๋ฉด, FTP-1์€ ์„ผ์„œ ๋‚ด๋ถ€์˜ ๊ณต๊ฐ„ ๊ตฌ์กฐ๋Š” encoder์— ๋งก๊ธฐ๊ณ  ์„ผ์„œ ์‚ฌ์ด์˜ ์ •๋ ฌ๋งŒ ๊ณ ์ • ์Šฌ๋กฏ์œผ๋กœ ์ค€๋‹ค๋Š” ์ ์ด ๋‹ค๋ฅด๋‹ค.

์ด‰๊ฐ VLA ์ชฝ์—์„œ ๋ณด๋ฉด, Tactile-VLA์™€ TacFiLM์€ ์‚ฌ์ „ํ•™์Šต๋œ VLA์— ์ด‰๊ฐ์„ adapter๋‚˜ FiLM์œผ๋กœ ๋ผ์›Œ ๋„ฃ๋Š” ๊ณ„์—ด์ด๋‹ค. FTP-1์€ ์ด‰๊ฐ์— ๋…๋ฆฝ expert๋ฅผ ์ฃผ๊ณ , ๊ทธ expert ์ž์ฒด๋ฅผ ์ด์งˆ์  ๋ฐ์ดํ„ฐ๋กœ ์‚ฌ์ „ํ•™์Šตํ•œ๋‹ค. ๊ฐ™์€ ๊ทธ๋ฃน์˜ Tactile-VLA๋ฅผ ๋ฒ ์ด์Šค๋ผ์ธ์œผ๋กœ ๋„ฃ์–ด โ€œadapter ์ฃผ์ž…์€ ์‚ฌ์ „ํ•™์Šต ์—†์ด ์˜คํžˆ๋ ค ํ•ด๊ฐ€ ๋  ์ˆ˜ ์žˆ๋‹คโ€๋ฅผ ๋ณด์˜€๋‹ค๋Š” ์ ์—์„œ ๊ณ„์—ด ๋‚ด๋ถ€์˜ ์ž๊ธฐ ๋น„ํŒ์— ๊ฐ€๊น๋‹ค.

generalist policy ์ชฝ์—์„œ ๋ณด๋ฉด, FTP-1์€ \pi_{0.5} ๊ตฌ์กฐ๋ฅผ ๊ฑฐ์˜ ๊ทธ๋Œ€๋กœ ์“ฐ๊ณ , action ์ชฝ ํ†ต์ผ์€ UniDex์˜ FAAS๋ฅผ ๊ฐ€์ ธ์˜จ๋‹ค. ๋ฐ์ดํ„ฐ ์ธก๋ฉด์—์„œ Sharpa ํ”Œ๋žซํผ(๋ธ”๋กœ๊ทธ์˜ Sharpa MoDE-VLA ๋ฆฌ๋ทฐ๊ฐ€ ๊ฐ™์€ SharpaNorthยทSharpaWave ํ•˜๋“œ์›จ์–ด๋ฅผ ๋‹ค๋ฃฌ๋‹ค)์˜ ์ž์ฒด ์ˆ˜์ง‘ ๋ฐ์ดํ„ฐ๊ฐ€ ๊ฐ€์žฅ ํฐ ๋‹จ์ผ ์ถœ์ฒ˜๋ผ๋Š” ์ ์€, ์ด ๋ชจ๋ธ์ด ๊ทธ ํ”Œ๋žซํผ์—์„œ ํŠนํžˆ ์ž˜ ์ž‘๋™ํ•  ์ด์œ ์ด๊ธฐ๋„ ํ•˜๋‹ค.

ํ‰๊ฐ€ ๋ฒค์น˜๋งˆํฌ๋กœ ์“ด UniVTAC์€ UniVTAC ๋ฆฌ๋ทฐ์— ์ •๋ฆฌ๋ผ ์žˆ๊ณ , ๊ทธ ๋ฆฌ๋ทฐ์— ๊ธฐ๋กํ•œ ์› ๋…ผ๋ฌธ ํ‘œ๊ฐ€ ์ด๋ฒˆ Table 1์˜ *ํ–‰ ๋Œ€์กฐ์™€ Insert HDMI ๋ฐฐ์ œ ๊ทผ๊ฑฐ์˜ ๊ฒ€ํ† ์— ์“ฐ์˜€๋‹ค.

์š”์•ฝ

FTP-1์€ ์ด‰๊ฐ ์‹ ํ˜ธ์˜ ์ด์งˆ์„ฑ์„ ์†์˜ ๊ธฐ๋Šฅ ์˜์—ญ 24์นธ์ด๋ผ๋Š” ๊ณตํ†ต ํ† ํฐ ๊ณต๊ฐ„(MTTS)์œผ๋กœ ํก์ˆ˜ํ•˜๊ณ , ์„ผ์„œ๋ณ„ ์ž‘์€ encoder์™€ ์„ผ์„œ ๋ฌด๊ด€ํ•œ tactile expert๋ฅผ \pi_{0.5} ์œ„์— ์–น์–ด 26๊ฐœ ์ถœ์ฒ˜ยท21๊ฐœ ์„ผ์„œ์˜ ๋ฐ์ดํ„ฐ๋กœ ์‚ฌ์ „ํ•™์Šตํ•œ ์ฒซ ์ด‰๊ฐ foundation policy๋‹ค. ๋‹ค์„ฏ ์…‹์—…์—์„œ seen ์„ผ์„œ UniVTAC 66.66%(์ฐจ์ˆœ์œ„ 49.16%), ์‹ค๊ธฐ 62.5%, unseen ์„ผ์„œ 46.6%(๋ฒ ์ด์Šค๋ผ์ธ 15.0%)๋ฅผ ๋ณด๊ณ ํ•˜๊ณ , NTP-1๊ณผ ๊ถค์  ๋…ธ์ถœ ๋Œ€์กฐ๊ตฐ์œผ๋กœ ์ด๋“์ด ์ด‰๊ฐ ์‚ฌ์ „ํ•™์Šต์—์„œ ์˜จ๋‹ค๋Š” ๊ฒƒ์„ ๋ณด์ด๋ ค ํ–ˆ๋‹ค. NTP-1 ๊ฒฐ๊ณผ๋ฅผ lift ์ œ์™ธ ์ง€ํ‘œ๋กœ ๋‹ค์‹œ ์ฝ์œผ๋ฉด ๊ทธ ์ฃผ์žฅ์€ ์ €์ž ์„œ์ˆ ๋ณด๋‹ค ์˜คํžˆ๋ ค ๊ฐ•ํ•ด์ง„๋‹ค.

ํ•œ๊ณ„๋Š” ์ฆ๊ฑฐ์˜ ๋‘๊ป˜์™€ ์ •ํ•ฉ์„ฑ์— ์žˆ๋‹ค. ์‹ค๊ธฐ๋Š” ๊ณผ์ œ๋‹น 20ํšŒยทํ•™์Šต 1ํšŒ์—ฌ์„œ ๊ณผ์ œ ๋‹จ์œ„๋กœ๋Š” ๋Œ€๋ถ€๋ถ„ ๊ตฌ๋ถ„๋˜์ง€ ์•Š๊ณ , ์‹ค๊ธฐ seen ์…‹์—…์—์„œ๋Š” ๊ฐ™์€ ๋กœ๋ด‡ ๋ฐ์ดํ„ฐ ๋…ธ์ถœ๊ณผ ์ด‰๊ฐ ์‚ฌ์ „ํ•™์Šต์ด ๋ถ„๋ฆฌ๋˜์ง€ ์•Š๋Š”๋‹ค. ์ดˆ๋ก์˜ +17.2%๋Š” ํ‘œ์˜ ํ‰๊ท  ์˜ค๊ธฐ์—์„œ ๋‚˜์™”๊ณ (์นธ ๊ธฐ์ค€ +16.7%p), Fig. 7๊ณผ Table 1์—๋„ ์˜ค๊ธฐ๊ฐ€ ์žˆ๋‹ค. probing์€ ํž˜ ์ •๋ณด๊ฐ€ ๊ฐœ์„ ๋˜์ง€ ์•Š์•˜์Œ์„ ๋ณด์—ฌ ์ฃผ๋Š”๋ฐ ๋ณธ๋ฌธ์€ ๊ทธ ํ–‰์„ ์š”์•ฝ์—์„œ ๋บ๋‹ค. ๊ณต๊ฐœ ์ฒดํฌํฌ์ธํŠธ์˜ tactile expert๋Š” ๋…ผ๋ฌธ์ด ์ ์€ 300M์ด ์•„๋‹ˆ๋ผ ์•ฝ 99M์ด๊ณ , ํ•™์Šต๋ฅ  ์Šค์ผ€์ค„๋„ ๋…ผ๋ฌธ ์„œ์ˆ ๊ณผ ๋‹ค๋ฅด๋‹ค. ๋‹ค์Œ์œผ๋กœ ๋ณด๊ณ  ์‹ถ์€ ๊ฒƒ์€ ์‹ค๊ธฐ ์…‹์—…์˜ ๋ฐ˜๋ณต ํ•™์Šต ๋ถ„์‚ฐ, Sharpa ์…‹์—…์—์„œ์˜ ๊ถค์  ๋…ธ์ถœ ๋Œ€์กฐ๊ตฐ, ์ƒˆ modality ๊ณ„์—ด ์„ผ์„œ๋กœ์˜ ์ „์ด, ๊ทธ๋ฆฌ๊ณ  ๋…ผ๋ฌธ ํ‘œ์™€ ๊ณต๊ฐœ ์ฒดํฌํฌ์ธํŠธ์˜ ๋Œ€์‘ ๊ด€๊ณ„๋‹ค.

์ด ๊ธ€์— ๋Œ€ํ•ด

  • ์ฝ์€ ๋ฒ”์œ„: arXiv v4(2026-10-27, 29์ชฝ)์˜ HTML๊ณผ PDF, ํ”„๋กœ์ ํŠธ ํŽ˜์ด์ง€ ๋ณธ๋ฌธ๊ณผ ์˜์ƒ, GitHub README์™€ ํ•ต์‹ฌ ๋ชจ๋ธ ํŒŒ์ผ(attention mask, model config, gemma ๋ณ€ํ˜• ์ •์˜, ํ•™์Šต ์Šคํฌ๋ฆฝํŠธ, UniVTAC ๋ฏธ์„ธ์กฐ์ • ์˜ˆ์‹œ), Hugging Face ์‚ฌ์ „ํ•™์ŠตยทUniVTAC ๋ฏธ์„ธ์กฐ์ • ์ฒดํฌํฌ์ธํŠธ์˜ ํŒŒ์ผ ๋ชฉ๋กยทconfig(๋ชจ๋ธยทํ•™์Šตยท๋ฐ์ดํ„ฐ ์„ค์ •)ยทsafetensors ํ—ค๋”, ModelScopeยทHugging Face ๋ฐ์ดํ„ฐ์…‹ ์นด๋“œ๋ฅผ ์ฝ์—ˆ๋‹ค. UniVTAC ์› ๋…ผ๋ฌธ(arXiv:2602.10093)์˜ Table I์„ ๋Œ€์กฐํ–ˆ๋‹ค.
  • ๊ทธ๋ฆผ: Fig. 1~16์€ arXiv HTML์˜ ์›๋ณธ ๊ทธ๋ฆผ์ด๋‹ค. ๊ทธ๋ฆผ ๋ฒˆํ˜ธ๋Š” ๋…ผ๋ฌธ ๊ธฐ์ค€์ด๋‹ค. ์˜์ƒ์€ ํ”„๋กœ์ ํŠธ ํŽ˜์ด์ง€์˜ mp4๋ฅผ ๊ทธ๋Œ€๋กœ ๋ถˆ๋Ÿฌ์˜จ๋‹ค.
  • ๊ณต๊ฐœ ์ƒํƒœ: ์ฝ”๋“œ๋Š” Apache-2.0(openpi ๊ธฐ๋ฐ˜, Gemma ๋ผ์ด์„ ์Šค ํŒŒ์ผ ๋™๋ด‰)์ด๋‹ค. ์‚ฌ์ „ํ•™์Šต ์ฒดํฌํฌ์ธํŠธ ftp1_v0426_50kstep๊ณผ UniVTAC 6๊ณผ์ œ ๋ฏธ์„ธ์กฐ์ • ์ฒดํฌํฌ์ธํŠธ๊ฐ€ Hugging Face์— ์žˆ๊ณ , ๋ชจ๋ธ ์นด๋“œ์—๋Š” ๋ผ์ด์„ ์Šค ํ‘œ๊ธฐ๊ฐ€ ์—†๋‹ค. ์ฒดํฌํฌ์ธํŠธ๋Š” PaliGemma ๊ฐ€์ค‘์น˜๋ฅผ ํฌํ•จํ•˜๋ฏ€๋กœ ๋ ˆํฌ์— ๋™๋ด‰๋œ Gemma ์ด์šฉ ์•ฝ๊ด€์˜ Model Derivative ๋ฐฐํฌ ์กฐ๊ฑด์ด ๊ฑธ๋ฆด ์ˆ˜ ์žˆ๋‹ค(์•ฝ๊ด€ ํ•ด์„์€ ํ•„์ž ํŒ๋‹จ์ด๋‹ค). ๋ฐ์ดํ„ฐ์…‹์€ ModelScope(MIT ํ‘œ๊ธฐ)์™€ Hugging Face์— 15๊ฐœ ์ถœ์ฒ˜ ๋ฌถ์Œ(FreeTacMan, MotionTrans, RDP, RDP-Bimanual, RH20T Cfg5ยท6ยท7, Unit, Unit-Bimanual, VLA-Touch, ViTaMIn, VTouch D-WHEELยทQINGLONG, exUMI, Sharpa)์ด ๊ณต๊ฐœ๋๊ณ , ์‚ฌ๋žŒ ์† ๋ฐ์ดํ„ฐ 5๊ฐœ ์ถœ์ฒ˜ ์ „๋ถ€์™€ HumanoidEverydayยทOpenLETDexยทDexUMI 2์ข…ยทTouchInTheWildยทREASSEMBLE์˜ 11๊ฐœ ์ถœ์ฒ˜๋Š” ์ด ๋ฆด๋ฆฌ์Šค์— ์—†๋‹ค. ์ด 11๊ฐœ๋Š” ์žฌ์ƒ˜ํ”Œ๋ง ํ›„ ํ•™์Šต ํ˜ผํ•ฉ์˜ 44.6%๋ฅผ ์ฐจ์ง€ํ•œ๋‹ค(Table 8๋กœ ํ•„์ž ๊ณ„์‚ฐ). ์‹ค๊ธฐ ์…‹์—…์˜ ๋ฏธ์„ธ์กฐ์ • ์ฒดํฌํฌ์ธํŠธ์™€ ๋ฒ ์ด์Šค๋ผ์ธ(Tactile-VLA, NTP, ๊ถค์  ๋…ธ์ถœ \pi_{0.5}) ์ฒดํฌํฌ์ธํŠธ๋Š” ๊ณต๊ฐœ ๋ชฉ๋ก์— ์—†๋‹ค. ๋…ผ๋ฌธ v4 ๋ณธ๋ฌธ์€ ์—ฌ์ „ํžˆ โ€œwill be released in the futureโ€๋ผ๊ณ  ์ ๋Š”๋‹ค.
  • ํ‘œ๊ธฐ: โ€œํ•„์ž ๊ณ„์‚ฐยท๊ฒ€์‚ฐโ€์€ ๋…ผ๋ฌธ ํ‘œ์˜ ๊ฐ’์œผ๋กœ ์ง์ ‘ ๊ณ„์‚ฐํ•œ ๊ฐ’(ํ‰๊ท  ์žฌ๊ณ„์‚ฐ, Fisher ์ •ํ™•๊ฒ€์ •, ๋น„์ค‘ ํ•ฉ, ํŒŒ๋ผ๋ฏธํ„ฐ ์ˆ˜)์ด๋‹ค. Fisher ๊ฒ€์ •์€ rollout์„ ๋…๋ฆฝ ์‹œํ–‰์œผ๋กœ ๋ณธ ๊ฒƒ์ด๋ฉฐ ํ•™์Šต seed ๊ฐ„ ๋ถ„์‚ฐ์€ ๋ฐ˜์˜ํ•˜์ง€ ์•Š๋Š”๋‹ค. ๊ณต๊ฐœ ์ฝ”๋“œยท์ฒดํฌํฌ์ธํŠธ ๋Œ€์กฐ๋Š” ํŒŒ์ผ๊ณผ ํ…์„œ shape๋ฅผ ์ฝ์€ ์ •์  ํ™•์ธ์ด๋‹ค. ์ด ๋ฆฌ๋ทฐ์˜ ๋ชจ๋“  ์„ฑ๊ณต๋ฅ ์€ ๋…ผ๋ฌธ์ด ๋ณด๊ณ ํ•œ ๊ฐ’์ด๋ฉฐ ์žฌํ˜„ ํ™•์ธ์€ ํ•˜์ง€ ์•Š์•˜๋‹ค.

Copyright 2026, JungYeon Lee