Curieux.JY
  • JungYeon Lee
  • Post
  • ๐Ÿ•ธ๏ธ Graph
  • Lecture
  • Note

On this page

  • ๐Ÿ” Ping Review
    • Figure 1 ์ž์„ธํžˆ ๋ณด๊ธฐ: VLA ยท WAM ยท JAM์„ ๊ฐ€๋ฅด๋Š” ์„ธ ๊ฐ€์ง€ ์„ ํƒ
      • ๊ณตํ†ต ๋ถ€๋ถ„: ์˜ค๋ฅธ์ชฝ ์—ด์˜ ํ–‰๋™ ๊ฒฝ๋กœ
      • ํŒจ๋„ (a) Vision-language-action model
      • ํŒจ๋„ (b) World-action model
      • ํŒจ๋„ (c) Just action model (JAM)
      • ์„ธ ์ถ•์œผ๋กœ ์ •๋ฆฌํ•˜๋ฉด
      • ๊ธฐํ˜ธ ์ •๋ฆฌ
      • ๊ทธ๋ฆผ์ด ๋ณด์—ฌ ์ฃผ์ง€ ์•Š๋Š” ๊ฒƒ
  • ๐Ÿ”” Ring Review
    • ํ•œ ์ค„๋กœ ์‹œ์ž‘ํ•˜๋ฉด
    • ๋ฐฐ๊ฒฝ: ์„ธ ์„ ํƒ์ด ํ•œ ๋ฌถ์Œ์œผ๋กœ ๋น„๊ต๋ผ ์™”๋‹ค (ยง1ยทยง2)
    • ๋ฐฉ๋ฒ• ์ƒ์„ธ 1: ๋ฌธ์ œ ์„ค์ • (ยง3.1)
    • ๋ฐฉ๋ฒ• ์ƒ์„ธ 2: Foresight supervision (ยง3.2, ๋ถ€๋ก A.3)
    • ๋ฐฉ๋ฒ• ์ƒ์„ธ 3: ์„ธ ์‚ฌ์ „ํ•™์Šต ๊ณ„์—ด๋กœ์˜ instantiation (ยง3.3, ๋ถ€๋ก A.2)
    • ์ง๊ด€: ์‹ค์ œ๋กœ ๋ฐ”๋€ ๊ฒƒ์€ ๋ฌด์—‡์ธ๊ฐ€
    • ์‹คํ—˜ ์„ค์ • (ยง4.1, ๋ถ€๋ก A.1)
    • ์‹คํ—˜ 1: ์ƒ์„ฑ์„ foresight๋กœ ๋ฐ”๊พธ๊ธฐ (ยง4.2, Table 1, Fig. 3ยท4ยท9)
    • ์‹คํ—˜ 2: ๋ฐฑ๋ณธ์„ ๋ฐ”๊พธ๊ธฐ (ยง4.3, Table 2ยท3ยท14, Fig. 5ยท8)
    • ์‹คํ—˜ 3: ๊ณต๊ฐœ ์‹œ์Šคํ…œ๊ณผ์˜ ๋น„๊ต (ยง4.4, Table 4ยท5ยท6ยท18ยท19, Fig. 6ยท7)
    • ์‹คํ—˜ 4: ํ•™์Šต ๋น„์šฉ (ยง4.5, ๋ถ€๋ก B, Table 9)
    • ์‹คํ—˜ 5: ๋ถ€๋ก์˜ ์ ˆ์ œ์™€ ํ†ต๊ณ„ (๋ถ€๋ก CยทD.7ยทD.8)
    • ๋น„ํŒ์ ์œผ๋กœ ๋ณด๋ฉด
      • ๊ฐ•์ 
      • ์•ฝ์ ยทํ•œ๊ณ„
    • ๊ด€๋ จ ์—ฐ๊ตฌ์™€์˜ ์ž๋ฆฌ ๋งค๊น€
    • ์ฝ”๋“œ ๊ณต๊ฐœ ์ƒํƒœ
    • ์š”์•ฝ
    • ์ด ๊ธ€์— ๋Œ€ํ•ด

๐Ÿ“ƒJAM ๋ฆฌ๋ทฐ

vla
world-model
manipulation
representation-learning
efficiency
Vision-Language-Action or World-Action? Just Action Models
Published

November 18, 2026

  • Paper Link (arXiv:2610.37219)

  • Code Link (github.com/y0urOy/JAM)

  • Project

  • Model (Hugging Face y0urOy/JAM)

  • Yonghao Yu, Lang Huang, Runyi Li, Toshihiko Yamasaki (The University of Tokyo ยท National Institute of Informatics)

  • arXiv preprint, 2026

  1. ๐Ÿ’ก VLA์™€ World-Action Model(WAM)์˜ ์šฐ์—ด ๋…ผ์Ÿ์€ ์‚ฌ์‹ค ๋ฐฑ๋ณธ ์ดˆ๊ธฐํ™” ยท ๋ฏธ๋ž˜ ๊ด€์ธก์— ๊ฑฐ๋Š” ๋ชฉ์ ํ•จ์ˆ˜ ยท ์ถ”๋ก  ๋•Œ ๋‚จ๊ธฐ๋Š” ๋ชจ๋“ˆ์ด๋ผ๋Š” ์„ธ ์„ ํƒ์„ ํ•œ๋ฐ ๋ฌถ์–ด ๋น„๊ตํ•œ ๊ฒƒ์ด๋ผ๊ณ  ์ง€์ ํ•˜๊ณ , ์ด๋ฅผ ํ’€์–ด ์“ด ๊ณตํ†ต ๋ ˆ์‹œํ”ผ JAM์„ ์ œ์•ˆํ•œ๋‹ค.
  2. โš™๏ธ ํ˜„์žฌ ํ”„๋ ˆ์ž„์˜ ๋ฐฑ๋ณธ feature๊ฐ€ ๊ณ ์ •๋œ future anchor๊ฐ€ ๋งŒ๋“  ๊ธฐ๋ก๋œ ๋ฏธ๋ž˜์˜ feature๋ฅผ cosine ๊ฑฐ๋ฆฌ๋กœ ๋งžํžˆ๋„๋ก ํ•™์Šต ์ „์šฉ prediction head๋ฅผ ๋ถ™์ด๊ณ (foresight loss \mathcal L_F), ํ•™์Šต์ด ๋๋‚˜๋ฉด ๊ทธ ๊ฐ€์ง€๋ฅผ ๋–ผ์–ด ๋‚ด ๋ฐฐํฌ ๊ฒฝ๋กœ๋Š” ๋ฐฑ๋ณธ + action expert ๊ทธ๋Œ€๋กœ ๋‘”๋‹ค. ์ด๋ฏธ์ง€ยท๋น„๋””์˜ค ์ƒ์„ฑ์€ ํ•™์Šต์—๋„ ์ถ”๋ก ์—๋„ ์—†๋‹ค.
  3. ๐ŸŽฏ ์•„ํ‚คํ…์ฒ˜ยท๋ฐ์ดํ„ฐยท์˜ˆ์‚ฐ์„ ๊ณ ์ •ํ•œ ๋น„๊ต์—์„œ LIBERO-Plus ์„ฑ๊ณต๋ฅ ์ด FastWAM 55.31โ†’64.54%, ImageWAM 65.27โ†’79.95%๋กœ ์˜ค๋ฅธ๋‹ค. ํ•™์Šต ๋น„์šฉ์€ ๋…ผ๋ฌธ v1์ด โ€œtarget์„ ์บ์‹œํ•œ JAM vs ์บ์‹œํ•˜์ง€ ์•Š์€ FastWAMโ€ ์กฐ๊ฑด์œผ๋กœ 73.0% ๊ฐ์†Œ๋ฅผ ์ฃผ์žฅํ•˜๊ณ , ๊ณต๊ฐœ ๋ ˆํฌ๋Š” ์ดํ›„ ์–‘์ชฝ ๋ชจ๋‘ ์บ์‹œํ•œ ์กฐ๊ฑด์˜ 57.7%๋กœ ํ‘œ๊ธฐ๋ฅผ ๋ฐ”๊ฟจ๋‹ค. FastWAM ์„ฑ๊ณต๋ฅ  ๊ธฐ์ค€๊ฐ’์€ ์ €์ž๋“ค์ด ๋‹ค์‹œ ํ•™์Šตยทํ‰๊ฐ€ํ•œ ๊ฐ’์ด๊ณ , Fast-WAM ๋…ผ๋ฌธ ์ž์ฒด์˜ ๋ณด๊ณ ๊ฐ’๊ณผ๋Š” ์ฐจ์ด๊ฐ€ ์žˆ๋‹ค.

๐Ÿ” Ping Review

๐Ÿ” Ping โ€” A light tap on the surface. Get the gist in seconds.

๋กœ๋ด‡ ์ •์ฑ…์„ ์‚ฌ์ „ํ•™์Šต ๋ชจ๋ธ์—์„œ ์ถœ๋ฐœ์‹œํ‚ค๋Š” ๋ฐฉ๋ฒ•์€ ํฌ๊ฒŒ ๋‘ ์ง„์˜์œผ๋กœ ๊ฐˆ๋ ค ์žˆ๋‹ค. VLA๋Š” vision-language model(VLM)์˜ ํ‘œํ˜„์„ ๋ฌผ๋ ค๋ฐ›์•„ ํ–‰๋™๋งŒ ํ•™์Šตํ•˜๊ณ , WAM์€ ์ด๋ฏธ์ง€ยท๋น„๋””์˜ค ์ƒ์„ฑ๊ธฐ์—์„œ ์ถœ๋ฐœํ•ด ํ–‰๋™๊ณผ ํ•จ๊ป˜ ๋‹ค์Œ์— ์˜ฌ ๊ด€์ธก์„ ์ƒ์„ฑํ•˜๋„๋ก ํ•™์Šตํ•œ๋‹ค. ๋‘ ์ง„์˜์˜ ์„ฑ๊ณต๋ฅ ์„ ๋งž๋Œ€๋Š” ๋…ผ์Ÿ์ด ์ด์–ด์ ธ ์™”๋Š”๋ฐ, ์ด ๋…ผ๋ฌธ์€ ๊ทธ ๋น„๊ต๊ฐ€ ์• ์ดˆ์— ๋ฌด์—‡์„ ๋น„๊ตํ•˜๋Š”์ง€ ๋ถˆ๋ถ„๋ช…ํ•˜๋‹ค๊ณ  ๋ณธ๋‹ค. ๋‘ ๊ณ„์—ด์€ ๋ฐฑ๋ณธ๋„ ๋‹ค๋ฅด๊ณ , ๋ฏธ๋ž˜ ๊ด€์ธก์„ ์“ฐ๋Š” ๋ฐฉ์‹๋„ ๋‹ค๋ฅด๊ณ , ์ถ”๋ก  ๋•Œ ๋‚จ๊ธฐ๋Š” ๋ชจ๋“ˆ๋„ ๋‹ค๋ฅด๋‹ค. ์„ฑ๊ณต๋ฅ  ์ฐจ์ด๋ฅผ ์ด ์…‹ ์ค‘ ์–ด๋””์— ๊ท€์†์‹œํ‚ฌ์ง€ ์ •ํ•  ์ˆ˜ ์—†๋‹ค.

JAM(Just Action Model)์€ ์ด ์…‹์„ ๋–ผ์–ด ๋†“์€ ๊ณตํ†ต ํ‹€์ด๋‹ค. ๋ฐฑ๋ณธ์€ VLM์ด๋“  ์ƒ์„ฑ๊ธฐ๋“  self-supervised ์ธ์ฝ”๋”๋“  ์ƒ๊ด€์—†๊ณ , ๋ฏธ๋ž˜ ๊ด€์ธก์€ ์ƒ์„ฑ ๋ชฉํ‘œ๊ฐ€ ์•„๋‹ˆ๋ผ ๊ณ ์ •๋œ feature target์œผ๋กœ๋งŒ ์“ฐ๋ฉฐ, ๊ทธ ๊ฐ๋… ๊ฐ€์ง€๋Š” ํ•™์Šต ํ›„ ๋ฒ„๋ฆฐ๋‹ค. ์ด๋ ‡๊ฒŒ ํ•˜๋ฉด FastWAMยทImageWAM ๊ฐ™์€ ๊ธฐ์กด WAM์˜ ์•„ํ‚คํ…์ฒ˜์™€ ๋ฐฐํฌ ๊ฒฝ๋กœ๋ฅผ ๊ทธ๋Œ€๋กœ ๋‘” ์ฑ„ ๋ชฉ์ ํ•จ์ˆ˜๋งŒ ๋ฐ”๊ฟ” ๋น„๊ตํ•  ์ˆ˜ ์žˆ๊ณ , ๊ฐ™์€ ๋ ˆ์‹œํ”ผ๋ฅผ DINOv2๋‚˜ Qwen3.5 ๊ฐ™์€ ๋น„์ƒ์„ฑ ๋ฐฑ๋ณธ์—๋„ ๊ทธ๋Œ€๋กœ ์ ์šฉํ•  ์ˆ˜ ์žˆ๋‹ค.

Figure 1 ์ž์„ธํžˆ ๋ณด๊ธฐ: VLA ยท WAM ยท JAM์„ ๊ฐ€๋ฅด๋Š” ์„ธ ๊ฐ€์ง€ ์„ ํƒ


Fig. 1 โ€” (a) VLA, (b) WAM, (c) JAM. ๋ฐฑ๋ณธ ์ดˆ๊ธฐํ™”, ๊ธฐ๋ก๋œ ๋ฏธ๋ž˜ ๊ด€์ธก์— ๊ฑฐ๋Š” ๋ชฉ์ ํ•จ์ˆ˜, ์ถ”๋ก  ๋•Œ ๋‚จ๋Š” ๋ชจ๋“ˆ์ด๋ผ๋Š” ์„ธ ์„ ํƒ์œผ๋กœ ์„ธ ๊ณ„์—ด์„ ๋‚˜๋ž€ํžˆ ๋†“์€ ์„ค๊ณ„ ๊ณต๊ฐ„ ๊ทธ๋ฆผ.

์ด ๊ทธ๋ฆผ์€ ๋…ผ๋ฌธ ์ „์ฒด์˜ ๋…ผ์ง€๋ฅผ ํ•œ ์žฅ์— ๋‹ด๋Š”๋‹ค. ์‹คํ—˜์€ ์ „๋ถ€ ์ด ๊ทธ๋ฆผ์˜ ์นธ์„ ํ•˜๋‚˜์”ฉ ๋ฐ”๊ฟ” ๋ผ์šฐ๋Š” ๋ฐฉ์‹์œผ๋กœ ์งœ์—ฌ ์žˆ์œผ๋ฏ€๋กœ, ๊ทธ๋ฆผ์„ ์ •ํ™•ํžˆ ์ฝ์–ด ๋‘๋ฉด ๋’ค์˜ ํ‘œ๊ฐ€ ์‰ฝ๊ฒŒ ์ฝํžŒ๋‹ค. ์„ธ ํŒจ๋„์€ ๊ฐ™์€ ๋ฌธ๋ฒ•์œผ๋กœ ๊ทธ๋ ค์ ธ ์žˆ๋‹ค. ์•„๋ž˜์ชฝ์ด ์ž…๋ ฅ๊ณผ ๋ฐฑ๋ณธ, ์˜ค๋ฅธ์ชฝ ์—ด์ด ํ–‰๋™ ๊ฒฝ๋กœ, ์œ„์ชฝ์ด ์†์‹ค ํ•จ์ˆ˜(์œก๊ฐํ˜•)๋‹ค. ๋ชจ๋“  ํŒจ๋„์˜ ์˜ค๋ฅธ์ชฝ ์—ด์€ ๋™์ผํ•˜๋‹ค. ์ด๊ฒƒ์ด ๊ทธ๋ฆผ์ด ๋จผ์ € ๋งํ•˜๋Š” ๊ฒƒ์ด๋‹ค. ํ–‰๋™์„ ๋‚ด๋Š” ๋ฐฉ์‹์€ ์„ธ ๊ณ„์—ด์ด ๊ฐ™๊ณ , ๋‹ค๋ฅธ ๊ฒƒ์€ ์™ผ์ชฝ๊ณผ ์œ„์ชฝ๋ฟ์ด๋‹ค.

๊ณตํ†ต ๋ถ€๋ถ„: ์˜ค๋ฅธ์ชฝ ์—ด์˜ ํ–‰๋™ ๊ฒฝ๋กœ

์„ธ ํŒจ๋„ ๋ชจ๋‘ ์˜ค๋ฅธ์ชฝ ์•„๋ž˜์— ์ ์„  ํ…Œ๋‘๋ฆฌ์˜ \tilde a_{1:H}๊ฐ€ ์žˆ๊ณ , ์ด๊ฒƒ์ด Action Expert๋กœ ๋“ค์–ด๊ฐ€ ์‹ค์„  ํ…Œ๋‘๋ฆฌ์˜ \hat a_{1:H}๊ฐ€ ๋‚˜์˜ค๋ฉฐ, ๊ทธ ์œ„์˜ ์ดˆ๋ก ์œก๊ฐํ˜• \mathcal L_{\mathrm{act}}๊ฐ€ ์ด๋ฅผ ๊ฐ๋…ํ•œ๋‹ค. ๋…ผ๋ฌธ ยง3.2์˜ ์ •์˜๋กœ \tilde a_{1:H}๋Š” ์ƒ˜ํ”Œ๋ง๋œ flow time์—์„œ noise๊ฐ€ ์„ž์ธ action chunk์ด๊ณ , action expert๋Š” flow matching์œผ๋กœ ํ•™์Šต๋œ๋‹ค(ยง3.1, ์‹ (1)). \hat a_{1:H}๋Š” action expert์˜ ์ถœ๋ ฅ, ์ฆ‰ ์˜ˆ์ธก๋œ ๊ธธ์ด H์˜ action chunk๋‹ค. ๋ฐฑ๋ณธ ์ƒ์ž์—์„œ Action Expert๋กœ ๊ฐ€๋Š” ๊ฐ€๋กœ ํ™”์‚ดํ‘œ๋Š” ๋ฐฑ๋ณธ์ด ๋งŒ๋“  ์‹œ๊ฐ context๋ฅผ action expert๊ฐ€ ์ฝ๋Š”๋‹ค๋Š” ๋œป์ด๋‹ค. ์ด ๊ฒฝ๋กœ๊ฐ€ ์„ธ ํŒจ๋„์—์„œ ๊ฐ™๋‹ค๋Š” ๊ฒƒ์€ ยง3.1์˜ ์„œ์ˆ (โ€œAll three recipes in Figure 1 share the same data and the same deployed interfaceโ€)๊ณผ ์ •ํ™•ํžˆ ๋งž๋Š”๋‹ค. ๋ฐฐํฌ๋œ ์ •์ฑ…์€ ์–ด๋А ๊ณ„์—ด์ด๋“  ํ˜„์žฌ ๊ด€์ธก f_0, ๋กœ๋ด‡ ์ƒํƒœ s_0, ์ง€์‹œ๋ฌธ c๋กœ a_{1:H}๋ฅผ ๋‚ธ๋‹ค(s_0์™€ c๋Š” Fig. 1์—๋Š” ์ƒ๋žต๋˜๊ณ  Fig. 2์— ๋‚˜์˜จ๋‹ค).

ํŒจ๋„ (a) Vision-language-action model

  • ๋ฐฑ๋ณธ: ์™ผ์ชฝ ์•„๋ž˜์˜ VLM ์ƒ์ž. ์ž…๋ ฅ์€ ํ˜„์žฌ ๊ด€์ธก f_0 ํ•˜๋‚˜๋‹ค.
  • ๋ฏธ๋ž˜ ๊ด€์ธก: ์™ผ์ชฝ ์œ„์— ํšŒ์ƒ‰ ์ ์„  ์ƒ์ž โ€œf_{1:T} Unusedโ€๊ฐ€ ์žˆ๋‹ค. ๋ฐ์ดํ„ฐ์—๋Š” ๊ธฐ๋ก๋œ ๋ฏธ๋ž˜ ๊ด€์ธก f_{1:T}๊ฐ€ ์žˆ์ง€๋งŒ ์“ฐ์ง€ ์•Š๋Š”๋‹ค๋Š” ๋œป์ด๋‹ค. ๊ทธ ์œ„์˜ ํšŒ์ƒ‰ ์ ์„  ์œก๊ฐํ˜• ์•ˆ์—๋Š” โ€œโ€”โ€๋งŒ ์žˆ๋‹ค. ๋ฏธ๋ž˜์— ๊ฑฐ๋Š” ์†์‹ค์ด ์—†๋‹ค๋Š” ํ‘œ์‹œ๋‹ค.
  • ๋ชฉ์ ํ•จ์ˆ˜: \mathcal L_{\mathrm{VLA}}=\mathcal L_{\mathrm{act}}(์‹ (1)). ์‚ฌ์ „ํ•™์Šต ์ง€์‹์€ ์ดˆ๊ธฐํ™”๋กœ๋งŒ ๋“ค์–ด์˜จ๋‹ค.
  • ์ถ”๋ก  ์‹œ ๋‚จ๋Š” ๊ฒƒ: VLM + Action Expert.

๋…ผ๋ฌธ์€ ์ด๊ฒƒ์„ โ€œaction-only VLA referenceโ€๋ผ ๋ถ€๋ฅธ๋‹ค. ์‹คํ—˜์—์„œ ์ด ์นธ์— ํ•ด๋‹นํ•˜๋Š” ๊ฒƒ์€ Qwen3.5-2BยทDINOv2-L์˜ action-only ๋Œ€์กฐ๊ตฐ(Table 14)๊ณผ, FastWAMยทImageWAM ์•„ํ‚คํ…์ฒ˜์—์„œ \mathcal L_G๋ฅผ ๋บ€ ํ–‰(Table 1์˜ ์ฒซ ํ–‰๋“ค)์ด๋‹ค.

ํŒจ๋„ (b) World-action model

  • ๋ฐฑ๋ณธ: Visual Generator ์ƒ์ž. ์ž…๋ ฅ์ด ๋‘˜์ด๋‹ค. ํ˜„์žฌ ๊ด€์ธก f_0(์‹ค์„ )์™€ ์ ์„  ํ…Œ๋‘๋ฆฌ์˜ \tilde f_{1:T}. ยง3.3์€ FastWAMยทImageWAM์˜ ์ƒ์„ฑ ๋ชฉ์ ํ•จ์ˆ˜๊ฐ€ โ€œdenoise future video latents and generate a future imageโ€๋ผ๊ณ  ์“ฐ๋ฏ€๋กœ, \tilde f_{1:T}๋Š” \tilde a์™€ ๊ฐ™์€ ํ‘œ๊ธฐ ๊ด€๋ก€๋กœ ์ฝ์œผ๋ฉด noise๊ฐ€ ์„ž์ธ ๋ฏธ๋ž˜ latent ์ž…๋ ฅ์ด๋‹ค(๊ทธ๋ฆผ์— ๋”ฐ๋กœ ์ •์˜๊ฐ€ ์—†์–ด ํ•„์ž๊ฐ€ ํ‘œ๊ธฐ ๊ด€๋ก€๋กœ ์ฝ์€ ๊ฒƒ์ด๋‹ค).
  • ๋ฏธ๋ž˜ ์˜ˆ์ธก๊ณผ ์†์‹ค: Visual Generator ์œ„๋กœ \hat f_{1:T}(์ƒ์„ฑ๋œ ๋ฏธ๋ž˜)๊ฐ€ ๋‚˜์˜ค๊ณ , ๊ทธ ์œ„์— ๊ธฐ๋ก๋œ ๋ฏธ๋ž˜ f_{1:T}๊ฐ€ ์žˆ๋‹ค. ๋‚จ์ƒ‰ ์œก๊ฐํ˜• \mathcal L_G๋กœ ๋“ค์–ด๊ฐ€๋Š” ํ™”์‚ดํ‘œ๊ฐ€ ๋‘ ๊ฐœ๋‹ค. f_{1:T}์—์„œ ์•„๋ž˜๋กœ๋ถ€ํ„ฐ ํ•˜๋‚˜, \hat f_{1:T}์—์„œ ์˜ค๋ฅธ์ชฝ์œผ๋กœ ๋Œ์•„ ์˜ฌ๋ผ๊ฐ€๋Š” ์„ ์ด ํ•˜๋‚˜. ์ƒ์„ฑ ๊ฒฐ๊ณผ๋ฅผ ๊ธฐ๋ก๋œ ๋ฏธ๋ž˜์™€ ๋งž์ถ”๋Š” ์ƒ์„ฑ ์†์‹ค์ด๋ผ๋Š” ๋œป์ด๋‹ค.
  • ๋ชฉ์ ํ•จ์ˆ˜: \mathcal L_{\mathrm{WAM}}=\mathcal L_{\mathrm{act}}+\lambda_G\mathcal L_G(์‹ (2)).
  • โ€œInfer.โ€ ์ ์„  ๋ฃจํ”„: \hat f_{1:T}์˜ ์˜ค๋ฅธ์ชฝ์—์„œ ํšŒ์ƒ‰ ์ ์„ ์ด ์•„๋ž˜๋กœ ๋‚ด๋ ค์™€ \tilde f_{1:T}๋กœ ๋˜๋Œ์•„๊ฐ„๋‹ค. ์บก์…˜์€ ์ด๋ฅผ โ€œgenerated future observations entering the action expert in some WAMsโ€๋ผ๊ณ  ์„ค๋ช…ํ•œ๋‹ค. ์ถ”๋ก  ๋•Œ ๋ฏธ๋ž˜๋ฅผ ๋จผ์ € ์ƒ์„ฑํ•˜๊ณ  ๊ทธ ์ƒ์„ฑ๋ฌผ์„ ๊ฑฐ์ณ ํ–‰๋™์„ ๋‚ด๋Š” imagine-then-execute ๊ณ„์—ด์„ ๊ฐ€๋ฆฌํ‚จ๋‹ค. ์ด ๋ฃจํ”„๋Š” โ€œsome WAMsโ€์—๋งŒ ํ•ด๋‹นํ•œ๋‹ค๋Š” ์ ์ด ์ค‘์š”ํ•˜๋‹ค. ์ด ๋…ผ๋ฌธ์ด ์ง์ ‘ ๋น„๊ตํ•˜๋Š” FastWAM๊ณผ ImageWAM์€ ์ด๋ฏธ ์ถ”๋ก  ๋•Œ ์ƒ์„ฑ์„ ํ•˜์ง€ ์•Š๋Š”๋‹ค(ยง3.2 ๋).
  • ์ถ”๋ก  ์‹œ ๋‚จ๋Š” ๊ฒƒ: Visual Generator + Action Expert, ์ผ๋ถ€ WAM์€ ์—ฌ๊ธฐ์— ์ƒ์„ฑ ๋ฃจํ”„๊นŒ์ง€.

ยง3.1์€ ์ด ์†์‹ค์ด ๋ ˆ์‹œํ”ผ๋ฅผ ์ƒ์„ฑ๊ธฐ์— ๋ฌถ๋Š”๋‹ค๊ณ  ์ง€์ ํ•œ๋‹ค. ํ•™์Šต ๋•Œ decoder๋‚˜ denoising ๊ฒฝ๋กœ๊ฐ€ ํ•„์š”ํ•˜๊ณ , ์ผ๋ถ€ WAM์€ ์ถ”๋ก  ๋•Œ๋„ ํ•„์š”ํ•˜๋‹ค.

ํŒจ๋„ (c) Just action model (JAM)

  • ๋ฐฑ๋ณธ: ๊ฐ€๋กœ๋กœ ๋„“์€ Backbone ์ƒ์ž ์•„๋ž˜์— โ€œVLM ยท Visual Generator ยท Visual Encoderโ€๊ฐ€ ์ ํ˜€ ์žˆ๋‹ค. (a)์™€ (b)์˜ ๋‘ ๋ฐฑ๋ณธ์„ ํฌํ•จํ•ด self-supervised ์ธ์ฝ”๋”๊นŒ์ง€ ๋ฌด์—‡์ด๋“  ๋“ค์–ด๊ฐˆ ์ˆ˜ ์žˆ๋‹ค๋Š” ํ‘œ์‹œ๋‹ค. ์ž…๋ ฅ์€ f_0 ํ•˜๋‚˜๋ฟ์ด๋‹ค. (b)์™€ ๋‹ฌ๋ฆฌ ๋ฏธ๋ž˜ latent๊ฐ€ ๋ฐฑ๋ณธ ์ž…๋ ฅ์œผ๋กœ ๋“ค์–ด๊ฐ€์ง€ ์•Š๋Š”๋‹ค.
  • ์ฃผํ™ฉ ์ ์„  ์˜์—ญ โ€œTraining Onlyโ€: ์œ„์ชฝ ์ „์ฒด๋ฅผ ๊ฐ์‹ผ๋‹ค. ๊ทธ ์•ˆ์— ๋„ค ์š”์†Œ๊ฐ€ ์žˆ๋‹ค.
    • f_{1:T} โ†’ Future Anchor: ๊ธฐ๋ก๋œ ๋ฏธ๋ž˜๋ฅผ ๊ณ ์ •๋œ ์ธ์ฝ”๋”(future anchor E_\phi)๊ฐ€ feature๋กœ ๋ฐ”๊พผ๋‹ค. ์ถœ๋ ฅ์ด g_k๋‹ค.
    • Prediction Head: ๋ฐฑ๋ณธ ๋ธ”๋ก \ell์˜ ํ˜„์žฌ ๊ด€์ธก ํ† ํฐ h^{(\ell)}๋ฅผ ๋ฐ›์•„ \hat g_k๋ฅผ ๋‚ธ๋‹ค. h^{(\ell)} ํ™”์‚ดํ‘œ๋Š” Backbone ์ƒ์ž์—์„œ ์ถœ๋ฐœํ•ด ์ฃผํ™ฉ ์˜์—ญ ๊ฒฝ๊ณ„๋ฅผ ๋„˜์–ด ๋“ค์–ด๊ฐ„๋‹ค.
    • ์ฃผํ™ฉ ์œก๊ฐํ˜• \mathcal L_F: g_k์™€ \hat g_k ๋‘ ํ™”์‚ดํ‘œ๊ฐ€ ๋“ค์–ด์˜จ๋‹ค. foresight loss๋‹ค.
  • ๋ชฉ์ ํ•จ์ˆ˜: \mathcal L_{\mathrm{JAM}}=\mathcal L_{\mathrm{act}}+\lambda_F\mathcal L_F(์‹ (3)).
  • ์ถ”๋ก  ์‹œ ๋‚จ๋Š” ๊ฒƒ: Backbone + Action Expert. ์ฃผํ™ฉ ์˜์—ญ์€ ํ†ต์งธ๋กœ ์‚ฌ๋ผ์ง„๋‹ค.

๊ทธ๋ฆผ์—์„œ ๊ฐ€์žฅ ์ค‘์š”ํ•œ ๊ฒƒ์€ ์ฃผํ™ฉ ์˜์—ญ์—์„œ Action Expert๋กœ ๊ฐ€๋Š” ํ™”์‚ดํ‘œ๊ฐ€ ํ•˜๋‚˜๋„ ์—†๋‹ค๋Š” ์ ์ด๋‹ค. ๋ฏธ๋ž˜ ๊ด€์ธก์€ g_k๋ผ๋Š” target์œผ๋กœ๋งŒ ์“ฐ์ด๊ณ  ํ–‰๋™ ๊ฒฝ๋กœ์˜ ์ž…๋ ฅ์ด ๋˜์ง€ ์•Š๋Š”๋‹ค. ๋ฏธ๋ž˜๊ฐ€ ํ–‰๋™์— ์˜ํ–ฅ์„ ์ฃผ๋Š” ์œ ์ผํ•œ ํ†ต๋กœ๋Š” \mathcal L_F์˜ gradient๊ฐ€ Prediction Head๋ฅผ ๊ฑฐ์ณ ๋ฐฑ๋ณธ์˜ h^{(\ell)}๋กœ ์—ญ์ „ํŒŒ๋˜๋Š” ๊ฒƒ๋ฟ์ด๋‹ค. (b)์—์„œ๋Š” ๋ฏธ๋ž˜ latent๊ฐ€ ๋ฐฑ๋ณธ ์ž…๋ ฅ์œผ๋กœ ๋“ค์–ด๊ฐ€๊ณ (์ ์„  \tilde f), ์ผ๋ถ€ WAM์—์„œ๋Š” ์ƒ์„ฑ๋œ ๋ฏธ๋ž˜๊ฐ€ ํ–‰๋™ ๊ฒฝ๋กœ๊นŒ์ง€ ๋“ค์–ด๊ฐ€๋Š” ๊ฒƒ(Infer. ๋ฃจํ”„)๊ณผ ์ •ํ™•ํžˆ ๋Œ€๋น„๋œ๋‹ค.

์„ธ ์ถ•์œผ๋กœ ์ •๋ฆฌํ•˜๋ฉด

๋ฐฑ๋ณธ ์ดˆ๊ธฐํ™” ๊ธฐ๋ก๋œ ๋ฏธ๋ž˜ f_{1:T}์— ๊ฑฐ๋Š” ๋ชฉ์ ํ•จ์ˆ˜ ์ถ”๋ก  ๋•Œ ๋‚จ๋Š” ๋ชจ๋“ˆ
(a) VLA VLM ์—†์Œ (โ€œUnusedโ€) VLM + Action Expert
(b) WAM Visual generator \mathcal L_G: ๋ฏธ๋ž˜ ๊ด€์ธก ์ƒ์„ฑ Generator + Action Expert (์ผ๋ถ€ WAM์€ ๋ฏธ๋ž˜ ์ƒ์„ฑ ๋ฃจํ”„ ํฌํ•จ)
(c) JAM VLM ยท visual generator ยท visual encoder ์ค‘ ์„ ํƒ \mathcal L_F: ๊ณ ์ • future anchor์˜ feature ์˜ˆ์ธก Backbone + Action Expert

๊ทธ๋ฆผ์˜ ์ƒ‰๋„ ์ด ๊ตฌ๋ถ„์„ ๋”ฐ๋ฅธ๋‹ค. ์‹œ๊ฐ ์ฒ˜๋ฆฌ ์ชฝ ์ƒ์ž(VLM, Visual Generator, Backbone, ๊ด€์ธก f)์™€ \mathcal L_G๋Š” ๋‚จ์ƒ‰ ๊ณ„์—ด, ํ–‰๋™ ์ชฝ(\tilde a, \hat a, Action Expert, \mathcal L_{\mathrm{act}})์€ ์ดˆ๋ก ๊ณ„์—ด, ํ•™์Šต ์ „์šฉ foresight ๊ฐ€์ง€๋Š” ์ฃผํ™ฉ ๊ณ„์—ด์ด๋‹ค.

๊ธฐํ˜ธ ์ •๋ฆฌ

๊ธฐํ˜ธ ์˜๋ฏธ ๊ทผ๊ฑฐ
f_0 ํ˜„์žฌ ๊ด€์ธก ยง3.1
f_{1:T} ๊ธฐ๋ก๋œ ๋ฏธ๋ž˜ ๊ด€์ธก ยง3.1
\hat f_{1:T}, \tilde f_{1:T} (b)์˜ ์ƒ์„ฑ๋œ ๋ฏธ๋ž˜ / ์ƒ์„ฑ๊ธฐ์˜ ๋ฏธ๋ž˜ ์ž…๋ ฅ ๊ทธ๋ฆผ + ยง3.3 (tilde ํ•ด์„์€ ํ•„์ž)
\tilde a_{1:H}, \hat a_{1:H} noise๊ฐ€ ์„ž์ธ action chunk / action expert ์ถœ๋ ฅ ยง3.2, Fig. 2
h^{(\ell)}\in\mathbb R^{N\times D} ๋ฐฑ๋ณธ ๋ธ”๋ก \ell์˜ ํ˜„์žฌ ๊ด€์ธก ํ† ํฐ ยง3.2
g_k,\ \hat g_k\in\mathbb R^{N\times D_g} readout offset k์˜ target / ์˜ˆ์ธก ยง3.2
\mathcal L_{\mathrm{act}},\ \mathcal L_G,\ \mathcal L_F ํ–‰๋™ flow matching / ์ƒ์„ฑ / foresight ์†์‹ค ์‹ (1)~(4)

๊ทธ๋ฆผ์ด ๋ณด์—ฌ ์ฃผ์ง€ ์•Š๋Š” ๊ฒƒ

Fig. 1์€ ์„ค๊ณ„ ๊ณต๊ฐ„์˜ ์ง€๋„์ด๊ณ , ์‹คํ—˜์—์„œ ๊ฒฐ์ •์ ์ด์—ˆ๋˜ ์„ธ๋ถ€๋Š” ๊ทธ๋ฆผ ๋ฐ–์— ์žˆ๋‹ค.

  • future anchor๊ฐ€ ๋ฌด์—‡์ด๋ƒ: ๊ทธ๋ฆผ์€ โ€œFuture Anchorโ€ ํ•œ ์ƒ์ž๋กœ ๊ทธ๋ฆฌ์ง€๋งŒ, ์‹ค์ œ๋กœ๋Š” ์‚ฌ์ „ํ•™์Šต ๋ฐฑ๋ณธ์˜ ๊ณ ์ • ๋ณต์‚ฌ๋ณธ, ๋ฌด์ž‘์œ„ ์ดˆ๊ธฐํ™”๋œ ๊ณ ์ • ๋ฐฑ๋ณธ, ์ •์ฑ…์ด ์ด๋ฏธ ์“ฐ๋Š” VAE tokenizer ๋‹จ๋…์ด ๋ชจ๋‘ ์‹œํ—˜๋๊ณ , Wan ๋ฐฑ๋ณธ์—์„œ๋Š” VAE ๋‹จ๋…์ด ๊ฐ€์žฅ ์ข‹์•˜๋‹ค(Table 1).
  • offset k: g_k์˜ k๊ฐ€ ํ˜„์žฌ(k=0)์ธ์ง€ ๋ฏธ๋ž˜(k\ge 1)์ธ์ง€๊ฐ€ ๋ชฉ์ ํ•จ์ˆ˜ ๊ต์ฒด ์ž์ฒด๋ณด๋‹ค ํฐ ์ฐจ์ด๋ฅผ ๋ƒˆ๋‹ค(Table 1).
  • (b)์˜ Infer. ๋ฃจํ”„๋Š” ๋น„๊ต ๋Œ€์ƒ์ด ์•„๋‹ˆ๋‹ค: ๋งค์นญ๋œ FastWAMยทImageWAM์€ ์ด๋ฏธ JAM๊ณผ ๊ฐ™์€ ์ถ”๋ก  ๊ฒฝ๋กœ๋ฅผ ์“ด๋‹ค. ๋”ฐ๋ผ์„œ ์ด ๋…ผ๋ฌธ์˜ matched ๋น„๊ต๊ฐ€ ์ธก์ •ํ•˜๋Š” ๊ฒƒ์€ ๊ทธ๋ฆผ์˜ ์„ธ ์ถ• ๊ฐ€์šด๋ฐ ๋‘ ๋ฒˆ์งธ ์ถ•(๋ฏธ๋ž˜ ๋ชฉ์ ํ•จ์ˆ˜) ํ•˜๋‚˜๋‹ค. ์„ธ ๋ฒˆ์งธ ์ถ•(์ถ”๋ก  ๋ชจ๋“ˆ)์˜ ํšจ๊ณผ๋Š” ์ด ๋…ผ๋ฌธ์ด ์ง์ ‘ ์žฌ์ง€ ์•Š๊ณ  FastWAM ๋…ผ๋ฌธ์˜ ์ธก์ •(4๋ฐฐ ์ด์ƒ ๋น ๋ฅธ ์ถ”๋ก )์„ ์ธ์šฉํ•œ๋‹ค(ยง4.4).
  • ๋ฐฑ๋ณธ ์นธ์˜ ์ฐจ์ด: (c)๋Š” Backbone์„ ํ•œ ์ƒ์ž๋กœ ๊ทธ๋ฆฌ์ง€๋งŒ, ๋ฐฑ๋ณธ๋งˆ๋‹ค action expert๊ฐ€ ๋ฐฑ๋ณธ์„ ์ฝ๋Š” ๋ฐฉ์‹(attention ๊ตฌ์กฐยทํ…์ŠคํŠธ ์ธ์ฝ”๋”)์ด ๋‹ค๋ฅด๋‹ค(Table 7). ๋…ผ๋ฌธ๋„ Table 3 ์บก์…˜์—์„œ ์ด ๋•Œ๋ฌธ์— ์‚ฌ์ „ํ•™์Šต ์ž์ฒด์˜ ํšจ๊ณผ๋ฅผ ๋ถ„๋ฆฌํ•  ์ˆ˜ ์—†๋‹ค๊ณ  ๋ฐํžŒ๋‹ค.

ํ”„๋กœ์ ํŠธ ํŽ˜์ด์ง€์˜ 59์ดˆ ํŠธ๋ ˆ์ผ๋Ÿฌ(๋ฌด์Œ)๊ฐ€ ์ด ๊ทธ๋ฆผ์˜ ์„ธ ํŒจ๋„์„ ์• ๋‹ˆ๋ฉ”์ด์…˜์œผ๋กœ ๋‹ค์‹œ ๋ณด์—ฌ ์ค€๋‹ค. โ€œLearn only the actionsโ€(VLA, ๋ฏธ๋ž˜ ํ”„๋ ˆ์ž„์— UNUSED ํ‘œ์‹œ) โ†’ โ€œLearn to generate the futureโ€(WAM) โ†’ โ€œPredict features, not framesโ€(JAM) โ†’ โ€œDeploy actions onlyโ€ ์ˆœ์„œ๋‹ค.


ํ”„๋กœ์ ํŠธ ํŽ˜์ด์ง€ ํŠธ๋ ˆ์ผ๋Ÿฌ(0:59, ๋ฌด์Œ).

ํ•ต์‹ฌ ๋ฐฉ๋ฒ•๋ก :

์ •์ฑ… ๋ฐฑ๋ณธ์ด ํ˜„์žฌ ๊ด€์ธก์„ ์ธ์ฝ”๋”ฉํ•œ ํ† ํฐ h^{(\ell)}์—์„œ, ํ•™์Šต ์ „์šฉ prediction head p๊ฐ€ ๊ฐ readout offset k๋งˆ๋‹ค \hat g_k๋ฅผ ๋‚ธ๋‹ค. ๊ณ ์ •๋œ future anchor E_\phi๊ฐ€ ๊ธฐ๋ก๋œ ๋ฏธ๋ž˜๋กœ ๋งŒ๋“  target g_k์™€์˜ cosine ๊ฑฐ๋ฆฌ๋ฅผ ํ† ํฐ๊ณผ offset์— ๋Œ€ํ•ด ํ‰๊ท ํ•œ ๊ฒƒ์ด foresight loss๋‹ค.

\mathcal L_F=\frac{1}{|\mathcal K|}\sum_{k\in\mathcal K}\mathbb E_n\Big[1-\cos\big(\hat g_{k,n},\ \mathrm{sg}[g_{k,n}]\big)\Big], \qquad \mathcal L_{\mathrm{JAM}}=\mathcal L_{\mathrm{act}}+\lambda_F\mathcal L_F

\mathcal K๋Š” ๊ฐ๋…ํ•  offset ์ง‘ํ•ฉ, n์€ ๊ณต๊ฐ„ ํ† ํฐ ์ธ๋ฑ์Šค, \mathrm{sg}๋Š” stop-gradient๋‹ค. gradient๋Š” head์™€ ์ •์ฑ… ๋ฐฑ๋ณธ์—๋งŒ ํ๋ฅธ๋‹ค. target ์ „์ฒ˜๋ฆฌ์™€ conditioning์ด ๊ณ ์ •์ด๋ฉด g_k๋Š” ํ•œ ๋ฒˆ ๊ณ„์‚ฐํ•ด ์บ์‹œํ•  ์ˆ˜ ์žˆ๋‹ค. ํ•™์Šต ํ›„ anchor์™€ head๋ฅผ ๋ฒ„๋ฆฌ๋ฏ€๋กœ ๋ฐฐํฌ ์•„ํ‚คํ…์ฒ˜์™€ ์ถ”๋ก  ๋น„์šฉ์€ ๋งค์นญ๋œ ๋ฒ ์ด์Šค๋ผ์ธ๊ณผ ๊ฐ™๋‹ค.

์ฃผ์š” ๊ฒฐ๊ณผ:

  • ์ƒ์„ฑ โ†’ foresight ๊ต์ฒด(Table 1, ์•„ํ‚คํ…์ฒ˜ยท๋ฐ์ดํ„ฐยท์˜ˆ์‚ฐ ๊ณ ์ •): FastWAM LIBERO 97.00 โ†’ 98.15%, LIBERO-Plus 55.31 โ†’ 64.54%(VAE target). ImageWAM(Ovis-U1) LIBERO 92.90 โ†’ 96.75%, LIBERO-Plus 65.27 โ†’ 79.95%(์‚ฌ์ „ํ•™์Šต Yak target).
  • offset: ํ˜„์žฌ ์œ„์น˜ target([0])๋ณด๋‹ค ๋ฏธ๋ž˜ ์œ„์น˜ target([1,2] ๋˜๋Š” [1])์ด ๋‚ซ๋‹ค. Wan์—์„œ LIBERO-Plus 44.62 โ†’ 55.58%.
  • ๋ฐฑ๋ณธ ๊ณ„์—ด ํ™•์žฅ(Table 3): LIBERO-Plus JAM(Qwen3.5-2B) 81.14%, JAM(Ovis-U1) 79.95%, JAM(DINOv2-L) 70.39%, JAM(DINOv2-B) 68.19%. ๋„ค ๊ฐœ ๋ชจ๋‘ ImageWAM(Ovis-U1) 65.27%๋ณด๋‹ค ๋†’๋‹ค.
  • RoboTwin 2.0(Table 4): JAM(FLUX.2) 93.1%๋กœ ํ‘œ์—์„œ ์ตœ๊ณ , JAM(Wan) 91.4% vs FastWAM(๊ณต๊ฐœ ๊ฐ€์ค‘์น˜) 91.2%.
  • ํ•™์Šต ๋น„์šฉ(Table 9, ๋…ผ๋ฌธ v1 ์ฃผ์žฅ๊ฐ’): JAM๋งŒ target์„ ์บ์‹œํ•œ ์กฐ๊ฑด์—์„œ ์ƒ˜ํ”Œ๋‹น ์ถ”์ • 15.52 โ†’ 4.19 TFLOPs(โˆ’73.0%, FastWAM ๋Œ€๋น„), 5.81 โ†’ 3.67(โˆ’36.8%, ImageWAM ๋Œ€๋น„). GH200 1์žฅ ์ธก์ • GPU ์‹œ๊ฐ„์€ โˆ’67.3% / โˆ’37.8%. ๋ฒ ์ด์Šค๋ผ์ธ๋„ ๋ฏธ๋ž˜ ์ธ์ฝ”๋”ฉ์„ ์บ์‹œํ•˜๋Š” ์กฐ๊ฑด์œผ๋กœ ๋งž์ถ”๋ฉด โˆ’57.7% / โˆ’31.9%์ด๊ณ , ๊ณต๊ฐœ ๋ ˆํฌ์™€ ํ”„๋กœ์ ํŠธ ํŽ˜์ด์ง€๋Š” ํ˜„์žฌ ์ด ๊ฐ’์„ ์“ด๋‹ค.
  • ํ†ต๊ณ„(Table 20ยท21): ๋„ค ๋น„๊ต ๋ชจ๋‘ 95% task-bootstrap ๊ตฌ๊ฐ„์ด 0์„ ๋ฐฐ์ œํ•˜๊ณ , ์„ ํƒ๋œ ๋‘ ์„ค์ •์˜ 3ํšŒ ๋…๋ฆฝ ํ•™์Šต์—์„œ LIBERO-Plus 64.61ยฑ0.37%(Wan), 79.92ยฑ0.43%(Ovis-U1).

๊ฒฐ๋ก :

JAM์˜ ์ƒˆ๋กœ์›€์€ ๋ชจ๋“ˆ์ด ์•„๋‹ˆ๋ผ ๋ถ„ํ•ด๋‹ค. โ€œVLA๋ƒ WAM์ด๋ƒโ€๋ผ๋Š” ์งˆ๋ฌธ์„ ๋ฐฑ๋ณธ ยท ๋ฏธ๋ž˜ ๋ชฉ์ ํ•จ์ˆ˜ ยท ์ถ”๋ก  ๋ชจ๋“ˆ์˜ ์„ธ ์ถ•์œผ๋กœ ์ชผ๊ฐœ๊ณ , ๋‘ ๋ฒˆ์งธ ์ถ•์—์„œ ์ƒ์„ฑ์„ ๊ณ ์ • feature ์˜ˆ์ธก์œผ๋กœ ๋ฐ”๊ฟ”๋„ ์„ฑ๋Šฅ์ด ์œ ์ง€๋˜๊ฑฐ๋‚˜ ์˜ค๋ฅธ๋‹ค๋Š” ๊ฒƒ์„ matched ๋น„๊ต๋กœ ๋ณด์˜€๋‹ค. ๋‹ค๋งŒ ๊ฐ€์žฅ ์ข‹์€ target์ด Wan์—์„œ๋Š” VAE latent์˜€๋‹ค๋Š” ์ , ๊ธฐ์ค€์ด ๋œ FastWAM ์ˆ˜์น˜๊ฐ€ ์› ๋…ผ๋ฌธ ๋ณด๊ณ ๊ฐ’๋ณด๋‹ค ๋‚ฎ๋‹ค๋Š” ์ , ์‹คํ—˜์ด ์ „๋ถ€ ์‹œ๋ฎฌ๋ ˆ์ด์…˜์ด๋ผ๋Š” ์ ์€ ๊ฒฐ๊ณผ๋ฅผ ์ฝ์„ ๋•Œ ๊ฐ™์ด ๋“ค๊ณ  ๊ฐ€์•ผ ํ•œ๋‹ค.

๐Ÿ”” Ring Review

๐Ÿ”” Ring โ€” An idea that echoes. Grasp the core and its value.

ํ•œ ์ค„๋กœ ์‹œ์ž‘ํ•˜๋ฉด

๋ฏธ๋ž˜ ๊ด€์ธก์ด ์ •์ฑ… ํ•™์Šต์— ๋„์›€์ด ๋œ๋‹ค๋Š” ๋ฐ์—๋Š” VLA ์ง„์˜๋„ WAM ์ง„์˜๋„ ์ด๊ฒฌ์ด ์—†๋‹ค. ๊ฐˆ๋ฆฌ๋Š” ์ง€์ ์€ โ€œ์–ด๋–ป๊ฒŒ ์“ฐ๋А๋ƒโ€๋‹ค. WAM์€ ๋ฏธ๋ž˜๋ฅผ ๊ทธ๋ ค ๋ณด๊ฒŒ ํ•˜๊ณ , JAM์€ ๋ฏธ๋ž˜๋ฅผ ์•Œ์•„๋งžํžˆ๊ฒŒ ํ•œ๋‹ค. ๊ณ ์ •๋œ ์ธ์ฝ”๋”๊ฐ€ ๊ธฐ๋ก๋œ ๋ฏธ๋ž˜๋ฅผ feature๋กœ ๋ฐ”๊ฟ” ๋‘๋ฉด, ์ •์ฑ…์€ ํ˜„์žฌ ํ”„๋ ˆ์ž„๋งŒ ๋ณด๊ณ  ๊ทธ feature๋ฅผ ๋งžํžˆ๋„๋ก ํ‘œํ˜„์„ ๋‹ค๋“ฌ๋Š”๋‹ค. ๊ทธ๋ฆผ์„ ๊ทธ๋ฆฌ๋Š” decoder๋„, denoising ๊ฒฝ๋กœ๋„, ๋ฏธ๋ž˜ ํ† ํฐ์„ ๋ฐฑ๋ณธ์— ๋„ฃ๋Š” ์ผ๋„ ํ•„์š” ์—†๋‹ค. ๊ทธ๋ ‡๊ฒŒ ํ•˜๋ฉด ์ƒ์„ฑ๊ธฐ ๋ฐฑ๋ณธ์ด๋“  VLM ๋ฐฑ๋ณธ์ด๋“  ๊ฐ™์€ ๋ฐฉ์‹์œผ๋กœ ๋ฏธ๋ž˜์—์„œ ๋ฐฐ์šธ ์ˆ˜ ์žˆ๊ณ , ์„ฑ๋Šฅ์€ ์ƒ์„ฑ์œผ๋กœ ๋ฐฐ์šด ์Œ๋‘ฅ์ด ์ •์ฑ…๊ณผ ๊ฐ™๊ฑฐ๋‚˜ ๋‚ซ๋‹ค. ์ด๊ฒƒ์ด ์ œ๋ชฉ์˜ โ€œJust Action Modelsโ€๊ฐ€ ๋œปํ•˜๋Š” ๋ฐ”๋‹ค. ๋‚จ๋Š” ๊ฒƒ์€ ํ–‰๋™ ๋ชจ๋ธ๋ฟ์ด๋‹ค.

๋ฐฐ๊ฒฝ: ์„ธ ์„ ํƒ์ด ํ•œ ๋ฌถ์Œ์œผ๋กœ ๋น„๊ต๋ผ ์™”๋‹ค (ยง1ยทยง2)

ํ‘œ์ค€ VLA(OpenVLA, \pi_0, \pi_{0.5})๋Š” VLM์— action ์ถœ๋ ฅ์„ ๋ถ™์—ฌ ํ–‰๋™๋งŒ ํ•™์Šตํ•œ๋‹ค. WAM์€ ์ด๋ฏธ์ง€ยท๋น„๋””์˜ค ์ƒ์„ฑ๊ธฐ์—์„œ ์ถœ๋ฐœํ•ด ํ–‰๋™๊ณผ ๋ฏธ๋ž˜ ๊ด€์ธก ์ƒ์„ฑ์„ ํ•จ๊ป˜ ํ•™์Šตํ•œ๋‹ค. ๋‘ ๊ณ„์—ด์˜ ์„ฑ๊ณต๋ฅ  ์ฐจ์ด๋Š” ํ”ํžˆ โ€œ์ƒ์„ฑ๊ธฐ ์‚ฌ์ „ํ•™์Šต์ด ๋” ์ข‹์€ ์ง€์‹์„ ์ค€๋‹คโ€ ํ˜น์€ โ€œVLM์˜ ์˜๋ฏธ ์ง€์‹์ด ๋” ๋‚ซ๋‹คโ€๋กœ ํ•ด์„๋ผ ์™”๋Š”๋ฐ, ์ €์ž๋“ค์€ ์ด ํ•ด์„์ด ์„ธ ๊ฐ€์ง€๋ฅผ ๋’ค์„ž๋Š”๋‹ค๊ณ  ๋ณธ๋‹ค.

  1. ๋ฐฑ๋ณธ ์ดˆ๊ธฐํ™”: VLM์—์„œ ์ถœ๋ฐœํ•˜๋Š”๊ฐ€, ์ƒ์„ฑ๊ธฐ์—์„œ ์ถœ๋ฐœํ•˜๋Š”๊ฐ€.
  2. ๋ฏธ๋ž˜ ๊ด€์ธก์— ๊ฑฐ๋Š” ๋ชฉ์ ํ•จ์ˆ˜: ๊ธฐ๋ก๋œ ๋ฏธ๋ž˜๋ฅผ ๋ฌด์‹œํ•˜๋Š”๊ฐ€, ์ƒ์„ฑํ•˜๋Š”๊ฐ€.
  3. ์ถ”๋ก  ๋•Œ ๋‚จ๋Š” ๋ชจ๋“ˆ: ๋ฏธ๋ž˜๋ฅผ ์ƒ์„ฑํ•œ ๋’ค ํ–‰๋™ํ•˜๋Š”๊ฐ€, ๋ฐ”๋กœ ํ–‰๋™ํ•˜๋Š”๊ฐ€.

์„ธ ๋ฒˆ์งธ ์ถ•์€ ์ด๋ฏธ ์ผ๋ถ€ ๋ถ„๋ฆฌ๋ผ ์žˆ์—ˆ๋‹ค. ๋ธ”๋กœ๊ทธ์—์„œ ๋ฆฌ๋ทฐํ•œ Fast-WAM์€ attention mask๋กœ ์ถ”๋ก  ๋•Œ์˜ ๋ฏธ๋ž˜ ์ƒ์ƒ์„ ๋–ผ์–ด ๋‚ด๊ณ , ํ•™์Šต ๋•Œ์˜ ๋น„๋””์˜ค co-training์ด ์ด๋“์˜ ๋Œ€๋ถ€๋ถ„์„ ๋‚ธ๋‹ค๊ณ  ๋ณด์˜€๋‹ค. ImageWAM์€ ๋น„๋””์˜ค ๋Œ€์‹  ์ด๋ฏธ์ง€ ํŽธ์ง‘ ๋ฐฑ๋ณธ๊ณผ ๋ฏธ๋ž˜ ์ด๋ฏธ์ง€ target์„ ์“ด๋‹ค. ๋‘ ๋…ผ๋ฌธ์€ ๊ฒฐ๊ณผ์ ์œผ๋กœ โ€œํ•™์Šต ๋•Œ์˜ ์‹œ๊ฐ ์ƒ์„ฑ ๋ชฉ์ ํ•จ์ˆ˜๊ฐ€ WAM์˜ ์—ด์‡ โ€๋ผ๋Š” ํ•ด์„์„ ๋‚จ๊ฒผ๋‹ค. JAM์€ ๊ทธ ๋‹ค์Œ ์งˆ๋ฌธ์„ ๋˜์ง„๋‹ค. ๊ธฐ๋ก๋œ ๋ฏธ๋ž˜์—์„œ ๋ฐฐ์šฐ๋Š” ๋ฐ ์‹œ๊ฐ ์ƒ์„ฑ ๋ชฉ์ ํ•จ์ˆ˜๊ฐ€ ๊ผญ ํ•„์š”ํ•œ๊ฐ€? ์•„๋‹ˆ๋ผ๋ฉด, ์ƒ์„ฑ ์‚ฌ์ „ํ•™์Šต์ด ์—†๋Š” ๋ฐฑ๋ณธ๋„ ๊ฐ™์€ ๋ฐฉ์‹์œผ๋กœ ๋ฏธ๋ž˜์—์„œ ๋ฐฐ์šธ ์ˆ˜ ์žˆ๋Š”๊ฐ€?

๋ฐฉ๋ฒ• ์ƒ์„ธ 1: ๋ฌธ์ œ ์„ค์ • (ยง3.1)

ํ•™์Šต ์˜ˆ์ œ๋Š” ํ˜„์žฌ ๊ด€์ธก f_0, ๋กœ๋ด‡ ์ƒํƒœ s_0, ์ง€์‹œ๋ฌธ c, action chunk a_{1:H}, ๊ธฐ๋ก๋œ ๋ฏธ๋ž˜ ๊ด€์ธก f_{1:T}๋กœ ์ด๋ฃจ์–ด์ง„๋‹ค. ๋ฐฐํฌ๋œ ์ •์ฑ…์€ (f_0,s_0,c)์—์„œ a_{1:H}๋ฅผ ์˜ˆ์ธกํ•œ๋‹ค. ์„ธ ๋ ˆ์‹œํ”ผ์˜ ๋ชฉ์ ํ•จ์ˆ˜๋Š” ๋‹ค์Œ๊ณผ ๊ฐ™๋‹ค.

\mathcal L_{\mathrm{VLA}}=\mathcal L_{\mathrm{act}}\quad(1),\qquad \mathcal L_{\mathrm{WAM}}=\mathcal L_{\mathrm{act}}+\lambda_G\mathcal L_G\quad(2),\qquad \mathcal L_{\mathrm{JAM}}=\mathcal L_{\mathrm{act}}+\lambda_F\mathcal L_F\quad(3)

\mathcal L_{\mathrm{act}}์€ action chunk์— ๋Œ€ํ•œ flow matching ์†์‹ค์ด๋‹ค. ์‹ (3)์ด ๋‘ ๊ณ„์—ด์„ ์ž‡๋Š” ๋‹ค๋ฆฌ๋ผ๋Š” ๊ฒƒ์ด ์ €์ž๋“ค์˜ ์ฃผ์žฅ์ด๋‹ค. ๋ฏธ๋ž˜๋Š” ์‹œ๊ฐ ์ƒ์„ฑ์ด ์•„๋‹ˆ๋ผ ๊ณ ์ • future anchor๋ฅผ ํ†ตํ•ด ๊ฐ๋…ํ•˜๊ณ , ๋ชฉ์ ํ•จ์ˆ˜๊ฐ€ ์ƒ์„ฑ๊ธฐ๋ฅผ ์š”๊ตฌํ•˜์ง€ ์•Š์œผ๋ฏ€๋กœ ๋ฐฑ๋ณธ์€ generative DiT, self-supervised ์ธ์ฝ”๋”, VLM ์ค‘ ๋ฌด์—‡์ด๋“  ๋  ์ˆ˜ ์žˆ๋‹ค. future anchor๋Š” ๊ณ ์ •๋œ ๋ฐฑ๋ณธ์ผ ์ˆ˜๋„ ์žˆ๊ณ  ์ •์ฑ…์˜ VAE tokenizer๋ฅผ ์žฌ์‚ฌ์šฉํ•  ์ˆ˜๋„ ์žˆ๋‹ค.

๋ฐฉ๋ฒ• ์ƒ์„ธ 2: Foresight supervision (ยง3.2, ๋ถ€๋ก A.3)


Fig. 2 โ€” JAM ํ•™์Šต(์™ผ์ชฝ)๊ณผ ๋ฐฐํฌ(์˜ค๋ฅธ์ชฝ). Input EncoderยทText EncoderยทFuture Anchor๋Š” ๊ณ ์ •(๋ณ„ํ‘œ), BackboneยทPrediction Head๋Š” ํ•™์Šต ๋Œ€์ƒ(์ ). ๊ธฐ๋ก๋œ ๋ฏธ๋ž˜ f_{1:T}๋Š” ์ ์„  ์ƒ์ž ์•ˆ์˜ future anchor ๊ฐ€์ง€๋กœ๋งŒ ๋“ค์–ด๊ฐ€๊ณ , ๋ฐฐํฌ ๋•Œ๋Š” ์ด ๊ฐ€์ง€์™€ prediction head๊ฐ€ ์‚ฌ๋ผ์ง„๋‹ค.

Fig. 2๋Š” Fig. 1(c)๋ฅผ ์‹ค์ œ ๋ฐ์ดํ„ฐ ํ๋ฆ„์œผ๋กœ ํŽผ์นœ ๊ทธ๋ฆผ์ด๋‹ค. Fig. 1์— ์—†๋˜ ๊ฒƒ์ด ์…‹ ๋ณด์ธ๋‹ค. โ‘  ํ˜„์žฌ ๊ด€์ธก์€ ์•„ํ‚คํ…์ฒ˜๋ณ„ Input Encoder(๊ณ ์ •)๋ฅผ ๊ฑฐ์ณ ๋ฐฑ๋ณธ์— ๋“ค์–ด๊ฐ„๋‹ค. โ‘ก ์ง€์‹œ๋ฌธ c๋Š” ๊ณ ์ • Text Encoder๋ฅผ ๊ฑฐ์ณ ๋ฐฑ๋ณธ์— ๋“ค์–ด๊ฐ„๋‹ค(ํ…์ŠคํŠธ ์ž…๋ ฅ์„ ๋ฐ›๋Š” ๋ฐฑ๋ณธ์˜ ๊ฒฝ์šฐ). โ‘ข ๋ฐฑ๋ณธ์—์„œ Action Expert๋กœ ๊ฐ€๋Š” ํ™”์‚ดํ‘œ์— KV๊ฐ€ ๋ถ™์–ด ์žˆ๋‹ค. action expert๋Š” ๋ฐฑ๋ณธ์˜ key/value๋ฅผ attention์œผ๋กœ ์ฝ๊ณ , ์—ฌ๊ธฐ์— ๋กœ๋ด‡ ์ƒํƒœ s_0์™€ noise๊ฐ€ ์„ž์ธ \tilde a_{1:H}๊ฐ€ ๋”ํ•ด์ง„๋‹ค. ์˜ค๋ฅธ์ชฝ ๋ฐฐํฌ ๊ทธ๋ฆผ์—๋Š” โ€œPut the bowl on the plateโ€์™€ f_0 โ†’ Backbone โ†’ KV โ†’ Action Expert โ†’ \hat a_{1:H}๋งŒ ๋‚จ๋Š”๋‹ค.

์ •์ฑ… feature์™€ target. h^{(\ell)}\in\mathbb R^{N\times D}๋Š” ๋ฐฑ๋ณธ ๋ธ”๋ก \ell์˜ ํ˜„์žฌ ๊ด€์ธก ํ† ํฐ์ด๋‹ค. ๊ณ ์ • future anchor E_\phi๋Š” ๊ธฐ๋ก๋œ ๋ฏธ๋ž˜ f_{1:T}๋ฅผ ์ž๊ธฐ conditioning ์ธํ„ฐํŽ˜์ด์Šค๋กœ ์ฒ˜๋ฆฌํ•œ๋‹ค(Yak, FLUX.2, Qwen์€ ์ด ์ธํ„ฐํŽ˜์ด์Šค๊ฐ€ f_0์™€ c๋„ ์ฝ๋Š”๋‹ค). target g_k\in\mathbb R^{N\times D_g}๋Š” anchor์˜ k๋ฒˆ์งธ ์‹œ๊ฐ„ ์œ„์น˜ feature์ด๊ณ , k=0์ด ํ˜„์žฌ ๊ด€์ธก, k\ge1์ด ๊ธฐ๋ก๋œ ๋ฏธ๋ž˜๋‹ค. ๋ฐฑ๋ณธ feature๋‚˜ VAE latent๋Š” h^{(\ell)}์˜ ๊ณต๊ฐ„ ํ† ํฐ ๋ฐฐ์น˜์— ๋งž์ถฐ ์ •๋ ฌ๋œ๋‹ค(๋ถ€๋ก A.2์— ๋”ฐ๋ฅด๋ฉด ๊ฐ์ฒด ์ถ”์  ์—†์ด ๊ณ ์ • ๊ณต๊ฐ„ ์ธ๋ฑ์Šค๋กœ).

Foresight loss. ์•ž์˜ ์‹ (4)๋‹ค. cosine ๊ฑฐ๋ฆฌ๋ฅผ ์“ฐ๋ฏ€๋กœ target์˜ ํฌ๊ธฐ๊ฐ€ ์•„๋‹ˆ๋ผ ๋ฐฉํ–ฅ๋งŒ ๋งžํžˆ๋ฉด ๋œ๋‹ค. head๋Š” ํ•™์Šต ๊ฐ€๋Šฅํ•œ D\to d ์ž…๋ ฅ projection, ๊ณต์œ ๋œ pre-LayerNorm transformer ๋ธ”๋ก 2๊ฐœ, offset๋งˆ๋‹ค ํ•˜๋‚˜์”ฉ์ธ ์„ ํ˜• d\to D_g readout์œผ๋กœ ์ด๋ฃจ์–ด์ง„๋‹ค(๋ถ€๋ก A.3). ์ด ๊ฐ€์ค‘์น˜ \lambda_F๋Š” FastWAM์—์„œ 0.6, ๊ทธ ๋ฐ–์˜ ์ •์ฑ…์—์„œ 0.3์ด๊ณ  offset ์ˆ˜์™€ ๋ฌด๊ด€ํ•˜๊ฒŒ \lambda_F/|\mathcal K|๋กœ ๋‚˜๋ˆ  ์ค€๋‹ค. offset์„ ๋Š˜๋ ค๋„ ์ด ๊ฐ๋… ์„ธ๊ธฐ๋Š” ๊ทธ๋Œ€๋กœ๋‹ค.

Prediction head ๊ธฐ๋ณธ ํฌ๊ธฐ (Table 8).
์ •์ฑ… ์ž…๋ ฅ D target D_g trunk d ์ธต head ์ˆ˜ FFN readout ์ˆ˜
JAM (Wan) 3072 3072 1536 2 24 6144 2
JAM (Ovis-U1) 1536 1536 768 2 12 3072 1
JAM (FLUX.2) 3072 3072 1536 2 24 6144 1
JAM (Qwen3.5-2B) 2048 2048 1024 2 16 4096 1
JAM (DINOv2-B) 768 768 384 2 6 1536 1
JAM (DINOv2-L) 1024 1024 512 2 8 2048 1

Readout offset์˜ ์‹ค์ œ ์˜๋ฏธ(ยง4.1). FastWAM์€ 4 ํ™˜๊ฒฝ ์Šคํ…๋งˆ๋‹ค ํ”„๋ ˆ์ž„์„ ๋ฝ‘๊ณ , causal VAE๊ฐ€ ํ˜„์žฌ ํ”„๋ ˆ์ž„์€ ๋‹จ๋…์œผ๋กœ, ์ดํ›„ ํ”„๋ ˆ์ž„์€ 4์žฅ์”ฉ ํ•œ latent ์œ„์น˜๋กœ ๋ฌถ๋Š”๋‹ค. ๋”ฐ๋ผ์„œ offset [0]์€ ํ˜„์žฌ ํ”„๋ ˆ์ž„, [1]์€ 4~16 ์Šคํ…, [2]๋Š” 20~32 ์Šคํ…์„ ๋ฎ๋Š”๋‹ค(32 ์Šคํ… action chunk ์ „์ฒด). ๋‚˜๋จธ์ง€ ๋ฐฑ๋ณธ์€ ๊ธฐ๋ก๋œ ๋ฏธ๋ž˜๊ฐ€ 16 ์Šคํ… ๋’ค ํ•œ ์žฅ์ด๊ณ , [1]์ด ๊ทธ ํ”„๋ ˆ์ž„, [0]์ด ํ˜„์žฌ๋‹ค.

์ถ”๋ก . ํ•™์Šต ํ›„ anchor์™€ head๋ฅผ ๋ฒ„๋ฆฌ๋ฏ€๋กœ target ์ถ”์ถœ๋„ ๋ฏธ๋ž˜ ์˜ˆ์ธก๋„ ์ถ”๋ก  ๋•Œ ์‹คํ–‰๋˜์ง€ ์•Š๋Š”๋‹ค. ์ •์ฑ…์€ f_0์™€ c๋ฅผ ํ•œ ๋ฒˆ ์ธ์ฝ”๋”ฉํ•˜๊ณ  action expert๊ฐ€ flow matching ์ƒ˜ํ”Œ๋ง ์Šคํ… ๋‚ด๋‚ด ๊ทธ context๋ฅผ ์žฌ์‚ฌ์šฉํ•œ๋‹ค. ์ €์ž๋“ค์ด ์งš๋“ฏ ์ด ์ถ”๋ก  ์ ˆ์ฐจ๋Š” ๋งค์นญ๋œ FastWAMยทImageWAM๊ณผ ๊ฐ™๋‹ค. ์ฐจ์ด๋Š” JAM์ด ํ•™์Šต ๋•Œ๋„ ์‹œ๊ฐ ์ƒ์„ฑ์„ ํ•˜์ง€ ์•Š๋Š”๋‹ค๋Š” ๊ฒƒ์ด๋‹ค.

๋ฐฉ๋ฒ• ์ƒ์„ธ 3: ์„ธ ์‚ฌ์ „ํ•™์Šต ๊ณ„์—ด๋กœ์˜ instantiation (ยง3.3, ๋ถ€๋ก A.2)

๋ฐฑ๋ณธ๋ณ„ target๊ณผ ์ถ”๋ก  context (Table 7).
future anchor target ํ”„๋ ˆ์ž„(ํ™˜๊ฒฝ ์Šคํ…) ๊ฐ๋… ์ธต / ์ „์ฒด readout ์ถ”๋ก  context
Wan / Wan VAE 0, 4, โ€ฆ, 32 18/30 [1, 2] Wan + UMT5
Yak18 0, 16 11/18 [1] Yak18 + Ovis
FLUX.2 0, 16 15/25 [1] FLUX.2 + Qwen3-4B
Qwen3.5-2B 0, 16 13/24 [1] causal imageโ€“text prefix
DINOv2-B / L 16 12/12, 24/24 [1] DINOv2 + Qwen3-4B
  • ์ƒ์„ฑ ๋ฐฑ๋ณธ. FastWAM์€ Wan2.2-5B ๋น„๋””์˜ค ๋ฐฑ๋ณธ(30๋ธ”๋ก)์— ๊ฐ™์€ ๊นŠ์ด์˜ action expert๋ฅผ directional mixed attention์œผ๋กœ ๋ถ™์ธ๋‹ค. ImageWAM(Ovis-U1)์€ Yak18 ์ด๋ฏธ์ง€ ํŽธ์ง‘ ๋ฐฑ๋ณธ(double-stream 6 + single-stream 12๋ธ”๋ก, ํญ 1,536)์— 18๋ธ”๋ก action expert๋ฅผ ๋ถ™์ธ๋‹ค. JAM์œผ๋กœ ๋ฐ”๊พธ๋ฉด ๋ฐฑ๋ณธ์€ ๊นจ๋—ํ•œ ํ˜„์žฌ ํ”„๋ ˆ์ž„๋งŒ ๋ณด๊ณ , ์ •์ฑ… ์•„ํ‚คํ…์ฒ˜์™€ action expert ์ดˆ๊ธฐํ™”๋Š” ๊ทธ๋Œ€๋กœ๋‹ค. ๊ณ ์ • ์ƒ์„ฑ๊ธฐ๊ฐ€ ๋ฌด์—‡์„ ๊ธฐ์—ฌํ•˜๋Š”์ง€ ๊ฐ€๋ฅด๊ธฐ ์œ„ํ•ด ๋‘ target์„ ๋” ๋‘”๋‹ค. ๊ฐ™์€ ๋ฐฑ๋ณธ์„ ๋ฌด์ž‘์œ„ ์ดˆ๊ธฐํ™”ํ•œ ๊ฒƒ(๊ตฌ์กฐ๋Š” ๊ฐ™๊ณ  ์‚ฌ์ „ํ•™์Šต feature๋งŒ ์—†์Œ)๊ณผ, VAE tokenizer ๋‹จ๋…(target transformer ์—†์ด latent patch feature๋ฅผ liftํ•œ ๊ฒƒ)์ด๋‹ค. Wan์˜ VAE-direct target์€ 2\times2\times48 patch๋ฅผ 192์ฐจ์›์—์„œ 3,072์ฐจ์›์œผ๋กœ ๊ณ ์ • ์ขŒํ‘œ ๋ฐ˜๋ณต๊ณผ ์Šค์ผ€์ผ๋ง์œผ๋กœ ๋Š˜๋ฆฐ ๊ฒƒ์ด๊ณ , norm๊ณผ cosine ์œ ์‚ฌ๋„๊ฐ€ ๋ณด์กด๋œ๋‹ค(๋ถ€๋ก A.2).
  • Self-supervised ๋ฐฑ๋ณธ. DINOv2๋Š” ์–ธ์–ด ์ž…๋ ฅ๋„ ์ƒ์„ฑ head๋„ ์—†๋‹ค. action expert๊ฐ€ ๋ฐฑ๋ณธ์˜ transformer ๋ธ”๋ก์„ ๊ทธ๋Œ€๋กœ ๋ณธ๋œจ๊ณ , ๋ณ„๋„ ํ…์ŠคํŠธ ์ธ์ฝ”๋”(๊ณ ์ • Qwen3-4B, 512 ํ† ํฐ)๊ฐ€ ์ง€์‹œ๋ฌธ์„ action expert์—๋งŒ ์ค€๋‹ค. DINOv2๋งŒ ์˜ˆ์™ธ์ ์œผ๋กœ ๋งˆ์ง€๋ง‰ ๋ธ”๋ก์„ ๊ฐ๋…ํ•œ๋‹ค.
  • VLM ๋ฐฑ๋ณธ. Qwen3.5-2B๋Š” ์ด๋ฏธ์ง€์™€ ํ…์ŠคํŠธ๋ฅผ ํ•จ๊ป˜ ์ฝ๋Š”๋‹ค. action expert๋Š” 18๊ฐœ GatedDeltaNet๊ณผ 6๊ฐœ full-attention ๋ธ”๋ก์˜ ํ˜ธํ™˜ ๊ฐ€์ค‘์น˜๋ฅผ ๋ณต์‚ฌํ•˜๊ณ , imageยทtextยทaction ํ† ํฐ ์œ„์˜ causal token mixing์œผ๋กœ ๋ฐฑ๋ณธ ํ† ํฐ์„ ์ฝ๋Š”๋‹ค. ์ €์ž๋“ค์€ ์ด๊ฒƒ์„ โ€œforesight loss๋ฅผ ๋”ํ•œ VLAโ€๋ผ๊ณ  ๋ถ€๋ฅธ๋‹ค.

์ง๊ด€: ์‹ค์ œ๋กœ ๋ฐ”๋€ ๊ฒƒ์€ ๋ฌด์—‡์ธ๊ฐ€

โ€œ์ƒ์„ฑ ๋Œ€์‹  ํ‘œํ˜„์„ ์˜ˆ์ธกํ•œ๋‹คโ€๋Š” ๋ฌธ์žฅ์€ JEPA ๊ณ„์—ด์˜ ์ต์ˆ™ํ•œ ๊ตฌํ˜ธ์ฒ˜๋Ÿผ ๋“ค๋ฆฌ์ง€๋งŒ, ์ด ๋…ผ๋ฌธ์˜ matched ๋น„๊ต์—์„œ ์‹ค์ œ๋กœ ๋ฐ”๋€ ๊ฒƒ์„ ํ•˜๋‚˜์”ฉ ์„ธ์–ด ๋ณด๋ฉด ๊ทธ๋ฆผ์ด ์กฐ๊ธˆ ๋” ๊ตฌ์ฒด์ ์ด๋‹ค.

์ฒซ์งธ, ๋ฐฑ๋ณธ์ด ๋ณด๋Š” ํ† ํฐ์ด ์ค„์—ˆ๋‹ค. FastWAM์˜ Wan ์‹œ๊ฐ expert๋Š” ์ƒ์„ฑ ํ•™์Šต ๋•Œ 294 ํ† ํฐ(ํ˜„์žฌ + ๋ฏธ๋ž˜ latent)์„ ์ฒ˜๋ฆฌํ•˜๊ณ , JAM์—์„œ๋Š” 98 ํ† ํฐ(ํ˜„์žฌ ํ”„๋ ˆ์ž„)๋งŒ ์ฒ˜๋ฆฌํ•œ๋‹ค(๋ถ€๋ก B.2). ์ƒ์„ฑ ๋ชฉ์ ํ•จ์ˆ˜๋Š” ๋ฏธ๋ž˜ latent๋ฅผ ๋ฐฑ๋ณธ ์ž…๋ ฅ์œผ๋กœ ๋„ฃ๊ณ  denoiseํ•˜๊ฒŒ ํ•˜๋ฏ€๋กœ, ๋ฐฑ๋ณธ ์šฉ๋Ÿ‰์˜ ์ผ๋ถ€๊ฐ€ โ€œ๋ฏธ๋ž˜ ํ”„๋ ˆ์ž„ ํ† ํฐ์„ ๋‹ค๋ฃจ๋Š” ์ผโ€์— ์“ฐ์ธ๋‹ค. JAM์—์„œ๋Š” ๋ฏธ๋ž˜๊ฐ€ ๋ฐฑ๋ณธ ์ž…๋ ฅ์—์„œ ๋น ์ง€๊ณ  ์ถœ๋ ฅ ์ชฝ์˜ ๋ณ„๋„ head์—์„œ๋งŒ ๋“ฑ์žฅํ•œ๋‹ค. ๋ฐฑ๋ณธ์€ ํ˜„์žฌ ํ”„๋ ˆ์ž„ ํ† ํฐ์—๋งŒ ์ง‘์ค‘ํ•˜๋˜, ๊ทธ ํ† ํฐ์ด ๋ฏธ๋ž˜๋ฅผ ๋งžํž ์ˆ˜ ์žˆ์„ ๋งŒํผ ์ •๋ณด๋ฅผ ๋‹ด๋„๋ก ์••๋ ฅ์„ ๋ฐ›๋Š”๋‹ค.

๋‘˜์งธ, Wan์—์„œ ์ด๊ธด target์€ โ€œ์˜๋ฏธ ํ‘œํ˜„โ€์ด ์•„๋‹ˆ๋ผ VAE latent์˜€๋‹ค. Table 1์—์„œ ์‚ฌ์ „ํ•™์Šต Wan anchor(55.58%)๋Š” FastWAM(55.31%)๊ณผ ์‚ฌ์‹ค์ƒ ๊ฐ™๊ณ , ๋ฌด์ž‘์œ„ Wan anchor(58.63%)๊ฐ€ ์˜คํžˆ๋ ค ๋†’์œผ๋ฉฐ, ๊ฐ€์žฅ ๋†’์€ ๊ฒƒ์€ VAE ๋‹จ๋…(64.54%)์ด๋‹ค. VAE latent๋Š” FastWAM์ด denoiseํ•˜๋˜ ๋ฐ”๋กœ ๊ทธ ๊ณต๊ฐ„์ด๋‹ค. ๊ทธ๋ ‡๋‹ค๋ฉด FastWAM ๋Œ€๋น„ ์ฐจ์ด๋Š” โ€œํ”ฝ์…€ ์ƒ์„ฑ vs ํ‘œํ˜„ ์˜ˆ์ธกโ€์ด๋ผ๊ธฐ๋ณด๋‹ค โ€œ๊ฐ™์€ latent๋ฅผ ๋ฐฑ๋ณธ ์•ˆ์—์„œ denoiseํ•˜๋А๋ƒ, ํ˜„์žฌ ํ† ํฐ์—์„œ ๋ณ„๋„ head๋กœ regressionํ•˜๋А๋ƒโ€์— ๊ฐ€๊น๋‹ค(ํ•„์ž ํ•ด์„). ์ €์ž๋“ค๋„ Wan VAE latent๊ฐ€ 32 ์Šคํ… ์ฐฝ์˜ ๋ณ€ํ™”๋ฅผ ์ด๋ฏธ ๋‹ด๊ณ  ์žˆ๋‹ค๋Š” ์ ์„ ๊ฐ€๋Šฅํ•œ ์„ค๋ช…์œผ๋กœ ๋“ ๋‹ค(ยง4.2).

์…‹์งธ, anchor์˜ ์ง€์‹์€ ๊ฑฐ์˜ ํ•„์š” ์—†๊ณ  ๋ฐฑ๋ณธ์˜ ์ง€์‹์€ ํ•„์š”ํ•˜๋‹ค. Table 2์—์„œ ๋ฐฑ๋ณธ์ด ์‚ฌ์ „ํ•™์Šต์ด๋ฉด anchor๊ฐ€ ๋ฌด์ž‘์œ„์—ฌ๋„ 97.65%, ๋ฐฑ๋ณธ์ด ๋ฌด์ž‘์œ„๋ฉด anchor๊ฐ€ ๋ฌด์—‡์ด๋“  71~84%๋กœ ๋ฌด๋„ˆ์ง„๋‹ค. ์ €์ž๋“ค์˜ ํ‘œํ˜„์œผ๋กœ๋Š” ๊ณ ์ •๋œ ๋ฌด์ž‘์œ„ anchor๋„ โ€œ๊ธฐ๋ก๋œ ๋ฏธ๋ž˜์˜ ์•ˆ์ •์ ์ด๊ณ  ์ž…๋ ฅ ์˜์กด์ ์ธ featureโ€๋ฅผ ์ œ๊ณตํ•œ๋‹ค. ๋ฌด์ž‘์œ„ ๋„คํŠธ์›Œํฌ๊ฐ€ ์ข‹์€ target์ด ๋  ์ˆ˜ ์žˆ๋‹ค๋Š” ๊ฒƒ์€ Random Network Distillation์ด๋‚˜ โ€œRandom teachers are good teachersโ€์—์„œ ์•Œ๋ ค์ง„ ํ˜„์ƒ์ด๊ณ , ๋ถ€๋ก E๋„ ์ด ๊ณ„๋ณด๋ฅผ ์ธ์šฉํ•œ๋‹ค. ์ •๋ฆฌํ•˜๋ฉด ์ด๋“์˜ ์›์ฒœ์€ target์˜ ์˜๋ฏธ ํ’ˆ์งˆ๋ณด๋‹ค โ€œํ˜„์žฌ feature๊ฐ€ ๋ฏธ๋ž˜์™€ ๊ฒฐ์ •์ ์œผ๋กœ ์—ฐ๊ฒฐ๋œ ์‹ ํ˜ธ๋ฅผ ๋งžํžˆ๋„๋ก ๊ฐ•์ œ๋œ๋‹คโ€๋Š” ๊ตฌ์กฐ ์ชฝ์— ์žˆ์–ด ๋ณด์ธ๋‹ค.

๋„ท์งธ, ํ˜„์žฌ ์œ„์น˜ target์€ ํ•ด๋กญ๋‹ค. Wan์—์„œ offset [0]๋งŒ ๊ฐ๋…ํ•˜๋ฉด LIBERO-Plus 44.62%๋กœ ๊ฐ๋…์ด ์•„์˜ˆ ์—†๋Š” FastWAM action-only(45.55%)๋ณด๋‹ค ๋‚ฎ๋‹ค. ์ €์ž๋“ค์€ Wan anchor๊ฐ€ bidirectional attention์œผ๋กœ [0] ์œ„์น˜์—๋„ ๋ฏธ๋ž˜ ๋งฅ๋ฝ์„ ์„ž์„ ์ˆ˜ ์žˆ๋‹ค๊ณ  ์“ฐ์ง€๋งŒ, ๊ฒฐ๊ณผ์ ์œผ๋กœ โ€œํ˜„์žฌ๋ฅผ ๋‹ค์‹œ ๋งžํžˆ๋Š”โ€ ์‹ ํ˜ธ๋Š” ํ–‰๋™์— ์“ธ ์ •๋ณด๋ฅผ ๋Š˜๋ฆฌ์ง€ ๋ชปํ•œ๋‹ค. ๋ฏธ๋ž˜ ์œ„์น˜๋ฅผ ๋ช…์‹œ์ ์œผ๋กœ ๊ฒจ๋ƒฅํ•ด์•ผ ํšจ๊ณผ๊ฐ€ ๋‚œ๋‹ค.

์‹คํ—˜ ์„ค์ • (ยง4.1, ๋ถ€๋ก A.1)

ํ•ญ๋ชฉ ๋‚ด์šฉ
ํ•˜๋“œ์›จ์–ดยท์ •๋ฐ€๋„ ๋ชจ๋“  ์‹คํ–‰ NVIDIA GH200 8์žฅ, bf16 mixed precision
optimizer AdamW (0.9, 0.95), weight decay 0.01, lr 10^{-4}, cosine decay + 5% warmup, gradient clip 1.0
LIBERO Spatial/Object/Goal/Long 40 ํƒœ์Šคํฌ ร— 50 rollout. FastWAM recipe 20k update, ImageWAM recipe(๊ทธ ์™ธ ๋ฐฑ๋ณธ) 21.7k update, global batch 128, 224ร—224 ์นด๋ฉ”๋ผ 2๋Œ€
LIBERO-Plus ImageWAM์˜ 15% subset ํ”„๋กœํ† ์ฝœ. suite๋ณ„ seeded draw๋กœ ๊ณ ์ • manifest 1,506 ํƒœ์Šคํฌ(Spatial 361 ยท Object 378 ยท Goal 389 ยท Long 378), ํƒœ์Šคํฌ๋‹น 1ํšŒ, task-weighted micro-average
RoboTwin 2.0 50 bimanual ํƒœ์Šคํฌ, Clean/Randomized ร— ํƒœ์Šคํฌ๋‹น 100 trial, unseen instruction, SAPIEN 3.0.0b1. 5 epoch: FastWAM 29,355 update(batch 1,024), ImageWAM recipe 69,900 update(batch 384)
์‚ฌ์ „ํ•™์Šต ์ €์ž ์ •์ฑ…์€ ๋ชจ๋‘ ๋กœ๋ด‡ ๋ฐ์ดํ„ฐ embodied pretraining ์—†์Œ(ํ‘œ์˜ P.T. ์—ด ร—)

๊ฐ ์ƒ์„ฑ ์ •์ฑ…๊ณผ ๊ทธ foresight ๋Œ€์‘ ์ •์ฑ…์€ ๋ฐ์ดํ„ฐ, ์ •์ฑ… ์•„ํ‚คํ…์ฒ˜, optimizer, ํ•™์Šต ์˜ˆ์‚ฐ, ํ‰๊ฐ€๊ธฐ๋ฅผ ๊ณต์œ ํ•œ๋‹ค. LIBERO-Plus๋Š” LIBERO๋กœ ํ•™์Šตํ•œ ์ •์ฑ…์„ ๊ต๋ž€ ์กฐ๊ฑด(camera, robot ์ดˆ๊ธฐ ์ƒํƒœ, language, light, background, sensor noise, layout)์—์„œ ํ‰๊ฐ€ํ•˜๋Š” robustness ๋ฒค์น˜๋งˆํฌ๋‹ค.

์‹คํ—˜ 1: ์ƒ์„ฑ์„ foresight๋กœ ๋ฐ”๊พธ๊ธฐ (ยง4.2, Table 1, Fig. 3ยท4ยท9)

์ƒ์„ฑ vs foresight, ์ •์ฑ… ์•„ํ‚คํ…์ฒ˜ ๊ณ ์ • (Table 1). ์„ฑ๊ณต๋ฅ  %. \lambda_F๋Š” Wan 0.6, Ovis-U1 0.3์œผ๋กœ ๊ณ ์ •ํ•˜๊ณ  offset ์ˆ˜๋กœ ๊ท ๋“ฑ ๋ถ„๋ฐฐ.
์ •์ฑ… \mathcal L_G \mathcal L_F future anchor readout LIBERO LIBERO-Plus
FastWAM (action-only) ร— ร— โ€“ โ€“ 95.25 45.55
FastWAM โœ“ ร— โ€“ โ€“ 97.00 55.31
JAM (Wan) ร— โœ“ Pretrained Wan [0] 95.80 44.62
JAM (Wan) ร— โœ“ Pretrained Wan [1, 2] 97.45 55.58
JAM (Wan) ร— โœ“ Pretrained Wan [0, 1, 2] 97.40 54.18
JAM (Wan) ร— โœ“ Random Wan [1, 2] 97.65 58.63
JAM (Wan) ร— โœ“ Pretrained VAE [1, 2] 98.15 64.54
ImageWAM (Ovis-U1, action-only) ร— ร— โ€“ โ€“ 90.20 61.35
ImageWAM (Ovis-U1) โœ“ ร— โ€“ โ€“ 92.90 65.27
JAM (Ovis-U1) ร— โœ“ Pretrained Yak [0] 94.55 69.65
JAM (Ovis-U1) ร— โœ“ Pretrained Yak [1] 96.75 79.95
JAM (Ovis-U1) ร— โœ“ Pretrained Yak [0, 1] 95.00 72.58
JAM (Ovis-U1) ร— โœ“ Random Yak [1] 95.55 72.51
JAM (Ovis-U1) ร— โœ“ Pretrained VAE [1] 95.75 76.43

์ด ํ‘œ์—์„œ ์ฝ์„ ๊ฒƒ์€ ์„ธ ์ค„์ด๋‹ค.

  1. ๋ชฉ์ ํ•จ์ˆ˜ ๊ต์ฒด ์ž์ฒด์˜ ํšจ๊ณผ๋Š” ๋ฐฑ๋ณธ์— ๋”ฐ๋ผ ํฌ๊ธฐ๊ฐ€ ๋‹ค๋ฅด๋‹ค. ์‚ฌ์ „ํ•™์Šต Wan anchor + ๋ฏธ๋ž˜ offset์ด๋ฉด FastWAM ๋Œ€๋น„ LIBERO 97.00 โ†’ 97.45, LIBERO-Plus 55.31 โ†’ 55.58๋กœ ๊ฑฐ์˜ ๊ฐ™๋‹ค. ๋ฐ˜๋ฉด Ovis-U1์€ ๊ฐ™์€ ์กฐ๊ฑด(์‚ฌ์ „ํ•™์Šต anchor + ๋ฏธ๋ž˜ offset)์—์„œ LIBERO-Plus 65.27 โ†’ 79.95๋กœ 14.68%p ์˜ค๋ฅธ๋‹ค. Ovis-U1์—์„œ๋Š” ์ƒ์„ฑ ์†์‹ค์ด action-only ๋Œ€๋น„ +3.92%p๋งŒ ์ฃผ๋Š”๋ฐ foresight๋Š” +18.60%p๋ฅผ ์ค€๋‹ค(ํ•„์ž ๊ณ„์‚ฐ).
  2. offset์ด ๋ชฉ์ ํ•จ์ˆ˜ ๊ต์ฒด๋ณด๋‹ค ํฌ๋‹ค. Wan [0] โ†’ [1,2]๋กœ 44.62 โ†’ 55.58, ๋‹ค์‹œ [0]์„ ๋”ํ•˜๋ฉด 54.18. Ovis-U1 [0] โ†’ [1]๋กœ 69.65 โ†’ 79.95. ๋‘ ๋ฐฑ๋ณธ ๋ชจ๋‘ ํ˜„์žฌ ์œ„์น˜๋ฅผ ์„ž์œผ๋ฉด ๋‚˜๋น ์ง„๋‹ค.
  3. ์„ ํ˜ธ target์€ ์•„ํ‚คํ…์ฒ˜๋งˆ๋‹ค ๋‹ค๋ฅด๋‹ค. Wan์€ VAE ๋‹จ๋…(64.54) > ๋ฌด์ž‘์œ„ Wan(58.63) > ์‚ฌ์ „ํ•™์Šต Wan(55.58), Ovis-U1์€ ์‚ฌ์ „ํ•™์Šต Yak(79.95) > VAE(76.43) > ๋ฌด์ž‘์œ„ Yak(72.51). ์ €์ž๋“ค์˜ ๊ฐ€๋Šฅํ•œ ์„ค๋ช…์€ target์˜ ์‹œ๊ฐ„ ์ •๋ณด๋‹ค. Wan์€ 32 ์Šคํ… action chunk ์œ„์˜ strided ์ฐฝ์„ ์ธ์ฝ”๋”ฉํ•˜๋ฏ€๋กœ VAE latent๊ฐ€ ์ด๋ฏธ ์žฅ๋ฉด ๋ณ€ํ™”๋ฅผ ๋”ฐ๋ผ๊ฐ€๊ณ , Yak์€ 16 ์Šคํ… ๋์  ํ•œ ์žฅ๋งŒ ์ธ์ฝ”๋”ฉํ•˜๋ฏ€๋กœ ์‚ฌ์ „ํ•™์Šต ์ธต์ด ๊ทธ๊ฒƒ์„ ํ˜„์žฌ ๊ด€์ธกยท์ง€์‹œ์™€ ์—ฐ๊ฒฐํ•ด ์ค˜์•ผ ํ•œ๋‹ค๋Š” ๊ฒƒ์ด๋‹ค.

Fig. 3 โ€” LIBERO-Long์—์„œ ImageWAM๊ณผ JAM(๋‘˜ ๋‹ค Ovis-U1)์˜ ๋งค์นญ๋œ rollout. (a) โ€œPut the black bowl in the bottom drawer and close itโ€: ImageWAM์€ ์„œ๋ž์„ ๋‹ซ์ง€ ๋ชปํ•œ๋‹ค. (b) โ€œPut both moka pots on the stoveโ€: ImageWAM์€ ๋‘ ๋ฒˆ์งธ ํฌํŠธ๋ฅผ ๋„˜์–ด๋œจ๋ฆฐ๋‹ค.

ํ”„๋กœ์ ํŠธ ํŽ˜์ด์ง€๋Š” Fig. 3์˜ ๋‘ ๊ณผ์ œ๋ฅผ ๊ฐ™์€ ์ดˆ๊ธฐ ์ƒํƒœ์—์„œ ์‹œ์ž‘ํ•œ ํ‰๊ฐ€๊ธฐ ๋…นํ™”๋กœ ๋ณด์—ฌ ์ค€๋‹ค. ์™ผ์ชฝ์ด ์ƒ์„ฑ ์†์‹ค๋กœ ํ•™์Šตํ•œ ImageWAM, ์˜ค๋ฅธ์ชฝ์ด foresight ์†์‹ค๋กœ ํ•™์Šตํ•œ JAM์ด๊ณ , ์งง์€ ์ชฝ ํด๋ฆฝ์€ ๋๋‚œ ๋’ค ๋งˆ์ง€๋ง‰ ํ”„๋ ˆ์ž„์— ๋ฉˆ์ถ˜๋‹ค.


LIBERO-Long "Put both moka pots on the stove"(Task 8 ยท Trial 9). ImageWAM(์™ผ์ชฝ)์€ ๋‘ ๋ฒˆ์งธ ํฌํŠธ๋ฅผ ๋„˜์–ด๋œจ๋ฆฌ๊ณ  JAM(์˜ค๋ฅธ์ชฝ)์€ ๋‘ ํฌํŠธ๋ฅผ ๋ชจ๋‘ ์Šคํ† ๋ธŒ์— ์˜ฌ๋ฆฐ๋‹ค. 2๋ฐฐ์†.


LIBERO-Long "Put the black bowl in the bottom drawer of the cabinet and close it"(Task 3 ยท Trial 21). ImageWAM์€ ์„œ๋ž์„ ์—ฐ ์ฑ„๋กœ ๋‘๊ณ  JAM์€ ๊ทธ๋ฆ‡์„ ๋„ฃ๊ณ  ์„œ๋ž์„ ๋‹ซ๋Š”๋‹ค. 2๋ฐฐ์†.

ํ˜„์žฌ feature์—์„œ ๋ฏธ๋ž˜๊ฐ€ ์–ผ๋งˆ๋‚˜ ์ฝํžˆ๋‚˜(Fig. 4ยท9, ๋ถ€๋ก D.6). ์ €์ž๋“ค์€ ๊ฐ ๊ณ ์ •๋œ Ovis-U1 ์ •์ฑ… ์œ„์— ํ† ํฐ๋ณ„ 2์ธต MLP probe ๋‘ ๊ฐœ๋ฅผ ํ•™์Šตํ•ด ํ˜„์žฌ ํ”„๋ ˆ์ž„๊ณผ 16 ์Šคํ… ๋’ค ํ”„๋ ˆ์ž„์„ ๋ณต์›ํ•˜๊ณ (ํ•™์Šตยทํ‰๊ฐ€ ์—ํ”ผ์†Œ๋“œ๋Š” ๋ถ„๋ฆฌ), ๋‘ ๋ณต์›์˜ ๋ถ€ํ˜ธ ์žˆ๋Š” ํœ˜๋„ ์ฐจ์ด๋ฅผ ๊ธฐ๋ก๋œ ๋ณ€ํ™”์™€ ๋น„๊ตํ•œ๋‹ค.


Fig. 4 โ€” โ€œput both moka pots on the stoveโ€์˜ ๋ณต์›๋œ ๋ฏธ๋ž˜ ๋ณ€ํ™”. ์™ผ์ชฝ๋ถ€ํ„ฐ ํ˜„์žฌ ํ”„๋ ˆ์ž„, ๊ธฐ๋ก๋œ ๋ฏธ๋ž˜(16 ์Šคํ… ๋’ค), ๊ธฐ๋ก๋œ ๋ณ€ํ™”, ๊ทธ๋ฆฌ๊ณ  ImageWAMยทJAM offset [0]ยทJAM offset [1] feature์—์„œ probe๋กœ ์ฝ์–ด ๋‚ธ ๋ณ€ํ™”. ๋นจ๊ฐ•์€ ๋ฐ์•„์ง€๋Š” ํ”ฝ์…€, ํŒŒ๋ž‘์€ ์–ด๋‘์›Œ์ง€๋Š” ํ”ฝ์…€(๊ณตํ†ต ์Šค์ผ€์ผ).

์ด ์˜ˆ์‹œ์—์„œ๋Š” offset [1]๋กœ ํ•™์Šตํ•œ JAM์˜ feature๊ฐ€ ํŒ”์ด ์ด๋™ํ•ด ๊ฐˆ ์˜ค๋ฅธ์ชฝ ์œ„ ์˜์—ญ์˜ ๋ฐ์•„์ง(๋นจ๊ฐ•)์„ ๊ฐ€์žฅ ์ง„ํ•˜๊ฒŒ ๋ณต์›ํ•œ๋‹ค. ๊ทธ๋Ÿฐ๋ฐ ์ •๋Ÿ‰ ์š”์•ฝ์€ ์ฐจ์ด๊ฐ€ ์ž‘๋‹ค.


Fig. 9 โ€” ๋ฏธ๋ž˜ ํ”„๋ ˆ์ž„ ํ‰๊ท  PSNR. Current(ํ˜„์žฌ ํ”„๋ ˆ์ž„์„ ๋ฏธ๋ž˜ ์˜ˆ์ธก์œผ๋กœ ๊ทธ๋Œ€๋กœ ์“ด ๊ธฐ์ค€) 16.13, ImageWAM 19.43, JAM [0] 19.29, JAM [1] 19.57 dB. ์„ธ๋กœ์ถ•์€ 15 dB์—์„œ ์‹œ์ž‘ํ•œ๋‹ค.

JAM [1]๊ณผ ImageWAM์˜ ์ฐจ์ด๋Š” 0.14 dB์ด๊ณ , ๋ถ„์‚ฐ์ด๋‚˜ ์—ํ”ผ์†Œ๋“œ ์ˆ˜๋Š” ๊ทธ๋ฆผ์— ์—†๋‹ค. ์„ธ๋กœ์ถ•์ด 15์—์„œ ์‹œ์ž‘ํ•ด ๋ง‰๋Œ€ ๋†’์ด ์ฐจ๊ฐ€ ์ปค ๋ณด์ธ๋‹ค. ์ €์ž๋“ค์˜ ์„œ์ˆ ์€ โ€œslightly aboveโ€์™€ โ€œsuggestโ€๋กœ ์ ˆ์ œ๋ผ ์žˆ์œผ๋‹ˆ, Fig. 4์˜ ์„ ๋ช…ํ•œ ๋Œ€๋น„๋Š” ํ•œ ์žฅ๋ฉด์˜ ์ •์„ฑ ์˜ˆ์‹œ๋กœ ์ฝ๋Š” ๊ฒƒ์ด ๋งž๋‹ค. ํ”„๋กœ์ ํŠธ ํŽ˜์ด์ง€๋Š” ๊ฐ™์€ ์—ํ”ผ์†Œ๋“œ ์ „์ฒด์— ๋Œ€ํ•ด probe ๊ฒฐ๊ณผ๋ฅผ ํ”„๋ ˆ์ž„ ๋‹จ์œ„๋กœ ๋ณด์—ฌ ์ฃผ๋Š” ์˜์ƒ์„ ํ•จ๊ป˜ ์‹ฃ๋Š”๋‹ค. ์žฅ๋ฉด๋งˆ๋‹ค ๋Œ€๋น„์˜ ํฌ๊ธฐ๊ฐ€ ๋‹ฌ๋ผ์ง€๋Š” ๊ฒƒ์„ ๋ณผ ์ˆ˜ ์žˆ๋‹ค.


ํ”„๋กœ์ ํŠธ ํŽ˜์ด์ง€์˜ probe ์˜์ƒ. Fig. 4์˜ ์—ฌ์„ฏ ์—ด(ํ˜„์žฌ ยท 16 ์Šคํ… ๋’ค ๊ธฐ๋ก ยท ๊ธฐ๋ก๋œ ๋ณ€ํ™” ยท ImageWAM ยท JAM [0] ยท JAM [1])์„ ์—ํ”ผ์†Œ๋“œ ์ง„ํ–‰์— ๋”ฐ๋ผ ๋ณด์—ฌ ์ค€๋‹ค.

์‹คํ—˜ 2: ๋ฐฑ๋ณธ์„ ๋ฐ”๊พธ๊ธฐ (ยง4.3, Table 2ยท3ยท14, Fig. 5ยท8)

๋ฐฑ๋ณธ๊ณผ anchor์˜ ์—ญํ•  ๋ถ„๋ฆฌ(Table 2, ๋ถ€๋ก D.1). JAM(Wan)์—์„œ ๋ฐฑ๋ณธ๊ณผ future anchor๋ฅผ ๊ฐ๊ฐ ์‚ฌ์ „ํ•™์Šต/๋ฌด์ž‘์œ„๋กœ ๋ฐ”๊พผ 2ร—2 ์‹คํ—˜์ด๋‹ค(action expert, ์ •์ฑ… VAE, ํ…์ŠคํŠธ conditioner๋Š” ๋ชจ๋“  ์„ค์ •์—์„œ ์‚ฌ์ „ํ•™์Šต ์ƒํƒœ).

(Table 2 / Table 17)
๋ฐฑ๋ณธ anchor Spatial Object Goal Long LIBERO ํ‰๊ท 
Pretrained Pretrained 98.2 100.0 97.6 94.0 97.45
Pretrained Random 98.0 99.8 97.0 95.8 97.65
Random Random 85.6 98.0 88.2 65.8 84.40
Random Pretrained 76.0 95.0 70.4 43.2 71.15

๋ฐฑ๋ณธ ์‚ฌ์ „ํ•™์Šต์€ ๋‘ ๋น„๊ต ๋ชจ๋‘์—์„œ ์ค‘์š”ํ•˜๊ณ  anchor๋Š” ๋ฌด์ž‘์œ„์—ฌ๋„ ๋œ๋‹ค. ๋‹ค๋งŒ ๋งˆ์ง€๋ง‰ ๋‘ ํ–‰์˜ ์—ญ์ „, ์ฆ‰ ๋ฌด์ž‘์œ„ ๋ฐฑ๋ณธ์—์„œ๋Š” ์‚ฌ์ „ํ•™์Šต anchor๊ฐ€ ๋ฌด์ž‘์œ„ anchor๋ณด๋‹ค 13.25%p ๋‚˜์˜๋‹ค(Long์€ 65.8 โ†’ 43.2)๋Š” ๊ฒฐ๊ณผ์— ๋Œ€ํ•ด์„œ๋Š” ๋…ผ๋ฌธ์— ์„ค๋ช…์ด ์—†๋‹ค.

๋ฐฑ๋ณธ ๊ณ„์—ด ๊ฐ„ ๋น„๊ต(Table 3)์™€ action-only ๋Œ€์กฐ๊ตฐ(Table 14).

(Table 3) ์•„ํ‚คํ…์ฒ˜์™€ action ์ธํ„ฐํŽ˜์ด์Šค๊ฐ€ ๋‹ฌ๋ผ ์‚ฌ์ „ํ•™์Šต ํšจ๊ณผ๋งŒ ๋ถ„๋ฆฌํ•  ์ˆ˜๋Š” ์—†๋‹ค๋Š” ๊ฒƒ์ด ์› ์บก์…˜์˜ ๋‹จ์„œ๋‹ค.
์ •์ฑ… ๋ฐฑ๋ณธ ํฌ๊ธฐ LIBERO LIBERO-Plus
ImageWAM (Ovis-U1), ์ƒ์„ฑ ์†์‹ค 1.13B 92.90 65.27
JAM (Ovis-U1) 1.13B 96.75 79.95
JAM (DINOv2-B) 86.6M 95.50 68.19
JAM (DINOv2-L) 304M 96.30 70.39
JAM (Qwen3.5-2B) 2.0B 96.55 81.14
LIBERO-Plus action-only ๋Œ€์กฐ๊ตฐ (Table 14). ๊ตต์€ ๊ธ€์”จ๋Š” ๋ฐฑ๋ณธ ์Œ ์•ˆ์—์„œ ๋†’์€ ์ชฝ.
๋ฐฉ๋ฒ• Camera Robot Language Light Background Noise Layout ํ‰๊ท 
Qwen3.5-2B action-only 80.00 68.02 72.54 92.55 54.84 93.02 75.45 77.03
JAM (Qwen3.5-2B) 80.83 61.42 71.72 93.79 81.72 94.19 84.55 81.14
DINOv2-L action-only 37.92 32.99 86.48 86.96 85.48 78.29 72.73 68.26
JAM (DINOv2-L) 55.42 39.09 80.74 85.71 77.42 81.40 73.18 70.39

foresight๋Š” Qwen3.5-2B์—์„œ +4.11%p, DINOv2-L์—์„œ +2.13%p๋ฅผ ์ค€๋‹ค(ํ•„์ž ๊ณ„์‚ฐ). ๋‘ ๋ฐฑ๋ณธ ๋ชจ๋‘ ์ผ๋ถ€ ๋ฒ”์ฃผ์—์„œ๋Š” action-only๊ฐ€ ๋” ๋‚ซ๋‹ค(Qwen์€ robotยทlanguage, DINOv2-L์€ languageยทlightยทbackground). ์—ฌ๊ธฐ์„œ ๋†“์น˜๊ธฐ ์‰ฌ์šด ์‚ฌ์‹ค์ด ํ•˜๋‚˜ ์žˆ๋‹ค. Qwen3.5-2B๋Š” foresight ์—†์ด๋„ LIBERO-Plus 77.03%๋กœ, ์ƒ์„ฑ ์†์‹ค์„ ์“ด ImageWAM(65.27%)๊ณผ JAM(Wan)(64.54%)๋ณด๋‹ค ์ด๋ฏธ ๋†’๋‹ค. Table 3์˜ โ€œ๋„ค ๋ฐฑ๋ณธ ๋ชจ๋‘ ImageWAM์„ ๋„˜๋Š”๋‹คโ€๋Š” ๊ฒฐ๊ณผ์—๋Š” foresight์˜ ๋ชซ๊ณผ ๋ฐฑ๋ณธยท์ธํ„ฐํŽ˜์ด์Šค์˜ ๋ชซ์ด ์„ž์—ฌ ์žˆ๊ณ , ๋…ผ๋ฌธ๋„ ๊ฒฐ๋ก ์—์„œ โ€œthe cross-backbone comparison does not isolate the effect of pretraining aloneโ€์ด๋ผ๊ณ  ์ ๋Š”๋‹ค.

Action-to-visual attention(Fig. 5ยท8, ๋ถ€๋ก D.5). action expert๊ฐ€ ์‹œ๊ฐ ํ† ํฐ์— ์ฃผ๋Š” attention์„ head, 16๊ฐœ action query, 10๊ฐœ denoising ์Šคํ…, ์ธต์— ๋Œ€ํ•ด ํ‰๊ท ํ•˜๊ณ , 20๊ฐœ LIBERO ํƒœ์Šคํฌ์˜ 100๊ฐœ ๊ธฐ์ค€ ํ”„๋ ˆ์ž„ ํ‰๊ท  ์ง€๋„๋ฅผ ๋บ€ ์–‘์ˆ˜ ๋ถ€๋ถ„๋งŒ ๋ณด์ธ ๊ฒƒ์ด๋‹ค.


Fig. 5 โ€” โ€œput the white mug on the left plate and put the yellow and white mug on the right plateโ€์˜ agent view(์œ„)์™€ wrist view(์•„๋ž˜). (a) Ovis-U1 ๋ฐฑ๋ณธ์˜ ImageWAMยทJAM๊ณผ ๊ทธ ์ฐจ์ด ์ง€๋„(๋นจ๊ฐ• = JAM์ด ๋” ๋†’์Œ, ํŒŒ๋ž‘ = ImageWAM์ด ๋” ๋†’์Œ). (b) DINOv2-LยทQwen3.5-2B ๋ฐฑ๋ณธ์˜ JAM. ๋ง‰๋Œ€๋Š” 100 ํ”„๋ ˆ์ž„ ํ‰๊ท  attention ๋น„์ค‘์œผ๋กœ ์œ„๋ถ€ํ„ฐ action(์ดˆ๋ก) ยท image(๋‚จ์ƒ‰) ยท text(์ฃผํ™ฉ) ํ‚ค.

๊ทธ๋ฆผ ์† ๋ง‰๋Œ€์˜ ์ˆซ์ž๋ฅผ ์˜ฎ๊ธฐ๋ฉด action/image/text ๋น„์ค‘์ด ImageWAM 21/29/50, JAM(Ovis-U1) 20/27/54, JAM(DINOv2-L) 21/71/8, JAM(Qwen3.5-2B) 57/27/16์ด๋‹ค. ๋ณธ๋ฌธ์˜ ์š”์•ฝ(DINOv2-L์€ image, JAM(Ovis-U1)์€ text, Qwen3.5-2B๋Š” action ํ‚ค์— ๊ฐ€์žฅ ํฐ ๋น„์ค‘)๊ณผ ์ผ์น˜ํ•œ๋‹ค. ๊ฐ™์€ Ovis-U1 ๋ฐฑ๋ณธ์—์„œ ImageWAM๊ณผ JAM์˜ ๋น„์ค‘์€ ๊ฑฐ์˜ ๊ฐ™๊ณ , ์ฐจ์ด๋Š” ์–ด๋””๋ฅผ ๋ณด๋А๋ƒ์— ์žˆ๋‹ค. ์ด ์˜ˆ์‹œ์—์„œ ImageWAM์€ ๊ทธ๋ฆฌํผ๊ฐ€ ์ฅ” ๋จธ๊ทธ์—, JAM์€ ์ง€์‹œ๋ฌธ์ด ์–ธ๊ธ‰ํ•œ ๋‘ ๋ฒˆ์งธ ๋จธ๊ทธ์— ๋” ์ง‘์ค‘ํ•œ๋‹ค. ์ €์ž๋“ค์€ ์ด ์ง€๋„๊ฐ€ attention์ด โ€œ์–ด๋””๋กœ ๊ฐ€๋Š”์ง€โ€๋ฅผ ๊ธฐ์ˆ ํ•  ๋ฟ์ด๋ผ๊ณ  ์„ ์„ ๊ธ‹๋Š”๋‹ค.


Fig. 8 โ€” agent view์˜ action-to-visual attention์„ LIBERO-Goal, LIBERO-Long, LIBERO-Plus camera, LIBERO-Plus layout ๋„ค ์žฅ๋ฉด์— ๋Œ€ํ•ด ํ™•์žฅํ•œ ๊ฒƒ. ๋ณธ๋ฌธ์— ๋”ฐ๋ฅด๋ฉด Ovis-U1์—์„œ ImageWAM์€ ๊ทธ๋ฆฌํผ์™€ ์ฅ” ๋ฌผ์ฒด๋ฅผ, JAM์€ ์ง€์‹œ๋œ ๋ฐฐ์น˜ ๋ชฉํ‘œ(ํ–‰ ์ˆœ์„œ๋Œ€๋กœ rack, plate, ๋‘ ๋ฒˆ์งธ plate, microwave)๋ฅผ ๊ฐ•์กฐํ•œ๋‹ค.

ํ”„๋กœ์ ํŠธ ํŽ˜์ด์ง€์˜ attention ์˜์ƒ. Fig. 5์˜ ์—ํ”ผ์†Œ๋“œ๋ฅผ ์ง„ํ–‰์— ๋”ฐ๋ผ ๋ณด์—ฌ ์ฃผ๋ฉฐ, ๋ง‰๋Œ€ ๋น„์ค‘๋„ ํ”„๋ ˆ์ž„๋งˆ๋‹ค ๋ฐ”๋€๋‹ค.

์‹คํ—˜ 3: ๊ณต๊ฐœ ์‹œ์Šคํ…œ๊ณผ์˜ ๋น„๊ต (ยง4.4, Table 4ยท5ยท6ยท18ยท19, Fig. 6ยท7)

์—ฌ๊ธฐ์„œ๋ถ€ํ„ฐ๋Š” matched ๋น„๊ต๊ฐ€ ์•„๋‹ˆ๋‹ค. โ€  ํ‘œ์‹œ ํ–‰๋งŒ ์ €์ž๋“ค์ด ํ‰๊ฐ€ํ–ˆ๊ณ , ๋‚˜๋จธ์ง€ ํ–‰์€ ์› ๋…ผ๋ฌธ์˜ ๊ฐ’๊ณผ ํ”„๋กœํ† ์ฝœ์„ ๊ทธ๋Œ€๋กœ ์˜ฎ๊ฒผ๋‹ค.

RoboTwin 2.0๊ณผ LIBERO (Table 4ยท5ยท18ยท19์—์„œ ๋ฐœ์ทŒ). LIBERO ํ‘œ์—๋Š” OpenVLA 76.5, GR00T N1 93.9๋„ ์žˆ๋‹ค.
๋ฐฉ๋ฒ• P.T. RoboTwin 2.0 (Clean / Rand. / ํ‰๊ท ) LIBERO ํ‰๊ท 
\pi_0 โœ“ 65.92 / 58.40 / 62.16 94.1
\pi_{0.5} โœ“ 82.74 / 76.76 / 79.75 96.9
JAM (Qwen3.5-2B)โ€  ร— 89.70 / 89.86 / 89.78 96.55
Motus โœ“ 88.66 / 87.02 / 87.84 97.7
LingBot-VA โœ“ 92.90 / 91.50 / 92.20 98.5
FastWAMโ€  ร— 91.22 / 91.20 / 91.21 (๊ณต๊ฐœ ๊ฐ€์ค‘์น˜) 97.00 (์žฌํ•™์Šต)
JAM (Wan)โ€  ร— 91.70 / 91.18 / 91.44 98.15
ImageWAM (Ovis-U1)โ€  ร— 89.14 / 88.78 / 88.96 (๋กœ์ปฌ ํ•™์Šต) 92.90
ImageWAM (FLUX.2)โ€  ร— 92.84 / 93.04 / 92.94 (๊ณต๊ฐœ ๊ฐ€์ค‘์น˜) โ€“
JAM (Ovis-U1)โ€  ร— 90.34 / 89.46 / 89.90 96.75
JAM (FLUX.2)โ€  ร— 92.88 / 93.28 / 93.08 โ€“

RoboTwin์—์„œ JAM(FLUX.2) 93.1%๊ฐ€ ํ‘œ์˜ ์ตœ๊ณ ์ด๊ณ , ๋น„๋””์˜ค ์ƒ์„ฑ๊ธฐ ๊ณ„์—ด์—์„œ๋Š” embodied pretraining์„ ์“ด LingBot-VA๊ฐ€ JAM(Wan)๋ณด๋‹ค 0.8%p ์•ž์„ ๋‹ค. LIBERO์—์„œ๋Š” \pi_{0.5}์™€ LingBot-VA๊ฐ€ JAM(Qwen3.5-2B)ยทJAM(Wan)๋ณด๋‹ค 0.3%p ์•ž์„ ๋‹ค. ์ €์ž๋“ค์€ JAM(Qwen3.5-2B)์ด RoboTwin์—์„œ JAM(FLUX.2)๋ณด๋‹ค 3.30%p ๋’ค์ง„๋‹ค๋Š” ์ ๋„ ๋ฐํžŒ๋‹ค.

LIBERO-Plus ๊ต๋ž€ ๋ฒ”์ฃผ๋ณ„ ์„ฑ๊ณต๋ฅ  (Table 6). ๊ณต๊ฐœ ํ–‰์€ ์› ํ”„๋กœํ† ์ฝœ(์ „์ฒด ์„ธํŠธ), โ€  ํ–‰์€ 1,506 ํƒœ์Šคํฌ ๊ณ ์ • subset.
๋ฐฉ๋ฒ• P.T. Camera Robot Language Light Background Noise Layout ํ‰๊ท 
UniVLA โœ“ 1.8 46.2 69.6 69.0 81.0 21.2 31.9 42.9
OpenVLA-OFT โœ“ 56.4 31.9 79.5 88.7 93.3 75.8 74.2 69.6
\pi_0 โœ“ 13.8 6.0 58.8 85.0 81.4 79.0 68.9 53.6
\pi_0-Fast โœ“ 65.1 21.6 61.0 73.2 73.2 74.4 68.8 61.6
WorldVLA ร— 0.1 27.9 41.6 43.7 17.1 10.9 38.0 25.0
JAM (Qwen3.5-2B)โ€  ร— 80.8 61.4 71.7 93.8 81.7 94.2 84.5 81.1
FastWAMโ€  ร— 24.2 50.3 78.7 87.6 49.5 39.9 67.3 55.3
JAM (Wan)โ€  ร— 25.4 68.0 92.6 93.2 46.2 52.7 81.4 64.5
ImageWAM (Ovis-U1)โ€  ร— 57.9 54.8 67.6 78.9 66.7 68.6 65.0 65.3
JAM (Ovis-U1)โ€  ร— 83.8 49.2 84.8 95.7 80.1 91.9 72.3 79.9

JAM์€ ์„ธ ๊ทธ๋ฃน ๋ชจ๋‘์—์„œ ํ‰๊ท  ์ตœ๊ณ ์ด๊ณ , FastWAM๊ณผ ImageWAM(Ovis-U1)์„ ๊ฐ๊ฐ 9.2%p, 14.7%p ์•ž์„ ๋‹ค. ๋ฒ”์ฃผ๋ณ„๋กœ ์ง€๋Š” ๊ณณ์€ FastWAM ๋Œ€๋น„ background, ImageWAM ๋Œ€๋น„ robot ์ดˆ๊ธฐ ์ƒํƒœ๋ฟ์ด๋‹ค. Wan ๊ณ„์—ด์€ camera ๊ต๋ž€์—์„œ ๋‘ ์ •์ฑ… ๋ชจ๋‘ 25% ์•ˆํŒŽ์œผ๋กœ ์•ฝํ•˜๋‹ค๋Š” ์ ์ด ๋ˆˆ์— ๋ˆ๋‹ค.


Fig. 6 โ€” 1,506๊ฐœ ๊ณต์œ  ํƒœ์Šคํฌ์˜ ์ง์ง€์€ ๊ฒฐ๊ณผ. ์™ผ์ชฝ ๋ง‰๋Œ€๋Š” ๋ฒ ์ด์Šค๋ผ์ธ๋งŒ ์„ฑ๊ณตํ•œ ํƒœ์Šคํฌ ์ˆ˜, ์˜ค๋ฅธ์ชฝ ๋ง‰๋Œ€๋Š” JAM๋งŒ ์„ฑ๊ณตํ•œ ํƒœ์Šคํฌ ์ˆ˜. (a) Ovis-U1, (b) Wan(VAE-direct [1,2]).

Fig. 6์˜ ๋ฒ”์ฃผ๋ณ„ ์ˆซ์ž๋ฅผ ๋”ํ•˜๋ฉด Ovis-U1์€ ๋ฒ ์ด์Šค๋ผ์ธ๋งŒ 89 / JAM๋งŒ 310, Wan์€ 111 / 250์œผ๋กœ, Table 20์˜ discordant ์ˆ˜์™€ ์ •ํ™•ํžˆ ์ผ์น˜ํ•œ๋‹ค(ํ•„์ž ๊ฒ€์‚ฐ). Ovis-U1์˜ ์ด๋“์€ camera(9 vs 71)์™€ sensor noise(7 vs 67)์—์„œ ๊ฐ€์žฅ ํฌ๊ณ , Wan์˜ ์ด๋“์€ robot ์ดˆ๊ธฐ ์ƒํƒœ(10 vs 45)์—์„œ ๊ฐ€์žฅ ํฌ๋ฉฐ camera๋Š” 31 vs 28๋กœ ๊ฑฐ์˜ ๋น„๊ธด๋‹ค.


Fig. 7 โ€” ์ถ”๊ฐ€ ๋งค์นญ rollout. (a)(b) Ovis-U1์˜ LIBERO-Plus cameraยทlayout ๊ต๋ž€, (c)(d) Wan์˜ RoboTwin 2.0 CleanยทRandomized. ๋ณธ๋ฌธ์— ๋”ฐ๋ฅด๋ฉด ImageWAM์€ (a)์—์„œ ํฌํŠธ๋ฅผ ๋“ค์ง€ ๋ชปํ•˜๊ณ  (b)์—์„œ ๊ฐ€์šด๋ฐ ๋Œ€์‹  ์œ„ ์„œ๋ž์„ ๋‹น๊ธฐ๋ฉฐ, FastWAM์€ (c)์—์„œ ์บ”์„ ๋„˜์–ด๋œจ๋ฆฌ๊ณ  (d)์—์„œ ์Œ“์€ ๋ธ”๋ก์„ ๋ฌด๋„ˆ๋œจ๋ฆฐ๋‹ค.

ํ”„๋กœ์ ํŠธ ํŽ˜์ด์ง€์˜ ๋‚˜๋จธ์ง€ ๋„ค ์Œ์ด Fig. 7์˜ ๋„ค ๊ณผ์ œ๋‹ค. RoboTwin ์Œ์˜ ๋ฒ ์ด์Šค๋ผ์ธ์€ ๋…ผ๋ฌธ Table 19์™€ ๊ฐ™์€ FastWAM ๊ณต๊ฐœ ๊ฐ€์ค‘์น˜๋‹ค.


LIBERO-Plus camera shift(Task 761) "Turn on the stove and put the moka pot on it". ์นด๋ฉ”๋ผ๊ฐ€ ์˜ฎ๊ฒจ์ง„ ์กฐ๊ฑด์—์„œ ImageWAM์€ ํฌํŠธ ์œ„์—์„œ ๋ฉˆ์ถ”๊ณ  JAM์€ ๊ณผ์ œ๋ฅผ ๋งˆ์นœ๋‹ค. 2๋ฐฐ์†.


LIBERO-Plus layout change(Task 1908) "Open the middle drawer of the cabinet". ImageWAM์€ ์œ„ ์„œ๋ž์„ ์—ด๊ณ  JAM์€ ๊ฐ€์šด๋ฐ ์„œ๋ž์„ ์—ฐ๋‹ค. 2๋ฐฐ์†.


RoboTwin 2.0 Clean "Put the can in the basket". FastWAM(๊ณต๊ฐœ ๊ฐ€์ค‘์น˜, ์™ผ์ชฝ)์€ ์บ”์„ ๋„˜์–ด๋œจ๋ฆฌ๊ณ  JAM(Wan)์€ ๋ฐ”๊ตฌ๋‹ˆ์— ๋„ฃ๋Š”๋‹ค. 2๋ฐฐ์†.


RoboTwin 2.0 Randomized "Stack the red, green, and blue blocks". FastWAM์€ 1,200 ์Šคํ… ์•ˆ์— ์Œ“๊ธฐ๋ฅผ ๋๋‚ด์ง€ ๋ชปํ•˜๊ณ  JAM(Wan)์€ ์„ธ ๋ธ”๋ก์„ ๋ชจ๋‘ ์Œ“๋Š”๋‹ค. 4๋ฐฐ์†.

์—ฌ์„ฏ ์Œ์€ ๋ชจ๋‘ โ€œ๋ฒ ์ด์Šค๋ผ์ธ ์‹คํŒจ / JAM ์„ฑ๊ณตโ€์ธ ์—ํ”ผ์†Œ๋“œ๋‹ค. ๊ฐ™์€ ์ดˆ๊ธฐ ์ƒํƒœ์—์„œ ์‹œ์ž‘ํ•ด ๊ณต์ •ํ•˜์ง€๋งŒ ๊ณ ๋ฅธ ์‚ฌ๋ก€์ด๊ณ , ๋ฐ˜๋Œ€ ๋ฐฉํ–ฅ์˜ discordant ์Œ(Ovis-U1 LIBERO-Plus์—์„œ 89๊ฑด)์€ ๋ณด์—ฌ ์ฃผ์ง€ ์•Š๋Š”๋‹ค.

์‹คํ—˜ 4: ํ•™์Šต ๋น„์šฉ (ยง4.5, ๋ถ€๋ก B, Table 9)

ํ•™์Šต ์ „์šฉ ํŒŒ๋ผ๋ฏธํ„ฐ, ์ƒ˜ํ”Œยทupdate๋‹น ์ถ”์ • online FLOPs, GH200 1์žฅ ์ธก์ • GPU ์‹œ๊ฐ„ (๋…ผ๋ฌธ v1 Table 9).
์ •์ฑ… target head ํŒŒ๋ผ๋ฏธํ„ฐ (M) anchor ํŒŒ๋ผ๋ฏธํ„ฐ (M) ์‹œ๊ฐ ํ–‰๋™ head target VAE ํ•ฉ๊ณ„ TFLOPs ๊ฐ์†Œ GPU ์‹œ๊ฐ„ (s/step) ๊ฐ์†Œ
FastWAM โ€“ โ€“ โ€“ 8.93 0.05 โ€“ โ€“ 6.54 15.52 โ€“ 0.789 โ€“
JAM (Wan) online 71 0 3.17 0.05 0.03 0 6.54 9.80 36.9% 0.532 32.6%
JAM (Wan) cached 71 0 3.17 0.05 0.03 0 0.93 4.19 73.0% 0.258 67.3%
ImageWAM (Ovis-U1) โ€“ โ€“ โ€“ 4.91 0.07 โ€“ โ€“ 0.83 5.81 โ€“ 0.473 โ€“
JAM (Ovis-U1) online 16.5 1,127 3.13 0.07 0.05 0.80 0.83 4.88 15.9% 0.403 14.8%
JAM (Ovis-U1) cached 16.5 0 3.13 0.07 0.05 0 0.41 3.67 36.8% 0.294 37.8%

FLOPs๋Š” ํ™œ์„ฑ ํŒŒ๋ผ๋ฏธํ„ฐ N, ํ† ํฐ T์— ๋Œ€ํ•ด forward๋ฅผ 2NT๋กœ ์ถ”์ •ํ•˜๊ณ , ํ•™์Šต ๋ธ”๋ก์€ forward 3๋ฐฐ(ImageWAM์˜ activation checkpointing์ด๋ฉด 4๋ฐฐ), ๊ณ ์ • anchor์™€ VAE๋Š” forward๋งŒ ์„ผ๋‹ค. attention score ํ•ญ, ํ…์ŠคํŠธ ์ธ์ฝ”๋”ฉ, optimizer update๋Š” ๋บ๋‹ค. GPU ์‹œ๊ฐ„์€ batch 16์—์„œ CUDA activity profiler๋กœ ์Ÿ€๋‹ค. Wan์˜ ์ธก์ • ๊ฐ์†Œ๊ฐ€ ์ถ”์ •์น˜๋ณด๋‹ค ์ž‘์€ ์ด์œ ๋ฅผ ์ €์ž๋“ค์€ normalizationยทactivationยทrotary embedding ์ปค๋„์ด ํ† ํฐ ์ˆ˜๋งŒํผ ์ค„์ง€ ์•Š๊ธฐ ๋•Œ๋ฌธ์ด๋ผ๊ณ  ์„ค๋ช…ํ•œ๋‹ค.

์ด ํ‘œ์˜ ๊ฐ์†Œ์œจ์€ ๋ชจ๋‘ ๋…ผ๋ฌธ v1์ด ์ฃผ์žฅํ•œ ๊ฐ’์ด๋‹ค. FastWAM ๋Œ€๋น„ 73.0%์˜ ๊ตฌ์„ฑ์„ ๋‚˜๋ˆ  ๋ณด๋ฉด, ์‹œ๊ฐ expert์˜ ํ† ํฐ ๊ฐ์†Œ(8.93 โ†’ 3.17)์™€ VAE ์ธ์ฝ”๋”ฉ ๊ฐ์†Œ(6.54 โ†’ 0.93)๊ฐ€ ๋‘˜ ๋‹ค ํฌ๋‹ค. ๋’ค์ชฝ์€ JAM๋งŒ target์„ ์บ์‹œํ•ด์„œ ์ƒ๊ธฐ๋Š” ์ ˆ๊ฐ์ด๋‹ค. ๋น„๊ต ์กฐ๊ฑด ๋ฌธ์ œ๋Š” ์•„๋ž˜ ๋น„ํŒ ์ ˆ์—์„œ ๋‹ค๋ฃฌ๋‹ค.

์‹คํ—˜ 5: ๋ถ€๋ก์˜ ์ ˆ์ œ์™€ ํ†ต๊ณ„ (๋ถ€๋ก CยทD.7ยทD.8)

๋ถ€๋ก C์˜ Wan ์ ˆ์ œ๋Š” ๋ชจ๋‘ ์‚ฌ์ „ํ•™์Šต Wan anchor, ๋ธ”๋ก 18, [1,2], \lambda_F=0.6, ํญ 1,536 transformer head, 20k update๋ฅผ ๊ธฐ๋ณธ์œผ๋กœ ํ•œ๋‹ค. ๋ณธ๋ฌธ์—์„œ ์ตœ์ข… ์„ ํƒ๋œ VAE target์ด ์•„๋‹ˆ๋‹ค.

์ ˆ์ œ ๊ฒฐ๊ณผ (LIBERO ํ‰๊ท  %, C.6ยทC.7์€ LIBERO-Plus ํฌํ•จ)
EMA anchor (C.1) ๊ณ ์ • ์‚ฌ์ „ํ•™์Šต 97.45 vs online EMA(decay 0.995) 96.40
๊ฐ๋… ๊นŠ์ด (C.2) ๋ธ”๋ก 14: 97.10 ยท 16: 97.30 ยท 18: 97.45 ยท 20: 95.45
\lambda_F (C.3) 0.4: 95.70 ยท 0.5: 96.90 ยท 0.6: 97.45 ยท 0.7: 96.75. FastWAM(97.00)์„ ๋„˜๋Š” ๊ฒƒ์€ 0.6๋ฟ, Long suite๊ฐ€ 90.0~97.0์œผ๋กœ ๊ฐ€์žฅ ๋ฏผ๊ฐ
head ์šฉ๋Ÿ‰ (C.4) Transformer ํญ 3072(145M) 96.30 ยท 1536(71M) 97.45 ยท 768(31M) 95.75 / MLP 3072(38M) 95.35 ยท 1536(17M) 95.05
๋ณ„๋„ expert vs ๊ณต์œ  ๊ฐ€์ค‘์น˜ (C.6) Qwen3.5 expert 3.60B: LIBERO-Plus 81.14 ยท ๊ณต์œ  2B(2.22B) 79.61 ยท ๊ณต์œ  4B(4.55B) 80.68
ํ•™์Šต ์˜ˆ์‚ฐ (C.7) FastWAM 10k 95.90/55.25, 20k 97.00/55.31 ยท JAM(Wan) 10k 97.45/48.47, 20k 97.45/55.58 (LIBERO/LIBERO-Plus)

C.7์€ ํฅ๋ฏธ๋กญ๋‹ค. JAM(Wan)์€ 10k update์—์„œ ์ด๋ฏธ LIBERO 97.45%๋กœ 20k FastWAM์„ ๋„˜์ง€๋งŒ LIBERO-Plus๋Š” 48.47%๋กœ 10k FastWAM(55.25%)๋ณด๋‹ค ๋‚ฎ๋‹ค. FastWAM์˜ LIBERO-Plus๋Š” 10k์—์„œ ๊ฑฐ์˜ ํฌํ™”ํ•˜๊ณ , JAM์€ ๋” ์˜ค๋ž˜ ํ•™์Šตํ•ด์•ผ robustness๊ฐ€ ๋”ฐ๋ผ์˜จ๋‹ค.

ํ†ต๊ณ„ ๊ฒ€์ •(Table 20).

๊ณ ์ • ์ฒดํฌํฌ์ธํŠธ ๋น„๊ต (Table 20). 100,000ํšŒ task-bootstrap.
๊ณ„์—ด ๋ฒค์น˜๋งˆํฌ \Delta (%p) 95% task-bootstrap CI ๋ฒ ์ด์Šค๋ผ์ธ๋งŒ / JAM๋งŒ McNemar p
FastWAM LIBERO +1.15 [+0.20, +2.10] 30 / 53 0.015
ImageWAM LIBERO +3.85 [+1.00, +7.40] 42 / 119 1.01\times10^{-9}
FastWAM LIBERO-Plus +9.23 [+6.84, +11.69] 111 / 250 1.91\times10^{-13}
ImageWAM LIBERO-Plus +14.67 [+12.15, +17.13] 89 / 310 9.69\times10^{-30}

\Delta๋Š” discordant ์ˆ˜์—์„œ ๊ทธ๋Œ€๋กœ ๋‚˜์˜ค๊ณ (์˜ˆ: (53โˆ’30)/2000 = 1.15%p), ๋„ค McNemar p๋„ exact binomial ๊ฒ€์ •์œผ๋กœ ๊ฐ™์€ ๊ฐ’์ด ์žฌํ˜„๋œ๋‹ค(ํ•„์ž ๊ฒ€์‚ฐ). ์„ ํƒ๋œ ๋‘ ์„ค์ •์€ 3ํšŒ ๋…๋ฆฝ ํ•™์Šต์—์„œ LIBERO 98.13ยฑ0.13%(Wan), 96.75ยฑ0.15%(Ovis-U1), LIBERO-Plus 64.61ยฑ0.37%, 79.92ยฑ0.43%๋กœ ๋ณ€๋™์ด ์ž‘๋‹ค(Table 21). Table 1๊ณผ ๋ณธ๋ฌธ ํ‘œ์— ์‹ค๋ฆฐ ๊ฐ’์€ ์ด ์„ธ ๋ฒˆ ์ค‘ 1๋ฒˆ run์ด๋‹ค.

๋น„ํŒ์ ์œผ๋กœ ๋ณด๋ฉด

๊ฐ•์ 

  • ์งˆ๋ฌธ์„ ๋ฐ”๊พผ ๊ฒƒ์ด ๊ฐ€์žฅ ํฐ ๊ธฐ์—ฌ๋‹ค. โ€œVLA๋ƒ WAM์ด๋ƒโ€๋ฅผ ์„ธ ์ถ•์œผ๋กœ ๋ถ„ํ•ดํ•œ Fig. 1์€ ์ดํ›„ ์ด ๋ถ„์•ผ์˜ ๋น„๊ต ์‹คํ—˜์„ ์„ค๊ณ„ํ•  ๋•Œ ๊ทธ๋Œ€๋กœ ์ฒดํฌ๋ฆฌ์ŠคํŠธ๋กœ ์“ธ ์ˆ˜ ์žˆ๋‹ค. ํŠนํžˆ โ€œ์ถ”๋ก  ๋ชจ๋“ˆโ€์„ โ€œ๋ฏธ๋ž˜ ๋ชฉ์ ํ•จ์ˆ˜โ€์™€ ๋–ผ์–ด ๋†“์€ ๋•๋ถ„์—, FastWAMยทImageWAM๊ณผ ์ถ”๋ก  ๊ฒฝ๋กœ๊ฐ€ ์™„์ „ํžˆ ๊ฐ™์€ ์ƒํƒœ์—์„œ ๋ชฉ์ ํ•จ์ˆ˜๋งŒ ๋น„๊ตํ•  ์ˆ˜ ์žˆ์—ˆ๋‹ค.
  • matched ๋น„๊ต์˜ ํ†ต์ œ๊ฐ€ ์—„๊ฒฉํ•˜๋‹ค. ๋ฐ์ดํ„ฐ, ์•„ํ‚คํ…์ฒ˜, optimizer, ์˜ˆ์‚ฐ, ํ‰๊ฐ€๊ธฐ๋ฅผ ๊ณต์œ ํ•˜๊ณ , action-only ํ–‰๊นŒ์ง€ ๊ฐ™์€ ํ‘œ์— ๋„ฃ์–ด ์ƒ์„ฑ ์†์‹ค๊ณผ foresight ์†์‹ค์ด ๊ฐ๊ฐ action-only ๋Œ€๋น„ ์–ผ๋งˆ๋ฅผ ์ฃผ๋Š”์ง€ ์ฝ์„ ์ˆ˜ ์žˆ๊ฒŒ ํ–ˆ๋‹ค. Ovis-U1์—์„œ ์ƒ์„ฑ +3.92%p ๋Œ€ foresight +18.60%p๋ผ๋Š” ๋Œ€๋น„๋Š” ์ด ์„ค๊ณ„ ๋•๋ถ„์— ๋ณด์ธ๋‹ค.
  • ํ†ต๊ณ„ ๋ณด๊ณ ๊ฐ€ ์ด ๋ถ„์•ผ ํ‰๊ท ๋ณด๋‹ค ๋‚ซ๋‹ค. task ๋‹จ์œ„ bootstrap CI, McNemar ๊ฒ€์ •, ์„ ํƒ ์„ค์ •์˜ 3ํšŒ ๋…๋ฆฝ ํ•™์Šต, ์ง์ง€์€ discordant ์ˆ˜๊นŒ์ง€ ๋‚ธ๋‹ค. Fig. 6์˜ ๋ฒ”์ฃผ๋ณ„ ์ˆ˜๊ฐ€ Table 20๊ณผ ์ •ํ™•ํžˆ ๋งž๊ณ , ๋„ค p๊ฐ’์ด ์žฌํ˜„๋œ๋‹ค.
  • ์ž๊ธฐ ๊ฒฐ๊ณผ์— ๋‹จ์„œ๋ฅผ ์ •ํ™•ํžˆ ๋ถ™์ธ๋‹ค. Table 3 ์บก์…˜์€ ์‚ฌ์ „ํ•™์Šต ํšจ๊ณผ๋ฅผ ๋ถ„๋ฆฌํ•  ์ˆ˜ ์—†๋‹ค๊ณ  ์“ฐ๊ณ , attention ์ง€๋„๋Š” โ€œ์–ด๋””๋กœ ๊ฐ€๋Š”์ง€โ€๋งŒ ๊ธฐ์ˆ ํ•œ๋‹ค๊ณ  ํ•˜๋ฉฐ, probe ๊ฒฐ๊ณผ๋Š” โ€œsuggestโ€๋กœ ์“ด๋‹ค. ์ž๊ธฐ์—๊ฒŒ ๋ถˆ๋ฆฌํ•œ ๊ฒฐ๊ณผ(offset [0]์ด action-only๋ณด๋‹ค ๋‚ฎ์Œ, LingBot-VAยท\pi_{0.5}๊ฐ€ ์•ž์„ฌ, Qwen์˜ RoboTwin ์—ด์„ธ)๋„ ๊ทธ๋Œ€๋กœ ์‹ฃ๋Š”๋‹ค.
  • ๋น„์šฉ์„ ์ถ”์ •์น˜์™€ ์‹ค์ธก ๋‘˜ ๋‹ค ๋ƒˆ๋‹ค. ์ถ”์ • FLOPs์™€ GPU ์‹œ๊ฐ„ ์ธก์ •์„ ๋‚˜๋ž€ํžˆ ๋‘๊ณ , ๋‘˜์ด ์–ด๊ธ‹๋‚˜๋Š” ์ด์œ ๊นŒ์ง€ ์„ค๋ช…ํ•œ๋‹ค.
  • ์žฌํ˜„ ์ž์‚ฐ์ด ๊ณต๊ฐœ๋๋‹ค. ์ฝ”๋“œ(MIT), ์„ ํƒ๋œ JAM ์„ค์ •์˜ ์—ด ๊ฐœ recipe, LIBERO ์ฒดํฌํฌ์ธํŠธ ์„ธ ๊ฐœ๊ฐ€ ๊ณต๊ฐœ๋ผ ์žˆ๋‹ค. ๋‹ค๋งŒ ๋ฒ ์ด์Šค๋ผ์ธ๊ณผ target ๋ณ€ํ˜• recipe๋Š” ์ตœ์‹  ์ฝ”๋“œ์— ์—†๋‹ค(์•„๋ž˜ ์ฝ”๋“œ ๊ณต๊ฐœ ์ƒํƒœ).

์•ฝ์ ยทํ•œ๊ณ„

  • FastWAM ๊ธฐ์ค€๊ฐ’์ด ์› ๋…ผ๋ฌธ ๋ณด๊ณ ๊ฐ’๋ณด๋‹ค ๋‚ฎ๋‹ค. ๋ธ”๋กœ๊ทธ์˜ Fast-WAM ๋ฆฌ๋ทฐ์— ๊ธฐ๋กํ•œ Fast-WAM ๋…ผ๋ฌธ ๊ฐ’์€ LIBERO 97.6%(98.2/100.0/97.0/95.2), RoboTwin 91.8%๋‹ค. JAM์ด ์“ด FastWAM์€ LIBERO 97.00%(๋กœ์ปฌ ์žฌํ•™์Šต, 97.4/99.6/96.6/94.4), RoboTwin 91.21%(๊ณต๊ฐœ ๊ฐ€์ค‘์น˜๋ฅผ ์ €์ž๋“ค์ด ํ‰๊ฐ€)๋‹ค. ์› ๋ณด๊ณ ๊ฐ’๊ณผ ๋Œ€์กฐํ•˜๋ฉด JAM(Wan)์˜ LIBERO ์šฐ์œ„๋Š” +1.15%p๊ฐ€ ์•„๋‹ˆ๋ผ +0.55%p์ด๊ณ , RoboTwin์—์„œ๋Š” 91.44% < 91.8%๋กœ ์˜คํžˆ๋ ค ๋‚ฎ๋‹ค(ํ•„์ž ๊ณ„์‚ฐ). ์ดˆ๋ก์˜ โ€œ97.0%/91.2% โ†’ 98.2%/91.4%โ€๋Š” ์ €์ž ํ‰๊ฐ€๊ฐ’๋ผ๋ฆฌ์˜ ๋น„๊ต๋‹ค. ๊ฐ™์€ ํ‰๊ฐ€๊ธฐ๋กœ ๋‹ค์‹œ ์žฐ ๊ฒƒ ์ž์ฒด๋Š” ๊ณต์ •ํ•œ ์„ ํƒ์ด์ง€๋งŒ, ์› ๋ณด๊ณ ๊ฐ’๊ณผ ์ฐจ์ด๊ฐ€ ๋‚˜๋Š” ์ด์œ ๋Š” ๋…ผ๋ฌธ์— ์—†๋‹ค. ๊ฐ™์€ ๋งฅ๋ฝ์—์„œ โ€œ๋น„๋””์˜ค ์†์‹ค์„ ๋บ€ FastWAMโ€๋„ Fast-WAM ๋…ผ๋ฌธ์˜ w.o. video co-train์€ LIBERO 93.5%, JAM์˜ FastWAM action-only๋Š” 95.25%๋กœ ๋‹ค๋ฅด๋‹ค. ๋˜ RoboTwin์˜ FastWAM๊ณผ ImageWAM(FLUX.2)์€ ๊ณต๊ฐœ ๊ฐ€์ค‘์น˜์ด๊ณ  JAM์€ ๋กœ์ปฌ ํ•™์Šต์ด๋ผ, Table 4์˜ RoboTwin ํ–‰์€ Table 1๊ณผ ๊ฐ™์€ ์˜๋ฏธ์˜ matched ๋น„๊ต๊ฐ€ ์•„๋‹ˆ๋‹ค.
  • target ์„ ํƒ์ด ํ‰๊ฐ€ ๋ฒค์น˜๋งˆํฌ์—์„œ ์ด๋ค„์กŒ๋‹ค. Wan์€ ๋‹ค์„ฏ target/offset ์กฐํ•ฉ, Ovis-U1๋„ ๋‹ค์„ฏ ์กฐํ•ฉ ๊ฐ€์šด๋ฐ LIBERO-Plus ์ตœ๊ณ ๊ฐ’์„ โ€œselected configurationโ€์œผ๋กœ ๊ณจ๋ผ ๋ณธ๋ฌธ ํ‘œ(Table 4~6)์™€ ํ†ต๊ณ„ ๊ฒ€์ •(Table 20)์— ์ผ๋‹ค. ๋ณ„๋„ validation ๋ถ„ํ• ์€ ์—†๋‹ค. 3ํšŒ ๋ฐ˜๋ณต์€ ์„ ํƒ๋œ ์„ค์ •์ด ์•ˆ์ •์ ์ด๋ผ๋Š” ๊ฒƒ์„ ๋ณด์ด์ง€๋งŒ ์„ ํƒ ํŽธํ–ฅ์„ ์—†์• ์ง€๋Š” ์•Š๋Š”๋‹ค. ๋ฒ ์ด์Šค๋ผ์ธ(FastWAM, ImageWAM)์˜ ํ•™์Šต ๋ถ„์‚ฐ์€ ๋ณด๊ณ ๋˜์ง€ ์•Š์•˜๋‹ค. ๋ถ€๋ก C์˜ ์ ˆ์ œ(๊นŠ์ด, \lambda_F, head)๋„ LIBERO๋งŒ ๋ณด๊ณ  LIBERO-Plus๋Š” ์—†๋‹ค.
  • โ€œ์ƒ์„ฑ ์—†์ดโ€๋ผ๋Š” ๋ฌธ์žฅ์˜ ๋ฒ”์œ„๋ฅผ ์ขํ˜€ ์ฝ์–ด์•ผ ํ•œ๋‹ค. Wan์—์„œ ์ตœ๊ณ  target์€ VAE latent๋ฅผ ๊ณ ์ • ๋ฐ˜๋ณต์œผ๋กœ liftํ•œ ๊ฒƒ์ด๊ณ , ์ด๋Š” FastWAM์ด denoiseํ•˜๋˜ ๊ณต๊ฐ„ ๊ทธ๋Œ€๋กœ๋‹ค. ์‚ฌ์ „ํ•™์Šต Wan feature๋ฅผ target์œผ๋กœ ์“ฐ๋ฉด FastWAM๊ณผ ๋™๋ฅ (55.58 vs 55.31)์ด๋‹ค. ๋”ฐ๋ผ์„œ Wan์—์„œ ํ™•์ธ๋œ ๊ฒƒ์€ โ€œ์˜๋ฏธ ํ‘œํ˜„์„ ์˜ˆ์ธกํ•˜๋ฉด ๋œ๋‹คโ€๋ณด๋‹ค โ€œ๊ฐ™์€ latent๋ฅผ ๋ฐฑ๋ณธ ์•ˆ์—์„œ denoiseํ•˜์ง€ ๋ง๊ณ  ํ˜„์žฌ ํ† ํฐ์—์„œ regressionํ•˜๋ผโ€์— ๊ฐ€๊น๋‹ค. ๋ฐ˜๋Œ€๋กœ Ovis-U1์—์„œ๋Š” ์‚ฌ์ „ํ•™์Šต Yak feature๊ฐ€ VAE๋ณด๋‹ค ๋‚ซ๋‹ค. ์ œ๋ชฉ๊ณผ ์ดˆ๋ก์˜ โ€œvisual generation์ด ํ•„์š” ์—†๋‹คโ€๋Š” ๊ฒฐ๋ก ์€ ๋งž์ง€๋งŒ, ๊ทธ ์ด์œ ๊ฐ€ โ€œํ‘œํ˜„ ๊ณต๊ฐ„์ด๋ผ์„œโ€์ธ์ง€ โ€œ๋ฐฑ๋ณธ ์ž…๋ ฅ์—์„œ ๋ฏธ๋ž˜ ํ† ํฐ์„ ๋บ๊ธฐ ๋•Œ๋ฌธโ€์ธ์ง€๋Š” ์ด ์‹คํ—˜๋“ค๋กœ ๊ฐˆ๋ผ์ง€์ง€ ์•Š๋Š”๋‹ค. ๋ฏธ๋ž˜ ํ† ํฐ์€ ๋ฐฑ๋ณธ ์ž…๋ ฅ์— ๋„ฃ๋˜ \mathcal L_G ๋Œ€์‹  \mathcal L_F๋กœ ๊ฐ๋…ํ•˜๋Š” ์กฐ๊ฑด์ด ์žˆ์œผ๋ฉด ๋‘˜์„ ๊ฐ€๋ฅผ ์ˆ˜ ์žˆ๋‹ค.
  • ๋ฐฑ๋ณธ ๋น„๊ต์˜ ๊ฒฐ๋ก ์ด ๊ฐ•ํ•˜๋‹ค. Qwen3.5-2B action-only(77.03%)๊ฐ€ ์ด๋ฏธ ์ƒ์„ฑ ๊ณ„์—ด matched ์ •์ฑ… ๋‘˜๋ณด๋‹ค ๋†’๋‹ค. foresight์˜ ์ฆ๋ถ„์€ ๋น„์ƒ์„ฑ ๋ฐฑ๋ณธ์—์„œ +2~4%p์ด๊ณ , ๋ฐฑ๋ณธ ๊ฐ„ ์ฐจ์ด๋Š” ๊ทธ๋ณด๋‹ค ํฌ๋‹ค. โ€œthe best generator-initialized and vision-language policies achieve comparable performanceโ€๋Š” ๋‹จ์ผ run์˜ 0.20/1.19%p ์ฐจ์ด์— ๊ธฐ๋Œ€๊ณ , DINOv2ยทOvis-U1ยทFLUX.2์—๋Š” action-only ๋Œ€์กฐ๊ตฐ๋„ ์ผ๋ถ€๋งŒ ์žˆ๋‹ค(Table 14๋Š” Qwen๊ณผ DINOv2-L๋งŒ).
  • ๋น„์šฉ 73%์—๋Š” ์บ์‹œ ์ด๋“์ด ์„ž์—ฌ ์žˆ๋‹ค. FastWAM์˜ 15.52 TFLOPs ๊ฐ€์šด๋ฐ 6.54๊ฐ€ ๋ฏธ๋ž˜ ์ฐฝ์˜ VAE ์ธ์ฝ”๋”ฉ์ด๋‹ค. ๊ณ ์ • VAE์ด๋ฏ€๋กœ FastWAM๋„ latent๋ฅผ ์บ์‹œํ•  ์ˆ˜ ์žˆ๋‹ค. Table 9์˜ ๊ฐ’์œผ๋กœ FastWAM์˜ VAE ํ•ญ์„ JAM ์บ์‹œ ์ˆ˜์ค€(0.93)์œผ๋กœ ๋ฐ”๊พธ๋ฉด 9.91 TFLOPs์ด๊ณ , JAM(Wan) ์บ์‹œ ๋Œ€๋น„ ๊ฐ์†Œ๋Š” ์•ฝ 57.7%๋‹ค(ํ•„์ž ๊ณ„์‚ฐ). ๊ณต๊ฐœ ๋ ˆํฌ๋Š” 2026-11-13 ์ปค๋ฐ‹์—์„œ README์™€ ํ”„๋กœ์ ํŠธ ํŽ˜์ด์ง€๋ฅผ ์ด ๊ฐ™์€ ์บ์‹œ ์กฐ๊ฑด ๋น„๊ต๋กœ ๋ฐ”๊ฟจ๋‹ค(โ€œwhen both cache their future encodings, uses 57.7% fewer estimated training FLOPs than FastWAMโ€, ImageWAM ๋Œ€๋น„ 31.9%). arXiv์—๋Š” ์•„์ง v1๋ฟ์ด๋ผ ๋…ผ๋ฌธ ๋ณธ๋ฌธ(73.0%, JAM๋งŒ ์บ์‹œํ•œ ๋น„๊ต)๊ณผ ๋ ˆํฌ๊ฐ€ ๋‹ค๋ฅธ ์ˆซ์ž๋ฅผ ๋งํ•˜๊ณ  ์žˆ๋‹ค. ํ”„๋กœ์ ํŠธ ํŽ˜์ด์ง€์˜ ํŠธ๋ ˆ์ผ๋Ÿฌ ์˜์ƒ ๋์—๋Š” โˆ’73%๊ฐ€ ๊ทธ๋Œ€๋กœ ๋‚จ์•„ ์žˆ๋‹ค. ๋˜ ์บ์‹œ ๋น„์šฉ(1ํšŒ target ์ถ”์ถœ)์€ ์ œ์™ธ๋๊ณ , JAM(Ovis-U1)์„ online์œผ๋กœ ๋Œ๋ฆฌ๋ฉด 1.13B ๊ณ ์ • Yak์ด ํ•™์Šต ์ค‘ ์ถ”๊ฐ€๋กœ ๋ฉ”๋ชจ๋ฆฌ์— ์˜ฌ๋ผ๊ฐ„๋‹ค.
  • probe์™€ attention์€ ํ•œ ์žฅ๋ฉด์˜ ์ •์„ฑ ๊ทผ๊ฑฐ๋‹ค. Fig. 4๋Š” ํ•œ ์˜ˆ์‹œ์ด๊ณ  ์ •๋Ÿ‰ ์š”์•ฝ(Fig. 9)์˜ ์ฐจ์ด๋Š” 0.14 dB, ์„ธ๋กœ์ถ•์€ 15 dB์—์„œ ์‹œ์ž‘ํ•œ๋‹ค. attention ์ง€๋„๋Š” 100 ํ”„๋ ˆ์ž„ ํ‰๊ท ์„ ๋บ€ ๋’ค 99 percentile๋กœ ๋‚˜๋ˆ„๊ณ  0.6 ๊ฑฐ๋“ญ์ œ๊ณฑ ์••์ถ•์„ ํ•ด์„œ ๋Œ€๋น„๋ฅผ ํ‚ค์šด ํ‘œ์‹œ(๋ถ€๋ก D.5)๋‹ค. ๋‘ ๋ถ„์„ ๋ชจ๋‘ ์ด๋“์˜ ๋ฉ”์ปค๋‹ˆ์ฆ˜์„ ๋ณด์—ฌ ์ค€๋‹ค๊ธฐ๋ณด๋‹ค โ€œ๊ทธ๋ ‡๊ฒŒ ํ•ด์„ํ•  ์—ฌ์ง€๊ฐ€ ์žˆ๋‹คโ€๋Š” ์ˆ˜์ค€์ด๊ณ , ์ €์ž๋“ค๋„ ๊ทธ๋ ‡๊ฒŒ ์“ด๋‹ค.
  • ์„ค๋ช… ์—†๋Š” ๊ฒฐ๊ณผ๊ฐ€ ๋‚จ์•„ ์žˆ๋‹ค. ๋ฌด์ž‘์œ„ ๋ฐฑ๋ณธ์—์„œ ์‚ฌ์ „ํ•™์Šต anchor๊ฐ€ ๋ฌด์ž‘์œ„ anchor๋ณด๋‹ค 13.25%p ๋‚˜์˜๋‹ค(Table 2). \lambda_F ์Šค์œ•์€ ๋‹จ์กฐ๋กญ์ง€ ์•Š์•„ Long suite๊ฐ€ 0.4์—์„œ 90.0, 0.5์—์„œ 97.0, 0.6์—์„œ 94.0์ด๋‹ค. ๋ธ”๋ก 20์œผ๋กœ ์˜ฎ๊ธฐ๋ฉด 2%p ๋–จ์–ด์ง„๋‹ค. ํ•˜์ดํผํŒŒ๋ผ๋ฏธํ„ฐ ๋ฏผ๊ฐ๋„๊ฐ€ ํฐ ํŽธ์ธ๋ฐ, ๊ฐ ์ ์ด ๋‹จ์ผ run์ธ์ง€(๋ถ€๋ก D.8์˜ ๋ฐ˜๋ณต์€ ์„ ํƒ๋œ ๋‘ ์„ค์ •๋งŒ) ํ‘œ์— ๋ช…์‹œ๋ผ ์žˆ์ง€ ์•Š๋‹ค.
  • ์‹œ๋ฎฌ๋ ˆ์ด์…˜๋ฟ์ด๋‹ค. LIBERO, LIBERO-Plus, RoboTwin 2.0๋งŒ ํ‰๊ฐ€ํ–ˆ๊ณ  ์‹ค๋ฌผ ๋กœ๋ด‡ ์‹คํ—˜์ด ์—†๋‹ค. ์œค๋ฆฌ ์ ˆ์ด โ€œphysical validation ์—†๋Š” ๋ฐฐํฌโ€๋ฅผ ์œ„ํ—˜์œผ๋กœ ์ ์„ ๋ฟ ๋ณ„๋„ limitations ์ ˆ๋„ ์—†๋‹ค. LIBERO-Plus๋Š” 15% subset์— ํƒœ์Šคํฌ๋‹น 1ํšŒ rollout์ด๊ณ , ๊ณต๊ฐœ ํ–‰(์ „์ฒด ์„ธํŠธ)๊ณผ ํ”„๋กœํ† ์ฝœ์ด ๋‹ฌ๋ผ Table 6์˜ ๊ทธ๋ฃน ๊ฐ„ ๋น„๊ต๋Š” ์ฐธ๊ณ ์šฉ์ด๋‹ค. ์ƒ์„ฑ ์†์‹ค์˜ ๊ฐ€์ค‘์น˜ \lambda_G ๊ฐ’๋„ ๋ณธ๋ฌธ์— ์ ํ˜€ ์žˆ์ง€ ์•Š๋‹ค(๋ฒ ์ด์Šค๋ผ์ธ recipe๋ฅผ ๋”ฐ๋ฅธ๋‹ค๋Š” ์„œ์ˆ ๋ฟ์ด๋‹ค).

๊ด€๋ จ ์—ฐ๊ตฌ์™€์˜ ์ž๋ฆฌ ๋งค๊น€

WAM์˜ โ€œ๋ฌด์—‡์ด ํ•„์š”ํ•œ๊ฐ€โ€ ๊ณ„๋ณด. Fast-WAM์€ ์ถ”๋ก  ๋•Œ์˜ ๋ฏธ๋ž˜ ์ƒ์ƒ์„ ๋นผ๊ณ  ํ•™์Šต ๋•Œ์˜ ๋น„๋””์˜ค co-training์ด ํ•ต์‹ฌ์ด๋ผ๊ณ  ๋ณด์˜€๊ณ , ImageWAM์€ ๋น„๋””์˜ค ๋Œ€์‹  ์ด๋ฏธ์ง€ ํŽธ์ง‘ ๋ฐฑ๋ณธ์œผ๋กœ ๊ฐ™์€ ์งˆ๋ฌธ์„ ๋˜์กŒ๋‹ค. JAM์€ ๊ทธ ๋‹ค์Œ ๋‹จ๊ณ„๋กœ ํ•™์Šต ๋•Œ์˜ ์ƒ์„ฑ๊นŒ์ง€ ๋บ€๋‹ค. ์„ธ ๋…ผ๋ฌธ์„ ์ด์–ด ์ฝ์œผ๋ฉด โ€œ๋ฏธ๋ž˜๋Š” ์ถ”๋ก  ์ž…๋ ฅ์œผ๋กœ๋„, ์ƒ์„ฑ ๋ชฉํ‘œ๋กœ๋„ ํ•„์š” ์—†๊ณ , ๊ณ ์ • target์ด๋ฉด ์ถฉ๋ถ„ํ•˜๋‹คโ€๋กœ ์งˆ๋ฌธ์ด ์ขํ˜€์ง„๋‹ค. ๋ฏธ๋ž˜ latent๋ฅผ ์ผ๋ถ€ denoiseํ•œ KV๋กœ ํ–‰๋™์„ ๋‚ด๋Š” Long-WAM์ด๋‚˜, ํ–‰๋™ยท๋ฏธ๋ž˜ ์ƒํƒœยท๊ฐ€์น˜๋ฅผ ๋น„๋””์˜ค ๋ชจ๋ธ์˜ latent ์‹œํ€€์Šค ์•ˆ์—์„œ ํ•จ๊ป˜ ์ƒ์„ฑํ•˜๋Š” Cosmos Policy์™€ ๋น„๊ตํ•˜๋ฉด, JAM์€ โ€œ๋ฏธ๋ž˜ ์˜ˆ์ธก ์••๋ ฅ์€ ์œ ์ง€ํ•˜๋˜ ์ƒ์„ฑ ๊ฒฝ๋กœ๋Š” ํ•™์Šต์—์„œ๋„ ์ถ”๋ก ์—์„œ๋„ ์—†์•ค๋‹คโ€๋Š” ๋ฐ˜๋Œ€์ชฝ ๋์— ์žˆ๋‹ค.

๋ฏธ๋ž˜ ํ‘œํ˜„ ๊ฐ๋…. VLA-JEPA๋Š” ์‚ฌ๋žŒ ์˜์ƒ ์‚ฌ์ „ํ•™์Šต์— ๊ณ ์ • ๋น„๋””์˜ค target์„ ์“ฐ๊ณ , JEPA-WAM์€ ๊ณ ์ • V-JEPA ๊ณต๊ฐ„์˜ ํ˜„์žฌยท๋ฏธ๋ž˜ ๊ฒฐํ•ฉ target์œผ๋กœ action ์ƒ์„ฑ predictor๋ฅผ ๊ฐ๋…ํ•œ๋‹ค. FLARE๋Š” ๋ฏธ๋ž˜ยท์ƒํƒœยทํ–‰๋™ ํ† ํฐ์„ action DiT ์•ˆ์—์„œ self-attention์œผ๋กœ ์„ž๋Š”๋‹ค. JAM์€ ์ด๋“ค๊ณผ ๊ฐ๋… ์œ„์น˜๊ฐ€ ๋‹ค๋ฅด๋‹ค. ๋ณ„๋„์˜ ํ•™์Šต ์ „์šฉ head๊ฐ€ ๊ธฐ์กด ๋ฐฑ๋ณธ ํ† ํฐ์„ ๊ฐ๋…ํ•˜๋ฏ€๋กœ ๋งค์นญ ์ •์ฑ…์˜ ํ–‰๋™ ๊ณ„์‚ฐ์ด ๊ทธ๋Œ€๋กœ ๋ณด์กด๋˜๊ณ , target ๊ณต๊ฐ„์€ ๋ฐฑ๋ณธ๋งˆ๋‹ค ๊ณ ๋ฅธ๋‹ค. ๊ฐ™์€ ๋‚  ๊ณต๊ฐœ๋œ ThinkWorld๋„ DINOv3 feature ๊ณต๊ฐ„์—์„œ ๋ฏธ๋ž˜๋ฅผ ์˜ˆ์ธกํ•˜๊ณ  ๋ฐฐํฌ ๋•Œ๋Š” ์ƒ์„ฑํ•˜์ง€ ์•Š๋Š”๋‹ค๋Š” ์ ์—์„œ ๊ฐ™์€ ํ๋ฆ„์ด์ง€๋งŒ, ๊ทธ์ชฝ์€ ์†๋ชฉ motion ํ† ํฐ๊ณผ RL์„ ์–น๋Š”๋‹ค. ์ €์ž๋“ค์˜ ์ „์ž‘ Mirai/Video-Mirai๋Š” ๊ฐ™์€ โ€œforesightโ€ ์•„์ด๋””์–ด๋ฅผ ์ด๋ฏธ์ง€ยท๋น„๋””์˜ค ์ƒ์„ฑ์— ์ ์šฉํ–ˆ๊ณ , JAM์€ ์ด๋ฅผ ๋กœ๋ด‡ ์ •์ฑ…์œผ๋กœ ์˜ฎ๊ธด ์…ˆ์ด๋‹ค.

๋ฌด์ž‘์œ„ target. ๋ฌด์ž‘์œ„ anchor๊ฐ€ ์ž˜ ์ž‘๋™ํ•œ๋‹ค๋Š” ๊ฒฐ๊ณผ๋Š” Random Network Distillation, โ€œRandom teachers are good teachersโ€, Noise As Targets๋กœ ์ด์–ด์ง€๋Š” ๊ณ„๋ณด์™€ ๋งž๋‹ฟ๊ณ , EMA target(BYOL ๊ณ„์—ด)์€ ์ด ์„ค์ •์—์„œ ๊ณ ์ • target๋ณด๋‹ค ๋‚˜๋นด๋‹ค(C.1).

VLA ์ชฝ. JAM(Qwen3.5-2B)์€ ์ €์ž ์ •์˜๋กœ โ€œforesight loss๋ฅผ ๋”ํ•œ VLAโ€๋‹ค. OpenVLA-OFT์ฒ˜๋Ÿผ VLM ๋ฐฑ๋ณธ์— continuous action head๋ฅผ ๋ถ™์ด๋Š” VLA์— ๋ณด์กฐ ์†์‹ค ํ•˜๋‚˜๋ฅผ ๋”ํ•˜๋Š” ํ˜•ํƒœ๋ผ, ๊ธฐ์กด VLA ํ•™์Šต ํŒŒ์ดํ”„๋ผ์ธ์— ๊ฐ€์žฅ ์‰ฝ๊ฒŒ ์ด์‹๋  ๋ถ€๋ถ„์ด๊ธฐ๋„ ํ•˜๋‹ค. ํ‰๊ฐ€ ๋ฒค์น˜๋งˆํฌ ์ค‘ RoboTwin 2.0์€ ๋ธ”๋กœ๊ทธ์— ๋”ฐ๋กœ ์ •๋ฆฌ๋ผ ์žˆ๋‹ค.

์ฝ”๋“œ ๊ณต๊ฐœ ์ƒํƒœ

GitHub README์™€ ๊ณต๊ฐœ ๋ ˆํฌ ๊ตฌ์„ฑ์„ ๊ธฐ์ค€์œผ๋กœ ์ •๋ฆฌํ•œ๋‹ค.

  • ๋ ˆํฌ: y0urOy/JAM, MIT. Wan ๋ฐฑ์—”๋“œ๋Š” FastWAM, ๋‚˜๋จธ์ง€(Ovis-U1, FLUX.2, Qwen3.5-2B, DINOv2-B/L) ๋ฐฑ์—”๋“œ๋Š” ImageWAM ์ฝ”๋“œ ์œ„์— ๊ตฌ์ถ•๋๋‹ค. LIBERO 6๊ฐœ, RoboTwin 2.0 4๊ฐœ, ๋ชจ๋‘ ์—ด ๊ฐœ JAM recipe๋ฅผ ํ•˜๋‚˜์˜ ๊ณต์œ  ํ™˜๊ฒฝ(Python 3.10, PyTorch 2.7.1, CUDA 12.8)์—์„œ ํ•™์Šตยทํ‰๊ฐ€ํ•˜๋„๋ก ๊ตฌ์„ฑ๋ผ ์žˆ๋‹ค.
  • ์ฒดํฌํฌ์ธํŠธ: Hugging Face y0urOy/JAM์— LIBERO ์ฒดํฌํฌ์ธํŠธ ์„ธ ๊ฐœ(JAM(Wan), JAM(Ovis-U1), JAM(Qwen3.5-2B))๊ฐ€ ์žˆ๊ณ  ๋ชจ๋ธ ์นด๋“œ์˜ ๋ผ์ด์„ ์Šค ํ‘œ๊ธฐ๋Š” Apache-2.0์ด๋‹ค. RoboTwin ์ฒดํฌํฌ์ธํŠธ๋Š” README ๋ชฉ๋ก์— ์—†๋‹ค.
  • ๊ณต๊ฐœ๋œ ๋ฒ”์œ„: ํ˜„์žฌ ์ฝ”๋“œ๊ฐ€ ๋ฐ”๋กœ ์žฌํ˜„ํ•˜๋Š” ๊ฒƒ์€ ๋ณธ๋ฌธ ํ‘œ์— ์“ฐ์ธ ์„ ํƒ๋œ JAM ์„ค์ •์ด๋‹ค. ๋งค์นญ ๋ฒ ์ด์Šค๋ผ์ธ(FastWAMยทImageWAM์˜ ์ƒ์„ฑ ์†์‹ค, action-only VLA)์€ ๊ฐ™์€ ์ •์ฑ… ํด๋ž˜์Šค ์•ˆ์˜ ์ฝ”๋“œ ๊ฒฝ๋กœ๋กœ๋Š” ๋‚จ์•„ ์žˆ์ง€๋งŒ ์‹คํ–‰์šฉ recipe๊ฐ€ ์—†๋‹ค. Table 1ยท2ยท10์˜ target ๋ณ€ํ˜•(์‚ฌ์ „ํ•™์Šตยท๋ฌด์ž‘์œ„ Wan anchor, EMA anchor, ๋ฌด์ž‘์œ„ Yak, Ovis-U1 VAE-direct, ๋ฌด์ž‘์œ„ ๋ฐฑ๋ณธ)์€ ์ตœ์‹  ์ฝ”๋“œ์—์„œ ์ง€์›ํ•˜์ง€ ์•Š๊ณ , ๊ณต๊ฐœ ์งํ›„์˜ ์ฒซ ์ปค๋ฐ‹์—๋งŒ ์žˆ๋‹ค. ์ฆ‰ ์ด ๋…ผ๋ฌธ์˜ ํ•ต์‹ฌ ์ฃผ์žฅ์ธ โ€œ์ƒ์„ฑ vs foresightโ€ ๋น„๊ต์™€ target ์„ ํƒ ๊ณผ์ •์€ ์ตœ์‹  ์ฝ”๋“œ๋งŒ์œผ๋กœ๋Š” ํ•œ ๋ฒˆ์— ๋‹ค์‹œ ๋Œ๋ฆด ์ˆ˜ ์—†๋‹ค.
  • head๋Š” โ€œ์‹คํ–‰๋˜์ง€ ์•Š์„ ๋ฟโ€์ด๋‹ค: ์ฝ”๋“œ๋ฅผ ์ฝ์–ด ๋ณด๋ฉด ๋ฐฐํฌ ๊ฒฝ๋กœ์—์„œ prediction head๋ฅผ ํ˜ธ์ถœํ•˜์ง€ ์•Š์ง€๋งŒ, ์ €์žฅ ์ฝ”๋“œ๋Š” head ๊ฐ€์ค‘์น˜๋ฅผ ์ฒดํฌํฌ์ธํŠธ์— ํ•จ๊ป˜ ๋‹ด๋Š” ๊ฒƒ์œผ๋กœ ์ฝํžŒ๋‹ค(Wan์€ ๋ช…์‹œ์ ์œผ๋กœ ์ €์žฅ, ์•ฝ 71M ํŒŒ๋ผ๋ฏธํ„ฐ). ๋ณธ๋ฌธ์˜ โ€œ๋ฐฐํฌ ๋•Œ ์‚ฌ๋ผ์ง„๋‹คโ€๋Š” ์ถ”๋ก  ๋น„์šฉ์—์„œ ๋น ์ง„๋‹ค๋Š” ๋œป์ด๋‹ค. ์ฒดํฌํฌ์ธํŠธ๋ฅผ ๋‚ด๋ ค๋ฐ›์•„ ํ™•์ธํ•œ ๊ฒƒ์€ ์•„๋‹ˆ๋‹ค.
  • ์™ธ๋ถ€ ์ž์‚ฐ: FLUX.2 recipe์˜ autoencoder๋Š” gated FLUX.2-dev ์ €์žฅ์†Œ(FLUX non-commercial license)์—์„œ ๋ฐ›์•„์•ผ ํ•œ๋‹ค. ๋ฐ์ดํ„ฐ๋Š” FastWAM์ด ์ „์ฒ˜๋ฆฌํ•œ LIBEROยทRoboTwin 2.0์„ ์“ด๋‹ค.
  • ํ•™์Šต ๋น„์šฉ ํ‘œ๊ธฐ ์ •์ •: ๋ ˆํฌ๋Š” 2026-11-13 ์ปค๋ฐ‹์—์„œ README์™€ ํ”„๋กœ์ ํŠธ ํŽ˜์ด์ง€์˜ ๋น„์šฉ ๋น„๊ต๋ฅผ ์–‘์ชฝ ๋ชจ๋‘ ์บ์‹œํ•œ ์กฐ๊ฑด(FastWAM ๋Œ€๋น„ โˆ’57.7%)์œผ๋กœ ๋ฐ”๊ฟจ๋‹ค. ๋…ผ๋ฌธ v1์˜ โˆ’73.0%๋Š” JAM๋งŒ ์บ์‹œํ•œ ์กฐ๊ฑด์ด๋‹ค.

์š”์•ฝ

JAM์€ VLA์™€ WAM์˜ ๋…ผ์Ÿ์„ ๋ฐฑ๋ณธ ์ดˆ๊ธฐํ™” ยท ๋ฏธ๋ž˜ ๋ชฉ์ ํ•จ์ˆ˜ ยท ์ถ”๋ก  ๋ชจ๋“ˆ์˜ ์„ธ ์ถ•์œผ๋กœ ์ชผ๊ฐœ๊ณ , ๋ฏธ๋ž˜ ๋ชฉ์ ํ•จ์ˆ˜ ์ถ•์—์„œ ์‹œ๊ฐ ์ƒ์„ฑ์„ ๊ณ ์ • future anchor feature์˜ ์˜ˆ์ธก์œผ๋กœ ๋ฐ”๊ฟจ๋‹ค. ๊ธฐ๋ก๋œ ๋ฏธ๋ž˜๋Š” target์œผ๋กœ๋งŒ ์“ฐ์ด๊ณ , ํ•™์Šต ์ „์šฉ head๋Š” ๋ฐฐํฌ ๋•Œ ์‚ฌ๋ผ์ง„๋‹ค. ์•„ํ‚คํ…์ฒ˜ยท์˜ˆ์‚ฐ์„ ๊ณ ์ •ํ•œ ๋น„๊ต์—์„œ LIBERO-Plus๊ฐ€ FastWAM 55.31 โ†’ 64.54%, ImageWAM 65.27 โ†’ 79.95%๋กœ ์˜ฌ๋ž๊ณ , ๊ฐ™์€ ๋ ˆ์‹œํ”ผ๊ฐ€ DINOv2์™€ Qwen3.5-2B์—๋„ ์ ์šฉ๋๋‹ค. ํ†ต๊ณ„ ๋ณด๊ณ ์™€ ๋‹จ์„œ ๋‹ฌ๊ธฐ๊ฐ€ ์„ฑ์‹คํ•˜๊ณ , Fig. 1์˜ ๋ถ„ํ•ด๋Š” ๊ทธ ์ž์ฒด๋กœ ์“ธ๋ชจ ์žˆ๋Š” ๋„๊ตฌ๋‹ค.

ํ•œ๊ณ„๋Š” ๊ฒฐ๊ณผ์˜ ํ•ด์„ ๋ฒ”์œ„์— ์žˆ๋‹ค. Wan์—์„œ ์ด๊ธด target์€ ์ƒ์„ฑ๊ธฐ๊ฐ€ denoiseํ•˜๋˜ VAE latent์˜€๊ณ , ์‚ฌ์ „ํ•™์Šต Wan feature๋กœ๋Š” FastWAM๊ณผ ๋น„๊ฒผ๋‹ค. ๊ทธ๋Ÿฌ๋‹ˆ โ€œ์ƒ์„ฑ์ด ํ•„์š” ์—†๋‹คโ€๋Š” ํ™•์ธ๋์ง€๋งŒ โ€œํ‘œํ˜„์„ ์˜ˆ์ธกํ•ด์•ผ ํ•œ๋‹คโ€๊นŒ์ง€๋Š” ํ™•์ธ๋˜์ง€ ์•Š์•˜๋‹ค. FastWAM ๊ธฐ์ค€๊ฐ’์€ ์› ๋…ผ๋ฌธ ๋ณด๊ณ ๊ฐ’๋ณด๋‹ค ๋‚ฎ์•„์„œ, ์› ๋ณด๊ณ ๊ฐ’๊ณผ ๋Œ€์กฐํ•˜๋ฉด LIBERO ์šฐ์œ„๋Š” 0.55%p๋กœ ์ค„๊ณ  RoboTwin์—์„œ๋Š” ์—ญ์ „๋œ๋‹ค. target์€ ํ‰๊ฐ€ ๋ฒค์น˜๋งˆํฌ์—์„œ ๊ณจ๋ž๊ณ , ๋ฐฑ๋ณธ ๋น„๊ต๋Š” Qwen์˜ action-only๊ฐ€ ์ด๋ฏธ ๊ฐ•ํ•˜๋‹ค๋Š” ์‚ฌ์‹ค ๋•Œ๋ฌธ์— foresight์˜ ๋ชซ์„ ๊ฐ€๋ฆฌ๊ธฐ ์–ด๋ ต๋‹ค. ๋…ผ๋ฌธ v1์˜ ๋น„์šฉ 73%์—๋Š” FastWAM๋„ ๋ˆ„๋ฆด ์ˆ˜ ์žˆ๋Š” ์บ์‹œ ์ด๋“์ด ์„ž์—ฌ ์žˆ๊ณ , ๋ ˆํฌ๋Š” ์ด๋ฏธ ๊ฐ™์€ ์กฐ๊ฑด์˜ 57.7%๋กœ ํ‘œ๊ธฐ๋ฅผ ๊ณ ์ณค๋‹ค. ์ตœ์‹  ์ฝ”๋“œ๋Š” ์„ ํƒ๋œ JAM ์„ค์ •๋งŒ ๋‹ด๊ณ  ์žˆ์–ด ์ƒ์„ฑ vs foresight ๋น„๊ต์™€ target ์„ ํƒ์„ ๊ทธ๋Œ€๋กœ ๋‹ค์‹œ ๋Œ๋ฆฌ๊ธฐ ์–ด๋ ต๋‹ค. ๋‹ค์Œ์œผ๋กœ ๋ณด๊ณ  ์‹ถ์€ ๊ฒƒ์€ ๋ฏธ๋ž˜ ํ† ํฐ์„ ๋ฐฑ๋ณธ ์ž…๋ ฅ์— ๋‘” ์ฑ„ \mathcal L_F๋กœ๋งŒ ๊ฐ๋…ํ•˜๋Š” ์กฐ๊ฑด, validation ๋ถ„ํ• ์—์„œ ๊ณ ๋ฅธ target, ๋ฒ ์ด์Šค๋ผ์ธ์˜ ๋ฐ˜๋ณต ํ•™์Šต ๋ถ„์‚ฐ, ๊ทธ๋ฆฌ๊ณ  ์‹ค๋ฌผ ๋กœ๋ด‡ ์‹คํ—˜์ด๋‹ค.

์ด ๊ธ€์— ๋Œ€ํ•ด

  • ์ฝ์€ ๋ฒ”์œ„: arXiv v1(2026-10-27, 24์ชฝ)์˜ PDF์™€ e-print LaTeX ์†Œ์Šค, ํ”„๋กœ์ ํŠธ ํŽ˜์ด์ง€ ๋ณธ๋ฌธ๊ณผ ์˜์ƒ, GitHub README, Hugging Face ๋ชจ๋ธ ์นด๋“œ, ๊ณต๊ฐœ ๋ ˆํฌ์˜ ๊ตฌ์„ฑ๊ณผ ์ปค๋ฐ‹ ๊ธฐ๋ก์„ ์ฝ์—ˆ๋‹ค. ๋น„๊ต๋ฅผ ์œ„ํ•ด ๋ธ”๋กœ๊ทธ์˜ Fast-WAM ๋ฆฌ๋ทฐ(Fast-WAM ๋…ผ๋ฌธ v2 ๊ธฐ์ค€)์— ๊ธฐ๋ก๋œ ์ˆ˜์น˜๋ฅผ ๋Œ€์กฐํ–ˆ๋‹ค.
  • ๊ทธ๋ฆผ: Fig. 1ยท2๋Š” e-print ์†Œ์Šค์˜ ์›๋ณธ ๊ทธ๋ฆผ ํŒŒ์ผ, Fig. 3~9๋Š” PDF ํŽ˜์ด์ง€์—์„œ ๊ฐ€์ ธ์™”๋‹ค. ๊ทธ๋ฆผ ๋ฒˆํ˜ธ๋Š” PDF ๊ธฐ์ค€์ด๋‹ค.
  • ์˜์ƒ: ๋กค์•„์›ƒ ์˜์ƒ ์—ฌ์„ฏ ๊ฐœ๋Š” ํ”„๋กœ์ ํŠธ ํŽ˜์ด์ง€์˜ ๋ฒ ์ด์Šค๋ผ์ธยทJAM ํด๋ฆฝ ์Œ์„ ์ขŒ์šฐ๋กœ ๋‚˜๋ž€ํžˆ ๋ฐฐ์น˜ํ•˜๊ณ  ๋ฐฐ์†์„ ์ ์€ ๊ฒƒ์ด๋‹ค. ์งง์€ ์ชฝ์€ ๋๋‚œ ๋’ค ๋งˆ์ง€๋ง‰ ํ”„๋ ˆ์ž„์— ๋ฉˆ์ถ˜๋‹ค. ํŠธ๋ ˆ์ผ๋Ÿฌยทprobeยทattention ์˜์ƒ์€ ํ”„๋กœ์ ํŠธ ํŽ˜์ด์ง€์˜ mp4๋ฅผ ๊ทธ๋Œ€๋กœ ๋ถˆ๋Ÿฌ์˜จ๋‹ค(๋ฌด์Œ). ๋ชจ๋“  ๋กค์•„์›ƒ์€ ํ”„๋กœ์ ํŠธ ํŽ˜์ด์ง€๊ฐ€ ๊ณ ๋ฅธ ๋Œ€ํ‘œ ์—ํ”ผ์†Œ๋“œ๋‹ค.
  • ํ‘œ๊ธฐ: โ€œํ•„์ž ๊ณ„์‚ฐยท๊ฒ€์‚ฐโ€์€ ๋…ผ๋ฌธ ํ‘œ์˜ ๊ฐ’์œผ๋กœ ์ง์ ‘ ๊ณ„์‚ฐํ•œ ๊ฐ’(์ฐจ์ด, ํ•ฉ๊ณ„, McNemar exact ๊ฒ€์ •, ์บ์‹œ ์กฐ๊ฑด์„ ๋งž์ถ˜ FLOPs ๋น„์œจ)์ด๋‹ค. Fig. 1์˜ \tilde f_{1:T} ํ•ด์„์€ ๊ทธ๋ฆผ์— ์ •์˜๊ฐ€ ์—†์–ด ๋…ผ๋ฌธ์˜ ํ‘œ๊ธฐ ๊ด€๋ก€๋กœ ์ฝ์€ ๊ฒƒ์ด๋‹ค. ์ด ๋ฆฌ๋ทฐ์˜ ๋ชจ๋“  ์„ฑ๊ณต๋ฅ ์€ ๋…ผ๋ฌธ์ด ๋ณด๊ณ ํ•œ ๊ฐ’์ด๋ฉฐ ์žฌํ˜„ ํ™•์ธ์€ ํ•˜์ง€ ์•Š์•˜๋‹ค.

Copyright 2026, JungYeon Lee