Curieux.JY
  • JungYeon Lee
  • Post
  • ๐Ÿ•ธ๏ธ Graph
  • Lecture
  • Note

On this page

  • ๐Ÿ” Ping Review
  • ๐Ÿ”” Ring Review
    • ํ•œ ์ค„๋กœ ์‹œ์ž‘ํ•˜๋ฉด
    • ๋ฐฐ๊ฒฝ: ์™œ PPO๊ฐ€ ๊ธฐ๋ณธ๊ฐ’์ด์—ˆ๊ณ , ์™œ ์ด์ œ ํ”๋“ค๋ฆฌ๋Š”๊ฐ€ (ยง1)
    • ๊ด€๋ จ ์—ฐ๊ตฌ: ์„ธ ๊ฐˆ๋ž˜๋ฅผ ํ•˜๋‚˜๋กœ ๋ฌถ๋Š”๋‹ค๋Š” ์ฃผ์žฅ (ยง2)
    • ์‚ฌ์ „์ง€์‹: SAC (ยง3)
    • ๋ฐฉ๋ฒ• ์ƒ์„ธ (ยง4)
      • (i) Fast training: update๋ฅผ ์ค„์ด๊ณ  ๋‚˜๋จธ์ง€๋ฅผ ํ‚ค์šด๋‹ค
      • (ii) Stable training: norm์„ ์ „๋ถ€ ๋ฌถ๋Š”๋‹ค
      • (iii) Exploration: ์ฐจ์›์— ๋งž์ถ˜ entropy์™€ ์ƒ๊ด€๋œ noise
    • ์ง๊ด€: ์™œ โ€œ์ ๊ฒŒ, ํฌ๊ฒŒโ€๊ฐ€ ํ†ตํ•˜๋Š”๊ฐ€
    • ์‹คํ—˜: ์…‹์—…
    • ์‹คํ—˜: ์ƒํƒœ ๊ธฐ๋ฐ˜, GPU ์‹œ๋ฎฌ๋ ˆ์ดํ„ฐ (ยง5.1)
    • ์‹คํ—˜: ์ƒํƒœ ๊ธฐ๋ฐ˜, CPU ์‹œ๋ฎฌ๋ ˆ์ดํ„ฐ (ยง5.2)
    • ์‹คํ—˜: ๋น„์ „ ๊ธฐ๋ฐ˜ (ยง5.3)
    • ์‹คํ—˜: Sim-to-real (ยง5.4, ๋ถ€๋ก D)
    • ๋ถ„์„ (ยง6)
      • Scaling ablation (ยง6.2, Fig. 8)
      • Architectural ablation (ยง6.3, Fig. 9)
      • Exploration ablation (ยง6.4, Fig. 10)
    • ๋น„ํŒ์ ์œผ๋กœ ๋ณด๋ฉด
      • ๊ฐ•์ 
      • ์•ฝ์ ยทํ•œ๊ณ„
    • ๊ณต๊ฐœ ์ฝ”๋“œ์™€ ๋Œ€์กฐ
    • ๊ด€๋ จ ์—ฐ๊ตฌ์™€์˜ ์ž๋ฆฌ ๋งค๊น€
    • ์š”์•ฝ
    • ์ด ๊ธ€์— ๋Œ€ํ•ด

๐Ÿ“ƒFlashSAC ๋ฆฌ๋ทฐ

rl
off-policy
scaling
humanoid
sim2real
FlashSAC: Fast and Stable Off-Policy Reinforcement Learning for High-Dimensional Robot Control
Published

November 14, 2026

  • Paper Link (arXiv:2604.04539)

  • Code Link (Holiday-Robot/FlashSAC, MIT)

  • Project

  • Donghu Kim*, Youngdo Lee*, Minho Park, Kinam Kim, I Made Aswin Nahrendra, Takuma Seno, Sehee Min, Daniel Palenicek, Florian Vogt, Danica Kragic, Jan Peters, Jaegul Choo, Hojoon Lee (* ๊ณต๋™ 1์ €์ž)

  • Holiday Robotics ยท KAIST ยท KRAFTON ยท Turing ยท TU Darmstadt ยท hessian.AI ยท KTH ยท DFKI ยท RIG / RSS 2026 (arXiv v2, 2026-05-15)

  1. ๐Ÿ’ก PPO๋Š” ์•ˆ์ •์ ์ด์ง€๋งŒ on-policy ๋ฐ์ดํ„ฐ๊ฐ€ ์ข์•„ ์†ยทํœด๋จธ๋…ธ์ด๋“œ ๊ฐ™์€ ๊ณ ์ฐจ์› ๋ฌธ์ œ์—์„œ ๊ฐ€์น˜ ํ‰๊ฐ€๊ฐ€ ์–ด๋ ต๊ณ , off-policy๋Š” ๋„“์€ ๋ฐ์ดํ„ฐ๋ฅผ ์“ฐ์ง€๋งŒ critic์„ ๋งž์ถ”๋ ค๋ฉด gradient update๊ฐ€ ๋งŽ์ด ํ•„์š”ํ•ด ๋А๋ฆฌ๊ณ  bootstrapping ์˜ค์ฐจ๊ฐ€ ์Œ“์ธ๋‹ค. FlashSAC์€ update ์ˆ˜๋ฅผ ํ™• ์ค„์ด๋Š” ๋Œ€์‹  ๋ชจ๋ธยท๋ฐฐ์น˜ยท๋ฐ์ดํ„ฐ ์ฒ˜๋ฆฌ๋Ÿ‰์„ ํ‚ค์šฐ๊ณ , ๊ทธ ๊ทœ๋ชจ์—์„œ critic์ด ๋ฌด๋„ˆ์ง€์ง€ ์•Š๋„๋ก weightยทfeatureยทgradient norm์„ ๋ชจ๋‘ ๋ฌถ๋Š”๋‹ค.
  2. โš™๏ธ SAC ์œ„์— 1,024 ๋ณ‘๋ ฌ ํ™˜๊ฒฝ, 10M replay, batch 2,048, ์ „์ด 1,024๊ฐœ๋‹น update 2ํšŒ๋ฅผ ์–น๊ณ , inverted residual ๋ธ”๋ก + BatchNorm + post-RMSNorm + distributional critic(๋ณด์ƒ ์Šค์ผ€์ผ๋ง) + weight normalization์œผ๋กœ critic์„ ์•ˆ์ •ํ™”ํ•˜๋ฉฐ, ํ–‰๋™ ์ฐจ์›์— ๋น„๋ก€ํ•˜๋Š” entropy target๊ณผ noise repetition์œผ๋กœ ํƒ์ƒ‰์„ ๋„“ํžŒ๋‹ค.
  3. ๐ŸŽฏ 10๊ฐœ ์‹œ๋ฎฌ๋ ˆ์ดํ„ฐ 60์—ฌ ๊ฐœ ๊ณผ์ œ์—์„œ ๊ณ ์ฐจ์› ๊ณผ์ œ์ผ์ˆ˜๋ก PPOยทFastTD3 ๋Œ€๋น„ ์šฐ์œ„๊ฐ€ ํฌ๊ณ (Fig. 1 ์ง‘๊ณ„: ๊ณ ์ฐจ์› 10๊ณผ์ œ ์ •๊ทœํ™” ์ ์ˆ˜ FlashSAC ์•ฝ 0.91(์•ฝ 22๋ถ„) vs PPO ์•ฝ 0.53(์•ฝ 76๋ถ„), ๊ทธ๋ฆผ์—์„œ ์ฝ์€ ๊ฐ’), Unitree G1 ์‹ค๊ธฐ ๋ณดํ–‰์€ ํ‰์ง€ ์•ฝ 20๋ถ„(PPO ์•ฝ 3์‹œ๊ฐ„), ๊ณ„๋‹จ ์•ฝ 4์‹œ๊ฐ„(PPO ์•ฝ 20์‹œ๊ฐ„) ํ•™์Šต์œผ๋กœ ์˜ฎ๊ฒจ์กŒ๋‹ค. ๋‹ค๋งŒ wall-clock์€ ์ธก์ •์ด ์•„๋‹ˆ๋ผ ๊ตฌ์„ฑ ์š”์†Œ ์‹œ๊ฐ„์˜ ํ•ฉ์œผ๋กœ ์ถ”์ •ํ•œ ๊ฐ’์ด๊ณ , sim-to-real ๋น„๊ต๋Š” ๋‘ ๋ฐฉ๋ฒ•์˜ ๋ณด์ƒ ๊ฐ€์ค‘์น˜๊ฐ€ ๋‹ค๋ฅด๋‹ค.

๐Ÿ” Ping Review

๐Ÿ” Ping โ€” A light tap on the surface. Get the gist in seconds.

์‹œ๋ฎฌ๋ ˆ์ด์…˜ ๊ธฐ๋ฐ˜ ๋กœ๋ด‡ RL์˜ ๊ธฐ๋ณธ๊ฐ’์€ ์˜ค๋žซ๋™์•ˆ PPO์˜€๋‹ค. ์‚ฌ์กฑ๋ณดํ–‰ยท๊ทธ๋ฆฌํผ ์กฐ์ž‘์ฒ˜๋Ÿผ ์ƒํƒœยทํ–‰๋™ ์ฐจ์›์ด ๋‚ฎ๊ณ  ์‹œ๋ฎฌ๋ ˆ์ดํ„ฐ๊ฐ€ ๋งค์šฐ ๋น ๋ฅธ ์˜์—ญ์—์„œ๋Š” ๋ฐ์ดํ„ฐ๋ฅผ ๋ฒ„๋ ค๋„ ๋‹ค์‹œ ๋ชจ์œผ๋ฉด ๊ทธ๋งŒ์ด๊ธฐ ๋•Œ๋ฌธ์ด๋‹ค. ์ด ๋…ผ๋ฌธ์€ ๊ทธ ์ „์ œ๊ฐ€ ํœด๋จธ๋…ธ์ด๋“œ, dexterous manipulation, ๋น„์ „ ์ œ์–ด๋กœ ๋„˜์–ด๊ฐ€๋ฉฐ ๊นจ์ง„๋‹ค๊ณ  ๋ณธ๋‹ค. ์ฐจ์›์ด ๋†’์•„์ง€๋ฉด ํ˜„์žฌ ์ •์ฑ… ์ฃผ๋ณ€์˜ ์ข์€ on-policy ๋ฐ์ดํ„ฐ๋งŒ์œผ๋กœ๋Š” ๊ฐ€์น˜ ํ‰๊ฐ€๊ฐ€ ์–ด๋ ค์›Œ์ง€๊ณ , ์‹œ๋ฎฌ๋ ˆ์ด์…˜๊ณผ ์ •์ฑ… ์ถ”๋ก ๋„ ๋น„์‹ธ์ง„๋‹ค(ยง1).

off-policy RL์€ replay buffer๋กœ ๋„“์€ ๋ถ„ํฌ๋ฅผ ์žฌ์‚ฌ์šฉํ•˜๋ฏ€๋กœ ์ž์—ฐ์Šค๋Ÿฌ์šด ๋Œ€์•ˆ์ด์ง€๋งŒ, ๋„“์€ ๋ฐ์ดํ„ฐ ์œ„์—์„œ critic์„ ๋งž์ถ”๋ ค๋ฉด update๊ฐ€ ๋งŽ์ด ํ•„์š”ํ•˜๊ณ , critic์ด ์ž๊ธฐ ์˜ˆ์ธก์„ target์œผ๋กœ ์“ฐ๋Š” bootstrapping ๋•Œ๋ฌธ์— ์˜ค์ฐจ๊ฐ€ ๋ฐ˜๋ณต update๋ฅผ ๋”ฐ๋ผ ์ฆํญ๋œ๋‹ค. FlashSAC์˜ ๊ฒฐ์ •์  ์ฐจ์ด๋Š” โ€œupdate๋ฅผ ๋งŽ์ด ํ•˜๋Š”โ€ ๋Œ€์‹  โ€œupdate๋ฅผ ์ ๊ฒŒ, ํฌ๊ฒŒ ํ•˜๋Š”โ€ ์ฒด์ œ๋กœ ์˜ฎ๊ธด ๊ฒƒ์ด๋‹ค. ์ง€๋„ํ•™์Šต์˜ scaling ๊ฒฝํ–ฅ(๊ฐ™์€ ์—ฐ์‚ฐ ์˜ˆ์‚ฐ์ด๋ฉด ํฐ ๋ชจ๋ธยทํฐ ๋ฐฐ์น˜ยท์ ์€ update๊ฐ€ ๋” ๋นจ๋ฆฌ ์ˆ˜๋ ดํ•œ๋‹ค)์„ off-policy RL์— ๊ฐ€์ ธ์˜ค๊ณ , ๊ทธ ์ฒด์ œ์—์„œ ํฐ critic์ด ๋ถˆ์•ˆ์ •ํ•ด์ง€๋Š” ๋ฌธ์ œ๋ฅผ norm ์ œ์•ฝ์œผ๋กœ ๋ง‰๋Š”๋‹ค.


๊ฒฐ๊ณผ ๊ฐœ์š”(Fig. 1). (a) ์ €์ฐจ์› 15๊ณผ์ œ(Franka ์กฐ์ž‘, ์‚ฌ์กฑ๋ณดํ–‰): FlashSAC์€ PPO์™€ ๋น„์Šทํ•˜๊ฑฐ๋‚˜ ์•ฝ๊ฐ„ ๋†’๋‹ค. (b) ๊ณ ์ฐจ์› 10๊ณผ์ œ(AllegroยทShadow ์†, G1ยทH1ยทT1 ํœด๋จธ๋…ธ์ด๋“œ): FlashSAC์ด ๋” ๋†’์€ ์ ์ˆ˜์— ๋” ์งง์€ ์‹œ๊ฐ„ ์•ˆ์— ๋„๋‹ฌํ•œ๋‹ค. (c) Unitree G1 sim-to-real ๋ณดํ–‰: ์‹œ๋ฎฌ๋ ˆ์ด์…˜ ์„ ์†๋„ ์ถ”์ข… ๊ณก์„ ์—์„œ FlashSAC์€ ์•ฝ 30๋ถ„, PPO๋Š” ์•ฝ 180๋ถ„.
ํ”„๋กœ์ ํŠธ ํŽ˜์ด์ง€์˜ Unitree G1 ์‹ค๊ธฐ ์˜์ƒ. ์™ผ์ชฝ๋ถ€ํ„ฐ ํ‰์ง€ ๋ณดํ–‰, ๋ฐ€๊ธฐ ์™ธ๋ž€, ๊ณ„๋‹จ ์˜ค๋ฅด๊ธฐ. ๋…ผ๋ฌธ์—๋Š” ์ด ์‹œ์—ฐ๋“ค์˜ ์‹œ๋„ ํšŸ์ˆ˜ยท์„ฑ๊ณต๋ฅ ์ด ์—†๋‹ค.

ํ•ต์‹ฌ ๋ฐฉ๋ฒ•๋ก :

์ถœ๋ฐœ์ ์€ off-policy critic์˜ bootstrapped Bellman ์†์‹ค์ด๋‹ค(Eq. 1).

\mathcal{L}_Q=\mathbb{E}_{(s,a,r,s')\sim\mathcal{D}}\Big[\big(Q_\theta(s,a)-(r+\gamma Q_\theta(s',a'))\big)^2\Big],\qquad a'\sim\pi(\cdot\mid s')

target์ด Q_\theta ์ž์‹ ์— ์˜์กดํ•˜๋ฏ€๋กœ, ๋ฐ์ดํ„ฐ๊ฐ€ ํฌ๋ฐ•ํ•œ (s',a')์—์„œ ์ƒ๊ธด ์˜ค์ฐจ๊ฐ€ update๋งˆ๋‹ค ํ˜„์žฌ ๊ฐ’์œผ๋กœ ๋ฒˆ์ง„๋‹ค. FlashSAC์€ ์„ธ ๋ฌถ์Œ์œผ๋กœ ๋Œ€์‘ํ•œ๋‹ค(ยง4).

  • Fast training: 1,024 ๋ณ‘๋ ฌ ํ™˜๊ฒฝ, ์ตœ๋Œ€ 10M replay, actorยทcritic ํ•ฉ๊ณ„ ์•ฝ 2.5M ํŒŒ๋ผ๋ฏธํ„ฐ์˜ 6์ธต ๋„คํŠธ์›Œํฌ, batch 2,048, update-to-data(UTD) ๋น„์œจ 2/1024.
  • Stable training: inverted residual ๋ธ”๋ก, ๊ฐ ๋น„์„ ํ˜• ์•ž์˜ BatchNorm, ๋งˆ์ง€๋ง‰ ๋ธ”๋ก ๋’ค RMSNorm, 101๊ฐœ atom์˜ categorical(distributional) critic, ๋งค step ๋’ค weight๋ฅผ ๋‹จ์œ„ ๊ตฌ๋กœ ํˆฌ์˜ํ•˜๋Š” weight normalization. distributional critic์˜ ๊ณ ์ •๋œ ์ง€์ง€์ง‘ํ•ฉ [G_{\min},G_{\max}] ์•ˆ์— return์ด ๋“ค์–ด์˜ค๋„๋ก ๋ณด์ƒ์„ ๋‚˜๋ˆˆ๋‹ค(Eq. 6).

\bar r_t=\frac{r_t}{\max\!\Big(\sqrt{\sigma^2_{t,G}+\epsilon},\; G_{t,\max}/G_{\max}\Big)}

\sigma^2_{t,G}๋Š” ํ• ์ธ return์˜ running ๋ถ„์‚ฐ, G_{t,\max}๋Š” ์ง€๊ธˆ๊นŒ์ง€ ๋ณธ return ํฌ๊ธฐ์˜ ์ตœ๋Œ“๊ฐ’์ด๋‹ค. ๋ถ„๋ชจ์˜ ๋‘ ๋ฒˆ์งธ ํ•ญ์ด โ€œ์ตœ๋Œ€ return์ด ์ง€์ง€์ง‘ํ•ฉ ๋์„ ๋„˜์ง€ ์•Š๊ฒŒโ€ ํ•˜๋Š” ํ•˜ํ•œ ์—ญํ• ์„ ํ•œ๋‹ค. - Exploration: entropy target์„ ๊ณผ์ œ๋งˆ๋‹ค ์ •ํ•˜์ง€ ์•Š๊ณ  ๊ณ ์ • ํ‘œ์ค€ํŽธ์ฐจ \sigma_{\text{tgt}}=0.15์˜ ๊ฐ€์šฐ์‹œ์•ˆ entropy๋กœ ์ •ํ•œ๋‹ค(Eq. 7). ํ–‰๋™ ์ฐจ์› |\mathcal A|์— ์„ ํ˜• ๋น„๋ก€ํ•œ๋‹ค.

\bar{\mathcal H}=\tfrac12|\mathcal A|\log\!\big(2\pi e\,\sigma_{\text{tgt}}^2\big)

์—ฌ๊ธฐ์— ํ–‰๋™ noise \epsilon\sim\mathcal N(0,I)๋ฅผ k step ๋™์•ˆ ์œ ์ง€ํ•˜๋Š” noise repetition์„ ๋”ํ•œ๋‹ค. k๋Š” Zeta ๋ถ„ํฌ P(k)\propto k^{-s}์—์„œ ๋ฝ‘๋Š”๋‹ค(s=2, ์ตœ๋Œ€ 16).

์ฃผ์š” ๊ฒฐ๊ณผ:

  • GPU ์‹œ๋ฎฌ๋ ˆ์ดํ„ฐ 25๊ณผ์ œ(Fig. 3): ์ €์ฐจ์› ๊ณผ์ œ์—์„œ๋Š” PPO์™€ ๋น„์Šทํ•˜๊ฑฐ๋‚˜ ์•ฝ๊ฐ„ ๋†’๊ณ , ๊ณ ์ฐจ์› ๊ณผ์ œ(์† 2์ข…ยทํœด๋จธ๋…ธ์ด๋“œ)์—์„œ๋Š” ์ ์ˆ˜์™€ ์‹œ๊ฐ„ ๋ชจ๋‘ ํฌ๊ฒŒ ์•ž์„ ๋‹ค. ์˜ˆ๋ฅผ ๋“ค์–ด Isaac Allegro Repose ์ตœ์ข… return์€ FlashSAC 5,199 vs FastTD3 4,306 vs PPO 912, Isaac G1 Rough๋Š” 41.2 vs 34.4 vs 11.3์ด๋‹ค(๊ณต๊ฐœ ๊ฒฐ๊ณผ CSV์˜ ์ตœ์ข… ํ‰๊ฐ€ 5-seed ํ‰๊ท , ํ•„์ž ๊ณ„์‚ฐ).
  • CPU ์‹œ๋ฎฌ๋ ˆ์ดํ„ฐ 39๊ณผ์ œ(Fig. 4): ๋‹จ์ผ ํ™˜๊ฒฝ 1M step์—์„œ XQCยทSimbaV2ยทMR.QยทTD-MPC2ยทPPO๋ฅผ ๋Œ€์ฒด๋กœ ์•ž์„ ๋‹ค. ๊ณต๊ฐœ CSV ์ตœ์ข…๊ฐ’์œผ๋กœ ์„ธ๋ฉด FlashSAC์€ ๊ณตํ†ต ๊ณผ์ œ ๊ธฐ์ค€ XQC๋ฅผ 25/29, SimbaV2๋ฅผ 35/39์—์„œ ๊ฐ™๊ฑฐ๋‚˜ ๋„˜๋Š”๋‹ค(ํ•„์ž ๊ณ„์‚ฐ).
  • ๋น„์ „ DMC 8๊ณผ์ œ(Fig. 5): 1M step์—์„œ ๊ฒฝ์Ÿ๋ ฅ ์žˆ๋Š” ์ ์ˆ˜์— ๋” ์งง์€ ์‹œ๊ฐ„ ์•ˆ์— ๋„๋‹ฌํ•œ๋‹ค. ๋‹จ, ๊ฐ™์€ 1M step ์ตœ์ข…๊ฐ’๋งŒ ๋ณด๋ฉด MR.Q๊ฐ€ 8๊ณผ์ œ ์ค‘ 5๊ณผ์ œ์—์„œ ๋” ๋†’๋‹ค(ํ•„์ž ๊ณ„์‚ฐ, ์•„๋ž˜ Ring ์ฐธ์กฐ).
  • Sim-to-real(G1, 29 DoF): ํ‰์ง€ ์•ฝ 20๋ถ„ vs PPO ์•ฝ 3์‹œ๊ฐ„, ๊ณ„๋‹จ(์‹ค๊ธฐ 15 cm) ์•ฝ 4์‹œ๊ฐ„ vs PPO ์•ฝ 20์‹œ๊ฐ„(ยง5.4, Fig. 6).
  • ๋ถ„์„(ยง6): replay buffer๋Š” 10M๊นŒ์ง€ ํ‚ค์šธ์ˆ˜๋ก ์ข‹๊ณ  50M์€ 50M step ์˜ˆ์‚ฐ์—์„œ๋Š” ๋А๋ฆฌ๋‹ค(Fig. 8a). ์•„ํ‚คํ…์ฒ˜ ๊ตฌ์„ฑ ์š”์†Œ๋ฅผ ํ•˜๋‚˜์”ฉ ๋”ํ•˜๋ฉด ์„ฑ๋Šฅ์ด ์˜ค๋ฅด๋Š”๋ฐ, ๊ฐ€์žฅ ํฐ ๋„์•ฝ์€ distributional critic + reward scaling์ด๋‹ค(Fig. 9).

๊ฒฐ๋ก :

FlashSAC์€ ์ƒˆ ์†์‹ค์ด๋‚˜ ์ƒˆ ์ด๋ก ๋ณด๋‹ค โ€œ์–ด๋А ์ฒด์ œ์—์„œ off-policy๋ฅผ ๋Œ๋ฆด ๊ฒƒ์ธ๊ฐ€โ€์— ๋Œ€ํ•œ ๊ณตํ•™์  ์ฒ˜๋ฐฉ์ด๋‹ค. SimbaV2ยทXQCยทCrossQ๊ฐ€ ์Œ“์•„ ์˜จ norm ์ œ์•ฝ ๊ณ„์—ด ์•ˆ์ •ํ™”๋ฅผ FastTD3๋ฅ˜ ๋Œ€๊ทœ๋ชจ ๋ณ‘๋ ฌ ์ฒด์ œ์— ๋ถ™์ด๊ณ , update๋ฅผ ์ค„์—ฌ wall-clock์„ ์•„๊ผˆ๋‹ค. ๊ณ ์ฐจ์› ๊ณผ์ œ์—์„œ์˜ ์ผ๊ด€๋œ ์ด๋“๊ณผ ํ•™์Šต๊ณก์„  ์›์ž๋ฃŒ ๊ณต๊ฐœ๋Š” ๊ฐ•์ ์ด๋‹ค. ๋ฐ˜๋ฉด wall-clock์€ ์ถ”์ •๊ฐ’์ด๊ณ , ์‹ค๊ธฐ ๋น„๊ต๋Š” ๋ณด์ƒ ์„ค๊ณ„๊ฐ€ ๋‹ค๋ฅด๋ฉฐ, ๋…ผ๋ฌธ ๋ณธ๋ฌธยท๋ถ€๋ก ํ‘œยท๊ณต๊ฐœ ์Šคํฌ๋ฆฝํŠธ ์‚ฌ์ด์— UTDยทn-stepยท๊ณผ์ œ ์ˆ˜ ๊ฐ™์€ ๋ถˆ์ผ์น˜๊ฐ€ ์—ฌ๋Ÿฟ ์žˆ๋‹ค.


๐Ÿ”” Ring Review

๐Ÿ”” Ring โ€” An idea that echoes. Grasp the core and its value.

ํ•œ ์ค„๋กœ ์‹œ์ž‘ํ•˜๋ฉด

off-policy RL์ด ๋А๋ฆฐ ์ด์œ ๋Š” update๊ฐ€ ๋งŽ์•„์•ผ ํ•˜๊ธฐ ๋•Œ๋ฌธ์ด๊ณ , update๊ฐ€ ๋งŽ์œผ๋ฉด bootstrapping ์˜ค์ฐจ๊ฐ€ ์Œ“์ธ๋‹ค. ๊ทธ๋Ÿฌ๋‹ˆ update๋ฅผ ์ค„์ด๊ณ  ๋Œ€์‹  ๋ฐ์ดํ„ฐยท๋ฐฐ์น˜ยท๋ชจ๋ธ์„ ํ‚ค์šฐ๋˜, ๊ทธ ํฐ critic์ด ์˜ค์ฐจ๋ฅผ ์ฆํญํ•˜์ง€ ์•Š๋„๋ก weightยทfeatureยทgradient์˜ ํฌ๊ธฐ๋ฅผ ๊ตฌ์กฐ์ ์œผ๋กœ ๋ฌถ์–ด๋ผ. FlashSAC์€ ์ด ์ฒ˜๋ฐฉ์„ ํ•˜๋‚˜์˜ ๊ณ ์ •๋œ ์„ค์ •์œผ๋กœ 60์—ฌ ๊ฐœ ๊ณผ์ œ์— ์ ์šฉํ•ด PPO๊ฐ€ ๊ธฐ๋ณธ๊ฐ’์ธ sim-to-real ํŒŒ์ดํ”„๋ผ์ธ์— off-policy๋ฅผ ๋„ฃ์„ ์ˆ˜ ์žˆ๋‹ค๊ณ  ์ฃผ์žฅํ•œ๋‹ค.

๋ฐฐ๊ฒฝ: ์™œ PPO๊ฐ€ ๊ธฐ๋ณธ๊ฐ’์ด์—ˆ๊ณ , ์™œ ์ด์ œ ํ”๋“ค๋ฆฌ๋Š”๊ฐ€ (ยง1)

PPO์˜ ์žฅ์ ์€ ์•ˆ์ •์„ฑ๊ณผ ํŠœ๋‹ ์šฉ์ด์„ฑ์ด๋‹ค. ๋ฒ„๋ฆฌ๋Š” ๋ฐ์ดํ„ฐ๊ฐ€ ์•„๊น์ง€ ์•Š์„ ๋งŒํผ ์‹œ๋ฎฌ๋ ˆ์ดํ„ฐ๊ฐ€ ๋น ๋ฅด๋ฉด ์ด ์žฅ์ ์ด ๋‹จ์ ์„ ์••๋„ํ•œ๋‹ค. ๊ทธ๋ž˜์„œ sim-to-real RL์ด ๊ฐ€์žฅ ์ž˜ ๋œ ์˜์—ญ์€ ์‚ฌ์กฑ๋ณดํ–‰๊ณผ ๊ทธ๋ฆฌํผ ์กฐ์ž‘, ์ฆ‰ ์ €์ฐจ์› + ์ดˆ๊ณ ์† ์‹œ๋ฎฌ๋ ˆ์ดํ„ฐ ์กฐํ•ฉ์ด์—ˆ๋‹ค.

๋…ผ๋ฌธ์€ ์ด ์กฐํ•ฉ์ด ํ˜„๋Œ€ ๋กœ๋ด‡ ํ•™์Šต์„ ๋Œ€ํ‘œํ•˜์ง€ ๋ชปํ•˜๊ฒŒ ๋๋‹ค๊ณ  ์ฃผ์žฅํ•œ๋‹ค. ๊ทผ๊ฑฐ๋Š” ์„ธ ๊ฐ€์ง€๋‹ค.

  1. ์ฐจ์›: ํœด๋จธ๋…ธ์ด๋“œยท๋‹ค์ง€ ์†ยท๋น„์ „ ์ œ์–ด๋Š” ์ƒํƒœยทํ–‰๋™ ์ฐจ์›์ด ํฌ๋‹ค. ํ˜„์žฌ ์ •์ฑ… ์ฃผ๋ณ€์— ๋ชฐ๋ฆฐ on-policy ๋ฐ์ดํ„ฐ๋กœ๋Š” ๊ฐ€์น˜ ํ‰๊ฐ€์™€ ์ •์ฑ… ๊ฐœ์„ ์ด ์–ด๋ ต๋‹ค.
  2. ์‹œ๋ฎฌ๋ ˆ์ด์…˜ ๋น„์šฉ: ๋ณต์žกํ•œ ์ ‘์ด‰ ์—ญํ•™์€ ์‹œ๋ฎฌ๋ ˆ์ดํ„ฐ๋ฅผ ๋А๋ฆฌ๊ฒŒ ํ•œ๋‹ค.
  3. ์ •์ฑ… ํฌ๊ธฐ: ํฐ ์ •์ฑ…์€ rollout ์ž์ฒด๊ฐ€ ๋น„์‹ธ๋‹ค.

on-policy๊ฐ€ ๊ณผ๊ฑฐ ๋ฐ์ดํ„ฐ๋ฅผ ์“ฐ๋ ค๋ฉด importance sampling์ด ํ•„์š”ํ•œ๋ฐ, ๊ณ ์ฐจ์› ์—ฐ์† ํ–‰๋™์—์„œ๋Š” importance weight์˜ ๋ถ„์‚ฐ์ด ๋„ˆ๋ฌด ์ปค์„œ ์‹ค์šฉ์ ์ด์ง€ ์•Š๋‹ค(ยง2.1). ๊ทธ๋ž˜์„œ off-policy๊ฐ€ โ€œ์ž์—ฐ์Šค๋Ÿฌ์šด ๋Œ€์•ˆโ€์ด์ง€๋งŒ, critic ํ•™์Šต์ด ๋А๋ฆฌ๊ณ  ๋ถˆ์•ˆ์ •ํ•ด์„œ sim-to-real์˜ ๊ธฐ๋ณธ๊ฐ’์ด ๋˜์ง€ ๋ชปํ–ˆ๋‹ค๋Š” ๊ฒƒ์ด ๋…ผ๋ฌธ์˜ ๋ฌธ์ œ ์„ค์ •์ด๋‹ค.

๊ด€๋ จ ์—ฐ๊ตฌ: ์„ธ ๊ฐˆ๋ž˜๋ฅผ ํ•˜๋‚˜๋กœ ๋ฌถ๋Š”๋‹ค๋Š” ์ฃผ์žฅ (ยง2)

์ด ๋…ผ๋ฌธ์€ ๊ด€๋ จ ์—ฐ๊ตฌ๋ฅผ ์„œ๋ก  ์งํ›„์— ๋‘๊ณ , ๊ธฐ์กด off-policy model-free ์—ฐ๊ตฌ๋ฅผ ์„ธ ๊ฐ€์ง€ ๊ณผ์ œ๋กœ ๋‚˜๋ˆˆ๋‹ค(v2์—์„œ exploration์ด ์„ธ ๋ฒˆ์งธ ๊ฐˆ๋ž˜๋กœ ์ถ”๊ฐ€๋๋‹ค).

๊ฐˆ๋ž˜ ๋Œ€ํ‘œ ์—ฐ๊ตฌ(๋…ผ๋ฌธ ์ธ์šฉ) ๋…ผ๋ฌธ์ด ์ง€์ ํ•˜๋Š” ํ•œ๊ณ„
์†๋„: ๋ณ‘๋ ฌ ์‹œ๋ฎฌยทํฐ replay PQL, FastTD3, FastSAC ์•ฝ 0.2M ํŒŒ๋ผ๋ฏธํ„ฐ์˜ ์ž‘์€ ๋„คํŠธ์›Œํฌ๋ผ ์ ๊ทผ ์„ฑ๋Šฅ์ด ์ œํ•œ๋˜๊ณ , ํ‚ค์šฐ๋ฉด ๋ถˆ์•ˆ์ •
์•ˆ์ •์„ฑ: featureยทweightยทgradient norm ์ œ์•ฝ CrossQ, BRO, SimBa, SimbaV2, CrossQ-WN, XQC ๋“ฑ ํฐ ๋ชจ๋ธ์ด ์ˆ˜๋ ดํ•˜๋ ค๋ฉด update๊ฐ€ ๋งŽ์•„ ๋ฐ์ดํ„ฐ๊ฐ€ ํ’๋ถ€ํ•œ ์‹œ๋ฎฌ ์ฒด์ œ์—์„œ ๋А๋ฆผ
ํƒ์ƒ‰: ์‹œ๊ฐ„ ์ƒ๊ด€ noise OU noise, pink noise, gSDE, ฮตz-greedy ๋А๋ฆฌ๊ณ  ๋ถˆ์•ˆ์ •ํ•œ ํ•™์Šต์ด๋ผ๋Š” ๋ณ‘๋ชฉ์€ ๊ทธ๋Œ€๋กœ

FlashSAC์˜ ์œ„์น˜๋Š” ์ด ํ‘œ์˜ ์„ธ ํ–‰์„ ํ•ฉ์นœ ๊ฒƒ์ด๋‹ค. ์•ˆ์ •์„ฑ ๊ฐˆ๋ž˜๋Š” ์ด ๋ธ”๋กœ๊ทธ์˜ SimBa ๋ฆฌ๋ทฐ์—์„œ ๋‹ค๋ฃฌ ๊ณ„๋ณด์™€ ์ง์ ‘ ์ด์–ด์ง„๋‹ค. FlashSAC์˜ ์ €์ž ์ค‘ Hojoon LeeยทDonghu KimยทJaegul ChooยทTakuma Seno๋Š” SimBa ์ €์ž์ด๊ณ , PalenicekยทPeters๋Š” CrossQ-WNยทXQC ์ชฝ ์ €์ž๋‹ค. ๊ทธ๋ž˜์„œ ์ด ๋…ผ๋ฌธ์€ โ€œ๋‚จ์˜ ๊ธฐ๋ฒ•์„ ์กฐํ•ฉํ–ˆ๋‹คโ€๊ธฐ๋ณด๋‹ค ๋‘ ์•ˆ์ •ํ™” ๊ณ„๋ณด์˜ ์ €์ž๋“ค์ด FastTD3์‹ ๋ณ‘๋ ฌ ์ฒด์ œ๋กœ ํ•จ๊ป˜ ์˜ฎ๊ฒจ ์˜จ ๊ฒฐ๊ณผ๋กœ ์ฝ๋Š” ํŽธ์ด ์ •ํ™•ํ•˜๋‹ค.

model-based(DreamerV3, TD-MPC2)๋Š” ๊ณ„ํšยท๋ชจ๋ธ ํ•™์Šต ๋น„์šฉ ๋•Œ๋ฌธ์— wall-clock์ด ์ค‘์š”ํ•œ ์ฒด์ œ์—์„œ ํ™•์žฅ์„ฑ์ด ๋–จ์–ด์ง„๋‹ค๊ณ  ์ •๋ฆฌํ•œ๋‹ค(ยง2.2).

์‚ฌ์ „์ง€์‹: SAC (ยง3)

FlashSAC์€ clipped double Q๋ฅผ ์“ฐ๋Š” SAC ์œ„์— ์„œ ์žˆ๋‹ค. ์ •์ฑ…์€ ์—”ํŠธ๋กœํ”ผ ๋ณด์ •๋œ Q๋ฅผ ์ตœ๋Œ€ํ™”ํ•˜๊ณ (Eq. 2), critic ๋‘ ๊ฐœ๋Š” EMA target ๋„คํŠธ์›Œํฌ \bar\phi_j\leftarrow\tau\phi_j+(1-\tau)\bar\phi_j๋กœ ๋งŒ๋“  target์„ ๋งž์ถ˜๋‹ค.

\mathcal L_\pi(\theta)=\mathbb E_{s\sim\mathcal D,\,a\sim\pi_\theta}\Big[\alpha\log\pi_\theta(a\mid s)-\min_{i=1,2}Q_{\phi_i}(s,a)\Big]

y=r+\gamma\Big(\min_{j=1,2}Q_{\bar\phi_j}(s',a')-\alpha\log\pi_\theta(a'\mid s')\Big),\qquad a'\sim\pi_\theta(\cdot\mid s')

๋…ผ๋ฌธ์€ SAC์˜ entropy ์ตœ๋Œ€ํ™”๊ฐ€ ๊ณ ์ฐจ์›์—์„œ ํŠนํžˆ ์ค‘์š”ํ•˜๋‹ค๊ณ  ๊ฐ•์กฐํ•œ๋‹ค. ํƒ์ƒ‰์ด ๋ถ€์กฑํ•˜๋ฉด replay buffer์˜ ์ปค๋ฒ„๋ฆฌ์ง€๊ฐ€ ๋‚˜๋น ์ง€๊ณ , ๊ทธ๊ฒƒ์ด ๊ณง critic์˜ ์™ธ์‚ฝ ์˜ค์ฐจ๊ฐ€ ๋œ๋‹ค๋Š” ์—ฐ๊ฒฐ์ด๋‹ค.

๋ฐฉ๋ฒ• ์ƒ์„ธ (ยง4)

(i) Fast training: update๋ฅผ ์ค„์ด๊ณ  ๋‚˜๋จธ์ง€๋ฅผ ํ‚ค์šด๋‹ค

ํ•ญ๋ชฉ ํ‘œ์ค€ off-policy ์„ค์ •(๋…ผ๋ฌธ ์„œ์ˆ ) FlashSAC(GPU ์‹œ๋ฎฌ)
๋ณ‘๋ ฌ ํ™˜๊ฒฝ ์†Œ์ˆ˜ 1,024
replay buffer 1M ์ตœ๋Œ€ 10M
๋„คํŠธ์›Œํฌ 0.2~0.5M ํŒŒ๋ผ๋ฏธํ„ฐ, 2~3์ธต MLP actorยทcritic ํ•ฉ๊ณ„ ์•ฝ 2.5M, 6์ธต
batch (๋ช…์‹œ ์—†์Œ) 2,048
UTD ์ „์ด๋‹น ์—ฌ๋Ÿฌ ๋ฒˆ ์ „์ด 1,024๊ฐœ๋‹น 2ํšŒ
๊ตฌํ˜„ (๋ช…์‹œ ์—†์Œ) PyTorch JIT(torch.compile), mixed precision(5~10% ๋‹จ์ถ•)

๋…ผ๋ฆฌ๋Š” ์ง€๋„ํ•™์Šต์˜ scaling ๋ฒ•์น™(Kaplan et al., 2020)์—์„œ ์™”๋‹ค. ๊ณ ์ • ์—ฐ์‚ฐ ์˜ˆ์‚ฐ์—์„œ ํฐ ๋ชจ๋ธ + ํฐ ๋ฐฐ์น˜ + ์ ์€ update๊ฐ€ ์ž‘์€ ๋ชจ๋ธ + ์žฆ์€ update๋ณด๋‹ค ๋นจ๋ฆฌ ์ˆ˜๋ ดํ•œ๋‹ค. ์ด๊ฒƒ์„ off-policy์— ๋ฐ”๋กœ ๋ชป ์“ด ์ด์œ ๋Š” ํฐ critic์ด bootstrapping์—์„œ ๋ถˆ์•ˆ์ •ํ•ด์ง€๊ธฐ ๋•Œ๋ฌธ์ด๊ณ , FlashSAC์€ (ii)๋กœ ๊ทธ ์žฅ์• ๋ฅผ ์น˜์šด ๋’ค ์ด ์ฒด์ œ๋ฅผ ํƒํ•œ๋‹ค.

๋‘ ๊ฐ€์ง€ ํ•ด์„ ํฌ์ธํŠธ๊ฐ€ ์žˆ๋‹ค.

  • โ€œ2.5M-parameter, 6-layer network for both the actor and criticโ€์€ ํ•ฉ๊ณ„๋‹ค. ๊ณต๊ฐœ ๊ธฐ๋ณธ ์„ค์ •(actor ์€๋‹‰ 128ยท๋ธ”๋ก 2, critic ์€๋‹‰ 256ยท๋ธ”๋ก 2, ํ™•์žฅ ๋น„์œจ 4, critic 2๊ฐœ)์œผ๋กœ ์„ธ๋ฉด G1 Flat ๊ธฐ์ค€ actor ์•ฝ 0.29M, critic ๋‘ ๊ฐœ ํ•ฉ ์•ฝ 2.24M, ํ•ฉ๊ณ„ ์•ฝ 2.53M์ด๋‹ค. โ€œ6์ธตโ€์€ ์ž…๋ ฅ ์ž„๋ฒ ๋”ฉ 1 + ๋ธ”๋ก 2๊ฐœ ร— Linear 2 + ์ถœ๋ ฅ head 1์ด๋‹ค. actor ํ˜ผ์ž์„œ๋Š” 0.3M ์ˆ˜์ค€์œผ๋กœ, ๋…ผ๋ฌธ์ด โ€œ์ž‘๋‹คโ€๊ณ  ํ•œ FastTD3(์•ฝ 0.2M)์™€ ํฐ ์ฐจ์ด๊ฐ€ ์—†๋‹ค. ํ‚ค์šด ๊ฒƒ์€ ์ฃผ๋กœ critic์ด๋‹ค.
  • UTD๋Š” ๋ณธ๋ฌธ(ยง4.1)์ด 2/1024, ๋ถ€๋ก Table 9๊ฐ€ 2/2048๋กœ ์„œ๋กœ ๋‹ค๋ฅด๋‹ค. ๊ณต๊ฐœ ์Šคํฌ๋ฆฝํŠธ๋Š” num_train_envs=1024, updates_per_interaction_step=2๋กœ ๋ณธ๋ฌธ๊ณผ ๊ฐ™๋‹ค. Fig. 8e์˜ ์ถ• ๋ผ๋ฒจ๋„ x/1024๋‹ค. ๋ถ€๋ก ํ‘œ๊ฐ€ ์˜ค๊ธฐ๋กœ ๋ณด์ธ๋‹ค.

(ii) Stable training: norm์„ ์ „๋ถ€ ๋ฌถ๋Š”๋‹ค


FlashSAC ์•„ํ‚คํ…์ฒ˜(Fig. 2). BatchNorm โ†’ Linear ์ž„๋ฒ ๋”ฉ, [Linear(dโ†’4d) โ†’ BatchNorm โ†’ ReLU โ†’ Linear(4dโ†’d) โ†’ BatchNorm โ†’ ReLU] + residual์„ N๋ฒˆ, ๋งˆ์ง€๋ง‰์— RMSNorm โ†’ Linear head.

๋‹ค์„ฏ ์žฅ์น˜๋Š” ๊ฐ์ž ๋‹ค๋ฅธ ํฌ๊ธฐ๋ฅผ ๋ฌถ๋Š”๋‹ค.

์žฅ์น˜ ๋ฌถ๋Š” ๋Œ€์ƒ ๋…ผ๋ฌธ์˜ ๊ทผ๊ฑฐ
Inverted residual ๋ธ”๋ก gradient ํ๋ฆ„ Transformer FFN์‹ ํ™•์žฅยท์ถ•์†Œ + skip(ยง4.2)
Post-RMSNorm ์ƒ˜ํ”Œ๋ณ„ feature norm ๋ถ„ํฌ ๋ฐ– ์ž…๋ ฅ์ด ๋ฌดํ•œ์ • ํฐ ํ™œ์„ฑ์„ ๋งŒ๋“ค์ง€ ์•Š๊ฒŒ(ยง4.2)
Pre-activation BatchNorm ํ™œ์„ฑ ์Šค์ผ€์ผ replay ๋ฐ์ดํ„ฐ๊ฐ€ ์—ฌ๋Ÿฌ ์ •์ฑ…์˜ ํ˜ผํ•ฉ์ด๋ผ ์ž…๋ ฅ ๋ถ„ํฌ๊ฐ€ ๋น„์ •์ƒ(non-stationary). LayerNorm ๋Œ€์‹  BN์„ ํƒํ•œ ์ด์œ ๋Š” ํฐ ๋ฐฐ์น˜ ํ†ต๊ณ„๋ฅผ ์จ์„œ loss landscape์˜ condition number๋ฅผ ๋‚ฎ์ถ”๊ธฐ ์œ„ํ•จ(ยง4.2)
Cross-batch value prediction BN ํ†ต๊ณ„์˜ ์ผ๊ด€์„ฑ ํ˜„์žฌ (s,a)์™€ ๋‹ค์Œ (s',a')๋ฅผ ํ•œ ๋ฐฐ์น˜๋กœ ๋ฌถ์–ด ๊ฐ™์€ BN ํ†ต๊ณ„๋ฅผ ์“ฐ๊ฒŒ ํ•œ๋‹ค(CrossQ ๋ฐฉ์‹)
Distributional critic + reward scaling target์˜ ์žก์Œ ๋ฏผ๊ฐ๋„, return ๋ฒ”์œ„ categorical cross-entropy๊ฐ€ loss landscape๋ฅผ ๋งค๋„๋Ÿฝ๊ฒŒ ํ•˜๊ณ , Eq. 6์ด return์„ ์ง€์ง€์ง‘ํ•ฉ ์•ˆ์— ๋‘”๋‹ค
Weight normalization weight norm ๋งค step ๋’ค ๊ฐ weight ๋ฒกํ„ฐ๋ฅผ ๋‹จ์œ„ ๊ตฌ๋กœ, ์ •๊ทœํ™”์ธต์˜ (\gamma,\beta)๋ฅผ norm \sqrt d๋กœ ํˆฌ์˜. ์ •๋ณด๋ฅผ ํฌ๊ธฐ๊ฐ€ ์•„๋‹ˆ๋ผ ๋ฐฉํ–ฅ์œผ๋กœ ํ‘œํ˜„ํ•˜๊ฒŒ ํ•œ๋‹ค

๋ณธ๋ฌธ ์„œ์ˆ ๋งŒ์œผ๋กœ๋Š” ์ž˜ ๋“œ๋Ÿฌ๋‚˜์ง€ ์•Š๊ณ  ๊ทธ๋ฆผยท๊ณต๊ฐœ ์ฝ”๋“œ์—์„œ ํ™•์ธ๋˜๋Š” ์„ธ๋ถ€๊ฐ€ ์…‹ ์žˆ๋‹ค.

  • ์ž…๋ ฅ ๊ด€์ธก์—๋„ BatchNorm์ด ์žˆ๋‹ค. Fig. 2์˜ ์ฒซ BatchNorm ๋ฐ•์Šค๊ฐ€ ๊ทธ๊ฒƒ์ด๊ณ , ๊ณต๊ฐœ ์ฝ”๋“œ๋„ ์ž„๋ฒ ๋”ฉ Linear ์•ž์— BN์„ ๋‘”๋‹ค. ๋ณธ๋ฌธ์€ โ€œ๊ฐ ๋น„์„ ํ˜• ์•ž์˜ BNโ€๋งŒ ์„ค๋ช…ํ•˜์ง€๋งŒ, ์ด BN์ด ์‚ฌ์‹ค์ƒ ๊ด€์ธก ์ •๊ทœํ™”๊ธฐ๋‹ค. SimBa๋Š” RSNorm(running ํ†ต๊ณ„ ๊ธฐ๋ฐ˜ ๊ด€์ธก ํ‘œ์ค€ํ™”)์„ ํ•ต์‹ฌ ๋ถ€ํ’ˆ์œผ๋กœ ๋‚ด์„ธ์› ๊ณ  BNยทLayerNorm์œผ๋กœ๋Š” ๊ทธ ํšจ๊ณผ๊ฐ€ ์•ˆ ๋‚ฌ๋‹ค๊ณ  ๋ณด๊ณ ํ–ˆ๋Š”๋ฐ(SimBa ๋ฆฌ๋ทฐ์˜ Fig. 12 ์ ˆ), FlashSAC์€ ๊ทธ ์ž๋ฆฌ๋ฅผ ๋ฐฐ์น˜ ํ†ต๊ณ„ BN์œผ๋กœ ๋ฐ”๊ฟจ๋‹ค. ๋…ผ๋ฌธ์€ ์ด ๊ต์ฒด์˜ ์ด์œ ๋ฅผ ๋”ฐ๋กœ ๋…ผํ•˜์ง€ ์•Š๋Š”๋‹ค.
  • cross-batch๋ฅผ ์“ฐ๋ฉด์„œ๋„ target network๋ฅผ ์œ ์ง€ํ•œ๋‹ค. CrossQ๋Š” ์ด concat์œผ๋กœ target network๋ฅผ ์—†์•ค ๊ฒƒ์ด ํ•ต์‹ฌ์ด์—ˆ๋Š”๋ฐ, FlashSAC์€ ฯ„=0.01์˜ EMA target critic์„ ๊ทธ๋Œ€๋กœ ๋‘๊ณ , target critic๋„ ๋ฐฐ์น˜ ํ†ต๊ณ„ ๋ชจ๋“œ๋กœ ๋Œ๋ฆฐ๋‹ค. ๊ณต๊ฐœ ์ฝ”๋“œ์˜ EMA๋Š” ํŒŒ๋ผ๋ฏธํ„ฐ๋งŒ ์˜ฎ๊ธฐ๊ณ  BN์˜ running ํ†ต๊ณ„ ๋ฒ„ํผ๋Š” ๋ณต์‚ฌํ•˜์ง€ ์•Š์œผ๋ฏ€๋กœ, target์„ ๋ฐฐ์น˜ ํ†ต๊ณ„๋กœ ๋Œ๋ฆฌ๋Š” ๊ฒƒ์ด ์ผ๊ด€๋œ ์„ ํƒ์ด๋‹ค.
  • residual ๊ฐ€์ง€์˜ ๋์ด ReLU๋‹ค. ๋ธ”๋ก ์ถœ๋ ฅ์ด x+\mathrm{ReLU}(\mathrm{BN}(\cdot))์ด๋ผ ๊ฐ ๋ธ”๋ก์€ feature์— 0 ์ด์ƒ์ธ ๊ฐ’๋งŒ ๋”ํ•œ๋‹ค. ๋ธ”๋ก์„ ์Œ“์„์ˆ˜๋ก feature ํฌ๊ธฐ๊ฐ€ ํ•œ ๋ฐฉํ–ฅ์œผ๋กœ ์ปค์งˆ ์ˆ˜ ์žˆ๊ณ , ๋งˆ์ง€๋ง‰ RMSNorm์ด ๊ทธ๊ฒƒ์„ ๋‹ค์‹œ ๋ฌถ๋Š” ๊ตฌ์กฐ๋‹ค. Fig. 9c์—์„œ โ€œ+Residualโ€ ๋‹จ๊ณ„์˜ feature norm์ด MLP๋ณด๋‹ค๋Š” ์ž‘์ง€๋งŒ ํ•™์Šต ๋‚ด๋‚ด ์ปค์ง€๋Š” ๊ฒƒ๊ณผ ๋งž์•„๋–จ์–ด์ง„๋‹ค(ํ•„์ž์˜ ํ•ด์„).

(iii) Exploration: ์ฐจ์›์— ๋งž์ถ˜ entropy์™€ ์ƒ๊ด€๋œ noise

Unified entropy target. SAC์˜ ์ž๋™ temperature ์กฐ์ •์€ target entropy๋ฅผ ๊ณผ์ œ๋งˆ๋‹ค ์ •ํ•ด์•ผ ํ•œ๋‹ค. FlashSAC์€ โ€œํ–‰๋™ ์ฐจ์›๋งˆ๋‹ค ํ‘œ์ค€ํŽธ์ฐจ 0.15์˜ ๊ฐ€์šฐ์‹œ์•ˆโ€์— ํ•ด๋‹นํ•˜๋Š” entropy๋ฅผ target์œผ๋กœ ์“ด๋‹ค. ์ฐจ์›๋‹น ๊ฐ’์€ \tfrac12\log(2\pi e\cdot0.15^2)\approx-0.478 nat์ด๋‹ค(ํ•„์ž ๊ณ„์‚ฐ). ํ–‰๋™์ด 37์ฐจ์›์ธ Isaac G1์ด๋ฉด ์•ฝ โˆ’17.7์ด๋‹ค.

Noise repetition. pink noiseยทOU noise๋Š” ํ™˜๊ฒฝ๋ณ„๋กœ ์ƒ๊ด€ noise ๊ณผ์ •์„ ์œ ์ง€ํ•ด์•ผ ํ•ด์„œ 1,024 ๋ณ‘๋ ฌ ํ™˜๊ฒฝ์—์„œ๋Š” ๊ณ„์‚ฐยท๋ฉ”๋ชจ๋ฆฌ ๋ถ€๋‹ด์ด ํฌ๋‹ค. FlashSAC์€ noise ๋ฒกํ„ฐ ํ•˜๋‚˜๋ฅผ k step ์œ ์ง€ํ•œ๋‹ค. s=2, ์ตœ๋Œ€ 16์˜ ์ ˆ๋‹จ Zeta ๋ถ„ํฌ๋ฉด P(k=1)\approx0.63, \mathbb E[k]\approx2.13, P(k\ge8)\approx0.046์ด๋‹ค(ํ•„์ž ๊ณ„์‚ฐ). ๋Œ€๋ถ€๋ถ„์€ ์งง๊ฒŒ, ๊ฐ€๋” ๊ธธ๊ฒŒ ๊ฐ™์€ ๋ฐฉํ–ฅ์œผ๋กœ ๋ฐ€์–ด ์ค€๋‹ค. ๊ณต๊ฐœ ์ฝ”๋“œ๋Š” ํ–‰๋™์„ \tanh(\mu+\sigma\epsilon)์œผ๋กœ ๋งŒ๋“ค๊ณ  \epsilon๋งŒ ๊ณ ์ •ํ•˜๋ฏ€๋กœ, ๊ฐ™์€ noise๊ฐ€ ์œ ์ง€๋˜๋Š” ๋™์•ˆ์—๋„ ์ •์ฑ…์˜ \mu,\sigma ๋ณ€ํ™”๋Š” ๊ทธ๋Œ€๋กœ ๋ฐ˜์˜๋œ๋‹ค. ๋˜ k์™€ ์นด์šดํ„ฐ๋Š” ๋ชจ๋“  ํ™˜๊ฒฝ์ด ๊ณต์œ ํ•˜๋Š” ์Šค์นผ๋ผ๋ผ์„œ 1,024๊ฐœ ํ™˜๊ฒฝ์ด ๊ฐ™์€ step์— ๋™์‹œ์— noise๋ฅผ ๋‹ค์‹œ ๋ฝ‘๋Š”๋‹ค. ์ด๊ฒƒ์€ ๋…ผ๋ฌธ์˜ โ€œwithout per-environment overheadโ€(ยง2.2), โ€œminimal local stateโ€(ยง4.3)์™€ ๋งž๋Š” ๊ตฌํ˜„์ด๋‹ค. ๋…ผ๋ฌธ์— ์ ํ˜€ ์žˆ์ง€ ์•Š์€ ๊ฒƒ์€ ์—ํ”ผ์†Œ๋“œ๊ฐ€ ๋๋‚˜ ํ™˜๊ฒฝ์ด reset๋ผ๋„ \epsilon์„ ๋‹ค์‹œ ๋ฝ‘์ง€ ์•Š๋Š”๋‹ค๋Š” ์  ํ•˜๋‚˜๋‹ค.

์ง๊ด€: ์™œ โ€œ์ ๊ฒŒ, ํฌ๊ฒŒโ€๊ฐ€ ํ†ตํ•˜๋Š”๊ฐ€

off-policy์˜ ๋А๋ฆผ์€ ๋ณดํ†ต โ€œ์ƒ˜ํ”Œ ํšจ์œจ์€ ์ข‹์€๋ฐ wall-clock์ด ๋‚˜์˜๋‹คโ€๋กœ ์š”์•ฝ๋œ๋‹ค. ์ „์ด ํ•˜๋‚˜๋‹น ์—ฌ๋Ÿฌ ๋ฒˆ updateํ•˜๋ฏ€๋กœ update ๋น„์šฉ์ด ์‹œ๋ฎฌ๋ ˆ์ด์…˜ ๋น„์šฉ์„ ์••๋„ํ•œ๋‹ค. FlashSAC์˜ ๊ด€์ ์€ ์ด ๋น„์šฉ ๊ตฌ์กฐ๋ฅผ ๋’ค์ง‘๋Š” ๊ฒƒ์ด๋‹ค. 1,024๊ฐœ ํ™˜๊ฒฝ์ด ๋งค step 1,024๊ฐœ ์ „์ด๋ฅผ ๋งŒ๋“ค๊ณ , ๊ทธ ์‚ฌ์ด update๋Š” ๋‘ ๋ฒˆ๋ฟ์ด๋‹ค. update ํ•œ ๋ฒˆ์ด ์ฒ˜๋ฆฌํ•˜๋Š” ์ •๋ณด๋Ÿ‰(batch 2,048 ร— ํฐ critic)์„ ๋Š˜๋ ค update ํšŸ์ˆ˜๋ฅผ ๋Œ€์‹ ํ•œ๋‹ค. GPU๋Š” ์ž‘์€ ํ–‰๋ ฌ ๊ณฑ ์—ฌ๋Ÿฌ ๋ฒˆ๋ณด๋‹ค ํฐ ํ–‰๋ ฌ ๊ณฑ ํ•œ ๋ฒˆ์— ํ›จ์”ฌ ํšจ์œจ์ ์ด๋ฏ€๋กœ ์ด ๊ตํ™˜์€ wall-clock์— ์œ ๋ฆฌํ•˜๋‹ค.

๊ทธ๋Ÿฌ๋ฉด ์™œ ๋‹ค๋ฅธ ์‚ฌ๋žŒ๋“ค์€ ์ด๊ฑธ ์•ˆ ํ–ˆ๋‚˜. update๊ฐ€ ์ ์œผ๋ฉด ํ•œ ๋ฒˆ์˜ update๊ฐ€ ํฌ๊ฒŒ ์›€์ง์—ฌ์•ผ ํ•˜๊ณ , ํฐ critic์€ ๊ทธ ํฐ ์›€์ง์ž„์—์„œ ์™ธ์‚ฝ ์˜ค์ฐจ๋ฅผ ๋” ํฌ๊ฒŒ ๋งŒ๋“ ๋‹ค. ์—ฌ๊ธฐ์„œ norm ์ œ์•ฝ์ด โ€œํฐ ๊ฑธ์Œ์„ ๊ฑธ์–ด๋„ ๋„˜์–ด์ง€์ง€ ์•Š๊ฒŒโ€ ํ•˜๋Š” ์—ญํ• ์„ ํ•œ๋‹ค. weight๋ฅผ ๋‹จ์œ„ ๊ตฌ์— ๋ถ™์žก์œผ๋ฉด effective learning rate๊ฐ€ weight ํฌ๊ธฐ์— ๋”ฐ๋ผ ์ค„์–ด๋“œ๋Š” ํ˜„์ƒ์ด ์‚ฌ๋ผ์ง€๊ณ , feature๋ฅผ RMSNorm์œผ๋กœ ๋ฌถ์œผ๋ฉด ์ฒ˜์Œ ๋ณด๋Š” (s',a')์—์„œ๋„ critic ์ถœ๋ ฅ์ด ํญ์ฃผํ•˜์ง€ ์•Š์œผ๋ฉฐ, distributional critic์€ target์˜ ์žก์Œ์„ cross-entropy๋กœ ํก์ˆ˜ํ•œ๋‹ค.

Fig. 7์€ ์ด ๋…ผ๋ฆฌ์˜ ๋‹ค๋ฅธ ์ชฝ ์ ˆ๋ฐ˜, ์ฆ‰ โ€œ์™œ off-policy ๋ฐ์ดํ„ฐ๊ฐ€ ํ•„์š”ํ•œ๊ฐ€โ€๋ฅผ ๋ณด์—ฌ ์ฃผ๋ ค๋Š” ๊ทธ๋ฆผ์ด๋‹ค.


Off-policy vs on-policy ๋ฐ์ดํ„ฐ ์ปค๋ฒ„๋ฆฌ์ง€(Fig. 7). Isaac Shadow Hand์—์„œ FlashSAC์„ 1M step ํ•™์Šตํ•œ ๋’ค (a) replay buffer์˜ ์ƒ˜ํ”Œ๊ณผ (b) ์ตœ์ข… ์ •์ฑ…์„ 1M step ๋” ๊ตด๋ฆฐ ์ƒ˜ํ”Œ์˜ ๋ฌผ์ฒด y ์œ„์น˜ ร— ์†๊ฐ€๋ฝ ๊ด€์ ˆ ํ–‰๋™ 2D ๋ฐ€๋„. (a)๊ฐ€ ํ›จ์”ฌ ๋„“๋‹ค.

๊ทธ๋ฆผ ์ž์ฒด๋Š” ์„ค๋“๋ ฅ์ด ์žˆ์ง€๋งŒ, ๋น„๊ต ๋Œ€์ƒ์ด PPO๊ฐ€ ๋ชจ์€ ๋ฐ์ดํ„ฐ๊ฐ€ ์•„๋‹ˆ๋ผ FlashSAC ์ตœ์ข… ์ •์ฑ…์˜ rollout์ด๋ผ๋Š” ์ ์— ์ฃผ์˜ํ•ด์•ผ ํ•œ๋‹ค. replay buffer๊ฐ€ ๋„“์€ ๊ฒƒ์€ ํ•™์Šต ์ดˆ๊ธฐ์˜ ์„œํˆฐ ์ •์ฑ… ๋ฐ์ดํ„ฐ๋ฅผ ๋‹ค ํ’ˆ๊ณ  ์žˆ๊ธฐ ๋•Œ๋ฌธ์ด๊ธฐ๋„ ํ•˜๋‹ค. ๊ทธ๋Ÿฌ๋‹ˆ ์ด ๊ทธ๋ฆผ์ด ๋ณด์—ฌ ์ฃผ๋Š” ๊ฒƒ์€ โ€œreplay buffer๊ฐ€ ์ตœ์ข… ์ •์ฑ… ๋ถ„ํฌ๋ณด๋‹ค ๋„“๋‹คโ€๋Š” ๊ฑฐ์˜ ์ •์˜์ƒ์˜ ์‚ฌ์‹ค์ด๊ณ , โ€œPPO์˜ on-policy ๋ฐ์ดํ„ฐ๊ฐ€ ํ•™์Šต ์ค‘์— ์ด๋งŒํผ ์ข์•„์„œ PPO๊ฐ€ ์ง„๋‹คโ€๋Š” ์ธ๊ณผ๊นŒ์ง€๋Š” ์•„๋‹ˆ๋‹ค. ๋…ผ๋ฌธ๋„ โ€œsuggestsโ€๋ผ๋Š” ๋™์‚ฌ๋ฅผ ์“ด๋‹ค(ยง6.1).

์‹คํ—˜: ์…‹์—…

์‹คํ—˜ ์‹œ๋ฎฌ๋ ˆ์ดํ„ฐ(๊ณผ์ œ ์ˆ˜) step ์˜ˆ์‚ฐ ๋น„๊ต ๋Œ€์ƒ FlashSAC ์„ค์ •
์ƒํƒœ ๊ธฐ๋ฐ˜, GPU(ยง5.1) IsaacLab 12, MuJoCo Playground 4, ManiSkill 6, Genesis 3 = 25 50M(PPO 200M) PPO(RSL-RL), FastTD3 ๋‹จ์ผ ์„ค์ •, ฮณ๋งŒ ์‹œ๋ฎฌ ๊ธฐ๋ณธ๊ฐ’
์ƒํƒœ ๊ธฐ๋ฐ˜, CPU(ยง5.2) MuJoCo 5, DMC 10, HumanoidBench 14, MyoSuite 10 1M(PPO 4M) PPO, XQC, SimbaV2, TD-MPC2, MR.Q batch 512, UTD 1
๋น„์ „(ยง5.3) DMC 8 1M, action repeat 2 DrQ-v2, MR.Q batch 256, UTD 0.5, 3-step, frame stack 3
Sim-to-real(ยง5.4) IsaacLab, G1 29 DoF blind ๋ณดํ–‰ ์‹œ๊ฐ„์œผ๋กœ ๋ณด๊ณ  PPO(DreamWaQ ํŒŒ์ดํ”„๋ผ์ธ) 4,096 ํ™˜๊ฒฝ, A100 1์žฅ

๊ณผ์ œ ๋ถ„๋ฅ˜๋Š” ์ €์ฐจ์› 15๊ฐœ(Franka, AnyMal-C/D, Go2)์™€ ๊ณ ์ฐจ์› 10๊ฐœ(Allegro, Shadow, G1, H1, Booster T1)๋‹ค. ๋ถ€๋ก ํ‘œ์˜ ์ฐจ์›์„ ๋ณด๋ฉด ๊ณ ์ฐจ์› ์ชฝ๋„ ํ–‰๋™ 16~37์ฐจ์›, ๊ด€์ธก 69~310์ฐจ์›์œผ๋กœ ๋Œ€๋ถ€๋ถ„ ์ˆ˜์‹ญ~์ˆ˜๋ฐฑ ์ฐจ์›์ด๋‹ค.

wall-clock์€ RTX 5090 + Ryzen 9 9950X3D ํ•œ ๋Œ€ ๊ธฐ์ค€์˜ ์ถ”์ •๊ฐ’์ด๋‹ค(๋ถ€๋ก A). ํ™˜๊ฒฝ ์ƒํ˜ธ์ž‘์šฉ ์‹œ๊ฐ„(์‹œ๋ฎฌ๋ ˆ์ด์…˜ยทbuffer ์—ฐ์‚ฐ)์€ ํ™˜๊ฒฝ๋งˆ๋‹ค, ์•Œ๊ณ ๋ฆฌ์ฆ˜ update ์‹œ๊ฐ„(์ถ”๋ก ยทgradient update)์€ ๋Œ€ํ‘œ ํ™˜๊ฒฝ(์ƒํƒœ ๊ธฐ๋ฐ˜์€ MuJoCo Humanoid-v4, ๋น„์ „์€ DMC Walker-run)์—์„œ ๋”ฐ๋กœ ์žฌ์„œ ๋”ํ•œ๋‹ค. ๋ชจ๋“  ๊ทธ๋ž˜ํ”„์˜ โ€œCompute Timeโ€ ์ถ•์ด ์ด ํ•ฉ์ด๋‹ค.

baseline์€ โ€œ๊ณต๊ฐœ ๊ฒฐ๊ณผ๊ฐ€ ์žˆ์œผ๋ฉด ๊ทธ๊ฒƒ์„, ์—†์œผ๋ฉด ๊ณต์‹ ๊ตฌํ˜„์œผ๋กœ ์žฌํ˜„โ€ํ–ˆ๋‹ค(ยง5.1). FlashSAC์€ ๋‹จ์ผ ์„ค์ •์ด๊ณ  baseline์€ ๊ณผ์ œ๋ณ„ ํŠœ๋‹์„ ์“ด๋‹ค๊ณ  ์ ๋Š”๋‹ค.

์‹คํ—˜: ์ƒํƒœ ๊ธฐ๋ฐ˜, GPU ์‹œ๋ฎฌ๋ ˆ์ดํ„ฐ (ยง5.1)


GPU ์‹œ๋ฎฌ๋ ˆ์ดํ„ฐ ๊ฒฐ๊ณผ(Fig. 3). (a) ์ €์ฐจ์›: Franka Open DrawerยทLift Cube, ManiSkill Pull Cube, Genesis Go2 Walk, Isaac AnymalC Rough. (b) ๊ณ ์ฐจ์›: Isaac AllegroยทShadow Repose, MJP Booster T1ยทG1 Flat, Isaac G1 Rough. x์ถ•์€ ์ถ”์ • compute time(๋ถ„).

๋…ผ๋ฌธ์˜ ์ฃผ์žฅ์€ ์„ธ ๋ฌธ์žฅ์ด๋‹ค. ์ €์ฐจ์›์—์„œ๋Š” PPO๋ณด๋‹ค ์•ฝ๊ฐ„ ๋‚ซ๊ณ , ๊ณ ์ฐจ์›์—์„œ๋Š” ๋ถ„๋ช…ํžˆ ๋‚ซ๊ณ , FastTD3๋ณด๋‹ค ์•ˆ์ •์ ์ด์–ด์„œ FastTD3๊ฐ€ ์‹คํŒจํ•˜๋Š” ๊ณผ์ œ(Go2 Walk, Franka Pull Cube)์—์„œ๋„ ์ˆ˜๋ ดํ•œ๋‹ค.

ํ•™์Šต๊ณก์„  ์›์ž๋ฃŒ๊ฐ€ ๋ ˆํฌ results/์— ๊ณต๊ฐœ๋ผ ์žˆ์–ด ๊ทธ๋ฆผ์˜ ๋์ ์„ ์ˆซ์ž๋กœ ํ™•์ธํ•  ์ˆ˜ ์žˆ๋‹ค. ์•„๋ž˜๋Š” ๊ทธ CSV์˜ ๋งˆ์ง€๋ง‰ ํ‰๊ฐ€๊ฐ’ 5-seed ํ‰๊ท ์ด๋‹ค(ํ•„์ž ๊ณ„์‚ฐ, ManiSkill์€ ์„ฑ๊ณต๋ฅ ).

๊ณผ์ œ ์ฐจ์›(๊ด€์ธก/ํ–‰๋™) FlashSAC FastTD3 PPO
Isaac Allegro Repose 124 / 16 5,199 4,306 912
Isaac Shadow Repose 157 / 20 9,800 9,618 4,901
Isaac G1 Rough 310 / 37 41.2 34.4 11.3
MJP G1 Flat 103 / 29 39.9 31.5 23.4
Isaac Franka Lift Cube 36 / 8 154.8 121.0 93.4 (seed ๊ฐ„ SD 62)
Genesis Go2 Walk 45 / 12 23.0 โˆ’0.3 18.5
ManiSkill PegInsertionSide (์„ฑ๊ณต๋ฅ ) 43 / 8 0.96 0.00 0.92
ManiSkill PokeCube (์„ฑ๊ณต๋ฅ ) 54 / 8 0.20 0.20 0.85

๊ณ ์ฐจ์› ์ชฝ์˜ ๊ฒฉ์ฐจ๋Š” ํ™•์‹คํ•˜๋‹ค. Allegro์—์„œ PPO๊ฐ€ 900 ์ˆ˜์ค€์— ๋จธ๋ฌด๋Š” ๋™์•ˆ FlashSAC์€ 5,000์„ ๋„˜๋Š”๋‹ค. ๋‹ค๋งŒ Shadow์—์„œ๋Š” FastTD3์™€์˜ ์ฐจ์ด๊ฐ€ 2% ๋‚จ์ง“์ด๊ณ , Allegro ๊ณก์„ ์€ 50M step ๋์—์„œ๋„ ์•„์ง ์˜ค๋ฅด๊ณ  ์žˆ๋‹ค(Fig. 3, ์ •๊ทœํ™” ๊ธฐ์ค€ 6,000์˜ ์•ฝ 87%).

์ €์ฐจ์› โ€œ์•ฝ๊ฐ„ ๋‚ซ๋‹คโ€์—๋Š” ๋ฐ˜๋ก€๊ฐ€ ๋ถ€๋ก์— ์žˆ๋‹ค. ManiSkill PokeCube์—์„œ PPO๋Š” ์„ฑ๊ณต๋ฅ  ์•ฝ 0.85, FlashSAC์€ ์•ฝ 0.20์ด๋‹ค(๋ถ€๋ก Fig. 17). FlashSAC์˜ seed ๊ฐ„ ํ‘œ์ค€ํŽธ์ฐจ๊ฐ€ 0.44์ธ ๊ฒƒ์€ 5๊ฐœ seed ์ค‘ ํ•˜๋‚˜๋งŒ ํ’€๊ณ  ๋‚˜๋จธ์ง€๋Š” ์‹คํŒจํ•œ ์ด๋ด‰ ๋ถ„ํฌ๋ผ๋Š” ๋œป์ด๋‹ค. ยง5.1์˜ โ€œconverging across all tasks where FastTD3 frequently failsโ€ ๋ฌธ์žฅ๋„ ์ด ๊ณผ์ œ์—์„œ๋Š” ์„ฑ๋ฆฝํ•˜์ง€ ์•Š๋Š”๋‹ค(FastTD3์™€ ๋˜‘๊ฐ™์ด 0.20). RollBallยทPickSingleYCB์—์„œ๋„ PPO์™€ ๋น„์Šทํ•˜๋‹ค.


ManiSkill 6๊ณผ์ œ ์„ฑ๊ณต๋ฅ (๋ถ€๋ก Fig. 17, x์ถ• env step). PokeCube์—์„œ PPO(์ฃผํ™ฉ)๊ฐ€ ์•ฝ 85~90%๋กœ FlashSAC(์ดˆ๋ก)์˜ ์•ฝ 20%๋ฅผ ํฌ๊ฒŒ ์•ž์„ ๋‹ค. ์ด ๊ทธ๋ฆผ์˜ PPO๋Š” 50M step๊นŒ์ง€๋งŒ ๊ทธ๋ ค์ ธ ์žˆ๋‹ค.

๋ช‡ ๊ฐ€์ง€ ์„ธ๋ถ€๋Š” ๋…ผ๋ฌธ ์„œ์ˆ ๊ณผ ๊ณต๊ฐœ ๋ฐ์ดํ„ฐ๊ฐ€ ๋‹ค๋ฅด๋‹ค.

  • PPO์˜ step ์˜ˆ์‚ฐ. ๋ณธ๋ฌธ์€ โ€œPPO๋Š” 200M stepโ€์ด๋ผ ํ•˜๊ณ  ๋ถ€๋ก ๊ทธ๋ฆผ ์บก์…˜๋„ โ€œAll methods 50M except PPO (200M)โ€์ด๋‹ค. ๊ณต๊ฐœ CSV์—์„œ 200M๊นŒ์ง€ ๊ฐ„ ๊ฒƒ์€ MuJoCo Playground 4๊ณผ์ œ๋ฟ์ด๊ณ , IsaacLabยทManiSkillยทGenesis์˜ PPO๋Š” 100M step์—์„œ ๋๋‚œ๋‹ค.
  • seed ์ˆ˜. ๋…ผ๋ฌธ์€ โ€œrandom seeds์˜ ํ‰๊ท , 95% bootstrap CIโ€๋ผ๊ณ ๋งŒ ํ•˜๊ณ  ๊ฐœ์ˆ˜๋ฅผ ๋ฐํžˆ์ง€ ์•Š๋Š”๋‹ค. CSV๋กœ๋Š” FlashSACยทFastTD3๊ฐ€ 5๊ฐœ, ManiSkill PPO๋Š” ๊ณผ์ œ๋งˆ๋‹ค 2~6๊ฐœ๋‹ค.
  • ๋ณด๊ณ ๋˜์ง€ ์•Š์€ ๊ณผ์ œ. ManiSkill CSV์—๋Š” ๋…ผ๋ฌธ ํ‘œ์— ์—†๋Š” UnitreeG1PlaceAppleInBowlยทUnitreeG1TransportBox๊ฐ€ ์žˆ๊ณ , TransportBox ์„ฑ๊ณต๋ฅ ์€ FlashSAC 0.20, PPO 1.00(3 seed)์ด๋‹ค. ํƒ์ƒ‰์šฉ ์‹คํ—˜์ด์—ˆ์„ ์ˆ˜๋„ ์žˆ์œผ๋‹ˆ ์˜๋„๋ฅผ ๋‹จ์ •ํ•  ์ˆ˜๋Š” ์—†์ง€๋งŒ, ๋…ผ๋ฌธ ๋ฐ– ์›์ž๋ฃŒ์— PPO๊ฐ€ ํฌ๊ฒŒ ์ด๊ธด ํœด๋จธ๋…ธ์ด๋“œ ์กฐ์ž‘ ๊ณผ์ œ๊ฐ€ ํ•˜๋‚˜ ๋” ์žˆ๋‹ค๋Š” ์‚ฌ์‹ค์€ ์ ์–ด ๋‘˜ ๋งŒํ•˜๋‹ค.

IsaacLab 12๊ณผ์ œ ์ „์ฒด(๋ถ€๋ก Fig. 12, x์ถ• compute time). ์ ์„ ์€ ์ •๊ทœํ™” ๊ธฐ์ค€์ ์ธ๋ฐ, ์ด ๊ธฐ์ค€์  ์ž์ฒด๊ฐ€ FlashSAC์„ ์˜ค๋ž˜ ๋Œ๋ ค ์–ป์€ ๊ทผ-์ ๊ทผ ์„ฑ๋Šฅ์ด๋‹ค(๋ถ€๋ก B.1).

์‹คํ—˜: ์ƒํƒœ ๊ธฐ๋ฐ˜, CPU ์‹œ๋ฎฌ๋ ˆ์ดํ„ฐ (ยง5.2)


CPU ์‹œ๋ฎฌ๋ ˆ์ดํ„ฐ ๊ฒฐ๊ณผ(Fig. 4). MuJoCo Humanoid-v4, DMC Humanoid WalkยทDog Run, HumanoidBench H1 Run, MyoSuite Key Turn Hard. ์ ์„ ์€ ์ •๊ทœํ™” ๊ธฐ์ค€(Humanoid-v4๋Š” TD7 5M step, HBench๋Š” ์„ฑ๊ณต ๋ฌธํ„ฑ).

๋ณ‘๋ ฌ ํ™˜๊ฒฝ์ด ์—†๋Š” ๊ณ ์ „ ์„ค์ •์—์„œ๋„ FlashSAC์€ batch๋ฅผ 512๋กœ, UTD๋ฅผ 1๋กœ๋งŒ ๋ฐ”๊พธ๊ณ  ๊ฐ™์€ ๊ตฌ์กฐ๋ฅผ ์“ด๋‹ค. ๋Œ€ํ‘œ ๊ณผ์ œ์—์„œ๋Š” ๊ฑฐ์˜ ๋ชจ๋“  baseline์„ ์•ž์„œ๊ณ , PPO๋Š” ์ด ๋‚ฎ์€ ์ƒ˜ํ”Œ ์˜ˆ์‚ฐ์—์„œ ํฌ๊ฒŒ ์ง„๋‹ค.

๊ณต๊ฐœ CSV์˜ 1M step ์ตœ์ข…๊ฐ’์œผ๋กœ 39๊ณผ์ œ๋ฅผ ์ „๋ถ€ ๋Œ€์กฐํ•˜๋ฉด ์ด ์ฃผ์žฅ์€ ๋Œ€์ฒด๋กœ ๋งž๋‹ค.

๋น„๊ต FlashSAC์ด ๊ฐ™๊ฑฐ๋‚˜ ๋†’์€ ๊ณผ์ œ FlashSAC์ด ๋‚ฎ์€ ๋Œ€ํ‘œ ๊ณผ์ œ
vs XQC(10 seed) 25 / 29 H1 Hurdle(107 vs 261), DMC cartpole balance_sparse, H1 Reach
vs SimbaV2(10 seed) 35 / 39 DMC cartpole swingup_sparse(482 vs 849), MyoSuite pen-twirl(1,450 vs 2,360)
vs MR.Q(10 seed) 13 / 15 cartpole swingup_sparse(482 vs 798)
vs TD-MPC2(1~3 seed) 25 / 29 H1 Hurdle(107 vs 387), cartpole swingup_sparse(482 vs 842)

(ํ•„์ž ๊ณ„์‚ฐ. ๊ฐ ๋ฐฉ๋ฒ• ๊ฐ ๊ณผ์ œ์˜ ๋งˆ์ง€๋ง‰ ํ‰๊ฐ€๊ฐ’ seed ํ‰๊ท  ๋น„๊ต์ด๋ฉฐ ํ†ต๊ณ„ ๊ฒ€์ •์€ ์•„๋‹ˆ๋‹ค. XQCยทMR.Q๋Š” ์ผ๋ถ€ ๋ฒค์น˜๋งˆํฌ ๊ฒฐ๊ณผ๊ฐ€ ์—†๋‹ค.)

ํฅ๋ฏธ๋กœ์šด ๊ฒƒ์€ ์•ฝ์ ์˜ ์œ„์น˜๋‹ค. DMC cartpole swingup_sparse๋Š” ๋ถ€๋ก B.6์ด โ€œํƒ์ƒ‰์„ ํ‰๊ฐ€ํ•˜๋ ค๊ณ โ€ ๋„ฃ์—ˆ๋‹ค๊ณ  ๋ฐํžŒ sparse ๋ณด์ƒ ๊ณผ์ œ์ธ๋ฐ, ์—ฌ๊ธฐ์„œ FlashSAC(482)์€ SimbaV2ยทMR.QยทTD-MPC2(์•ฝ 800~850)์— ํฌ๊ฒŒ ๋’ค์ง„๋‹ค. noise repetition์„ ๋‚ด์„ธ์šด ๋ฐฉ๋ฒ•์ด ์ •์ž‘ sparse ํƒ์ƒ‰ ๊ณผ์ œ์—์„œ ์•ฝํ•˜๋‹ค๋Š” ์ ์€ ยง6.4์˜ ํƒ์ƒ‰ ๋ถ„์„(์กฐ๋ฐ€ํ•œ ๋ณด์ƒ์˜ IsaacLab 4๊ณผ์ œ์—์„œ๋งŒ ์ธก์ •)์ด ๋‹ค๋ฃจ์ง€ ์•Š๋Š” ๋ถ€๋ถ„์ด๋‹ค.

๋˜ Fig. 4์˜ DMC Dog Run์„ ๋ณด๋ฉด XQC(๋…ธ๋ž‘)๊ฐ€ ๊ฐ™์€ ์‹œ๊ฐ„๋Œ€์—์„œ๋Š” FlashSAC๋ณด๋‹ค ์œ„์— ์žˆ๋‹ค๊ฐ€ 1M step ๋(์•ฝ 40๋ถ„)์—์„œ ๋ฉˆ์ถ”๊ณ , FlashSAC์€ ์•ฝ 55๋ถ„๊นŒ์ง€ ๊ฐ€์„œ ๋” ๋†’์ด ๋๋‚œ๋‹ค. โ€œcompute ํšจ์œจ๊ณผ ์ ๊ทผ ์„ฑ๋Šฅ ๋ชจ๋‘์—์„œ ์•ž์„ ๋‹คโ€๋Š” ์บก์…˜์€ ์ด ๊ทธ๋ฆผ์—์„œ๋Š” ์ ๊ทผ ์ชฝ์—๋งŒ ํ•ด๋‹นํ•œ๋‹ค.

๋ถ€๋ก ํ‘œ์˜ ๊ณผ์ œ ์ˆ˜ ํ•ฉ(5 + 10 + 14 + 10)์€ 39์ด๊ณ  CSV๋„ 39๊ณผ์ œ์ธ๋ฐ, ๋ณธ๋ฌธ์€ โ€œ40 tasksโ€๋ผ๊ณ  ์“ด๋‹ค.

์‹คํ—˜: ๋น„์ „ ๊ธฐ๋ฐ˜ (ยง5.3)


๋น„์ „ DMC ๊ฒฐ๊ณผ(Fig. 5). Acrobot Swingup, Finger Turn Hard, Hopper Hop, Cheetah Run, Walker Run์—์„œ FlashSACยทDrQ-v2ยทMR.Q. 1M step, action repeat 2.

๋น„์ „์€ ๋ Œ๋”๋ง ๋น„์šฉ ๋•Œ๋ฌธ์— ์ฒ˜๋ฆฌ๋Ÿ‰์ด ๋‚ฎ์•„ off-policy๋ผ๋ฆฌ๋งŒ ๋น„๊ตํ•œ๋‹ค. ์ธ์ฝ”๋”๋Š” DrQ-v2 ํ‘œ์ค€(conv 3์ธต + linear bottleneck), 3ํ”„๋ ˆ์ž„ stack(84ร—84ร—9), 3-step return์„ ์“ด๋‹ค. ๋ณธ๋ฌธ์€ โ€œ๋ชจ๋“  baseline์— ๋Œ€ํ•ด ์ ๊ทผ ์„ฑ๋Šฅ์ด ๊ฐ™๊ฑฐ๋‚˜ ๋†’๊ณ  wall-clock ์ˆ˜๋ ด์ด ๋น ๋ฅด๋‹คโ€๊ณ  ํ•˜๊ณ , DrQ-v2๋Š” Finger Turn Hard ๋“ฑ์—์„œ ์ˆ˜๋ ด์— ์‹คํŒจํ•˜๋ฉฐ MR.Q๋Š” ๋ณด์กฐ dynamics ๋ชจ๋ธ ๋•Œ๋ฌธ์— ์—ฐ์‚ฐ์ด ๋” ๋“ ๋‹ค๊ณ  ์„ค๋ช…ํ•œ๋‹ค.

โ€œ์ ๊ทผ ์„ฑ๋Šฅ์ด ๊ฐ™๊ฑฐ๋‚˜ ๋†’๋‹คโ€๋Š” ๊ฐ™์€ step ๊ธฐ์ค€์œผ๋กœ๋Š” ์ง€์ง€๋˜์ง€ ์•Š๋Š”๋‹ค. ๋ถ€๋ก Fig. 28์—๋Š” ๋ณธ๋ฌธ์— ์—†๋Š” XQCยทDrM๊นŒ์ง€ ๋‹ค์„ฏ ๋ฐฉ๋ฒ•์ด ๊ทธ๋ ค์ ธ ์žˆ๊ณ , 1M step ์ตœ์ข…๊ฐ’์€ ๋‹ค์Œ๊ณผ ๊ฐ™๋‹ค(๊ณต๊ฐœ CSV, ํ•„์ž ๊ณ„์‚ฐ).


๋น„์ „ DMC 8๊ณผ์ œ ์ „์ฒด(๋ถ€๋ก Fig. 28, x์ถ• compute time). ๋ณธ๋ฌธ Fig. 5์— ์—†๋Š” XQC(๋…ธ๋ž‘)ยทDrM(ํŒŒ๋ž‘)์ด ํ•จ๊ป˜ ๊ทธ๋ ค์ ธ ์žˆ๋‹ค. AcrobotยทWalker RunยทCheetah Run์—์„œ ์ผ๋ถ€ baseline์ด FlashSAC๋ณด๋‹ค ๋†’๊ฒŒ ๋๋‚œ๋‹ค.
๊ณผ์ œ FlashSAC(5) MR.Q(10) XQC(10) DrM(3) DrQ-v2(10)
acrobot-swingup 316 287 382 362 173
cartpole-swingup-sparse 798 797 729 795 785
cheetah-run 762 776 724 839 735
finger-turn-easy 931 954 666 959 645
finger-turn-hard 904 933 692 737 315
hopper-hop 184 270 249 258 225
reacher-hard 969 966 937 961 664
walker-run 553 616 684 652 543

(๊ด„ํ˜ธ๋Š” seed ์ˆ˜. DrQ-v2ยทDrM์€ 1.02M step ๊ทผ์ฒ˜์˜ ๋งˆ์ง€๋ง‰ ๊ฐ’. ๊ตต์€ ๊ธ€์”จ๋Š” ํ–‰ ์ตœ๋Œ“๊ฐ’.)

๊ฐ™์€ 1M step์—์„œ MR.Q๋Š” 8๊ณผ์ œ ์ค‘ 5๊ณผ์ œ์—์„œ FlashSAC๋ณด๋‹ค ๋†’๋‹ค. ์ฐจ์ด๊ฐ€ ์ž‘์€ ๊ณผ์ œ(cheetah 762 vs 776)๋„ ๋งŽ์•„ โ€œFlashSAC์ด ์ง„๋‹คโ€๊ณ  ๋งํ•  ์ˆ˜์ค€์€ ์•„๋‹ˆ์ง€๋งŒ, ์ •ํ™•ํ•œ ์š”์•ฝ์€ โ€œ์ ์ˆ˜๋Š” ๊ฒฝ์Ÿ๋ ฅ ์žˆ๋Š” ์ˆ˜์ค€์ด๊ณ , ๊ทธ ์ ์ˆ˜์— ๋” ์งง์€ ์ถ”์ • ์‹œ๊ฐ„ ์•ˆ์— ๋„๋‹ฌํ•œ๋‹คโ€์ด๋‹ค. ๋ณธ๋ฌธ Fig. 5๊ฐ€ XQCยทDrM์„ ๋นผ๊ณ  ๋‹ค์„ฏ ๊ณผ์ œ๋งŒ ๊ณ ๋ฅธ ๊ฒƒ๋„ ์ด ์ธ์ƒ์„ ํ‚ค์šด๋‹ค. ๋˜ ๋น„์ „ ์ธ์ฝ”๋” ์ฝ”๋“œ๋Š” ๊ณต๊ฐœ ๋ ˆํฌ์— ์—†์–ด์„œ ์ด ์ ˆ์€ CSV ๋ง๊ณ ๋Š” ํ™•์ธํ•  ๊ธธ์ด ์—†๋‹ค.

์‹คํ—˜: Sim-to-real (ยง5.4, ๋ถ€๋ก D)

G1(29 DoF) blind ๋ณดํ–‰์„ ํ”ผ๋ผ๋ฏธ๋“œ ๊ณ„๋‹จยท๊ฒฉ์žยทํŒŒ๋„ยท๊ตฌ๋ฉ์ด ์ง€ํ˜• ์ปค๋ฆฌํ˜๋Ÿผ(10๋‹จ๊ณ„, ๊ณ„๋‹จ ๋†’์ด 0~23 cm)๊ณผ domain randomization์œผ๋กœ ํ•™์Šตํ•œ๋‹ค. baseline์€ DreamWaQ ํŒŒ์ดํ”„๋ผ์ธ์˜ PPO์ด๊ณ , FlashSAC๋„ ๊ฐ™์€ sim-to-real ์žฅ์น˜๋ฅผ ์“ด๋‹ค. ๊ด€์ธก ์ด๋ ฅ์œผ๋กœ ๊ธฐ์ € ์†๋„์™€ ์ž ์žฌ๋ฅผ ์ถ”์ •ํ•˜๋Š” context estimator(CENet)๋ฅผ actor์— ๋„ฃ๊ณ , critic์€ ์‹ค์ œ ๊ธฐ์ € ์„ ์†๋„ยท๋ฐœ ์ ‘์ด‰ยท๋†’์ด ์ง€๋„๋ฅผ ๋ฐ›๋Š” asymmetric actor-critic์ด๋‹ค. ์ด ํŒŒ์ดํ”„๋ผ์ธ์€ ์ด ๋ธ”๋กœ๊ทธ์˜ DreamWaQ ๋ฆฌ๋ทฐ์—์„œ ๋‹ค๋ฃฌ ๊ฒƒ์ด๊ณ , DreamWaQ์˜ ์ œ1์ €์ž Nahrendra๊ฐ€ ์ด ๋…ผ๋ฌธ์˜ ๊ณต์ €์ž๋‹ค.


ํ•™์Šต ์ง€ํ˜•(๋ถ€๋ก Fig. 11). ํ”ผ๋ผ๋ฏธ๋“œ ๊ณ„๋‹จ(์ตœ๋Œ€ 23 cm), ๋ฌด์ž‘์œ„ ๊ฒฉ์ž(15 cm), ๋ฌด์ž‘์œ„ ๊ท ์ผ ์ง€ํ˜•(โˆ’2~4 cm), ํŒŒ๋„(์ง„ํญ 20 cm), ๊ตฌ๋ฉ์ด(30 cm). 50% ์ด์ƒ ํ†ต๊ณผํ•˜๋ฉด ๋‹จ๊ณ„๋ฅผ ์˜ฌ๋ฆฐ๋‹ค.

G1 ์‹ค๊ธฐ ๊ณ„๋‹จ ์˜ค๋ฅด๊ธฐ์™€ ์‹œ๋ฎฌ๋ ˆ์ด์…˜ terrain level ๊ณก์„ (Fig. 6). FlashSAC์€ ์•ฝ 4์‹œ๊ฐ„์— terrain level ์•ฝ 3.9, PPO๋Š” ์•ฝ 4์‹œ๊ฐ„๊นŒ์ง€ 0์— ๋จธ๋ฌผ๋‹ค ์•ฝ 20์‹œ๊ฐ„์— ๊ฐ™์€ ์ˆ˜์ค€์— ์ด๋ฅธ๋‹ค.

๊ฒฐ๊ณผ๋Š” ํ‰์ง€ ์•ฝ 20๋ถ„ vs 3์‹œ๊ฐ„, 15 cm ๊ณ„๋‹จ(๋†’์ด 15 cm, ๋„ˆ๋น„ 60 cm, ํ”Œ๋žซํผ 1.5 m, ํ•™์Šต๊ณผ ๋‹ค๋ฅธ ์น˜์ˆ˜) ์•ฝ 4์‹œ๊ฐ„ vs 20์‹œ๊ฐ„์ด๋‹ค. ์ •์ฑ…์€ ์žฌํ•™์Šต ์—†์ด ์ „ยทํ›„ยท์˜† ๋ฐฉํ–ฅ ๋ณดํ–‰์„ ์ง€์›ํ•œ๋‹ค.

์ด ์ ˆ์€ ๋…ผ๋ฌธ์˜ ๊ฐ€์žฅ ๊ฐ•ํ•œ ๋ฉ”์‹œ์ง€(โ€œhours to minutesโ€)๋ฅผ ๋‹ด๊ณ  ์žˆ์ง€๋งŒ, ๊ทผ๊ฑฐ์˜ ์ข…๋ฅ˜๋ฅผ ๋‚˜๋ˆ  ๋ณด๋ฉด ๋น„๊ต์˜ ๊ณต์ •์„ฑ์— ๊ฑธ๋ฆฌ๋Š” ์ง€์ ์ด ๋งŽ๋‹ค.

โ‘  ๋ณด์ƒ์ด ๊ฐ™์ง€ ์•Š๋‹ค. ๋ณธ๋ฌธ ยง5.4๋Š” โ€œBoth methods share identical reward design and coefficientsโ€๋ผ๊ณ  ์“ด๋‹ค. ๊ทธ๋Ÿฐ๋ฐ ๋ถ€๋ก Table 14์™€ ยงD.4๋Š” โ€œํ•™์Šต ๋™์—ญํ•™์ด ๋‹ฌ๋ผ ๋‹ค๋ฅธ ๋ณด์ƒ ๊ฐ€์ค‘์น˜๊ฐ€ ํ•„์š”ํ–ˆ๋‹คโ€๊ณ  ํ•˜๊ณ , ์‹ค์ œ ๊ฐ€์ค‘์น˜๋Š” ํฌ๊ฒŒ ๋‹ค๋ฅด๋‹ค.

๋ณด์ƒ ํ•ญ(๋ถ€๋ก Table 14 ์ผ๋ถ€) FlashSAC PPO
์„ ์†๋„ ์ถ”์ข… 2.0 (ฯƒ=0.25) 1.5 (ฯƒ=0.5)
Feet air time 1.0 0.25
Action rate โˆ’0.5 โˆ’0.01
Body orientation โˆ’52.0 โˆ’2.0
Stand still โˆ’5.0 โˆ’0.25
Undesired contacts โˆ’5.0 โˆ’1.0
Termination penalty ์—†์Œ โˆ’200
Alive bonus 1.0 ์—†์Œ

๋ถ€๋ก์€ PPO๊ฐ€ termination ๊ธฐ๋ฐ˜ shaping ์—†์ด๋Š” ์•ˆ์ • ๋ณดํ–‰๊นŒ์ง€ ์˜ค๋ž˜ ๊ฑธ๋ฆฐ๋‹ค๊ณ  ์„ค๋ช…ํ•œ๋‹ค. ์ด์œ ๊ฐ€ ์žˆ๋Š” ์กฐ์ •์ด์ง€๋งŒ, ๊ฒฐ๊ณผ์ ์œผ๋กœ Fig. 1cยทFig. 6์˜ ์‹œ๊ฐ„ ๋น„๊ต๋Š” ์•Œ๊ณ ๋ฆฌ์ฆ˜ + ๋ณด์ƒ ์„ค๊ณ„์˜ ์ฐจ์ด๋‹ค. ๋ณธ๋ฌธ ๋ฌธ์žฅ์€ ๋ถ€๋ก๊ณผ ์ •๋ฉด์œผ๋กœ ์–ด๊ธ‹๋‚œ๋‹ค.

โ‘ก ํ•˜๋“œ์›จ์–ด์™€ ์„ค์ •์ด ๋ณธ๋ฌธ์˜ ๋‹ค๋ฅธ ์ ˆ๊ณผ ๋‹ค๋ฅด๋‹ค. ๋ณธ๋ฌธ์€ wall-clock์„ RTX 5090 ๊ธฐ์ค€์œผ๋กœ ์Ÿ€๋‹ค๊ณ  ํ•˜์ง€๋งŒ, ๋ถ€๋ก D.1์€ sim-to-real ํ•™์Šต์ด 4,096 ๋ณ‘๋ ฌ ํ™˜๊ฒฝ, A100 ํ•œ ์žฅ์—์„œ ์•ฝ 4์‹œ๊ฐ„์ด๋ผ๊ณ  ํ•œ๋‹ค. PPO๊ฐ€ ์–ด๋–ค GPU์—์„œ 20์‹œ๊ฐ„ ๊ฑธ๋ ธ๋Š”์ง€๋Š” ์ ํ˜€ ์žˆ์ง€ ์•Š๋‹ค. ๋˜ ยง5.4๋Š” FlashSAC์ด โ€œstate-based experiments(ยง5.2)์™€ ๊ฐ™์€ ๊ตฌ์กฐยทํ•˜์ดํผํŒŒ๋ผ๋ฏธํ„ฐโ€๋ฅผ ์ผ๋‹ค๊ณ  ์ฐธ์กฐํ•˜๋Š”๋ฐ, ยง5.2๋Š” batch 512ยทUTD 1์˜ CPU ์„ค์ •์ด๋‹ค. GPU ์„ค์ •(ยง5.1)์˜ ์˜ค๊ธฐ๋กœ ๋ณด์ด์ง€๋งŒ ํ™•์ธํ•  ์ˆ˜ ์—†๋‹ค.

โ‘ข ์‹ค๊ธฐ ํ‰๊ฐ€๊ฐ€ ์ •์„ฑ์ ์ด๋‹ค. ์‹ค๊ธฐ ๊ฒฐ๊ณผ๋Š” ์˜์ƒ ํ”„๋ ˆ์ž„๊ณผ ์‹œ๊ฐ„ ์ง„์ˆ ๋ฟ์ด๊ณ , ์‹œ๋„ ํšŸ์ˆ˜ยท์„ฑ๊ณต๋ฅ ยท์†๋„ ์ถ”์ข… ์˜ค์ฐจ ๊ฐ™์€ ์‹ค๊ธฐ ์ง€ํ‘œ๊ฐ€ ์—†๋‹ค. โ€œcomparable performanceโ€์˜ ๊ธฐ์ค€๋„ ์ •์˜๋˜์ง€ ์•Š์•˜๋‹ค. Fig. 1cยทFig. 6์˜ ๊ณก์„ ์€ ์‹œ๋ฎฌ๋ ˆ์ด์…˜ ์ง€ํ‘œ์ด๊ณ , ๊ณต๊ฐœ CSV๋กœ๋Š” ๋‘ ๋ฐฉ๋ฒ• ๋ชจ๋‘ seed 1๊ฐœ(42)๋‹ค. ๋งˆ์ง€๋ง‰ ๊ฐ’์€ ํ‰์ง€ ์ถ”์ข… 0.876(FlashSAC) vs 0.947(PPO), terrain level 3.87 vs 4.04๋กœ PPO๊ฐ€ ์กฐ๊ธˆ ๋†’๊ฒŒ ๋๋‚œ๋‹ค.

โ‘ฃ terrain level ์•ฝ 3.9๋Š” ์ƒ๊ฐ๋ณด๋‹ค ๋‚ฎ์€ ๋‹จ๊ณ„๋‹ค. ์ปค๋ฆฌํ˜๋Ÿผ ์ตœ๋Œ€๋Š” 10์ด๊ณ , ๋ถ€๋ก์€ โ€œ์ธ์ง€ ๋ชจ๋“ˆ ์—†์ด 5๋‹จ๊ณ„๋ฅผ ๋„˜๊ธฐ๋Š” ์–ด๋ ต๋‹คโ€๊ณ  ์ ๋Š”๋‹ค. Legged Gym์‹์œผ๋กœ ๋†’์ด๊ฐ€ ๋‹จ๊ณ„์— ์„ ํ˜• ๋น„๋ก€ํ•œ๋‹ค๊ณ  ๊ฐ€์ •ํ•˜๋ฉด ํ‰๊ท  3.9๋‹จ๊ณ„๋Š” ์•ฝ 9 cm ๊ณ„๋‹จ์— ํ•ด๋‹นํ•œ๋‹ค(ํ•„์ž ์ถ”์ •, ๋…ผ๋ฌธ์€ ๋‹จ๊ณ„โ†”๏ธŽ๋†’์ด ๋Œ€์‘์„ ๋ฐํžˆ์ง€ ์•Š๋Š”๋‹ค). ๊ทธ๋Ÿฐ๋ฐ ์‹ค๊ธฐ ๊ณ„๋‹จ์€ 15 cm๋‹ค. ํ™˜๊ฒฝ๋งˆ๋‹ค ๋‹จ๊ณ„๊ฐ€ ๋‹ค๋ฅด๋‹ˆ ์ผ๋ถ€ ํ™˜๊ฒฝ์€ ๋” ๋†’์€ ๋‹จ๊ณ„์—์„œ ํ•™์Šตํ–ˆ์„ ๊ฒƒ์ด๊ณ , ํ•™์Šต ๋ถ„ํฌ๋ฅผ ๋„˜์–ด์„  ์ผ๋ฐ˜ํ™”์ผ ์ˆ˜๋„ ์žˆ๋‹ค. ์–ด๋А ์ชฝ์ด๋“  ๋…ผ๋ฌธ์€ ์ด ๊ฐ„๊ทน์„ ์„ค๋ช…ํ•˜์ง€ ์•Š๋Š”๋‹ค.

โ‘ค domain randomization ๋ฒ”์œ„๊ฐ€ ์—†๋‹ค. ยง5.4๋Š” โ€œfull details are provided in ยงDโ€๋ผ๊ณ  ํ•˜์ง€๋งŒ ๋ถ€๋ก์—๋Š” ์ง€ํ˜•ยท๊ด€์ธกยทPD gainยท๋ณด์ƒ ํ‘œ๋งŒ ์žˆ๊ณ  randomization ๋ฒ”์œ„ ํ‘œ๋Š” ์—†๋‹ค.

๋ถ„์„ (ยง6)

๋ถ„์„์€ ์ „๋ถ€ IsaacLab 4๊ณผ์ œ(AllegroยทShadow ํ๋ธŒ ์žฌ๋ฐฐ์น˜, G1 ํ‰์ง€ยทํ—˜์ง€)์—์„œ ํ–ˆ๋‹ค.

Scaling ablation (ยง6.2, Fig. 8)


Scaling ๊ฒฐ๊ณผ(Fig. 8, x์ถ• compute time). (a) replay buffer, (b) batch, (c) critic ํญ(actor๋Š” ๋น„๋ก€ ์ถ•์†Œ), (d) ๋ธ”๋ก ์ˆ˜, (e) UTD. ๊ตต์€ ๋ผ๋ฒจ์ด ๊ธฐ๋ณธ๊ฐ’.

๋ณธ๋ฌธ ๊ฒฐ๋ก ์€ โ€œbuffer๋Š” 10M๊นŒ์ง€ ์ข‹๊ณ  50M์€ ๋А๋ฆฌ๋‹คโ€, โ€œbatchยท๋ชจ๋ธ์„ ํ‚ค์šฐ๊ณ  UTD๋ฅผ ์ค„์ด๋ฉด ์ˆ˜๋ ด์ด ๋นจ๋ผ์ง„๋‹ค(scaling ๋ฒ•์น™๊ณผ ์ผ์น˜)โ€์ด๋‹ค.

๊ทธ๋ฆผ์„ ์ถ•๋ณ„๋กœ ๋‹ค์‹œ ์ฝ์œผ๋ฉด ๊ฒฐ๋ก ์ด ์กฐ๊ธˆ ๋‹ค๋ฅด๋‹ค. wall-clock ๊ธฐ์ค€์œผ๋กœ batch 8K, ํญ 1024, UTD 8/1024๋Š” ๊ธฐ๋ณธ๊ฐ’๋ณด๋‹ค ๋А๋ฆฌ๊ณ , batch 0.5K, ํญ 64, UTD 0.5/1024๋Š” ๋‚˜์˜๋‹ค. ์ฆ‰ ๋‹จ์กฐ ๊ฒฝํ–ฅ์ด ์•„๋‹ˆ๋ผ ๊ธฐ๋ณธ๊ฐ’ ๊ทผ์ฒ˜์— sweet spot์ด ์žˆ๋Š” ์—ญ U์ž๋‹ค. โ€œํ‚ค์šธ์ˆ˜๋ก ๋นจ๋ผ์ง„๋‹คโ€๋Š” ๋ฒ”์œ„๋ฅผ ํ•œ์ •ํ•ด์•ผ ๋งž๋Š” ๋ฌธ์žฅ์ด๋‹ค.

๊ณต๊ฐœ CSV๋กœ 50M step ๋์˜ ์ ์ˆ˜(4๊ณผ์ œ์˜ ์ •๊ทœํ™” return ํ‰๊ท , ํ•„์ž ๊ณ„์‚ฐ)๋ฅผ ๋ณด๋ฉด ๋˜ ๋‹ค๋ฅธ ๊ทธ๋ฆผ์ด ๋‚˜์˜จ๋‹ค.

์ถ• ๊ธฐ๋ณธ๊ฐ’ ๊ธฐ๋ณธ๊ฐ’ ์ ์ˆ˜ ๋” ํฐ ์„ค์ •์˜ ์ ์ˆ˜
critic ํญ 256 0.90 512: 1.05, 1024: 1.08
๋ธ”๋ก ์ˆ˜ 2 0.90 3: 1.01, 4: 1.04
batch 2K 0.90 4K: 1.01, 8K: 1.00
UTD 2/1024 0.90 4/1024: 1.03, 8/1024: 0.99
replay 10M 0.90 1M: 0.88, 50M: 0.80, 0.1M: 0.52

(๊ฐ ๊ณผ์ œ์˜ ๋งˆ์ง€๋ง‰ ํ‰๊ฐ€ return์„ ๋ถ€๋ก Table 1์˜ normalize score๋กœ ๋‚˜๋ˆˆ 4๊ณผ์ œ ํ‰๊ท . ๋…ผ๋ฌธ ๊ทธ๋ฆผ์˜ ์ง‘๊ณ„ ๋ฐฉ์‹์ด ๋ช…์‹œ๋˜์ง€ ์•Š์•„ ์ ˆ๋Œ€๊ฐ’์€ ๊ทธ๋ฆผ๊ณผ ์กฐ๊ธˆ ๋‹ค๋ฅด๋‹ค.)

๊ฐ™์€ 50M step์ด๋ฉด ๋” ํฐ ์„ค์ •์ด ์ผ๊ด€๋˜๊ฒŒ ๋” ๋†’์€ ์ ์ˆ˜๋กœ ๋๋‚œ๋‹ค. ๊ธฐ๋ณธ๊ฐ’์€ ์ ๊ทผ ์„ฑ๋Šฅ์ด ์•„๋‹ˆ๋ผ wall-clock ๋Œ€๋น„ ์ ์ˆ˜๋ฅผ ๊ธฐ์ค€์œผ๋กœ ๊ณ ๋ฅธ ์ง€์ ์ด๋‹ค. ์ด๊ฒƒ์€ ๋…ผ๋ฌธ์˜ โ€œfastโ€ ์ฃผ์žฅ๊ณผ ์ผ๊ด€๋˜์ง€๋งŒ, ํ”„๋กœ์ ํŠธ ํŽ˜์ด์ง€์˜ โ€œhighest asymptotic performance in the shortest wall-clock timeโ€์ด๋ผ๋Š” ๋ฌธ๊ตฌ์™€๋Š” ๊ธด์žฅ ๊ด€๊ณ„๋‹ค. ๊ธฐ๋ณธ ์„ค์ •์€ ๊ฐ™์€ ๋ฐ์ดํ„ฐ์—์„œ 0.1~0.18 ์ •๋„์˜ ์ ์ˆ˜๋ฅผ ๋‚จ๊ฒจ ๋‘๊ณ  ์žˆ๋‹ค.

์ด ํ‘œ๋Š” FlashDexRetarget ๋ฆฌ๋ทฐ์˜ ์„ค์ • ์„ ํƒ๊ณผ๋„ ๋งž๋ฌผ๋ฆฐ๋‹ค. ๊ทธ ๋…ผ๋ฌธ์€ FlashSAC ๊ธฐ๋ณธ๊ฐ’(replay 10M, critic ํญ 256)์„ 50Mยท1024๋กœ ํ‚ค์› ๊ณ  300M step์„ ํ•™์Šตํ–ˆ๋‹ค. ์ด ๋…ผ๋ฌธ์˜ Fig. 8a๋„ โ€œ50M buffer๋Š” ์ถฉ๋ถ„ํ•œ ์‹œ๊ฐ„์ด ์ฃผ์–ด์ง€๋ฉด ์•ฝ๊ฐ„ ๋” ๋†’์€ ์ ๊ทผ ์„ฑ๋Šฅ์— ์ด๋ฅผ ์ˆ˜ ์žˆ๋‹คโ€๊ณ  ์“ฐ๋ฏ€๋กœ, ๊ธด ํ•™์Šต์—์„œ ๋” ํฐ bufferยท๋” ๋„“์€ critic์„ ์“ฐ๋Š” ๊ฒƒ์€ ์ €์ž ๊ทธ๋ฃน ์Šค์Šค๋กœ์˜ ํ•ด์„๊ณผ ๊ฐ™์€ ๋ฐฉํ–ฅ์ด๋‹ค. ๋‹ค๋งŒ Fig. 8a ์ž์ฒด๋Š” 50M step์—์„œ ๋๋‚˜์„œ, 50M buffer๊ฐ€ ๊ฒฐ๊ตญ ๋”ฐ๋ผ์žก๋Š”๋‹ค๋Š” ๋ฌธ์žฅ์€ ์ด ๋…ผ๋ฌธ ์•ˆ์—์„œ๋Š” ์ธก์ •๋˜์ง€ ์•Š์•˜๋‹ค(50M step ์˜ˆ์‚ฐ์—์„œ 50M buffer๋Š” ์ „์ด๋ฅผ ํ•˜๋‚˜๋„ ๋ฒ„๋ฆฌ์ง€ ์•Š๋Š”๋‹ค).

Architectural ablation (ยง6.3, Fig. 9)


์•„ํ‚คํ…์ฒ˜ ablation(Fig. 9). MLP์—์„œ ์‹œ์ž‘ํ•ด Residual โ†’ BatchNorm โ†’ RMSNorm โ†’ Distributional critic(+reward scaling) โ†’ Weight norm์„ ๋ˆ„์ ํ•ด ๋”ํ•œ๋‹ค. (a) ์ •๊ทœํ™” ์ ์ˆ˜, (b) weight norm, (c) feature norm, (d) gradient norm, (e) critic condition number(๋กœ๊ทธ ์Šค์ผ€์ผ).

๋ณธ๋ฌธ์€ โ€œ๊ตฌ์„ฑ ์š”์†Œ๋ฅผ ๋”ํ• ์ˆ˜๋ก norm์ด ๋ฌถ์ด๊ณ , condition number๊ฐ€ ๋‹จ์กฐ ๊ฐ์†Œํ•˜๋ฉฐ, ์ด ์•ˆ์ •์„ฑ์ด ์„ฑ๋Šฅ์œผ๋กœ ์ด์–ด์ง„๋‹คโ€๊ณ  ์“ด๋‹ค. ์บก์…˜๋„ โ€œEach addition stabilizes trainingโ€์ด๋‹ค.

๊ทธ๋ฆผ๊ณผ CSV๋ฅผ ๋‹จ๊ณ„๋ณ„๋กœ ๋Œ€์กฐํ•˜๋ฉด ๋‹ค์Œ๊ณผ ๊ฐ™๋‹ค.

๋‹จ๊ณ„ 50M step ์ ์ˆ˜(ํ•„์ž ๊ณ„์‚ฐ) ๊ทธ๋ฆผ์—์„œ ๋ณด์ด๋Š” ๊ฒƒ
MLP 0.28 feature norm์ด ์•ฝ 1.8 โ†’ 6์œผ๋กœ ๊ณ„์† ์ฆ๊ฐ€
+ Residual 0.43 gradient norm์ด ์•ฝ 40M step์—์„œ ์•ฝ 11.7๊นŒ์ง€ ํŠ€๊ณ , condition number๊ฐ€ MLP๋ณด๋‹ค ๋†’๋‹ค
+ BatchNorm 0.37 ์ ์ˆ˜๊ฐ€ ์ด์ „ ๋‹จ๊ณ„๋ณด๋‹ค ๋‚ฎ๋‹ค
+ RMSNorm 0.60 condition number๋Š” ๋‚ด๋ ค๊ฐ€์ง€๋งŒ weight norm์ด ์•ฝ 0.26๊นŒ์ง€ ์ปค์ ธ MLP๋ณด๋‹ค ๋†’๋‹ค
+ Dist. critic 0.90 ๊ฐ€์žฅ ํฐ ๋„์•ฝ(+0.30). condition number๊ฐ€ ์•ฝ 11~12๋Œ€๋กœ ๋‚ด๋ ค๊ฐ„๋‹ค
+ Weight norm(FlashSAC) 0.90 ์ ์ˆ˜ ๋ณ€ํ™” ์—†์Œ. weightยทgradient norm๋งŒ ํ‰ํ‰ํ•ด์ง„๋‹ค

๋”ฐ๋ผ์„œ โ€œmonotonicallyโ€์™€ โ€œeach addition stabilizesโ€๋Š” ์ •ํ™•ํ•˜์ง€ ์•Š๋‹ค. Residual์€ condition number์™€ gradient norm์„ ์˜คํžˆ๋ ค ํ‚ค์šฐ๊ณ , BatchNorm์€ ์ ์ˆ˜๋ฅผ ๊นŽ๊ณ , RMSNorm์€ weight norm์„ ํ‚ค์šด๋‹ค. ์„ฑ๋Šฅ์˜ ๋Œ€๋ถ€๋ถ„์€ distributional critic + reward scaling ํ•œ ๋‹จ๊ณ„์—์„œ ๋‚˜์˜จ๋‹ค. weight normalization์€ ์ €์ž ์Šค์Šค๋กœ โ€œmodest gainsโ€๋ผ๊ณ  ํ•˜๊ณ , ์ƒ˜ํ”Œ์ด ์ œํ•œ๋œ ์ฒด์ œ์—์„œ ๊ฐ•๊ฑด์„ฑ์ด ์ข‹์•„์„œ ๋‚จ๊ฒผ๋‹ค๊ณ  ์„ค๋ช…ํ•˜์ง€๋งŒ ๊ทธ ๊ฐ•๊ฑด์„ฑ ์‹คํ—˜์€ ์ œ์‹œ๋˜์ง€ ์•Š์•˜๋‹ค.

๋˜ ํ•˜๋‚˜ ์งš์„ ์ ์€ ๋ˆ„์  ablation์ด๋ผ๋Š” ๊ตฌ์กฐ๋‹ค. distributional critic์ด RMSNormยทBatchNorm ์—†์ด๋„ ๊ฐ™์€ ๋„์•ฝ์„ ๋งŒ๋“œ๋Š”์ง€, ํ˜น์€ weight norm์„ ๋นผ๋ฉด ๋ฌด์—‡์ด ๊นจ์ง€๋Š”์ง€(์ œ๊ฑฐํ˜• ablation)๋Š” ์•Œ ์ˆ˜ ์—†๋‹ค. ๊ณต๊ฐœ CSV์—๋Š” condition number ์™ธ์— srankยทdormant neuron ๊ฐ™์€ ์ง„๋‹จ ์ง€ํ‘œ๋„ ๋“ค์–ด ์žˆ๋Š”๋ฐ ๋…ผ๋ฌธ์—๋Š” ์“ฐ์ด์ง€ ์•Š์•˜๋‹ค.

Exploration ablation (ยง6.4, Fig. 10)


ํƒ์ƒ‰ ablation(Fig. 10, x์ถ• env step). (a) ฯƒ_tgt โˆˆ {0.05, 0.1, 0.15, 0.2, 0.25}๋Š” ๋น„์Šทํ•˜๊ฒŒ ์ˆ˜๋ ด, (b) noise repetition์„ ๋„๋ฉด ์ˆ˜๋ ด์ด ๋А๋ฆฌ๊ณ  ์ตœ์ข… ์ ์ˆ˜๊ฐ€ ๋‚ฎ๋‹ค.

๋ณธ๋ฌธ์€ ฯƒ_tgt์— โ€œlargely insensitiveโ€ํ•˜๊ณ  ๊ธฐ๋ณธ๊ฐ’ 0.15๊ฐ€ ์ž˜ ์ž‘๋™ํ•œ๋‹ค๊ณ  ์“ฐ๊ณ , ์บก์…˜์€ ์ตœ์ ์ด 0.15~0.2๋ผ๊ณ  ์“ด๋‹ค. CSV๋กœ 50M step ์ ์ˆ˜๋ฅผ ๋ณด๋ฉด 0.05: 0.89, 0.1: 0.92, 0.15: 0.90, 0.2: 0.97, 0.25: 0.96์ด๋‹ค(ํ•„์ž ๊ณ„์‚ฐ). ๋ฏผ๊ฐ๋„๊ฐ€ ์ž‘๋‹ค๋Š” ๊ฒฐ๋ก ์€ ๋งž์ง€๋งŒ, ๊ธฐ๋ณธ๊ฐ’์€ ์ด sweep์—์„œ ์ตœ์ ์ด ์•„๋‹ˆ๋‹ค. noise repetition์€ ๋„๋ฉด 0.84๋กœ ๋–จ์–ด์ง„๋‹ค. ๋‹ค๋งŒ ์ด ๋น„๊ต๋Š” ์กฐ๋ฐ€ํ•œ ๋ณด์ƒ์˜ IsaacLab 4๊ณผ์ œ์—์„œ๋งŒ ํ–ˆ๊ณ , ์•ž์„œ ๋ณธ sparse ํƒ์ƒ‰ ๊ณผ์ œ์—์„œ์˜ ํšจ๊ณผ๋Š” ์ธก์ •๋˜์ง€ ์•Š์•˜๋‹ค.

๋น„ํŒ์ ์œผ๋กœ ๋ณด๋ฉด

๊ฐ•์ 

  • ๋ฌธ์ œ ์„ค์ •์ด ์ •ํ™•ํ•˜๋‹ค. โ€œoff-policy๊ฐ€ ๋А๋ฆฐ ๊ฒƒ์€ update๊ฐ€ ๋งŽ๊ธฐ ๋•Œ๋ฌธโ€์ด๋ผ๋Š” ์ง„๋‹จ๊ณผ โ€œupdate๋ฅผ ์ค„์ด๊ณ  ๋‹ค๋ฅธ ์ถ•์„ ํ‚ค์šด๋‹คโ€๋Š” ์ฒ˜๋ฐฉ์ด ๋ช…๋ฃŒํ•˜๋‹ค. ์ง€๋„ํ•™์Šต์˜ scaling ์ง๊ด€์„ RL์˜ ๋น„์šฉ ๊ตฌ์กฐ์— ๋งž๊ฒŒ ์˜ฎ๊ฒผ๋‹ค.
  • ๋„“์€ ํ‰๊ฐ€ ๋ฒ”์œ„. 10๊ฐœ ์‹œ๋ฎฌ๋ ˆ์ดํ„ฐ, GPUยทCPUยท๋น„์ „ยท์‹ค๊ธฐ๋ฅผ ํ•˜๋‚˜์˜ ์„ค์ •์œผ๋กœ ๋Œ๋ ธ๋‹ค. ๋‹จ์ผ ์„ค์ •์ด๋ผ๋Š” ์‚ฌ์‹ค ์ž์ฒด๊ฐ€ ์‹ค์šฉ์  ๊ฐ€์น˜๋‹ค. ๊ณต๊ฐœ CSV๋กœ ์„ธ์–ด ๋ณด๋ฉด CPU 39๊ณผ์ œ์—์„œ XQCยทSimbaV2๋ฅผ ๋Œ€๋ถ€๋ถ„ ๋„˜๋Š”๋‹ค.
  • ๊ณ ์ฐจ์› ์šฐ์œ„๋Š” ์›์ž๋ฃŒ๋กœ ํ™•์ธ๋œ๋‹ค. AllegroยทShadowยทํœด๋จธ๋…ธ์ด๋“œ์—์„œ PPO์™€์˜ ๊ฒฉ์ฐจ๊ฐ€ ์ˆ˜ ๋ฐฐ์ด๊ณ , FastTD3๋ณด๋‹ค๋„ ์ผ๊ด€๋˜๊ฒŒ ๋†’๋‹ค.
  • ์›์ž๋ฃŒ ๊ณต๊ฐœ. ๊ฑฐ์˜ ๋ชจ๋“  ๊ทธ๋ฆผ์˜ ํ•™์Šต๊ณก์„  CSV(๋Œ€๊ฐœ 5 seed)๊ฐ€ ๋ ˆํฌ์— ์žˆ์–ด ๋…์ž๊ฐ€ ๊ทธ๋ฆผ์„ ๋‹ค์‹œ ๊ทธ๋ฆฌ๊ณ  ์ฃผ์žฅ์„ ์ˆซ์ž๋กœ ๋Œ€์กฐํ•  ์ˆ˜ ์žˆ๋‹ค. ์ด ๋ฆฌ๋ทฐ์˜ ํ‘œ ๋Œ€๋ถ€๋ถ„์ด ๊ทธ ๋•๋ถ„์— ๊ฐ€๋Šฅํ–ˆ๋‹ค. ์ด๋Ÿฐ ๊ณต๊ฐœ๋Š” ๋“œ๋ฌผ๋‹ค.
  • ํ•ด์„ ๊ฐ€๋Šฅํ•œ ์ง„๋‹จ ์ง€ํ‘œ. normยทcondition number๋ฅผ ๊ฐ™์ด ๊ทธ๋ ค โ€œ์™œ ์•ˆ์ •ํ•ด์กŒ๋‚˜โ€๋ฅผ ๋งํ•˜๋ ค ํ•œ ์ ์€ ์„ฑ๋Šฅ ๊ณก์„ ๋งŒ ๋ณด์—ฌ ์ฃผ๋Š” ๋…ผ๋ฌธ๋ณด๋‹ค ๋‚ซ๋‹ค.

์•ฝ์ ยทํ•œ๊ณ„

  • wall-clock์ด ์ธก์ •์ด ์•„๋‹ˆ๋ผ ์ถ”์ •์ด๋‹ค. ๋ชจ๋“  ์‹œ๊ฐ„ ์ถ•์€ ํ™˜๊ฒฝ ์‹œ๊ฐ„๊ณผ ์•Œ๊ณ ๋ฆฌ์ฆ˜ update ์‹œ๊ฐ„์„ ๋”ฐ๋กœ ์žฌ์„œ ๋”ํ•œ ๊ฐ’์ด๊ณ , update ์‹œ๊ฐ„์€ ๋Œ€ํ‘œ ํ™˜๊ฒฝ ํ•˜๋‚˜(MuJoCo Humanoid-v4 / DMC Walker-run)์—์„œ ์žฐ ๊ฒƒ์„ ์ „ ๊ณผ์ œ์— ์žฌ์‚ฌ์šฉํ•œ๋‹ค(๋ถ€๋ก A). ์ƒํƒœ ์ฐจ์› 310์˜ G1 Rough์™€ 376์˜ Humanoid-v4๋Š” ๋น„์Šทํ•˜๋”๋ผ๋„, IsaacLab์˜ GPUโ†’CPUโ†’GPU ์™•๋ณต์ด๋‚˜ 1,024 ํ™˜๊ฒฝ Python ๋ฃจํ”„ ๊ฐ™์€ ๋น„์šฉ์€ ์ด ์ถ”์ •์— ๋“ค์–ด๊ฐ€์ง€ ์•Š์„ ์ˆ˜ ์žˆ๋‹ค. ํ”„๋กœ์ ํŠธ ํŽ˜์ด์ง€์˜ scaling ๊ทธ๋ฆผ(Fig. 8 ๋Œ€์‘)์€ arXiv ํŒ๊ณผ ๊ณก์„  ๋ชจ์–‘์€ ๊ฐ™์€๋ฐ ์‹œ๊ฐ„ ์ถ• ๋๊ฐ’์ด ๋‹ค๋ฅด๋‹ค(ํญ 1024๊ฐ€ ์•ฝ 42๋ถ„ vs ์•ฝ 49๋ถ„). ์ด ๋…ผ๋ฌธ์˜ ํ—ค๋“œ๋ผ์ธ์ด โ€œ์‹œ๊ฐ„โ€์ธ ๋งŒํผ, ์‹ค์ธก end-to-end wall-clock ํ•œ ํ‘œ๊ฐ€ ํ•„์š”ํ–ˆ๋‹ค.
  • Sim-to-real ๋น„๊ต์˜ ๊ณต์ •์„ฑ. ๋ณธ๋ฌธ์€ ๊ฐ™์€ ๋ณด์ƒ์ด๋ผ ํ•˜์ง€๋งŒ ๋ถ€๋ก์€ ๊ฐ€์ค‘์น˜๊ฐ€ ํฌ๊ฒŒ ๋‹ค๋ฅด๋‹ค(termination penalty โˆ’200 vs ์—†์Œ, body orientation โˆ’52 vs โˆ’2 ๋“ฑ). ํ•˜๋“œ์›จ์–ด(A100 vs ๋ณธ๋ฌธ์˜ RTX 5090), seed 1๊ฐœ, ์‹ค๊ธฐ ์„ฑ๊ณต๋ฅ ยท์‹œ๋„ ํšŸ์ˆ˜ ๋ถ€์žฌ, domain randomization ๋ฒ”์œ„ ๋ถ€์žฌ๊นŒ์ง€ ๊ฒน์ณ โ€œhours to minutesโ€๋Š” ๋ฐฉ๋ฒ• + ๋ณด์ƒ ์„ค๊ณ„ ๋ฌถ์Œ์˜ ์‹œ์—ฐ์œผ๋กœ ์ฝ์–ด์•ผ ํ•œ๋‹ค.
  • ๋ณธ๋ฌธ ๋ฌธ์žฅ๊ณผ ๊ทธ๋ฆผยท๋ถ€๋ก์˜ ์–ด๊ธ‹๋‚จ. โ€œcondition number monotonically decreasesโ€(Fig. 9์™€ ๋‹ค๋ฆ„), โ€œeach addition stabilizesโ€(ResidualยทBNยทRMSNorm ๋‹จ๊ณ„์—์„œ ๋ฐ˜๋ก€), โ€œmatches or exceeds all baselines in asymptotic performanceโ€(๋น„์ „์—์„œ ๊ฐ™์€ step ๊ธฐ์ค€์œผ๋กœ๋Š” MR.Q๊ฐ€ 5/8 ์šฐ์œ„, ๋ถ€๋ก์—๋Š” ๋ณธ๋ฌธ์— ์—†๋Š” XQCยทDrM์ด ๋” ๋†’์€ ๊ณผ์ œ๋„ ์žˆ๋‹ค), โ€œconverging across all tasksโ€(ManiSkill PokeCube 1/5 seed๋งŒ ์„ฑ๊ณต), โ€œPPO 200M stepโ€(CSV๋Š” ๋Œ€๋ถ€๋ถ„ 100M).
  • ์„ค์ • ์„œ์ˆ  ๋ถˆ์ผ์น˜. UTD(๋ณธ๋ฌธ 2/1024 vs Table 9 2/2048), n-step(Table 9 GPU n=1 vs IsaacLab ์Šคํฌ๋ฆฝํŠธ n=3), โ€œฮณ๋งŒ ๋ฐ”๊พผ๋‹คโ€(๊ณต๊ฐœ ์Šคํฌ๋ฆฝํŠธ๋Š” IsaacLab๋งŒ n-step 3, MuJoCo PlaygroundยทGenesis๋Š” asymmetric ๊ด€์ธก์„ ์ผ ๋‹ค), ๊ณผ์ œ ์ˆ˜(๋ณธ๋ฌธ 40 vs ํ‘œยทCSV 39), โ€œ2.5M for both actor and criticโ€(ํ•ฉ๊ณ„), ๋น„์ „ ํ•˜์ดํผํŒŒ๋ผ๋ฏธํ„ฐ(โ€œCPU ์„ค์ • ์žฌ์‚ฌ์šฉโ€์ด๋ผ ํ–ˆ์ง€๋งŒ batch 256ยทUTD 0.5๋กœ ๋‹ค๋ฆ„), ยง5.4์˜ ยง5.2 ์ฐธ์กฐ, LR ์Šค์ผ€์ค„(Table 9๋Š” 1.5e-4๊นŒ์ง€ ๊ฐ์†Œ์ง€๋งŒ ๊ณต๊ฐœ ์ฝ”๋“œ์—์„œ actorยทtemperature๋Š” ์•ฝ 2.25e-4์—์„œ ๋๋‚จ). ๋Œ€๋ถ€๋ถ„ ์‚ฌ์†Œํ•˜์ง€๋งŒ ์žฌํ˜„์ž์—๊ฒŒ๋Š” n-step 3์ฒ˜๋Ÿผ ๊ฒฐ๊ณผ๋ฅผ ๋ฐ”๊ฟ€ ์ˆ˜ ์žˆ๋Š” ํ•ญ๋ชฉ์ด ์„ž์—ฌ ์žˆ๋‹ค. ๋ถ„์„ ์ ˆ(ยง6)์ด ์ „๋ถ€ IsaacLab 4๊ณผ์ œ์—์„œ ์ด๋ค„์กŒ์œผ๋ฏ€๋กœ ๊ทธ ๊ณก์„ ๋“ค๋„ n-step 3์œผ๋กœ ํ•™์Šต๋์„ ๊ฐ€๋Šฅ์„ฑ์ด ๋†’๋‹ค.
  • ๊ธฐ๋ณธ๊ฐ’์€ ์ ๊ทผ ์„ฑ๋Šฅ์ด ์•„๋‹ˆ๋ผ ์†๋„์— ๋งž์ถฐ์ ธ ์žˆ๋‹ค. ๊ฐ™์€ 50M step์ด๋ฉด ํญยท๊นŠ์ดยทbatchยทUTD๋ฅผ ํ‚ค์šด ์„ค์ •์ด ๋ชจ๋‘ ๋” ๋†’๊ฒŒ ๋๋‚œ๋‹ค. โ€œ๊ฐ€์žฅ ๋†’์€ ์ ๊ทผ ์„ฑ๋Šฅ์„ ๊ฐ€์žฅ ์งง์€ ์‹œ๊ฐ„์—โ€๋ผ๋Š” ํ™๋ณด ๋ฌธ๊ตฌ๋Š” ๋‘˜ ์ค‘ ํ•˜๋‚˜๋ฅผ ๊ณ ๋ฅธ ๊ฒฐ๊ณผ๋‹ค.
  • ์ •๊ทœํ™” ๊ธฐ์ค€์˜ ์ž๊ธฐ ์ฐธ์กฐ. IsaacLab ์ •๊ทœํ™” ์ ์ˆ˜๋Š” FlashSAC์„ ์˜ค๋ž˜ ๋Œ๋ฆฐ ์„ฑ๋Šฅ์„ ๊ธฐ์ค€์œผ๋กœ ์‚ผ๋Š”๋‹ค. ์ƒ๋Œ€ ๋น„๊ต์—๋Š” ์˜ํ–ฅ์ด ์—†์ง€๋งŒ, ์ง‘๊ณ„ ๊ทธ๋ž˜ํ”„์˜ โ€œ1.0โ€์ด ๋ฌด์—‡์„ ๋œปํ•˜๋Š”์ง€(=FlashSAC์˜ ์žฅ๊ธฐ ์„ฑ๋Šฅ) ์•Œ๊ณ  ์ฝ์–ด์•ผ ํ•œ๋‹ค.
  • Fig. 7์˜ ๋น„๊ต ๋Œ€์ƒ. on-policy ์ชฝ์ด PPO์˜ ๋ฐ์ดํ„ฐ๊ฐ€ ์•„๋‹ˆ๋ผ FlashSAC ์ตœ์ข… ์ •์ฑ…์˜ rollout์ด๋ผ, โ€œon-policy๊ฐ€ ์ข์•„์„œ ์ง„๋‹คโ€๋Š” ์ธ๊ณผ๋ฅผ ์ง์ ‘ ๋ณด์—ฌ ์ฃผ์ง€ ์•Š๋Š”๋‹ค.
  • sparse ํƒ์ƒ‰์—์„œ์˜ ์•ฝ์ ์ด ๋‹ค๋ค„์ง€์ง€ ์•Š๋Š”๋‹ค. ํƒ์ƒ‰ ์žฅ์น˜๋ฅผ ๊ธฐ์—ฌ๋กœ ๋‚ด์„ธ์šฐ๋ฉด์„œ ๋ถ€๋ก์ด ํƒ์ƒ‰ ํ‰๊ฐ€์šฉ์ด๋ผ ํ•œ cartpole swingup_sparse์—์„œ ๋‹ค๋ฅธ ๋ฐฉ๋ฒ•๋“ค์— ํฌ๊ฒŒ ๋’ค์ง„๋‹ค.
  • ๊ณต๊ฐœ ๋ฒ”์œ„๊ฐ€ ์ ˆ๋ฐ˜์ด๋‹ค. ๋ณธ ๋ ˆํฌ์—๋Š” ์ƒํƒœ ๊ธฐ๋ฐ˜ ํ•™์Šต ์ฝ”๋“œ์™€ CSV๊ฐ€ ์žˆ์ง€๋งŒ ๋น„์ „ ์ธ์ฝ”๋” ์ฝ”๋“œ์™€ sim-to-real ์ฝ”๋“œ๊ฐ€ ์—†๊ณ , ์ฒดํฌํฌ์ธํŠธ๋„ ์—†๋‹ค. sim-to-real ๊ตฌ์„ฑ ์š”์†Œ๋Š” ๋‚˜์ค‘์— ์ƒ๊ธด ํ˜•์ œ ๋ ˆํฌ Holiday-Robot/rsl_rl_flashsac์— ๋“ค์–ด ์žˆ๋‹ค. ๊ฐ™์€ ์ฝ”๋“œ๋กœ ๋…ผ๋ฌธ ๊ฒฐ๊ณผ๋ฅผ ๋ƒˆ๋Š”์ง€๋Š” ์•Œ ์ˆ˜ ์—†๋‹ค.

๊ณต๊ฐœ ์ฝ”๋“œ์™€ ๋Œ€์กฐ

์•„๋ž˜๋Š” Holiday-Robot/FlashSAC upstream ์ปค๋ฐ‹ 87edc90(2026-04-09)์„ ์ฝ์–ด ํ™•์ธํ•œ ์‚ฌ์‹ค์ด๋‹ค. ํ•™์Šต์€ ๋Œ๋ฆฌ์ง€ ์•Š์•˜๊ณ , ๋…ผ๋ฌธ ์ˆ˜์น˜๋Š” ๋ชจ๋‘ โ€œ๋…ผ๋ฌธ์ด ์ฃผ์žฅํ•œ ๊ฐ’โ€์ด๋‹ค.

ํ•ญ๋ชฉ ๋…ผ๋ฌธ ๊ณต๊ฐœ ์ฝ”๋“œ ํŒ์ •
UTD(GPU) ๋ณธ๋ฌธ 2/1024, Table 9 2/2048 num_train_envs=1024, updates_per_interaction_step=2 ๋ณธ๋ฌธ๊ณผ ์ผ์น˜, ํ‘œ ์˜ค๊ธฐ
n-step(GPU) Table 9 n=1 IsaacLab ์Šคํฌ๋ฆฝํŠธ๋งŒ n_step=3, ๋‚˜๋จธ์ง€ GPU ์‹œ๋ฎฌ 1 ์„œ์ˆ ๊ณผ ๋‹ค๋ฆ„
์‹œ๋ฎฌ๋ณ„ ์กฐ์ • โ€œฮณ๋งŒ ์‹œ๋ฎฌ ๊ธฐ๋ณธ๊ฐ’์— ๋งž์ถคโ€ ฮณ = 0.99(IsaacLab)ยท0.97(MJP)ยท0.9(ManiSkill)ยท0.95(Genesis, MyoSuite). MJPยทGenesis๋Š” asymmetric ๊ด€์ธก ์ผฌ ฮณ ์™ธ ์กฐ์ •๋„ ์žˆ์Œ
๋ชจ๋ธ ํฌ๊ธฐ โ€œ2.5M, 6-layer for both actor and criticโ€ actor ์•ฝ 0.29M + critic 2๊ฐœ ์•ฝ 2.24M = ์•ฝ 2.53M, Linear 6์ธต ํ•ฉ๊ณ„
๋ธ”๋ก ๊ตฌ์กฐ Fig. 2 Linear(hโ†’4h, bias ์—†์Œ) โ†’ BN โ†’ ReLU โ†’ Linear(4hโ†’h) โ†’ BN โ†’ ReLU โ†’ + residual ์ผ์น˜
์ž…๋ ฅ ์ •๊ทœํ™” Fig. 2 ์ฒซ BatchNorm ๋ฐ•์Šค(๋ณธ๋ฌธ ์„ค๋ช… ์—†์Œ) ์ž„๋ฒ ๋”ฉ ์•ž BatchNorm ์ผ์น˜(๋ณธ๋ฌธ ์„œ์ˆ  ๋ณด์ถฉ)
Cross-batch CrossQ ๋ฐฉ์‹ concatํ•˜์ง€๋งŒ EMA target critic(ฯ„=0.01)์„ ์œ ์ง€ํ•˜๊ณ  target๋„ ๋ฐฐ์น˜ ํ†ต๊ณ„๋กœ forward(EMA๊ฐ€ BN ๋ฒ„ํผ๋ฅผ ๋ณต์‚ฌํ•˜์ง€ ์•Š์Œ) ๋…ผ๋ฌธ ๋ฐ– ์„ธ๋ถ€
Actor update์˜ Q ์„œ์ˆ  ์—†์Œ actor ์†์‹ค์˜ Q๋Š” critic์„ BN running ํ†ต๊ณ„(eval ๋ชจ๋“œ, momentum 0.01)๋กœ ํ‰๊ฐ€. critic ํ•™์Šต์€ ๋ฐฐ์น˜ ํ†ต๊ณ„ ๋…ผ๋ฌธ ๋ฐ– ์„ธ๋ถ€
LR ์Šค์ผ€์ค„ cosine decay 3e-4 โ†’ 1.5e-4(Table 9) actorยทcriticยทtemperature๊ฐ€ ๊ฐ™์€ ์Šค์ผ€์ค„(์ด update ์ˆ˜ ๊ธฐ์ค€)์„ ์“ฐ์ง€๋งŒ actorยทtemperature ์Šค์ผ€์ค„๋Ÿฌ๋Š” actor update(2ํšŒ์— 1๋ฒˆ)์—์„œ๋งŒ step โ†’ ํ•™์Šต ๋์— cosine ์ ˆ๋ฐ˜ ์ง€์ , LR ์•ฝ 2.25e-4. critic๋งŒ 1.5e-4์— ๋„๋‹ฌ ์„œ์ˆ ๊ณผ ๋‹ค๋ฆ„(actorยทtemperature)
Weight norm ๋ฒ”์œ„ ๊ฐ weight ๋ฒกํ„ฐ โ†’ ๋‹จ์œ„ norm, (\gamma,\beta) โ†’ \sqrt d ๋ชจ๋“  Linear ํ–‰ ์ •๊ทœํ™”, BN์˜ [\gamma;\beta]๋ฅผ ํ•จ๊ป˜ \sqrt d๋กœ. value headยทpolicy head bias๋Š” ์ œ์™ธ ์ผ์น˜(์„ธ๋ถ€ ๋ณด์ถฉ)
Noise repetition ฮต๋ฅผ k step ์œ ์ง€, k~Zeta(2), ์ตœ๋Œ€ 16, โ€œminimal local stateโ€ ๊ฐ™์Œ. k์™€ ์นด์šดํ„ฐ๋Š” ์ „ ํ™˜๊ฒฝ ๊ณต์œ  ์Šค์นผ๋ผ(๋…ผ๋ฌธ ์„œ์ˆ ๊ณผ ๋ถ€ํ•ฉ). ์—ํ”ผ์†Œ๋“œ reset ๋•Œ ฮต๋ฅผ ๋‹ค์‹œ ๋ฝ‘์ง€ ์•Š์Œ reset ์ฒ˜๋ฆฌ๋งŒ ๋…ผ๋ฌธ์— ์—†์Œ
Reward scaling Eq. 6 ํ• ์ธ return์˜ running ๋ถ„์‚ฐ๊ณผ ์ตœ๋Œ€ ํฌ๊ธฐ๋กœ ๋‚˜๋ˆ”, ํ‰๊ท ์€ ๋นผ์ง€ ์•Š์Œ, G_{\max}=5 ์ผ์น˜
๋น„์ „ยทsim-to-real ยง5.3, ยง5.4 ๋ณธ ๋ ˆํฌ์— ์—†์Œ(CSV๋งŒ) ๋ฏธ๊ณต๊ฐœ

์žฌํ˜„ํ•  ๋•Œ ์•Œ์•„ ๋‘˜ ํ•จ์ •๋„ ๋ช‡ ๊ฐ€์ง€ ์žˆ๋‹ค. IsaacLabยทGenesis๋Š” ํ•œ ํ”„๋กœ์„ธ์Šค์— ์‹œ๋ฎฌ๋ ˆ์ดํ„ฐ๋ฅผ ํ•˜๋‚˜๋งŒ ๋„์šธ ์ˆ˜ ์žˆ์–ด ํ‰๊ฐ€ ํ™˜๊ฒฝ = ํ•™์Šต ํ™˜๊ฒฝ์ด๊ณ , ํ‰๊ฐ€๊ฐ€ ํ•™์Šต ํ™˜๊ฒฝ์„ resetํ•œ ๋’ค ํ•™์Šต ๋ฃจํ”„๋Š” ํ‰๊ฐ€ ์ „์˜ ๊ด€์ธก์œผ๋กœ ์ด์–ด์ง„๋‹ค. ๊ธฐ๋ณธ ์„ค์ •์—์„œ ํ‰๊ฐ€ 10ํšŒ ร— 1,024๊ฐœ ์ „์ด์˜ ๊ด€์ธก ์ง์ด ์–ด๊ธ‹๋‚œ๋‹ค. ๋˜ IsaacLab์€ ์ข…๋ฃŒ ์ง์ „ ๊ด€์ธก์„ ์ฃผ์ง€ ์•Š์•„ time-out bootstrap์ด reset ํ›„ ๊ด€์ธก์„ ์“ด๋‹ค(์ฝ”๋“œ ์ฃผ์„์ด IsaacLab ์ด์Šˆ๋ฅผ ๋“ค์–ด ์ธ์ •ํ•œ๋‹ค). ๋‘˜ ๋‹ค ์„ฑ๋Šฅ์— ๋ฏธ์น˜๋Š” ํฌ๊ธฐ๋Š” ์ž‘์„ ๊ฒƒ์œผ๋กœ ๋ณด์ด์ง€๋งŒ, ๊ณต๊ฐœ ์ฝ”๋“œ๋กœ ์žฌํ•™์Šตํ•œ ์ˆ˜์น˜๊ฐ€ ๋…ผ๋ฌธ ๊ณก์„ ๊ณผ ๋ฏธ์„ธํ•˜๊ฒŒ ๋‹ค๋ฅผ ์ˆ˜ ์žˆ๋Š” ์ด์œ ๋‹ค. Python 3.10/3.11์— ๋”ฐ๋ผ Isaac Sim 4.5 + IsaacLab 2.1 ๋˜๋Š” 5.1 + 2.3์„ ์“ฐ๋„๋ก ๊ณ ์ •๋ผ ์žˆ๊ณ , IsaacLab extra๋Š” GenesisยทHumanoidBench์™€ ๊ฐ™์€ ํ™˜๊ฒฝ์— ์„ค์น˜๋˜์ง€ ์•Š๋Š”๋‹ค.

๊ด€๋ จ ์—ฐ๊ตฌ์™€์˜ ์ž๋ฆฌ ๋งค๊น€

FlashSAC์€ off-policy ์•ˆ์ •ํ™” ๊ณ„๋ณด์˜ โ€œ๊ทœ๋ชจโ€ ๋ฒ„์ „์ด๋‹ค. SimBa๊ฐ€ โ€œ์•„ํ‚คํ…์ฒ˜๋งŒ ๋ฐ”๊ฟ”๋„ ํŒŒ๋ผ๋ฏธํ„ฐ scaling์ด ์—ด๋ฆฐ๋‹คโ€๋ฅผ ๋ณด์˜€๊ณ , SimbaV2๊ฐ€ hypersphere ์ •๊ทœํ™”ยทdistributional criticยท๋ณด์ƒ ์Šค์ผ€์ผ๋ง์œผ๋กœ ์ด๋ฅผ ๋‹ค๋“ฌ์—ˆ์œผ๋ฉฐ, CrossQยทXQC๋Š” BatchNorm๊ณผ condition number๋กœ ๊ฐ™์€ ๋ฌธ์ œ๋ฅผ ๋‹ค๋ฅธ ์ชฝ์—์„œ ํ’€์—ˆ๋‹ค. ์ด๋“ค์ด ์ฃผ๋กœ ์ƒ˜ํ”Œ ํšจ์œจ(1M step, ๋‹จ์ผ ํ™˜๊ฒฝ)์—์„œ ๊ฒฝ์Ÿํ–ˆ๋‹ค๋ฉด, FlashSAC์€ ๊ฐ™์€ ๋ถ€ํ’ˆ์„ FastTD3ยทFastSAC์ด ์—ฐ ๋Œ€๊ทœ๋ชจ ๋ณ‘๋ ฌยทwall-clock ์ฒด์ œ๋กœ ์˜ฎ๊ฒผ๋‹ค. ๊ทธ๋ž˜์„œ CPU ๋ฒค์น˜๋งˆํฌ์—์„œ๋Š” XQCยทSimbaV2์™€ ๊ทผ์†Œํ•œ ์ฐจ์ด๋กœ ์•ž์„œ๊ณ , GPU ๋ฒค์น˜๋งˆํฌ์—์„œ๋Š” FastTD3์™€ PPO๋ฅผ ํฌ๊ฒŒ ์•ž์„ ๋‹ค. ๊ธฐ์—ฌ์˜ ๋ฌด๊ฒŒ๋Š” ์ƒˆ ๋ถ€ํ’ˆ๋ณด๋‹ค ์ฒด์ œ์˜ ์ด๋™๊ณผ ๊ทธ ์ฒด์ œ์—์„œ์˜ ํ•˜์ดํผํŒŒ๋ผ๋ฏธํ„ฐ ์„ ํƒ์— ์žˆ๋‹ค.

์‘์šฉ ์ชฝ์—์„œ๋Š” ์ด๋ฏธ ํ›„์† ์‚ฌ์šฉ์ด ๋ณด์ธ๋‹ค. FlashDexRetarget์€ FlashSAC์„ replay 50Mยทcritic ํญ 1024๋กœ ํ‚ค์›Œ ์ˆ˜์‹ญ~์ˆ˜์ฒœ ๊ฐœ ์† ๋™์ž‘ ๋ฆฌํƒ€๊ฒŒํŒ…์„ ํ•œ ์ •์ฑ…์— ๋‹ด์•˜๊ณ , ๊ทธ ๋…ผ๋ฌธ์˜ ์–ด๋ธ”๋ ˆ์ด์…˜์—์„œ ํ•™์Šต๊ธฐ ๊ต์ฒด(PPO โ†’ FlashSAC โ†’ ํฐ buffer โ†’ ํฐ critic)๊ฐ€ ๊ฐ€์žฅ ํฐ ํšจ๊ณผ๋ฅผ ๋ƒˆ๋‹ค. CHORD v2 ๋ถ€๋ก๋„ FlashSAC์œผ๋กœ ํ•™์Šตํ–ˆ๋‹ค๊ณ  ์ ๋Š”๋‹ค. ํœด๋จธ๋…ธ์ด๋“œ ๋ฒค์น˜๋งˆํฌ ์ชฝ์—์„œ๋Š” HumanoidBench์˜ 14๊ณผ์ œ๊ฐ€ CPU ์‹คํ—˜์— ๋“ค์–ด ์žˆ๊ณ , FlashSAC์€ ๋Œ€๋ถ€๋ถ„์˜ ๊ณผ์ œ์—์„œ TD-MPC2๋ฅผ ๋„˜์ง€๋งŒ H1 Hurdle์ฒ˜๋Ÿผ ํฌ๊ฒŒ ๋’ค์ง€๋Š” ๊ณผ์ œ๋„ ์žˆ๋‹ค.

์ƒ๋ณด์ ์ธ ๋ฐฉํ–ฅ์œผ๋กœ๋Š” ์ €์ž๋“ค์ด ๊ฒฐ๋ก ์—์„œ ๋“  ๊ฒƒ๋“ค์ด ์žˆ๋‹ค. ๋ฐ์ดํ„ฐ ํšจ์œจ์ด ์ข‹์•„์ง€๋ฉด ๋” ํฐ ์ •์ฑ…, ๋น„์ „ยท์ด‰๊ฐ ๊ฐ™์€ ํ’๋ถ€ํ•œ ๊ฐ๊ฐ, ๋А๋ฆฌ์ง€๋งŒ ์‚ฌ์‹ค์ ์ธ ์‹œ๋ฎฌ๋ ˆ์ดํ„ฐ, ์‹œ์—ฐ๊ณผ ์ž๊ธฐ ์ˆ˜์ง‘ ๋ฐ์ดํ„ฐ์˜ ํ˜ผํ•ฉ(RLPD์‹)์„ ์“ธ ์ˆ˜ ์žˆ๋‹ค๋Š” ๊ฒƒ์ด๋‹ค. ํŠนํžˆ ์ด‰๊ฐ ๊ธฐ๋ฐ˜ ํ•™์Šต์œผ๋กœ์˜ ํ™•์žฅ์„ future work๋กœ ๋ช…์‹œํ•œ๋‹ค(ยง7). ๋‹ค๋งŒ ์ด ๋…ผ๋ฌธ์ด ์‹ค์ œ๋กœ ๋ณด์ธ ๊ฒƒ์€ ์ƒํƒœ ๊ธฐ๋ฐ˜ ๊ณ ์ฐจ์› ์ œ์–ด๊นŒ์ง€์ด๊ณ , ๋น„์ „์€ ๊ณต๊ฐœ ์ฝ”๋“œ ์—†์ด CSV๋งŒ ์žˆ๋‹ค.

์š”์•ฝ

FlashSAC์€ off-policy RL์„ โ€œupdate๋ฅผ ์ ๊ฒŒ, ํฌ๊ฒŒโ€ ํ•˜๋Š” ์ฒด์ œ๋กœ ์˜ฎ๊ธฐ๊ณ , ๊ทธ ์ฒด์ œ์—์„œ ํฐ critic์ด ๋ฌด๋„ˆ์ง€์ง€ ์•Š๋„๋ก BatchNormยทRMSNormยทdistributional criticยทweight normalization์œผ๋กœ norm์„ ๋ฌถ์€ SAC ๋ณ€ํ˜•์ด๋‹ค. ํ•˜๋‚˜์˜ ์„ค์ •์œผ๋กœ 10๊ฐœ ์‹œ๋ฎฌ๋ ˆ์ดํ„ฐ 60์—ฌ ๊ฐœ ๊ณผ์ œ๋ฅผ ๋Œ๋ ค ๊ณ ์ฐจ์› ๊ณผ์ œ(๋‹ค์ง€ ์†, ํœด๋จธ๋…ธ์ด๋“œ)์—์„œ PPO์™€ FastTD3๋ฅผ ํฌ๊ฒŒ ์•ž์„ฐ๊ณ , G1 ์‹ค๊ธฐ ๋ณดํ–‰ ํ•™์Šต ์‹œ๊ฐ„์„ ๋ช‡ ์‹œ๊ฐ„์—์„œ ๋ช‡์‹ญ ๋ถ„์œผ๋กœ ์ค„์˜€๋‹ค๊ณ  ๋ณด๊ณ ํ•œ๋‹ค. ํ•™์Šต๊ณก์„  ์›์ž๋ฃŒ๊นŒ์ง€ ๊ณต๊ฐœํ•ด ์ฃผ์žฅ์„ ์ˆซ์ž๋กœ ๋Œ€์กฐํ•  ์ˆ˜ ์žˆ๊ฒŒ ํ•œ ์ ์€ ์ด ๋ถ„์•ผ์—์„œ ๋“œ๋ฌธ ๋ฏธ๋•์ด๋‹ค.

๋™์‹œ์— ์ด ๋ฆฌ๋ทฐ๊ฐ€ ์›์ž๋ฃŒ๋กœ ํ™•์ธํ•œ ๋ฒ”์œ„์—์„œ๋Š” ์„œ์ˆ ์˜ ์ •๋ฐ€๋„๊ฐ€ ๊ฒฐ๊ณผ์˜ ๊ฐ•๋„๋ฅผ ๋”ฐ๋ผ๊ฐ€์ง€ ๋ชปํ•œ๋‹ค. wall-clock์€ ์ถ”์ •๊ฐ’์ด๊ณ , sim-to-real์˜ ์‹œ๊ฐ„ ๋น„๊ต๋Š” ๋‹ค๋ฅธ ๋ณด์ƒ ์„ค๊ณ„ ์œ„์—์„œ ๋‚˜์™”์œผ๋ฉฐ, โ€œ์ ๊ทผ ์„ฑ๋Šฅ์—์„œ๋„ ์•ž์„ ๋‹คโ€๋Š” ๋ฌธ์žฅ์€ ๋น„์ „ยท์ผ๋ถ€ ์ €์ฐจ์› ๊ณผ์ œ์—์„œ ์›์ž๋ฃŒ์™€ ๋งž์ง€ ์•Š๋Š”๋‹ค. ์•„ํ‚คํ…์ฒ˜ ์ด๋“์˜ ๋Œ€๋ถ€๋ถ„์€ distributional critic ํ•œ ๋‹จ๊ณ„์—์„œ ์˜ค๊ณ , ๊ธฐ๋ณธ ์„ค์ •์€ ์ ๊ทผ ์„ฑ๋Šฅ์ด ์•„๋‹ˆ๋ผ ์†๋„์— ๋งž์ถฐ ๊ณจ๋ž๋‹ค. ๊ทธ๋ž˜์„œ ๊ฐ€์žฅ ์ •ํ™•ํ•œ ํ•œ ์ค„์€ ์ด๋ ‡๋‹ค. ๊ณ ์ฐจ์› ์ƒํƒœ ๊ธฐ๋ฐ˜ ์ œ์–ด์—์„œ PPO ๋Œ€์‹  ๋จผ์ € ์‹œ๋„ํ•ด ๋ณผ ๋งŒํ•œ, ์ž˜ ํŠœ๋‹๋œ off-policy ๊ธฐ๋ณธ๊ฐ’. โ€œIf youโ€™re using PPO, try FlashSACโ€๋ผ๋Š” ํ”„๋กœ์ ํŠธ ํŽ˜์ด์ง€์˜ ์ œ์•ˆ์€ ๊ทธ ๋ฒ”์œ„ ์•ˆ์—์„œ ์„ค๋“๋ ฅ์ด ์žˆ๋‹ค.

์ด ๊ธ€์— ๋Œ€ํ•ด

  • ์ฝ์€ ๋ฒ”์œ„: arXiv v2(2026-05-15)์˜ ๋ณธ๋ฌธยท๋ถ€๋ก์„ e-print ์†Œ์Šค์™€ HTML๋กœ ๋Œ€์กฐํ•ด ์ฝ์—ˆ๋‹ค. v1(2026-04-06)๊ณผ์˜ ์ฐจ์ด๋Š” ๊ด€๋ จ ์—ฐ๊ตฌ ์ ˆ(ํƒ์ƒ‰ ๊ฐˆ๋ž˜ ์ถ”๊ฐ€)๋ฟ์ด๊ณ  ๊ทธ๋ฆผยทํ‘œ๋Š” ๊ฐ™๋‹ค. ๊ฒŒ์žฌ์ฒ˜ RSS 2026์€ arXiv ๋ฉ”ํƒ€๋ฐ์ดํ„ฐ์˜ comment ๊ธฐ์ค€์ด๋‹ค. Fig. 1~10ยท12ยท17ยท28์€ e-print์˜ ๊ทธ๋ฆผ PDF๋ฅผ, Fig. 11์€ arXiv HTML ๊ทธ๋ฆผ์„ ์ผ๋‹ค.
  • ๋…ผ๋ฌธ ๋ฐ– ์ž๋ฃŒ: ๊ณต๊ฐœ ๋ ˆํฌ Holiday-Robot/FlashSAC(MIT, upstream ์ปค๋ฐ‹ 87edc90)์˜ scripts/ยทresults/ยท์•Œ๊ณ ๋ฆฌ์ฆ˜ ์ฝ”๋“œ์™€ ํ”„๋กœ์ ํŠธ ํŽ˜์ด์ง€(์˜์ƒ, ๊ทธ๋ฆผ)๋ฅผ ์ฐธ๊ณ ํ–ˆ๊ณ , ๋ณธ๋ฌธ์—์„œ ๊ทธ๋•Œ๋งˆ๋‹ค ์ถœ์ฒ˜๋ฅผ ๋ฐํ˜”๋‹ค. ๋ ˆํฌ์—๋Š” ์ฒดํฌํฌ์ธํŠธยท๋น„์ „ ์ฝ”๋“œยทsim-to-real ์ฝ”๋“œ๊ฐ€ ์—†๋‹ค.
  • ํ‘œ๊ธฐ: โ€œํ•„์ž ๊ณ„์‚ฐโ€์€ ๊ณต๊ฐœ results/ CSV์˜ ๋งˆ์ง€๋ง‰ ํ‰๊ฐ€๊ฐ’ seed ํ‰๊ท ์œผ๋กœ ๊ณ„์‚ฐํ•œ ๊ฐ’(๊ณผ์ œ๋ณ„ ์ตœ์ข…๊ฐ’, 4๊ณผ์ œ ์ •๊ทœํ™” ํ‰๊ท , ์ŠนํŒจ ์ˆ˜) ๋˜๋Š” ๋…ผ๋ฌธ ์„ค์ •์œผ๋กœ ๊ณ„์‚ฐํ•œ ๊ฐ’(ํŒŒ๋ผ๋ฏธํ„ฐ ์ˆ˜, Zeta ๋ถ„ํฌ ํ†ต๊ณ„, entropy target)์ด๋‹ค. ๋…ผ๋ฌธ ๊ทธ๋ฆผ์˜ ์ง‘๊ณ„ ๋ฐฉ์‹๊ณผ ๋‹ค๋ฅผ ์ˆ˜ ์žˆ์–ด ์ ˆ๋Œ€๊ฐ’์€ ๊ทธ๋ฆผ๊ณผ ์กฐ๊ธˆ ๋‹ค๋ฅผ ์ˆ˜ ์žˆ๊ณ , ํ†ต๊ณ„ ๊ฒ€์ •์€ ํ•˜์ง€ ์•Š์•˜๋‹ค. โ€œ๊ทธ๋ฆผ์—์„œ ์ฝ์€ ๊ฐ’โ€์€ ๊ทธ๋ž˜ํ”„์˜ ์ ์„ ๋ˆˆ์œผ๋กœ ์ฝ์€ ๊ทผ์‚ฌ์น˜๋‹ค. โ€œํ•„์ž์˜ ํ•ด์„ยท์ถ”์ •โ€์€ ๋…ผ๋ฌธ์ด ์ง์ ‘ ์ฃผ์žฅํ•˜์ง€ ์•Š์€ ํ•ด์„์ด๋‹ค. ์ฝ”๋“œ ํ™•์ธ์€ ์ •์  ๋…ํ•ด์ด๋ฉฐ ์‹คํ–‰ ๊ฒ€์ฆ์ด ์•„๋‹ˆ๋‹ค. ์ธ์šฉ์€ ๋Œ€๊ด„ํ˜ธ ๋ฒˆํ˜ธ ๋Œ€์‹  ์ €์ž-์—ฐ๋„ยท๋…ผ๋ฌธ ์ด๋ฆ„์œผ๋กœ ์ ์—ˆ๋‹ค.

Copyright 2026, JungYeon Lee