PoEM: Predicting RL Outcomes from Existing Policies

AI Digest

PoEM框架通过现有策略预测新奖励函数的RL结果，减少重复训练成本。

PoEM predicts RL outcomes on new rewards using existing policies, avoiding redundant training.

Key points

• 新奖励可分解为现有奖励的线性组合时，策略可直接合成 Linear reward combinations enable policy synthesis
• 非线性奖励场景下log-policy仍呈现低秩子空间特性 Log-policies form low-rank subspaces even non-linearly
• 仅需样本奖励/策略输出即可估计组合权重系数 Weights estimated from sample reward/policy outputs
• 无需额外RL训练即可近似目标策略 Target policy approximated without RL training
• 验证覆盖文本/图像多模态合成与真实奖励 Validated across synthetic/real rewards in multiple modalities

Takeaway: PoEM通过复用已有模型显著降低RL训练成本。 / PoEM reduces RL training costs by reusing existing models.

Why it matters 该方法为多奖励场景提供高效策略预测方案，具实际应用价值。