#physicalfoundationmodels — Public Fediverse posts

One thing to understand about physical foundation models or robotic foundation models is in-context learning.

You should aim to frame the problem and the data in a fashion where the model can learn to control the embodiment in-context, rather than training it without a possibility to calibrate and discover where it is in the start of the session.

Otherwise you won't get truly universal models, but models which constantly hedge their bets and are forced to make their control signal not only generalist, but generalist across all training worlds and embodiments *at the same time*.

This means that you'll be stuck in a frame where you will need a control adapter layer separately trained per embodiment, because the foundation model is incapable of discovering in-context what it inhabits, so its outputs are by necessity the kind that should work somewhat ok for all possible worlds.

The model also becomes unable to learn embodiment-specific control policies without hacks.

I believe the fact that people don't realize they need to consider in-context learning for these foundation models for embodiment calibration is a root of many practical problems down the line.

#PhysicalFoundationModels #UniversalEmbodiment #robots #FoundationModels