Minecraft is an extremely complex environment which provides players with visual, auditory, and informational observation of many complex data types. Furthermore, players interact with Minecraft using more than just embodied actions: players can craft, build, destroy, smelt, enchant, manage their inventory, and even communicate with other players via a text chat.
To provide a unified interface with which agents can obtain and perform
similar observations and actions as players, we have provided
first-class for support for this multi-modality in the environment:
the observation and action spaces of environments are
gym.spaces.Dict spaces. These observation and action
dictionaries are comprised of individual fields we call handlers.
In the documentation of every environment we provide a listing
of the exact
gym.space of the observations returned by and actions expected by the environment’s step function. We are slowly
building documentation for these handlers, and you can click those highlighted with blue for more information!
Observation and action spaces that are
Enum are encoded as strings by default (e.g. “none”,
“log”, and “sandstone#2”) when they are returned from
When building an action to pass into
env.step(act), the Enum component of the action dict can
be encoded as either a string or an integer.
The Enum integer value that corresponds to each Enum string value can be accessed via
Enum.values_map[string_value]. For example, to get the integer value corresponding to the
equip action “dirt” in
MineRLBasaltBuildVillageHouse, you can
Visual Observations -
- pov : Box(width, height, nchannels)
An RGB image observation of the agent’s first-person perspective.
- third-person : Box(width, height, nchannels)
An RGB image observation of the agent’s third-person perspective.
This observation is not yet supported by any environment.
- compass-observation : Box(1)
The current position of the minecraft:compass object from 0 (behind agent left) to 0.5 in front of agent to 1 (behind agent right)
This observation uses the default Minecraft game logic which includes compass needle momentum. As such it may change even when the agent has stoped moving!
Equip Observations -
- equipped_items.mainhand.type : Enum('none', 'air', ..., 'other'))
This observation is an Enum type. See Enum Spaces for more information.
The type of the item that the player has equipped in the mainhand slot. If the mainhand slot is empty then the value is ‘air’. If the mainhand slot contains an item not inside this observation space, then the value is ‘other’.
Camera Control -
- camera : Box(2) [delta_pitch, delta_yaw]
This action changes the orientation of the agent’s head by the corresponding number of degrees. When the
povobservation is available, the camera changes its orientation pitch by the first component and its yaw by the second component. Both
delta_yaware limited to [-180, 180] inclusive
- attack : Discrete(1) [attack]
This action causes the agent to attack.
Tool Control -
- equip : Enum('none', 'air', ..., 'other'))
This is action is an Enum type. See Enum Spaces for more information.
This action equips the first instance of the specified item from the agents inventory to the main hand if the specified item is present, otherwise does nothing.
airmatches any empty slot in an agent’s inventory and functions as an un-equip, or equip-nothing action.
equip 'other'are both no-op actions. In other words, they leave the currently equipped item unchanged. However, in the MineRL dataset,
othertakes on a special meaning.
otheris the wildcard equip action that is recorded in the dataset whenever a player equipped an item that wasn’t included in this action space’s Enum.
env.step(act) typically will not process the equip action for two ticks (i.e., you will not see the observation value equipped_items change until two more calls to env.step.)
This is due to a limitation with the current version of Malmo, our Minecraft backend.
- use : Discrete(1) [use]
This action is equivalent to right-clicking in Minecraft. It causes the agent to use the item it is holding in the mainhand slot, or to open doors or gates when it is facing an applicable Minecraft structure.