Table of Contents

Class QMIXAgent<T>

Namespace
AiDotNet.ReinforcementLearning.Agents.QMIX
Assembly
AiDotNet.dll

QMIX agent for multi-agent value-based reinforcement learning.

public class QMIXAgent<T> : DeepReinforcementLearningAgentBase<T>, IRLAgent<T>, IFullModel<T, Vector<T>, Vector<T>>, IModel<Vector<T>, Vector<T>, ModelMetadata<T>>, IModelSerializer, ICheckpointableModel, IParameterizable<T, Vector<T>, Vector<T>>, IFeatureAware, IFeatureImportance<T>, ICloneable<IFullModel<T, Vector<T>, Vector<T>>>, IGradientComputable<T, Vector<T>, Vector<T>>, IJitCompilable<T>, IDisposable

Type Parameters

T

The numeric type used for calculations.

Inheritance
QMIXAgent<T>
Implements
IFullModel<T, Vector<T>, Vector<T>>
IModel<Vector<T>, Vector<T>, ModelMetadata<T>>
IParameterizable<T, Vector<T>, Vector<T>>
ICloneable<IFullModel<T, Vector<T>, Vector<T>>>
IGradientComputable<T, Vector<T>, Vector<T>>
Inherited Members
Extension Methods

Remarks

QMIX factorizes joint action-values into per-agent values using a mixing network that monotonically combines them.

For Beginners: QMIX solves multi-agent problems by letting each agent learn its own Q-values, then using a "mixing network" to combine them into a team Q-value.

Key innovation:

  • Value Factorization: Team value = mix(agent1_Q, agent2_Q, ...)
  • Mixing Network: Ensures individual and joint actions are consistent
  • Monotonicity: If one agent improves, team improves
  • Decentralized Execution: Each agent acts independently

Think of it like: Each player estimates their contribution, and a coach combines these to determine the team's overall score.

Famous for: StarCraft II micromanagement, cooperative games

Constructors

QMIXAgent(QMIXOptions<T>, IOptimizer<T, Vector<T>, Vector<T>>?) 

public QMIXAgent(QMIXOptions<T> options, IOptimizer<T, Vector<T>, Vector<T>>? optimizer = null)

Parameters

options QMIXOptions<T>
optimizer IOptimizer<T, Vector<T>, Vector<T>>

Properties

FeatureCount

Gets the number of input features (state dimensions).

public override int FeatureCount { get; }

Property Value

int

Methods

ApplyGradients(Vector<T>, T)

Applies gradients to update the agent.

public override void ApplyGradients(Vector<T> gradients, T learningRate)

Parameters

gradients Vector<T>
learningRate T

Clone()

Clones the agent.

public override IFullModel<T, Vector<T>, Vector<T>> Clone()

Returns

IFullModel<T, Vector<T>, Vector<T>>

ComputeGradients(Vector<T>, Vector<T>, ILossFunction<T>?)

Computes gradients for the agent.

public override Vector<T> ComputeGradients(Vector<T> input, Vector<T> target, ILossFunction<T>? lossFunction = null)

Parameters

input Vector<T>
target Vector<T>
lossFunction ILossFunction<T>

Returns

Vector<T>

Deserialize(byte[])

Deserializes the agent from bytes.

public override void Deserialize(byte[] data)

Parameters

data byte[]

GetMetrics()

Gets the current training metrics.

public override Dictionary<string, T> GetMetrics()

Returns

Dictionary<string, T>

Dictionary of metric names to values.

GetModelMetadata()

Gets model metadata.

public override ModelMetadata<T> GetModelMetadata()

Returns

ModelMetadata<T>

GetParameters()

Gets the agent's parameters.

public override Vector<T> GetParameters()

Returns

Vector<T>

LoadModel(string)

Loads the agent's state from a file.

public override void LoadModel(string filepath)

Parameters

filepath string

Path to load the agent from.

Predict(Vector<T>)

Makes a prediction using the trained agent.

public override Vector<T> Predict(Vector<T> input)

Parameters

input Vector<T>

Returns

Vector<T>

PredictAsync(Vector<T>) 

public Task<Vector<T>> PredictAsync(Vector<T> input)

Parameters

input Vector<T>

Returns

Task<Vector<T>>

ResetEpisode()

Resets episode-specific state (if any).

public override void ResetEpisode()

SaveModel(string)

Saves the agent's state to a file.

public override void SaveModel(string filepath)

Parameters

filepath string

Path to save the agent.

SelectAction(Vector<T>, bool)

Selects an action given the current state observation.

public override Vector<T> SelectAction(Vector<T> state, bool training = true)

Parameters

state Vector<T>

The current state observation as a Vector.

training bool

Whether the agent is in training mode (affects exploration).

Returns

Vector<T>

Action as a Vector (can be discrete or continuous).

SelectActionForAgent(int, Vector<T>, bool)

Select action for a specific agent using epsilon-greedy.

public Vector<T> SelectActionForAgent(int agentId, Vector<T> state, bool training = true)

Parameters

agentId int
state Vector<T>
training bool

Returns

Vector<T>

Serialize()

Serializes the agent to bytes.

public override byte[] Serialize()

Returns

byte[]

SetParameters(Vector<T>)

Sets the agent's parameters.

public override void SetParameters(Vector<T> parameters)

Parameters

parameters Vector<T>

StoreExperience(Vector<T>, Vector<T>, T, Vector<T>, bool)

Stores an experience tuple for later learning.

public override void StoreExperience(Vector<T> state, Vector<T> action, T reward, Vector<T> nextState, bool done)

Parameters

state Vector<T>

The state before action.

action Vector<T>

The action taken.

reward T

The reward received.

nextState Vector<T>

The state after action.

done bool

Whether the episode terminated.

StoreMultiAgentExperience(List<Vector<T>>, List<Vector<T>>, T, List<Vector<T>>, Vector<T>, Vector<T>, bool)

Store multi-agent experience with global state.

public void StoreMultiAgentExperience(List<Vector<T>> agentStates, List<Vector<T>> agentActions, T teamReward, List<Vector<T>> nextAgentStates, Vector<T> globalState, Vector<T> nextGlobalState, bool done)

Parameters

agentStates List<Vector<T>>
agentActions List<Vector<T>>
teamReward T
nextAgentStates List<Vector<T>>
globalState Vector<T>
nextGlobalState Vector<T>
done bool

Train()

Performs one training step, updating the agent's policy/value function.

public override T Train()

Returns

T

The training loss for monitoring.

TrainAsync() 

public Task TrainAsync()

Returns

Task
Edit this page