Table of Contents

Class E2FGVI<T>

Namespace
AiDotNet.Video.Inpainting
Assembly
AiDotNet.dll

E2FGVI - End-to-End Framework for Flow-Guided Video Inpainting.

public class E2FGVI<T> : NeuralNetworkBase<T>, INeuralNetworkModel<T>, INeuralNetwork<T>, IFullModel<T, Tensor<T>, Tensor<T>>, IModel<Tensor<T>, Tensor<T>, ModelMetadata<T>>, IModelSerializer, ICheckpointableModel, IParameterizable<T, Tensor<T>, Tensor<T>>, IFeatureAware, IFeatureImportance<T>, ICloneable<IFullModel<T, Tensor<T>, Tensor<T>>>, IGradientComputable<T, Tensor<T>, Tensor<T>>, IJitCompilable<T>, IInterpretableModel<T>, IInputGradientComputable<T>, IDisposable

Type Parameters

T

The numeric type used for calculations.

Inheritance
E2FGVI<T>
Implements
IFullModel<T, Tensor<T>, Tensor<T>>
IModel<Tensor<T>, Tensor<T>, ModelMetadata<T>>
IParameterizable<T, Tensor<T>, Tensor<T>>
ICloneable<IFullModel<T, Tensor<T>, Tensor<T>>>
IGradientComputable<T, Tensor<T>, Tensor<T>>
Inherited Members
Extension Methods

Remarks

For Beginners: E2FGVI removes unwanted objects from videos and fills in the gaps with realistic content. It uses optical flow (motion information) to propagate known content into missing regions across frames.

Use cases:

  • Remove watermarks or logos from videos
  • Remove unwanted people or objects
  • Repair damaged or corrupted video frames
  • Video restoration and cleanup

Technical Details: - End-to-end trainable flow-guided inpainting - Bidirectional flow propagation - Transformer-based content hallucination - Temporal consistency enforcement

Constructors

E2FGVI(NeuralNetworkArchitecture<T>, int) 

public E2FGVI(NeuralNetworkArchitecture<T> architecture, int numFeatures = 128)

Parameters

architecture NeuralNetworkArchitecture<T>
numFeatures int

Properties

SupportsTraining

Gets whether training is supported.

public override bool SupportsTraining { get; }

Property Value

bool

Methods

CreateNewInstance()

Creates a new instance of the same type as this neural network.

protected override IFullModel<T, Tensor<T>, Tensor<T>> CreateNewInstance()

Returns

IFullModel<T, Tensor<T>, Tensor<T>>

A new instance of the same neural network type.

Remarks

For Beginners: This creates a blank version of the same type of neural network.

It's used internally by methods like DeepCopy and Clone to create the right type of network before copying the data into it.

DeserializeNetworkSpecificData(BinaryReader)

Deserializes network-specific data that was not covered by the general deserialization process.

protected override void DeserializeNetworkSpecificData(BinaryReader reader)

Parameters

reader BinaryReader

The BinaryReader to read the data from.

Remarks

This method is called at the end of the general deserialization process to allow derived classes to read any additional data specific to their implementation.

For Beginners: Continuing the suitcase analogy, this is like unpacking that special compartment. After the main deserialization method has unpacked the common items (layers, parameters), this method allows each specific type of neural network to unpack its own unique items that were stored during serialization.

GetModelMetadata()

Gets the metadata for this neural network model.

public override ModelMetadata<T> GetModelMetadata()

Returns

ModelMetadata<T>

A ModelMetaData object containing information about the model.

InitializeLayers()

Initializes the layers of the neural network based on the architecture.

protected override void InitializeLayers()

Remarks

For Beginners: This method sets up all the layers in your neural network according to the architecture you've defined. It's like assembling the parts of your network before you can use it.

Inpaint(List<Tensor<T>>, List<Tensor<T>>)

Inpaints a video sequence by filling in masked regions.

public List<Tensor<T>> Inpaint(List<Tensor<T>> frames, List<Tensor<T>> masks)

Parameters

frames List<Tensor<T>>

Input video frames.

masks List<Tensor<T>>

Binary masks indicating regions to fill (1 = fill, 0 = keep).

Returns

List<Tensor<T>>

Inpainted video frames.

Exceptions

ArgumentNullException

Thrown when frames or masks is null.

ArgumentException

Thrown when frames and masks have different counts or are empty.

Predict(Tensor<T>)

Predicts the inpainted output for a single masked frame.

public override Tensor<T> Predict(Tensor<T> input)

Parameters

input Tensor<T>

Input tensor containing frame and mask concatenated [B, C+1, H, W].

Returns

Tensor<T>

Inpainted frame.

RemoveObject(List<Tensor<T>>, List<Tensor<T>>)

Removes an object from video based on mask sequence.

public List<Tensor<T>> RemoveObject(List<Tensor<T>> frames, List<Tensor<T>> objectMasks)

Parameters

frames List<Tensor<T>>
objectMasks List<Tensor<T>>

Returns

List<Tensor<T>>

RepairVideo(List<Tensor<T>>, List<Tensor<T>>)

Repairs corrupted regions in video frames.

public List<Tensor<T>> RepairVideo(List<Tensor<T>> frames, List<Tensor<T>> corruptionMasks)

Parameters

frames List<Tensor<T>>
corruptionMasks List<Tensor<T>>

Returns

List<Tensor<T>>

SerializeNetworkSpecificData(BinaryWriter)

Serializes network-specific data that is not covered by the general serialization process.

protected override void SerializeNetworkSpecificData(BinaryWriter writer)

Parameters

writer BinaryWriter

The BinaryWriter to write the data to.

Remarks

This method is called at the end of the general serialization process to allow derived classes to write any additional data specific to their implementation.

For Beginners: Think of this as packing a special compartment in your suitcase. While the main serialization method packs the common items (layers, parameters), this method allows each specific type of neural network to pack its own unique items that other networks might not have.

Train(Tensor<T>, Tensor<T>)

Trains the model on a masked frame and its ground truth.

public override void Train(Tensor<T> input, Tensor<T> expectedOutput)

Parameters

input Tensor<T>

Input tensor containing masked frame [B, C, H, W].

expectedOutput Tensor<T>

Ground truth unmasked frame [B, C, H, W].

UpdateParameters(Vector<T>)

Updates the network's parameters with new values.

public override void UpdateParameters(Vector<T> parameters)

Parameters

parameters Vector<T>

The new parameter values to set.

Remarks

For Beginners: During training, a neural network's internal values (parameters) get adjusted to improve its performance. This method allows you to update all those values at once by providing a complete set of new parameters.

This is typically used by optimization algorithms that calculate better parameter values based on training data.

Edit this page