sign_language_translator.models.language_models.mixer module

This module provides the MixerLM class, which is a language model that allows sampling from multiple language models.

Classes: - MixerLM: A language model that combines the outputs of multiple language models based on a selection strategy. It extends the abstract base class LanguageModel.

class sign_language_translator.models.language_models.mixer.MixerLM(models: List[LanguageModel], selection_probabilities: List[float] | None = None, unknown_token='<unk>', name=None, model_selection_strategy='choose')

Bases: LanguageModel

The MixerLM class is a language model that combines multiple language models using a mixing strategy. It extends the abstract base class LanguageModel.

Attributes: - models (List[LanguageModel]): List of language models to be combined. - selection_probabilities (List[float] | None): The selection probabilities for each language model.

If not provided, equal probabilities are assigned.

• unknown_token (str): The token representation used for unknown or out-of-vocabulary tokens.

• model_selection_strategy (str): The strategy for selecting the next token from the language models.

  Possible values: “choose” (selects one model and infers through it).

  “merge” (infers through all models & combines their output probabilities).

• name (str): The name of the mixer language model object (optional).

Methods: - next(self, context: Iterable) -> Tuple[Any, float]: Generates the next token based on the given context. - next_all(self, context: Iterable) -> Tuple[List[Any], List[float]]: Generates all next

tokens and their associated probabilities based on the given context.

• save(self, model_path: str) -> None: saves the mixer model as a pickle file.

• load(model_path: str) -> MixerLM: loads the mixer model from a pickle file.

• __str__(self) -> str: Returns a string representation of the MixerLM instance.

static load(model_path: str) → MixerLM

Loads a MixerLM model from the given model path.

Parameters:

model_path (str) – The path to the model file.

Returns:

The loaded MixerLM model.

Return type:

MixerLM

next(context: Iterable) → Tuple[Any, float]

Generates the next token based on the given context and also returns its probability.

Parameters:

context (Iterable) – A piece of sequence used as the context for generating the next token.

Returns:

The next token and its associated probability.

Token has the same type as the items in the context iterable.

Return type:

Tuple[Any, float]

next_all(context: Iterable) → Tuple[List[Any], List[float]]

Computes probabilities for all next tokens based on the given context and returns them both.

If model selection strategy is “choose” then selects one model and infers through it. If model_selection_strategy is “merge” then for each language model, it generates the all next tokens and probabilities. It combines the tokens and probabilities from all models to create a list of unique next tokens and their corresponding weighted probabilities.

Parameters:

context (Iterable) – A piece of sequence like the training examples.

Returns:

A tuple containing a list of unique next tokens and their corresponding probabilities.

Return type:

Tuple[List[Any], List[float]]

save(model_path: str, overwrite=False) → None

Save the model to a file.

Parameters:

• model_path (str) – The path to save the model.

• overwrite (bool, optional) – Whether to overwrite an existing file. Defaults to False.

Raises:

FileExistsError – If a file already exists at model_path and overwrite is False.