Evals

Create and manage model evaluation jobs.

Create

Create a new model evaluation job.

tg evals create

Parameters

Flag	Description
--type [classify|score|compare]	Type of evaluation to create. required
--judge-model [string]	Name or URL of the judge model to use for evaluation. required
--judge-model-source [serverless|dedicated|external]	Source of the judge model. required
--judge-external-api-token [string]	Optional external API token for the judge model.
--judge-external-base-url [string]	Optional external base URLs for the judge model.
--judge-system-template [string]	System template for the judge model. required
--input-data-file-path [string]	Path to the input data file. required
--model-field [string]	Name of the field in the input file containing text generated by the model. Cannot be used when model-a-name and other model config parameters are specified.
--model-to-evaluate [string]	Model name when using the detailed config
--model-to-evaluate-source [serverless|dedicated|external]	Source of the model to evaluate.
--model-to-evaluate-external-api-token [string]	Optional external API token for the model to evaluate.
--model-to-evaluate-external-base-url [string]	Optional external base URL for the model to evaluate.
--model-to-evaluate-max-tokens [integer]	Max tokens for model-to-evaluate
--model-to-evaluate-temperature [float]	Temperature for model-to-evaluate
--model-to-evaluate-system-template [string]	System template for model-to-evaluate
--model-to-evaluate-input-template [string]	Input template for model-to-evaluate
--labels [string]	Classification labels - comma-separated list
--pass-labels [string]	Labels considered as passing (required for classify type). A comma-separated list.
--min-score [float]	Minimum score value (required for score type).
--max-score [float]	Maximum score value (required for score type).
--pass-threshold [float]	Threshold score for passing (required for score type).
--model-a-field [string]	Name of the field in the input file containing text generated by Model A. Can not be used when model-a-name and other model config parameters are specified
--model-a [string]	Model name or URL for model A when using detailed config.
--model-a-source [serverless|dedicated|external]	Source of model A.
--model-a-external-api-token [string]	Optional external API token for model A.
--model-a-external-base-url [string]	Optional external base URL for model A.
--model-a-max-tokens [integer]	Max tokens for model A.
--model-a-temperature [float]	Temperature for model A.
--model-a-system-template [string]	System template for model A.
--model-a-input-template [string]	Input template for model A.
--model-b-field [string]	Name of the field in the input file containing text generated by Model B. Can not be used when model-b-name and other model config parameters are specified
--model-b [string]	Model name or URL for model B when using detailed config.
--model-b-source [serverless|dedicated|external]	Source of model B.
--model-b-external-api-token [string]	Optional external API token for model B.
--model-b-external-base-url [string]	Optional external base URL for model B.
--model-b-max-tokens [integer]	Max tokens for model B.
--model-b-temperature [float]	Temperature for model B.
--model-b-system-template [string]	System template for model B.
--model-b-input-template [string]	Input template for model B.

List

List all eval jobs.

tg evals list

Parameters

Flag	Description
--status [pending|queued|running|completed|error|user_error]	Filter by job status.
--limit [integer]	Limit number of results (max 100).

Retrieve

Get the details for a specific evaluation job.

tg evals retrieve [EVALUATION_ID]

Status

Get the status and results of a specific evaluation job

tg evals status [EVALUATION_ID]