Metadata allows you to add custom tags to your LLM calls. This can be useful for grouping and filtering data by environment (i.e. dev, stage, prod), regions (i.e. us-east-1, eu-west-1) or model/device versioning.

Note: Metada works best with >50 unique key-value pairs. Metadata is not useful for storing highly unique strings (such as user id’s, randomly generated values, e.g. anything with high cardinality). Columns with high cardinality with result in perfromance issues. Contact support@lytix.co if you need to store highly unique values.

Note: There are some reserved keys you should not use:

  • env: This is used to track the environment of the LLM call. This is a key managed by Lytix and only accepts values of dev, stage or prod.

​
Use Optimodel to push metadata (see code below)

from optimodel import queryModel, ModelTypes, ModelMessage, ModelMessageContentEntry

response = await queryModel(
    model=ModelTypes.llama_3_70b_instruct,
    messages=[
        ModelMessage(
            role="system",
            content="Always respond with a JSON object",
        ),
        ModelMessage(role="user", content=[
            ModelMessageContentEntry(type="text", text="Hello how are you again?"),
        ]),
    ],
    # Just set this field
    metadata={
        "key": "value"
    }
)

​
Use the native OpenAI client to push metadata

response = client.chat.completions.create(
    model=ModelTypes.mistral_large_latest.name,
    temperature=0,
    max_tokens=1000,
    messages=[
        {"role": "system", "content": "You are a helpful assistant. Always JSON"},
        {"role": "user", "content": "How are you?"},
    ],
    extra_headers={
        # Set this field, replace {METADATA_KEY} and {METADATA_VALUE} with your metadata
        "lytix-metadata-key-{METADATA_KEY}": "{METADATA_VALUE}"
    }
)