ChatSambaNovaCloud

This will help you getting started with SambaNovaCloud chat models. For detailed documentation of all ChatSambaNovaCloud features and configurations head to the API reference.

SambaNova's SambaNova Cloud is a platform for performing inference with open-source models

Overview

Integration details

Class	Package	Local	Serializable	JS support	Package downloads	Package latest
ChatSambaNovaCloud	langchain-community	❌	❌	❌

Model features

Tool calling	Structured output	JSON mode	Image input	Audio input	Video input	Token-level streaming	Native async	Token usage	Logprobs
✅	❌	❌	❌	❌	❌	✅	✅	✅	❌

Setup

To access ChatSambaNovaCloud models you will need to create a SambaNovaCloud account, get an API key, install the langchain_community integration package, and install the SSEClient Package.

pip install langchain-community
pip install sseclient-py

Credentials

Get an API Key from cloud.sambanova.ai and add it to your environment variables:

export SAMBANOVA_API_KEY="your-api-key-here"

import getpass
import os

if not os.getenv("SAMBANOVA_API_KEY"):
    os.environ["SAMBANOVA_API_KEY"] = getpass.getpass(
        "Enter your SambaNova Cloud API key: "
    )

If you want to get automated tracing of your model calls you can also set your LangSmith API key by uncommenting below:

# os.environ["LANGCHAIN_TRACING_V2"] = "true"
# os.environ["LANGCHAIN_API_KEY"] = getpass.getpass("Enter your LangSmith API key: ")

Installation

The LangChain SambaNovaCloud integration lives in the langchain_community package:

%pip install -qU langchain-community
%pip install -qu sseclient-py

Instantiation

Now we can instantiate our model object and generate chat completions:

from langchain_community.chat_models.sambanova import ChatSambaNovaCloud

llm = ChatSambaNovaCloud(
    model="Meta-Llama-3.1-70B-Instruct",
    max_tokens=1024,
    temperature=0.7,
    top_k=1,
    top_p=0.01,
)

API Reference:ChatSambaNovaCloud

Invocation

messages = [
    (
        "system",
        "You are a helpful assistant that translates English to French. Translate the user sentence.",
    ),
    ("human", "I love programming."),
]
ai_msg = llm.invoke(messages)
ai_msg

AIMessage(content="J'adore la programmation.", additional_kwargs={}, response_metadata={'finish_reason': 'stop', 'usage': {'acceptance_rate': 14, 'completion_tokens': 8, 'completion_tokens_after_first_per_sec': 164.48250980392157, 'completion_tokens_after_first_per_sec_first_ten': 854.0507017671442, 'completion_tokens_per_sec': 33.17909116130268, 'end_time': 1731001001.274775, 'is_last_response': True, 'prompt_tokens': 55, 'start_time': 1731001001.0004687, 'time_to_first_token': 0.2317485809326172, 'total_latency': 0.24111570630754744, 'total_tokens': 63, 'total_tokens_per_sec': 261.2853428952586}, 'model_name': 'Meta-Llama-3.1-70B-Instruct', 'system_fingerprint': 'fastcoe', 'created': 1731001000}, id='415968fc-1a58-47e0-a8a3-7cb80383980b')

print(ai_msg.content)

J'adore la programmation.

Chaining

We can chain our model with a prompt template like so:

from langchain_core.prompts import ChatPromptTemplate

prompt = ChatPromptTemplate(
    [
        (
            "system",
            "You are a helpful assistant that translates {input_language} to {output_language}.",
        ),
        ("human", "{input}"),
    ]
)

chain = prompt | llm
chain.invoke(
    {
        "input_language": "English",
        "output_language": "German",
        "input": "I love programming.",
    }
)

API Reference:ChatPromptTemplate

AIMessage(content='Ich liebe das Programmieren.', additional_kwargs={}, response_metadata={'finish_reason': 'stop', 'usage': {'acceptance_rate': 2.3333333333333335, 'completion_tokens': 6, 'completion_tokens_after_first_per_sec': 100.67263842084546, 'completion_tokens_after_first_per_sec_first_ten': 205.99550965424336, 'completion_tokens_per_sec': 36.087854367623336, 'end_time': 1731001005.4113064, 'is_last_response': True, 'prompt_tokens': 50, 'start_time': 1731001005.2245064, 'time_to_first_token': 0.13713407516479492, 'total_latency': 0.1662609236580985, 'total_tokens': 56, 'total_tokens_per_sec': 336.8199740978178}, 'model_name': 'Meta-Llama-3.1-70B-Instruct', 'system_fingerprint': 'fastcoe', 'created': 1731001005}, id='0990bec1-1721-41f9-a14c-d0bea8ac6263')

Streaming

system = "You are a helpful assistant with pirate accent."
human = "I want to learn more about this animal: {animal}"
prompt = ChatPromptTemplate.from_messages([("system", system), ("human", human)])

chain = prompt | llm

for chunk in chain.stream({"animal": "owl"}):
    print(chunk.content, end="", flush=True)

Yer lookin' fer some knowledge about owls, eh? Alright then, matey, settle yerself down with a pint o' grog and listen close.

Owls be a fascinatin' lot, with their big round eyes and silent wings. They be birds o' prey, which means they hunt other creatures fer food. There be over 220 species o' owls, rangin' in size from the tiny Elf Owl (which be smaller than a parrot) to the Great Grey Owl (which be as big as a small eagle).

Here be some o' the most interestin' facts about owls:

1. **Night vision**: Owls have eyes that be specially designed fer seein' in the dark. Their eyes be big and round, with a reflective layer called the tapetum lucidum that helps 'em see in low light.
2. **Silent flyers**: Owls have special feathers on their wings that help 'em fly silently. This be useful fer sneakin' up on prey, matey!
3. **Turnin' heads**: Owls can turn their heads up to 270 degrees, which be useful fer seein' what be goin' on around 'em without havin' to move their whole body.
4. **Huntin' skills**: Owls be skilled hunters, with sharp talons and strong wings. They can catch prey in mid-air, and some species can even catch fish right out o' the water!
5. **Monogamous**: Many owl species be monogamous, which means they mate fer life. Some species even stay together fer many years, which be a rare thing in the animal kingdom.

So there ye have it, matey! Owls be amazin' creatures, and there be much more to learn about 'em. Now go forth and spread the word about these fascinatin' birds!

Async

prompt = ChatPromptTemplate.from_messages(
    [
        (
            "human",
            "what is the capital of {country}?",
        )
    ]
)

chain = prompt | llm
await chain.ainvoke({"country": "France"})

AIMessage(content='The capital of France is Paris.', additional_kwargs={}, response_metadata={'finish_reason': 'stop', 'usage': {'acceptance_rate': 1, 'completion_tokens': 7, 'completion_tokens_after_first_per_sec': 326.8246386410566, 'completion_tokens_after_first_per_sec_first_ten': 0, 'completion_tokens_per_sec': 28.56260299906024, 'end_time': 1731001010.9238367, 'is_last_response': True, 'prompt_tokens': 42, 'start_time': 1731001010.678761, 'time_to_first_token': 0.22671723365783691, 'total_latency': 0.24507570266723633, 'total_tokens': 49, 'total_tokens_per_sec': 199.93822099342168}, 'model_name': 'Meta-Llama-3.1-70B-Instruct', 'system_fingerprint': 'fastcoe', 'created': 1731001010}, id='c17f7e9b-2db4-4c53-bf1a-980c773cf554')

Async Streaming

prompt = ChatPromptTemplate.from_messages(
    [
        (
            "human",
            "in less than {num_words} words explain me {topic} ",
        )
    ]
)
chain = prompt | llm

async for chunk in chain.astream({"num_words": 30, "topic": "quantum computers"}):
    print(chunk.content, end="", flush=True)

Quantum computers use quantum bits (qubits) to process info, leveraging superposition and entanglement to perform calculations exponentially faster than classical computers for certain complex problems.

Tool calling

SambaNovaCloud supports tool calling

from datetime import datetime

from langchain_core.messages import HumanMessage, SystemMessage, ToolMessage
from langchain_core.tools import tool


@tool
def get_time(kind: str = "both") -> str:
    """Returns current date, current time or both.
    Args:
        kind: date, time or both
    """
    if kind == "date":
        date = datetime.now().strftime("%m/%d/%Y")
        return f"Current date: {date}"
    elif kind == "time":
        time = datetime.now().strftime("%H:%M:%S")
        return f"Current time: {time}"
    else:
        date = datetime.now().strftime("%m/%d/%Y")
        time = datetime.now().strftime("%H:%M:%S")
        return f"Current date: {date}, Current time: {time}"


tools = [get_time]


def invoke_tools(tool_calls, messages):
    available_functions = {tool.name: tool for tool in tools}
    for tool_call in tool_calls:
        selected_tool = available_functions[tool_call["name"]]
        tool_output = selected_tool.invoke(tool_call["args"])
        print(f"Tool output: {tool_output}")
        messages.append(ToolMessage(tool_output, tool_call_id=tool_call["id"]))
    return messages

API Reference:HumanMessage | SystemMessage | ToolMessage | tool

llm_with_tools = llm.bind_tools(tools=tools)
messages = [
    HumanMessage(
        content="I need to schedule a meeting for two weeks from today. Can you tell me the exact date of the meeting?"
    )
]

response = llm_with_tools.invoke(messages)
while len(response.tool_calls) > 0:
    print(f"Intermediate model response: {response.tool_calls}")
    messages.append(response)
    messages = invoke_tools(response.tool_calls, messages)
response = llm_with_tools.invoke(messages)

print(f"final response: {response.content}")

Intermediate model response: [{'name': 'get_time', 'args': {'kind': 'date'}, 'id': 'call_4092d5dd21cd4eb494', 'type': 'tool_call'}]
Tool output: Current date: 11/07/2024
final response: The meeting will be exactly two weeks from today, which would be 25/07/2024.

API reference

For detailed documentation of all ChatSambaNovaCloud features and configurations head to the API reference: https://api.python.langchain.com/en/latest/chat_models/langchain_community.chat_models.sambanova.ChatSambaNovaCloud.html

Related

• Chat model conceptual guide
• Chat model how-to guides