Fashionable agentic functions hardly ever speak to a single mannequin or a single device, so how do you retain that stack maintainable when suppliers, fashions and instruments hold altering each few weeks. Moonshot AI’s Kosong targets this drawback as an LLM abstraction layer for agent functions. Kosong unifies message constructions, asynchronous device orchestration and pluggable chat suppliers so groups can construct brokers with out arduous wiring enterprise logic to a single API. It’s also the layer that powers Moonshot’s Kimi CLI.
What Kosong offers?
Kosong is a Python library that sits between your agent logic and LLM suppliers. It as an LLM abstraction layer for contemporary agent functions and reveals instance code that makes use of a Kimi chat supplier along with excessive stage helper features generate and step.
The general public API floor is deliberately stored small. On the high stage you import kosong.generate, kosong.step and the outcome sorts GenerateResult and StepResult. Supporting modules outline chat_provider, message, tooling, and tooling.easy. These modules wrap supplier particular streaming codecs, token accounting and gear calls behind one constant interface.
ChatProvider and message mannequin
The core integration level is the ChatProvider abstraction. Moonshot staff reveals a supplier implementation for Kimi in kosong.chat_provider.kimi. A Kimi object is initialized with base_url, api_key and the mannequin identify, for instance kimi-k2-turbo-preview. This supplier is then handed into kosong.generate or kosong.step along with a system immediate, instruments and a message historical past.
Messages are represented by the Message class from kosong.message. Within the examples, a message is constructed with a job, similar to "person", and a content material argument. The kind of content material is documented as both a string or a listing of content material elements, which lets the library assist richer multimodal payloads whereas protecting the fundamental chat instance easy for brand spanking new customers.
Kosong additionally exposes a streaming unit StreamedMessagePart by way of kosong.chat_provider. Supplier implementations emit these elements throughout era, and the library merges them into the ultimate Message. The non-compulsory TokenUsage construction tracks token counts in a supplier unbiased approach, which is then connected to the outcome objects for logging and monitoring.
Tooling, Toolset and SimpleToolset
Most agent stacks want instruments similar to search, code execution or database calls. Kosong fashions this via the tooling module. The instance within the GitHub repo defines a device by subclassing CallableTool2 with a Pydantic parameter mannequin. The instance AddTool units identify, description and params, and implements __call__ to return a ToolOk worth which is a legitimate ToolReturnType.
Instruments are registered in a SimpleToolset from kosong.tooling.easy. Within the instance, a SimpleToolset is instantiated after which augmented with the AddTool occasion utilizing the += operator. This toolset is handed into kosong.step, not into generate. The toolset is answerable for resolving device calls from the mannequin and routing them to the proper async perform, whereas step manages the orchestration round a single conversational flip.
generate for single shot completion
The generate perform is the entry level for plain chat completion. You present the chat_provider, a system_prompt, an express checklist of instruments, which might be empty, and a historical past of Message objects. The Kimi instance reveals a minimal utilization sample the place a single person message is handed as historical past and instruments=[].
generate helps streaming via an on_message_part callback. Within the GitHub repo, the analysis staff illustrates this by defining a easy output perform that prints every StreamedMessagePart. After streaming is full, generate returns a GenerateResult that accommodates the merged assistant message and an non-compulsory utilization construction with token counts. This sample lets functions each show incremental output and nonetheless work with a clear remaining message object.
step for device utilizing brokers
For device utilizing brokers, Kosong exposes the step perform. The instance within the Git Repo reveals kosong.step being referred to as with a Kimi supplier, a SimpleToolset that accommodates AddTool, a system immediate and person historical past that instructs the mannequin to name the add device.
step returns a StepResult. The instance prints outcome.message after which awaits outcome.tool_results(). This methodology collects all device outputs produced in the course of the step and returns them to the caller. The orchestration of device calls, together with argument parsing into the Pydantic parameter mannequin and conversion into ToolReturnType outcomes, is dealt with inside Kosong so agent authors would not have to implement their very own dispatch loop for every supplier.
In-built demo and relationship with Kimi CLI
Kosong ships with a inbuilt demo agent that may be run regionally. The Git README paperwork atmosphere variables KIMI_BASE_URL and KIMI_API_KEY, and reveals a launch command utilizing uv run python -m kosong kimi --with-bash. This demo makes use of Kimi because the chat supplier and exposes a terminal agent that may name instruments, together with shell instructions when the choice with bash is enabled.
Key Takeaways
- Kosong is an LLM abstraction layer from Moonshot AI that unifies message constructions, asynchronous device orchestration and pluggable chat suppliers for agent functions.
- The library exposes a small core API,
generatefor plain chat andstepfor device utilizing brokers, backed by abstractions similar toChatProvider,Message,Device,ToolsetandSimpleToolset. - Kosong at present ships a
Kimichat supplier focusing on the Moonshot AI API, and defines theChatProviderinterface so groups can plug in extra backends with out altering agent logic. - Device definitions use Pydantic parameter fashions and
ToolReturnTypeoutcomes, which lets Kosong deal with argument parsing, validation and orchestration of device calls insidestep. - Kosong powers Moonshot’s Kimi CLI, offering the underlying LLM abstraction layer whereas Kimi CLI focuses on the command line agent expertise that may goal Kimi and different backends.
Kosong appears to be like like a practical transfer from Moonshot AI, it cleanly separates agent logic from LLM and gear backends whereas protecting the floor space small for early builders. By centering every thing on ChatProvider, Message and Toolset, it offers Kimi CLI and different stacks a constant strategy to evolve fashions and tooling with out rewriting orchestration. For groups constructing long run agent methods, Kosong could possibly be the proper of minimal infrastructure.
Take a look at the Repo and Docs. Be at liberty to take a look at our GitHub Web page for Tutorials, Codes and Notebooks. Additionally, be at liberty to comply with us on Twitter and don’t neglect to affix our 100k+ ML SubReddit and Subscribe to our E-newsletter. Wait! are you on telegram? now you may be part of us on telegram as properly.
Michal Sutter is a knowledge science skilled with a Grasp of Science in Knowledge Science from the College of Padova. With a strong basis in statistical evaluation, machine studying, and information engineering, Michal excels at reworking complicated datasets into actionable insights.
