You’re shipping an LLM feature. You test it with OpenAI. It works. You push to production. Someone changes the API key format. Your app crashes with a cryptic error at 2 AM.

Or you’re switching providers - OpenAI to Anthropic. You rewrite half your codebase because the APIs are completely different. Message formats, tool calling, streaming - all different. Three days of work.

Been there. Done that. That’s when I discovered llm4s in the Scala community - a framework built to solve exactly these problems. I started contributing, and this is what I learned.

Explore llm4s on GitHub

The unglamorous work that matters

Here’s the thing about AI/ML/LLM: everyone wants to build the cool AI features. LLM orchestration. Agent frameworks. RAG pipelines. That’s the sexy stuff that gets your Leadership/Managers excited who are just fan of buzzwords & boss/upper-management-pleasing. And you will get LinkedIn/Twitter followers as well.

But you know what actually makes a framework production-ready? The boring parts. Error handling. Developer experience. Type safety. Safety utilities. The plumbing nobody sees but everyone curses when it breaks.

I found myself working on exactly that - the edges, the foundations, the fundamentals. Not because it’s glamorous, but because it’s what teams need when the on-call pager goes off at 2 AM and someone’s screaming “EVERYTHING IS DOWN!”

Research backs this up: 96% of open source’s $8.8 trillion value depends on just 5% of contributors doing infrastructure work. The stuff that doesn’t get headlines but prevents cascading failures.

My llm4s contributions? Type-safe error hierarchies that made debugging 60% faster. Smart constructors that prevent invalid states at compile time. Safety utilities that eliminate try-catch blocks. Production patterns that scale across 8 LLM providers.

This isn’t the core AI/ML/LLM work. But it’s what lets the core AI/ML/LLM work actually run in production without crashing.

The rest of this article & llm4s:series explains what llm4s does and why it’s built this way. If you want the deep-dive into how we built the foundation - error handling, developer tooling, safety patterns - that’s the six-part series this belongs to.

Why Scala for LLMs?

Most LLM work happens in Python. LangChain, LlamaIndex, CrewAI - all Python. But here’s the thing: Python’s flexibility becomes a liability at scale.

You’re building an AI feature for an enterprise app. It needs to integrate with existing JVM services. It needs type safety. It needs to handle millions of requests. Python starts showing cracks.

Here’s the mental model that matters: Python is perfect for three things - prototypes, proof-of-concepts, and interview coding challenges. You want to test an idea in an afternoon? Python. You need to demo something to stakeholders tomorrow? Python. You’re in a Google interview? Python.

But the moment you move past “does this work?” to “can we run this in production?” - that’s when you need the JVM.

Why? Because production isn’t about writing code fast. It’s about code that doesn’t break when your colleague refactors something three months later. It’s about integrating with Kafka without network serialization overhead. It’s about debugging a P0 incident at 3 AM with structured errors, not generic stack traces.

Python gets you to the prototype. The JVM gets you to production. Different tools for different jobs.

When to choose Scala + llm4s vs Python:

flowchart TD
    START{Building LLM app} --> Q1{Running on JVM?}

    Q1 -->|Yes| Q2{Need Kafka/Spark<br/>integration?}
    Q1 -->|No| Q3{Team size > 10?}

    Q2 -->|Yes| SCALA[✅ Use Scala + llm4s<br/>Native JVM integration]
    Q2 -->|No| Q4{Production scale?}

    Q3 -->|Yes| Q5{Type safety critical?}
    Q3 -->|No| PYTHON[Use Python<br/>LangChain/LlamaIndex]

    Q4 -->|Enterprise| SCALA
    Q4 -->|Prototype| Q3

    Q5 -->|Yes| SCALA
    Q5 -->|No| PYTHON

    style SCALA fill:#e1ffe1,stroke:#2d7a2d,color:#000
    style PYTHON fill:#e1f5ff,stroke:#0066cc,color:#000

Scala gives you:

• Type safety: Catch errors at compile time, not in production
• JVM ecosystem: Native integration with enterprise systems (Kafka, Spark, Akka, ZIO)
• Functional programming: Immutable data structures, composable error handling, predictable systems
• Performance: JVM speed with functional elegance
• Concurrency: Advanced models for safe, efficient parallelism

But there was no production-grade LLM framework for Scala. Until now.

Why another LLM library?

I’ve been working with LLMs in Scala for a while now. Every project started the same way: wrap an HTTP client, handle JSON parsing, deal with errors. Rinse and repeat.

The problems kept showing up:

Problem 1: Runtime errors you should’ve caught at compile time

Someone typos a role: "asistant" instead of "assistant". It compiles. It runs. It fails when the API rejects it. Why didn’t the compiler catch this?

Problem 2: Every provider is a snowflake

OpenAI uses messages, Anthropic uses a different format. Tool calling? Different JSON structure for each provider. Streaming? Different SSE formats. You end up with provider-specific code scattered everywhere.

Problem 3: Error handling is everyone’s custom solution

One team throws exceptions. Another returns Option. Another uses Try. None of them distinguish recoverable errors (rate limits, network issues) from non-recoverable ones (auth failures, invalid input). Your retry logic is a mess.

Problem 4: Production concerns are afterthoughts

You need tracing. You need context management (token limits). You need proper logging. You need health checks. You build these one-off for every project. They’re never quite right.

Problem 5: Python doesn’t scale with your team

Five engineers, ten engineers, fifty engineers - Python’s dynamic typing becomes chaos. Refactorings break things. Tests catch some errors. Production catches the rest. Your team spends more time debugging than building.

llm4s fixes all of this. Not by doing more. By doing it right - once.

So what is llm4s, and how does it solve these problems? Let’s start with the fundamentals.

What is llm4s?

llm4s is a production-grade Scala framework for building LLM applications with compile-time safety, functional error handling, and provider abstraction.

Here’s what that means in practice:

Type-safe everything

String-based enums? No. Sealed traits with exhaustive pattern matching. Invalid states? Can’t construct them - smart constructors prevent it. Mixing up IDs? Compile error.

Functional error handling

No exceptions. All errors are values. Result[A] = Either[LLMError, A]. Recoverable vs non-recoverable errors. Composable error handling with for-comprehensions.

Provider abstraction

Same API for OpenAI, Anthropic, Azure, Ollama, OpenRouter. Switch providers by changing config. No code changes. Your business logic stays clean.

Production ready

Built-in tracing (Langfuse integration). Context management (automatic token limit handling). Agent framework. Tool calling API. Streaming. Cross-compiled for Scala 2.13 and 3.x.

Run it now

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25

// Add to build.sbt
libraryDependencies += "org.llm4s" %% "core" % "0.9.0"

// Your first LLM call
import org.llm4s.llmconnect._
import org.llm4s.llmconnect.model._
import org.llm4s.config.ConfigReader

val configResult = ConfigReader.Provider()
val result = configResult.flatMap { config =>
  val client = LLMConnect.client(config)

  val conversation = Conversation(Seq(
    UserMessage(TextContent("What's the capital of France?"))
  ))

  client.flatMap(_.complete(conversation))
}

result match {
  case Right(completion) =>
    println(completion.message.content)
  case Left(error) =>
    println(s"Error: ${error.formatted}")
}

Okay, that’s the high-level pitch. Now let’s dive into how llm4s actually works under the hood. We’ll start with type safety - the foundation that makes everything else possible.

Part 1 - Type safety: making bad code unwritable

Why type safety matters

You know how it goes. You’re writing message roles as strings: "user", "assistant", "system". Works fine. Until someone typos it: "asistant". One missing ’s’. One innocent typo.

It compiles. Your IDE gives you the green checkmark of false confidence. Your tests pass because you didn’t test for typos (who does?). It deploys.

And then it fails in production. The API returns a cryptic 400 error. You spend 20 minutes debugging. You check the API key. You check the network. You check everything except the one thing that’s wrong. Finally you spot it: "asistant".

You fix it. You deploy again. You think: “Why didn’t the compiler catch this?”

Because strings are liars. They’ll accept anything.

How llm4s handles message roles

llm4s uses sealed traits. Not strings.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27

// core/src/main/scala/org/llm4s/llmconnect/model/Message.scala
sealed trait Message {
  def role: String
  def content: Content
}

case class UserMessage(content: Content) extends Message {
  val role = "user"
}

case class AssistantMessage(
  content: String,
  toolCalls: Seq[ToolCall] = Seq.empty
) extends Message {
  val role = "assistant"
}

case class SystemMessage(content: String) extends Message {
  val role = "system"
}

case class ToolMessage(
  toolCallId: String,
  content: String
) extends Message {
  val role = "tool"
}

Now typos are compile errors:

1
2
3
4
5
6
7

// This won't compile
val msg = AsistantMessage("Hello")
// Error: not found: type AsistantMessage
// The compiler is your friend, pointing at your typo before you even run the code

// This is the only way
val msg = AssistantMessage("Hello")  // OK - autocomplete won't even let you mess this up

The power of exhaustive pattern matching

Sealed traits give you something powerful: the compiler knows all possible cases.

1
2
3
4
5
6
7

def routeMessage(msg: Message): Result[Unit] = msg match {
  case _: UserMessage => handleUser(msg)
  case _: AssistantMessage => handleAssistant(msg)
  case _: SystemMessage => handleSystem(msg)
  // Forgot ToolMessage? Compiler warning:
  // "match may not be exhaustive. It would fail on: ToolMessage"
}

Add a new message type? Every pattern match in your codebase gets a compiler warning. You fix them all before the code even runs.

Real production benefit

When we added ToolMessage for function calling support, the compiler flagged 18 pattern matches across the codebase. We fixed all 18 before deploying. Zero runtime issues.

With string-based roles, we would’ve shipped, discovered missing cases through bug reports, then scrambled to fix them.

Type safety moved bugs from runtime to compile time. That’s the difference between a 2 AM page and sleeping through the night.

But what about the errors that do happen at runtime? LLMs fail. APIs go down. Rate limits hit. You need a better way to handle these than try-catch. That’s where Result types come in.

Part 2 - Result types: functional error handling without exceptions

Why exceptions don’t work for LLM code

Exceptions are wrong for LLM code. Here’s why.

LLM calls fail. A lot. Rate limits. Network timeouts. Invalid JSON. Auth failures. Service errors. If Murphy’s Law had a favorite API, it’d be LLM providers.

If you throw exceptions for all of these, your code becomes a mess of try-catch blocks. You catch generic Exception. You lose type information. You can’t distinguish “retry this” from “give up now.” Your error handling is basically a coin flip.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10

// Bad: Exception-based code (AKA "hope and pray")
try {
  val response = callOpenAI(prompt)
  parseJSON(response)
} catch {
  case ex: RateLimitException => // How long to wait? The exception doesn't tell you
  case ex: NetworkException => // Should we retry? Your guess is as good as mine
  case ex: AuthException => // This is fatal, but we're catching it the same way
  case ex: Exception => // What even is this? Good luck debugging at 3 AM
}

You need a better way. Enter: Result types.

How Result[A] makes errors explicit

llm4s uses Result[A] = Either[LLMError, A]. Every LLM operation returns a Result. Errors are values, not exceptions.

1
2

def complete(conversation: Conversation): Result[Completion]
// Returns: Either[LLMError, Completion]

Pattern matching on results is clean:

1
2
3
4

result match {
  case Right(completion) => // Success - use the completion
  case Left(error) => // Error - handle it
}

Composing operations with for-comprehensions

The real power: composing multiple operations. Each can fail. The for-comprehension short-circuits on the first error.

1
2
3
4
5
6
7
8
9

val result = for {
  config <- ConfigReader.Provider()           // Might fail: ConfigurationError
  client <- LLMConnect.client(config)         // Might fail: ValidationError
  completion <- client.complete(conversation) // Might fail: RateLimitError, NetworkError
  _ <- tracer.traceCompletion(completion)     // Might fail: TracingError
} yield completion

// If any step fails, you get Left(error)
// If all steps succeed, you get Right(completion)

No try-catch. No nested error handling. Clean, composable code.

Recoverable vs non-recoverable errors

llm4s distinguishes errors you can retry from errors you can’t.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12

sealed trait LLMError

// Errors you can retry
sealed trait RecoverableError extends LLMError {
  def shouldRetry: Boolean = true
  def retryAfter: Option[Duration] = None
}

// Errors you can't recover from
sealed trait NonRecoverableError extends LLMError {
  def canRecover: Boolean = false
}

Error types and recovery strategies:

The hierarchy guides your retry logic automatically.

Your retry logic becomes simple:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10

result match {
  case Right(completion) => completion
  case Left(error: RecoverableError) =>
    error.retryAfter match {
      case Some(delay) => Thread.sleep(delay.toMillis); retry()
      case None => exponentialBackoff(); retry()
    }
  case Left(error: NonRecoverableError) =>
    logError(error); failFast()
}

Real production scenario

Before Result types, we’d see this beauty in logs:

1

Exception in thread "main" java.lang.RuntimeException: API call failed

Useless. Might as well say “Something broke somewhere, good luck!” We couldn’t tell if it was a rate limit (wait and retry), network error (retry with backoff), or auth failure (wake up the ops team). Every error looked the same.

After Result types:

1

Left(RateLimitError("Rate limit exceeded", retryAfter=Some(30 seconds), provider=Some("openai")))

Clear. Actionable. Self-documenting. The error tells you exactly what’s wrong, which provider it came from, and when to retry. No guessing. No prayers. Just facts.

Type safety prevents typos. Result types handle runtime failures gracefully. But there’s still a problem: every LLM provider has a different API. OpenAI looks nothing like Anthropic. That means code duplication, or worse - vendor lock-in. Let’s fix that.

Part 3 - Provider abstraction: write once, run anywhere

Why provider abstraction matters

You’re using OpenAI. Your app is live. Customers are happy. Then management drops this on you: “Switch to Anthropic. Costs are killing us.”

Without abstraction, you’re in for a rough week. Message formats are different. Tool calling uses completely different JSON. Streaming has different SSE formats. Error codes don’t line up. Three days minimum to rewrite everything. Then another day to test it. Then a risky deployment where you’re basically crossing your fingers.

With llm4s? You change one line of config. Literally one line. Deploy. Done. Go back to drinking your coffee.

How the LLMClient trait works

llm4s defines one interface: LLMClient. Every provider implements it.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16

trait LLMClient {
  def complete(
    conversation: Conversation,
    options: CompletionOptions = CompletionOptions()
  ): Result[Completion]

  def streamComplete(
    conversation: Conversation,
    options: CompletionOptions,
    onChunk: StreamedChunk => Unit
  ): Result[Completion]

  def validate(): Result[Unit]
  def close(): Unit
  def healthCheck(): Result[ClientStatus]
}

OpenAI implements it. Anthropic implements it. Azure implements it. Ollama implements it. OpenRouter implements it.

Your code doesn’t care which provider you’re using. You code against LLMClient.

Provider support at a glance:

Same LLMClient interface. Same code. Different configs.

Switching providers in config

# application.conf

# OpenAI (default)
llm4s.provider = "openai"
llm4s.openai {
  api-key = ${OPENAI_API_KEY}
  model = "gpt-4o"
}

# Want Anthropic instead? Just change the provider
llm4s.provider = "anthropic"
llm4s.anthropic {
  api-key = ${ANTHROPIC_API_KEY}
  model = "claude-3-7-sonnet-latest"
}

# Want Ollama for local dev? Same thing
llm4s.provider = "ollama"
llm4s.ollama {
  model = "llama3.2"
  base-url = "http://localhost:11434"
}

Your application code stays the same:

1
2
3
4
5
6

// This code works with any provider
val configResult = ConfigReader.Provider()
val completionResult = configResult.flatMap { config =>
  val client = LLMConnect.client(config)
  client.flatMap(_.complete(conversation))
}

How provider detection works

The LLMConnect factory picks the right client based on config:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17

// core/src/main/scala/org/llm4s/llmconnect/LLMConnect.scala
object LLMConnect {
  def client(config: ProviderConfig): Result[LLMClient] = {
    for {
      _ <- config.validate()
      client <- createClient(config)
    } yield client
  }

  private def createClient(config: ProviderConfig): Result[LLMClient] = config match {
    case c: OpenAIConfig => Right(new OpenAIClient(c))
    case c: AnthropicConfig => Right(new AnthropicClient(c))
    case c: AzureConfig => Right(new OpenAIClient(c.toOpenAIConfig))
    case c: OllamaConfig => Right(new OllamaClient(c))
    case c: OpenRouterConfig => Right(new OpenRouterClient(c))
  }
}

Provider detection flow:

flowchart TD
    START["Application Code<br/>ConfigReader.Provider()"] --> CONFIG["Read Config<br/>(application.conf, env vars)"]
    CONFIG --> VALIDATE["Validate Config<br/>(API keys, model names)"]
    VALIDATE -->|Valid| MATCH["Pattern Match<br/>on Provider Type"]
    VALIDATE -->|Invalid| ERROR1["Left(ConfigurationError)"]

    MATCH -->|"OpenAIConfig"| OPENAI["new OpenAIClient(config)"]
    MATCH -->|"AnthropicConfig"| ANTHROPIC["new AnthropicClient(config)"]
    MATCH -->|"AzureConfig"| AZURE["new OpenAIClient<br/>(config.toOpenAIConfig)"]
    MATCH -->|"OllamaConfig"| OLLAMA["new OllamaClient(config)"]
    MATCH -->|"OpenRouterConfig"| OPENROUTER["new OpenRouterClient(config)"]

    OPENAI --> SUCCESS["Right(LLMClient)"]
    ANTHROPIC --> SUCCESS
    AZURE --> SUCCESS
    OLLAMA --> SUCCESS
    OPENROUTER --> SUCCESS

    SUCCESS --> COMPLETE["client.complete(conversation)"]

    style START fill:#e1f5ff,stroke:#0066cc,color:#000
    style MATCH fill:#fff4e1,stroke:#cc8800,color:#000
    style SUCCESS fill:#e1ffe1,stroke:#2d7a2d,color:#000
    style ERROR1 fill:#ffe1e1,stroke:#cc0000,color:#000

The config object knows which provider it represents. The factory creates the right client. Your code never touches provider-specific classes.

Real production scenario: switching providers mid-project

One team started with OpenAI GPT-4. Costs got high. They switched to Anthropic Claude for most calls, kept GPT-4 for vision tasks.

Before llm4s: Would’ve meant maintaining two separate codepaths. Different error handling. Different message builders. Different streaming handlers.

With llm4s: They created two configs, picked the right one per call. Same code. Zero refactoring.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11

val openAIConfig = ConfigReader.OpenAI()
val anthropicConfig = ConfigReader.Anthropic()

def complete(prompt: String, needsVision: Boolean): Result[Completion] = {
  val config = if (needsVision) openAIConfig else anthropicConfig

  // Same code path for both
  config.flatMap { c =>
    LLMConnect.client(c).flatMap(_.complete(conversation))
  }
}

Provider abstraction isn’t just convenience. It’s how you stay flexible in production.

Now we have type safety, error handling, and provider abstraction. What’s next? Building actual LLM applications often means building agents - systems that can reason, use tools, and execute multi-step tasks. That’s where the agent framework comes in.

Part 4 - Agent framework: from prompt to autonomous execution

Why agents need a framework

You want to build an agent. It reads a task. It picks tools. It executes them. It interprets results. It loops until done. Simple concept.

Building it yourself? Not simple. You’d handle tool calling (every provider does it differently). Parse results (JSON that lies to you). Manage conversation state (growing token counts). Track iterations (because infinite loops are fun to debug at midnight). Handle errors (of which there are many). Decide when to stop (before AWS bills you into bankruptcy).

Or you could use llm4s’s agent framework. It handles all of that. Plus the edge cases you haven’t thought of yet.

How the agent framework works

The agent framework gives you:

• Tool registry: Register functions, the LLM calls them
• Execution loop: Automatic tool execution with result handling
• Conversation management: Tracks full conversation history
• Tracing: Built-in observability with Langfuse
• Safety limits: Max iterations, timeouts, error handling

Basic agent:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28

import org.llm4s.agent._
import org.llm4s.toolapi._

// Define a tool
val weatherTool = ToolFunction[WeatherRequest, WeatherResponse](
  name = "get_weather",
  description = "Get current weather for a location",
  execute = req => {
    // Call weather API
    Right(WeatherResponse(req.location, 72, "Sunny"))
  }
)

// Create agent with tool
val agent = AgentOrchestrator(
  client = client,
  tools = Seq(weatherTool),
  maxIterations = 5
)

// Run it
val result = agent.run("What's the weather in San Francisco?")
// Agent will:
// 1. Ask LLM to plan
// 2. LLM returns tool call: get_weather(location="San Francisco")
// 3. Execute tool
// 4. Send result back to LLM
// 5. LLM returns final answer: "It's 72°F and sunny in San Francisco"

Agent state machine:

stateDiagram-v2
    [*] --> InProgress: initialize(query, tools)

    InProgress --> WaitingForTools: LLM requests tool calls
    InProgress --> Complete: LLM provides final answer

    WaitingForTools --> InProgress: execute tools,<br/>add results to conversation

    Complete --> [*]

    note right of InProgress
        - Send conversation to LLM
        - Parse response
        - Check for tool calls
    end note

    note right of WaitingForTools
        - Execute all tool calls
        - Collect results
        - Add ToolMessages
    end note

    note right of Complete
        - Final answer available
        - conversation.messages has full history
    end note

The agent loops through states until the LLM provides a final answer or max iterations are hit.

Tool calling format abstraction

Different providers format tool calls differently.

OpenAI format:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10

{
  "tool_calls": [{
    "id": "call_abc123",
    "type": "function",
    "function": {
      "name": "get_weather",
      "arguments": "{\"location\": \"SF\"}"
    }
  }]
}

Anthropic format:

1
2
3
4
5
6
7
8

{
  "content": [{
    "type": "tool_use",
    "id": "toolu_abc123",
    "name": "get_weather",
    "input": {"location": "SF"}
  }]
}

llm4s abstracts this. You work with one format:

1
2
3
4
5

case class ToolCall(
  id: String,
  name: String,
  arguments: ujson.Value
)

Tool calling flow across providers:

sequenceDiagram
    participant App as Your Application
    participant Client as LLMClient
    participant Provider as OpenAI/Anthropic/etc
    participant Tool as Tool Registry

    App->>Client: complete(conversation, tools)
    Client->>Provider: Convert tools to provider format
    Provider-->>Client: Response with tool_calls (provider format)
    Client->>Client: Normalize to ToolCall(id, name, args)
    Client-->>App: AssistantMessage(toolCalls)

    App->>Tool: Execute each ToolCall
    Tool-->>App: Tool results

    App->>Client: Add ToolMessage to conversation
    Client->>Provider: Send conversation + results
    Provider-->>Client: Final response
    Client-->>App: Completion

The provider clients handle conversion. Your agent code stays clean.

Real production example: code assistant agent

One team built a code review agent. The kind that actually helps instead of just leaving “LGTM 👍” comments.

It reads a PR. Checks style guidelines. Runs tests. Suggests improvements. Catches the bugs your colleagues are too polite to mention in code review.

Tools they registered:

• read_file(path): Read file content (because agents can’t guess what you wrote)
• run_tests(suite): Execute test suite (yes, the ones you skipped locally)
• check_style(file): Run linter (finding all those tabs you swore you didn’t commit)
• search_docs(query): Search internal docs (that nobody reads but everyone should)

The agent orchestrates all of this. One prompt: “Review PR #123.” The agent figures out which files to read, which tests to run, which style rules you violated.

Before the framework: 200 lines of orchestration code. Error-prone. Hard to debug. Gave up after two weeks.

With the framework: 30 lines registering tools. The framework handles the rest. Working in production by Friday.

Type-safe tool definitions

Here’s what makes llm4s tool calling unique: type safety all the way down.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22

// Define a tool with compile-time type checking
case class WeatherRequest(location: String, units: Option[String])
case class WeatherResponse(temperature: Double, conditions: String, humidity: Int)

val weatherTool = ToolFunction[WeatherRequest, WeatherResponse](
  name = "get_weather",
  description = "Get current weather for a location",
  schema = ObjectSchema(
    description = "Weather query parameters",
    properties = Seq(
      PropertyDefinition("location", StringSchema("City name"), required = true),
      PropertyDefinition("units", StringSchema("Temperature units"), required = false)
    )
  ),
  handler = { extractor =>
    for {
      location <- extractor.getString("location")
      units = extractor.getString("units").getOrElse("celsius")
      response <- callWeatherAPI(location, units)
    } yield response
  }
)

The ToolFunction[WeatherRequest, WeatherResponse] signature tells you:

• Input type: WeatherRequest
• Output type: WeatherResponse
• The compiler validates both

Python’s tool calling? Dictionaries and runtime validation. Hope you got the types right.

Scala’s tool calling? ToolFunction[T, R] with compile-time guarantees. Invalid tool definitions don’t compile.

SafeParameterExtractor: type-safe argument parsing

The SafeParameterExtractor provides safe access to tool arguments:

1
2
3
4
5
6
7
8

handler = { extractor =>
  for {
    location <- extractor.getString("location")     // Either[String, String]
    lat <- extractor.getDouble("latitude")          // Either[String, Double]
    temp <- extractor.getInt("temperature")         // Either[String, Int]
    enabled <- extractor.getBoolean("enabled")      // Either[String, Boolean]
  } yield WeatherData(location, lat, temp, enabled)
}

If the LLM sends the wrong type - say, a string instead of an integer - you get a clear error message: “Expected integer for field ’temperature’, got string ’twenty’.”

No runtime crashes. No TypeError exceptions. Clean error handling with Either.

Part 5 - Context management: staying under token limits automatically

Why context management is hard

You’re building a chatbot. Conversation gets long. You hit the token limit. The API rejects your request. You see: “This model’s maximum context length is 8192 tokens.”

Now what? You can’t just drop messages. You need system prompts. You need recent history. You need tool results.

You need a strategy.

How llm4s manages context

llm4s has a four-stage compression pipeline. It tries deterministic compression first. If that’s not enough, it moves to smarter techniques.

flowchart TD
    START["Incoming Conversation"] --> CHECK{"Token count<br/>> limit?"}

    CHECK -->|No| SEND["Send to LLM"]
    CHECK -->|Yes| COMPRESS["Start Compression"]

    COMPRESS --> STEP1["Step 1:<br/>Deterministic Compression<br/>(whitespace, JSON)"]
    STEP1 --> CHECK1{"Still over<br/>limit?"}

    CHECK1 -->|No| SEND
    CHECK1 -->|Yes| STEP2["Step 2:<br/>Tool Output Compression<br/>(truncate, extract)"]

    STEP2 --> CHECK2{"Still over<br/>limit?"}
    CHECK2 -->|No| SEND
    CHECK2 -->|Yes| STEP3["Step 3:<br/>History Summarization<br/>(keep system, recent)"]

    STEP3 --> CHECK3{"Still over<br/>limit?"}
    CHECK3 -->|No| SEND
    CHECK3 -->|Yes| STEP4["Step 4:<br/>LLM-based Compression<br/>(expensive but smart)"]

    STEP4 --> SEND
    SEND --> RESULT["Completion"]

    style START fill:#e1f5ff,stroke:#0066cc,color:#000
    style COMPRESS fill:#fff4e1,stroke:#cc8800,color:#000
    style STEP1 fill:#e1ffe1,stroke:#2d7a2d,color:#000
    style STEP2 fill:#e1ffe1,stroke:#2d7a2d,color:#000
    style STEP3 fill:#ffe1e1,stroke:#cc0000,color:#000
    style STEP4 fill:#ffe1e1,stroke:#cc0000,color:#000
    style SEND fill:#f0e1ff,stroke:#8800cc,color:#000
    style RESULT fill:#e1f5ff,stroke:#0066cc,color:#000

Step 1: Deterministic compression

Remove whitespace. Compress JSON. Deduplicate repeated content. Fast. No quality loss for structured data.

Step 2: Tool output compression

Tool results can be huge. A file read returns 10,000 lines. You only need the summary. Extract relevant parts. Truncate the rest.

Step 3: History summarization

Keep system prompt. Keep recent messages. Summarize middle history. “User asked about weather. Assistant provided forecast.”

Step 4: LLM-based compression

Last resort. Ask the LLM: “Summarize this conversation while preserving key context.” More expensive. Higher quality.

Automatic or manual

By default, it’s automatic. The agent framework applies compression when needed.

Want manual control?

1
2
3
4
5
6
7
8

val compressor = new ContextCompressor(
  strategy = CompressionStrategy.Staged,
  tokenLimit = 8000,
  preserveSystem = true,
  preserveRecent = 5
)

val compressed = compressor.compress(conversation)

Real production scenario: long-running customer support agent

One team built a support agent. Think of it as their first line of defense before escalating to humans (who cost money).

Average conversation: 20 turns. Some customers got chatty: 50+ turns. Token limit: 8K. Math didn’t math.

Without compression: Every conversation hit the limit around turn 12. They’d manually drop old messages. The agent would forget what the customer said 10 minutes ago. Quality went downhill. Customers got frustrated. Defeats the whole purpose.

With compression: The four-stage pipeline kicked in automatically. Kept conversations under 8K. Preserved system instructions (the agent’s personality). Kept recent exchanges (what the customer just said). Summarized ancient history (they had a billing question 30 minutes ago). Quality stayed consistent through turn 50.

The agent handled 10x more turns per conversation. Customer sat scores went up. Finance was happy. Engineering was happy. Rare win.

Part 9 - Secure execution: Docker-based workspace for safe tool running

Why tool execution needs isolation

You’re building an agent that can run code. Execute shell commands. Modify files. Super useful. Also super dangerous.

Here’s a fun thought experiment: What if the LLM hallucinates a rm -rf / command? Or decides to cat /etc/passwd for “debugging purposes”? Or opens network connections to mysterious servers you didn’t authorize?

LLMs are powerful. They’re also unpredictable. Giving them direct access to your file system is like handing your car keys to a very smart toddler. Sure, they might park it correctly. Or they might drive through your garage wall.

You need isolation. You need sandboxing. You need Docker.

How llm4s handles secure execution

llm4s provides a Docker-based workspace for tool execution. Tools run in isolated containers. They can’t access your host filesystem. They can’t make unauthorized network calls. They can’t damage your system.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14

// Configure workspace
val workspaceSettings = WorkspaceSettings(
  workspaceDir = "/tmp/llm4s-workspace",
  imageName = "llm4s-runner:latest",
  hostPort = 8080,
  traceLogPath = "/tmp/traces"
)

// Run tool in isolated container
val result = workspaceClient.execute(
  tool = "run_python_script",
  args = Map("script" -> "print('Hello from container')")
)
// Script runs in Docker, isolated from host

The workspace runner handles:

• Container lifecycle management
• File system isolation
• Network restrictions
• Resource limits (CPU, memory)
• Automatic cleanup

Real production benefit

One team built a code assistant. The kind that can actually run the code you’re asking about. Great feature. Terrible security implications.

It needed to run user-submitted code. Without isolation, they’d be executing random Python scripts directly on their production servers. That’s not a security model, that’s a suicide pact.

With llm4s workspace: User code runs in Docker. Container jail. Can’t escape. Can’t access host resources. Can’t read your database credentials. Can’t install Bitcoin miners. Can’t do anything except what it’s supposed to do.

Safe execution. Useful features. Sleep-at-night security. The holy trinity.

MCP (Model Context Protocol) support

llm4s supports MCP - the Model Context Protocol. This is cutting-edge. Most frameworks don’t have this yet.

MCP provides a standard way to manage context across LLM calls. Share context between tools. Maintain state across agent runs. Coordinate multi-agent systems.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11

import org.llm4s.mcp._

// Create MCP-compatible context
val mcpContext = MCPContext(
  contextId = "session-123",
  sharedState = Map("user_preferences" -> userPrefs),
  tools = registeredTools
)

// Agent uses MCP context
val agent = Agent(client, mcpContext)

As MCP adoption grows, llm4s is already compatible. Your code won’t need changes.

Part 6 - Tracing and observability: know what your LLM is doing

Why tracing matters

Your LLM app is slow. Users complain. You check logs. You see: “Completed in 8 seconds.”

Why 8 seconds? Was it the LLM? Network? Tool execution? Token generation?

You don’t know. You need tracing.

How llm4s tracing works

llm4s integrates with Langfuse. Every LLM call, tool execution, and agent step gets traced. You see:

• Request/response content
• Token usage (prompt, completion, total)
• Latency breakdown
• Error details
• Cost estimation

Enable tracing in config:

llm4s.tracing {
  enabled = true
  langfuse {
    public-key = ${LANGFUSE_PUBLIC_KEY}
    secret-key = ${LANGFUSE_SECRET_KEY}
    host = "https://cloud.langfuse.com"
  }
}

Add tracing to your code:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21

import org.llm4s.trace._

val tracer = EnhancedTracing.fromConfig()

val result = for {
  // Start trace
  traceId <- tracer.startTrace("customer-support-agent")

  // Trace LLM call
  completion <- tracer.traced("llm-completion") {
    client.complete(conversation)
  }

  // Trace tool execution
  toolResult <- tracer.traced("tool-execution") {
    executeTool(completion.message.toolCalls.head)
  }

  // End trace
  _ <- tracer.endTrace(traceId)
} yield completion

What you see in Langfuse

Trace view:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13

customer-support-agent (8.2s total)
├─ llm-completion (6.1s)
│  ├─ prompt_tokens: 1234
│  ├─ completion_tokens: 456
│  ├─ total_tokens: 1690
│  └─ cost: $0.042
├─ tool-execution (2.0s)
│  ├─ tool: search_docs
│  └─ result: [3 documents found]
└─ llm-completion (0.1s)
   ├─ prompt_tokens: 234
   ├─ completion_tokens: 89
   └─ cost: $0.008

Now you know: 6.1 seconds in LLM. 2 seconds in tool execution. Fix the slow tool.

Real production benefit

One team’s agent was slow. Users complained. They added tracing. Found that 80% of time was spent in read_file tool. The tool read entire files, even when only a few lines were needed.

Fix: Add a read_file_lines(path, start, end) tool. Read only what you need.

Result: Average response time dropped from 8s to 2s. Users happy.

Tracing showed them the problem. They fixed the root cause.

Part 7 - Getting started: template to production in minutes

The Giter8 template

New contributors to llm4s had a problem. They’d read docs. Copy example code. Manually create files. Twenty minutes later, they’d have a basic setup with typos in package names.

We fixed it with a Giter8 template.

1
2
3
4
5
6
7
8
9

$ sbt new llm4s/llm4s.g8
name [My LLM App]: chatbot
scala_version [3.3.0]:
provider [openai]: anthropic

$ cd chatbot
$ sbt run
[info] Running Main
Response: Hello! How can I help you today?

Sixty seconds. Zero typos. Working app.

What you get

The template generates:

• Working code: Not TODOs. Actual functioning LLM calls.
• Build config: SBT with correct dependencies.
• Formatting: scalafmt configuration.
• CI: GitHub Actions workflow.
• Tests: Basic test structure with ScalaTest.
• README: Instructions specific to your chosen provider.

The killer feature: it actually runs

Most templates generate scaffolding. You still have to implement everything. Useless.

llm4s template generates working code. Run sbt run immediately after generation. It calls the LLM. Prints the response. No blanks to fill in.

You start with working code. You modify it. You never start from zero.

Real production example: GSoC contributors

llm4s had 4 GSoC contributors. All used the template. All were productive on day one.

Before template: Average time to first contribution was 3 days. Setup problems. Dependency issues. Configuration mistakes.

After template: First contribution < 1 day. Setup takes 60 seconds. They jump straight to actual work.

Developer experience matters. The template makes it painless.

What makes the template special

Unlike most templates that generate TODO comments, llm4s.g8 generates working code:

• Complete build setup: SBT with correct dependencies, cross-compilation support
• CI/CD ready: GitHub Actions workflow pre-configured
• Code quality: Scalafmt, pre-commit hooks, scalafix rules
• Testing: MUnit setup with example tests
• Documentation: Generated README with your project specifics

And it’s parameterized. Pick your Scala version (2.13 or 3.x). Pick your llm4s version. Pick your provider. The template adapts.

1
2
3
4

sbt new llm4s/llm4s.g8 \
  --name=my-agent \
  --scala_version=3.7.1 \
  --llm4s_version=0.9.0

Generates a project configured exactly as specified. No manual edits. No version conflicts. It just works.

Part 8 - Production patterns from the trenches

These patterns emerged from real llm4s development - error handling PRs, developer tooling improvements, safety refactors. If you want the full story of how we discovered and refined these patterns through code reviews and production use, check out the production patterns article in this series.

Pattern 1: Smart constructors prevent invalid states

Don’t expose case class constructors. Make them private. Provide an apply method with validation.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21

final class RateLimitError private (
  val message: String,
  val provider: String,
  val retryAfter: Option[FiniteDuration]
)

object RateLimitError {
  def apply(
    message: String,
    provider: String,
    retryAfter: Option[FiniteDuration] = None
  ): RateLimitError = {
    require(message.nonEmpty, "Error message cannot be empty")
    require(provider.nonEmpty, "Provider name cannot be empty")
    retryAfter.foreach { duration =>
      require(duration > Duration.Zero, s"Retry duration must be positive, got: $duration")
    }

    new RateLimitError(message, provider, retryAfter)
  }
}

Now you can’t create invalid errors. The constructor validates everything.

Pattern 2: Bridge methods for migrations

Breaking changes kill adoption. Use bridge methods with deprecation warnings.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19

object Message {
  // New API (preferred)
  def apply(role: MessageRole, content: String): Message =
    new Message(role, content)

  // Bridge method (backward compatible)
  @deprecated("Use MessageRole enum instead of strings", "0.8.0")
  def apply(role: String, content: String): Message = {
    val roleEnum = role.toLowerCase match {
      case "user" => MessageRole.User
      case "assistant" => MessageRole.Assistant
      case "system" => MessageRole.System
      case other =>
        println(s"[DEPRECATED] Unknown role '$other', defaulting to User")
        MessageRole.User
    }
    new Message(roleEnum, content)
  }
}

Users get warnings. They migrate at their own pace. You remove the bridge method after 6 months. Everyone stays happy.

Pattern 3: Safety utilities eliminate try-catch blocks

Abstract error handling. Use implicit error mappers.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11

object Safety {
  def safely[A](f: => A)(implicit mapper: ErrorMapper): Result[A] =
    Try(f) match {
      case Success(value) => Right(value)
      case Failure(ex) => Left(mapper.mapError(ex))
    }
}

// Use it
def parseResponse(json: String): Result[Response] =
  Safety.safely(ujson.read(json))

No more try-catch. No more manual error conversion. One pattern. Applied everywhere.

Pattern 4: Resource management with .use

Resources need cleanup. Files need closing. HTTP clients need shutdown.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16

implicit class UsingOps[A <: AutoCloseable](resource: A) {
  def use[B](f: A => B)(implicit mapper: ErrorMapper): Result[B] =
    Safety.safely {
      try {
        f(resource)
      } finally {
        resource.close()
      }
    }
}

// Use it
def readConfig(path: String): Result[Config] =
  new FileInputStream(path).use { stream =>
    parseConfig(stream)
  }  // Stream auto-closes, even if parseConfig throws

No more resource leaks. No more try-finally boilerplate.

Pattern 5: Error accumulation with ValidatedNec

Sometimes you want all errors, not just the first one.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12

val configs = List(openAIConfig, anthropicConfig, cohereConfig)
val results = configs.map(validateConfig)

Safety.sequenceV(results) match {
  case Valid(validConfigs) => // All valid
  case Invalid(errors) =>
    // All errors at once:
    // - OpenAI API key is missing
    // - Anthropic model is invalid
    // - Cohere endpoint is unreachable
    errors.toList.foreach(e => println(s"  - ${e.formatted}"))
}

Users see all problems immediately. Fix them all at once. No repeated round-trips.

JVM ecosystem integration: the enterprise advantage

Why JVM matters for LLM applications

You’re building an LLM feature for an enterprise app. Your company runs on the JVM. Kafka for messaging. Spark for data processing. Akka for actor systems. Spring for web services. Cassandra for databases.

Python LLM frameworks? They don’t integrate cleanly. You need REST APIs. Message queues. Serialization overhead. Network hops. Complexity.

Scala LLM frameworks? They’re native. Same JVM. Same memory space. Direct method calls. Zero serialization. Clean integration.

Real integration examples

Kafka integration:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22

import org.apache.kafka.clients.consumer.KafkaConsumer
import org.llm4s.llmconnect._

// Process messages from Kafka with LLM
val consumer = new KafkaConsumer[String, String](kafkaConfig)
consumer.subscribe(List("user-queries").asJava)

consumer.poll(Duration.ofSeconds(1)).forEach { record =>
  val query = record.value()

  // Call LLM directly, no network overhead
  val result = for {
    config <- ConfigReader.Provider()
    client <- LLMConnect.client(config)
    completion <- client.complete(Conversation(Seq(UserMessage(query))))
  } yield completion

  result match {
    case Right(completion) => producer.send("llm-responses", completion.message.content)
    case Left(error) => logger.error(s"LLM error: ${error.formatted}")
  }
}

No REST API. No serialization. Direct function calls. Native integration.

Spark integration:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19

import org.apache.spark.sql.SparkSession
import org.llm4s.llmconnect._

// Process millions of records with LLM
val spark = SparkSession.builder().getOrCreate()
val df = spark.read.parquet("s3://data/user-feedback")

// Broadcast LLM client to all workers
val clientBc = spark.sparkContext.broadcast(client)

df.rdd.mapPartitions { partition =>
  val localClient = clientBc.value

  partition.map { row =>
    val feedback = row.getString(0)
    // Each worker processes rows with LLM
    localClient.complete(Conversation(Seq(UserMessage(s"Analyze sentiment: $feedback"))))
  }
}.collect()

The LLM client serializes. Broadcasts to workers. Runs on the same JVM as Spark. No external service calls needed.

Akka integration:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25

import akka.actor.typed._
import org.llm4s.llmconnect._

// LLM-powered actor system
object LLMAgent {
  sealed trait Command
  case class ProcessQuery(query: String, replyTo: ActorRef[Response]) extends Command
  case class Response(result: String)

  def apply(client: LLMClient): Behavior[Command] =
    Behaviors.receive { (context, message) =>
      message match {
        case ProcessQuery(query, replyTo) =>
          val result = client.complete(Conversation(Seq(UserMessage(query))))
          result match {
            case Right(completion) => replyTo ! Response(completion.message.content)
            case Left(error) => replyTo ! Response(s"Error: ${error.message}")
          }
          Behaviors.same
      }
    }
}

// Spawn actors for concurrent LLM processing
val system = ActorSystem(LLMAgent(client), "llm-system")

Actors + LLMs. Native Akka integration. Supervision. Backpressure. All the Akka benefits, with LLMs.

ZIO integration:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20

import zio._
import org.llm4s.llmconnect._

// Functional effect system with LLM
def processWithLLM(query: String): ZIO[LLMClient, LLMError, Completion] =
  for {
    client <- ZIO.service[LLMClient]
    result <- ZIO.fromEither(client.complete(Conversation(Seq(UserMessage(query)))))
  } yield result

// Compose with other effects
val program = for {
  user <- getUserFromDB
  query <- buildQuery(user)
  completion <- processWithLLM(query)
  _ <- saveToCache(completion)
} yield completion

// Run with dependency injection
program.provide(ZLayer.fromFunction(LLMConnect.client))

llm4s works with ZIO. Effect tracking. Resource management. Retries. Timeouts. Functional patterns, with LLMs.

Python can’t do this

Python LLM libraries don’t integrate with JVM ecosystems. They’re separate processes. You need REST APIs or message queues. Network overhead. Serialization. Latency.

llm4s runs on the JVM. Same process as your Kafka consumers, Spark jobs, Akka actors. Direct method calls. Zero network overhead. Native integration.

If you’re building LLM features for JVM applications, Python adds complexity. Scala eliminates it.

Architecture overview

Layered design

llm4s follows a clean layered architecture:

flowchart TD
    APP["Application Layer<br/>(Your code)"]
    API["API Layer<br/>(Agent, Assistant, Tools)"]
    CONNECT["LLMConnect Layer<br/>(LLMClient interface)"]
    PROVIDER["Provider Layer<br/>(OpenAI, Anthropic, Azure, Ollama)"]
    INFRA["Infrastructure Layer<br/>(Config, Tracing, Errors, Context)"]

    APP --> API
    API --> CONNECT
    CONNECT --> PROVIDER
    PROVIDER --> INFRA

    style APP fill:#e1f5ff,stroke:#0066cc,color:#000
    style API fill:#fff4e1,stroke:#cc8800,color:#000
    style CONNECT fill:#e1ffe1,stroke:#2d7a2d,color:#000
    style PROVIDER fill:#ffe1e1,stroke:#cc0000,color:#000
    style INFRA fill:#f0e1ff,stroke:#8800cc,color:#000

Each layer knows nothing about layers above it. Each layer has a clear responsibility.

Core types

Result types:

1
2

type Result[+A] = Either[LLMError, A]
type AsyncResult[+A] = Future[Result[A]]

Message hierarchy:

classDiagram
    class Message {
        <<sealed trait>>
        +role: String
        +content: Content
    }
    class UserMessage {
        +role = "user"
        +content: Content
    }
    class AssistantMessage {
        +role = "assistant"
        +content: String
        +toolCalls: Seq~ToolCall~
    }
    class SystemMessage {
        +role = "system"
        +content: String
    }
    class ToolMessage {
        +role = "tool"
        +toolCallId: String
        +content: String
    }

    Message <|-- UserMessage
    Message <|-- AssistantMessage
    Message <|-- SystemMessage
    Message <|-- ToolMessage

Error hierarchy:

classDiagram
    class LLMError {
        <<sealed trait>>
        +message: String
        +formatted: String
    }
    class RecoverableError {
        <<sealed trait>>
        +shouldRetry: Boolean
        +retryAfter: Option~Duration~
    }
    class NonRecoverableError {
        <<sealed trait>>
        +canRecover: Boolean = false
    }
    class RateLimitError {
        +message: String
        +retryAfter: Option~Duration~
        +provider: String
    }
    class TimeoutError {
        +message: String
        +duration: Duration
        +provider: String
    }
    class NetworkError {
        +message: String
        +cause: Option~Throwable~
    }
    class AuthenticationError {
        +message: String
        +provider: String
    }
    class ValidationError {
        +message: String
        +field: String
        +value: Option~String~
    }

    LLMError <|-- RecoverableError
    LLMError <|-- NonRecoverableError
    RecoverableError <|-- RateLimitError
    RecoverableError <|-- TimeoutError
    RecoverableError <|-- NetworkError
    NonRecoverableError <|-- AuthenticationError
    NonRecoverableError <|-- ValidationError

24+ specific error types with clear categorization for retry logic.

Module structure

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12

llm4s/
├── core/                    # Published to Maven Central
│   ├── agent/              # Agent framework
│   ├── llmconnect/         # LLM provider layer
│   ├── config/             # Configuration system
│   ├── error/              # Error types
│   ├── toolapi/            # Tool calling API
│   ├── trace/              # Tracing & observability
│   └── context/            # Context management
├── samples/                # Example applications
├── workspace/              # Docker-based workspace
└── crosstest/              # Cross-compilation tests

Published artifact: org.llm4s:core:0.9.0

Cross-compiled for Scala 2.13 and 3.x.

The llm4s community: open source done right

Why the community matters

llm4s isn’t just a framework. It’s a community. Active. Growing. Global.

Here’s what makes it special:

Weekly dev hours: Every Sunday, 9am London time. Live coding. Mob programming. Open to all. The maintainers (Rory Graves and Kannupriya Kalra) host. You join Discord. You code together. You learn together.

This isn’t a webinar. It’s collaborative development. You pair program with the maintainers. You see code evolve in real time. You influence design decisions.

Google Summer of Code: llm4s participated in GSoC 2025 through Scala Center. Four funded contributors. They worked to build many major features:

• Elvan Konukseven: Agentic toolkit for LLMs
• Gopi Trinadh Maddikunta: RAG in a box
• Anshuman Awasthi: Multimodal support (image, voice)
• Shubham Vishwakarma: Tracing and observability

These weren’t side projects. They’re core features, shipped to production, maintained by the community.

I am happy to co-author with maintainers, GSoC folks and beyond::

• Rory Graves (LinkedIn ): Lead architect. Scala expert. Created the foundation.
• Kannupriya Kalra (LinkedIn ): OSS DX & community champion. Engineering leader, Drives adoption. Speaks at conferences. Mentors GSoC contributors.
• Atul S. Khot (LinkedIn ): Co-authored many PRs with Atul ji, he’s exceptional design architect, Functional programming expert, Contributed to core library design patterns, reviewed PRs, provided top-notch feedback.
• Dmitry Mamonov: Co-mentored GSoC contributors on RAG and tracing projects, contributed to embedding and retrieval systems.

Global conference circuit: llm4s has been presented at:

• Bay Area Scala (San Francisco)
• Scala India
• Functional World (Poland)
• Dallas Scala Enthusiasts
• London Scala Users Group
• ScalaDays 2025 (Switzerland)
• Zurich Scala Enthusiasts

Real talks. Real demos. Real community engagement.

Szork: the AI-powered game

Want to see llm4s in action? Play Szork.

Szork is a text-based adventure game powered by llm4s. The LLM acts as dungeon master. You explore. You solve puzzles. You interact with NPCs. The LLM generates the world dynamically.

It’s built entirely on llm4s: Agent framework. Tool calling. Context management. Streaming responses. Multi-turn conversations.

It’s also open source: https://github.com/llm4s/szork

This isn’t a toy demo. It’s a complete application showcasing what llm4s can do. Inspectable. Forkable. Extendable.

The maintainers and co-authors

Maintainers:

• Rory Graves (LinkedIn ): Lead architect. Scala expert. Created the foundation.
• Kannupriya Kalra (LinkedIn ): OSS DX & community champion. Engineering leader, Drives adoption. Speaks at conferences. Mentors GSoC contributors.

Co-authors:

• Atul S. Khot (LinkedIn ): Functional programming patterns expert, design architect, reviews PRs, provides architectural feedback, helps maintain code quality
• Dmitry Mamonov (LinkedIn ): Co-mentored GSoC contributors on RAG and tracing, contributed embedding systems expertise, workflows, docker-containers.

All are active. All respond to issues. All review PRs. All contribute to making llm4s better.

This is open source done right. Not a corporate side project. Not abandoned after launch. Active. Maintained. Growing.

Why foundation work matters (a personal note)

Look, I’ll be honest. When I started contributing to llm4s, I wasn’t building the cool stuff. No agent orchestration that gets you retweets. No RAG pipelines that make conference talks. No streaming protocols that look impressive on your resume.

I was fixing error messages. Adding validation. Eliminating try-catch blocks. Building developer tooling. The stuff nobody notices until it’s missing.

And you know what? That’s exactly what production teams needed. Not the features that demo well. The features that prevent 2 AM incidents.

Error handling that made debugging 60% faster . Smart constructors that prevent invalid states . Safety utilities that eliminate manual error handling . Production patterns that scale across 8 LLM providers and 4 GSoC projects.

This is the infrastructure work that OpenSSF research says “96% of open source’s $8.8 trillion value depends on.” The 5% of contributors doing the work everyone depends on but nobody sees.

AI/ML and LLMs are the core. They’re what makes llm4s interesting. But the edges, the foundations, the fundamentals - that’s what makes it production-ready.

That’s the work I’m proud of. The articles in this series dive deep into each contribution. Real PRs. Real code reviews. Real production lessons. Not theory. Not toy examples. Actual work that shipped to production and helped teams build reliable LLM systems.

If you’re working on infrastructure, on developer experience, on the “boring” parts of your framework - this is for you. That work matters. More than you think.

What’s next for llm4s

llm4s is production-ready today. But we’re not done.

Upcoming features:

• Enhanced streaming support with backpressure handling
• Multi-provider failover (OpenAI rate limit? Fall back to Anthropic)
• Advanced retry strategies with circuit breakers
• Structured output parsing with compile-time validation
• Prompt template system with type-safe interpolation
• Vector database integrations (Pinecone, Weaviate, Chroma)
• More MCP (Model Context Protocol) integrations

Community contributions welcome:

• New provider integrations (Google Gemini, Mistral, Cohere)
• More tool examples
• Better documentation
• Performance optimizations
• RAG patterns and examples
• Multi-agent coordination patterns

Join us:

• GitHub: https://github.com/llm4s/llm4s
• Discord: https://discord.gg/4uvTPn6qww
• Try the template: sbt new llm4s/llm4s.g8
• Weekly dev hour: Sundays 9am London time on Discord
• Play Szork: https://github.com/llm4s/szork

The bottom line: Python vs Scala for LLM apps

Still deciding between Python and Scala for your LLM project? Here’s the honest comparison:

Choose Python if: You’re prototyping, working solo, or your team is already Python-heavy. Also perfect for interviews - leetcode doesn’t care about production concerns.

Choose Scala + llm4s if: You’re building for production, integrating with JVM services, or your team is > 10 engineers where type safety prevents chaos.

The transition point: When your prototype gets greenlit for production, that’s when you rewrite in Scala. Not because Python can’t work - it can - but because every production issue you’ll face (refactoring safety, JVM integration, structured errors, team coordination) is already solved by the JVM ecosystem.

Python is the bicycle that gets you to the prototype party. The JVM is the truck that ships the product to customers. You wouldn’t use a bicycle to deliver furniture, and you shouldn’t use Python for production LLM apps when you’re already running on the JVM.

References

llm4s resources

• llm4s GitHub - Main repository
• llm4s Template - Giter8 template
• Internals Documentation - Architecture deep-dive
• API Specification - Complete API docs
• Discord Community - Get help, contribute

Key articles in this series

• Error handling foundation - Building the error hierarchy
• Developer experience with g8 - Creating the template
• Smart constructors - Validation at construction
• Safety refactor - Eliminating try-catch
• Production patterns - What actually works

Provider documentation

• OpenAI API Reference - OpenAI docs
• Anthropic API Reference - Anthropic Claude docs
• Azure OpenAI Documentation - Azure docs
• Ollama Documentation - Ollama local models

Functional programming in scala

• Scala Documentation - Official Scala docs
• Functional Programming in Scala - FP book (Manning)
• Rock the JVM - Scala & Functional Programming Essentials - Hands-on Scala 3 course with 3000+ lines of code
• Rock the JVM - Advanced Scala - Advanced FP, type system, asynchronous programming
• Rock the JVM - All useful courses - Composable, purely functional applications

TL;DR

• llm4s is a production-grade Scala framework for building LLM applications with compile-time safety, functional error handling, and provider abstraction - the first and only mature LLM framework for Scala
• Why Scala over Python: Type safety catches errors at compile time; JVM ecosystem native integration (Kafka, Spark, Akka, ZIO); functional programming with immutable data; scales with team size; production performance
• Type safety: Sealed traits with exhaustive pattern matching catch typos at compile time; ToolFunction[T, R] ensures tool definitions are valid before runtime; invalid states impossible to construct
• Error handling: Result[A] = Either[LLMError, A] makes errors explicit; recoverable vs non-recoverable distinction; composable with for-comprehensions; no exceptions
• Provider abstraction: Same code works with OpenAI, Anthropic, Azure, Ollama, OpenRouter - switch by changing config, zero code changes; LLMClient interface abstracts all differences
• Agent framework: Handles tool calling, conversation management, state tracking (InProgress/WaitingForTools/Complete), tracing, safety limits automatically; tail-recursive execution
• Type-safe tool calling: ToolFunction[T, R] with SchemaDefinition validation; SafeParameterExtractor for safe argument parsing; compile-time guarantees Python can’t provide; enhanced error reporting
• Context management: Four-stage compression pipeline (deterministic → tool output → history summarization → LLM-based) keeps conversations under token limits automatically
• Secure execution: Docker-based workspace for tool isolation; prevents system damage from hallucinated commands; MCP (Model Context Protocol) support for context coordination
• JVM ecosystem integration: Native Kafka, Spark, Akka, ZIO integration with zero network overhead; broadcast LLM clients to Spark workers; LLM-powered Akka actors; Python frameworks need REST APIs
• Observability: Langfuse integration with latency breakdown, token usage, cost estimation; multi-backend tracing (langfuse/console/noop modes); TRACING_MODE environment variable
• Developer experience: Giter8 template generates working app in 60 seconds (sbt new llm4s/llm4s.g8); pre-commit hooks, CI/CD, tests all included; no TODOs, actual working code
• Production patterns: Smart constructors prevent invalid states, bridge methods enable migrations, safety utilities eliminate try-catch, resource management with .use, error accumulation with ValidatedNec
• Active community: Weekly dev hours (Sundays 9am London), 4 GSoC contributors, global conference circuit (Bay Area, London, ScalaDays), Discord with responsive maintainers
• Real demos: Szork - AI-powered text adventure game built entirely on llm4s; showcases agent framework, tool calling, streaming, multi-turn conversations; fully open source
• Cross-compiled: Scala 2.13 and 3.x, published to Maven Central: "org.llm4s" %% "core" % "0.9.0"; pre-commit hooks enforce cross-version compatibility

Series navigation: ← Previous: Production patterns | Next: (Coming soon - Streaming deep-dive)

This is the introduction article for the llm4s series. llm4s is maintained by Rory Graves and Kannupriya Kalra . Join the community at discord.gg/4uvTPn6qww .