{"id":2088,"date":"2025-06-03T19:23:06","date_gmt":"2025-06-03T19:23:06","guid":{"rendered":"https:\/\/rssfeedtelegrambot.bnaya.co.il\/index.php\/2025\/06\/03\/how-to-make-an-ai-chatbot-from-scratch-using-docker-model-runner\/"},"modified":"2025-06-03T19:23:06","modified_gmt":"2025-06-03T19:23:06","slug":"how-to-make-an-ai-chatbot-from-scratch-using-docker-model-runner","status":"publish","type":"post","link":"https:\/\/rssfeedtelegrambot.bnaya.co.il\/index.php\/2025\/06\/03\/how-to-make-an-ai-chatbot-from-scratch-using-docker-model-runner\/","title":{"rendered":"How to Make an AI Chatbot from Scratch using Docker Model Runner"},"content":{"rendered":"

Today, we\u2019ll show you how to build a fully functional Generative AI chatbot using Docker Model Runner<\/a> and powerful observability tools, including Prometheus, Grafana, and Jaeger. We\u2019ll walk you through the common challenges developers face when building AI-powered applications, demonstrate how Docker Model Runner solves these pain points, and then guide you step-by-step through building a production-ready chatbot with comprehensive monitoring and metrics.<\/p>\n

By the end of this guide, you\u2019ll know how to make an AI chatbot and run it locally. You\u2019ll also learn how to set up real-time monitoring insights, streaming responses, and a modern React interface \u2014 all orchestrated through familiar Docker workflows.<\/p>\n

The current challenges with GenAI development<\/h2>\n

Generative AI (GenAI) is revolutionizing software development, but creating AI-powered applications comes with significant challenges. First, the current AI landscape is fragmented \u2014 developers must piece together various libraries, frameworks, and platforms that weren\u2019t designed to work together. Second, running large language models efficiently requires specialized hardware configurations that vary across platforms, while AI model execution remains disconnected from standard container workflows. This forces teams to maintain separate environments for their application code and AI models.<\/p>\n

Third, without standardized methods for storing, versioning, and serving models, development teams struggle with inconsistent deployment practices. Meanwhile, relying on cloud-based AI services creates financial strain through unpredictable costs that scale with usage. Additionally, sending data to external AI services introduces privacy and security risks, especially for applications handling sensitive information.<\/p>\n

These challenges combine to create a frustrating developer experience that hinders experimentation and slows innovation precisely when businesses need to accelerate their AI adoption. Docker Model Runner addresses these pain points by providing a streamlined solution for running AI models locally<\/a>, right within your existing Docker workflow.<\/p>\n

How Docker is solving these challenges<\/h2>\n

Docker Model Runner<\/a> offers a revolutionary approach to GenAI development by integrating AI model execution directly into familiar container workflows.\u00a0<\/p>\n

\n<\/div>\n

Figure 1: Comparison diagram showing complex multi-step traditional GenAI setup versus simplified Docker Model Runner single-command workflow<\/em><\/p>\n\n

Many developers successfully use containerized AI models<\/a>, benefiting from integrated workflows, cost control, and data privacy. Docker Model Runner builds on these strengths by making it even easier and more efficient to work with models. By running models natively on your host machine while maintaining the familiar Docker interface, Model Runner delivers.<\/p>\n

Simplified Model Execution<\/strong>: Run AI models locally with a simple Docker CLI command, no complex setup required.<\/p>\n

Hardware Acceleration<\/strong>: Direct access to GPU resources without containerization overhead<\/p>\n

Integrated Workflow<\/strong>: Seamless integration with existing Docker tools and container development practices<\/p>\n

Standardized Packaging<\/strong>: Models are distributed as OCI artifacts through the same registries you already use<\/p>\n

Cost Control<\/strong>: Eliminate unpredictable API costs by running models locally<\/p>\n

Data Privacy<\/strong>: Keep sensitive data within your infrastructure with no external API calls<\/p>\n

This approach fundamentally changes how developers can build and test AI-powered applications, making local development faster, more secure, and dramatically more efficient.<\/p>\n\n

How to create an AI chatbot with Docker<\/h2>\n

In this guide, we\u2019ll build a comprehensive GenAI application that showcases how to create a fully-featured chat interface powered by Docker Model Runner, complete with advanced observability tools to monitor and optimize your AI models.<\/p>\n

Project overview<\/h3>\n

The project is a complete Generative AI interface that demonstrates how to:<\/p>\n

Create a responsive React\/TypeScript chat UI with streaming responses<\/p>\n

Build a Go backend server that integrates with Docker Model Runner<\/p>\n

Implement comprehensive observability with metrics, logging, and tracing<\/p>\n

Monitor AI model performance with real-time metrics<\/p>\n

Architecture<\/h3>\n

The application consists of these main components:<\/p>\n

The frontend sends chat messages to the backend API<\/p>\n

The backend formats the messages and sends them to the Model Runner<\/p>\n

The LLM processes the input and generates a response<\/p>\n

The backend streams the tokens back to the frontend as they\u2019re generated<\/p>\n

The frontend displays the incoming tokens in real-time<\/p>\n

Observability components collect metrics, logs, and traces throughout the process<\/p>\n

\n<\/div>\n

Figure 2: Architecture diagram showing data flow between frontend, backend, Model Runner, and observability tools like Prometheus, Grafana, and Jaeger.<\/em><\/p>\n\n

Project structure<\/h3>\n

The project<\/a> has the following structure:<\/p>\n

\ntree -L 2
\n.
\n\u251c\u2500\u2500 Dockerfile
\n\u251c\u2500\u2500 README-model-runner.md
\n\u251c\u2500\u2500 README.md
\n\u251c\u2500\u2500 backend.env
\n\u251c\u2500\u2500 compose.yaml
\n\u251c\u2500\u2500 frontend
\n..
\n\u251c\u2500\u2500 go.mod
\n\u251c\u2500\u2500 go.sum
\n\u251c\u2500\u2500 grafana
\n\u2502 \u2514\u2500\u2500 provisioning
\n\u251c\u2500\u2500 main.go
\n\u251c\u2500\u2500 main_branch_update.md
\n\u251c\u2500\u2500 observability
\n\u2502 \u2514\u2500\u2500 README.md
\n\u251c\u2500\u2500 pkg
\n\u2502 \u251c\u2500\u2500 health
\n\u2502 \u251c\u2500\u2500 logger
\n\u2502 \u251c\u2500\u2500 metrics
\n\u2502 \u251c\u2500\u2500 middleware
\n\u2502 \u2514\u2500\u2500 tracing
\n\u251c\u2500\u2500 prometheus
\n\u2502 \u2514\u2500\u2500 prometheus.yml
\n\u251c\u2500\u2500 refs
\n\u2502 \u2514\u2500\u2500 heads
\n..\n

21 directories, 33 files\n<\/p><\/div>\n\n

We\u2019ll examine the key files and understand how they work together throughout this guide.<\/p>\n

Prerequisites<\/h3>\n

Before we begin, make sure you have:<\/p>\n

Docker Desktop<\/a> (version 4.40 or newer)\u00a0<\/p>\n

Docker Model Runner<\/a> enabled<\/p>\n

At least 16GB of RAM for running AI models efficiently<\/p>\n

Familiarity with Go (for backend development)<\/p>\n

Familiarity with React and TypeScript (for frontend development)<\/p>\n

Getting started<\/h3>\n

To run the application:<\/p>\n

Clone the repository:\u00a0<\/p>\n

\ngit clone
\nhttps:\/\/github.com\/dockersamples\/genai-model-runner-metrics\n

cd genai-model-runner-metrics<\/p>\n<\/div>\n

Enable Docker Model Runner in Docker Desktop:<\/p>\n

Go to Settings > Features in Development > Beta tab<\/p>\n

Enable \u201cDocker Model Runner\u201d<\/p>\n

Select \u201cApply and restart\u201d<\/p>\n

\n<\/div>\n

Figure 3: Screenshot of Docker Desktop Beta Features settings panel with Docker AI, Docker Model Runner, and TCP support enabled.<\/em><\/p>\n

Download the model<\/p>\n

For this demo, we\u2019ll use Llama 3.2, but you can substitute any model of your choice:<\/p>\n

\ndocker model pull ai\/llama3.2:1B-Q8_0\n<\/div>\n

Just like viewing containers, you can manage your downloaded AI models directly in Docker Dashboard under the Models section. Here you can see model details, storage usage, and manage your local AI model library.<\/p>\n

\n<\/div>\n

Figure 4: View of Docker Dashboard showing locally downloaded AI models with details like size, parameters, and quantization.<\/em><\/p>\n

Start the application:\u00a0<\/p>\n

\ndocker compose up -d –build\n<\/div>\n
\n<\/div>\n

Figure 5: List of active running containers in Docker Dashboard, including Jaeger, Prometheus, backend, frontend, and genai-model-runner-metrics.<\/em><\/p>\n

Open your browser and navigate to the frontend URL at http:\/\/localhost:3000<\/a> . You\u2019ll be greeted with a modern chat interface (see screenshot) featuring:\u00a0<\/p>\n

Clean, responsive design with dark\/light mode toggle<\/p>\n

Message input area ready for your first prompt<\/p>\n

Model information displayed in the footer<\/p>\n

\n<\/div>\n

Figure 6: GenAI chatbot interface showing live metrics panel with input\/output tokens, response time, and error rate.<\/em><\/p>\n

Click on Expand to view the metrics like:<\/p>\n

Input tokens<\/p>\n

Output tokens<\/p>\n

Total Requests<\/p>\n

Average Response Time<\/p>\n

Error Rate<\/p>\n

\n<\/div>\n

Figure 7: Expanded metrics view with input and output tokens, detailed chat prompt, and response generated by Llama 3.2 model.<\/em><\/p>\n

Grafana allows you to visualize metrics through customizable dashboards. Click on View Detailed Dashboard to open up Grafana dashboard.<\/p>\n

\n<\/div>\n

Figure 8: Chat interface showing metrics dashboard with prompt and response plus option to view detailed metrics in Grafana.<\/em><\/p>\n

Log in with the default credentials (enter \u201cadmin\u201d as user and password) to explore pre-configured AI performance dashboards (see screenshot below) showing real-time metrics like tokens per second, memory usage, and model performance.\u00a0<\/p>\n

Select Add your first data source. Choose Prometheus as a data source. Enter \u201chttp:\/\/prometheus:9090<\/a>\u201d as Prometheus Server URL. Scroll down to the end of the site and click \u201cSave and test\u201d. By now, you should see \u201cSuccessfully queried the Prometheus API\u201d as an acknowledgement. Select Dashboard and click Re-import for all these dashboards.<\/p>\n

By now, you should have a Prometheus 2.0 Stats dashboard up and running.<\/p>\n

\n<\/div>\n

Figure 9: Grafana dashboard with multiple graph panels monitoring GenAI chatbot performance, displaying time-series charts for memory consumption, processing speeds, and application health<\/em><\/p>\n\n

Prometheus allows you to collect and store time-series metrics data. Open the Prometheus query interface http:\/\/localhost:9091<\/a> and start typing \u201cgenai\u201d in the query box to explore all available AI metrics (as shown in the screenshot below). You\u2019ll see dozens of automatically collected metrics, including tokens per second, latency measurements, and llama.cpp-specific performance data.\u00a0<\/p>\n

\n<\/div>\n

Figure 10: Prometheus web interface showing dropdown of available GenAI metrics including genai_app_active_requests and genai_app_token_latency<\/em><\/p>\n\n

Jaeger provides a visual exploration of request flows and performance bottlenecks. You can access it via http:\/\/localhost:16686<\/a><\/p>\n

Implementation details<\/h3>\n

Let\u2019s explore how the key components of the project work:<\/p>\n

Frontend implementation<\/p>\n

The React frontend provides a clean, responsive chat interface built with TypeScript and modern React patterns. The core App.tsx component manages two essential pieces of state: dark mode preferences for user experience and model metadata fetched from the backend\u2019s health endpoint.\u00a0<\/p>\n

When the component mounts, the useEffect hook automatically retrieves information about the currently running AI model. It displays details like the model name directly in the footer to give users transparency about which LLM is powering their conversations.<\/p>\n

\n\/\/ Essential App.tsx structure
\nfunction App() {
\n const [darkMode, setDarkMode] = useState(false);
\n const [modelInfo, setModelInfo] = useState<ModelMetadata | null>(null);\n

\/\/ Fetch model info from backend
\n useEffect(() => {
\n fetch(‘http:\/\/localhost:8080\/health’)
\n .then(res => res.json())
\n .then(data => setModelInfo(data.model_info));
\n }, []);<\/p>\n

return (
\n <div className=”min-h-screen bg-white dark:bg-gray-900″>
\n <Header toggleDarkMode={() => setDarkMode(!darkMode)} \/>
\n <ChatBox \/>
\n <footer>
\n Powered by Docker Model Runner running {modelInfo?.model}
\n <\/footer>
\n <\/div>
\n );
\n}<\/p>\n<\/div>\n

The main App component orchestrates the overall layout while delegating specific functionality to specialized components like Header for navigation controls and ChatBox for the actual conversation interface. This separation of concerns makes the codebase maintainable while the automatic model info fetching demonstrates how the frontend seamlessly integrates with the Docker Model Runner through the Go backend\u2019s API, creating a unified user experience that abstracts away the complexity of local AI model execution.<\/p>\n

Backend implementation: Integration with Model Runner<\/p>\n

The core of this application is a Go backend that communicates with Docker Model Runner. Let\u2019s examine the key parts of our main.go file:<\/p>\n

\nclient := openai.NewClient(
\n option.WithBaseURL(baseURL),
\n option.WithAPIKey(apiKey),
\n)\n<\/div>\n

This demonstrates how we leverage Docker Model Runner\u2019s OpenAI-compatible API. The Model Runner exposes endpoints that match OpenAI\u2019s API structure, allowing us to use standard clients. Depending on your connection method, baseURL is set to either:<\/p>\n

http:\/\/model-runner.docker.internal\/engines\/llama.cpp\/v1\/ (for Docker socket)<\/p>\n

http:\/\/host.docker.internal:12434\/engines\/llama.cpp\/v1\/ (for TCP)<\/p>\n

How metrics flow from host to containers<\/h3>\n

One key architectural detail worth understanding: llama.cpp runs natively on your host (via Docker Model Runner), while Prometheus and Grafana run in containers. Here\u2019s how they communicate:<\/p>\n

The Backend as Metrics Bridge:<\/strong><\/p>\n

Connects<\/strong> to llama.cpp via Model Runner API (http:\/\/localhost:12434)<\/p>\n

Collects<\/strong> performance data from each API call (response times, token counts)<\/p>\n

Calculates<\/strong> metrics like tokens per second and memory usage<\/p>\n

Exposes<\/strong> all metrics in Prometheus format at http:\/\/backend:9090\/metrics<\/p>\n

Enables<\/strong> containerized Prometheus to scrape metrics without host access<\/p>\n

This hybrid architecture gives you the performance benefits of native model execution with the convenience of containerized observability.<\/p>\n

LLama.cpp metrics integration<\/h3>\n

The project provides detailed real-time metrics specifically for llama.cpp models:<\/p>\n

\n

Metric<\/strong><\/p>\n

Description<\/strong><\/p>\n

Implementation in Code<\/strong><\/p>\n

Tokens per Second<\/p>\n

Measure of model generation speed<\/p>\n

LlamaCppTokensPerSecond in metrics.go<\/p>\n

Context Window Size<\/p>\n

Maximum context length in tokens<\/p>\n

LlamaCppContextSize in metrics.go<\/p>\n

Prompt Evaluation Time<\/p>\n

Time spent processing input prompt<\/p>\n

LlamaCppPromptEvalTime in metrics.go<\/p>\n

Memory per Token<\/p>\n

Memory efficiency measurement<\/p>\n

LlamaCppMemoryPerToken in metrics.go<\/p>\n

Thread Utilization<\/p>\n

Number of CPU threads used<\/p>\n

LlamaCppThreadsUsed in metrics.go<\/p>\n

Batch Size<\/p>\n

Token processing batch size<\/p>\n

LlamaCppBatchSize in metrics.go<\/p>\n<\/div>\n

One of the most powerful features is our detailed metrics collection for llama.cpp models. These metrics help optimize model performance and identify bottlenecks in your inference pipeline.<\/p>\n

\n\/\/ LlamaCpp metrics
\nllamacppContextSize = promautoFactory.NewGaugeVec(
\n prometheus.GaugeOpts{
\n Name: “genai_app_llamacpp_context_size”,
\n Help: “Context window size in tokens for llama.cpp models”,
\n },
\n []string{“model”},
\n)\n

llamacppTokensPerSecond = promautoFactory.NewGaugeVec(
\n prometheus.GaugeOpts{
\n Name: “genai_app_llamacpp_tokens_per_second”,
\n Help: “Tokens generated per second”,
\n },
\n []string{“model”},
\n)<\/p>\n

\/\/ More metrics definitions…<\/p>\n<\/div>\n

These metrics are collected, processed, and exposed both for Prometheus scraping and for real-time display in the front end. This gives us unprecedented visibility into how the llama.cpp inference engine is performing.<\/p>\n\n

Chat implementation with streaming<\/h3>\n

The chat endpoint implements streaming for real-time token generation:<\/p>\n

\n

\/\/ Set up streaming with a proper SSE format
\nw.Header().Set(“Content-Type”, “text\/event-stream”)
\nw.Header().Set(“Cache-Control”, “no-cache”)
\nw.Header().Set(“Connection”, “keep-alive”)<\/p>\n

\/\/ Stream each chunk as it arrives
\nif len(chunk.Choices) > 0 && chunk.Choices[0].Delta.Content != “” {
\n outputTokens++
\n _, err := fmt.Fprintf(w, “%s”, chunk.Choices[0].Delta.Content)
\n if err != nil {
\n log.Printf(“Error writing to stream: %v”, err)
\n return
\n }
\n w.(http.Flusher).Flush()
\n}<\/p>\n<\/div>\n

This streaming implementation ensures that tokens appear in real-time in the user interface, providing a smooth and responsive chat experience. You can also measure key performance metrics like time to first token and tokens per second.<\/p>\n

Performance measurement<\/h3>\n

You can measure various performance aspects of the model:<\/p>\n

\n\/\/ Record first token time
\nif firstTokenTime.IsZero() && len(chunk.Choices) > 0 &&
\nchunk.Choices[0].Delta.Content != “” {
\n firstTokenTime = time.Now()\n

\/\/ For llama.cpp, record prompt evaluation time
\n if strings.Contains(strings.ToLower(model), “llama”) ||
\n strings.Contains(apiBaseURL, “llama.cpp”) {
\n promptEvalTime := firstTokenTime.Sub(promptEvalStartTime)
\n llamacppPromptEvalTime.WithLabelValues(model).Observe(promptEvalTime.Seconds())
\n }
\n}<\/p>\n

\/\/ Calculate tokens per second for llama.cpp metrics
\nif strings.Contains(strings.ToLower(model), “llama”) ||
\n strings.Contains(apiBaseURL, “llama.cpp”) {
\n totalTime := time.Since(firstTokenTime).Seconds()
\n if totalTime > 0 && outputTokens > 0 {
\n tokensPerSecond := float64(outputTokens) \/ totalTime
\n llamacppTokensPerSecond.WithLabelValues(model).Set(tokensPerSecond)
\n }
\n}<\/p>\n<\/div>\n

These measurements help us understand the model\u2019s performance characteristics and optimize the user experience.<\/p>\n

Metrics collection<\/h3>\n

The metrics.go file is a core component of our observability stack for the Docker Model Runner-based chatbot. This file defines a comprehensive set of Prometheus metrics that allow us to monitor both the application performance and the underlying llama.cpp model behavior.<\/p>\n

Core metrics architecture<\/h3>\n

The file establishes a collection of Prometheus metric types:<\/p>\n

Counters<\/strong>: For tracking cumulative values (like request counts, token counts)<\/p>\n

Gauges<\/strong>: For tracking values that can increase and decrease (like active requests)<\/p>\n

Histograms<\/strong>: For measuring distributions of values (like latencies)<\/p>\n

Each metric is created using the promauto factory, which automatically registers metrics with Prometheus.<\/p>\n

Categories of metrics<\/h2>\n

The metrics can be divided into three main categories:<\/p>\n

1. HTTP and application metrics<\/h3>\n
\n\/\/ RequestCounter counts total HTTP requests
\nRequestCounter = promauto.NewCounterVec(
\n prometheus.CounterOpts{
\n Name: “genai_app_http_requests_total”,
\n Help: “Total number of HTTP requests”,
\n },
\n []string{“method”, “endpoint”, “status”},
\n)\n

\/\/ RequestDuration measures HTTP request durations
\nRequestDuration = promauto.NewHistogramVec(
\n prometheus.HistogramOpts{
\n Name: “genai_app_http_request_duration_seconds”,
\n Help: “HTTP request duration in seconds”,
\n Buckets: prometheus.DefBuckets,
\n },
\n []string{“method”, “endpoint”},
\n)<\/p>\n<\/div>\n

These metrics monitor the HTTP server performance, tracking request counts, durations, and error rates. The metrics are labelled with dimensions like method, endpoint, and status to enable detailed analysis.<\/p>\n

2. Model performance metrics<\/h3>\n
\n\/\/ ChatTokensCounter counts tokens in chat requests and responses
\nChatTokensCounter = promauto.NewCounterVec(
\n prometheus.CounterOpts{
\n Name: “genai_app_chat_tokens_total”,
\n Help: “Total number of tokens processed in chat”,
\n },
\n []string{“direction”, “model”},
\n)\n

\/\/ ModelLatency measures model response time
\nModelLatency = promauto.NewHistogramVec(
\n prometheus.HistogramOpts{
\n Name: “genai_app_model_latency_seconds”,
\n Help: “Model response time in seconds”,
\n Buckets: []float64{0.1, 0.5, 1, 2, 5, 10, 20, 30, 60},
\n },
\n []string{“model”, “operation”},
\n)<\/p>\n<\/div>\n

These metrics track the LLM usage patterns and performance, including token counts (both input and output) and overall latency. The FirstTokenLatency metric is particularly important as it measures the time to get the first token from the model, which is a critical user experience factor.<\/p>\n

3. llama.cpp specific metrics<\/h3>\n
\n\/\/ LlamaCppContextSize measures the context window size
\nLlamaCppContextSize = promauto.NewGaugeVec(
\n prometheus.GaugeOpts{
\n Name: “genai_app_llamacpp_context_size”,
\n Help: “Context window size in tokens for llama.cpp models”,
\n },
\n []string{“model”},
\n)\n

\/\/ LlamaCppTokensPerSecond measures generation speed
\nLlamaCppTokensPerSecond = promauto.NewGaugeVec(
\n prometheus.GaugeOpts{
\n Name: “genai_app_llamacpp_tokens_per_second”,
\n Help: “Tokens generated per second”,
\n },
\n []string{“model”},
\n)<\/p>\n<\/div>\n

These metrics capture detailed performance characteristics specific to the llama.cpp inference engine used by Docker Model Runner. They include:<\/p>\n

1. Context Size:\u00a0<\/h4>\n

It represents the token window size used by the model, typically ranging from 2048 to 8192 tokens. The optimization goal is balancing memory usage against conversation quality.\u00a0 When memory usage becomes problematic, reduce context size to 2048 tokens for faster processing<\/p>\n

2. Prompt Evaluation Time<\/h4>\n

It measures the time spent processing input before generating tokens, essentially your time-to-first-token latency with a target of under 2 seconds. The optimization focus is minimizing user wait time for the initial response. If evaluation time exceeds 3 seconds, reduce context size or implement prompt compression techniques.<\/p>\n

3. Tokens Per Second<\/h4>\n

It measures the time spent processing input before generating tokens, essentially your time-to-first-token latency with a target of under 2 seconds. The optimization focus is minimizing user wait time for the initial response. If evaluation time exceeds 3 seconds, reduce context size or implement prompt compression techniques.\u00a0<\/p>\n

4. Tokens Per Second<\/h4>\n

It indicates generation speed, with a target of 8+ TPS for good user experience. This metric requires balancing response speed with model quality. When TPS drops below 5, switch to more aggressive quantization (Q4 instead of Q8) or use a smaller model variant.\u00a0<\/p>\n

5. Memory Per Token<\/h4>\n

It tracks RAM consumption per generated token, with optimization aimed at preventing out-of-memory crashes and optimizing resource usage. When memory consumption exceeds 100MB per token, implement aggressive conversation pruning to reduce memory pressure. If memory usage grows over time during extended conversations, add automatic conversation resets after a set number of exchanges.\u00a0<\/p>\n

6. Threads Used<\/h4>\n

It monitors the number of CPU cores actively processing model operations, with the goal of maximizing throughput without overwhelming the system. If thread utilization falls below 50% of available cores, increase the thread count for better performance.\u00a0<\/strong><\/p>\n

7. Batch Size<\/h4>\n

It controls how many tokens are processed simultaneously, requiring optimization based on your specific use case balancing latency versus throughput. For real-time chat applications, use smaller batches of 32-64 tokens to minimize latency and provide faster response times.<\/p>\n

In nutshell, these metrics are crucial for understanding and optimizing llama.cpp performance characteristics, which directly affect the user experience of the chatbot.<\/p>\n

Docker Compose: LLM as a first-class service<\/h2>\n

With Docker Model Runner integration, Compose makes AI model deployment as simple as any other service. One docker-compose.yml file defines your entire AI application:<\/p>\n

Your AI models (via Docker Model Runner)<\/p>\n

Application backend and frontend<\/p>\n

Observability stack (Prometheus, Grafana, Jaeger)<\/p>\n

All networking and dependencies<\/p>\n

The most innovative aspect is the llm service using Docker\u2019s model provider, which simplifies model deployment by directly integrating with Docker Model Runner without requiring complex configuration. This composition creates a complete, scalable AI application stack with comprehensive observability.<\/p>\n

\n llm:
\n provider:
\n type: model
\n options:
\n model: ${LLM_MODEL_NAME:-ai\/llama3.2:1B-Q8_0}\n<\/div>\n

\u200b\u200bThis configuration tells Docker Compose<\/a> to treat an AI model as a standard service in your application stack, just like a database or web server.\u00a0<\/p>\n

The provider syntax is Docker\u2019s new way of handling AI models natively. Instead of building containers or pulling images, Docker automatically manages the entire model-serving infrastructure for you.\u00a0<\/p>\n

The model: ${LLM_MODEL_NAME:-ai\/llama3.2:1B-Q8_0} line uses an environment variable with a fallback, meaning it will use whatever model you specify in LLM_MODEL_NAME, or default to Llama 3.2 1B if nothing is set.<\/p>\n

\n

Docker Compose: One command to run your entire stack<\/p>\n

Why is this revolutionary? Before this, deploying an LLM required dozens of lines of complex configuration \u2013 custom Dockerfiles, GPU device mappings, volume mounts for model files, health checks, and intricate startup commands.<\/p>\n

Now, those four lines replace all of that complexity. Docker handles downloading the model, configuring the inference engine, setting up GPU access, and exposing the API endpoints automatically. Your other services can connect to the LLM using simple service names, making AI models as easy to use as any other infrastructure component. This transforms AI from a specialized deployment challenge into standard infrastructure-as-code.<\/p>\n<\/div>\n

Here\u2019s the full compose.yml file that orchestrates the entire application:<\/p>\n

\n services:
\n backend:
\n env_file: ‘backend.env’
\n build:
\n context: .
\n target: backend
\n ports:
\n – ‘8080:8080’
\n – ‘9090:9090’ # Metrics port
\n volumes:
\n – \/var\/run\/docker.sock:\/var\/run\/docker.sock # Add Docker socket access
\n healthcheck:
\n test: [‘CMD’, ‘wget’, ‘-qO-‘, ‘http:\/\/localhost:8080\/health’]
\n interval: 3s
\n timeout: 3s
\n retries: 3
\n networks:
\n – app-network
\n depends_on:
\n – llm\n

frontend:
\n build:
\n context: .\/frontend
\n ports:
\n – ‘3000:3000’
\n depends_on:
\n backend:
\n condition: service_healthy
\n networks:
\n – app-network<\/p>\n

prometheus:
\n image: prom\/prometheus:v2.45.0
\n volumes:
\n – .\/prometheus\/prometheus.yml:\/etc\/prometheus\/prometheus.yml
\n command:
\n – ‘–config.file=\/etc\/prometheus\/prometheus.yml’
\n – ‘–storage.tsdb.path=\/prometheus’
\n – ‘–web.console.libraries=\/etc\/prometheus\/console_libraries’
\n – ‘–web.console.templates=\/etc\/prometheus\/consoles’
\n – ‘–web.enable-lifecycle’
\n ports:
\n – ‘9091:9090’
\n networks:
\n – app-network<\/p>\n

grafana:
\n image: grafana\/grafana:10.1.0
\n volumes:
\n – .\/grafana\/provisioning:\/etc\/grafana\/provisioning
\n – grafana-data:\/var\/lib\/grafana
\n environment:
\n – GF_SECURITY_ADMIN_PASSWORD=admin
\n – GF_USERS_ALLOW_SIGN_UP=false
\n – GF_SERVER_DOMAIN=localhost
\n ports:
\n – ‘3001:3000’
\n depends_on:
\n – prometheus
\n networks:
\n – app-network<\/p>\n

jaeger:
\n image: jaegertracing\/all-in-one:1.46
\n environment:
\n – COLLECTOR_ZIPKIN_HOST_PORT=:9411
\n ports:
\n – ‘16686:16686’ # UI
\n – ‘4317:4317’ # OTLP gRPC
\n – ‘4318:4318’ # OTLP HTTP
\n networks:
\n – app-network<\/p>\n

# New LLM service using Docker Compose’s model provider
\n llm:
\n provider:
\n type: model
\n options:
\n model: ${LLM_MODEL_NAME:-ai\/llama3.2:1B-Q8_0}<\/p>\n

volumes:
\n grafana-data:<\/p>\n

networks:
\n app-network:
\n driver: bridge<\/p>\n<\/div>\n

This compose.yml defines a complete microservices architecture for the application with integrated observability tools and Model Runner support:<\/p>\n

backend<\/strong><\/p>\n

Go-based API server with Docker socket access for container management<\/p>\n

Implements health checks and exposes both API (8080) and metrics (9090) ports<\/p>\n

frontend<\/strong><\/p>\n

React-based user interface for an interactive chat experience<\/p>\n

Waits for backend health before starting to ensure system reliability<\/p>\n

prometheus<\/strong><\/p>\n

Time-series metrics database for collecting and storing performance data<\/p>\n

Configured with custom settings for monitoring application behavior<\/p>\n

grafana<\/strong><\/p>\n

Data visualization platform for metrics with persistent dashboard storage<\/p>\n

Pre-configured with admin access and connected to the Prometheus data source<\/p>\n

jaeger<\/strong><\/p>\n

Distributed tracing system for visualizing request flows across services<\/p>\n

Supports multiple protocols (gRPC\/HTTP) with UI on port 16686<\/p>\n

How Docker Model Runner integration works<\/h3>\n

The project integrates with Docker Model Runner through the following mechanisms:<\/p>\n

Connection Configuration<\/strong>:<\/p>\n

Using internal DNS: http:\/\/model-runner.docker.internal\/engines\/llama.cpp\/v1\/<\/p>\n

Using TCP via host-side support: localhost:12434<\/p>\n

Docker\u2019s Host Networking<\/strong>:<\/p>\n

The extra_hosts configuration maps host.docker.internal to the host\u2019s gateway IP<\/p>\n

Environment Variables<\/strong>:<\/p>\n

BASE_URL: URL for the model runner<\/p>\n

MODEL: Model identifier (e.g., ai\/llama3.2:1B-Q8_0)<\/p>\n

API Communication<\/strong>:<\/p>\n

The backend formats messages and sends them to Docker Model Runner<\/p>\n

It then streams tokens back to the frontend in real-time<\/p>\n

Why this approach excels<\/h3>\n

Building GenAI applications with Docker Model Runner and comprehensive observability offers several advantages:<\/p>\n

Privacy and Security<\/strong>: All data stays on your local infrastructure<\/p>\n

Cost Control<\/strong>: No per-token or per-request API charges<\/p>\n

Performance Insights<\/strong>: Deep visibility into model behavior and efficiency<\/p>\n

Developer Experience<\/strong>: Familiar Docker-based workflow with powerful monitoring<\/p>\n

Flexibility<\/strong>: Easy to experiment with different models and configurations<\/p>\n

Conclusion<\/h2>\n

The genai-model-runner-metrics project demonstrates a powerful approach to building AI-powered applications with Docker Model Runner while maintaining visibility into performance characteristics. By combining local model execution with comprehensive metrics, you get the best of both worlds: the privacy and cost benefits of local execution with the observability needed for production applications.<\/p>\n

Whether you\u2019re building a customer support bot, a content generation tool, or a specialized AI assistant, this architecture provides the foundation for reliable, observable, and efficient AI applications. The metrics-driven approach ensures you can continuously monitor and optimize your application, leading to better user experiences and more efficient resource utilization.<\/p>\n

Ready to get started? Clone the repository, fire up Docker Desktop<\/a>, and experience the future of AI development \u2014 your own local, metrics-driven GenAI application is just a docker compose up away!<\/p>\n\n\n

Learn more<\/h3>\n

Read our quickstart guide to Docker Model Runner<\/a>.<\/p>\n

Find documentation for Model Runner<\/a>.<\/p>\n

Subscribe to the Docker Navigator Newsletter<\/a>.<\/p>\n

New to Docker? Create an account<\/a>.\u00a0<\/p>\n

Have questions? The Docker community is here to help<\/a>.<\/p>","protected":false},"excerpt":{"rendered":"

Today, we\u2019ll show you how to build a fully functional Generative AI chatbot using Docker Model Runner and powerful observability […]<\/p>\n","protected":false},"author":0,"featured_media":0,"comment_status":"","ping_status":"","sticky":false,"template":"","format":"standard","meta":{"site-sidebar-layout":"default","site-content-layout":"","ast-site-content-layout":"default","site-content-style":"default","site-sidebar-style":"default","ast-global-header-display":"","ast-banner-title-visibility":"","ast-main-header-display":"","ast-hfb-above-header-display":"","ast-hfb-below-header-display":"","ast-hfb-mobile-header-display":"","site-post-title":"","ast-breadcrumbs-content":"","ast-featured-img":"","footer-sml-layout":"","ast-disable-related-posts":"","theme-transparent-header-meta":"","adv-header-id-meta":"","stick-header-meta":"","header-above-stick-meta":"","header-main-stick-meta":"","header-below-stick-meta":"","astra-migrate-meta-layouts":"default","ast-page-background-enabled":"default","ast-page-background-meta":{"desktop":{"background-color":"var(--ast-global-color-4)","background-image":"","background-repeat":"repeat","background-position":"center center","background-size":"auto","background-attachment":"scroll","background-type":"","background-media":"","overlay-type":"","overlay-color":"","overlay-opacity":"","overlay-gradient":""},"tablet":{"background-color":"","background-image":"","background-repeat":"repeat","background-position":"center center","background-size":"auto","background-attachment":"scroll","background-type":"","background-media":"","overlay-type":"","overlay-color":"","overlay-opacity":"","overlay-gradient":""},"mobile":{"background-color":"","background-image":"","background-repeat":"repeat","background-position":"center center","background-size":"auto","background-attachment":"scroll","background-type":"","background-media":"","overlay-type":"","overlay-color":"","overlay-opacity":"","overlay-gradient":""}},"ast-content-background-meta":{"desktop":{"background-color":"var(--ast-global-color-5)","background-image":"","background-repeat":"repeat","background-position":"center center","background-size":"auto","background-attachment":"scroll","background-type":"","background-media":"","overlay-type":"","overlay-color":"","overlay-opacity":"","overlay-gradient":""},"tablet":{"background-color":"var(--ast-global-color-5)","background-image":"","background-repeat":"repeat","background-position":"center center","background-size":"auto","background-attachment":"scroll","background-type":"","background-media":"","overlay-type":"","overlay-color":"","overlay-opacity":"","overlay-gradient":""},"mobile":{"background-color":"var(--ast-global-color-5)","background-image":"","background-repeat":"repeat","background-position":"center center","background-size":"auto","background-attachment":"scroll","background-type":"","background-media":"","overlay-type":"","overlay-color":"","overlay-opacity":"","overlay-gradient":""}},"footnotes":""},"categories":[4],"tags":[],"class_list":["post-2088","post","type-post","status-publish","format-standard","hentry","category-docker"],"_links":{"self":[{"href":"https:\/\/rssfeedtelegrambot.bnaya.co.il\/index.php\/wp-json\/wp\/v2\/posts\/2088","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/rssfeedtelegrambot.bnaya.co.il\/index.php\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/rssfeedtelegrambot.bnaya.co.il\/index.php\/wp-json\/wp\/v2\/types\/post"}],"replies":[{"embeddable":true,"href":"https:\/\/rssfeedtelegrambot.bnaya.co.il\/index.php\/wp-json\/wp\/v2\/comments?post=2088"}],"version-history":[{"count":0,"href":"https:\/\/rssfeedtelegrambot.bnaya.co.il\/index.php\/wp-json\/wp\/v2\/posts\/2088\/revisions"}],"wp:attachment":[{"href":"https:\/\/rssfeedtelegrambot.bnaya.co.il\/index.php\/wp-json\/wp\/v2\/media?parent=2088"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/rssfeedtelegrambot.bnaya.co.il\/index.php\/wp-json\/wp\/v2\/categories?post=2088"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/rssfeedtelegrambot.bnaya.co.il\/index.php\/wp-json\/wp\/v2\/tags?post=2088"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}