The landscape of generative AI development is evolving rapidly but comes with significant challenges. API usage costs can quickly add up, especially during development. Privacy concerns arise when sensitive data must be sent to external services. And relying on external APIs can introduce connectivity issues and latency.<\/p>\n
Enter Gemma 3 and <\/strong>Docker Model Runner<\/strong><\/a>, a powerful combination that brings state-of-the-art language models to your local environment, addressing these challenges head-on.<\/p>\n In this blog post, we\u2019ll explore how to run Gemma 3<\/a> locally using Docker Model Runner. We\u2019ll also walk through a practical case study: a Comment Processing System<\/a> that analyzes user feedback about a fictional AI assistant named Jarvis.<\/p>\n Before diving into the implementation, let\u2019s look at why local GenAI development is becoming increasingly important:<\/p>\n Cost efficiency<\/strong>: With no per-token or per-request charges, you can experiment freely without worrying about usage fees.<\/p>\n Data privacy<\/strong>: Sensitive data stays within your environment, with no third-party exposure.<\/p>\n Reduced network latency<\/strong>: Eliminates reliance on external APIs and enables offline use.<\/p>\n Full control<\/strong>: Run the model on your terms, with no intermediaries and full transparency.<\/p>\n Docker Model Runner provides an OpenAI-compatible API interface to run models locally. Here\u2019s how to set it up with Gemma 3<\/a>:<\/p>\n Once setup is complete, the OpenAI-compatible API provided by the Model Runner is available at: http:\/\/localhost:12434\/engines\/v1<\/a><\/p>\n To demonstrate the power of local GenAI development, we\u2019ve built a Comment Processing System<\/a> that leverages Gemma 3 for multiple NLP tasks. This system:<\/p>\n Generates synthetic user comments about a fictional AI assistant<\/p>\n Categorizes comments as positive, negative, or neutral<\/p>\n Clusters similar comments together using embeddings<\/p>\n Identifies potential product features from the comments<\/p>\n Generates contextually appropriate responses<\/p>\n All tasks are performed locally with no external API calls.<\/strong><\/p>\n To make this work, we configure the OpenAI SDK to point to the Docker Model Runner:<\/p>\n export default { \/\/ Initialize OpenAI client with local endpoint One key benefit of running models locally<\/a> is the ability to experiment freely with different configurations for each task without worrying about API costs or rate limits.<\/p>\n In our case:<\/p>\n Synthetic comment generation uses a higher temperature for creativity.<\/p>\n Categorization uses a lower temperature and a 10-token limit for consistency.<\/p>\n Clustering allows up to 20 tokens to improve semantic richness in embeddings.<\/p>\n This flexibility lets us iterate quickly, tune for performance, and tailor the model\u2019s behavior to each use case.<\/p>\n To simulate user feedback, we use Gemma 3\u2019s ability to follow detailed, context-aware prompts.<\/p>\n switch (type) { return `Generate a realistic ${sentiment} user comment about an AI assistant called Jarvis, focusing on its ${topic}.<\/p>\n The comment should sound natural, as if written by a real user who has been using Jarvis. Examples:<\/p>\n “Jarvis is seriously impressive \u2013 the speed at which it responds is incredible! I\u2019ve never used an AI assistant that\u2019s so quick and efficient, it\u2019s a game changer.\n<\/p><\/div>\n The ability to produce realistic feedback on demand is incredibly useful for simulating user data with zero API cost.<\/strong><\/p>\n We also use Gemma 3 to simulate polite, on-brand support responses to user comments. Here\u2019s the prompt logic:<\/p>\n Guidelines for responses: ${featuresContext}<\/p>\n Generate a polite, helpful response to this user comment.` Examples:<\/p>\n For a positive comment:<\/p>\n The Jarvis Team<\/p>\n<\/div>\n For a negative comment:<\/p>\n The Jarvis Team\n<\/p><\/div>\n This approach ensures a consistent, human-like support experience generated entirely locally.<\/strong><\/p>\n Beyond generating and responding to comments, we also use Gemma 3 to analyze user feedback and identify actionable insights. This helps simulate the role of a product analyst, surfacing recurring themes, user pain points, and opportunities for improvement.<\/p>\n Here, we provide a prompt instructing the model to identify up to three potential features or improvements based on a set of user comments.\u00a0<\/p>\n For each set of comments, identify up to 3 potential features or improvements that could address the user feedback.<\/p>\n For each feature, provide: Format your response as a JSON array of features, with each feature having the fields: name, description, type, and priority.` ${commentsText}` try { Here\u2019s an example of what the model might return:<\/p>\n And just like everything else in this project, it\u2019s generated locally with no external services<\/strong>.<\/p>\nThe power of local GenAI development<\/h2>\n
Setting up Docker Model Runner with Gemma 3<\/h2>\n
It is included in Docker Desktop for macOS, starting with version 4.40.0<\/a>.<\/p>\n
\ndocker model pull ai\/gemma3\n<\/div>\nCase study: Comment processing system<\/h2>\n
Implementation details<\/h2>\n
Configuring the OpenAI SDK to use local models<\/h3>\n
\n \/\/ Model configuration
\n openai: {
\n baseURL: “http:\/\/localhost:12434\/engines\/v1”, \/\/ Base URL for Docker Model Runner
\n apiKey: ‘ignored’,
\n model: “ai\/gemma3”,
\n commentGeneration: { \/\/ Each task has its own configuration, for example temperature is set to a high value when generating comments for creativity
\n temperature: 0.3,
\n max_tokens: 250,
\n n: 1,
\n },
\n embedding: {
\n model: “ai\/mxbai-embed-large”, \/\/ Model for generating embeddings
\n },
\n },
\n \/\/ … other configuration options
\n};<\/p>\n<\/div>\n
\nimport config from ‘.\/config.js’;\n
\nconst client = new OpenAI({
\n baseURL: config.openai.baseURL,
\n apiKey: config.openai.apiKey,
\n});<\/p>\n<\/div>\nTask-specific configuration<\/h3>\n
Generating synthetic comments<\/h3>\n
\n * Create a prompt for comment generation
\n * @param {string} type – Type of comment (positive, negative, neutral)
\n * @param {string} topic – Topic of the comment
\n * @returns {string} – Prompt for OpenAI
\n *\/
\nfunction createPromptForCommentGeneration(type, topic) {
\n let sentiment = ”;\n
\n case ‘positive’:
\n sentiment = ‘positive and appreciative’;
\n break;
\n case ‘negative’:
\n sentiment = ‘negative and critical’;
\n break;
\n case ‘neutral’:
\n sentiment = ‘neutral and balanced’;
\n break;
\n default:
\n sentiment = ‘general’;
\n }<\/p>\n
\nKeep the comment concise (1-3 sentences) and focused on the specific topic.
\nDo not include ratings (like “5\/5 stars”) or formatting.
\nJust return the comment text without any additional context or explanation.`;
\n}<\/p>\n<\/div>\nGenerating contextual responses<\/h3>\n
\n model: config.openai.model,
\n messages: [
\n {
\n role: “system”,
\n content: `You are a customer support representative for an AI assistant called Jarvis. Your task is to generate polite, helpful responses to user comments.\n
\n1. Be empathetic and acknowledge the user’s feedback
\n2. Thank the user for their input
\n3. If the comment is positive, express appreciation
\n4. If the comment is negative, apologize for the inconvenience and assure them you’re working on improvements
\n5. If the comment is neutral, acknowledge their observation
\n6. If relevant, mention that their feedback will be considered for future updates
\n7. Keep responses concise (2-4 sentences) and professional
\n8. Do not make specific promises about feature implementation or timelines
\n9. Sign the response as “The Jarvis Team”`
\n },
\n {
\n role: “user”,
\n content: `User comment: “${comment.text}”
\nComment category: ${comment.category || ‘unknown’}<\/p>\n
\n }
\n ],
\n temperature: 0.7,
\n max_tokens: 200
\n });<\/p>\n<\/div>\nExtracting product features from user feedback<\/h3>\n
\n * Extract features from comments
\n * @param {string} commentsText – Text of comments
\n * @returns {Promise<Array>} – Array of identified features
\n *\/
\nasync function extractFeaturesFromComments(commentsText) {
\n const response = await client.chat.completions.create({
\n model: config.openai.model,
\n messages: [
\n {
\n role: “system”,
\n content: `You are a product analyst for an AI assistant called Jarvis. Your task is to identify potential product features or improvements based on user comments.\n
\n1. A short name (2-5 words)
\n2. A brief description (1-2 sentences)
\n3. The type of feature (New Feature, Improvement, Bug Fix)
\n4. Priority (High, Medium, Low)<\/p>\n
\n },
\n {
\n role: “user”,
\n content: `Here are some user comments about Jarvis. Identify potential features or improvements based on these comments:<\/p>\n
\n }
\n ],
\n response_format: { type: “json_object” },
\n temperature: 0.5
\n });<\/p>\n
\n const result = JSON.parse(response.choices[0].message.content);
\n return result.features || [];
\n } catch (error) {
\n console.error(‘Error parsing feature identification response:’, error);
\n return [];
\n }
\n}<\/p>\n<\/div>\n
\n {
\n “name”: “Enhanced Visual Customization”,
\n “description”: “Allows users to personalize the Jarvis interface with more themes, icon styles, and display options to improve visual appeal and user preference.”,
\n “type”: “Improvement”,
\n “priority”: “Medium”,
\n “clusters”: [
\n “1”
\n ]
\n },\n<\/div>\nConclusion<\/h3>\n