{"id":1074,"date":"2024-07-23T18:10:38","date_gmt":"2024-07-23T18:10:38","guid":{"rendered":"https:\/\/rssfeedtelegrambot.bnaya.co.il\/index.php\/2024\/07\/23\/local-llm-messenger-chat-with-genai-on-your-iphone\/"},"modified":"2024-07-23T18:10:38","modified_gmt":"2024-07-23T18:10:38","slug":"local-llm-messenger-chat-with-genai-on-your-iphone","status":"publish","type":"post","link":"https:\/\/rssfeedtelegrambot.bnaya.co.il\/index.php\/2024\/07\/23\/local-llm-messenger-chat-with-genai-on-your-iphone\/","title":{"rendered":"Local LLM Messenger: Chat with GenAI on Your iPhone"},"content":{"rendered":"

In this AI\/ML Hackathon post<\/a>, we want to share another winning project from last year\u2019s Docker AI\/ML Hackathon<\/a>. This time we will dive into Local LLM Messenger<\/a>, an honorable mention winner created by Justin Garrison<\/a>.<\/p>\n

Developers are pushing the boundaries to bring the power of artificial intelligence (AI) to everyone. One exciting approach involves integrating Large Language Models (LLMs) with familiar messaging platforms like Slack and iMessage. This isn\u2019t just about convenience; it\u2019s about transforming these platforms into launchpads for interacting with powerful AI tools.<\/p>\n

Imagine this: You need a quick code snippet or some help brainstorming solutions to coding problems. With LLMs integrated into your messaging app, you can chat with your AI assistant directly within the familiar interface to generate creative ideas or get help brainstorming solutions. No more complex commands or clunky interfaces \u2014 just a natural conversation to unlock the power of AI.<\/p>\n

Integrating with messaging platforms can be a time-consuming task, especially for macOS users. That\u2019s where Local LLM Messenger (LoLLMM) steps in, offering a streamlined solution for connecting with your AI via iMessage.\u00a0<\/p>\n

What makes LoLLM Messenger unique?<\/h2>\n

The following demo, which was submitted to the AI\/ML Hackathon, provides an overview of LoLLM Messenger (Figure 1).<\/p>\n

\n<\/div>\n

Figure 1: <\/strong>Demo of LoLLM Messenger as submitted to the AI\/ML Hackathon.<\/p>\n

The LoLLM Messenger bot allows you to send iMessages to Generative AI (GenAI) models running directly on your computer. This approach eliminates the need for complex setups and cloud services, making it easier for developers to experiment with LLMs locally.<\/p>\n

Key features of LoLLM Messenger<\/h2>\n

LoLLM Messenger includes impressive features that make it a standout among similar projects, such as:<\/p>\n

Local execution<\/strong>: Runs on your computer, eliminating the need for cloud-based services and ensuring data privacy.<\/p>\n

Scalability<\/strong>: Handles multiple AI models simultaneously, allowing users to experiment with different models and switch between them easily.<\/p>\n

User-friendly interface<\/strong>: Offers a simple and intuitive interface, making it accessible to users of all skill levels.<\/p>\n

Integration with Sendblue<\/strong>: Integrates seamlessly with Sendblue<\/a>, enabling users to send iMessages to the bot and receive responses directly in their inbox.<\/p>\n

Support for ChatGPT<\/strong>: Supports the GPT-3.5 Turbo and DALL-E 2 models, providing users with access to powerful AI capabilities.<\/p>\n

Customization:<\/strong> Allows users to customize the bot\u2019s behavior by modifying the available commands and integrating their own AI models.<\/p>\n

How does it work?<\/h2>\n

The architecture diagram shown in Figure 2 provides a high-level overview of the components and interactions within the LoLLM Messenger project. It illustrates how the main application, AI models, messaging platform, and external APIs work together to enable users to send iMessages to AI models running on their computers.<\/p>\n

<\/a>Figure 2:<\/strong> Overview of the components and interactions in the LoLLM Messenger project.<\/p>\n

By leveraging Docker, Sendblue, and Ollama, LoLLM Messenger offers a seamless and efficient solution for those seeking to explore AI models without the need for cloud-based services. LoLLM Messenger utilizes Docker Compose<\/a> to manage the required services.\u00a0<\/p>\n

Docker Compose simplifies the process by handling the setup and configuration of multiple containers, including the main application, ngrok (for creating a secure tunnel), and Ollama (a server that bridges the gap between messaging apps and AI models).<\/p>\n

Technical stack<\/h2>\n

The LoLLM Messenger tech stack includes:<\/p>\n

Lollmm service<\/strong>: This service is responsible for running the main application. It handles incoming iMessages, processing user requests, and interacting with the AI models. The lollmm service communicates with the Ollama model, which is a powerful AI model for text and image generation.<\/p>\n

Ngrok<\/strong>: This service is used to expose the main application\u2019s port 8000 to the internet using ngrok. It runs in the Alpine image and forwards traffic from port 8000 to the ngrok tunnel. The service is set to run in the host network mode.<\/p>\n

Ollama:<\/strong> This service runs the Ollama model, which is a powerful AI model for text and image generation. It listens on port 11434 and mounts a volume from .\/run\/ollama to \/home\/ollama. The service is set to deploy with GPU resources, ensuring that it can utilize an NVIDIA GPU if available.<\/p>\n

Sendblue:<\/strong> The project integrates with Sendblue to handle iMessages. You can set up Sendblue by adding your API Key and API Secret in the app\/.env file and adding your phone number as a Sendblue contact.<\/p>\n

Getting started<\/h2>\n

To get started, ensure that you have installed and set up the following components:<\/p>\n

Install the latest Docker Desktop<\/a>.<\/p>\n

Register for Sendblue https:\/\/app.sendblue.co\/auth\/login<\/a>.\u00a0<\/p>\n

Create an ngrok account using your preferred way and get authtoken https:\/\/dashboard.ngrok.com\/signup<\/a>.<\/p>\n

Clone the repository<\/h2>\n

Open a terminal window and run the following command to clone this sample application:<\/p>\n

\ngit clone https:\/\/github.com\/dockersamples\/local-llm-messenger\n<\/div>\n

You should now have the following files in your local-llm-messenger directory:<\/p>\n

\n.
\n\u251c\u2500\u2500 LICENSE
\n\u251c\u2500\u2500 README.md
\n\u251c\u2500\u2500 app
\n\u2502 \u251c\u2500\u2500 Dockerfile
\n\u2502 \u251c\u2500\u2500 Pipfile
\n\u2502 \u251c\u2500\u2500 Pipfile.lock
\n\u2502 \u251c\u2500\u2500 default.ai
\n\u2502 \u251c\u2500\u2500 log_conf.yaml
\n\u2502 \u2514\u2500\u2500 main.py
\n\u251c\u2500\u2500 docker-compose.yaml
\n\u251c\u2500\u2500 img
\n\u2502 \u251c\u2500\u2500 banner.png
\n\u2502 \u251c\u2500\u2500 lasers.gif
\n\u2502 \u2514\u2500\u2500 lollm-demo-1.gif
\n\u251c\u2500\u2500 justfile
\n\u2514\u2500\u2500 test
\n \u251c\u2500\u2500 msg.json
\n \u2514\u2500\u2500 ollama.json\n

4 directories, 15 files\n<\/p><\/div>\n

The script main.py file under the \/app directory is a Python script that uses the FastAPI framework to create a web server for an AI-powered messaging application. The script interacts with OpenAI\u2019s GPT-3 model and an Ollama endpoint for generating responses. It uses Sendblue\u2019s API for sending messages.<\/p>\n

The script first imports necessary libraries, including FastAPI, requests, logging, and other required modules.<\/p>\n

\nfrom dotenv import load_dotenv
\nimport os, requests, time, openai, json, logging
\nfrom pprint import pprint
\nfrom typing import Union, List\n

from fastapi import FastAPI
\nfrom pydantic import BaseModel<\/p>\n

from sendblue import Sendblue\n<\/p><\/div>\n

This section sets up configuration variables, such as API keys, callback URL, Ollama API endpoint, and maximum context and word limits.<\/p>\n

\nSENDBLUE_API_KEY = os.environ.get(“SENDBLUE_API_KEY”)
\nSENDBLUE_API_SECRET = os.environ.get(“SENDBLUE_API_SECRET”)
\nopenai.api_key = os.environ.get(“OPENAI_API_KEY”)
\nOLLAMA_API = os.environ.get(“OLLAMA_API_ENDPOINT”, “http:\/\/ollama:11434\/api”)
\n# could also use request.headers.get(‘referer’) to do dynamically
\nCALLBACK_URL = os.environ.get(“CALLBACK_URL”)
\nMAX_WORDS = os.environ.get(“MAX_WORDS”)\n<\/div>\n

Next, the script performs the logging configuration, setting the log level to INFO. Creates a file handler for logging messages to a file named app.log.<\/p>\n

It then defines various functions for interacting with the AI models, managing context, sending messages, handling callbacks, and executing slash commands.<\/p>\n

\ndef set_default_model(model: str):
\n try:
\n with open(“default.ai”, “w”) as f:
\n f.write(model)
\n f.close()
\n return
\n except IOError:
\n logger.error(“Could not open file”)
\n exit(1)\n

def get_default_model() -> str:
\n try:
\n with open(“default.ai”) as f:
\n default = f.readline().strip(“n”)
\n f.close()
\n if default != “”:
\n return default
\n else:
\n set_default_model(“llama2:latest”)
\n return “”
\n except IOError:
\n logger.error(“Could not open file”)
\n exit(1)<\/p>\n

def validate_model(model: str) -> bool:
\n available_models = get_model_list()
\n if model in available_models:
\n return True
\n else:
\n return False<\/p>\n

def get_ollama_model_list() -> List[str]:
\n available_models = []<\/p>\n

tags = requests.get(OLLAMA_API + “\/tags”)
\n all_models = json.loads(tags.text)
\n for model in all_models[“models”]:
\n available_models.append(model[“name”])
\n return available_models<\/p>\n

def get_openai_model_list() -> List[str]:
\n return [“gpt-3.5-turbo”, “dall-e-2”]<\/p>\n

def get_model_list() -> List[str]:
\n ollama_models = []
\n openai_models = []
\n all_models = []
\n if “OPENAI_API_KEY” in os.environ:
\n # print(openai.Model.list())
\n openai_models = get_openai_model_list()<\/p>\n

ollama_models = get_ollama_model_list()
\n all_models = ollama_models + openai_models
\n return all_models<\/p>\n

DEFAULT_MODEL = get_default_model()<\/p>\n

if DEFAULT_MODEL == “”:
\n # This is probably the first run so we need to install a model
\n if “OPENAI_API_KEY” in os.environ:
\n print(“No default model set. openai is enabled. using gpt-3.5-turbo”)
\n DEFAULT_MODEL = “gpt-3.5-turbo”
\n else:
\n print(“No model found and openai not enabled. Installing llama2:latest”)
\n pull_data = ‘{“name”: “llama2:latest”,”stream”: false}’
\n try:
\n pull_resp = requests.post(OLLAMA_API + “\/pull”, data=pull_data)
\n pull_resp.raise_for_status()
\n except requests.exceptions.HTTPError as err:
\n raise SystemExit(err)
\n set_default_model(“llama2:latest”)
\n DEFAULT_MODEL = “llama2:latest”<\/p>\n

if validate_model(DEFAULT_MODEL):
\n logger.info(“Using model: ” + DEFAULT_MODEL)
\nelse:
\n logger.error(“Model ” + DEFAULT_MODEL + ” not available.”)
\n logger.info(get_model_list())<\/p>\n

pull_data = ‘{“name”: “‘ + DEFAULT_MODEL + ‘”,”stream”: false}’
\n try:
\n pull_resp = requests.post(OLLAMA_API + “\/pull”, data=pull_data)
\n pull_resp.raise_for_status()
\n except requests.exceptions.HTTPError as err:
\n raise SystemExit(err)<\/p>\n

def set_msg_send_style(received_msg: str):
\n “””Will return a style for the message to send based on matched words in received message”””
\n celebration_match = [“happy”]
\n shooting_star_match = [“star”, “stars”]
\n fireworks_match = [“celebrate”, “firework”]
\n lasers_match = [“cool”, “lasers”, “laser”]
\n love_match = [“love”]
\n confetti_match = [“yay”]
\n balloons_match = [“party”]
\n echo_match = [“what did you say”]
\n invisible_match = [“quietly”]
\n gentle_match = []
\n loud_match = [“hear”]
\n slam_match = []<\/p>\n

received_msg_lower = received_msg.lower()
\n if any(x in received_msg_lower for x in celebration_match):
\n return “celebration”
\n elif any(x in received_msg_lower for x in shooting_star_match):
\n return “shooting_star”
\n elif any(x in received_msg_lower for x in fireworks_match):
\n return “fireworks”
\n elif any(x in received_msg_lower for x in lasers_match):
\n return “lasers”
\n elif any(x in received_msg_lower for x in love_match):
\n return “love”
\n elif any(x in received_msg_lower for x in confetti_match):
\n return “confetti”
\n elif any(x in received_msg_lower for x in balloons_match):
\n return “balloons”
\n elif any(x in received_msg_lower for x in echo_match):
\n return “echo”
\n elif any(x in received_msg_lower for x in invisible_match):
\n return “invisible”
\n elif any(x in received_msg_lower for x in gentle_match):
\n return “gentle”
\n elif any(x in received_msg_lower for x in loud_match):
\n return “loud”
\n elif any(x in received_msg_lower for x in slam_match):
\n return “slam”
\n else:
\n return\n<\/p><\/div>\n

Two classes, Msg and Callback, are defined to represent the structure of incoming messages and callback data. The code also includes various functions and classes to handle different aspects of the messaging platform, such as setting default models, validating models, interacting with the Sendblue API, and processing messages. It also includes functions to handle slash commands, create messages from context, and append context to a file.<\/p>\n

\nclass Msg(BaseModel):
\n accountEmail: str
\n content: str
\n media_url: str
\n is_outbound: bool
\n status: str
\n error_code: int | None = None
\n error_message: str | None = None
\n message_handle: str
\n date_sent: str
\n date_updated: str
\n from_number: str
\n number: str
\n to_number: str
\n was_downgraded: bool | None = None
\n plan: str\n

class Callback(BaseModel):
\n accountEmail: str
\n content: str
\n is_outbound: bool
\n status: str
\n error_code: int | None = None
\n error_message: str | None = None
\n message_handle: str
\n date_sent: str
\n date_updated: str
\n from_number: str
\n number: str
\n to_number: str
\n was_downgraded: bool | None = None
\n plan: str<\/p>\n

def msg_openai(msg: Msg, model=”gpt-3.5-turbo”):
\n “””Sends a message to openai”””
\n message_with_context = create_messages_from_context(“openai”)<\/p>\n

# Add the user’s message and system context to the messages list
\n messages = [
\n {“role”: “user”, “content”: msg.content},
\n {“role”: “system”, “content”: “You are an AI assistant. You will answer in haiku.”},
\n ]<\/p>\n

# Convert JSON strings to Python dictionaries and add them to messages
\n messages.extend(
\n [
\n json.loads(line) # Convert each JSON string back into a dictionary
\n for line in message_with_context
\n ]
\n )<\/p>\n

# Send the messages to the OpenAI model
\n gpt_resp = client.chat.completions.create(
\n model=model,
\n messages=messages,
\n )<\/p>\n

# Append the system context to the context file
\n append_context(“system”, gpt_resp.choices[0].message.content)<\/p>\n

# Send a message to the sender
\n msg_response = sendblue.send_message(
\n msg.from_number,
\n {
\n “content”: gpt_resp.choices[0].message.content,
\n “status_callback”: CALLBACK_URL,
\n },
\n )<\/p>\n

return<\/p>\n

def msg_ollama(msg: Msg, model=None):
\n “””Sends a message to the ollama endpoint”””
\n if model is None:
\n logger.error(“Model is None when calling msg_ollama”)
\n return # Optionally handle the case more gracefully<\/p>\n

ollama_headers = {“Content-Type”: “application\/json”}
\n ollama_data = (
\n ‘{“model”:”‘ + model +
\n ‘”, “stream”: false, “prompt”:”‘ +
\n msg.content +
\n ” in under ” +
\n str(MAX_WORDS) + # Make sure MAX_WORDS is a string
\n ‘ words”}’
\n )
\n ollama_resp = requests.post(
\n OLLAMA_API + “\/generate”, headers=ollama_headers, data=ollama_data
\n )
\n response_dict = json.loads(ollama_resp.text)
\n if ollama_resp.ok:
\n send_style = set_msg_send_style(msg.content)
\n append_context(“system”, response_dict[“response”])
\n msg_response = sendblue.send_message(
\n msg.from_number,
\n {
\n “content”: response_dict[“response”],
\n “status_callback”: CALLBACK_URL,
\n “send_style”: send_style,
\n },
\n )
\n else:
\n msg_response = sendblue.send_message(
\n msg.from_number,
\n {
\n “content”: “I’m sorry, I had a problem processing that question. Please try again.”,
\n “status_callback”: CALLBACK_URL,
\n },
\n )
\n return\n<\/p><\/div>\n

Navigate to the app\/ directory and create a new file for adding environment variables.<\/p>\n

\ntouch .env
\nSENDBLUE_API_KEY=your_sendblue_api_key
\nSENDBLUE_API_SECRET=your_sendblue_api_secret
\nOLLAMA_API_ENDPOINT=http:\/\/host.docker.internal:11434\/api
\nOPENAI_API_KEY=your_openai_api_key\n<\/div>\n

Next, add the ngrok authtoken to the Docker Compose file. You can get the authtoken from this link<\/a>.<\/p>\n

\nservices:
\n lollm:
\n build: .\/app
\n # command:
\n # – sleep
\n # – 1d
\n ports:
\n – 8000:8000
\n env_file: .\/app\/.env
\n volumes:
\n – .\/run\/lollm:\/run\/lollm
\n depends_on:
\n – ollama
\n restart: unless-stopped
\n network_mode: “host”
\n ngrok:
\n image: ngrok\/ngrok:alpine
\n command:
\n – “http”
\n – “8000”
\n – “–log”
\n – “stdout”
\n environment:
\n – NGROK_AUTHTOKEN=2i6iXXXXXXXXhpqk1aY1
\n network_mode: “host”
\n ollama:
\n image: ollama\/ollama
\n ports:
\n – 11434:11434
\n volumes:
\n – .\/run\/ollama:\/home\/ollama
\n network_mode: “host”\n<\/div>\n

Running the application stack<\/h2>\n

Next, you can run the application stack, as follows:<\/p>\n

\n$ docker compose up\n<\/div>\n

You will see output similar to the following:<\/p>\n

\n[+] Running 4\/4
\n \u2714 Container local-llm-messenger-ollama-1 Create… 0.0s
\n \u2714 Container local-llm-messenger-ngrok-1 Created 0.0s
\n \u2714 Container local-llm-messenger-lollm-1 Recreat… 0.1s
\n ! lollm Published ports are discarded when using host network mode 0.0s
\nAttaching to lollm-1, ngrok-1, ollama-1
\nollama-1 | 2024\/06\/20 03:14:46 routes.go:1011: INFO server config env=”map[OLLAMA_DEBUG:false OLLAMA_FLASH_ATTENTION:false OLLAMA_HOST:http:\/\/0.0.0.0:11434 OLLAMA_KEEP_ALIVE: OLLAMA_LLM_LIBRARY: OLLAMA_MAX_LOADED_MODELS:1 OLLAMA_MAX_QUEUE:512 OLLAMA_MAX_VRAM:0 OLLAMA_MODELS:\/root\/.ollama\/models OLLAMA_NOHISTORY:false OLLAMA_NOPRUNE:false OLLAMA_NUM_PARALLEL:1 OLLAMA_ORIGINS:[http:\/\/localhost https:\/\/localhost http:\/\/localhost:* https:\/\/localhost:* http:\/\/127.0.0.1 https:\/\/127.0.0.1 http:\/\/127.0.0.1:* https:\/\/127.0.0.1:* http:\/\/0.0.0.0 https:\/\/0.0.0.0 http:\/\/0.0.0.0:* https:\/\/0.0.0.0:* app:\/\/* file:\/\/* tauri:\/\/*] OLLAMA_RUNNERS_DIR: OLLAMA_TMPDIR:]”
\nollama-1 | time=2024-06-20T03:14:46.308Z level=INFO source=images.go:725 msg=”total blobs: 0″
\nollama-1 | time=2024-06-20T03:14:46.309Z level=INFO source=images.go:732 msg=”total unused blobs removed: 0″
\nollama-1 | time=2024-06-20T03:14:46.309Z level=INFO source=routes.go:1057 msg=”Listening on [::]:11434 (version 0.1.44)”
\nollama-1 | time=2024-06-20T03:14:46.309Z level=INFO source=payload.go:30 msg=”extracting embedded files” dir=\/tmp\/ollama2210839504\/runners
\nngrok-1 | t=2024-06-20T03:14:46+0000 lvl=info msg=”open config file” path=\/var\/lib\/ngrok\/ngrok.yml err=nil
\nngrok-1 | t=2024-06-20T03:14:46+0000 lvl=info msg=”open config file” path=\/var\/lib\/ngrok\/auth-config.yml err=nil
\nngrok-1 | t=2024-06-20T03:14:46+0000 lvl=info msg=”starting web service” obj=web addr=0.0.0.0:4040 allow_hosts=[]
\nngrok-1 | t=2024-06-20T03:14:46+0000 lvl=info msg=”client session established” obj=tunnels.session
\nngrok-1 | t=2024-06-20T03:14:46+0000 lvl=info msg=”tunnel session started” obj=tunnels.session
\nngrok-1 | t=2024-06-20T03:14:46+0000 lvl=info msg=”started tunnel” obj=tunnels name=command_line addr=http:\/\/localhost:8000 url=https:\/\/94e1-223-185-128-160.ngrok-free.app
\nollama-1 | time=2024-06-20T03:14:48.602Z level=INFO source=payload.go:44 msg=”Dynamic LLM libraries [cpu cuda_v11]”
\nollama-1 | time=2024-06-20T03:14:48.603Z level=INFO source=types.go:71 msg=”inference compute” id=0 library=cpu compute=”” driver=0.0 name=”” total=”7.7 GiB” available=”3.9 GiB”
\nlollm-1 | INFO: Started server process [1]
\nlollm-1 | INFO: Waiting for application startup.
\nlollm-1 | INFO: Application startup complete.
\nlollm-1 | INFO: Uvicorn running on http:\/\/0.0.0.0:8000 (Press CTRL+C to quit)
\nngrok-1 | t=2024-06-20T03:16:58+0000 lvl=info msg=”join connections” obj=join id=ce119162e042 l=127.0.0.1:8000 r=[2401:4900:8838:8063:f0b0:1866:e957:b3ba]:54384
\nlollm-1 | OLLAMA API IS http:\/\/host.docker.internal:11434\/api
\nlollm-1 | INFO: 2401:4900:8838:8063:f0b0:1866:e957:b3ba:0 – “GET \/ HTTP\/1.1” 200 OK\n<\/div>\n

If you\u2019re testing it on a system without an NVIDIA GPU, then you can skip the deploy attribute of the Compose file.\u00a0<\/p>\n

Watch the output for your ngrok endpoint. In our case, it shows: https:\/\/94e1-223-185-128-160.ngrok-free.app\/<\/a><\/p>\n

Next, append \/msg to the following ngrok webhooks URL:\u00a0https:\/\/94e1-223-185-128-160.ngrok-free.app\/<\/a><\/p>\n

Then, add it under the webhooks URL section on Sendblue and save it (Figure 3).\u00a0 The ngrok service is configured to expose the lollmm service on port 8000 and provide a secure tunnel to the public internet using the ngrok.io domain.\u00a0<\/p>\n

The ngrok service logs indicate that it has started the web service and established a client session with the tunnels. They also show that the tunnel session has started and has been successfully established with the lollmm service.<\/p>\n

The ngrok service is configured to use the specified ngrok authentication token, which is required to access the ngrok service. Overall, the ngrok service is running correctly and is able to establish a secure tunnel to the lollmm service.<\/p>\n

<\/a>Figure 3: <\/strong>Adding ngrok authentication token to webhooks.<\/p>\n

Ensure that there are no error logs when you run the ngrok container (Figure 4).<\/p>\n

<\/a>Figure 4: <\/strong>Checking the logs for errors.<\/p>\n

Ensure that the LoLLM Messenger container is actively up and running (Figure 5).<\/p>\n

<\/a>Figure 5: <\/strong>Ensure the LoLLM Messenger container is running. <\/p>\n

The logs show that the Ollama service has opened the specified port (11434) and is listening for incoming connections. The logs also indicate that the Ollama service has mounted the \/home\/ollama directory from the host machine to the \/home\/ollama directory within the container.<\/p>\n

Overall, the Ollama service is running correctly and is ready to provide AI models for inference.<\/p>\n

Testing the functionality<\/h2>\n

To test the functionality of the lollm service, you first need to add your contact number to the Sendblue dashboard. Then you should be able to send messages to the Sendblue number and observe the responses from the lollmm service (Figure 6).<\/p>\n

<\/a>Figure 6:<\/strong> Testing functionality of lollm service.<\/p>\n

The Sendblue platform will send HTTP requests to the \/msg endpoint of your lollmm service, and your lollmm service will process these requests and return the appropriate responses.<\/p>\n

The lollmm service is set up to listen on port 8000.<\/p>\n

The ngrok tunnel is started and provides a public URL, such as https:\/\/94e1-223-185-128-160.ngrok-free.app<\/a>.<\/p>\n

The lollmm service receives HTTP requests from the ngrok tunnel, including GET requests to the root path (\/) and other paths, such as \/favicon.ico, \/predict, \/mdg, and \/msg.<\/p>\n

The lollmm service responds to these requests with appropriate HTTP status codes, such as 200 OK<\/strong> for successful requests and 404 Not Found<\/strong> for requests to paths that do not exist.<\/p>\n

The ngrok tunnel logs the join connections, indicating that clients are connecting to the lollmm service through the ngrok tunnel.<\/p>\n

Figure 7: Sending requests and receiving responses. <\/p>\n

The first time you chat with LLM by typing \/list (Figure 7), you can check the logs as shown:<\/p>\n

\nngrok-1 | t=2024-07-09T02:34:30+0000 lvl=info msg=”join connections” obj=join id=12bd50a8030b l=127.0.0.1:8000 r=18.223.220.3:44370
\nlollm-1 | OLLAMA API IS http:\/\/host.docker.internal:11434\/api
\nlollm-1 | INFO: 18.223.220.3:0 – “POST \/msg HTTP\/1.1” 200 OK
\nngrok-1 | t=2024-07-09T02:34:53+0000 lvl=info msg=”join connections” obj=join id=259fda936691 l=127.0.0.1:8000 r=18.223.220.3:36712
\nlollm-1 | INFO: 18.223.220.3:0 – “POST \/msg HTTP\/1.1” 200 OK\n<\/div>\n

Next, let\u2019s install the codellama model by typing \/install codellama:latest (Figure 8).<\/p>\n

Figure 8: Installing the `codellama` model.<\/p>\n

You can see the following container logs once you set the default model to codellama:latest as shown:<\/p>\n

\nngrok-1 | t=2024-07-09T03:39:23+0000 lvl=info msg=”join connections” obj=join id=026d8fad5c87 l=127.0.0.1:8000 r=18.223.220.3:36282
\nlollm-1 | setting default model
\nlollm-1 | INFO: 18.223.220.3:0 – “POST \/msg HTTP\/1.1” 200 OK\n<\/div>\n

The lollmm service is running correctly and can handle HTTP requests from the ngrok tunnel. You can use the ngrok tunnel URL to test the functionality of the lollmm service by sending HTTP requests to the appropriate paths (Figure 9).<\/p>\n

Figure 9: Testing the messaging functionality.<\/p>\n

Conclusion<\/h2>\n

LoLLM Messenger is a valuable tool for developers and enthusiasts looking to push the boundaries of LLM integration within messaging apps. It allows developers to craft custom chatbots for specific needs, add real-time sentiment analysis to messages, or explore entirely new AI features in your messaging experience.\u00a0<\/p>\n

To get started, you can explore the LoLLM Messenger project on GitHub<\/a> and discover the potential of local LLM.<\/p>\n

Learn more<\/h2>\n

Subscribe to the Docker Newsletter<\/a>.\u00a0<\/p>\n

Read the AI\/ML Hackathon<\/a> collection.<\/p>\n

Get the latest release of Docker Desktop<\/a>.<\/p>\n

Vote on what\u2019s next! Check out our public roadmap<\/a>.<\/p>\n

Have questions? The Docker community is here to help<\/a>.<\/p>\n

New to Docker? Get started<\/a>.<\/p>","protected":false},"excerpt":{"rendered":"

In this AI\/ML Hackathon post, we want to share another winning project from last year\u2019s Docker AI\/ML Hackathon. This time […]<\/p>\n","protected":false},"author":0,"featured_media":0,"comment_status":"","ping_status":"","sticky":false,"template":"","format":"standard","meta":{"site-sidebar-layout":"default","site-content-layout":"","ast-site-content-layout":"default","site-content-style":"default","site-sidebar-style":"default","ast-global-header-display":"","ast-banner-title-visibility":"","ast-main-header-display":"","ast-hfb-above-header-display":"","ast-hfb-below-header-display":"","ast-hfb-mobile-header-display":"","site-post-title":"","ast-breadcrumbs-content":"","ast-featured-img":"","footer-sml-layout":"","ast-disable-related-posts":"","theme-transparent-header-meta":"","adv-header-id-meta":"","stick-header-meta":"","header-above-stick-meta":"","header-main-stick-meta":"","header-below-stick-meta":"","astra-migrate-meta-layouts":"default","ast-page-background-enabled":"default","ast-page-background-meta":{"desktop":{"background-color":"var(--ast-global-color-4)","background-image":"","background-repeat":"repeat","background-position":"center center","background-size":"auto","background-attachment":"scroll","background-type":"","background-media":"","overlay-type":"","overlay-color":"","overlay-opacity":"","overlay-gradient":""},"tablet":{"background-color":"","background-image":"","background-repeat":"repeat","background-position":"center center","background-size":"auto","background-attachment":"scroll","background-type":"","background-media":"","overlay-type":"","overlay-color":"","overlay-opacity":"","overlay-gradient":""},"mobile":{"background-color":"","background-image":"","background-repeat":"repeat","background-position":"center center","background-size":"auto","background-attachment":"scroll","background-type":"","background-media":"","overlay-type":"","overlay-color":"","overlay-opacity":"","overlay-gradient":""}},"ast-content-background-meta":{"desktop":{"background-color":"var(--ast-global-color-5)","background-image":"","background-repeat":"repeat","background-position":"center center","background-size":"auto","background-attachment":"scroll","background-type":"","background-media":"","overlay-type":"","overlay-color":"","overlay-opacity":"","overlay-gradient":""},"tablet":{"background-color":"var(--ast-global-color-5)","background-image":"","background-repeat":"repeat","background-position":"center center","background-size":"auto","background-attachment":"scroll","background-type":"","background-media":"","overlay-type":"","overlay-color":"","overlay-opacity":"","overlay-gradient":""},"mobile":{"background-color":"var(--ast-global-color-5)","background-image":"","background-repeat":"repeat","background-position":"center center","background-size":"auto","background-attachment":"scroll","background-type":"","background-media":"","overlay-type":"","overlay-color":"","overlay-opacity":"","overlay-gradient":""}},"footnotes":""},"categories":[4],"tags":[],"class_list":["post-1074","post","type-post","status-publish","format-standard","hentry","category-docker"],"_links":{"self":[{"href":"https:\/\/rssfeedtelegrambot.bnaya.co.il\/index.php\/wp-json\/wp\/v2\/posts\/1074","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/rssfeedtelegrambot.bnaya.co.il\/index.php\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/rssfeedtelegrambot.bnaya.co.il\/index.php\/wp-json\/wp\/v2\/types\/post"}],"replies":[{"embeddable":true,"href":"https:\/\/rssfeedtelegrambot.bnaya.co.il\/index.php\/wp-json\/wp\/v2\/comments?post=1074"}],"version-history":[{"count":0,"href":"https:\/\/rssfeedtelegrambot.bnaya.co.il\/index.php\/wp-json\/wp\/v2\/posts\/1074\/revisions"}],"wp:attachment":[{"href":"https:\/\/rssfeedtelegrambot.bnaya.co.il\/index.php\/wp-json\/wp\/v2\/media?parent=1074"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/rssfeedtelegrambot.bnaya.co.il\/index.php\/wp-json\/wp\/v2\/categories?post=1074"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/rssfeedtelegrambot.bnaya.co.il\/index.php\/wp-json\/wp\/v2\/tags?post=1074"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}