{"id":1510,"date":"2024-12-09T18:18:19","date_gmt":"2024-12-09T18:18:19","guid":{"rendered":"https:\/\/rssfeedtelegrambot.bnaya.co.il\/index.php\/2024\/12\/09\/using-local-ai-models-with-net-aspire\/"},"modified":"2024-12-09T18:18:19","modified_gmt":"2024-12-09T18:18:19","slug":"using-local-ai-models-with-net-aspire","status":"publish","type":"post","link":"https:\/\/rssfeedtelegrambot.bnaya.co.il\/index.php\/2024\/12\/09\/using-local-ai-models-with-net-aspire\/","title":{"rendered":"Using Local AI models with .NET Aspire"},"content":{"rendered":"

Using local AI models can be a great way to experiment on your own machine without needing to deploy resources to the cloud. In this post, we\u2019ll look at how use .NET Aspire with Ollama<\/a> to run AI models locally, while using the Microsoft.Extensions.AI abstractions to make it transition to cloud-hosted models on deployment.<\/p>\n

Setting up Ollama in .NET Aspire<\/h2>\n

We\u2019re going to need a way to use Ollama from our .NET Aspire application, and the easiest way to do that is using the Ollama hosting integration<\/a> from the .NET Aspire Community Toolkit. You can install the Ollama hosting integration from NuGet via the Visual Studio tooling, VS Code tooling, or the .NET CLI. Let\u2019s take a look at how to install the Ollama hosting integration via the command line into our app host project:<\/p>\n

dotnet add package CommunityToolkit.Aspire.Hosting.Ollama<\/p>\n

Once you\u2019ve installed the Ollama hosting integration, you can configure it in your Program.cs file. Here\u2019s an example of how you might configure the Ollama hosting integration:<\/p>\n

var ollama =
\n builder.AddOllama(“ollama”)
\n .WithDataVolume()
\n .WithOpenWebUI();<\/p>\n

Here, we\u2019ve used the AddOllama extension method to add the container to the app host. Since we\u2019re going to download some models, we\u2019re going to want to persist that data volume across container restarts (it means we don\u2019t have to pull several gigabytes of data every time we start the container!). Also, so we\u2019ve got a playground, we\u2019ll add the OpenWebUI container, which will give us a web interface to interact with the model outside of our app.<\/p>\n

Running a local AI model<\/h2>\n

The ollama resource that we created in the previous step is only going to be running the Ollama server, we still need to add some models to it, and we can do that with the AddModel method. Let\u2019s use the Llama 3.2 model<\/a>:<\/p>\n

var chat = ollama.AddModel(“chat”, “llama3.2”);<\/p>\n

If we wanted to use a variation of the model, or a specific tag, we could specify that in the AddModel method, such as ollama.AddModel(“chat”, “llama3.2:1b”) for the 1b tag of the Llama 3.2 model. Alternatively, if the model you\u2019re after isn\u2019t in the Ollama library, you can use the AddHuggingFaceModel method to add a model from the Hugging Face model hub.<\/p>\n

Now that we have our model, we can add it as a resource to any of the other services in the app host:<\/p>\n

builder.AddProject<Projects.MyApi>(“api”)
\n .WithReference(chat);<\/p>\n

When we run the app host project, the Ollama server will start up and download the model we specified (make sure you don\u2019t stop the app host before the download completes), and then we can use the model in our application. If you want the resources that depend on the model to wait until the model is downloaded, you can use the WaitFor method with the model reference:<\/p>\n

builder.AddProject<Projects.MyApi>(“api”)
\n .WithReference(chat)
\n .WaitFor(chat);<\/p>\n

In the above screenshot of the dashboard, we\u2019ll see that the model is being downloaded. The Ollama server is running but unhealthy because the model hasn\u2019t been downloaded yet, and the api resource hasn\u2019t started as it\u2019s waiting for the model to download and become healthy.<\/p>\n

Using the model in your application<\/h2>\n

With our API project set up to use the chat model, we can now use the OllamaSharp<\/a> library to connect to the Ollama server and interact with the model, and to do this, we\u2019ll use the OllamaSharp integration<\/a> from the .NET Aspire Community Toolkit:<\/p>\n

dotnet add package CommunityToolkit.Aspire.OllamaSharp<\/p>\n

This integration allows us to register the OllamaSharp client as the IChatClient or IEmbeddingsGenerator service from the Microsoft.Extensions.AI package<\/a>, which is an abstraction that means we could switch out the local Ollama server for a cloud-hosted option such as Azure OpenAI Service without changing the code using the client:<\/p>\n

builder.AddOllamaSharpChatClient(“chat”);<\/p>\n

Note: If you are using an embedding model and want to register the IEmbeddingsGenerator service, you can use the AddOllamaSharpEmbeddingsGenerator method instead.<\/p>\n

To make full use of the Microsoft.Extensions.AI pipeline, we can provide that service to the ChatClientBuilder:<\/p>\n

builder.AddKeyedOllamaSharpChatClient(“chat”);
\nbuilder.Services.AddChatClient(b => b
\n .UseFunctionInvocation()
\n .UseOpenTelemetry(configure: t => t.EnableSensitiveData = true)
\n .UseLogging()
\n \/\/ Use the OllamaSharp client
\n .Use(b.Services.GetRequiredKeyedService<IChatClient>(“chat”)));<\/p>\n

Lastly, we can inject the IChatClient into our route handler:<\/p>\n

app.MapPost(“\/chat”, async (IChatClient chatClient, string question) =>
\n{
\n var response = await chatClient.CompleteAsync(question);
\n return response.Message;
\n});<\/p>\n

Supporting cloud-hosted models<\/h2>\n

While Ollama is great as a local development tool, when it comes to deploying your application, you\u2019ll likely want to use a cloud-based AI service like Azure OpenAI Service. To handle this, we\u2019ll need to update the API project to register a different implementation of the IChatClient service when running in the cloud:<\/p>\n

if (builder.Environment.IsDevelopment())
\n{
\n builder.AddKeyedOllamaSharpChatClient(“chat”);
\n}
\nelse
\n{
\n builder.AddKeyedAzureOpenAIClient(“chat”);
\n}<\/p>\n

builder.Services.AddChatClient(b => b
\n .UseFunctionInvocation()
\n .UseOpenTelemetry(configure: t => t.EnableSensitiveData = true)
\n .UseLogging()
\n \/\/ Use the previously registered IChatClient, which is either Ollama or Azure OpenAI
\n .Use(b.Services.GetRequiredKeyedService<IChatClient>(“chat”)));<\/p>\n

Conclusion<\/h2>\n

In this post, we\u2019ve seen how, with only a few lines of code, we can set up an Ollama server with .NET Aspire, specify a model that we want to use, have it downloaded for us, and then integrated into a client application. We\u2019ve also seen how we can use the Microsoft.Extensions.AI abstractions to make it easy to switch between local and cloud-hosted models. This is a powerful way to experiment with AI models on your local machine before deploying them to the cloud.<\/p>\n

Check out the eShop sample application<\/a> for a full example of how to use Ollama with .NET Aspire.<\/p>\n

The post Using Local AI models with .NET Aspire<\/a> appeared first on .NET Blog<\/a>.<\/p>","protected":false},"excerpt":{"rendered":"

Using local AI models can be a great way to experiment on your own machine without needing to deploy resources […]<\/p>\n","protected":false},"author":0,"featured_media":0,"comment_status":"","ping_status":"","sticky":false,"template":"","format":"standard","meta":{"site-sidebar-layout":"default","site-content-layout":"","ast-site-content-layout":"default","site-content-style":"default","site-sidebar-style":"default","ast-global-header-display":"","ast-banner-title-visibility":"","ast-main-header-display":"","ast-hfb-above-header-display":"","ast-hfb-below-header-display":"","ast-hfb-mobile-header-display":"","site-post-title":"","ast-breadcrumbs-content":"","ast-featured-img":"","footer-sml-layout":"","ast-disable-related-posts":"","theme-transparent-header-meta":"","adv-header-id-meta":"","stick-header-meta":"","header-above-stick-meta":"","header-main-stick-meta":"","header-below-stick-meta":"","astra-migrate-meta-layouts":"default","ast-page-background-enabled":"default","ast-page-background-meta":{"desktop":{"background-color":"var(--ast-global-color-4)","background-image":"","background-repeat":"repeat","background-position":"center center","background-size":"auto","background-attachment":"scroll","background-type":"","background-media":"","overlay-type":"","overlay-color":"","overlay-opacity":"","overlay-gradient":""},"tablet":{"background-color":"","background-image":"","background-repeat":"repeat","background-position":"center center","background-size":"auto","background-attachment":"scroll","background-type":"","background-media":"","overlay-type":"","overlay-color":"","overlay-opacity":"","overlay-gradient":""},"mobile":{"background-color":"","background-image":"","background-repeat":"repeat","background-position":"center center","background-size":"auto","background-attachment":"scroll","background-type":"","background-media":"","overlay-type":"","overlay-color":"","overlay-opacity":"","overlay-gradient":""}},"ast-content-background-meta":{"desktop":{"background-color":"var(--ast-global-color-5)","background-image":"","background-repeat":"repeat","background-position":"center center","background-size":"auto","background-attachment":"scroll","background-type":"","background-media":"","overlay-type":"","overlay-color":"","overlay-opacity":"","overlay-gradient":""},"tablet":{"background-color":"var(--ast-global-color-5)","background-image":"","background-repeat":"repeat","background-position":"center center","background-size":"auto","background-attachment":"scroll","background-type":"","background-media":"","overlay-type":"","overlay-color":"","overlay-opacity":"","overlay-gradient":""},"mobile":{"background-color":"var(--ast-global-color-5)","background-image":"","background-repeat":"repeat","background-position":"center center","background-size":"auto","background-attachment":"scroll","background-type":"","background-media":"","overlay-type":"","overlay-color":"","overlay-opacity":"","overlay-gradient":""}},"footnotes":""},"categories":[7],"tags":[],"class_list":["post-1510","post","type-post","status-publish","format-standard","hentry","category-dotnet"],"_links":{"self":[{"href":"https:\/\/rssfeedtelegrambot.bnaya.co.il\/index.php\/wp-json\/wp\/v2\/posts\/1510","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/rssfeedtelegrambot.bnaya.co.il\/index.php\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/rssfeedtelegrambot.bnaya.co.il\/index.php\/wp-json\/wp\/v2\/types\/post"}],"replies":[{"embeddable":true,"href":"https:\/\/rssfeedtelegrambot.bnaya.co.il\/index.php\/wp-json\/wp\/v2\/comments?post=1510"}],"version-history":[{"count":0,"href":"https:\/\/rssfeedtelegrambot.bnaya.co.il\/index.php\/wp-json\/wp\/v2\/posts\/1510\/revisions"}],"wp:attachment":[{"href":"https:\/\/rssfeedtelegrambot.bnaya.co.il\/index.php\/wp-json\/wp\/v2\/media?parent=1510"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/rssfeedtelegrambot.bnaya.co.il\/index.php\/wp-json\/wp\/v2\/categories?post=1510"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/rssfeedtelegrambot.bnaya.co.il\/index.php\/wp-json\/wp\/v2\/tags?post=1510"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}