One of the most interesting ways to enhance your existing applications with AI is to enable more ways for your users to interact with it. Today you probably handle text input, and perhaps some touch gestures for your power users. Now it\u2019s easier than ever to expand that to voice and vision, especially when your users\u2019 primary input is via mobile device.<\/p>\n
At Microsoft Build 2025 I demonstrated<\/a> expanding the .NET MAUI sample \u201cto do\u201d app from text input to supporting voice and vision when those capabilities are detected. Let me show you how .NET MAUI and our fantastic ecosystem of plugins makes this rather painless to do with a single implementation that works across all platforms starting with voice.<\/p>\n\n Being able to talk to an app isn\u2019t anything revolutionary. We\u2019ve all spoken to Siri, Alexa, and our dear Cortana a time or two, and the key is in knowing the keywords and recipes of things they can comprehend and act on. \u201cStart a timer\u201d, \u201cturn down the volume\u201d, \u201ctell me a joke\u201d, and everyone\u2019s favorite \u201cI wasn\u2019t talking to you\u201d.<\/p>\n The new and powerful capability we now have with large language models is having them take our unstructured ramblings and make sense of that in order to fit the structured format our apps expect and require.<\/p>\n The first thing to do is add the Plugin.Maui.Audio NuGet which helps us request permissions to the microphone and start capturing a stream. The plugin is also capable of playback.<\/p>\n dotnet add package Plugin.Maui.Audio –version 4.0.0<\/p>\n In MauiProgram.cs configure the recording settings and add the IAudioService from the plugin to the services container.<\/p>\n public static class MauiProgram builder.Services.AddSingleton<IAudioService, AudioService>(); Be sure to also review and implement any additional configuration steps in the documentation<\/a>.<\/p>\n Now the app is ready to capture some audio. In VoicePage the user will tap the microphone button, start speaking, and tap again to end the recording.<\/p>\n This is a trimmed version of the actual code<\/a> for starting and stopping the recording.<\/p>\n [RelayCommand] _recorder = _audioManager.CreateRecorder(); IsRecording = true; \/\/ more code<\/p>\n TranscribeAsync(); Once it has the audio stream it can start transcribing and processing it. (source<\/a>)<\/p>\n private async Task TranscribeAsync() if (_audioSource != null) Transcript = await _transcriber.TranscribeAsync(audioFilePath, CancellationToken.None);<\/p>\n await ExtractTasksAsync(); In this sample app, I used Microsoft.Extensions.AI with OpenAI to perform the transcription with the whisper-1 model trained specifically for this use case<\/a>. There are certainly other methods of doing this including on-device with SpeechToText<\/a> in the .NET MAUI Community Toolkit.<\/p>\n By using Microsoft.Extensions.AI I can easily swap out another cloud based AI service, use a local LLM with ONNX<\/a>, or later choose another on-device solution.<\/p>\n using Microsoft.Extensions.AI; namespace Telepathic.Services;<\/p>\n public class WhisperTranscriptionService : ITranscriptionService try return result.Value.Text.Trim(); Once I have the transcript, I can have my AI service make sense of it to return projects and tasks using the same client. This happens in the ExtractTasksAsync method referenced above. The key parts of this method are below. (source<\/a>)<\/p>\n private async Task ExtractTasksAsync() Here’s the transcript: {Transcript}”;<\/p>\n var chatClient = _chatClientService.GetClient(); if (response?.Result != null) The _chatClientService is an injected service class that handles the creation and retrieval of the IChatClient instance provided by Microsoft.Extensions.AI. Here I use the GetResponseAsync method along with passing a strong type and a prompt, and the LLM (gpt-4o-mini in this case) returns a ProjectsJson response. The response includes a Projects list with which I can proceed.<\/p>\n Now I\u2019ve gone from having an app that only took data entry input via a form, to an app that can also take unstructured voice input and produce structure data. While I was tempted to just insert the results into the database and claim success, there was yet more to do to make this a truly satisfying experience.<\/p>\n There\u2019s a reasonable chance that the project name needs to be adjusted for clarity, or some task was misheard or worse yet omitted. To address this, I add a step of approval where the use can see the projects and tasks as recommendations and choose to accept them as-is with changes. This is not much different than the experience we have now in Copilot when changes are make but we have the option to iterate further, keep, or discard.<\/p>\n For more guidance like this for designing great AI experiences in your apps, consider checking out the HAX Toolkit<\/a> and Microsoft AI Principles<\/a>.<\/p>\n Here are key resources mentioned in this article to help you implement multimodal AI capabilities in your .NET MAUI apps:<\/p>\n AI infused mobile & desktop app development with .NET MAUI <\/a> In this article, we explored how to enhance .NET MAUI applications with multimodal AI capabilities, focusing on voice interaction. We covered how to implement audio recording using Plugin.Maui.Audio<\/a>, transcribe speech using Microsoft.Extensions.AI<\/a> with OpenAI\u2019s Whisper model<\/a>, and extract structured data from unstructured voice input.<\/p>\n By combining these technologies, you can transform a traditional form-based app into one that accepts voice commands and intelligently processes them into actionable data. The implementation works across all platforms with a single codebase, making it accessible for any .NET MAUI developer.<\/p>\n With these techniques, you can significantly enhance user experience by supporting multiple interaction modes, making your applications more accessible and intuitive, especially on mobile devices where voice input can be much more convenient than typing.<\/p>\n The post Multimodal Voice Intelligence with .NET MAUI<\/a> appeared first on .NET Blog<\/a>.<\/p>","protected":false},"excerpt":{"rendered":" One of the most interesting ways to enhance your existing applications with AI is to enable more ways for your […]<\/p>\n","protected":false},"author":0,"featured_media":0,"comment_status":"","ping_status":"","sticky":false,"template":"","format":"standard","meta":{"site-sidebar-layout":"default","site-content-layout":"","ast-site-content-layout":"default","site-content-style":"default","site-sidebar-style":"default","ast-global-header-display":"","ast-banner-title-visibility":"","ast-main-header-display":"","ast-hfb-above-header-display":"","ast-hfb-below-header-display":"","ast-hfb-mobile-header-display":"","site-post-title":"","ast-breadcrumbs-content":"","ast-featured-img":"","footer-sml-layout":"","ast-disable-related-posts":"","theme-transparent-header-meta":"","adv-header-id-meta":"","stick-header-meta":"","header-above-stick-meta":"","header-main-stick-meta":"","header-below-stick-meta":"","astra-migrate-meta-layouts":"default","ast-page-background-enabled":"default","ast-page-background-meta":{"desktop":{"background-color":"var(--ast-global-color-4)","background-image":"","background-repeat":"repeat","background-position":"center center","background-size":"auto","background-attachment":"scroll","background-type":"","background-media":"","overlay-type":"","overlay-color":"","overlay-opacity":"","overlay-gradient":""},"tablet":{"background-color":"","background-image":"","background-repeat":"repeat","background-position":"center center","background-size":"auto","background-attachment":"scroll","background-type":"","background-media":"","overlay-type":"","overlay-color":"","overlay-opacity":"","overlay-gradient":""},"mobile":{"background-color":"","background-image":"","background-repeat":"repeat","background-position":"center center","background-size":"auto","background-attachment":"scroll","background-type":"","background-media":"","overlay-type":"","overlay-color":"","overlay-opacity":"","overlay-gradient":""}},"ast-content-background-meta":{"desktop":{"background-color":"var(--ast-global-color-5)","background-image":"","background-repeat":"repeat","background-position":"center center","background-size":"auto","background-attachment":"scroll","background-type":"","background-media":"","overlay-type":"","overlay-color":"","overlay-opacity":"","overlay-gradient":""},"tablet":{"background-color":"var(--ast-global-color-5)","background-image":"","background-repeat":"repeat","background-position":"center center","background-size":"auto","background-attachment":"scroll","background-type":"","background-media":"","overlay-type":"","overlay-color":"","overlay-opacity":"","overlay-gradient":""},"mobile":{"background-color":"var(--ast-global-color-5)","background-image":"","background-repeat":"repeat","background-position":"center center","background-size":"auto","background-attachment":"scroll","background-type":"","background-media":"","overlay-type":"","overlay-color":"","overlay-opacity":"","overlay-gradient":""}},"footnotes":""},"categories":[7],"tags":[],"class_list":["post-2124","post","type-post","status-publish","format-standard","hentry","category-dotnet"],"_links":{"self":[{"href":"https:\/\/rssfeedtelegrambot.bnaya.co.il\/index.php\/wp-json\/wp\/v2\/posts\/2124","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/rssfeedtelegrambot.bnaya.co.il\/index.php\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/rssfeedtelegrambot.bnaya.co.il\/index.php\/wp-json\/wp\/v2\/types\/post"}],"replies":[{"embeddable":true,"href":"https:\/\/rssfeedtelegrambot.bnaya.co.il\/index.php\/wp-json\/wp\/v2\/comments?post=2124"}],"version-history":[{"count":0,"href":"https:\/\/rssfeedtelegrambot.bnaya.co.il\/index.php\/wp-json\/wp\/v2\/posts\/2124\/revisions"}],"wp:attachment":[{"href":"https:\/\/rssfeedtelegrambot.bnaya.co.il\/index.php\/wp-json\/wp\/v2\/media?parent=2124"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/rssfeedtelegrambot.bnaya.co.il\/index.php\/wp-json\/wp\/v2\/categories?post=2124"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/rssfeedtelegrambot.bnaya.co.il\/index.php\/wp-json\/wp\/v2\/tags?post=2124"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}Talk to me<\/h2>\n
Listening to audio<\/h3>\n
\n{
\n public static MauiApp CreateMauiApp()
\n {
\n var builder = MauiApp.CreateBuilder();
\n builder
\n .UseMauiApp<App>()
\n .AddAudio(
\n recordingOptions =>
\n {
\n#if IOS || MACCATALYST
\n recordingOptions.Category = AVFoundation.AVAudioSessionCategory.Record;
\n recordingOptions.Mode = AVFoundation.AVAudioSessionMode.Default;
\n recordingOptions.CategoryOptions = AVFoundation.AVAudioSessionCategoryOptions.MixWithOthers;
\n#endif
\n });<\/p>\n
\n \/\/ more code
\n }
\n}<\/p>\n
\nprivate async Task ToggleRecordingAsync()
\n{
\n if (!IsRecording)
\n {
\n var status = await Permissions.CheckStatusAsync<Permissions.Microphone>();
\n if (status != PermissionStatus.Granted)
\n {
\n status = await Permissions.RequestAsync<Permissions.Microphone>();
\n if (status != PermissionStatus.Granted)
\n {
\n \/\/ more code
\n return;
\n }
\n }<\/p>\n
\n await _recorder.StartAsync();<\/p>\n
\n RecordButtonText = “\u23f9 Stop”;
\n }
\n else
\n {
\n _audioSource = await _recorder.StopAsync();
\n IsRecording = false;
\n RecordButtonText = “\ud83c\udfa4 Record”;<\/p>\n
\n }
\n}<\/p>\n
\n{
\n string audioFilePath = Path.Combine(FileSystem.CacheDirectory, $”recording_{DateTime.Now:yyyyMMddHHmmss}.wav”);<\/p>\n
\n {
\n await using (var fileStream = File.Create(audioFilePath))
\n {
\n var audioStream = _audioSource.GetAudioStream();
\n await audioStream.CopyToAsync(fileStream);
\n }<\/p>\n
\n }
\n}<\/p>\n
\nusing OpenAI;<\/p>\n
\n{
\n public async Task<string> TranscribeAsync(string path, CancellationToken ct)
\n {
\n var openAiApiKey = Preferences.Default.Get(“openai_api_key”, string.Empty);
\n var client = new OpenAIClient(openAiApiKey);<\/p>\n
\n {
\n await using var stream = File.OpenRead(path);
\n var result = await client
\n .GetAudioClient(“whisper-1”)
\n .TranscribeAudioAsync(stream, “file.wav”, cancellationToken: ct);<\/p>\n
\n }
\n catch (Exception ex)
\n {
\n \/\/ Will add better error handling in Phase 5
\n throw new Exception($”Failed to transcribe audio: {ex.Message}”, ex);
\n }
\n }
\n}<\/p>\nMaking sense and structure<\/h3>\n
\n{
\n var prompt = $@”
\n Extract projects and tasks from this voice memo transcript.
\n Analyze the text to identify actionable tasks I need to keep track of. Use the following instructions:
\n 1. Tasks are actionable items that can be completed, such as ‘Buy groceries’ or ‘Call Mom’.
\n 2. Projects are larger tasks that may contain multiple smaller tasks, such as ‘Plan birthday party’ or ‘Organize closet’.
\n 3. Tasks must be grouped under a project and cannot be grouped under multiple projects.
\n 4. Any mentioned due dates use the YYYY-MM-DD format<\/p>\n
\n var response = await chatClient.GetResponseAsync<ProjectsJson>(prompt);<\/p>\n
\n {
\n Projects = response.Result.Projects;
\n }
\n}<\/p>\nCo-creation<\/h2>\n
Resources<\/h2>\n
\nPlugin.Maui.Audio<\/a> \u2013 NuGet package for handling audio recording and playback in .NET MAUI apps
\nMicrosoft.Extensions.AI<\/a> \u2013 Framework for integrating AI capabilities into .NET applications
\nWhisper Model<\/a> \u2013 OpenAI\u2019s speech-to-text model used for audio transcription
\nSpeechToText in .NET MAUI Community Toolkit<\/a> \u2013 On-device alternative for speech recognition
\nONNX Runtime<\/a> \u2013 For running local LLMs on mobile devices
\nHAX Toolkit<\/a> \u2013 Design guidance for AI experiences in applications
\nMicrosoft AI Principles<\/a> \u2013 Guidelines for responsible AI implementation
\nTelepathy Sample App Source Code<\/a> \u2013 Complete implementation example referenced in this article<\/p>\nSummary<\/h2>\n