Gemini Live Can Guide You In The Real World Once You Let The AI See It

Google unveiled the new Pixel 10 phones, the Pixel Watch 4 models, and the affordable Pixel Buds 2A earbuds earlier this week, but Gemini AI was also a big star of the Made by Google event. Artificial intelligence is at the center of all these products, with Google having announced a few interesting AI experiences. Magic Cue, Camera Coach, and conversational edits in Google Photos are highlights, but Gemini Live also got a few upgrades this week, including the ability to see what you see and interact with the real world via the Pixel phone's display.

The new Gemini Live feature will not be a surprise to any Pixel user who saw Google's Project Astra demo at I/O 2025 a few months ago. At the time, Google shared concepts for Gemini Live, including the AI assistant's ability to recognize objects in the real world and answer questions in real time while highlighting those objects on the screen.

Maybe you're trying to repair something following an online tutorial and you don't know which tool to pick from a box or table. Sharing your screen with Gemini Live and asking the AI to identify the tool mentioned in the tutorial could do the trick. The AI would look at the camera stream, identify the object, and then highlight the right tool by drawing a rectangle around it.

Visual guidance will be available in Gemini Live on Pixel 10 phones first. The feature will roll out in time for the phones' August 28 release date. The functionality will be available on other Android devices during the same week. iPhone owners who use Gemini Live will get visual guidance after that.

To have Gemini Live provide guidance in the real world, you'll need to share your phone's camera with the AI once you've updated the app. Start asking questions about the objects around you, and Gemini Live will respond while highlighting items on the screen that match what the camera sees.

The new Gemini Live feature might be useful when shopping. You can ask the AI for help when choosing between two similar products, or tell it to identify something on your list. Gemini Live will provide quick answers by voice while highlighting corresponding items on the phone's display. The feature will be even more useful with AI smart glasses where you won't have to point your phone to what you're seeing. AI smart glasses already feature built-in cameras. But that's just speculation for now. It's likely that Google's Android XR smart glasses will support Gemini Live's visual help in the near future.

Visual guidance is the most exciting new Gemini Live feature from Google, but it's not the only one. Google gave Gemini Live a few other handy features. The AI assistant will be able to connect to more Google apps than the ones it supports right now (Google Calendar, Keep, and Tasks). In the future, Messages, Phone, and Clock will get Gemini Live integrations. Google will also expand the Google Maps capabilities of Gemini Live.

The idea is to make the most of those apps while interacting with Gemini Live by voice. Google says you'll be able to make calls and send messages while you're already talking to Gemini Live about something else: "Imagine you're talking with Gemini to find the fastest subway route. If you realize you're running behind, you can simply interrupt and say, 'This route looks good. Now, send a message to Alex that I'm running about 10 minutes late.' Gemini can draft the text, and you can get back to your navigation."

In the example above, you'd be using Gemini Live, Google Maps, Contacts, and Messages at the same time. The AI assistant would be pulling information from Google Maps while also taking action inside the Messages app after identifying the recipient.

Finally, Google plans to "dramatically improve" Gemini Live's ability to understand human speech, including nuances in it from intonation, rhythm, and pitch. Also, Google's model updates will help you customize the Gemini Live voice experience. You'll be able to change the speed of Gemini's voice and its accent. The AI will also be able to change its personality for entertainment, like storytelling.