OpenAI has taken a giant leap forward with the latest update to ChatGPT, introducing a feature that allows the AI to interact with real-world visuals through your phone’s camera and screen. This new capability, part of the Advanced Voice Mode, was unveiled during OpenAI’s “Shipmas” event, following several months of teasers and delays since its announcement in May’s Spring Update.
This update brings a new dimension to ChatGPT’s interaction model, enabling it to provide “realtime visual context” for a variety of tasks. Whether you’re looking for design tips for your living space, need help with a math problem, or require assistance in crafting a quick reply in your Messages app, ChatGPT can now offer insights based on what it sees through your device.
How It Works
To engage with this feature, users simply activate the Advanced Voice Mode in the ChatGPT app and tap on the video icon located at the bottom left of the interface. This allows the AI to see live video feeds, providing a hands-free, conversational experience. For screen sharing, the process involves selecting “Share Screen” from the app’s drop-down menu, giving ChatGPT the ability to view and interact with your device’s interface.
During the launch demonstration, OpenAI showcased how ChatGPT could guide an employee through the nuances of making pour-over coffee, adjusting to real-time queries and providing personalized feedback. In another example, the AI was used to interpret facial expressions for emotional analysis or assist with educational tasks like solving an equation on a whiteboard.
Despite some initial hiccups during earlier demos, such as misidentifying objects or jumping to conclusions too quickly, the current rollout appears more refined. We tested the feature by showing it an office plant, where it successfully identified the plant as an Aloe Vera, noted signs of distress in the plant’s health, and advised on watering schedules — all with commendable accuracy.
This feature is currently rolling out to Team, Plus, and Pro users via the latest version of the ChatGPT mobile app, with some geographical restrictions. Users in the EU, Switzerland, Iceland, Norway, and Liechtenstein will have to wait a bit longer for this functionality, but OpenAI has promised an imminent rollout.
This update not only enhances user engagement but also positions ChatGPT as a more versatile assistant in daily life, bridging the gap between digital advice and real-world application. It’s a testament to how AI can evolve to become more integrated and intuitive in our everyday environments, offering not just information but contextual, visual assistance.
As we continue to explore the limits of AI, features like this push the envelope on what’s possible, making technology not just a tool but a collaborator in our physical and digital spaces.
Your First 10 AI Skills
10 practical AI skills, copy-paste prompts and a 7-day plan to start using AI with confidence.
Download the guide →