Calvin Wankhede / Android Authority
TL;DR
- OpenAI is testing a extra environment friendly strategy to let ChatGPT see your display screen on Android.
- It might use Android’s Bubbles function and Accessibility choices to view what’s in your display screen.
- This methodology makes use of system sources extra effectively in comparison with the present screen-sharing methodology that entails casting.
OpenAI is on the lookout for a approach out through the use of an unusual methodology that would change how display screen sharing with ChatGPT works on Android.
ChatGPT on Android at the moment makes use of the MediaProjection API, which can be used for display screen recording or casting the display screen to an exterior show, to see what’s in your display screen. This methodology can really feel clunky for varied causes, together with the totally different permissions dialogs and warning pop-ups that seem, which may hinder your workflow. For the reason that API additionally repeatedly information the display screen, this may result in excessive useful resource utilization, slowing down different processes.
To deal with these considerations, OpenAI is creating a brand new methodology that would serve the identical operate whereas utilizing fewer sources. In model 1.2026.118 of the ChatGPT app on Android, we’ve found that OpenAI is toying with the concept of utilizing Android’s Accessibility settings, together with Android’s Bubbles multitasking function, to let the chatbot see what’s in your display screen.
The function modifications the workflow for Display screen Sharing in ChatGPT. When it’s enabled for the primary time, customers will likely be required to allow sure settings so ChatGPT can “learn seen textual content, buttons, and display screen particulars.” They’re first guided to allow “ChatGPT display screen assist” in Android’s Accessibility settings, following which they would want to allow notifications and dialog bubbles for the app. These settings would forestall the app from being killed.
As soon as permissions are granted, a dialog bubble will seem in your telephone’s display screen, and you need to use it to see what’s in your display screen, whether or not you’re on the house display screen or in an app.
Don’t wish to miss the perfect from Android Authority?
It is possible for you to to work together with ChatGPT by tapping this bubble, as you’ll be able to see within the video beneath. Right here you should have the choice to ask particular questions in regards to the display screen’s contents, and it’ll ignore any context from the prevailing chat, focusing solely on what it at the moment sees in your display screen.
Whereas this strategy bypasses the extreme useful resource utilization of the prevailing screen-sharing methodology, granting Accessibility entry to an AI chatbot is also a priority for some cautious customers. We’re uncertain how OpenAI plans to take care of these apprehensions, or how quickly we might see the function being applied.
⚠️ An APK teardown helps predict options which will arrive on a service sooner or later primarily based on work-in-progress code. Nevertheless, it’s potential that such predicted options could not make it to a public launch.
Thanks for being a part of our neighborhood. Learn our Remark Coverage earlier than posting.
