Computer Use

Computer Use lets Fluent do things in your Mac's other apps: it reads what's in an app's window, then clicks, types, scrolls and drags there with a pointer of its own. Your own mouse and keyboard stay yours, so you can keep working while it does.

Ask in the Smart Panel or the chat window, and name the app when more than one could do it.

Try Asking

AskWhat Fluent does
"Fill in the shipping form in Safari with my address from Memory"Finds the form's fields and types into each one
"Copy the totals from this PDF into the open Numbers sheet" (with the PDF attached)Reads the PDF, then types each total into its cell
"In Music, find Kind of Blue and play it"Searches the catalog and starts the album
"What do you see on my screen?"Takes a screenshot of the display and describes it
"Fill in the rest of October in my timesheet in Chrome"Works down the whole page, row by row, to its end

How It Works

  1. Fluent reads the app's buttons, fields, lists and text through Accessibility, in the background: the app doesn't have to be in front.
  2. It acts with its own pointer, drawn over the app, then looks at the app again to check what changed. The pointer stays in the app's window until the task is done. It moves with the window and hides when another window covers it.
  3. The answer's badge says what it's doing ("Working with Music"); click the badge for the details.

Some apps show a button only while the pointer is over an item, like Play on an album's artwork. macOS shows those under your own pointer only, not Fluent's, so Fluent opens the item's right-click menu instead, where apps usually offer the same command.

On a long web page, Fluent sees the rows in view first and scrolls for the rest, so it can fill in a whole timesheet or form.

Fluent can work in any app that's running, except Fluent itself. It works best in apps whose controls Accessibility can read, as most Mac apps' are. For a game or a drawing canvas, it can look at screenshots instead.

Turn It On

Fluent – Computer Use Settings

Computer Use is on by default. Click it in Settings → Integrations to open its page:

PermissionNeeded forOn its page
AccessibilityReading and controlling appsAllow Accessibility, which opens System Settings
Screenshots (Screen Recording)Only when Fluent should look at a window or your screen as a pictureAllow Screenshots

A permission that's granted shows Allowed. Below them, the Browser section sets up how it works in your browser, see In Your Browser below. To keep Computer Use out of your requests, turn it off with the switch at the top of its page, or its checkbox in the list of integrations.

Stop Or Approve

  • Esc stops Fluent's pointer in the middle of a step, and Fluent is told that you stopped it. With the Smart Panel in front, Esc also closes the panel, which ends the request too unless Keep requests running in background is on.
  • To end the whole request, click the ring in the prompt row, as for any answer.
  • Computer Use doesn't ask before it acts. To confirm each step, tick Ask Before Running for its Interact tool, on the same page: Fluent then waits for Approve before each batch of clicks and keys.

In Your Browser

In a browser, Computer Use works as it does in any app, signed in to your sites as you are, which can bring the browser's window forward. To have it work in the background, set up one of these in the Browser section of Computer Use's page:

Chrome extension (recommended)Remote debugging
Works inChrome and the browsers built on it, such as Arc, Brave and EdgeChrome, Edge, Brave, Vivaldi, Arc, Helium and Chromium
In the backgroundIn background tabs, even while you're awayIn the tabs you have open; the browser asks for your approval after every restart, so not while you're away
Used byComputer Use and your teammatesComputer Use only
Set it upClick Add to Chrome (or your browser's name), then add Fluent from the Chrome Web Store. The row then says Connected.Turn it on, click Open Browser Settings, and turn on Allow remote debugging on the page that opens. Approve Fluent when the browser asks.

Neither is required. When a browser isn't connected, Computer Use says so once and keeps working in it the usual way. Remote debugging gives Fluent broad access to the browser, so turn it on only if you want Fluent working in your tabs.

A Tab Group For Each Chat

With the Chrome extension, the websites a chat opens go into a tab group of that chat's own, titled Fluent. Each chat's group gets its own color, so several chats stay apart. The blue Fluent group is your teammates'.

The tabs are yours: Fluent never closes them, and teammates don't work in them. A group you collapse, rename or recolor stays as you left it. While Fluent works in a tab, its pointer shows in the page and glides to each click.

Several Sites At Once

A request with several parts is planned, and with the Chrome extension the plan's steps use the browser side by side, each in tabs of its own in the chat's group. Ask "What do Linear, Jira and Trello cost for 10 people?", and the three pricing pages are read side by side, not one after another. Click the plan's line in the reply to watch every step. With remote debugging, or in another app, the steps take turns.

The Extension's Menu

Fluent – The Chrome Extension's Menu

Click Fluent's button in the browser's toolbar:

ItemWhat it does
Ask FluentOpens the Smart Panel with the page in its prompt
Add to MemoryKeeps the page's text in Memory's Inbox, read as you see it, signed in
Hire a TeammateStarts a hire for the page in the chat window, for example a teammate that tells you when its price drops. See Team
Working NowThe tabs Fluent is working in, each with Stop
Fluent's TabsThe tabs in Fluent's groups: your chats' and your teammates'

While Fluent clicks or types in a tab, or takes a picture of it, the browser shows a bar saying Fluent started debugging it, until the task is done. To stop Fluent, click Stop in the menu, or cancel the bar.

Tips

  • Keep the teammates' group collapsed. Collapse the blue Fluent tab group and drag it to the far left of your tabs: it takes little room, and your teammates keep working in it. Teammates' tabs go into it without opening it, and keep running there, as in an open group.
  • Teammates stay in their group. Teammates and scheduled actions work only in the tabs of the blue Fluent group. They read your other tabs but never click or type in them. To let them work in a tab of yours, drag it into the group.
  • Teammates reuse their tabs. A teammate keeps one tab for each site it works on. Its next task there opens in that tab, not in a new one.
  • One group for teammates. Teammates that start at the same time share one blue Fluent group, and extra blue groups in a window merge into one. A chat's group is never merged.
  • Tabs the extension can't reach. The extension can't work in the browser's own tabs, like its settings. Fluent works in such a tab only when you ask for it, and says first that the browser may ask for permission.
  • Your own tabs. A teammate's job about a page you already have open, like a chat in WhatsApp or Telegram Web, reads your tab as it is rather than opening another. In a group you collapsed, the browser pauses your tabs, except those a teammate follows live: to read a paused one, Fluent wakes it for a moment, and the browser's bar shows meanwhile.
  • A tab you take over is yours. Bring one of Fluent's tabs to the front and go to another page in it, and it's yours from then on: its job opens its page in a new tab next time.
  • Moving Fluent's tabs. Move a teammate's tab to another window or into a group of yours, or pin it, and it's still Fluent's: its job keeps using it. Taken out of every group, it goes back into a Fluent group.
  • Restarts. When the browser or Fluent restarts, teammates find their tabs again and carry on. A tab the browser unloaded to save memory is loaded again at its job's next check.

If It Doesn't Work

What happensWhat to do
Fluent says it can't read or control the appAllow Fluent in System Settings → Privacy & Security → Accessibility, or click Allow Accessibility in Computer Use's settings
It can't take a screenshotClick Allow Screenshots and turn on Fluent under Screen Recording; reopen Fluent if macOS asks
The browser comes to the frontAdd the Chrome extension, or turn on Remote debugging, in Computer Use's Browser section
The extension's menu says Fluent isn't running, or the browser isn't set upOpen Fluent; in Settings → Integrations → Computer Use, the Chrome extension row says what it needs, such as folder access for that browser
It works in the wrong appName the app in your request: "in Numbers, …"