🐴 AI can take over your computer now

The new Claude feature and its impact on product designers

Oct 24, 2024

☀️ Happy Thursday!

Two days ago, Anthropic launched the upgraded Claude 3.5 Sonnet and introduced a groundbreaking new feature: computer use.

The new Claude 3.5 Sonnet has demonstrated significant improvements across industry benchmarks. However, the most jaw-dropping advancement to me is its ability to control a computer—allowing the AI to view the screen, move the cursor, click buttons, and perform a task autonomously.

Anthropic acknowledges that this feature is still experimental. However, I can already start imagining an exciting (yet somewhat scary) future. Today, I’ll walk you through how this new feature works and discuss its potential impact on product designers.

1. A simple demo

Check out this demo by Mustafa Mergisi. It’s a simple and straightforward automation that searches for information online and compiles it into a spreadsheet.

This is the prompt he used:

Search Amazon for 3 wireless earbuds:
- Find price
- Rating
- Brand name
Make a simple Excel file 'earbuds.xlsx':
- Put the information in a basic table
- Add colors to the headers
- Sort by price
Show me the results

All you have to do next is sit back and watch as Claude takes control of your computer and completes the task for you.

2. More demos from Anthropic

1) Ask AI to control the computer to code a website.

It appears that the system runs into several errors, requiring human intervention to fix them. As Anthropic also mentions, "We’re releasing computer use early for feedback from developers and expect the capability to improve rapidly over time." It’s easy to imagine that AI will soon be able to handle end-to-end tasks like this seamlessly.

2) Ask AI to control the computer to find sunrise viewing spots and figure out the logistics.

It navigates to google.com, finds the appropriate location, opens Maps to check the distance from the user’s location, determines the sunrise time, and then adds an event to the calendar with all the relevant details. Give the video a watch. It is fascinating—and a little unsettling—to think about what AI agents might be capable of in the future…

3. What it means for product designers

I can already imagine multiple scenarios where this capability could empower product designers to automate repetitive tasks in the near future.

1) Audit existing flows

Auditing existing product flows provides the team with a clear understanding of the current state of a product, but documenting and maintaining these flows can be really time-consuming and tedious. (It is one of my least favorite tasks.) AI can streamline this process by navigating through each screen in the flow, taking snapshots, and organizing them in a centralized location for easy review. When updates are made in the future, AI can repeat the process, ensuring that the documentation remains up-to-date with minimal effort.

2) Run through Quality Assurance Checklists

This is another repetitive and tedious task that requires thorough careful review to maintain quality. If there’s a QA checklist (e.g., for accessibility, consistency, or responsiveness), AI can systematically review each design element, flagging any issues that do not comply with set standards.

3) Automate content entry

Let’s say you've collected a large amount of data from user research and are deep into organizing and summarizing it. For repetitive data entry tasks, like inputting survey responses or relevant user quotes into a spreadsheet or FigJam board, AI can streamline the process by automatically copying the content from the source and pasting it into the correct location.

4) Gather design inspirations

If you've created a draft design but are struggling to move it forward, you can ask AI for assistance. It can browse UX/UI inspiration websites, find similar products with comparable elements, take screenshots, and place relevant examples directly into Figma right next to your designs for inspiration.

5) Check design consistency

You probably remember the last time you detached an instance from design system :) We’ve all done this… As the file grows larger, it can become increasingly difficult to keep track of or spot these inconsistencies. AI can help review every element in a design file, identifying and flagging any discrepancies in fonts, colors, or styles that deviate from the default design system. Then it’s much easier for you to review and make a decision whether to update them or not.

6) Assist in research

AI can gather information online or offline, navigate various sources, and compile relevant data into an organized report. For example, let’s say you’re designing a product checkout flow. You can start by asking AI to review all previous documents related to the checkout process, gather key decisions made, experiment results, and past research findings. Then, ask AI to compile and summarize this information into a report, with linked sources for easy reference.

And actually, if we think further, AI could do so much more that what I listed above, even including potentially replacing designers—which is a bit scary to think about at this stage. Since Halloween hasn’t arrived yet, I’m not thinking that far ahead for this article, but that would’ve made a perfect Halloween edition…

That’s it for this week. Thanks for reading.

What’s one task in design you’d ask AI to automate for you?

Let me know in the comments.

Xinran

P.S. When I first started Design with AI newsletter, I wondered if there would be enough to write about after a few posts. Now it’s the complete opposite—there are so many exciting advancements in AI that I have to manage a content queue. What a world we live in!