Understanding UiPath’s AI Computer Vision

In today’s tech landscape, AI dominates conversations. And when it comes to efficiency, there’s a power duo: RPA & AI.
Let’s dive into one example: AI Computer Vision.

❓ What is AI Computer Vision?

AI Computer Vision is a machine-learning-based method that visually identifies all UI elements on a computer screen. It allows our UiPath robots to interact with these elements, simulating human interaction. Unlike traditional selectors, AI Computer Vision doesn’t rely on underlying properties of applications. Instead, it focuses on the aspect and relationship of screen elements.

🌟 How Does It Work?

Visual Element Detection: AI Computer Vision locates UI elements using machine learning. It combines element detection and text (OCR) detection to understand the UI comprehensively.
Anchoring System: The relationship between detected elements is encoded into a multi-anchor descriptor, uniquely identifying the targeted element.

💡 Why AI Computer Vision?

We need Computer Vision because it solves the challenges of automating dynamic interfaces and remote desktops, such as Citrix, VMWare, Microsoft RDP, VNC, and others. These environments often have unreliable or non-existent selectors, making it hard for robots to recognize and interact with the UI elements they need. 🚀

🔧 How to Implement Computer Vision in UiPath:

Install the UI Automation package: It contains specialized Computer Vision activities.
Refactored UI Automation activities: Classic activities like Click, Type Into, or Get Text are now powered by Computer Vision’s neural network for precise UI element identification.
CV Screen Scope activity: Define the screen area you want to automate and use other CV activities within it to perform actions on UI elements efficiently.

In the upcoming articles, we’ll explore this topic further, focusing on its activities and features in UiPath.

Stay tuned, and happy automating!