UIED

About UIED

Detecting Graphical User Interface (GUI) elements in GUI images is a domain-specific object detection task. It supports many software engineering tasks, such as GUI animation and testing, GUI search and code generation.
We provide a UIED, a toolkit designed to provide user with a simple and easy-to-use platform to achieve accurate GUI element detection.

We provide support on

GUI Element Detection

3 Computer Vision & 3 Deep Learning approaches
Interactive Dashborad

Editable dashbord to adjust the detection result, including position and tag
Software Engineering Output

Output reusable precise detection result for further application (Testing, UI2CODE)

Our Detection Approach

In addition to several classic detection techniques, we proposed our own top-down divide-and-conquer approach to detect GUI text and GUI elements separately

Text Detection

We use a state-of-the-art scene text detector EAST to detect the text element on GUI.
Graphic Element Detection

We adopt a top-down strategy that first segments layout block and then locates atomic elements.
- Block Detection - Segment the UI by layout blocks
- Component Extraction Locate components from binary map of each block
- Element Classification Classify categories of extracted element
Merge

The approach incorporates results from the two branches to produce the final result.

Demonstration

UIED detects GUI elements from the input GUI image and provides a editable dashboard to adjust the detection.
It produces an easy to use json file for any further developement.

Home Page

1

Go to Tool page to use UIED
2

Select given example or upload your own GUI image

UIED is able to handle various GUI, including mobile app, web page and even hand-drawn sketch
3

Select given example or upload your own GUI image

We provide totally 6 approaches to perform element detection, but due to hardware limitation, only 3 of them are available now

For our own approach, the best set of key parameters may vary with GUI size and type, feel free to adjust them if you are unsatisfied of the detection

Dashboard

4

Go to Tool page to use UIED

We provide user a M A G I C Dashboard for editing the detection result

You can Drag & Drop UI elements to adjust their position

You can Change the size and tag of UI elements

You can Delete any unwanted UI elements

All the elements are Collected and Categorized

You can Add more collected elements from the storage

You can apply More Approach to detect UI elements

All the new detected elements are also collected

You can Compound the result to get the best one
5

Export the result as reusable file for further application

The adjusted precise element detection is useful for various application and can be easily expanded