We provide
for your UI designs

Get Started Our Tool

About UIED

Detecting Graphical User Interface (GUI) elements in GUI images is a domain-specific object detection task. It supports many software engineering tasks, such as GUI animation and testing, GUI search and code generation.
We provide a UIED, a toolkit designed to provide user with a simple and easy-to-use platform to achieve accurate GUI element detection.

We provide support on

  • GUI Element Detection

    3 Computer Vision & 3 Deep Learning approaches

  • Interactive Dashborad

    Editable dashbord to adjust the detection result, including position and tag

  • Software Engineering Output

    Output reusable precise detection result for further application (Testing, UI2CODE)

Our Detection Approach

In addition to several classic detection techniques, we proposed our own top-down divide-and-conquer approach to detect GUI text and GUI elements separately

  • Text Detection

    We use a state-of-the-art scene text detector EAST to detect the text element on GUI.

  • Graphic Element Detection

    We adopt a top-down strategy that first segments layout block and then locates atomic elements.

    • Block Detection - Segment the UI by layout blocks

    • Component Extraction Locate components from binary map of each block

    • Element Classification Classify categories of extracted element

  • Merge

    The approach incorporates results from the two branches to produce the final result.


UIED detects GUI elements from the input GUI image and provides a editable dashboard to adjust the detection.
It produces an easy to use json file for any further developement.

Home Page

  • 1

    Go to Tool page to use UIED

  • 2

    Select given example or upload your own GUI image

    UIED is able to handle various GUI, including mobile app, web page and even hand-drawn sketch

  • 3

    Select given example or upload your own GUI image

    We provide totally 6 approaches to perform element detection, but due to hardware limitation, only 3 of them are available now

    For our own approach, the best set of key parameters may vary with GUI size and type, feel free to adjust them if you are unsatisfied of the detection


  • 4

    Go to Tool page to use UIED

    We provide user a M A G I C Dashboard for editing the detection result

    You can Drag & Drop UI elements to adjust their position

    You can Change the size and tag of UI elements

    You can Delete any unwanted UI elements

    All the elements are Collected and Categorized

    You can Add more collected elements from the storage

    You can apply More Approach to detect UI elements

    All the new detected elements are also collected

    You can Compound the result to get the best one

  • 5

    Export the result as reusable file for further application

    The adjusted precise element detection is useful for various application and can be easily expanded


GUI Element Detection

Convert Your GUI Design Image Or Screenshot To High-Precision Element

The UIED algorithm would be upgraded irregularly, please check the GitHub repo for the latest version for better detection

Quick Start

Using examples to see how the tool works

Upload Custom UI

Try the tool with your upload UI image

Select a processing method

GUI Element Detection & Perceptual Grouping

Detect and Grouping GUI Elements into Layout Blocks

Quick Start

Using examples to see how the tool works

Upload UI Image

Try the tool with your upload UI image

Element Detection by UIED + Element Grouping

Detect & Group Elements

Web Code Generation

Generate Web Code (HTML + CSS) from the Input GUI Image

Quick Start

Using examples to see how the tool works

Upload Web UI Image

Try the tool with your upload UI image

Test Version

Code Generation Currently only Supports Web GUI

Get Started

GUI Element Detection


Web Codes Generation

Meet The Team


Mulong Xie

Frontend, Backend and Detection Algorithm

Sidong Feng



Jieshan Chen


Zhenchang Xing

Associate Professor

Chunyang Chen


Our Collaborators

Contact Us

Office N223, Building 108
Research School of Computer Science
Australian National University
Canberra ACT 2600 Australia


+61 0450674929

Your message has been sent. Thank you!