A screenshot-based task mining framework for disclosing the drivers behind variable human actions

Abstract

Robotic Process Automation (RPA) enables experts to use the graphical user interface as a means to automate and integrate systems. This is a fast method to automate repetitive tasks. To avoid building software robots from scratch, Task Mining (TM) can be used to monitor human behavior through timestamped events, such as mouse clicks and keystrokes. From a so-called User Interface log (UI Log), it is possible to discover the process model behind this behavior. However, when the discovered process model shows different process variants, it is hard to determine what drives a human’s decision to execute one variant over the other. Existing works analyze UI Logs looking for the underlying rules, but neglect what can be seen on the screen. As a result, a major part of the human decision-making remains hidden. To address this gap, this paper describes a TM framework that uses the screenshot of each event in the UI Log as an additional source of information. From such enriched UI Log, a decision tree is created, which offers a more complete explanation of the human decision-making. This can be graphically represented, explicitly identifying which elements in the screenshots are relevant to make the decision. The framework has been evaluated through a case study that involves a process with real-life screenshots, indicating a satisfactorily high accuracy of the overall approach.