Abstract
Intelligent Process Automation (IPA) is emerging as a sub-field of AI to support the automation of long-tail processeswhichrequiresthecoordinationoftasksacrossdifferentsys-tems. So far, the field of IPA has been largely driven by sys-tems and use cases, lacking a more formal definition of thetask and its assessment. This paper aims to address this gapby providing a formalisation of IPA and by proposing specificmetrics to support the empirical evaluation of IPA systems.This work also compares and contrasts IPA against relatedtaskssuchasend-userprogrammingandprogramsynthesis.
Introduction
Robotic Process Automation (RPA) aims to provide a sup-porting framework to automate the long-tail of processeswhich involve routine tasks, structured data and determin-istic outcomes (Aguirre and Rodriguez 2017). RPA supportsend-users in the automation of existing processes withouttherequirementofaprogramminglanguage.
Even though there has been an increase in investments inthe area of RPA, it still relies on the explicit encoding ofrules and configurations for process generation, with lim-ited use of AI methods. Agostinelli et al. (Agostinelli, Mar-rella,andMecella2019)providesananalysisofseveralRPAtools,noneofthestudiedtoolshasself-learningabilityand,are not able to automatically understand which actions be-long to which process (intra-routine learning) and whichprocesses are available for automation (inter-routine learn-ing).
The goal of Intelligent Process Automation (IPA) is togeneralise RPA, proving the tools to create complex work-flows, with minimal user interference (Reddy, Harichan-dana, and T Alekhya 2019). IEEE Standards Associationdefines Intelligent Process Automation as “a preconfiguredsoftware instance that combines business rules, experiencebased context determination logic, and decision criteria toinitiate and execute multiple interrelated human and auto-matedprocessesinadynamiccontext.”(8072017).
Consider,forexample,auserthatworksintheITdepart-ment of a company and is responsible for redirecting tick-etsfromtherequestsystemtothecorrectdepartment.RPA
∗DellZhangisonleavefromBirkbeck,UniversityofLondon.
would allow the user to automate this process by manuallygeneratingasetofrulescapableofredirectingeachticketto the appropriate department; however, if the process iscomplex, with several variables, the implementation mightbe costly and slow. IPA systems, on the other hand, wouldobserve the user’s actions and detect the patterns betweendifferent requests and the redirected departments. With suf-ficient examples and the clarification of the user intents, itwould automate the process, minimising human interven-tion.
Sofar,thefieldofIPAhasbeenlargelydrivenbysystemsand use cases, lacking a more formal definition of the taskansitssystematicevaluation.Thispaperaimstoaddressthisgapbyfocusingonthefollowingcontributions:
ProvidingaformalisationofIPA.
Proposingspecificmetricstosupporttheempiricalevalu-ationofIPAmethodsandsystems.
IntroducinganewbenchmarkfortheevaluationofIPA.
ComparingandcontrastingIPAagainstrelatedtaskssuchasend-userprogrammingandprogramsynthesis.
Thispaperisorganisedasfollows:Section2presentstheformalisation of IPA and related concepts. Section 3 intro-duces the modalities and tasks that are part of IPA. Section4 analyses research areas that are similar to IPA. Section 5thenusestheformalismdefinedpreviouslytodefinemetricsfor the different IPA tasks. Section 6 presents the method-ologyusedforconstructingabenchmarkforevaluatingIPAtasks.Finally,Section7concludesthiswork.
FormalisingIPA
At the center of IPA is the capture and formalisation of aworkflow (the interaction between end-user actions, soft-wareanddataartefactswithinanend-userinterfaceenviron-ment)inaformallanguagewhichcanbeusedtore-enacttheworkflow.
IPA formal languages have particular properties whichaims at eliciting the end-user actions embedded within aworkflow. The formalisation of these properties is a pre-requisiteforthedefinitionofanevaluationmethodology.Itisthroughthislanguagethatwemayunderstandand
evaluate a given interactor, such as the Sikuli GUI automa-tiontool1.Operatingdirectlywiththeinteractorallowsusto evaluate the interaction of the end processes with thesame interfaces a human would encounter, rather than theextremely varied computations that underlie the systemswith which it interacts. To this end, we propose an abstractinteraction language with which to describe the kind ofinterface-centred activity to be mimicked by IPA. We firstdefine it purely syntactically, and then describe its intendedinstantiationintermsofGUIsandGUIinteractions.
Definition1.AninterfaceisafinitesetIofsymbols
{i1,...,in}whichwillbereferredtoasinterfaceelements.
Definition2.Anenvironmenteisdefinedby a 4-tuple( , ,Ω,S)where = I1,...,Inisafinitesetof interfaces, Ω is a countably infinite set of variables(arg1, arg2, . . .) or value symbols and Fis a set of finitaryfunction symbols f (s, arg1, . . . argm) where each arg maybeanyinterfaceelementiIjoravaluevΩ.Swillbereferredtoasthestatespace.
The following example serves to introduce the intendedconcrete realisations of the described language, after whichweturntothedefinitionsrelevanttoitsinterpretation.
Figure1:Asimplifiedenvironmentewithtwointerfaces,I1
andI2.
Example 1.An end-user may work on a single desktopconfiguration with multiple windows, each of which can beidentified as a separate interface. We may assume the inter-faceelementsarerepresentedbyuniqueIDs.
interpretationfunction[[]]whichwillbedescribedcompre-hensivelyinthenextdefinition.
Definition4.InterpretationFunction:
Given an environment e = (, F, Ω, S) and a fixed inter-actorlanguage,wedefinethefollowinginterpretationfunc-tion:
[[S]] is abstractly defined to be the computational statespace of the machine on which we are hypothetically op-erating.
ForeachinterfaceelementsymboliI,[[i]]mapsitoauniquemeaningfulsegmentoftheinterfacesuchasabut-ton,alistitem,aform,aparagraphoftextorascrollbar.
[[I]] is the set[[i]]i I , so that [[I]] refers to a com-pleteinterfaceorwindow.
[[Ω]]is the set of all values which are possible interac-tor inputs: in the case of a computerized interactor, thiswouldbeasetofallpossibledataartifactssuchasstrings,booleans,integersorimagesthatwouldberecognisedbythe interactor’s own program language as program argu-ments.
[[F ]]is the set of all valid action functions in the inter-actor language, so that each functional symbol fF ismapped to a valid action function of matching arity, witharguments interpreted pointwise according to the interpre-tationfunction.
Fornotationalsimplicity,wewillhenceforthconflateen-vironmental elements with their interpretations (for exam-ple, I for [[I]]) and suppress the state space argument to ac-tionfunctions.
Definition5.Aninterfaceelementiispositionallyrealisedif there exists a coordinate pair bb(i) = ((x0, y0), (x1, y1))defining the bounding box associated with the instantiationofiinthegraphicaluserinterface.
(1)(1)
In Figure 1, let I1=i1, i2, where these identify theforminputboxandthesubmitbuttonrespectively.Thestatesymbolswillrepresentthecomputationalstateunderthehood, while the function set F is determined by the validactionsrelevanttotheinterfaces:here,forexample,onecanimagine these to include a click action f1(s, i) and a textinputactionf2(s,i,‘exampletextinput’).
ThesetΩisthatofallpossibleuserinputstothetextbox.Tosimplify,supposethatinthiscaseitisΩ={‘a’,...,‘z’,‘’}∗(where∗istheKleenestar).
Definition 3. A model or instantiation of an environment eis defined with respect to a working computer desktop withfinitelymanygraphicaluserinterfaces,aninteractor(which
Figure 2: Interface element i(1)with a positional realisationvisualisedasaboundingbox.
A soft typing system may be implemented if one intro-duces an environmental vocabulary, by which descriptivetypes may be assigned to both interface elements and inputvalues.
Definition 6. An environment vocabulary
may either be automatic, such as a Sikuli program instance,orahumanend-user)whichhasitsownlanguageandan
T ‘a’,...,‘z’,‘’∗isasetofdescriptivestrings
strings.Definetype: {I1,...,In,Ω}→ T tobea
1http://sikulix.com/
descriptorfunctionforinterfaceelementsandvalueswhich
mayserveasactionfunctionarguments.
Example2.Infigure1,onemaychoosetodefineanenvi-ronmentvocabularysuchthattype(i1)=‘button’.
A descriptive vocabulary (or soft typing system) is notstrictly necessary, but may serve for greater interpretabilityand potentially be used to introduce constraints on functionarguments.Ifthesystemdesignerwishes,onemaysimilarlyintroduce an action vocabulary to which action functionsmaybemapped.
text2process
process2text
demo2text
The purpose of the formalization in this section has beento formally define a process in the context of IPA. The fol-lowingdefinitionscompletethisgoalandallowsustodefinethecoretasksofIPAandtosuggestevaluationmetrics.
Definition 7.A process p(withrespect to ainterpreted en-vironmente)isanorderedsequenceofactionfunctions
p= f0(arg(0),...,arg(0)),
Figure3:IPAtasks
demo2process(D2P)
AtthecenterofIPAisthetaskoftransformingademonstra-tion to a program that is capable of re-executing the sameprocess demonstrated by the user. We assume that all tasksare result-oriented, i.e., the user wants to achieve a specificresultbyexecutingthetask.Theresultscanbespecificob-
0 n1
.
asetofactions,suchassendingane-mailtoaclientinquir-
ingaboutthepaymentofaservice.
ReturningtotheexampleinFigure4,theresultingpro-
fm(arg(m),...,arg(m))
cessisimplementedusingtheSikuliGUIAutomationTool.
withvalidactionfunctionsandargumentsinterpreted
(andvalid)inthespecifiedenvironment.
Definition 8.In the case that an environment e is instanti-ated(orrealised)withrespecttoacomputationalinteractor,its programming language is referred to as an IPA Realisa-tionLanguage.
Definition9.AnIPAprogramisaprocesspinanenviron-mentewhichisinstantiatedwithrespecttoanIPARealisa-tionLanguage.
Although the definition of a process is purposefully en-vironment agnostic, for the purpose of the tasks describedin the next section we specifically assume process to meana process which is an IPA program. In particular, the tasksnameddemo2processandtext2processrequiretheoutputtobe an IPA process with respect to some IPA realisation lan-guage.
IPAModalities&Tasks
Intelligent Process Automation can be defined as a com-bination of four different tasks: demo2text, demo2process,text2process and process2text, as shown in Figure 3. In thissectionwewilldescribeeachoneofthetasks,usingamoti-vationalexamplepresentedinFigure4.
Withrespecttoourformalization,thesetasksmaybeun-derstood as the exercise of defining a mapping between thethree different environment interpretations which refer to thesame desktop configuration interpreting the interface set Ibutwithactionfunctionsandargumentsinstantiatedwithre-spect to three different interactor “languages”: the recordedactivityofahuman(whichconstitutesademo),anyIPARe-alisation Language (which defines the target process) andhumannaturallanguage(instructionaltext).
spreadsheetasininthedemonstration.
demo2text(D2T)
The demo2text task requires the conversion from a demon-stration of a user executing an activity to a natural lan-guage description of the executed activity. In the example,the demonstration is a screen-recording of a user’s desktop,divided into different time segments, where each segmenthas a corresponding natural language description of the ac-tions. The natural language description should be a set ofimperative sentences, where each sentence describes a singleaction that should be taken in order to replicate the demon-stration.
The generation of natural language descriptions allowsusers to understand better the process that is being demon-strated,especiallyforusersthatarenotexecutingtheaction,allowingnewuserstoquicklylearnhowtoexecutethepro-cessthemselveswithoutexternalinterference.
In Figure 4,we present the conversion from a videodemonstrating a user manipulating a spreadsheet with stu-dent’sgradestothecorrespondingnaturallanguagedescrip-tionofthetask.
Demo2text is similar to the task of automated video cap-tioning. However, video captioning is the task of describ-ing general videos using natural language (Pan et al. 2017),while demo2text focuses on the description of Desktop ac-tions (usually materialised in video form) with imperativesentences.
text2process(T2P)andprocess2text(P2T)
Instead of having the end-user instance workflow demon-stration,itispossibletotargetatextdescriptionofthework-flow. The text2process task is particularly useful when theuserwantstogeneratetheprocessfromthedescriptionof
Naturallanguagedescription Videodemonstrations
Figure4:ExampleofIPAtasks
the activity. The task is similar to end-user programming andsemantic parsing, giving that users can convert from natu-ral language to a program that implements a process (Sales,Handschuh,andFreitas2017)andProgramSynthesis,sincethefinalgoalisgeneratingaprogram.Givenasetofimper-ativesentences,wewanttogenerateaprogramthatexecutesthedescribedactions.
Similarly, the user might also want to generate the NLdescription of a process that is already implemented, espe-ciallyincaseswheretheprocessbecomestoocomplextobeanalysedbyhumans.
RelatedWork
Intelligent process automation aims to replicate and im-prove, iteratively, activities carried out by humans. Com-paringtheperformanceofdifferentIPAtechniquesrequireswell-defined tasks and metrics that reflect the relevant as-pectswhenautomatingaprocess.Inthissection,wepresentthebenchmarkspresentinsimilarresearchareas.
Even though IPA applications, such as software intelli-genceandRPA,haveparticularresearchquestions(Aalstetal. 2018), there are only a few published datasets designedtohelpaddressingthisdemand.
One conspicuously related dataset is the Mini World ofBits (MiniWoB) (Shi et al. 2017), a benchmark of 100 re-inforcement learning environments containing many of thecharacteristicsoflivewebtasks,createdinacontrolledcon-text. Each MiniWoB environment is an HTML page with aresolutionof210x160pixels,andanaturallanguagetask
description, such as “Click on the ‘Next’ button.”. The en-vironment provides a precise evaluation metric, rewardingsimulated behaviour, with rewards ranging from -1.0 (fail-ure)to1.0(success),accordingtotheresultsofeachaction,i.e., if the action shifts the current state to a state closer totheenvironmentgoalornot.
Thesamework(Shietal.2017)alsoproposesFormWoB,which consists of four web tasks based on real flight book-ing websites, and QAWoB, which approaches web tasks asquestion answering, soliciting questions from crowd work-ers. Even though MiniWoB provides an essential baselinefor IPA, it is still a synthetic dataset, not reflecting real userapplications. It has a specific screen size, a limited numberof applications and user actions, creating a controlled andclosedenvironment.
An example of a real (non-synthetic) dataset for soft-wareintelligenceisthePhotoShopOperationVideo(PSOV)Dataset (Cheng et al. 2018),containing videos and densecommandannotationsforPhotoshopSoftware,with74hours of videos and 29,204 labelled commands. DespitePSOV presenting real-world use cases, it is still limited toonespecificsoftware,i.e.,itisnotgeneralisabletootherap-plications.
Comparable benchmarks can be found in the researchareaofvideocaptioning.(Rohrbachetal.2015)presentstheMPIIMovieDescriptiondataset(MPII-MD)whichcontainstranscribed and aligned audio descriptions and script datasentencesofasetof55moviesofdiversegenres.(Torabietal.2015)introducesadatasetincluding84.6hoursofpaired
video/sentencesfrom92DVDs,withhigh-qualitynaturallanguagephrasesdescribingthevisualcontentinagiven
Eimagearg ifargisanimage
Earg(arg,argÙ¨)= Esymbarg ifargisasymbolic
segmentoftime.
Aprocesscanbeseenasaformalrepresentationofatask;therefore, there is an evident alignment between IPA andsemantic parsing (SP). Unlike IPA, several benchmarks areavailableforSP,fromwhichwecanobtaininsights.
Mostofthesebenchmarksfocusonevaluatingtheconver-sionfromanaturallanguagespecificationtoaprogramwrit-ten in a specific programming language. WikiSQL (Zhong,Xiong, and Socher 2017) is a collection of questions, corre-spondingSQLqueries,andSQLtables.NL2Bash(Linetal.2018)isacorpuswithfrequentlyusedBashcommandswithits respective natural language description. Other applica-tions include converting from a visual object to code (Ling etal.2016;Beltramelli2018)andfromsourcecodetoPseudo-codeindifferentlanguages(Odaetal.2015).
Process mining is also another closely related researcharea; it aims to discover, monitor and improve real processes,extracting knowledge from event logs (Van Der Aalst et al.2011). Unlike IPA, it does not have its main focus on au-tomation, but on finding answers for domain-specific ques-tions, such as analysing patient treatment procedures (Boseand van der Aalst 2011),and discovering the roles of thepeopleinvolvedinthevariousstagesofaspecificpro-cess(vanDongen2015).
EvaluationMetrics
D2P&T2P
ThissectionconcentratesonaweightedquantificationofthecorrectnessandcompletenessofthegeneratedIPAprogramoutput. Correctness and completeness are defined against agold reference IPA program which was produced by one ormorehumanprogrammers.
GivenasetÎ =p1, . . . pkofgeneratedIPApro-gramswhereeachpj=f0(arg0,...,argn)....
fm(arg0,...,argn)andacorrespondinggoldstandardÎ ,
value
Foranimageargument,theintersectionoverunion(IoU)isusedtodefineEimagearg.Giventheboundingboxofthecorrespondinginterfaceelement(bb(i(arg)))andthegoldstandardboundingbox(bbÙ¨(i(arg))):
areaof(bb(i(arg)) bbÙ¨(i(arg)))IoU=areaof(bb(i(arg))∪bbÙ¨(i(arg)))
where Eimagearg(arg) = 0 if IoU > 0.5 and
Eimagearg(arg)=0 otherwise.
IoU assumes that Imagesyscan be registered within thegsreferenceScreenshotgs.
An alternative way to define Eimageargis by directlycomparing the two argument images using means squarederror (MSE) or structural similarity index (SSIM) (Wang etal.2004):
m−1n−1
MSE= [I(i,j) K(i,j)]
mn
i=0j=0
(2µxµy+c1)(2σxy+c2)
SSIM(x,y)=
(µ2+µ2+c1)(σ2+σ2+c2)
Thepreviousmeasuresdonottakeintoaccount theorderoftheprogramstatements. In order to capturethe sequential nature of an IPA program we include asequence-based metric which is based on the longestcommonsubsequence(LCS)function.GiventwosequencesX = (x1, x2, . . . , xm) and Y= (y1, y2, . . . , yn), and giventhat the prefixes of X are X1,2,...,mand the prefixes of YareY1,2,...,n,LCSisdefinedas:
∅
we define a set of metrics of varying granularity that can beseenasapproximatemeasuresofprogramcorrectness.
Westartwiththefollowingerrorfunctions:
LCS(X,Y)=
(ifi=0orj =0)
LCS(Xi−1,Yj−1)ˆxi
StrictError:
0ifpj=pÙ¨
i j (ifi,j>0andxi=yj)
1
MAEstrict(Î )=
(ifi,j> 0andxi/= yj).
pΣj∈Î
Estrict(pj,pÙ¨)
|Î |
In order to compute the maximal program fragment gen-eratedweencodetheprogramsÎ Ù¨andÎ asasequenceof unique symbols sifrom a hash table derived from thestatementsofÎ Ù¨∪Î .Themaximumprogramoverlap
Predicate/ArgumentSensitiveError:
(MPO)isdefinedas:
y
Esensitive
(p,pÙ¨)=
MPO(Î ,Î Ù¨)=lcs(S,S )
|S|
f∈{f0,...,fm},
arg∈{arg0,...,argn}
Earg(arg,argÙ¨)+Epred(f,fÙ¨)
|Î |
P2T&D2T
EvaluatinggeneratedtextisarequisiteforareasrelatedtoNaturalLanguageGeneration,suchasMachineTrans-
Epred(f,f
Ù¨)=
iff=fÙ¨
otherwise
lation,AutomaticSummarisation,Image/VideoCaptioning,andDocumentSimilarity.BLEUhasbeenwidelyapplied
and accepted as an evaluation metric for the tasks in the men-tionedareas(GattandKrahmer2018).
Similarly, we apply BLEU (Papineni et al. 2002) to eval-uatetextgeneratedfromdemonstrationsandprocesses.Ourgoal is to automatically evaluate for a demonstration Dior a process Pihow well a candidate generated sentencecimatches the set of of demonstration/process descriptionsSi=si1, ..., sim. BLEU computes the n-gram overlapbetweenthegeneratedtextdescriptionandthereferencede-scription. BLEU score depends on two different other fac-tors:modifiedn-gramprecisionandbrevitypenalty.
Modified precision score computes the fraction of wordsmatched between candidate descriptions and reference de-scriptions in the entire test corpus. Differently from preci-sion,itclipsthen-gramswhenitmatcheswiththereference,avoiding high-precision results obtained with only repeti-tions of correct words. Modified n-gram precision is com-putedasfollows:
Each video has an average of 56 seconds and is dividedinto time intervals, where each interval has a correspondingnaturallanguagedescription.
There are two types of natural language description for thetask:amoregeneralideaofthetaskbeingperformed,i.e.,asummarisation, and a detailed step-by-step description. Thesummarised text is used as a guide for the annotator, in or-der to generate the video, description and program. The de-taileddescriptioniswrittenusingimperativesentences,suchas“Clickonthebutton‘Send”.
RealWoB also contains a program for every task. The pro-grams were implemented based on the videos and the nat-ural language step-by-step. Every sentence in natural lan-guage corresponds to one or more commands in the pro-gram/process. Figure 5 presents an example of pair of nat-ural language step-by-step description process implementa-tioninSikuli.
The100tasksaredividedinto10categories:
Spreadsheetuse:10tasks
Σ Σ Countclip(n-gram)
Spreadsheetandbrowseruse-simple:10tasks
pn= Σ
Σ Count(n-gram)
Spreadsheetandbrowseruse-elaborate:10tasks
C∈{Candidates}n-gram∈C
The brevity penalty factor penalizes candidates descrip-tions,c,shorterthanthereferencedescriptions,r,computedasfollows:
BP 1 if c > re(1−r/c) ifc≤r
Finally,theBLEUscoreiscalculatedas
N
BLEU=BP·exp( wnlogpn)
n=1
where wnare positive weights summing to one and N isthesizeofthen-grams.
RealWorldofBitsBenchmark
Tothebestofourknowledge,therearenodatasetsavailablethat can be used as a benchmark for all tasks defined in thiswork. Therefore, we create a benchmark for evaluation ofIPAapproaches.InspiredbytheMiniWoB(Shietal.2017),wenameourdatasetRealWorldofBits(RealWoB).Inthissectionwedescribehowwegeneratedourbenchmark.
RealWoBcontains100differententries,whereeachentryisrelatedtoonespecifictask(e.g.,searchingforaflight)anditcontains:
Ascreenrecording(video)ofauserperformingthetask;
Anaturallanguagesummarisationofthetask;
A natural languagestep-by-step descriptionof thetask;Aprogramthatiscapableofre-executingthetask,writteninSikuliGUIAutomationToolandTagUI2.
2https://github.com/kelaberetiv/TagUI
Webmailsuiteuse:10tasks
SpreadsheetandWebmailsuiteuse:10tasks
Webmailsuiteandbrowseruse:10tasks
Browser,spreadsheetandWebmailsuiteuse:10tasks
Browser(socialmedia)use:10tasks
Browser (social media) and spreadsheet use: 10 tasksRandomselectionofprevioustasksexecutedinadifferentoperatingsystem:10tasks
The selected group of tasks/computer applications arecommonforofficeworkers,withahighnumberofpotentialend users. We identified the tasks as suitable for automation,wherenoneorlittlehumaninputisrequired.
The automation of tasks using webmail suites and webbrowsingisparticularlyhardduetothedynamicstateoftheweb.Fortasksinvolvingreceiving/sendinge-mails,wecon-sider that all e-mails follow a pre-defined template. For theones that involve browsing the web, we cache a version ofthe used webpage and assume it as the current state of thepage.
Forillustrationpurposes,Table1presentsoneexampleofeachtask.
RealWoB was annotated by four different annotators, A,B, C and D, where C is a specialist in IPA. It was constructedwiththefollowingsteps:
We defined 90 different tasks based on investigating ev-erydayofficeworkertasks.
Annotator A recorded 90 videos, running the tasks onWindows10.
Annotator B wrote a summary of the task being per-formed, a step by step description (for every video seg-ment) and a program capable of executing the task auto-matically.
Processimplementation
Step-by-stepdescription
Type"Totalhours"inordertodescribethecell.
Type "=SUM(" to initiate the summationformulawitharguments:cellsD14,D25,D36,D47 and D58.
Type")"and"Enter"toapplytheformula.
Figure5:NLdescriptionandrespectiveSikuliprocessimplementation.
Table1:ExamplesoftasksindatasetRealWoB.
Category TaskExample(Summary)
Openaspreadsheet<filelocation>containingcolumnsfornamesofstudentsandmarksforfiveassignments.Createanewcolumncontainingthemeanofthemarksforeachstudent.
Openthewebsite:https://veggiedesserts.co.uk/vegan-carrot-cake/
Spreadsheet+browser
Spreadsheet+browser
Findtheingredientsfortherecipeonthepage.
Openaspreadsheet<filelocation>containingacolumnforingredientsandacolumnforquantity.Copytherecipedetailsintothespreadsheet.
Openaspreadsheet<filelocation>containingalistofproducts.Searcheachitemonhttps://www.idealo.co.uk/
Addthelowestpriceinthecolumnnexttoeachproductname.
Openoutlook.
Writeanemailto<hidden>requestingtobookaflightfromManchestertoTokyoonthe01/08/2019.
Openaspreadsheetcontainingnamesofpeople,places(originanddestination)anddates(inaspreadsheet),
Spreadsheet+Webmail
Webmail + browser
Browser+spreadsheet+webmail
Openoutlook.
Sendanemailto<hidden>requestingaflightbookingforeachrowinthespreadsheet(onee-mailforeach).
Openoutlook.
Readanemailrequestingthebookingofaflight.
Usehttps://www.skyscanner.net/toverifythepriceoftheflight.
Openoutlook.
Readoneemailrequestingthebookingofaflight.
Usehttps://www.skyscanner.net/toverifythepriceoftheflight.
Addthenameofthepersonandthecheapestflighttoa(pre-created)spreadsheet.
Gotohttps://www.imdb.com/chart/top?ref=nvmv250.
Searchforthetrailersofthetop5moviesofalltimeonYoutubeusingthenameofthemovieandthewordtrailer.
Findthecurrenttop10hashtagsonTwitter.Addthehashtagsittoapre-createdspreadsheet.
Randomtask-DifferentOS AnyoftheabovetasksexecutedusingtheOSUbuntu.
AnnotatorCverifiedandcorrectedthevideosandthean-notations.
Fromthe90differenttasks,wechosetenrandomtasks.
AnnotatorBrecordedthetentasksusingUbuntu16.04.
Annotator D wrote a summary of the task being per-formed, a step by step description (for every video seg-ment) and a program capable of executing the task auto-matically.
AnnotatorCverifiedandcorrectedthevideosandthean-notations.
Conclusion
In this work, we present a formalisation for IPA using abottom-up approach, defining several composing blocks ofan IPA program. We also define the modalities and tasks thatarepartofIPA,demo2text,demo2process,text2processandprocess2text, and we present how these tasks relate to simi-larresearchareas.
With the formalisation, it was also possible to define ap-propriate metrics for each one of the tasks, evaluating theresulting process or text. We also describe how we built adatasetthatcanbeusedasabenchmarkfortheIPAtasks.
WeenvisionthattheresearchareainIPAwillseesignifi-cant progress in thenextfew years, following the increased
useofRPAtools.TheformalisationpresentinthisworkisastartingsteptowardsbuildingcomplexIPAsystems.
While we have provided the formalisation, evaluationtechniques and methodology for designing a benchmark,there is an important next step in this research: joining to-gether these contributions and creating a baseline for eachoneoftheIPAtasks.
Acknowledgements
The authors would like to thank Jacques Cali and the BluePrism team for their support of this project. We also thanktheanonymousreviewersforthevaluablefeedback.
References
[807 2017] 2017. Ieee guide for terms and concepts in intel-ligentprocessautomation.IEEEStd2755-20171–16.
[Aalst et al. 2018] Aalst,W. M.;Bichler, M.;Heinzl,A.;et al.2018.Robotic process automation.Business & In-formation Systems Engineering: The International JournalofWIRTSCHAFTSINFORMATIK60(4):269–272.
[Agostinelli, Marrella, and Mecella 2019] Agostinelli,S.;Marrella, A.; and Mecella, M.2019.Research challengesfor intelligent robotic process automation.Workshop onArtificial Intelligence for Business Process Management(AI4BPM19)held in conjuction with the 17th Int. Confer-ence on Business Process Management (BPM’19). Vienna,Austria1-6September,2019.
[AguirreandRodriguez2017]Aguirre,S.,andRodriguez,
A.2017.Automation of a business process using roboticprocessautomation(rpa):Acasestudy.InFigueroa-Garc´Ä±a,
J.C.;Lo´pez-Santana,E.R.;Villa-Ram´Ä±rez,J.L.;andFerro-Escobar, R., eds., Applied Computer Sciences in Engineer-ing,65–71.Cham:SpringerInternationalPublishing.
[Beltramelli 2018] Beltramelli, T.2018.pix2code: Gener-ating code from a graphical user interface screenshot.InProceedings of the ACM SIGCHI Symposium on Engineer-ingInteractiveComputingSystems,3.ACM.
[Bose and van der Aalst 2011] Bose, R. J. C., and van derAalst, W. M.2011.Analysis of patient treatment proce-dures. InBusinessProcessManagementWorkshops(1),vol-ume99,165–166.
[Cheng et al. 2018] Cheng, J.; Hsu, H.-K.; Fang, C.; Jin, H.;Wang,S.;andYang,M.-H.2018.Learningfromphotoshopoperation videos: The psov dataset. In Asian Conference onComputerVision,223–239.Springer.
[Gatt and Krahmer 2018] Gatt, A., and Krahmer, E.2018.Surveyofthestateoftheartinnaturallanguagegeneration:Coretasks,applicationsandevaluation.JournalofArtificialIntelligenceResearch61:65–170.
[Lin et al. 2018] Lin, X. V.; Wang, C.; Zettlemoyer, L.; andErnst, M. D. 2018. Nl2bash: A corpus and semantic parserfornaturallanguageinterfacetothelinuxoperatingsystem.arXivpreprintarXiv:1802.08979.
[Lingetal.2016]Ling, W.; Grefenstette, E.; Hermann,
K.M.;Kocˇisky`,T.;Senior,A.;Wang,F.;andBlunsom,P. 2016. Latent predictor networks for code generation. arXivpreprintarXiv:1603.06744.
[Oda et al. 2015] Oda, Y.; Fudaba, H.; Neubig, G.; Hata, H.;Sakti,S.;Toda,T.;andNakamura,S.2015.Learningtogeneratepseudo-codefromsourcecodeusingstatisti-cal machine translation (t).In 2015 30th IEEE/ACM In-ternationalConferenceonAutomatedSoftwareEngineering(ASE),574–584.IEEE.
[Pan et al. 2017] Pan, Y.; Yao, T.; Li, H.; and Mei, T. 2017.Video captioning with transferred semantic attributes.InProceedingsoftheIEEEconferenceoncomputervisionandpatternrecognition,6504–6512.
[Papineni et al. 2002] Papineni, K.; Roukos, S.; Ward, T.;and Zhu, W.-J. 2002. Bleu: a method for automatic evalua-tion of machine translation. In Proceedings of the 40th an-nual meeting on association for computational linguistics,311–318.AssociationforComputationalLinguistics.
[Reddy,Harichandana,andTAlekhya2019]Reddy, K.
P. N.; Harichandana, U.; and T Alekhya, R. S. M. 2019. ; AStudy of Robotic Process Automation Among Artificial In-telligence; International Journal of Scientific and ResearchPublications(IJSRP)9(2)(ISSN:2250-3153).
[Rohrbach et al. 2015]Rohrbach, A.; Rohrbach, M.; Tandon,N.; and Schiele, B. 2015. A dataset for movie description.In Proceedings of the IEEE conference on computer visionandpatternrecognition,3202–3212.
[Sales, Handschuh, and Freitas 2017] Sales, J.; Handschuh,S.; and Freitas, A. 2017. Semeval-2017 task 11: End-user development using natural language.In Proceedingsofthe11thInternationalWorkshoponSemanticEvaluation(SemEval-2017),556–564.
[Shi et al. 2017] Shi, T. T.; Karpathy, A.; Fan, L. J.; Hernan-dez, J.; and Liang, P. 2017. World of bits: An open-domainplatform for web-based agents.In Proceedings of the 34thInternationalConferenceonMachineLearning-Volume70,3135–3144.JMLR.org.
[Torabi et al. 2015] Torabi, A.; Pal, C.; Larochelle, H.; andCourville, A. 2015. Using descriptive video services to cre-ate a large data source for video annotation research. arXivpreprintarXiv:1503.01070.
[Van Der Aalst et al. 2011] Van Der Aalst, W.; Adriansyah,A.;DeMedeiros,A.K.A.;Arcieri,F.;Baier,T.;Blickle,T.;Bose, J. C.; Van Den Brand, P.; Brandtjen, R.; Buijs, J.; et al.2011.Process mining manifesto.In International Confer-enceonBusinessProcessManagement,169–194.Springer.
[van Dongen 2015] van Dongen, B. F. 2015. Bpi challenge2015. In 11th International Workshop on Business ProcessIntelligence(BPI2015).
[Wang et al. 2004]Wang, Z.; Bovik, A. C.; Sheikh, H. R.; Si-moncelli, E. P.; et al. 2004. Image quality assessment: fromerrorvisibilitytostructuralsimilarity.IEEEtransactionsonimageprocessing13(4):600–612.
[Zhong, Xiong, and Socher 2017] Zhong, V.; Xiong, C.; andSocher, R.2017.Seq2sql: Generating structured queriesfrom natural language using reinforcement learning.arXivpreprintarXiv:1709.00103.
Post a Comment