A dynamic framwork for intelligent inspection

CAD Computer Aided Design typically involves the design manufacture and inspection of a me chanical part. The problem of reverse engineering is to take an existing mechanical part as the point of departure and to inspect or produce a design and perhaps a manufacturing process for the part. We propose to explore the feasibility of a new approach to inspection and reverse engineering applications. In particular we investigate the use of discrete event dynamic systems DEDS to guide and control the active exploration and sensing of mechanical parts for industrial inspection?? The proposed framework utilizes DEDS for constructing an observer for inspection purposes??

A Dynamic Framework for Intelligent Inspection Tarek M􀀀 Sobh Chris Jaynes and Thomas C􀀀Henderson􀀀 UUCS􀀀 􀀀 Department of Computer Science University of Utah Salt Lake City UT USA October Abstract CAD Computer Aided Design typically involves the design manufacture and inspection of a me chanical part􀀀 The problem of reverse engineering is to take an existing mechanical part as the point of departure and to inspect or produce a design and perhaps a manufacturing process for the part􀀀 We propose to explore the feasibility of a new approach to inspection and reverse engineering applications􀀀 In particular we investigate the use of discrete event dynamic systems DEDS to guide and control the active exploration and sensing of mechanical parts for industrial inspection􀀀 The proposed framework utilizes DEDS for constructing an observer for inspection purposes􀀀 􀀀This work was supported in part by DARPA grant N􀀀􀀀􀀀 J and NSF grant CDA 􀀀 All opinions ndings conclusions or recommendations expressed in this document are those of the author and do not necessarily re ect the views of the sponsoring agencies 􀀀 Introduction Developing environments for inspection applications is an essential activity in many engineering disciplines􀀀 Usually too much time is spent in designing hardware and software environments in order to be able to attack a speci c problem􀀀 One of the purposes of this work is to provide a basis for solving a class of inspection and reverse engineering problems􀀀 The technique to be explored can hopefully be used for a variety of applications􀀀 We use an observer agent to sense the current world environment and make some measurements then supply relevant information to a control module that will be able to make some design choices that will later a ect manufacturing and or inspection activities􀀀 This involves both autonomous and semi autonomous sensing􀀀 Objectives and Questions The objective of this research project is to explore the basis for a consistent software and hardware environment and a exible system that is capable of performing a variety of inspection activities􀀀 In particular we will concentrate on the adaptive automatic extraction of some properties of the world to be sensed and on the subsequent use of the sensed data for producing reliable descriptions of the sensed environments for manufacturing and or description re nement purposes􀀀 We use an observer agent with some sensing capabilities vision and touch to actively gather data measurements of mechanical parts􀀀 Our thesis is that 􀀀 Discrete Event Dynamical Systems DEDS provide the base for de ning consistent and adaptive control structures for the inspection and reverse engineering problem􀀀 If this is true then we will be able to answer the following questions 􀀀 What is a suitable algorithm to coordinate sensing inspection design and manufacturing 􀀀 What is a suitable control strategy for sensing the mechanical part 􀀀 Which parts should be implemented in hardware vs􀀀 software 􀀀 What are suitable language tools for constructing a reverse engineering and or inspection strategy We describe DEDS in more detail later but they can be simply described as Dynamic systems typically asynchronous in which state transitions are triggered by dis crete events in the system􀀀 It is possible to control and observe hybrid systems systems that involve continuous discrete and symbolic parameters under uncertainty using DEDS formulations 􀀀 The applications of this work are numerous e􀀀g􀀀 automatic inspection of mechanical or electronic components and reproduction of mechanical parts􀀀 Moreover the experience gained in performing this research will allow us to study the subdivision of the solution into reliable reversible and an easy to modify software and hardware environments􀀀 Methodology for Inspection In this section we describe the solution methodology and discuss the components separately􀀀 The control ow is also described and the methods speci c equipment and procedures to be designed and implemented are also discussed in detail􀀀 We use a vision sensor B W CCD camera and a coordinate measuring machine CMM with the necessary software interfaces to a Sun Sparcstation as the sensing devices􀀀 The object is to inspected by the co operation of the observer camera and the probing CMM a DEDS is used as the high level framework for exploring the mechanical part􀀀 We next discuss DEDS in general and then we proceed to apply the framework for sensing the inspection process􀀀 􀀀 Discrete Event Dynamic Systems Discrete event dynamic systems are dynamic systems typically asynchronous in which state transitions are triggered by the occurrence of discrete events in the system􀀀 DEDS are usually modeled by nite state automata with partially observable events together with a mechanism for enabling and disabling a subset of state transitions 􀀀 We propose that this model is a suitable framework for many reverse engineering tasks􀀀 In particular we use the model as a high level structuring technique for our system􀀀 We can represent a DEDS by the following quadruple G X􀀀 􀀀 U􀀀 where X is the nite set of states is the nite set of possible events U is the set of admissible control inputs consisting of a speci ed collection of subsets of corresponding to the choices of sets of controllable events that can be enabled and is the set of observable events􀀀 We can visualize the concept of DEDS by means of the example in Figure 􀀀 The graphical representa tion is quite similar to a classical nite automaton􀀀 Here circles denote states and events are represented by arcs􀀀 The rst symbol in each arc label denotes the event while the symbol following denotes the corresponding output if the event is observable 􀀀 Finally we mark the controllable events by u 􀀀 Thus in this example X f 􀀀 􀀀 􀀀 g f 􀀀 􀀀 g f 􀀀 g and is controllable at state but not at state 􀀀 An alive state is a state that can never undergo transitions leading to a state that has no outgoing transitions a dead state 􀀀 A system A is alive if all its states are alive􀀀 Stability can be de ned with respect to the states of a DEDS automaton􀀀 Assuming that we have identi ed the set of good states E that we would like our DEDS to stay within or to not stay outside for an in nite time then stabilizability can be formally de ned as follows Given a live system A and some E X x X is stabilizable with respect to E or E stabilizable if there exists a combination of controllable events control pattern K such that x is alive and does not stay outside E forever E stable when K is used􀀀 A set of states Q is a stabilizable set if there exists a control pattern K so that every x Q is alive and stable in AK A under the control pattern K and A is a stabilizable system if X is a stabilizable set􀀀 A DEDS is termed observable if we can use any sequence of observable events to determine the current state exactly at intermittent points in time separated by a bounded number of events􀀀 More formally take any su ciently long string s that can be generated from any initial state x􀀀 For any observable system / / : u / 1 2 0 3 Figure A Simple DEDS Example we can then nd a pre x p of s such that p takes x to a unique state y and the length of the remaining su x is bounded by some integer no􀀀 Also for any other string t from some initial state x 􀀀 such that t has the same output string as p we require that t takes x 􀀀 to the same unique state y􀀀 The basic idea behind strong output stabilizability is that we will know that the system is in state E i the observer state is a subset of E􀀀 The compensator should then force the observer to a state corresponding to a subset of E at intervals of at most a nite integer i of observable transitions􀀀 If Z is the set of states of the observer then A is strongly output E stabilizable if there exists a state feedback K for the observer O such that OK is stable with respect to EO f x Z j x Eg􀀀 We advocate an approach in which a stabilizable semi autonomous visual sensing interface would be capable of making decisions about the state of the observed machine part and the probe􀀀 Thus providing both symbolic and parametric descriptions to the reverse engineering and or inspection control module􀀀 The DEDS based active sensing interface will be discussed in the following section􀀀 Modeling and Constructing an Observer The tasks that the autonomous observer system executes can be modeled e ciently within a DEDS frame work􀀀 We use the DEDS model as a high level structuring technique to preserve and make use of the information we know about the way in which a mechanical part should be explored􀀀 The state and event description is associated with di erent visual cues for example appearance of objects speci c D move ments and structures interaction between the touching probe and part and occlusions􀀀 A DEDS observer serves as an intelligent sensing module that utilizes existing information about the tasks and the environ ment to make informed tracking and correction movements and autonomous decisions regarding the state of the system􀀀 In order to know the current state of the exploration process we need to observe the sequence of events occurring in the system and make decisions regarding the state of the automaton􀀀 State ambiguities are allowed to occur however they are required to be resolvable after a bounded interval of events􀀀 The goal will be to make the system a strongly output stabilizable one and or construct an observer to satisfy speci c task oriented visual requirements􀀀 Many D visual cues for estimating D world behavior can be used􀀀 Examples include image motion shadows color and boundary information􀀀 The uncertainty in the sensor acquisition procedure and in the image processing mechanisms should be taken into consideration to compute the world uncertainty􀀀 Foveal and peripheral vision strategies could be used for the autonomous focusing on relevant aspects of the scene􀀀 Pyramid vision approaches and logarithmic sensors could be used to reduce the dimensionality and computational complexity for the scene under consideration􀀀 Error States and Sequences We can utilize the observer framework for recognizing error states and sequences􀀀 The idea behind this recognition task is to be able to report on visually incorrect sequences􀀀 In particular if there is a pre determined observer model of a particular inspection task under observation then it would be useful to determine if something goes wrong with the exploration actions􀀀 The goal of this reporting procedure is to alert the an operator or autonomously supply feedback to the inspecting robot so that it could correct its actions􀀀 An example of errors in inspection is unexpected occlusions between the observer camera and the inspection environment or probing the part in a manner that might break the probe􀀀 The correct sequences of automata state transitions can be formulated as the set of strings that are acceptable by the observer automaton􀀀 This set of strings represents precisely the language describing all possible visual task evolution steps􀀀 T 1 T 2 T 3 Figure A Hierarchy of Tasks􀀀 Hierarchical Representation Figure shows a hierarchy of three submodels􀀀 Motives behind establishing hierarchies in the DEDS modeling of di erent exploration tasks includes reducing the search space of the observer and exhibiting modularity in the controller design􀀀 This is done through the designer who subdivides the task space of the exploring robot into separate submodels that are inherently independent􀀀 Key events cause the transfer of the observer control to new submodels within the hierarchical description􀀀 Transfer of control through the observer hierarchy of models allows coarse to ne shift of attention in recovering events and asserting state transitions􀀀 Mapping Module The object of having a mapping module is to dispense with the need for the manual design of DEDS automaton for various platform tasks􀀀 In particular we would like to have an o line module which is to be supplied with some symbolic description of the task under observation and whose output would be the code for a DEDS automata that is to be executed as the observer agent􀀀 A graphical representation of the mapping module is shown in Figure 􀀀 The problem reduces to guring out what is an appropriate form for the task description􀀀 The error state paradigm motivated regarding this problem as the inverse problem of determining acceptable languages for a speci c DEDS observer automaton􀀀 In particular we suggest a skeleton for the mapping module that transform a collection of input strings into an automaton model􀀀 Mapping Module DEDS Automaton Conditions Transition Task Language Figure The Mapping Module􀀀 The idea is to supply the mapping module with a collection of strings that represents possible state transition sequences􀀀 The input highly depends on the task under observation what is considered as relevant states and how coarse the automaton should be􀀀 The sequences are input by an operator􀀀 It should be obvious that the Garbage in garbage out principle holds for the construction process in particular if the set of input strings is not representative of all possible scene evolutions then the automaton would be a faulty one􀀀 The experience and knowledge that the operator have would in uence the outcome of the resulting model􀀀 However it should be noticed that the level of experience needed for providing these sets of strings is much lower than the level of experience needed for a designer to actually construct a DEDS automaton manually􀀀 The description of the events that cause transitions between di erent symbols in the set of strings should be supplied to the module in the form of a list􀀀 As an illustrative example suppose that the task under consideration is simple grasping of one object and that all we care to know is three con gurations whether the hand is alone in the scene whether there is an object in addition to the hand and whether enclosure has occurred􀀀 If we represent the con gurations by three states h ho and hc then the operator would have to supply the mapping module with a list of strings in a language whose alphabet consists of those three symbols and those strings should span the entire language so that the resulting automaton would accept all possible con guration sequences􀀀 The mapping from a set of strings in a regular language into a minimal equivalent automaton is a solved problem in automata theory􀀀 One possible language to describe this simple automaton is L hh􀀀hoh􀀀 ohch􀀀 c and a corresponding DEDS automaton is shown in Figure 􀀀 H HO HC Figure An Automaton for Simple Grasping􀀀 The best case scenario would have been for the operator to supply exactly the language L to the mapping module with the appropriate event de nitions􀀀 However it could be the case that the set of strings that the operator supplies do not represent the task language correctly and in that case some learning techniques would have to be implemented which in e ect augment the input set of strings into a language that satis es some pre determined criteria􀀀 For example y􀀀 is substituted for any string of y s having a length greater than n and so on􀀀 In that case the resulting automaton would be correct up to a certain degree depending on the operator s experience and the correctness of the learning strategy􀀀 􀀀 Sensing Strategy We use a B W CCD camera mounted on a tripod and a coordinate measuring machine CMM to sense the mechanical part􀀀 A discrete event dynamic system DEDS algorithm is used to facilitate the state recovery of the inspection process􀀀 DEDS are suitable for modeling robotic observers as they provide a means for tracking the continuous discrete and symbolic aspects of the scene under consideration 􀀀 Thus the DEDS controller will be able to model and report the state evolution of the inspection process􀀀 In inspection the DEDS guides the sensing machines to the parts of the objects where discrepancies occur between the real object or a CAD model of it and the recovered structure data points and or parameters􀀀 The DEDS formulation also compensates for noise in the sensor readings both ambiguities and uncertainties using a probabilistic approach for computing the D world parameters 􀀀 The recovered data from the sensing module is then used to drive the CAD module􀀀 The DEDS sensing agent is thus used to collect data of a passive element for designing structures an exciting extension is to use a similar DEDS observer for moving agents and subsequently design behaviors through a learning stage􀀀 Experiment An experiment was performed for inspecting a mechanical part using a camera and the coordinate measur ing machine􀀀 A prede ned DEDS state machine was used as the observer agent skeleton and the mechanical Figure Experimental Setup part was put on a black velvet background on top of the coordinate measuring machine table to simplify the vision algorithms􀀀 The camera was placed on a stationary tripod at the base of the table so that the part was always in view􀀀 The probe could then extend into the eld of view and come into contact with the part as shown in Figure 􀀀 The experiment proceeds as follows􀀀 First an image is captured from the camera􀀀 Next the appropriate image processing takes place to nd the position of the part the number of features observed and the location of the probe􀀀 A program using this information produces a state signal that is appropriate for the scene􀀀 The signal is read by the state machine and the next state is produced and reported􀀀 DEDS State Machine The speci c automata chosen for the test was a state machine G􀀀 Where X fInitial EOF Error A B C Dg and f eofg􀀀 The state transitions were controlled by the input signals supplied by intermediate vision programs􀀀 There are four stable states A B C and D that describe the state of the probe and part in the scene􀀀 The three other states Initial Error and EOF specify the actual state of the system in special cases􀀀 The states can interpreted as 􀀀 Initial State Waiting for rst input signal 􀀀 A Part Alone in Scene 􀀀 B Probe and Part in Scene probe is far from part􀀀 􀀀 C Probe and Part in Scene probe is close to part􀀀 􀀀 D Probe touching or overlapping part􀀀 􀀀 Error An invalid signal was received􀀀 􀀀 EOF The End of File signal was received􀀀 not 1,2,3,4,5,6,7 not 2,3,4,5,6,7,8,9 not 1 not 2,3,4,89 2,3,4 5,6,7 not 1,5,6,7 1 EOF EOF EOF EOF EOF 2,3,4 d < 50 8,9 8,9 5,6,7 d > 50 d < 50 2,3,4 1 d > 50 1 5,6,7 EOF ERROR Inital State A B C D Figure State Machine Used in Test Vision and Touch A layer of processing must exist between the state machine and the camera that provides the proper input signals􀀀 This layer is made up of several vision lters that are applied to each image as it is captured􀀀 Several things must be known about the scene before a signal is produced􀀀 The location of the part the location of the probe the pixel distance between them the number of features on the part and the distance to the closest feature􀀀 Because the part to be inspected is known before the test is run we are able to adjust the vision lters for the task at hand􀀀 We are also able to make speci c assumptions about the images we expect to receive􀀀 The vision problem then becomes a simple series of lters􀀀 First the image is thresholded at a gray level that optimizes the loss of background while retaining the part and probe􀀀 Next a median lter is applied that removes small regions of noise􀀀 The image is then parsed to nd all segments separated by an appropriate pixel distance and labels them with a unique region identi er􀀀 We are able to assume that the probe if in the scene will always intersect the image border􀀀 The probe tip is the farthest point on the probe region from the border􀀀 This holds true because of the geometry of the probe􀀀 An image with one region that intersects the border is the case in which the probe is touching the part􀀀 If we have more than one region we must discover the distance between the tip of the probe region and the part􀀀 This is done through an edge following algorithm that gives us the x􀀀 y positions of the pixels on the edge of each region􀀀 We then nd the Euclidean distances between the edge points and the probe tip􀀀 The closest point found is used in producing the signal to the state machine􀀀 Features on the part are considered to be edge responses that are interior to the part boundary􀀀 Distance to the closest feature from the probe tip is found using the same method as before􀀀 􀀀 Results A typical sequence from a probing task was run􀀀 The probe was introduced to the scene and moved in a legal way accepted by stable states in the machine towards the part until contact was made􀀀 Next the probe backed o and again approached until the probe and part overlapped􀀀 The automaton was forced into an error state by approaching from the other side of the part much too fast􀀀 The probe was not seen until it was too close to the object body􀀀 Because a transition from state A to C is invalid and error state is reached􀀀 Each image was displayed on a terminal window as it was captured along with the corresponding state of the automaton􀀀 Current Developments The application environment we eventually intend to develop consists of three major working elements the sensing design and manufacturing modules􀀀 The ultimate goal is to establish a computational framework that is capable of deriving designs for machine parts or objects inspect and re ne them while creating a exible and consistent engineering environment that is extensible􀀀 The control ow is from the sensing module to the design module and then to the manufacturing component􀀀 Feedback can be re supplied to the sensing agent to inspect manufactured parts compare them to the originals and continue the ow in the loop until a certain tolerance is met􀀀 The system is intended to be ultimately as autonomous as possible􀀀 We intend to study what parts of the system can be implemented in hardware􀀀 Some parts seem to be inherently suited to hardware which will be discussed later some other parts of the system may be possible to put in hardware but experimentation will provide the basis for making that decision􀀀 Providing language interfaces between the di erent components in the inspection and reverse engineering control loop is an integral part of the project􀀀 Robotics and Sensing We intend to use a robot arm a PUMA a vision sensor B W CCD camera mounted on the end e ector and a coordinate measuring machine CMM 􀀀 A discrete event dynamic system DEDS algorithm will be used to coordinate the movement of the robot sensor and the CMM􀀀 The DEDS control algorithm will also guide the CMM to the relevant parts of the objects that need to be explored in more detail curves holes complex structures etc􀀀 Computer Aided Design and Manufacturing The data and parameters derived from the sensing agent are then to be fed into the CAD system for designing the geometry of the part s under inspection􀀀 We intend to use the design environment for that purpose􀀀 The goal is to provide automatic programming interfaces from the data obtained in the sensing module to the programming environment􀀀 The parametric and D point descriptions STATE A: Probe Not In Scene STATE B: Probe Approaching Object STATE C: Probe Close To Object STATE D: Probe Touching Object STATE C: Probe Close To Object STATE B: Probe Approaching Object STATE B: Probe Approaching Object STATE C: Probe Close To Object STATE D: Probe Touching Object STATE C: Probe Close To Object STATE B: Probe Approaching Object STATE A: Probe Not In Scene ERROR STATE STATE A: Probe Not In Scene STATE B: Probe Approaching Object STATE A: Probe Not In Scene are to be integrated to provide consistent and e cient surface descriptions for the CAD tool􀀀 For pure inspection purposes the computer aided geometric description of parts could be used as a driver for guid ing both the robotic manipulator and the coordinate measuring machine for exploring the object and recognizing discrepancies between the real part and the model􀀀 The computer aided design parameters are then to be used for manufacturing the prototypes􀀀 Con siderable e ort has been made for automatically moving from a computer aided geometric model to a process plan for making the parts on the appropriate NC machines and then to automatically generate the appropriate machine instructions 􀀀 We intend to use the Monarch VMC milling machine as the manufacturing host􀀀 The system will produce the NC code for manufacturing the parts􀀀 􀀀 VLSI and Languages The software and hardware requirements of the environment are the backbone for this project􀀀 We intend to select parts of the system implementation and study the possibility of hardwiring them􀀀 There has been considerable e ort and experience in VLSI chip design and one of the sub problems would be to study the need and e ciency of making customized chips in the environment􀀀 The DEDS model as an automaton is very suitable for Path Programmable Logic PPL implementation􀀀 A number of the visual sensing algorithms could be successfully implemented in PPL saving considerable computing time􀀀 Integrated circuits for CAGD surface manipulation is an e ort that is already underway􀀀 We intend to investigate a new area the possibility of implementing the DEDS part of the system in integrated circuitry􀀀 There is a lot of interfacing involved in constructing the inspection and reverse engineering environments under consideration􀀀 Using multi language object based communication and control methodology between the three major components Sensing CAD and CAM is essential􀀀 We intend to use a common shared database for storing data about the geometric model and the rules governing the interaction of the di erent phases in the reproduction and inspection paradigms 􀀀 We also intend to use a graphical behavior design tool for the automatic production of the sensing DEDS automata code from a given control language description􀀀 Conclusions We propose a new strategy for inspection and or reverse engineering􀀀 We concentrate on the inspection of machine parts􀀀 We also describe a framework for constructing a full environment for generic inspection and reverse engineering􀀀 The problem is divided into sensing design and manufacturing components with an underlying software and hardware backbone􀀀 This project aims at developing control strategies for sensing the world and coordinating the di erent activities between the phases􀀀 We use a DEDS framework to construct an intelligent observer module for inspection􀀀 The developed framework utilizes existing knowledge to formulate an adaptive and goal directed strategy for exploring mechanical parts􀀀 References R􀀀 Bajcsy Active Perception Proceedings of the IEEE Vol􀀀 No􀀀 August 􀀀 A􀀀 Benveniste and P􀀀 L􀀀 Guernic Hybrid Dynamical Systems Theory and the SIGNAL Language IEEE Transactions on Automatic Control Vol􀀀 No􀀀 May 􀀀 M􀀀 J􀀀 Bradakis Reactive Behavior Design Tool Master s Thesis Computer Science Department University of Utah January 􀀀 T􀀀 M􀀀 Carter K􀀀 F􀀀 Smith S􀀀 R􀀀 Jacobs and R􀀀 M􀀀 Ne Cell Matrix Methodologies for Integrated Circuit Design Integration􀀀 The VLSI Journal 􀀀 F􀀀 Chaumette and P􀀀 Rives Vision Based Control for Robotic Tasks In Proceedings of the IEEE International Workshop on Intelligent Motion Control Vol􀀀 pp􀀀 August 􀀀 S􀀀 Drake and S􀀀 Sela A Foundation for Features Mechanical Engineering January 􀀀 J􀀀 Gu and K􀀀 Smith A Structured Approach for VLSI Circuit Design IEEE Computer 􀀀 C􀀀 D􀀀 Hansen and T􀀀 C􀀀 Henderson CAGD Based Computer Vision IEEE Transactions on Pattern Analysis and Machine Intelligence PAMI 􀀀 G􀀀 Lindstrom J􀀀 Maluszynski and T􀀀 Ogi Using Types to Interface Functional and Logic Program ming July pp􀀀 technical summary submitted to SIGPLAN Symposium on Principles of Programming Languages􀀀 A􀀀 Nerode and J􀀀 B􀀀 Remmel A Model for Hybrid Systems Presented at the Hybrid Systems Workshop Mathematical Sciences Institute Cornell University May 􀀀 C􀀀 M􀀀 Ozveren Analysis and Control of Discrete Event Dynamic Systems A State Space Approach Ph􀀀D􀀀 Thesis Massachusetts Institute of Technology August 􀀀 R􀀀 F􀀀 Riesenfeld Mathematical Methods in Computer Aided Geometric Design chapter Design Tools for Shaping Spline Academic Press 􀀀 T􀀀 M􀀀 Sobh and R􀀀 Bajcsy A Model for Observing a Moving Agent Proceedings of the Fourth International Workshop on Intelligent Robots and Systems IROS Osaka Japan November 􀀀 M􀀀 Swanson R􀀀 Kessler Domains e cient mechanisms for specifying mutual exclusion and disci plined data sharing in concurrent scheme First U S Japan Workshop on Parallel August 􀀀 J􀀀 A􀀀 Thingvold and E􀀀 Cohen Physical Modeling with B Spline Surfaces for Interactive Design and Animation Proceedings of the Symposium on Interactive D graphics ACM March 􀀀