Journal of information and
communication convergence engineering
(J. Inf. Commun. Converg. Eng.)

Smart factory based on Information and Communication Technology (ICT) is essential to proactively respond to changes in the market conditions and remain competitive. Recent developments in digital technologies such as smart devices, big data, cloud computing, cyber physical systems, the Internet of Things (IoT), and paradigm shifts have influenced the industry, accelerating changes in the manufacturing environment in these smart factories. In manufacturing, smart factories can implement more advanced solutions in contrast to traditional manufacturing conditions and realize informatization and optimization of the manufacturing process. Th trend continues to drive the automation of production systems using ICT-based smart technologies [1]. The idea of smart factory systems is a valuable and necessary goal, but this approach may be difficult for small companies to fully adopt owing to practical challenges such as poor technology, lack of sufficient personnel, and budget limitations. Therefore, developing such systems in stages by partial application is reasonable starting with simpler parts of the manufacturing process.

The production of wire harnesses is typically performed using traditional production methods in conventional manufacturing environments. A wire harness is a wiring assembly that bundles several wires together, and is typically used to connect or disconnect powered components and power sources of electronic devices. In the manufacturing processes of such wire harnesses, which mainly depends on manual labor, the product inspection process is especially easy to automate, and the benefits of implementing the smart factory idea are considerable. Automation and autonomy in the product inspection process thus seem to be a necessary step towards developing the smart factory idea throughout the entire wire harness manufacturing process.

Automation of the wire harness product inspection process is primarily based on machine vision technology, which refers to a computer application system that automatically performs the intended function by processing images acquired through a camera [2-4]. As this technology processes inputs faster and more precisely than the human eye, it has been applied to various industrial fields, and has contributed considerably to the improvement of productivity in industrial sites [2,5-11]. In inspection areas for products, smart automatic inspection based on machine vision has become an important element of the manufacturing process, which can evolve into elements of the smart factory idea by increasing inspection accuracy and processing efficiency.

Currently, some multi-harness manufacturers have partially adopt machine vision inspection technology in the final product inspection process. Since it contains parts for which vision inspection techniques are difficult to apply owing to the nature of the manufacturing harness, most of them still rely on visual aids such as magnifying glasses and visual inspection by skilled workers. For the inspection method by skilled workers, the processing efficiency or throughput of the entire manufacturing process is subject to visual inspection. This method has challenges such as difficulty in securing skilled workers and high training costs. In addition, it is difficult to maintain the consistency of product inspection since the skill, condition, and external environment of the worker can affect inspection quality. Thus, the inspection method by such skilled workers is becoming a major challenge for improving the productivity and quality of the product, resulting in deteriorating accuracy and efficiency. Therefore, to enseure the reliability and efficiency of the wire harness product manufacturing process and to maintain and supply a stable quality of the product, it is necessary to introduce an automated inspection technology specialized for the wire harness industry.

In this study, we propose a technology system designed to provide automation and real-time autonomous operation services for the entire process related to product inspection during the wire harness manufacturing process. In this work, we first discuss a key requirement for solving the obstacles to applying machine vision technology to the automatic inspection of wire harness products and propose technical solutions to solve them. Then, we describe the design and implementation of a system based on the proposed technology and presents the results. In addition, the proposed technology was verified through an evaluation of the implemented system. As a performance verification for this purpose, we consider seven evaluation indices including precision, detection recognition rate, recall rate, inspection (elapsed) time etc.

According to the research results of this study, automation and improved flexibility of the inspection process can make inspection work easy even for unskilled workers, increasing the efficiency of automatic inspection. In addition, remote real-time monitoring of the inspection process is possible anytime and anywhere, thereby reducing the cost of the entire manufacturing process. Therefore, the proposed technology is a sound system solution that can significantly alleviate or eliminate the limitations and obstacles of existing schemes. This ultimately contributes to facilitating a smart factory approach in harness manufacturing.

The remainder of this study is organized as follows. In Section 2, the related technologies and research work are discussed. Section 3 proposes a solution to this requirement and describes the system design and implementation. Section 4 presents the results of implementing the system solution and discusses its evaluation and verification. This paper concludes with Section 5.

Research works related to this study include the field of machine vision inspection and autonomous operations of inspection-oriented manufacturing processes. Machine vision technology has been applied in a wide range of fields, demonstrating its superior performance and reliability. This technology is based on video detection by cameras, and image processing techniques acquired by cameras have traditionally been developed and applied over a long period of time. Machine vision technologies may be applied in every living without being particularly noticeable [11-14].

An automatic inspection system based on machine vision is a representative technology applied by taking advantage of high performance and reliability using the machine vision technology. Research on automatic inspection systems based on machine vision has been conducted over a long period of, and it has been widely applied to various fields in actual industrial settings [5,6-10]. Most of the conditions of machine vision technology require very strictly different inspection conditions for the product to be inspected, and in reality, these methods have been developed and applied as highly specialized techniques [11,14-18].

Among these methods, automatic inspection of wire harnesses has been studied along with methods of detecting the components by the pattern matching method, or measuring the length using the edge detection method [19]. Studies have also considered error detection [20] and systems for testing automatic connectors for testing flat connectors with different types and numbers of wires [21].

Studies on the autonomous operation of inspection-oriented manufacturing of wire harnesses are rare. The present work aims to address that issue. In this study, we presents a machine vision-based automatic inspection and autonomous operation service solution for color multi-wire harnesses. In contrast to previous studies, we considered color distinctions owing to color variation required in the process of automatic vision inspection by an image camera, simultaneous inspection of multiple product models, improving the recognition direction of the target object that can detect the image of the test object regardless of the position and direction, and the autonomous operation service solution of the entire wire harness inspection process.

A. Ensuring Consistency of the Reference

The requirement of ensuring the consistency of the reference scoping that distinguishes the color of each wire is practically very difficult to follow. There is some variation in the gray or RGB component values of the coated color owing to the heterogeneity of the raw material mixture during the wire production stage. If wires are sourced from different companies, inconsistencies in the raw material formulation of the wire production stages and inhomogeneities of mixing may occur. This results in variations in the gray or RGB component values of the color coated on the wire, even for wires of the same color. This problem can still occur for wires produced by the same company at a lesser degree for various reasons, such as environmental, climatic, and other conditions at the time of manufacture.

In this study, we propose a system to solve this problem. Fig. 1 shows a possible system solution proposed to ensure the consistency of reference scoping. Wire harness products of multiple models were manufactured simultaneously using wire rolls received from external wire manufacturers. Each manufactured harness can be successfully classified into good and bad products through a machine vision-based automatic inspection process.

Figure 1. Proposed solution for ensuring Consistency of the Reference scoping

Fig. 2 shows the operational flow of the autonomous mode set-up and inspection for machine vision-based wire harness inspection. Wire rolls of various colors are manufactured and received by external manufacturers, and in the harness manufacturing stage, wire rolls of the required color are selected and applied in manufacturing. Therefore, a wire harness product produced at the manufacturing stage consists of wires of different colors, each of which is used from its own wire roll. Each colored wire roll was manufactured and delivered by a wire manufacturer. This means that the reference value setting data of each color to be applied in the inspection step may differ for each harness product model, worker, and working time. Thus, wire harness products have several models and each model has different inspection reference data values, according to the wire roll identifier applied to each color. This created multiple modes for the single-wire harness product model. Therefore, in the harness manufacturing step, when the product model number is to be manufactured by the worker, the unique IDs of the wire rolls are applied thereto, and the worker information is acquired by the tag sensor and sent to the server. The server combines this input information to configure the Model & Inspection Modes.

Figure 2. Operation flow of Autonomous mode set-up and Inspection for Machine Vision-based Wire harness inspection

In the inspection phase of the wire harness product, when the product produced in the harness manufacturing step arrives at the inspection device and is ready for inspection, the model number of the product is automatically acquired and transmitted to the server. The server then selects the inspection mode information for a given model number and returns it to the device. The inspection device automatically sets inspection criteria based on the inspection mode information returned from the server, and inspection of the product is performed using these criteria. The inspection result information is then transferred back to the server for storage and management, and the finished harness product is moved to the next step.

Such a process is automatically acquired in real time by a tag sensor or the like in the course of work, and is automatically performed from the manufacture of the harness product to the next step through the inspection step. The administrator or operator can monitor the status information of the product stored and managed on the server with a local monitor or mobile terminal in real time.

B. Flexibility of Recognition Directionality

The second point that must be considered in the inspection of the wire harness is the flexibility of the recognition directionality of the product under test. Therefore, there is a need for an inspection technique that has a relatively low (almost no) constraint on the recognition direction or location of the target product without using any additional dedicated jig for various models. This increases the flexibility of the inspection by automatically setting the inspection system environment according to the inspection target product during the inspection process of the harness, and extracting image data of a specific part regardless of the position or direction in which the inspection target product is placed. This can also increase the performance of the inspection process. This is also essential to ultimately realize automation and autonomous service solutions for all processes, such as the manufacturing, inspection, and management of wire harnesses.

The inspection method for colored wire harnesses proposed in this work improves the recognition direction of the target object for automatic machine vision-based inspection and makes it possible to detect the target image regardless of the position and orientation of the object. This method does not require any additional equipment such as a dedicated jig. The inspection focuses on the mis-wiring of the color wire harness, and the proposed method consists of three stages: detecting the target image, maintaining the orientation of the image (i.e., rotation of the image), and reading the color. The first stage for detecting the target image is a preprocessing step of acquiring an image from a camera and detecting a wire region as an inspection region. The orientation of the image was maintained by rotating the inspection target image around the point of interest to enable reading the color to proceed to the next step. The reading of the color as the last step is to read out the mis-wiring of the color wire harness. This process involves recognizing the color of the wire using the color reference information presented in the inspection mode and comparing the recognized color with a predefined color sequence to check for defects.

C. Design

This study establishes a conceptual model of a machine vision automatic inspection system for a multi-wire harness, and designs and implements the system solution, including its core functions and system components. The system is based on the conceptual model and includes two principal concepts, as discussed in previous sections. They are autonomous mode set-ups for ensuring consistency of the reference scoping and automatic machine vision-based inspection, which improves the recognition direction of the target object. The system also provides a technological solution for autonomous operation and real time monitoring services.

Based on the established system conceptual model, the target system to be implemented is defined as SVI-CH (Smart System for Vision-based Inspecting Colors of a Multiple-wire Harness). This system consists of the Image Capture (IC), Inspection and Interpretation (II), Autonomous and Operational Services (AOS) subsystems, and a communication connection network for data and information transfer between them. The II subsystem comprises software programs for image processing, inspection and interpretation, while the IC consists of vision inspection equipment such as a CCD camera, a lens and an image data grabber. The AOS includes the real-time data detection function for each process step, database and management function, data transmission/reception and communication function, and real-time monitoring access function.

Fig. 3 shows the proposed operational architecture of the machine vision-based color inspection system solution of the multi-wire harness by means of the development target system, SVI-CH. Fig. 4 shows the interaction flow structure among the system elements of the overall system operation.

Figure 3. Operational architecture of the Proposed system solution

Figure 4. Interaction flow structure among System Elements

Tags and tag readers are primarily used for the real-time data detection function of each process step. To obtain information on the wire harness product to be inspected before inspection, a tag is attached to the product carrier box during the production process of the harness, which includes information about the producer, time, and product model. The box with this tag is moved to the inspection standby area, where it is registered as an inspection standby by the tag reader. The contents of this box read by the tag reader are registered in the DB through the server, by creating a new task, and recording the inspection state as “Waiting for Inspection”. The production of harnesses is carried out on several production lines simultaneously, and several harness product carrier boxes are registered at the inspection waiting place and wait for inspection.

In the vision inspection process, the inspector begins the inspection by reading the tags of the harness product carrier box under test and the inspector’s own tags onto the tag reader. In this way, the inspector information, time data, and harness product carrier box ID (identifier) to be inspected are transmitted to the server, and the task with this ID is searched among the tasks that are in the “Waiting for Inspection” state in the database (DB). It updates such necessary information about the searched task as inspector information, time information, etc. and changes the state of the task from “Waiting for Inspection” to “Under Inspection”. The inspector then registers the IDs of both the good and defective product boxes in the DB by reading their tags in the tag reader. These boxes contain inspected harness products as either good or defective. This is useful for automatically acquiring history information about the harness product box, which is inspected by the tag reader in the next process step. Furthermore, this approach is ultimately necessary for existing manufacturing facilities to evolve into smart factory systems. When the inspection is completed in the inspection process step, the inspection result information, such as inspection completion time, total quantity of inspected products, and quantities of good or defective products after inspection, stored in the II is delivered to the server, and the series of inspection processes is completed by updating the DB information of the task. At this time, the state of the task is changed from “Under Inspection” to “Inspection Completed”.

An operations manager can monitor these processes and results in real-time at any time through the operation manager terminal at the local site. Outside, it is possible to monitor these processes and results in real time by accessing the web server via the Internet.

A. Implementation Result

Fig. 5 shows a monitor screen capture of the message transmitting and receiving execution between the tag readers, vision inspection device (i.e., II), and the cloud server. When the message is successfully received, the server returns a “200” to designate the communication protocol, indicating that the message communication was successful. If an error occurs while processing a command, it displays an error message.

Figure 5. Screen capture of Message Communication between Tag readers, Vision inspection device and Cloud server.

Fig. 6 is a manager monitor screen accessed remotely through the Internet, showing the real-time status of the inspection work in progress: (a) a screen capture of a manager’s monitor, and (b) its menu translated in English.

Figure 6. Manager’s monitor screen showing Real-time Inspection work status

This figure shows the overall execution results of the system solution proposed and implemented in this study. It is possible to see the same screen in real time anytime, anywhere via a web browser on a mobile smart terminal or PC and accessing the given URL address. The screen updates automatically at regular intervals; therefore, it always shows the current states.

Registration No is the task number generated when the harness product carrier box is first registered using Tag Reader #1. One task includes a number of elements showing Registration Time, Progress Status, Waiting Carrier ID, Product Model, Quantity to be Inspected, Total Inspection Quantity, Worker ID, Inspector ID, Inspection Start Time, Inspection Completion Time, Carrier ID for Goods, Carrier ID for Defective, and No of Goods and No of Defective Goods. The Progress Status element indicates one of three states: Waiting for Inspection, Under Inspection, or Inspection Completed. Each of these elements is updated in real time as the inspection proceeds.

The results displayed on the screen during the overall execution of this system solution usually include three types of state change, as described above. At the manufacturing stage, the worker places a number of manufactured wire harness products into a carrier box and moves them to the inspection site. Being ready for inspection, the box is registered through the tag attached to it and the tag reader provided at the inspection site. The server then assigns a registration number during the initial registration and displays the element data. The element of Progress Status indicates in this case Waiting for Inspection as shown in Fig. 7.

Figure 7. State of Waiting for Inspection

The inspection starts with the inspector registering his tag and recognizing the tags attached to the goods carrier box and the defective goods carrier box, respectively. These two boxes are intended to contain the inspected products classified as good or defective after the completion of the inspection. The inspection then proceeds by displaying data of elements such as Inspector ID, Inspection Start Time, Goods, Defective Goods Carrier Boxes IDs, etc., at which time the state value of Progress Status changes to Under Inspection. Fig. 8 illustrates this situation.

Figure 8. Transition to the state of Under Inspection

When the inspector completes the product inspection and sends the inspection completion signal to the server, the Progress Status transitions to the state, Inspection Completed. Simultaneously, it is possible to confirm information such as Total Inspection Quantity, Inspection Result Numbers of Goods and Defective Goods, Inspection Completion Time. Fig. 9 shows the status after the completion of the inspection.

Figure 9. Transition to the state of Inspection Completed

Fig. 10 shows the overall test environment for the results of implementing this system solution. It shows the machine vision-based inspection device (i.e. IC and II), the computer running the software and a display monitor, tags and tag readers, etc. The WiFi network environment and a cloud server are in different locations.

Figure 10. Test environment for Implemented system solution

Fig. 11 is a screen capture of the monitor displaying the results of the automatic machine vision-based inspection of a wire harness product executed by the proposed system solution.

Figure 11. Screen capture of the Monitor displaying Final decision result

Fig. 12 and Fig. 13 are monitor screens showing the results of wire harness inspection executed by this system solution, Goods and Defective Goods, respectively.

Figure 12. Monitor screens displaying Final decision “Good” of Inspection

Figure 13. Monitor screens displaying Final decision “NG (Not Good, Defective)” of Inspection

B. Verification and Test Results

As a result of the proposed and actual implementation in this study, we verified the function and performance of the system solution through a self-test. There are no standardized reference values for these test items, which are usually determined within the industry itself. To ensure objectivity, seven test items were set up, and an evaluation test was requested to Korea Conformity Laboratories (KCL), an external accredited test institute.

The seven test item sets were color discrimination capability, inspection time, precision, detection recognition rate, recall rate, success rate of autonomous positioning and automatic mode setting. During this evaluation test, 200 samples of real products, including 196 good and four defective wire harnesses, were applied to the system solution implemented using the proposed technology and used as inspection targets to determine good or defective units. The precision is the ratio of the number of recognized good products to the total number of good products, the detection recognition rate is the ratio of correctly recognized good and defective goods by adding a limited number of good and defective items to the vision inspection system. The recall is the ratio of the number of actual goods classified as good when the wire-harness products of good or defective quality are placed in the vision inspection system. In addition, the success rate of autonomous positioning involves checking the autonomous positioning of the image after positioning the inspection object, that is, a wire harness product, arbitrarily in the inspection area. When placing an object for inspection, it is not limited to specific criteria or directions. Tests for this item were included in the other tests above.

In this evaluation test, precision, detection recognition rate, recall rate and success rate of autonomous positioning were all 100%. The goal was also achieved in the ten kinds of color discrimination capability, the inspection (elapsed) time from image capture to final decision within 0.3 second, and the four automatic mode settings that were originally set. From these results, it can be seen that the machine vision automatic inspection and operation service system solution of the multi-wire harness proposed in this study was justified, and the proposed technologies were verified.

In product inspection areas in manufacturing, smart automatic inspection based on machine vision has become an important element of the manufacturing process, which has the potential to support the development of into a smart factory systems by increasing inspection accuracy and processing efficiency. The manufacturing process of wire harnesses can benefit from automation because of its current reliance on manual labor and processes. Automation and autonomy in the product inspection process thus seem to be a necessary part of the smart factory in the entire wire harness manufacturing process.

This study proposes a specialized technology and system solution to provide automation and a real-time autonomous operation service for the entire process related to product inspection during the wire harness manufacturing process. A technical solution to solve the obstacles in applying machine vision technology to the automatic inspection of wire harness products is presented. The solution addresses how to ensure the consistency of the reference scoping that distinguishes the color of each wire, how to enable the flexibility of the recognition directionality of the product under test, and how to provide autonomous and operational services for the machine vision-based inspection of multiple-wire harnesses and real-time monitoring of the inspection status. The proposed technological solution not only lowers the cost of inspection, but also enables multiple inspections to be performed simultaneously, resulting in the improvement of inspection performance and the efficiency and flexibility of the entire manufacturing operation.

Next, the design and implementation of a system solution based on the proposed technologies are presented, along with the results. The implemented system was defined as SVI-CH which means a smart system for vision-based inspecting colors of a multiple-wire harness. This system consists of image capture, inspection, interpretation and autonomous and operational service subsystems, and a communication connection network for data and information transfer between them. The autonomous and operational service subsystem includes the real-time data detection function of each process step, database and management function, data transmission/reception and communication function, and real-time monitoring access function. In this study, we have described how this system can autonomously inspect wire harness products and their associated overall process operations.

In addition, we verified the effectiveness of the proposed technology through an evaluation of the implemented system solution. Seven test items were set up for the evaluation test, which was requested by an accredited test institute. A total of 200 samples of real products including 196 good and four defective wire harnesses were applied during the evaluation test. From these evaluation tests, the target values were achieved for all seven test items. It can be seen from these results that the machine vision-based automatic inspection and operation service system solution for the multi-wire harness proposed in this paper was justified, and the proposed technologies were also verified.

The technologies and implemented results presented in this paper not only lower the incidental costs for the inspection but also increase the performance of the inspection by enabling simultaneous and multiple automatic tests of multiple models. In addition, remote real-time monitoring is available anytime and anywhere. This results in improved efficiency and flexibility for the entire manufacturing industry. Accordingly, the inspection process is shortened, and the flexibility of the inspection work is improved to facilitate inspection work even by unskilled workers, thereby increasing the automatic inspection efficiency and reducing the overall cost. These ultimately become the basis toward smart factories.

This work was supported by a grant from Inje University for Research in 2018.