Multisource and color lighting for detection of small protuberances

Proceedings Chapter

Reference

Multisource and color lighting for detection of small protuberances

BRUNET, Denis, PUN, Thierry

Abstract

In typical machine vision problems such as quality control or object location, it is often the case that elements of interest are small protuberances over a surface. We present an innovative and robust approach aiming at detecting such protuberances. Its basic ideas are to detect the shadows produced by the protuberances and to use several light sources simultaneously to enhance detection. Each light source produces a different set of shadows;

combining the shadows produced by all light sources helps to locate the protuberance, because these shadows are the only significantly varying patterns between views. Rather than using several white light sources in sequence, it is possible to use simultaneous color sources with appropriate filters to separate the image into independent channels. The approach has been validated on a concrete problem with highly variable protuberances and nonplanar surfaces. The results confirm the robustness of this approach, which could be used for other problems as well.

BRUNET, Denis, PUN, Thierry. Multisource and color lighting for detection of small

protuberances. In: SPIE Conference on Advances in Intelligent Robotic Systems. Optics, Illumination and Image Sensing for Machine Vision VI. SPIE, 1991. p. 23-34

DOI : 10.1117/12.57967

Available at:

http://archive-ouverte.unige.ch/unige:47724

Disclaimer: layout of this document may differ from the published version.

Multisource and color lighting for detection of small protuberance

Denis Brunet, Thierry Pun

Computer Vision Group, University of Geneva, 12 rue du Lac CH - 1207 Geneva, Switzerland

ABSTRACT

In typical machine vision problems such as quality control or object location, it is often the case that ele- ments of interest are small protuberances over a surface. We present an innovative and robust approach aim- ing at detecting such protuberances. Its basic ideas are: a) to detect the shadows produced by the protuber- ances, b) to use several light sources simultaneously to enhance detection.

Each light source produces a different set of shadows; combining the shadows produced by all light sources allows to locate the protuberance, since these shadows are the only significantly varying patterns between views. Rather than using several white light sources in sequence, it is possible to use simultaneous color sources with appropriate filters to separate the image into independent channels.

The approach has been validated on a concrete problem with highly variable protuberances and non-planar surfaces. The results confirm the robustness of this approach, that could be used for other problems as well.

1. INTRODUCTION

The present work was initially performed in the context of the Potato Operation^1,2,3, where potatoes taken from a bunch have to be tested for the presence of viruses. This is done by extracting some pulp with a drill, which is plunged into the potato under the biggest germ or group of germs. It is therefore necessary to have a robust method for locating small germs on the surface of a potato^4,5,6. Many difficulties render this seem- ingly easy task fairly difficult. First, germs as well as skin have various colors and shapes; second, the con- trast between germ and skin is often poor; third, defects of the skin such as scars, spots or highlights can introduce false detections if only small germs are present.

The solution presented here consists in detecting the shadows generated by the germs, and using several col- or light sources to cross-check each set of shadows and therefore enforce detection reliability. Section 2.

briefly presents the geometrical model for the potato and the germs. Section 3. studies some characteristics of the shadows that are generated by a single source, and Section 4. deals with the constraints inherent to multisource configurations. Section 5. presents some concrete results on potato images.

2. MODELING

A sphere of centerO and radius1 represents the potato. A cone of heighth and slopeα, with its axis orthog- onal to the sphere surface, is a germ. All light sources are assumed punctual. The observer or camera is rep- resented by its optical center and perspective projection is assumed (Fig. 1).

Proc. SPIE Conf. onOptics, Illumination, and Image Sensing for Machine Vision VI, Boston, USA, Nov. 10-15, 1991.

Fig. 1: Model used for the potato, the germ, the observer and the punctual light source.

An Euler-like coordinate system is used, with its origin as the centerO of the sphere and the axis passing throughO and the observerV being the reference. The distance fromO toV isd_o (Fig. 1). Two light sources s₁ ands₂ are defined by their respective distanced_s1 andd_s2 from the origin, their nutation anglesν₁ and ν₂, and their relative precession angleΠ12 (Fig. 2). The point on the surface with its normal toward the ob- server is called the poleP_o, and the points with their normal toward each source are the polesP_s1 andP_s2.

Fig. 2: Two sources case. The position of the sources is defined by their distanced_si, their nuta- tion angle νi, as well as their relative precession angle Π between them. The polesP_o,P_s1and P_s2 are shown on the sphere.

1 O

α h

punctual light source

observer d_o

d_s

ν₁

ν₂

Π12

P_s1 P_o

P_s2 observerV

lights₁

lights₂

3. SHADOW CHARACTERISTICS FOR ONE LIGHT SOURCE 3.1. Classification of illuminated areas

Before extending the problem to multiple sources, the single source case is considered. In particular, the geometrical classification of the shadows is studied here. In view of the cone model used for the protuber- ance, it is of particular concern to determine whether or not shadows are produced on the potato surface, according to the light position.

As shown in Fig. 3.a, an incoming light ray with incident angleβ does not generates any shadow if either or , since in the first case the cone is completely illuminated and in the second the cone is in the shaded area of the sphere. We add the constraint that “good” shadows are those lying completely on the sphere surface, that is for which with . This not only ensures that the whole shadow is projected on the surface, but also eliminates low contrasted shadows since irradiance would approach0 whenβ approachesπ / 2. A perspective view of the role ofβ is depicted in Fig. 3.b; the surface of the sphere is divided into four areas: the shaded area with no light reaching it, the shadowless area where cones are totally illuminated and do not generate any shadow, the imprecise area on which shadows partially exist and the remaining significant area. The surface of this significant area, for which , increases asα decreases orh decreases, therefore favoring cones with small values forα and h.

Fig. 3: Single source; a) two dimensional slice with the three incident anglesα,βmax andπ / 2 and five possible cones; b) perspective view of the subdivision due to the three incident anglesα, βmax andπ / 2.

As light rays falling with the same incident angle define a virtual cone (Fig. 3.a), it is possible to visualize the areas as being defined by three virtual cones, all having the punctual source as apex and intersecting the sphere with anglesα,βmax andβ respectively.

Another observation is that a single light source is inadequate for reliable shadow-based detection since only a fraction of the whole illuminated surface contains pertinent shadows, independently of the observer’s po- β α< β π> ⁄2

β β< _max β_max = asin(1⁄ (1+h))

α β β< < _max

β_max α π/2

O

shaded area

imprecise area

shadowless area

significant area

a) b)

P_o

h

sition. Worse than that, the significant area as seen by the observer decreases due to perspective projection asν tends toπ/2. Two or three sources at least are consequently needed to provide adequate detection over the entire visible surface.

3.2. Partially self-hidden shadow area

It is useful to determine if a cone possibly hides its own shadow to the observer, and if so, in which propor- tion. More generally, it is necessary to quantify the angle after perspective projection between the cone and its shadow. Such information can be useful for predicting the appearance of the pattern cone-shadow during the detection process. Denoting byε the above mentioned angle, it appears that if the shadow is entirely visible, and if the cone progressively hides its shadow until (Fig. 4).

Fig. 4: Self-hidden shadows;ε is the angle between the cone axis and the corresponding axis in its shadow, as seen in projection by the observer.

Fig. 5 represents the values of on the visible surface of the sphere. A black circle can be seen passing through the pointsP_o andP_s; it corresponds to the locations where a shadow is seen perpendicular to its cone. The inner part of the circle corresponds to an angleε greater thanπ / 2, and the outer part toε smaller thanπ / 2. As the circle necessarily passes through the two poles, the bigger the angleν the bigger the pro- portion of area for which a cone can not hide its own shadow.

Fig. 5: Cosine of the angleε for and (0…1

➝

Consequently, simultaneously maximizing the significant area (Section 3.1.) and minimizing the self-hid- den shadow area leads to conflicting requirements. This confirms that using a single source leads to an un- satisfactory solution in terms of detection quality. The answer is therefore to consider several independent sources, that help balance these contradictory requirements.

ε π– <π⁄2

ε π– >π⁄2 ε π– = π

π 3.π/2

π/2

0 2π

ε

corresponding angleε cone and shadow

ε ( ) cos

ε < π/2 ε = π/2 ε > π/2

shadowless area limit

ν = 45° d_s = ∞

4. MULTISOURCE ARRANGEMENT 4.1. General considerations

It is understood by multisource arrangement the fact that the detection algorithm uses simultaneously more than one view of the scene, each view differing only by its lighting and the set of shadows generated. Each view is independent from each other, and the acquisition mechanism for these views must treat them as in- dependent channels. One way to do so is by using lights with different colors, for example selected by nar- row band filters, and capturing images with the corresponding filters. A simple set up would be composed of three lights of respective colors red, green and blue, and a color video camera that would digitizes simul- taneously the three planes. Another possibility would be to move a single light, without changing its color, and to perform successive acquisitions without modifying the position of the sphere.

The purpose of this section is therefore to specify the constraints between those several sources and then to deduce the minimal number of sources as well as their respective positions.

First, all light sources are considered to be symmetrically arranged around the axisOV. Thus, the sources are assumed to be at same distanced_s from the origin. Two arrangements are studied, circular symmetry and triangular symmetry respectively.

Second, the multisource arrangement can be thought of as a tessellation of the sphere by virtual cones (Sec- tion 3.1.). Each source provides the two virtual cones that delimit its significant area, so only relations be- tween virtual cones of neighboring sources have to be defined.

4.2. Conditions pertaining to two neighboring sources

Any two neighboring sources are constrained as follows, independently of the observer (Fig. 6):

- the shadowless area of the first source has to be in the significant area of the second source;

- the two shadowless areas must not intersect;

- the larger angle between the sources that allows these two requirements to be verified is taken.

Fig. 6: Perspective view of the constraints between two neighboring sources, showing how sig- nificant areas intersect.

number of significant area(s) 2

1 0

shadowless area 1

significant area 1 light 1 light 2

shadowless area 2 significant area 2

significant areas 1+2

Formally, these constraints are shown in Fig. 7.a together with the definition of anglesγ,δ andσ. Fig. 7.b defining the angleφ that delimits the visible area of the sphere.

Fig. 7: Two sources; a) various angles defined by the sources if they satisfy to the neighboring conditions; b) the distance to the observer delimits the visible part of the surface.

Simple geometrical manipulations provide the following formulas:

(1) (2) (3) (4) (5)

The neighboring condition can therefore be expressed as , which guarantees the non-intersection of the two shadowless areas.

4.3. Circular positioning of light sources

In this configuration, all the sources have the same angleν and are separated with the same angleΠ, so their poles are distributed along an imaginary circle centered on theOV line. Fig. 8 shows the corresponding poles for two to five sources.

There are three constraints required in order to insure the consistency of this setting. First, any pair of sources must satisfy the neighboring conditions, thus preserving the consistency along the circle. The circle consistency constraint limits the maximum angleν by an angleνmax,circle. Second, both the border and the center of the visible area are to be reached by all significant areas, the whole visible surface being thus op- timally covered. This yields a minimal and a maximal angle forν denoted byνmin,border andνmax,center re-

α

β β_max

γ σ δ

π/2 φ

a) b)

d_o d_s

β_max = asin(1⁄ (1+h))

β = α–asin((sinα) ⁄ (1+d_s)) γ = β_max– asin((sinβ_max) ⁄ (1+d_s)) δ = γ β– σ = δ⁄2

φ = acos(1⁄ (1+d_o))

σ β≥

spectively.

Fig. 8: Circular positioning for two to five lights, with the poleP_o in black and polesP_si in white.

After algebraic manipulations, and denotingn the number of sources:

if ,

otherwise (6)

(7) (8) The conditions for the circular positioning are therefore:

if thenν has to verify

These constraints are depicted Fig. 9, the inner part of the meshes representing all possible simultaneous values forα,h,d andn.

Fig. 9: Constraints for the circular positioning, withα in degree,h in percentage of sphere radius, d_o=d_s=d, the numbern of sources varying from 2 to 10; a)d=1, b)d=5, c)d=∞.

2 3 4 5

ν Π

ν_{max circle,} = π⁄2 σ π≥ ⁄n

ν_{max circle,} = asin((sinσ) ⁄ (sin(π⁄n))) ν_{min border,} = φ γ–

ν_{max center,} = γ

σ β≥

ν_{min border,} ≤ ≤ν min(ν_{max circle,} ,ν_{max center,} )

50

25

0 22.5 45 0 22.5 45 22.5 45

n

0 50

25

50

25 h [%]

α ^[˚]

a) b) c)

41

89 67 45 23

10

35

h [%]

α ^[˚]

h [%]

α ^[˚]

The following observations regarding sources positioning can be made from Fig. 9:

- to insure existence of shadows, the slope of the cones cannot exceed approximately 35˚;

- the smallerh, the bigger the range of possibleα;

- the greaterd, the greater the possible values ofh, independently of the numbern of sources;

- the greaterd, the smaller the maximumα, independently ofn; - asn increases, the total ranges ofα andh increase too;

- up to 6 sources, the maximum height increases asn increases, but past 6 there is a saturation: 41%

of sphere radius is the maximum forh;

- increasing the number of sources from 6 to 9 yields a gain in α; past 10 sources no profit can be gained, either inh or inα;

Generally speaking, the ranges ofα andh are somewhat small, due to the very restrictive definition of the neighboring condition. Although sufficient in many cases, those ranges can be increased by lowering this condition, for example by allowing the imprecise area to be included into the significant area.

4.4. Triangular positioning of light sources

For the sake of comparison, another configuration is presented. The triangular positioning is obtained by repetition over the visible surface of the same spherical triangle with source poles as vertices. Fig. 10 shows the poles corresponding to 3, 6 and 12 sources; the triangulation maximizes the surface under the constraints expressed below.

Fig. 10: Poles of the sources in the triangular disposition, for 3, 6 and 12 lights respectively.

There are only two constraints in this case, since the center of the visible surface is always illuminated by the three closest sources. The first constraint is that each of the lights belonging to the same triangle satisfy the neighboring condition with the other two; this guarantees consistency of the triangulation and yields a maximum for the angleν of the outer sources, denotedνmax,n, wheren is the number of sources. The second constraint expresses that the border of the visible area has to be reached by every significant area of the outer sources, the whole visible surface being thus optimally covered.

After a few spherical triangles geometry, the resulting formulas are (with three different equations for νmax,nforn=3, 6 and 12 sources):

(9)

if ,

otherwise (10)

3 6 12

ν_{min border,} = φ γ–

ν_max,3 = π⁄2 σ π≥ ⁄3

ν_max,3 = asin((sinσ) ⁄ (sin(π⁄3)))

(11) (12) (13)

For the triangular positioning, the conditions are, forn=3, 6 or 12 sources:

if thenν has to verify

In a manner similar to Fig. 9, Fig. 11 depicts the different ranges of parametersα,h,d andn that are simul- taneously allowed.

Fig. 11: Constraints for the triangular positioning, withα in degree,h in percentage of sphere radius,d_o=d_s=d, the numbern of sources taking the values 3, 6 and 12; a)d=1, b)d=5, c)d=∞. The main remarks that can be made by comparing Fig. 9 and Fig. 11 are:

- the maximum heighth also increases asd increases, but does not saturate at 41%;

- forn= 6 andn= 12, the triangular setting is better as it provides a much larger range forh, and a slightly larger range forα;

- more than 12 sources (18, 21, 27…) can be positioned with the triangular setting, leading to more increases in the ranges ofh andα;

- for 12 sources andd =∞,h can be up to 90% of the radius of the sphere ifα is less than 5˚.

The triangular setting can overcome some of the limitations of the circular one, but with an increasing com- plexity in terms of positioning. A good choice can be to opt for the circular positioning for 2 to 5 sources and for the triangular positioning past 6, the ranges of possible parameters of the triangular-6 exceeding those of the circular-10.

To summarize, this section has answered the question of the minimal number of sources required for a given ν_max,6 = ν_max,3+2×acos((cosν_max,3) ⁄ (cosσ))

ζ = 3 2⁄ ×acos(cosδ–cos²δ) ⁄sin²σ

ν_max,12 = acos(cosδ×cosν_max,6+ sinδ×sinν_max,6× cosζ)

σ β≥

ν_{min border,} ≤ ≤ν ν_{max n,}

0 0 22.5 45

n

0 50

25 h [%]

α ^[˚]

a) b) c)

612 3 93

75

22.5 45 22.5 45

50

25 75

50

25 75

h [%]

α ^[˚]

h [%]

α ^[˚]

cone configuration and distance to the sphere. It has also proposed two different configurations for the po- sitions of the sources. Since dealing with 12 sources can be somewhat awkward, a compromise can be made if only a region of the sphere is known to be relevant, by using only those lights with their significant areas covering this region. However, in case the cone heights exceed 90% of sphere radius, the shadow detection approach may not be well suited. It must be noted that the question of the optimal number of sources is still open. It might be answered by the choice of the detection algorithm.

5. PROCESSINGS AND RESULTS 5.1. Type of processing

Givenn images corresponding ton sources positioned in accordance with the above conditions (Section 4.), the aim of the processing is to produce a kind of “conspicuity map” indicating possible locations for germs together with a confidence value. The general scheme that has been implemented consists in defining a se- quence of processing for a single image, then in specifying along this sequence when to merge then images and the manner to do it (Fig. 12).

Fig. 12: Typical processing scheme as experimented, shown here for 3 sources. After separate processings corresponding to the 3 sources, a rendezvous is inserted for merging the 3 channels.

The result is a map with a larger spot where larger germs are expected.

Requirements for the algorithm are its ability to detect the highly variable patterns generated by a germ and its shadow without confusion with a scar, a hole or the border of the potato. The difficulty resides in the appearance of these patterns, the germ and its shadow, having many combinations of lengths and orienta- tions over the visible surface (Fig. 13).

Fig. 13: Example of the types of patterns to be detected over the significant area.

3 2

1 Merging Final

processings Separate

processings Image

Map

The solution that was found the most satisfactory⁵ consisted first of using a Roberts’like edge detector to extract the contrasted boundary between the cone and its shadow, then of eliminating the long contours gen- erated by the border of the potato. The next step is a grouping of similar junctions around the location of a germ. A simple procedure for achieving this grouping is to keep only junctions appearing in images from at least two neighboring sources; this is achieved by computing the logical sum between each pair of neigh- boring sources planes. After a final morphological closing operation, the resulting binary image contains regions whose size are proportional to the confidence of having a germ.

5.2. Results

Fig. 14 presents the results obtained using three sources configuration for four representative potatoes, by using the algorithm described in Section 5.1.

Fig. 14: The upper part shows one from the three views of four representative potatoes, the lower part the corresponding results (average image size: 400x600). Potatoes a) and d) are typical spec- imens while c) has only small germs in hollows; b) is a pathological case with highlights, scars, hollows and a slanting germ.

Despite an imperfect matching of the planes due to the acquisition procedure, the results appeared to be ro- bust even with such a simple sequence of operations. It is interesting to notice that with appropriate hardware

a) b) c) d)

(SIMD or MIMD machines) the processing can be done simultaneously on every plane. The speed of oper- ations could also be increased by using red, green and blue lights in conjunction with a video color camera which would directly provide three image planes.

6. CONCLUSION

From the specific problem of the Potato operation, a robust detection method has been derived that applies to small and poorly contrasted protuberances on a spherical shape. The contrasted patterns of shadows pro- duced by simultaneous but independent light sources are explicitly used. A geometrical modeling followed by a detailed study of constraints between several sources has permitted the specification of the minimal number of source and their relative positioning. Real case images have finally demonstrated the feasibility of the approach with a three-light configuration.

From there, without changing the basic ideas, the approach could easily be extended to plane or undulated surfaces and to different models for the protuberances.

7. ACKNOWLEDGEMENTS

This work is supported in part by grants from the Swiss National Fund for Scientific Research (FNRS 20- 26475.89) and from the Swiss National Research Program 23 “AI and Robotics” (PNR23 4023-27036).

We thank our preferred greengrocer, M^r Durand, for his biologically cultivated potatoes. We also thank M.

Lefebvre, S. Gil, L. Chachere and the Office 313 gang for helpful comments and support.

8. REFERENCES

1. T. Pun, ‘‘Projet P.d.t. (Opération Patate)’’, AI and Vision Group Technical Memo, Sept. 1989.

2. M. Lefebvre, D. Brunet, J.-D. Dessimoz, P. Gugerli, R. Strasser, T. Pun, “The Potato Operation: gen- eral overview”, AI and Vision Group Report 91.01, Computing Science Center, University of Geneva, January 1991.

3. T. Pun, M Lefebvre, S. Gil, D. Brunet, J.-D. Dessimoz, P. Gugerli, “The Potato Operation: Computer vision for agricultural robotics”, SPIE Conference on Advances in Intelligent Robotic Systems, High- speed architectures and systems in machine vision, Boston, USA, Nov. 10-15, 1991.

4. M. Lefebvre, D. Brunet, T. Pun, “The Potato Operation: germs detection using contour based mea- sures”, AI and Vision Group Report 91.02, Computing Science Center, University of Geneva, January 1991.

5. D. Brunet, T. Pun, “The Potato operation: germ detection by shadows analysis and controlled light- ing”, AI and Vision Group Report 91.03, Computing Science Center, University of Geneva, April 1991.

6. M.-A. Glassey, ‘‘Projet Patate’’, Semester and Diploma Work, Institute of Microtechnology, Swiss Federal Institute of Technology (EPFL), Lausanne, Switzerland, 1991.