Automatic nanopositioning in SEM
5.4 Nanopositioning using intensity-based visual servoing
In this section, a control law to perform automatic nanopositioning in SEM using the pixel intensity values is explained. First the derivation of control law is explained fol-lowed by the experimental validations.
5.4 Nanopositioning using intensity-based visual servoing 103 5.4.1 Intensity-based visual servoing
This method is based on considering the intensity values of all the pixels present in an image as visual features i.e. s =I. By considering all the pixels in an imagef(x, y) of sizeM×N, the visual feature is given by
s=I= I(1,1), I(1,2)...I(M,N)⊤
(5.13) where, I(u,v) is the intensity of a pixel at location (u, v) andIis a column vector of size M×N. In this case, the error is
e=I−I∗ (5.14)
If we consider the problem of error minimization as an optimization problem, the primary goal will be to minimize the cost C given by equation (5.15).
C=e⊤e= (I−I∗)⊤(I−I∗) (5.15) The goal Image I∗ is defined by teaching approach where the platform is moved to a desired position and the image is acquired. Once, the target is defined, the main goal of the controller is to regulate the cost function from an unknown position. When C is minimum, the current position corresponds to the desired position. To visually reflect this cost function, a set of images are acquired by moving the platform around the target position. The cost is computed using these images offline and is shown in the figure5.9. From the figure, the cost becomes minimum at the desired position; however, the convergence is not smooth. The main reason is that the intensity variations in SEM imaging are not constant.
Figure 5.9: Visual representation of the cost function for intensity-based visual servoing.
Now to control the motion of the positioning stage, an estimation of the interaction matrix is required to link the temporal variation of pixel intensities with the camera instantaneous velocities. As shown by Marchand [Mar07], it can be derived by consider-ing theoptical flow constraint equation (OFCE) provided by Horn and Schunck [HS81].
According to OFCE, the intensity of a pixel I(x, y, t) in an image is same before and after a small displacement (dx, dy) for a time interval dt. We then have
I(x, y, t) =I(x+dx, y+dy, t+dt) (5.16) The first order Taylor expansion of (5.16) gives (5.17).
∂I
∂xdx+ ∂I
∂ydy+∂I
∂tdt= 0 (5.17)
On dividing (5.17) by dt, we get:
∇Ixx˙+∇Iyy˙+ ˙I = 0 (5.18) where,∇Ix = ∂I∂x and ∇Iy = ∂I∂y.
Now, if we consider the interaction matrix related to a 2D point m(x, y) (projection of 3D pointM(X, Y, Z)) in the image which is given by equation (5.19).
L(x,y) =
−Z1 0 Zx xy −(1 +x2) y 0 −Z1 Zy (1 +y2) xy −x
(5.19) UsingL(x,y), equation (5.2) can be decomposed into (5.20).
˙
x=Lxvand ˙y=Lyv (5.20)
By substituting (5.20) in (5.18), we get
˙I=−[∇IxLx+∇IyLy]v (5.21) From equation (5.21), the interaction matrix LI(x,y) is
LI(x,y)=−[∇IxLx+∇IyLy] (5.22) Here,LI(x,y) is the interaction matrix of size 1×6 for one pixel. For entire image, it can be rewritten as
. UsingLI, the time variation of pixel intensities is given by
˙I=LIv (5.24)
The deduced interaction matrix allows us to control up to 6 degrees of freedom. However, the used positioning platform can be controlled only 2 degrees of freedom (x and y).
Now, using equations (5.22) and (5.14), the control law given by equation (5.5) can be used. However, it has been shown that, a more feasible solution can be by using a control
5.4 Nanopositioning using intensity-based visual servoing 105 law derived of the form Levenberg-Maquardt optimization technique [Mal04,CMC08].
The control law derived of this form can be highly non-linear and improve the robustness [CM11]. The final control law is then given by equation (5.25).
v=−λcVp(H+µdiag(H))−1LI⊤e (5.25) where, λ and µ are positive gains, H = LI⊤LI is the Hessian matrix and cVp is the transformation matrix from camera frameRc to platform frameRpand can be obtained from sensor calibration.
As mentioned earlier, the platform can be controlled only with the voltages and a voltage-displacement model has been derived. So the velocities computed from control law are now converted to displacementsd(x,y) of the platform using equation (5.26).
d(x,y)=vavgt (5.26)
where, vavg = v0+v2cur is the average velocity, v0 is initial velocity, vcur is the current velocity andt is the sampling time taken.
For each iteration, the displacement vector is updated as given by equation (5.27) and the corresponding voltages computed from equations (5.9), (5.10), (5.11) and (5.12) are used to control the platform.
dnew=dcur+dprev (5.27)
where,dnew,dcuranddprev are the updated, current and previous displacements respec-tively. The block diagram depicting the overall control is shown in figure5.10.
Control law
Figure 5.10: Block diagram depicting intensity-based visual servoing.
5.4.2 Nanopositioning at optimal scan speed
The initial experiments are performed to position the silicon microparts with a normal scan speed of 720 nanoseconds per pixel providing a frame rate of 2.2 frames per second.
The magnification is fixed to 300 ×. For this test, the user selected voltages are 50V for x channel and 60V for channel. Figures 5.11(a) and 5.11(b) show respectively the images acquired at desired and initial positions. Figures 5.11(c) to 5.11(f) show error (I −I∗) at different locations. The final error at the end of the positioning task is
Figure 5.11: Series of images depicting the intensity-based visual servoing at normal scan speed. (a) Represent user selected desired position. (b) Initial image in the process. (c) - (f) Errors at different positions. (g) Final error.
shown in figure 5.11(g). Fig. 5.12(a) and Fig. 5.12 show the displacement and voltage variations with each iteration during the positioning process. Figure 5.12(c) shows the cost variation with each iteration.
5.4.3 Nanopositioning at high scan speed
Second tests are performed to position the microparts using the images acquired with an increased raster scanning speed. Normally in SEM imaging, high scanning rates during image acquisition leads to the increased noise levels in images. More details about noise in SEM imaging noise are explained in chapter 3. This test has been performed to check the methods efficiency in reaching the desired position at noisy conditions. For this test a scan speed of 360 nanoseconds (maximum allowed) per pixel that provides a frame rate of 3.1 frames per second has been used. Initially selected voltages for desired location are 50 V and 60 V forx and y channels respectively. Figures 5.13(a) and 5.13(b) show images at desired position and initial location respectively. Figures 5.13(c) and 5.13(d) show the error at first and final iterations during the positioning.
Figures5.14(a), 5.14(b) and 5.14(c) show the variation of displacements, voltages and cost during the process.
5.4.4 Nanopositioning at increased magnification
This experiments are performed to position the micropats at high magnification. This task has been performed at a magnification of 800×. Simultaneously, the method is also validated with increased scan speed at the selected magnification. The selected scan time is 360 nanoseconds per pixel. The voltages selected for desired position are 30V
5.4 Nanopositioning using intensity-based visual servoing 107
Figure 5.12: (a) Displacement (b) voltage (c) cost variations in x and y axes of the positioning stage with each iteration using optimal scan speed.
Figure 5.13: (a) Image acquired at desired location. (b) initial image (c) error at initial position (d) error at final position using high scan speed.
and 60 V respectively for x and y axes. Figures 5.15(a) and 5.15(b) show the images at desired position and initial position respectively. Figure 5.15(c) shows the error at initial position and figure5.15(d) shows the error at final position during the positioning
0 20 40 60 80 100
Figure 5.14: (a) Displacement (b) Voltage (c) cost variations during the positioning task using intensity-based visual servoing at high scan speed.
Figure 5.15: Images acquired at (a) desired location (b) initial position. Error images at (c) initial position (d) final position during the Fourier-based visual servoing process at high magnification (800×).
5.4 Nanopositioning using intensity-based visual servoing 109 task. The displacement and voltage variation plots are shown in Figures 5.16(a) and 5.16(b) respectively. Figure 5.16(c) shows the cost variation with each iteration at the selected magnification.
Figure 5.16: (a) Displacement (b) Velocity variations during the positioning task using intensity-based visual servoing at high magnification (800×).
5.4.5 Nanopositioning at unstable conditions
The final experiments are conducted to perform the positioning task at unstable con-ditions i.e. at varying brightness and contrast and to check the method’s efficiency.
For this test, once the desired position is selected and the positioning has started, the contrast and brightness are varied manually. The selected magnification is 300× and scan speed is 720 nanoseconds per pixel. Eventually, the nanopositioning task failed during this test since it requires constant or less variations in the intensity. Figures 5.17(a) and 5.17(b) show the reference and initial images. Figures 5.17(c) and 5.17(d) show the images where the contrast and brightness values are changed during this test.
The displacement, voltage and error variations are shown in figures5.18(a),5.18(b) and 5.18(c) respectively. The zero voltage during the test indicates that the voltage is out of the range.
Figure 5.17: Series of images acquired during nanopositioning at unstable conditions using intensity-based method. (a) Desired image. (b) - (d) Images acquired during the process.
Figure 5.18: (a) Displacement (b) Velocity variations during the positioning task using intensity-based visual servoing at unstable conditions.