Space non-invariant point-spread function and its estimation in fluorescence microscopy

HAL Id: inria-00438719

https://hal.inria.fr/inria-00438719v2

Submitted on 18 Dec 2009

HAL is a multi-disciplinary open access

archive for the deposit and dissemination of

sci-entific research documents, whether they are

pub-L’archive ouverte pluridisciplinaire HAL, est

destinée au dépôt et à la diffusion de documents

scientifiques de niveau recherche, publiés ou non,

estimation in fluorescence microscopy

Praveen Pankajakshan, Laure Blanc-Féraud, Zvi Kam, Josiane Zerubia

To cite this version:

Praveen Pankajakshan, Laure Blanc-Féraud, Zvi Kam, Josiane Zerubia. Space non-invariant

point-spread function and its estimation in fluorescence microscopy. [Research Report] RR-7157, INRIA.

2009, pp.54. �inria-00438719v2�

a p p o r t

d e r e c h e r c h e

9

-6

3

9

9

IS

R

N

IN

R

IA

/R

R

--7

1

5

7

--F

R

+

E

N

G

Thème COG

Space non-invariant point-spread function and its

estimation in fluorescence microscopy.

Praveen PANKAJAKSHAN, Laure BLANC-FÉRAUD, Zvi KAM and Josiane ZERUBIA

N° 7157

Praveen PANKAJAKSHAN

∗

,Laure BLANC-FÉRAUD

∗

, Zvi KAM

†

and Josiane ZERUBIA

∗

ThèmeCOGSystèmes ognitifs

ProjetARIANA

Rapportdere her he n°7157De ember 200956pages

Abstra t: Inthis resear h report, we re all briey how the dira tion-limited nature of

an opti al mi ros ope's obje tive, and the intrinsi noise an ae t the observed images'

resolution. A blind de onvolution algorithm an restore the lost frequen ies beyond the

dira tionlimit. However,underotherimaging onditions,theapproximationof

aberration-freeimaging, isnot appli able,and thephase aberrations of theemerging wavefront from

a spe imen immersion medium annot be ignored any more. We show that an obje t's

lo ationand itsoriginal intensitydistribution an bere overed by retrievingtherefra ted

wavefront's phase from the observed intensity images. We demonstratethis by retrieving

thepoint-spread fun tionfromanimaged mi rosphere. Thenoise andtheinuen e ofthe

mi rosphere size an be mitigated and sometimes ompletely removed from the observed

imagesby usinga maximuma posteriori estimate. However,due tothe in oherent nature

ofthea quisitionsystem,phaseretrievalfromtheobservedintensitieswillbepossibleonly

if the phase is onstrained. We have used geometri al opti s to model the phase of the

refra tedwavefront,andtestedthealgorithmonsomesimulatedimages.

Key-words: uores en emi ros opy,point-spreadfun tion,blindde onvolution,spheri al

aberration,maximuma posteriori,maximumlikelihood, expe tationmaximization

∗

ARIANAProje t-team,INRIA/I3S/UNS,2004Routedeslu ioles-BP93,06902Sophia-AntipolisCedex, Fran e

†

Résumé : Danslapremièrepartiede e rapportdere her he,nousrappelonsbrièvement

omment lanature limitée de dira tionde l'obje tif d'un mi ros opeoptique, et le bruit

intrinsèque peuvent ae terla résolution d'une image observée. Unalgorithme de

dé on-volution aveugle a été proposé en vue de restaurer les fréquen es manquants au delà de

lalimite de dira tion. Cependant, sous d'autres onditions, l'approximationdu systéme

imageur l'imageriesansaberrationn'est plusvalideet don lesaberrations delaphasedu

frontd'ondeémergeantd'unmédium nesontplusignorées. Dansladeuxièmepartie de e

rapportdere her he,nousmontronsqueladistributiond'intensitéoriginelleet la

lo alisa-tiond'unobjetpeuventêtreretrouvéesuniquementenobtenantdelaphasedufrontd'onde

réfra té,àpartir d'imagesd'intensitéobservées. Nousdémontrons elaparobtentiondela

fon tiondeoua partird'unemi rosphèreimagée. Lebruit etl'inuen edelatailledela

mi rosphèrepeuventêtrediminuésetparfois omplètementsupprimesdesimagesobservées

enutilisantunestimateur maximuma posteriori. Néanmoins,a ause del'in ohéren edu

systèmed'a quisition,uneré upérationdephaseapartird'intensitésobservéesn'est

possi-blequesilarestaurationdelaphaseest ontrainte. Nousavonsutilisél'optiquegéométrique

pourmodéliser laphase dufrontd'onde réfra té, et nous avons teste l'algorithmesur des

imagesimulées.

Mots- lés: mi ros opieàuores en e,fon tiondeou,dé onvolutionaveugle,aberration

A knowlegement

Thisresear hwas fundedbytheP2RFran o-IsraeliCollaborativeResear hProgram 1

. We

wouldliketothankINRIAforsupportingthePhDoftherstauthorthroughaCORDI

fel-lowship. Theauthorsgratefullya knowledgeDr. BoZhang(PhilipsMedi alresear h),Prof.

OlivierHaeberléandProf. AlainDieterlen(UniversitédeHaute-Alsa e,Mulhouse),andDr.

Jean-ChristopheOlivo-Marin (Pasteur Institute, Paris) for several interesting dis ussions.

Additionally, oursin eregratitudegoesto Dr. Gilbert Engler(INRASophia-Antipolis)for

theimagespresentedinFigs.20and21ofAppendix D.

1

Contents

1 Ba kground 1

1.1 ImagingModel . . . 3

1.1.1 Ba kgroundFluores en e . . . 3

1.1.2 Noise . . . 5

1.1.3 SimulatingBand-LimitedObje t . . . 7

1.2 ProblemStatement. . . 7

1.3 OrganizationofthisResear hReport. . . 9

2 Analyti al Point-Spread Fun tion Model 10 2.1 ReviewoftheS alarDira tionTheory . . . 10

2.1.1 MaxwellandHelmholtz'sEquations . . . 10

2.1.2 Fundamentals ofVe torCal ulus . . . 11

2.1.3 S alarDira tionTheory . . . 12

2.2 TheoryofAberrations . . . 18

2.2.1 Spheri alAberrations . . . 18

2.2.2 ThePhase Fa tor. . . 19

2.3 ApproximatingthePoint-SpreadFun tion . . . 20

2.3.1 NonlinearPhaseApproximation . . . 20

2.3.2 LinearPhase Approximation . . . 21

2.3.3 ApodizationFun tionApproximation . . . 21

2.4 Analyti alExpression . . . 21

3 Literature Review on BlindDe onvolution 26 3.1 MaximumLikelihoodApproa h . . . 26

3.2 Penalized MaximumLikelihoodApproa h . . . 27

3.3 OtherApproa hes . . . 30

4 Point-Spread Fun tion Estimation 32 4.1 MaximumLikelihoodApproa h . . . 32

4.2 MaximumAPosteriori Approa h . . . 33

4.2.1 Obje tandPoint-SpreadFun tionParametersEstimation . . . 34

4.2.2 UniformIntensityandApodizationEstimation . . . 36

5 Implementationand Analysis 38 5.1 InitializationoftheAlgorithm. . . 38

5.2 PreliminaryResults . . . 38

6 Con lusions and Future Work 42

A Appendix: Maximum LikelihoodExpe tation Maximization (MLEM) 43

C Appendix: Gradient Cal ulations 46

C.1 FortheObje tFun tion . . . 46

C.2 ForthePoint-SpreadFun tion . . . 46

List of Figures

1 S hemati ofaCLSM . . . 1

2 Anillustration ofdira tion . . . 2

3 Observedmi rosphereimage sli esandestimatedba kground . . . 4

4 Histogramofobservationandba kgroundsubtra tedobservation . . . 5

5 Simulationofobje tandobservation . . . 6

6 AxialintensityprolesofPSFandobservation . . . 8

7 Lightas anele tromagneti wave . . . 10

8 Depi tionforGreen'sidentityanddivergen etheorem . . . 12

9 Dira tionbya planars reen . . . 13

10 Amplitudepupilfun tionofaCLSM . . . 17

11 Fo usingoflightforrefra tiveindexmismat h . . . 18

12 Spheri alaberrationdueto refra tiveindexmismat h . . . 20

13 Numeri ally omputedCLSMPSF . . . 25

14 Blindde onvolutionresultsusingMLEM . . . 28

15 Comparisonoftrueobje tandPSFwithMLEMestimates . . . 29

16 Segmentationalongradialandaxialplane . . . 39

17 Algorithmprogression . . . 40

18 Axialintensityproles omparingobje t,observationandrestoration. . . 41

19 S hemati oftheexperimental pro edure. . . 49

20 Observedimagesofmi rospheresshowingradial invarian e . . . 50

Abbreviations

3

D Threedimension

2

D Twodimension

AFP A tualfo alposition

AU Airyunits

CCD Charge- oupleddevi e

CG Conjugategradient

CLSM Confo allasers anningmi ros ope

EM Expe tationmaximization

FWHM Full-widthat halfmaximum

GFP Greenuores entprotein

i ifandonlyif

LSI Linearspa einvariant

LSM Lasers anningmi ros ope

LSNI Linearspa enoninvariant

MAP Maximumaposteriori

ML Maximumlikelihood

MLE Maximumlikelihoodestimate

MSE Meansquarederror

NA Numeri alaperture

NFP Nominalfo alposition

OTF Opti altransferfun tion

pdf Probabilitydensityfun tion

PMT Photomultiplier tube

PSF Point-spreadfun tion

SNR Signaltonoiseratio

SA Spheri alaberration

WFM Wideeldmi ros ope

Notations

(

· ∗ ·)

Linearspa einvariant onvolution

(

·)

∗

Complex onjugationoperation

(b

·)

Estimate

α

Semi-apertureangleoftheobje tivelens

A(

_·)

Apodizationfun tion

b

Ba kgroundsignal

B

Fourier-BesseltransformorHankeltransformofzero-order

D

DiameteroftheCLSMpinhole

δ(

_·)

Dira -deltafun tion

∆

xy

Radialsampling size

∆

z

Axialsamplingsize

E(·, ·)

Time-varyingele tri eld

ǫ

Algorithm onvergen efa tor

ǫ

(·)

Permitivityofa medium

F(·)

Fourier transform

F

−1

₍

_·)

InverseFouriertransform

γ

Re ipro al ofthephoton onversionfa tor

h

Point-spreadfun tion(PSF)ofanimagingsystem

H(·, ·)

Time-varyingmagneti eld

i(

_·)

Observedimage

j

Imaginaryunitofa omplexnumber

J

0

Besselfun tionoftherstkindoforder zero

k

Ve torof oordinatesinthefrequen yspa e

λ

Averagephotonux

λ

em

Emissionwavelength

λ

ex

Ex itationwavelength

J (·)

Energyfun tion

(µ, σ

2

₎

Meanandvarian eofanormaldistribution

µ

(·)

Permeabilityofamedium

n

Indexofiterationfortheestimationalgorithm

∈ N

+

n

(med)

Refra tiveindexofa medium

n

Outwardnormaltoa surfa e

∇(·)

Gradientofave toreld

∇

2

₍

·)

Lapla ianofa s alareld

N

g

(

·)

AdditiveGaussiannoise

N

p

(

·)

Poissondistribution

(N

x

, N

y

)

Number ofpixelsintheradialplane

(N

z

)

Number ofaxialsli esorplanes

o(

_·)

Spe imenorobje timaged

ω

_(·)

Parameterve tor tobeestimated

Ω

s

Dis retespatialdomain

Ω

f

Dis retefrequen ydomain

p(

_·)

Probabilitydensityfun tion

P r(

_·|·)

Conditionalprobability

P

Pupilfun tion

(ρ, φ, z)

Cylindri al oordinates

(ρ, φ, θ)

Spheri al oordinates

Σ

Dira tingaperture

Θ

Parameterspa e(

∈ R

)

ϕ

Opti alphasedieren e

ϕ

d

Defo usphaseterm

ϕ

a

Aberrationphaseterm

x

Ve torof oordinatesin obje t/imagespa e(

2

Dor

3

D) (

∈ R

2

or

∈ R

3

)

1 Ba kground

Fluores entmi ros opes[1,2 ℄use a highlyfo usedlaser spot to s an biologi al spe imens

inthreedimension(

3

D)andtoobtainopti alimagese tionsofthevolumeofinterest. The spe imenistreatedwitha dyesu hasthegreenuores entprotein(GFP) andituores e

on ex itation by an in ident laser beam. By hanging the obje tive to fo us at dierent

depths inside the spe imen, and by olle ting the emitteduores en e at ea h plane, one

an visualize the ells, tissuesand embryos in

3

D.In Fig.1 , wesee a simple s hemati of a onfo al laser s anning mi ros ope (CLSM) where the emissioneld energy is olle ted

bypla ing aphotomultiplier tube(PMT) at theposition oftheemission beamfo us. The

dieren ebetweenthe lassi aluores entmi ros opessu haswideeldmi ros ope(WFM)

is that in the CLSM a pinhole is addedbefore the dete tion stage. This pinhole restri ts

the total amount of light olle ted to the plane that is in fo us (as shown by solid line

in thes hemati ;dotted line representsthe outof fo us planes). Themajor advantage of

Figure 1: S hemati ofa CLSM. A) Laser, B) ex itation lter, C)di hromati mirror, D)

obje tivelens, E) in-fo usplane ofthe spe imen,F) pinhole aperture, G)photomultiplier

usingCLSMforimagingbiologi alspe imensisthatitimagese ientlythosepartsofthe

spe imenthatlieinthefo alregionoftheex itationlight. Thelightfromthelayersoutside

this region isgreatly attenuated anddoesnot ontribute signi antly to the nal output.

However,even under ideal onditions, the resolution of theobserved obje tis ae ted by

thedira tion-limitednatureoftheopti alsystem. Thisisbe ausewhenlightfromapoint

sour epassesthroughasmall ir ularaperture,itdoesnotprodu eabrightdotasanimage,

butratheradiused ir ulardis knownasAirydis surroundedbymu hfainter on entri

ir ularrings(seeFig.2 ). Thisexampleofdira tionisofgreatimportan ebe ausemany

Figure 2: An illustration of the dira tion pattern for a ir ular aperture (©

Ariana-INRIA/I3S).

opti alinstruments(in ludingthehumaneye) have ir ular apertures. If thissmearing of

theimageofthepointsour eislargerthanthatprodu edbytheaberrationsofthesystem,

theimaging pro ess is saidto bedira tion-limited,and that is thebest resolutionwhi h

anbephysi allyobtainedfrom thatsizeofaperture.

In additionto blurring, theimage measurement is orrupted bybothintrinsi and

ex-trinsi noise sour es. In digital mi ros opy, the sour e of noise is either the signal itself

(so- alledphotonshotnoise),orthedigitalimagingsystem. Asimagingismadepossibleby

the onversionoflightenergytophoto-ele tronsatthedete torelementofthemi ros ope,

by tra king the a umulated photon to ele tron onversions (photo-ele trons) alone over

time,we an observethat itrevealsanunderlyingPoissondistribution ofevents[3 ℄. Thus,

photon emission has the fundamental property of being sto hasti with respe t to time.

Ontheotherhand,ifanimage istakenwith nolight providedto thedete tor, the

signal-independentele trons olle tand forma distribution hara terizedbya meanvalueand a

standarddeviation. Itisforthisreasonthatthesignal dependentnoiseis hara terizedby

1.1 Imaging Model

Let

O(Ω

s

) =

{o = (o

xyz

) : Ω

s

⊂ N

3

7→ R}

denote all possible observableobje tson the dis retespatialdomain

Ω

s

=

{(x, y, z) : 0 ≤ x ≤ N

x

−1, 0 ≤ y ≤ N

y

−1, 0 ≤ z ≤ N

z

−1}

and

h : Ω

s

7→ R

themi ros opepointspread fun tion(PSF). If

{i(x) : x ∈ Ω

s

}

(assumedto be boundedand positive) denotethe observedvolume,then theobservation anbeexpressed

as

i(x) =

N

p

([h

∗ o](x) + b(x)) + N

g

(x), x

∈ Ω

s

,

(1) where,

N

p

(

·)

denotes voxel-wise noise fun tionmodeled as a Poissonianpro ess,

N

g

isthe dete tor noise approximated by additive zero-mean Gaussian distribution

(0, σ

2

g

)

, and

b :

Ω

s

7→ R

denotes the low-frequen y ba kground signal aused by s attered photons and auto-uores en efromthesample. Sin ethePMTina onfo almi ros ope(likeZeissLSM

510usedinourexperiments)usuallyhaseithera

8

-bitor

12

-bitsampling,theintensitygray levelsareassumedtolieeitherbetween

[0, 255]

orbetween

[0, 4095]

.

1.1.1 Ba kground Fluores en e

InEq. (1) , we have assumedthat the imaging systemhasbeena priori alibrated so that

thereisnegligibleosetinthedete torandthattheilluminationisuniform;thatisno

mis-alignment in theillumination lamp. This assumption isjustied in our ase as elu idated

bythefollowingexample.

Fig. 3(a)and (b) show therst and the last sli e of a real observed volume for a

mi- rosphereimmersedin water. Fig.3 ( )and Fig.3(d) showthe estimatedba kground

ˆb(x)

obtainedbymorphologi allyopeningtherst(Fig.3 (a))andthelast(Fig.3(b))sli esusing

a ir ularstru turalelement[4℄. Therstandthelastsli eswere hosenbe ausetheywere

foundtobefreeoftheobje tuores en e. We an seefrom thehistogramsthatthereis a

dominan eofasingleintensityvalue. Hen e,theba kgroundisalmostuniformandforour

model we an assumeit to bea onstant. What we an also infer from thegures is that

there is a uniformity in the illumination and noalignment problems. The mean value of

theba kgroundsignal fromthisestimationpro edurewasfoundtobeabout

3.13

intensity levelunits (IU). Fig.4(a)shows thehistogram al ulatedforthe sli e in Fig.3(b). Inthe

histogram shown in Fig. 4(b) for the ba kground subtra ted sli e in Fig. 3(d), we noti e

now that thedominant intensities are essentiallyzero due to thesubtra tion. This shows

that this kind of pro edure works well in estimating the ba kground. Next, the overall

histogramoftheimagevolumewassmoothedandaPoissondistributionwasttothedata.

The parameters of the distribution were estimated by using a maximum likelihood (ML)

algorithm. Theempiri almeanofthedistributionwasestimatedtobebetween

3.9683

and

3.9702

IU with a

95%

onden e. Sin e the obje t uores en e was sparsely populated, wendthat thereis notmu h dieren ebetweentheoverall distribution meanestimation

and the individual ba kground estimated. This is valid in most of the images taken

us-ing a CLSM where the obje tuores en e is sparsely distributed throughout thevolume.

Formoredetailsonba kgroundestimationinuores en emi ros opy,theinterestedreader

(a) (b)

( ) (d)

Figure 3: (a) The rst and (b) the last sli es of an observed mi rosphere image volume

havingsparseobje tuores en e andthe orrespondingestimatedba kgrounduores en e

(a) (b)

Figure4: Histogramof(a)asli eintheoriginalvolume,and(b)theba kgroundsubtra ted

sli e(© Ariana-INRIA/I3S).

Forthesimulationsusedin therest ofthisresear hreport,a onstantba kground

uo-res en eofintensity

10

wasaddedtotheblurredobje tandthisisillustratedinFig.5(b). We an note the shift in axial position between the original obje t in Fig. 5(a) and its

blurred version in Fig. 5(b). This axial shiftwill befurther dis ussed in Subse tion2.2.1.

It is importantto noti e that theradial enter for theobje tand the observation remain

un hanged.

1.1.2 Noise

ForaPMToperatinginthephoton- ountingmode,we anassumethatthereisnoreadout

or quantum noise, and the total photonsdete ted due to obje t uores en e and various

othernoisesour esarePoissondistributed. Thus, thesimulateddataset

i

anbeobtained asa Poissonrandomvariablewithmean

([h

∗ o](x) + b(x))

andEq.(1) an berewrittenin termsoftheobservedphoton ountsas

where

γ

∈ R

+

isthe re ipro al ofthe photon onversion fa tor, and

γi(x)

isthe observed photon number at the dete tor. For uores en e mi ros opy,

γ

is proportional to several physi alparameters,su hastheintegrationtimeandthequantume ien yofthedete tor.

Itisimportanttonotethatthelinearshiftinvariant(LSI)approximationforthe onvolution

operation in themodel ofEq. (2 ) is not validwhen handling spa evarying PSFs. In this

ase,wewillhaveto providea newexpression thatisbasedona newlineardepth varying

(LDV)observationmodel.

Although the Poisson statisti s provide a realisti noise model, in many appli ations

additiveGaussiannoisemodelisassumedtosimplifythenumeri al omputation. However,

itisimportanttonotethatunderlowsignaltonoiseratio(SNR),theadditiveGaussiannoise

modelprovidesapoorerdes riptionoftheuores en emi ros opyimagingthanthePoisson

model. ThehighSNR ase anbeaddressedbyemployingthe entrallimittheorem(CLT)

forlargenumbersofmeasurementdata, wheretheadditiveGaussiannoisemodelperforms

satisfa torily. InFig.5( ),themi rosphereofFig.5(a)issimulatedasobservedbyaZeiss

LSM 510 mi ros ope with a 40X oil immersion obje tivelens of numeri alaperture (NA)

1.3.

(a) (b) ( )

Figure 5: Simulationofa (a) mi rosphere obje t, (b) theblurred observation showingthe

ba kgrounduores en e, ( )theblurred observationwiththeba kgrounduores en eand

1.1.3 Simulating Band-Limited Obje t

Sin ewefrequentlyusesimulateddatasetfortestingor omparingtheperforman eofour

algorithm, it isne essary to numeri allyapproximate a band-limitedobje tthat ould be

usedtorepresentarealobje tthatweareinterestedinimaging. Thegeometryoftheobje t

wewishtos anhere,namelythemi robead,isa sphere. Su hspheresservewellasa good

modelforuores en e- oatedbeads.

Thesespheres anbegeneratedfromtheiranalyti alexpressionsinthefrequen ydomain

as[6 ℄

O(k) =

sin(2πR

q

k

2

x

+ k

2

y

+ k

2

z

)

− (2πR

q

k

2

x

+ k

y

2

+ k

z

2

) cos(2πR

q

k

2

x

+ k

2

y

+ k

z

2

)

π

2

₍

q

_k

2

x

+ k

2

y

+ k

2

z

)

3

(3)

where

R

is the radiusof the desiredsphere, and the sampling in the frequen ydomain is arried out so that

∆

k

x

= 1/N

x

,

∆

k

y

= 1/N

y

, and

∆

k

z

= ∆

xy

/(∆

z

N

z

)

.

∆

xy

and

∆

z

are respe tively the radial and the axial sampling in the image spa e. To avoid Gibbs

phenomenon,thenumeri alapproximationof

O(k

x

, k

y

, k

z

)

was multipliedbya

ℓ

∞

normal-ized

3

DGaussianfun tionwithvarian es

(N

x

, N

y

, N

z

)/(2π)

. Fig.5 (a)showsthesimulated spherewithmanufa turespe iedradiusof

250

nm. Theobje tintensity isassumedto lie between

[0, 200]

.

Thesimulationisperformedas ifthemi rosphereimageswerea quirefora ZeissLSM

510mi ros opeequippedwithanArgonions anninglaserofwavelength

488

nm. The emis-sionBandPass(BP) ltersareBP

505

− 530

foraGreenFluores en eProtein(GFP)type staining.

1.2 Problem Statement

Theuores en edistributioninanobje t anberestoredbyde onvolutionwiththe

knowl-edgeofthePSFoftheimagingsystem. Ithasbeenwidelya eptedintheliteraturethatthe

de onvolutionperforman eoftheGaussian noisemodel isinferiortothephysi ally orre t

Poisson noise model. Wethus justify theuse of the model in Eq. (2 ). We observed that

theoreti ally al ulatedPSFsoftenla ktheexperimental ormi ros opespe i signatures,

while the empiri ally obtained images of mi rospheres are either over sized or (and) too

noisy. Fig.6 showstheaxialintensityprolesalonga radialplanefor anumeri ally

al u-lated PSF, a simulated obje t blurred with the PSF and a simulated observation. As we

dealwithlive ells,oftenonlyasingleobservationofthevolumeisavailableforrestoration.

Thisisbe ause,theveryplantandanimal ellsthatwewishtoobserve ouldbeirreversibly

damagedwhenexposedtohighenergyradiationforalongtime. Photoblea hingofthedye

is anotherreason foravoidinglong exposures. Thenon-repeatability ofthe whole pro ess

makes it di ultto restore the lost frequen iesbeyond the dira tion limitswithout any

onstraintsontheobje torthePSF.Intherstphaseofourwork[7 ,8,9 ,10 ℄,weproposed

a blindde onvolutionalgorithmthat estimated thedira tion-limited PSF(under

Figure6:Axialintensityprolesofa al ulatedPSF,ablurredmi rosphereandtheobserved

mi rosphere. Theintensities are s aled and the peaks mat hed for visual omparison (©

Ariana-INRIA/I3S).

minimizedanenergyfun tionalternatively,rstwithrespe tto theintensityofanimaged

obje tandthenwithrespe ttothePSFofthemi ros ope. Atea hiterationtheestimated

obje tintensitywasregularizedusingatotalvariationalpotentialfun tion,andthePSFwas

proje tedonto a parametri spa e. Weproje tedtheestimated PSFon to a

3

D Gaussian model [11 ℄, and restored thelost frequen iesbeyond the dira tion limits by regularizing

theobje tusingatotalvariational(TV)term.

In most ases the degradation is often inuen ed only by the dira tion ee t.

How-ever, if imaging into deeper se tions of the spe imen, spheri al aberrations (SA) annot

beignored. This isbe ause,the refra tive indexmismat h betweenthe spe imen andthe

immersion medium ofthe obje tivelensbe omessigni ant[12 ℄. Anadditionaldieren e

inthepathisintrodu edintheemergingwavefrontofthelightduetothisdieren einthe

index. Theaimofthisreportisto deriveamethodologyforretrievingthetruePSFofthe

imaging system from theobserved images of mi rospheres in the presen e of aberrations.

Hen eforth,weshalladdresstheimagesofmi rospheresasexperimentalPSF.Itistheusual

pro eduretousethisexperimentalPSFforde onvolvingaseparatesetofspe imen

observa-tiondata. In[13 ℄, itwas noti edthat aniterativenonlinearde onvolutionalgorithmwhen

applied isvery sensitiveto theamountof randomnessin the experimental PSF. Moreover

an estimation pro edure arried out on the observational data showed that the re overed

obje tsaresigni antlyimprovedbyusingasmoothedexperimentalPSFratherthanusing

theexperimental PSFdire tly. Similarly,foradeterministi algorithmtested onsimulated

obje ts, theroot meansquared error(RMSE)is signi antlysmallerforthesmoothed

ex-perimental PSF thanusing thenon-smoothedone. Anadditionalpoint that hasnotbeen

aretoo largetorepresentanidealpointsour e. In ontrast,ifthesize ofthebeadsaretoo

small,thentheobservationisplaguedwithnoiseproblemsasthebeadisweaklyuores ent.

Wesummarizethemotivationfortheresear h inthisreport asfollows

ˆ experimentally obtained PSFs by imaging uores ent beads are low in ontrast and

highlynoisy,

ˆ theyarelargein size omparedtothea tualPSF,

ˆ the PSFs so onstru ted annot bereused if theexperimental onditions arevaried,

and

ˆ knowledge ofthePSFof animagingsystemhelps inre overing theoriginalintensity

distribution ofanimaged obje tbyde onvolution.

1.3 Organization of this Resear h Report

Thisresear h report isorganized as follows. InSe tion 2 ,weretra ethes alar dira tion

theory for nally introdu ing the Stokseth's [14 ℄ approximation for the dira tion-limited

analyti al PSF expression. This model is extendedin Subse tion2.2 to alsoin lude

aber-rations. InSe tion3,weenlisttheexistingliteratureonthesubje tofblindde onvolution

andwesee why thesemethods annot beapplied for solvingthe above problem. Drawing

inspirationfromourearlierwork onthesubje tof blindde onvolution,weproposein

Se -tion4 anapproa h of estimatingthea tual PSF fromthe observationdata with Bayesian

framework. Thedieren ehereisthatwearemoreinterestedinestimatingPSFsthatvary

withthea quisition onditions;butwithsomeknowledgeoftheobje t. Totestour

hypoth-esis,in Se tion 5,wehaveshown some experimentsperformedonsimulated data. Finally

wedis ussthepossibilityofextendingthisworktoretrievethePSFdire tlyfrom observed

2 Analyti al Point-Spread Fun tion Model

In this se tion we review some of the basi s of the dira tion theory that will help us in

eventually deriving an analyti al expression for the PSF. Although ele tromagneti elds

are ve torial in nature, we have onsidered only the s alar properties of light here. Our

reasonsforignoringtheve torialnatureoflightwillbemade learinthedis ussionatthe

endofthisse tion.

2.1 Review of the S alar Dira tion Theory

2.1.1 Maxwell and Helmholtz's Equations

Intheabsen eoffree harge,Maxwell'sequationsare[15 ℄

∇ × E = −µ

m

∂

∂t

H,

(4)

∇ × H = ǫ

m

∂

∂t

E,

(5)

∇ · ǫ

m

E = 0,

(6)

∇ · µ

m

H = 0,

(7)

where

E(x, t)

and

H(x, t)

are the orthogonally time-varying ele tri and magneti elds respe tively(seeFig.7).

∇ × E

givesthe url

(E)

and

∇ · ǫ

m

E

givesthediv

(ǫ

m

E)

.

µ

m

isthe permeabilityand

ǫ

m

isthepermittivityofthemedium. Ifthemediumishomogeneous,

ǫ

is onstantthroughouttheregionofpropagation. Themediumissaidtobenon-dispersiveto

light if

ǫ

isindependentofthewavelength

λ

of thelightused. All mediaarenon-magneti hen e the permeability of the medium is the same as that in va uum. The ve tor wave

equationforanytime-varyingeld

V

_(t)

anthusbewritten as

∇

2

V

_(t)

₋

n

2

c

2

∂

2

∂t

2

V

(t) = 0,

(8) where,

∇

2

istheLapla ianoperator,

n

m

= (ǫ

m

/ǫ

0

)

1/2

istherefra tiveindexofthemedium,

and

c = 1/(µ

0

ǫ

0

)

1/2

thespeedoflightin va uum. However,fornonhomogeneous medium,

the oupling between the ele tri and magneti elds annot be reje ted and hen e the

aboveequation hastobemodiedto in lude alsothevariationin therefra tiveindex. At

theboundaries, ouplingisintrodu edbetweentheele tri andmagneti eldsandintheir

individual omponents as well. Theerror is small provided that theboundary onditions

haveee toveranareathatisasmallpartoftheareathroughwhi hawavemaybepassing.

Wedenea stri tlymono hromati time-harmoni s alareld by

U (x, t) = U

x

(x) exp(

−jωt).

(9)

Theaboves alareld alsosatisesEq.(8 ),andhen etheHelmholtzequation

(

_∇

2

+ k

2

)U = 0,

(10)

where

k = (2πnµ)/c = 2πn/λ

is the wave number, and

λ

is the wavelength of the light in freespa e (

λ = c/µ

). If thiseld hasnoevanes ent omponent, it an bewritten as a weightedsumofplanewavesoftheform

U (x) =

Z

k

A(k) exp(jk

· x)dΩ,

(11)

where

k

isa unitve torthatdes ribesthedire tionofpropagationoftheplanewaves.

2.1.2 Fundamentalsof Ve tor Cal ulus

Divergen e Theorem TheDivergen e theorem ortheGauss' theorem isthe higher

di-mensionalformofthefundamentaltheoremof al ulus.

Theorem 2.1. Let

V

beasimple solidregionand

S

the boundarysurfa eof

V

,givenwith thepositiveoutwardorientation

n

. Let

V

beave toreldwhose omponentfun tionshave ontinuouspartialderivatives onan open region that ontains

V

. Then

Z

V

(

∇ · V)dV =

I

S

V

_{· ndS.}

(12)

Green'sse ond identity Green'sidentitiesareasetofve torderivative/integral

identi-tiesthat an beusefulin derivingtheFresnel-Kir ho dira tionequationsandtheDebye

integral approximation. Sin e we areonly interestedin these ond identity, it isstated as

follows.

If

G

and

U

are ontinuouslydierentiables alarelds on

V

in

R

3

,then

Z

V

(G

∇

2

_U

− U∇

2

_{G)dV =}

I

S

(G

∂

∂n

U

− U

∂

∂n

G)

· dS.

(13)

Here,

G(P )

is a s alar eld asa fun tionof position

P

, and

∂/∂n

is thepartial derivative alongtheoutwardnormaldire tion(seeFig.8)in thesurfa eelement

dS

. Itis straightfor-wardtoshowthatforthes alarelds

G

and

U

satisfyingtheHelmholtzequationEq. (10 ) , thelefthandsideofGreen'sse ondidentityEq. (13 )iszero.

Figure8: Solidregiondepi tionforGreen'sidentityandthedivergen etheorem(©

Ariana-INRIA/I3S).

2.1.3 S alar Dira tionTheory

Weintrodu ebelowthedira tiontheoryforlightpropagationinahomogeneous medium.

Themostimportantapproximationhereis thetreatmentof light asa s alar phenomenon,

negle tingthefundamentallyve torialnatureoftheele tromagneti elds. Thes alartheory

yieldsa urateresultsiftwo onditions aremet:

1. thedira tionaperture

Σ

isverylargein omparisontothewavelengthoflight

λ

,and 2. thedira tingeldsmustnotbeobservedtoo loseto theaperture

r

01

≫ λ

.

Forallmi ros opesoperatinginthefar-eldregion,theaboveapproximationsarejustied.

InFig.9,if

P

0

isthepointofobservation, fordira tionbyanaperture

Σ

,theKir ho

G

Figure9: Dira tionbya planars reenilluminatedbya singlespheri al wave(©

Ariana-INRIA/I3S).

atanarbitrarypoint

P

1

thatisasolutiontoEq.(13 )is

G(P

1

) =

exp(jk

0

r

o1

)

r

01

.

(14)

Here,

r

01

isthedistan efromtheaperture

Σ

totheobservationpoint

P

0

,and

k

0

= 2π/λ

is thewavenumberin va uum.

∂

∂n

G(P

1

) = cos(n, r

01

)(jk

0

−

1

r

01

)

exp(jk

0

r

01

)

r

01

,

(15)

where,

n

istheoutwardnormalto

Σ

,and

cos(n, r

01

)

isthe osineoftheanglebetweenthe normal

n

and

r

01

.

Theorem 2.2. The integral theorem of HelmholtzEq. ( 10 )andthe Kir ho Eq. (14 )give

theeldatanypoint

P

0

expressedintermsoftheboundaryvaluesofthewaveonany losed surfa e

S

surroundingthat point. A ordingly

U (P

0

) =

1

4π

I

S

exp(jk

0

r

01

)

r

01

∂

∂n

U

− U

∂

∂n

exp(jk

0

r

01

)

r

01

d

S.

(16)

Thisrepresentsthebasi equationof s alardira tiontheory.

Fresnel-Kir ho Dira tion Formula

Theorem 2.3. If weassume that an aperture

Σ

is illuminated by asingle spheri al wave originating from

P

2

and that

r

01

≫ λ

,

r

21

≫ λ

the disturban e at

P

0

with Kir ho's boundary onditions is[16℄

U (P

0

) =

A

jλ

Z

Σ

exp(jk

0

(r

21

+ r

01

))

r

21

r

01

³ cos(n, r

01

)

− cos(n, r

21

)

2

´

d

_S.

(17)

Proof. Bytheorem2.2andapplyingtheKir ho'sboundary onditionsweget

U (P

0

) =

1

4π

Z

Σ

(G

∂

∂n

U

− U

∂

∂n

G)d

S.

(18)

Thefringing ee ts an benegle ted ifthedimension ofthe aperture is mu h larger than

thewavelength

λ

ofthelightused. If

k

0

≫ 1/r

01

,thenEq.(15 ) an bewritten as

∂

∂n

G(P

1

)

≈ jk

0

cos(n, r

01

)

exp(jk

0

r

01

)

r

01

.

(19)

SubstitutingEq. (19 )in Eq.(18 ) ,weget

U (P

0

) =

1

4π

Z

Σ

exp(jk

0

r

01

)

r

01

³ ∂

∂n

U

− jk

0

U cos(n, r

01

)

´

d

_S.

(20)

If we assume that the aperture is illuminated by a single spheri al wave arising from

P

2

, then

U (P

1

) = A

exp(jk

0

r

21

)

r

21

.

(21) Hen e

U (P

0

) =

A

jλ

Z

Σ

exp(jk

0

(r

21

+ r

01

))

r

21

r

01

³ cos(n, r

01

)

− cos(n, r

21

)

2

´

d

S.

(22)

Remarks

(a)Bythere ipro itytheoremofHelmholtz,theee tofpla ingthepointsour eat

P

2

and observingat

P

0

isequivalentto pla ingthepoint-sour eat

P

0

andobservingat

P

2

. (b)TheFresnel-Kir hodira tionformulaessentially onrmstheHuygensprin iple. The

eldat

P

0

arisesfromaninnitenumberof titiousse ondarypointsour eslo atedwithin the aperture itself. The se ondary sour es here ontain amplitudes and phases that are

relatedtotheilluminationwavefront,andtheanglesofilluminationandobservation.

( ) The Fresnel-Kir ho dira tion approximation is similar to the Rayleigh-Sommereld

theoryforsmalldira tionangles.

Debye approximation From the Kir ho-Fresnel formulation, the Debye integral

ap-proximationfora ir ularaperture an beobtainedas [15℄

U (P

0

) =

−j

λ

2π

Z

0

α

Z

0

A(θ

i

) exp(

−jk

i

ρ sin θ

i

cos(φ

− ψ)) exp(jk

i

z cos θ

i

) sin θ

i

dθ

i

dφ,

(23)

where

α

is the semi-aperture angle of the obje tive,

A

is the apodization fun tion,

k

i

=

2πn

i

/λ

, and

λ/n

i

isthe wavelength in the medium ofrefra tiveindex

n

i

. Fora lenswith a uniform aperture, the apodization fun tion is radially symmetri al with respe t to the

opti axis and an berepresentedby

A(θ

i

)

. Theintensityproje tedfrom an isotropi ally illuminatingpointsour esu hasaurophore,ona(at)pupilplaneisboundtobeenergy

onservation onstraint. Therefore the amplitude as a fun tion of

θ

i

in the pupil plane shouldvaryas

(cos θ

i

)

−1/2

andtheenergyas

(cos θ

i

)

−1

[6 ℄. Thus,

A(θ

i

)

forthedete tionis givenby

A(θ

i

) = (cos θ

i

)

−

1

2

,

(24)

andfortheilluminationas

A(θ

i

) = (cos θ

i

)

1

2

.

(25)

Eqs.(24 )and(25)areknownasthesine onditionandtheyguaranteethatasmallregion

ofthe obje t planein theneighborhood ofthe opti axis is imaged sharplyby a familyof

rayswhi h an haveanyangulardivergen e. This onstitutesanaplanati imaging system

anditexhibits

2

D transverseshiftinvarian e.

Remarks TheDebyeapproximationholdsgood onlyif

(a)

r

21

≫ a

, (

a

istheradiusoftheaperture)

(b)thespheri al waveletsfromtheaperture

Σ

areapproximatedbyplanewavelets, ( )

cos(n, r

01

)

≈ −1

,

Stokseth approximation Stokseth [14 ℄ also arrived at the above approximation, in

Eq. (23 ), by extending the work of Hopkins [17 ℄. Hopkins essentially worked out an

ap-proximation for small amounts of defo using. WhileStokseth's is a simple analyti al

ap-proximationforlargedefo usings. ThePSFisdenedhereastheirradian edistribution in

theimageplaneofapointsour eintheobje tplane. AstheOTFforin oherentillumination

andthePSFarerelatedbyFouriertransforms,wesay

h(x) =

F

−1

_{[OT F (k)].}

(26)

The omplexamplitudePSF anhen ebewrittenas

h

A

(x) =

Z

k

OT F

A

(k) exp(jk

· x)dk,

(27) where

x

and

k

are the

3

D oordinatesin theimage andtheFourier spa e. Bymakingthe axialFourierspa e o-ordinate

k

z

afun tionoflateral o-ordinates,

k

z

= (k

2

i

−(k

x

2

+ k

y

2

))

1/2

,

the

3

DFouriertransformisredu edto

h

A

(x, y, z) =

Z

k

x

Z

k

y

P (k

x

, k

y

, z) exp(j(k

x

x + k

y

y))dk

y

dk

x

,

(28)

where

P (

·, ·, ·)

des ribestheoverall omplexelddistributioninthepupilofanunaberrated obje tive lens. The pupil fun tion is a des ription of the magnitude and phase of the

wavefrontthat a point sour eprodu esat theexit pupil ofthe imaging system. In simple

termstheaboveexpressionstatesthattheelddistributionatapoint

(x, y, z)

intheimage spa e an be obtained by applying Fourier transform onthe overall pupil fun tion. For a

mi ros opesueringfromdefo us,thepupil fun tion an bewrittenas

P (k

x

, k

y

, z) =

(

A(θ

i

) exp(jk

0

ϕ(θ

i

, θ

s

, z)),

if

√

_k

₂

x

+k

y

2

k

i

<

N A

n

i

0,

otherwise (29)

where

ϕ(θ

i

, θ

s

, z)

istheopti alphasedieren ebetweenthewavefrontemergingfromtheexit pupilandthereferen espheremeasuredalongtheextremeray,and

θ

i

= sin

−1

_(k

2

x

+k

y

2

)

1/2

/k

i

and

θ

s

= sin

−1

_(k

2

x

+ k

2

y

)

1/2

/k

s

(seeg.12 ).

θ

s

and

θ

i

arerelatedbySnell'slawas

n

i

sin θ

i

= n

s

sin θ

s

.

(30) Infa t,

ϕ(θ

s

, θ

s

, z)

is a sumof two terms,the defo us term

ϕ

d

(θ

s

, z)

andthe termdue to aberrations

ϕ

a

(θ

i

, θ

s

)

. For spheri al aberrationfree imaging onditions, it is to be noted that

θ

i

≈ θ

s

,andhen e

ϕ

a

(θ

i

, θ

s

)

≈ 0

. Thiswillbedis ussedinsubse tion 2.2.1.

If

f

isthedistan ebetweentheexitpupilandthegeometri alimagepoint,thenStokseth obtainedtheexa texpressionforthisdefe toffo usas

ϕ

d

(θ

s

, z) =

−f − z cos θ

s

+ (f

2

+ 2f z + z

2

cos

2

θ

s

)

1

Figure10: Theamplitudeofthepupilfun tionofaCLSMforthe(a)ideal,(b)illumination,

and( )emission ases(©Ariana-INRIA/CNRS).

A small angle approximation an be obtained by making a series expansion of

cos θ

s

and

cos

2

_θ

s

, omittingtermsof

θ

s

withhigherorderthantwo

ϕ

d

(θ

s

, z) =

1

2

zθ

2

s

³

1

₋

z

f + z

´

, θ

s

≪ 1.

(32)

For smalldefo usingsandsmallangles

z

2

_cos

2

_θ

s

≈ z

2

, andhen eEq.(31 )be omes

ϕ

d

(θ

s

, z) = z(1

− cos θ

s

)

(33) Werewritein thespheri al oordinatesas

k

_{· x = k}

_i

_{sin θ}

_i

_{(x cos φ + y sin φ),}

(34) and

d

2

k

_{= k}

2

i

sin θ

i

cos θ

i

dθ

i

dφ.

(35) ByusingEqs.(29),(33 ) ,(34 ),(35 )intoEq. (28 ),weget:

h

A

(x, y, z) = k

2

i

α

Z

0

2π

Z

0

(cos θ

i

)

−

1

2

exp(jk

_i

sin θ

_i

(x cos φ + y sin φ))

·

(36)

exp(jk

s

z(1

− cos θ

s

)) sin θ

i

cos θ

i

dθ

i

dφ.

If

x = ρ cos ψ

and

y = ρ sin ψ

, then

ρ = (x

2

_{+ y}

2

₎

1/2

,and

h

A

(x, y, z) = k

i

2

α

Z

0

2π

Z

0

(cos θ

i

)

1

2

exp(jk

_i

ρ sin θ

_i

cos(φ

− ψ)) exp(jk

_s

z(1

− cos θ

_s

)) sin θ

_i

dθ

_i

dφ.

(37)

(a) The extra term

exp(jkz)

and

exp(jπ/2)

is an eled when the omplex amplitude is squaredtogetthePSF.

(b)Thenal intensitiesarenormalizedso thattheysumtounity.

2.2 Theory of Aberrations

Under ideal onditions,a highNA obje tivelens transformstheplanarwavefrontin ident

onitto a spheri al wavefrontat thefo al region. However,under pra ti alsituations, the

refra tedwavefrontsoprodu edhastogothroughseveralopti alelementsandthespe imen.

Duetothisreasontheemergingwavefrontisrarelyspheri alinnature. Aberrantwavefront

meansthat the resultingobserved images will be distorted as well. Whilethere aremany

aberrations that exist for the mi ros ope, werestri t our analysis to spheri al aberration

(SA)asthisisthedominantandthemostobservableform.

2.2.1 Spheri al Aberrations

Spheri alaberration(SA)isanopti alee to urringwhentheobliqueraysenteringalens

arefo usedinadierentlo ationthanthe entralrays. Themismat hbetweentherefra tive

indexoftheobje tivelensimmersionmediumandthespe imenembeddingmedium auses

SA in uores en e mi ros ope. When light rosses the boundary between materials with

dierent refra tive indi es, it bends a ross the boundary surfa e dierently depending on

the angle of in iden e (light refra tion); the oblique rays are bent more than the entral

rays. If the mismat h is large, e.g. when going from oil lens immersion medium into a

wateryspe imenembeddingmedium, SA ausesthePSFtobe omeasymmetri at depths

of even a few mi rons. Also, the amount of light olle ted from a point sour e de reases

Figure 11: S hemati des ribing the fo using of light when traveling betweenmedium of

withdepthbe auseofanaxialbroadeningofthePSF.Whenmeasuredexperimentallyusing

uorophores[18 ℄,itwasfoundthatthePSF hangesfromafairlysymmetri axialshapeto

anasymmetri shape. Itisimportanttorememberthefollowingfeaturesofadepthvarying

PSF(DVPSF)

ˆ in theabsen eofotheraberrations,thePSFremainsradiallysymmetri al,

ˆ itspeakintensityde reaseswithin reasein thedepth offo us

d

,

ˆ there isanin reasein thewidth ofthePSFin theaxialbutnotthelateraldire tion

within reaseinthedepthoffo us

d

,

ˆ the totalamount oflight olle ted from a givenobje tdoesnot hange with depth.

This is be ause thede rease in peakamplitudeis oin ident with anin reasein the

prolewidthsu hthatthetotal olle tedlightremainsessentially onstant.

We onsider the situation where the obje tive lens fo uses through an interfa e between

mediawith dierent refra tiveindi es as shown in Fig.11 . Thedistan e

d

isthe nominal fo usingdepth, (i.e. thedepthoffo usinamat hedmedium)andthefo usismadefroma

mediumofrefra tiveindex

µ

i

toamediumofrefra tiveindex

µ

s

. Pra ti alimagingsystems are rarelyaberration-free. Withaberrations, the wavefront emerging from the lens is not

a spheri al surfa e. Under su h onditions, the imaging system also loses its property of

shift-invarian e.

2.2.2 The Phase Fa tor

Consider a pointsour e lo ated at a depth

d

below the over slip in a mounting medium of index

n

s

, observed with an obje tive lens designed for an immersion medium with a refra tiveindex

n

i

(see Fig.12 ). In Eq.(37 ),wehadignored theaberrations byassuming that

n

i

≈ n

s

andhen e

θ

i

≈ θ

s

. Whenthisisnotthe ase,wehaveto al ulatetheaberration fun tion

ϕ

a

(θ

i

, θ

s

)

, dueto themismat h of

n

s

and

n

i

. Thephase hange isdeterminedby thedieren ebetweentheopti alpathlengthtraveledtothepupilbyaraythatleavesthe

sour eat anangle

θ

s

relativeto theopti axisandisrefra tedtotheangle

θ

i

uponleaving the mounting medium, and the opti al path length that a ray with angle

θ

i

would have traveledifthemountingmedium indexweretheidealindex

n

i

. Intheirpaper,Gibsonand Lanni[19 ℄ mention that there are

8

parameters (out of a totalof

18

)that may vary from theirdesign onditions as re ommended bythe mi ros opemanufa turer. However, when

ami ros opeisproperly alibrated,thereareonly

3

parametersthatessentiallyvaryunder experimental onditions. These are

1. thespe imenrefra tiveindex

n

s

,

2. theimmersionmedium refra tiveindex

n

i

oftheobje tivelens,and 3. thedepth

d

underthe overslipwhereliestheplaneoffo us.

Figure 12: Spheri al aberration due to refra tive index mismat h

n

s

6= n

i

(© Ariana-INRIA/I3S).

Therefra tiveindexoftheimmersion mediumissensitiveto hangesin temperature

espe- ially if anoil immersion lens is used. Anerror in this parameter signi antlyae ts the

numeri ally omputedPSF.

Using simplegeometri alopti s, we an showthat thisshiftin theopti al path an be

analyti allyexpressedas

ϕ

a

(θ

i

, θ

s

; d, n

i

, n

s

) = d(n

s

cos θ

s

− n

i

cos θ

i

),

(38) where

d

is thefo using at a nominal depth into the spe imen of refra tiveindex

n

s

from a medium ofrefra tiveindex

n

i

. The above phasetermrelies onthe assumptionthat the errorduetomismat hintherefra tiveindi esbetweenthe overglass

n

g

andtheobje tive lenshaseither been ompensated by the orre tion ollaror isminimal. Ifthe over glass

isusedwithanobje tivelensthat issigni antlydierentthanitsdesignspe i ation,an

additionalphasetermshouldbein luded,and

d

repla edbythethi knessofthe overslip. 2.3 Approximating the Point-Spread Fun tion

2.3.1 Nonlinear Phase Approximation

ThephaseterminEq.(38 ) anberewrittenas

ϕ

a

(θ

i

, θ

s

; d, n

i

, n

s

) = d sec θ

i

(n

s

cos θ

s

cos θ

i

− n

i

cos

2

θ

i

),

as

(sin

2

_θ

i

+cos

2

θ

i

) = 1

. Butsin e

n

s

sin θ

s

= n

i

sin θ

i

fromtheSnell'slawand

cos(θ

s

−θ

i

) =

(cos θ

s

cos θ

i

+ sin θ

s

sin θ

i

)

,thephaseis

ϕ

a

(θ

i

, θ

s

; d, n

i

, n

s

)

= d sec θ

i

(n

s

cos θ

s

cos θ

i

− n

i

+ n

s

sin θ

s

sin θ

i

),

= d sec θ

i

(n

s

cos θ

s

cos θ

i

− n

i

+ n

s

sin θ

s

sin θ

i

),

= d sec θ

i

(n

s

cos(θ

s

− θ

i

)

− n

i

)

(40)

Ifweassumethat theaberrationisnotvery large,i.e.

θ

s

≈ θ

i

,sothat

ϕ

a

(θ

i

, θ

s

; d, n

i

, n

s

)

≈ d sec θ

i

(n

s

− n

i

).

(41)

2.3.2 Linear Phase Approximation

Thephasetermin Eq.(38 ) an beapproximatedbyusinga set of ir ular basisfun tions

alled Zernikepolynomials [20 ℄. These polynomials form a ompleteorthogonalset onthe

unit disk. Sin e there is no azimuthal variation, it is su ient to onsider Zernike ir le

polynomials oforder

n

and zerokind

(Z

0

n

(ρ; d, n

i

))

. Here

ρ (0

≤ ρ ≤ 1)

is the normalized radial oordinate.

ϕ

a

(ρ; d, n

i

) = d

NA

(

4

X

n=0

c

n

Z

n

0

(ρ; d, n

i

))

(42)

where the expansion oe ients

(c

n

)

need to be al ulated or determined. Sin e we are interestedonlyindefo usandtherstorderSA,

ϕ

a

(ρ; d, n

i

, n

s

)

isapproximatedtoonlythe Zernikepolynomialsoforderupto

4

.

2.3.3 Apodization Fun tion Approximation

Forthegradient al ulationstobedis ussedinAppendixC,wemakeanapproximationon

theapodizationfun tion

A(θ

i

)

ofEq.(29 ). Wesimplyassumetheapodization fun tionfor ex itationandemissionareapproximatelysame andequalto one. Thatis

A(θ

i

)

≈ 1.

(43)

2.4 Analyti al Expression

Inthis se tion wesummarize the theory that hasbeen dis ussed earlierfor obtaining the

analyti alPSFexpression. Wepro eedrstwiththefollowingdenitions,

Denition Besselfun tionofrstkindandintegerordern

∀ x ∈ R, n ∈ N, J

n

(x) =

1

π

Z

π

0

cos(nθ

_{− x sin θ)dθ}

(44)

Corollary2.4. Besselfun tion of rstkind withorderzero

∀ x ∈ R, J

0

(x) =

1

π

Z

π

0

cos(x sin θ)dθ

(45)

Thus forthenonparaxial ase(i.e. NA

> 0.7

),Eq.(28 ) anbemodiedtonowin ludethe additionalphasetermas

h

A

(x; λ)

∝

α

Z

0

A(θ

i

) sin θ

i

J

0

(k

i

ρ sin θ

i

) exp(jk

0

ϕ

d

(θ

s

, z; n

i

)) exp(jk

0

ϕ

a

(θ

i

, θ

s

; d, n

i

, n

s

))dθ

i

.

(46)

and

h

A

(x; λ) = 2πk

2

i

exp(jk

s

z)

α

Z

0

(cos θ

i

)

1

2

sin θ

_i

J

₀

(k

_i

ρ sin θ

_i

)

·

(47)

exp(jk

0

ϕ

a

(θ

i

, θ

s

; d, n

i

, n

s

)) exp(

−jk

s

z cos θ

s

)dθ

i

.

Byusinga normalizedradial o-ordinate

t = sin θ

i

/ sin α

,theeldat

P

0

be omes

h

A

(x; λ) = 2πk

i

2

exp(jk

s

z)

1

Z

0

(1

_{− t}

2

sin

2

α)

−

1

2

t sin

2

αJ

₀

(ktρ sin α)

·

(48)

exp(jk

0

ϕ

a

(t, θ

s

; d, n

i

, n

s

)) exp(

−jk

s

z(1

− t

2

sin

2

α)

1

2

)dt.

For smallangles(i.e. NA

≤ 0.7

),

sin θ

i

≈ θ

i

andhen e

Eq.

(48)

= 2πk

2

i

sin

2

α

1

Z

0

J

0

(ktρ sin α) exp(jk

0

ϕ(t, θ

s

; d, n

i

, n

s

))tdt,

(49) where

ϕ(t, θ

s

; d, n

i

, n

s

) = ϕ

a

(t, α, θ

s

; d, n

o

, n

s

) + ϕ

d

(t, z, α),

(50) and

ϕ

d

(t, z, α) =

k

s

z sin

2

_α

2k

0

t

2

(51)

isthequadrati defo usterm.

TheFouriertransformof a ir ularlysymmetri al fun tion

g(ρ)

written as

B(g(ρ))

an begivenas

B(g(ρ)) = G

k

= 2π

∞

Z

0

rg(r)J

0

(2πrk)dr

(52)

where,

J

0

isthebesselfun tionoftherstkindoforderzero. Fora ir ularlysymmetri al fun tion

f (ρ, θ)

≡ f(ρ)

,theFourier-Bessel transform or Hankeltransformof zeroorder is givenas:

B[f(ρ)] = 2π

∞

Z

0

ρf (ρ)J

0

(2πρk)dρ.

(53)

ThustheFouriertransformofa ir ularlysymmetri fun tionisitself ir ularlysymmetri al.

Denition Cir lefun tion

circ(ρ) =











1,

ρ < 1

1

2

, ρ = 1

0,

otherwise. (54)

TheFourier-Besseltransformofthe ir le fun tion anbewrittenas

B[circ(ρ)] = 2π

1

Z

0

ρJ

0

(2πρk)dρ

(55)

Usinga hange ofvariable

ρ

′

_{= 2πρk}

andtheidentity

R

x

0

ξJ

0

(ξ)dξ = xJ

1

(x)

,

B[circ(ρ)] =

_2πk

1

2

2πk

Z

0

ρ

′

_J

0

(ρ

′

)dr

′

=

1

k

J

1

(2πk)

(56)

where,

J

1

(

·)

is a Bessel fun tionof therstkind andorder one. By using the notationof Fourier-BesseltransformorHankeltransformofzero-order,Eq.(49 )issimpliedas

h

A

(x; λ) = k

2

i

sin

2

α

B[exp(jk

0

ϕ(t, θ

s

; d, n

i

, n

s

))].

(57) Summarizingtheresultsobtained earlier,theamplitudedistribution an beeither written

as

h

A

(x)

∝

α

Z

0

A(θ

i

) sin(θ

i

)J

0

(k

i

ρ sin θ

i

) exp(jk

0

ϕ(θ

i

, θ

s

, z; d, n

i

, n

s

))dθ

i

,

(58)

whi h isasimpli ationofEq.(37 )orEq.(49 )as(byignoringapodization)

h

A

(x, y, z; λ)

∝

1

Z

0

J

0

(k

i

ρ sin αt) exp(jk

0

ϕ

d

) exp(jk

i

ϕ

a

)tdt,

(59)

where

J

0

is the Bessel fun tion of the rst kind of order zero,

ϕ

d

= (z sin

2

_αt

2

_)/2

,

ϕ

i

=

d(n

s

cos θ

s

− n

i

cos θ

i

)

,

n

i

sin θ

i

= n

s

sin θ

s

and

n

i

sin α =

NA.

u = (2πn

o

/λ)ρ sin α

and

For a singlephoton(

1

-p)ex itation, whenthe uorophorereturns to thegroundstate theemittedwavelength is longerthan theex itation wavelength (Stoke's shift). Fromthe

Helmholtzre ipro itytheorem,thePSFiswrittenasaprodu toftheex itationdistribution

andtheemissiondistributionas

h

T h

∝ |h

A

(x; λ

ex

)

|

2

·

Z

x

2

1

+y

2

1

≤

D2

4

|h

A

(x

− x

1

, y

− y

1

, z; λ

em

)

|

2

dx

1

dy

1

,

(60)

where

λ

ex

and

λ

em

arerespe tivelytheex itation andemissionwavelengths,

D

istheba k proje teddiameter of the ir ular pinhole. Fig.13 showsa numeri ally omputed PSFby

usingtheEqs.(33 ),(38 ),(50 )and(58 )inEq.(60 ). Themi ros opeusesanex itationbeam

ofwavelength

λ

ex

= 488

nm,andtheemissionpeakhasawavelengthof

λ

em

= 520

nm. The obje tivelens has a NA of

1.3

,

40

X magni ation and is an immersion oil typerefra tive index

n

i

= 1.518

.

In in oherent imaging,the distribution of intensity in the image plane is found by

in-tegratingtheintensitydistributions in thedira tionimagesasso iatedwith ea hpointin

theobje t. Thusif

o(x

′

₎

istheintensityat

x

′

in theobje tplane,theintensityatthepoint

x

intheimageisobtained as(ignoringtheinuen eofnoise):

i(x) =

Z

x∈Ω

s

o(x

′

)h(x

− x

′

)dx

′

_.

(61)

(a)

(b)

Figure 13: Maximum intensity proje tion (MIP) of a numeri ally omputed spheri ally

aberrated onfo al PSF along the (a) opti axis giving the radial plane, (b) lateral axis

3 Literature Review on Blind De onvolution

In this se tion we review some existing methods on blind de onvolution applied to

mi- ros opy. It is di ult to make anexhaustive reviewof all theliterature available in this

eld. So,wehave hosenonlythosethatresembletheproblemwearehandlingormethods

thathaveraised onsiderableinterestsonthissubje t. Wehaveskippedthesurveyon

de- onvolutionalgorithmsasthisexer ise was arriedoutearlierin [21 ,22 ℄andmore re ently

inourresear h report[8℄withdierentobje tregularization.

3.1 Maximum Likelihood Approa h

Ifweassumethattheobservedimage

i(x)

isarealizationofanindependentPoissonpro ess atea h voxel,thenthelikelihood anbewrittenas

Pr(i

|o, h) =

Y

x∈Ω

s

[(h

∗ o) + b](x)

i(x)

_e

−[(h∗o)+b](x)

i(x)!

,

(62)

where, the mean of the Poisson pro ess is given by

[(h

∗ o) + b](x)

. Conventional blind de onvolutionalgorithmsestimatetheobje tandthePSFdire tlyfromtheaboveequation.

Thatis

(ˆ

o, ˆ

h) = arg max

(o,h)

{Pr(i|o, h)}

(63)

Amaximumlikelihood (ML)approa h anbeusedtore overboththeobje tandPSF as

ˆ

o

ML

= arg max

o

{Pr(i|o, h)}

= arg min

o

{− log(Pr(i|o, h))}.

(64)

ˆ

h

ML

=

arg max

h

{Pr(i|o, h)}

=

arg min

h

{− log(Pr(i|o, h))}.

(65)

An expli it iterative multipli ative algorithm basedon Maximum Likelihood Expe tation

Maximization(MLEM) [9 ℄formalism an bederivedfromtheaboveexpressionas

ˆ

o

n+1

ML

(x) = ˆo

n

ML

(x)

·

µ

i(x)

(ˆ

o

n

ML

∗ h)(x)

∗ h(−x)

¶

q

o

,

∀x ∈ Ω

s

.

(66)

Themethodinvolvesalternatingbetweenobje tminimizationaboveandthePSF

minimiza-tion

ˆ

h

n+1

ML

(x) = ˆh

n

ML

(x)

·

µ

i(x)

(o

_{∗ ˆh}

n

ML

)(x)

∗ o(−x)

¶

q

h

,

∀x ∈ Ω

s

,

(67)

where

n

is the index of iteration of the algorithm. For those unfamiliar with MLEM, its derivationfromtheBayes'stheoremandasformalizedbyRi hardson[23 ℄issummarizedin

AppendixA. Theparameters

q

o

, q

h

∈ [1, 1.5]

ontrolsthe onvergen eofthetwoiterations. When

q

o

and

q

h

is unity, then wearriveat the lassi al MLEMalgorithm and whenthey aremorethan

3

, they onverge(andsometimes diverge) rapidly.

In Fig. 14, we show the results of applying Eqs. (66 ) and (67 ) to the observation of

Fig.5( ). As this algorithm has to be manually terminated, we show the results for two

iterations with

n = 70

and

n = 200

. Fig. 14(a) and (d) gives the true aberrated PSF and the obje t used for simulating the observation. Sin e the algorithm does not have

anyinformationabouttheobje tor thePSF (otherthanpositivityandux onservation),

theestimate of theobje t

o

ˆ

(n)

_(x)

in Fig. 14(e)and (f), resembles losely thetrue PSF in

Fig. 14 (a), in shape and in the position of the COG. Similarly, the estimate of the PSF

ˆ

h

(n)

_(x)

in Fig. 14 (b) and ( ) resembles more the imaged axially entered mi rosphere in

Fig.14 (d). Thus,havingnopriorknowledgeontheobje torthePSF anmakeitdi ult

to separate or distinguish them from ea h other. From this simple example, we learly

observe that in the MLEM estimation, the hara teristi s of the PSF is absorbed by the

obje t andvi e versa. Also,when theiterationsareallowed to ontinue, progressivelythe

resultsstartdeterioratingas anbeseenfromFig.14 ( )after

200

iterationsofthea elerated blind de onvolution. We perform another experiment, where the obje t is axially shifted

fromits enterbytwoplanesasshowninFig.15 (a). Fig.15( )showsthesametruePSFas

inFig.14 (a). Fig.15 (b)and(d)show

o

ˆ

n

_(x)

and

ˆ

h

n

_(x)

after

n = 70

iterations. Theresults ofthisexperimentleadsusto on ludethatthetrueaxialpositionoftheobje t annotalso

bere overedusingthismethodofMLEMblindde onvolution.

3.2 Penalized Maximum Likelihood Approa h

Inthe lassi al MLEM,the stopping riteriaisusually thenumber ofiterations. Thisis a

onstraintintrodu edinthealgorithmthatpreventsearlydivergen eoftheresults. The

ear-liestblindde onvolutionapproa hthatwasappliedtoin oherentquantum-limitedimaging

andappli ation touores entmi ros opeswas by Holmesin [24 ℄. Atea h iterationofthe

MLEMforthePSF, a unitsummation onstraintisenfor ed onthethePSF. Inaddition,

the energy of the PSF is onstrained to lie within an hourglass-shaped region by using a

Ger hberg-Saxton[25 ℄typealgorithm. Thispreventsportionoftheba kgrounduores en e

b(x)

from beingerroneously onsidered ashavingoriginatedfromthePSF. Finally,a ban-dlimiting and positivity riterion is introdu ed. This form of grouped oordinate des ent

is typi ally referred to as an iterative blind de onvolution (IBD). Biggsand Andrews [26 ℄

provided an alternativeapproa h for a elerating theIBD. Sin e theobje testimate

on-vergesfasterthanthePSFestimates,amodi ationwasproposedwhereinthea eleration

wasa hievedbyperformingseveraliterationsforthePSFafterea hobje testimation. The

numberof y lesofPSF iterationsto applyafter ea h iterationoftheobje testimatewas

experimentally hosen. Givenanobservationdata,theestimationoftheparameters

{q

o

, q

h

}

orthe numberof iterationsin Biggsmethod [26 ℄ remainsstillan openproblem. Although

(a)TruePSF

h(x)

(b)

ˆ

h

(n)

_(x)

,

n = 70

( )

ˆ

h

(n)

_(x)

,

n = 200

(d)trueobje t

o(x)

(e)

o

ˆ

(n)

_(x)

,

n = 70

(f)

o

ˆ

(n)

_(x)

,

n = 200

Figure 14: Blind de onvolution results based on the MLEM algorithm after

70

and

200

iterations(© Ariana-INRIA/I3S).