Chapter 17:Image Post Processing and Analysis PDF

Chapter 17:Image Post Processing and Analysis Slide set of 176 slides based on the chapter authored by P.A.Yushkevich of the IAEA publication (ISBN 978-92-0-131010-1): Diagnostic Radiology Physics. A Handbook forTeachers and Students Objective: To familiarize the s...

Chapter 17:Image Post Processing and Analysis Slide set of 176 slides based on the chapter authored by P.A.Yushkevich of the IAEA publication (ISBN 978-92-0-131010-1): Diagnostic Radiology Physics. A Handbook forTeachers and Students Objective: To familiarize the student with the most common problems in image post processing and analysis, and the algorithms to address them. Slide set prepared by E.Berry (Leeds,UK and The Open University in London) IAEA International Atomic Energy Agency CHAPTER17 TABLE OF CONTENTS 17.1.Introduction 17.2. Deterministic Image Processing and Feature Enhancement 17.3. Image Segmentation 17.4. - Image Registration 17.5. Open-source tools for image analysis IAEA Diagnostic Radiology Physics:A Handbook for Teachers and Students-17.Slide 1 (02/176) 17.1INTRODUCTION 17.1 IAEA Diagnostic Radiology Physics:A Handbook for Teachers and Students-17.1 Slide 1 (03/176) 17.1INTRODUCTION 17.1 Introduction (1 of 2) For decades,scientists have used computers to enhance and analyze medical images Initially simple computer algorithms were used to enhance the appearance of interesting features in images, helping humans read and interpret them better Later,more advancedalgorithms were developed,where the computer would not only enhanceimages,but also participate in understanding their content IAEA Diagnostic Radiology Physics:A Handbook for Teachers and Students-17.1 Slide 2 (04/176 17.1INTRODUCTION 17.1 Introduction 2 of2) Segmentation algorithms were developed to detect and extract specific anatomical objects in images, such as malignant lesions in mammograms Registration algorithms were developed to align images of different modalities and to find corresponding anatomical locations in images from different subjects These algorithms have made computer-aided detection and diagnosis computer-guided surgery,and other highly complex medical technologies possible Today, the field of image processing and analysis is a complex branch of science that lies at the intersection of applied mathematics computer science,physics,statistics, and biomedical sciences IAEA Diagnostic Radiology Physics:A Handbook for Teachers and Students-17.1 Slide 3 (05/176 17.1INTRODUCTION 17.1 Overview This chapter is divided into two main sections classical image processing algorithms image filtering, noise reduction, and edge/feature extraction from images more modern image analysis approaches including segmentation and registration IAEA Diagnostic Radiology Physics:A Handbook for Teachers and Students-17.1 Slide 4 (06/176) 17.1INTRODUCTION 17.1 Image processing vs.Image analysis The main feature that distinguishes image analysis from image processing is the use of external knowledge about the objects appearing in the image This external knowledge can be based on heuristic knowledge physical models data obtained from previous analysis of similar images Image analysis algorithms use this external knowledge to fill in the information that is otherwise missing or ambiguous in the images IAEA Diagnostic Radiology Physics:A Handbook for Teachers and Students-17.1 Slide 5 (07/176 17.1INTRODUCTION 17.1 Example of image analysis A biomechanical model of the heart may be used by an image analysis algorithm to help find the boundaries of the heart in a CT or MR image This model can help the algorithm tell true heart boundaries from various other anatomical boundaries that have similar appearance in the image IAEA Diagnostic Radiology Physics:A Handbook for Teachers and Students-17.1 Slide 6 (08/176 17.1INTRODUCTION 17.1 The most important limitation of image processing Image processing cannot increase the amount of information available in the input image Applying mathematical operations to images can only remove information present in an image sometimes,removing information that is not relevant can make it easier for humans to understand images Image processing is always limited by the quality of the input data if an imaging system provides data of unacceptable quality,it is better to try to improve the imaging system, rather than hope that the "magic' of image processing will compensate for poor imaging IAEA Diagnostic Radiology Physics:A Handbook for Teachers and Students-17.1 Slide 7 (09/176 17.2 DETERMINISTICIMAGEPROCESSING AND FEATURE ENHANCEMENT 17.2 IAEA Diagnostic Radiology Physics:A Handbook for Teachers and Students-17.2 Slide 1 (12/176) 17.2 DETERMINISTICIMAGEPROCESSING AND FEATURE ENHANCEMENT 17.2.1SPATIAL FILTERING AND NOISEREMOVAL IAEA Diagnostic Radiology Physics:A Handbook for Teachers and Students-17.2.1 Slide 1 (13/176) 17.2 DETERMINISTICIMAGE PROCESSING AND FEATURE ENHANCEMENT 17.2.1 Spatial Filtering and Noise Removal Filtering Filtering is an operation that changes the observable quality of an image,in terms of resolution contrast noise Typically,filtering involves applying the same or similar mathematical operation at every pixel in an image for example,spatial filtering modifies the intensity of each pixel in an image using some function of the neighbouring pixels Filtering is one of the most elementary image processing operations IAEA Diagnostic Radiology Physics:A Handbook for Teachers and Students-17.2.1 Slide2 14/176 17.2 DETERMINISTICIMAGE PROCESSING AND FEATURE ENHANCEMENT 17.2.1 Spatial Filtering and Noise Removal Mean filtering in the image domain A very simple example of a spatial filter is the mean filter Replaces each pixel in an image with the mean of the N x N neighbourhood around the pixel The output of the filter is an image that appears more 'smooth"and less"noisy'than the input image Input image convolved Input X ray image with a 7x7mean filter Averaging over the small neighbourhood reduces the magnitude of the intensity discontinuities in the image IAEA Diagnostic Radiology Physics:A Handbook for Teachers and Students-17.2.1 Slide 3 (15/176 17.2 DETERMINISTICIMAGE PROCESSING AND FEATURE ENHANCEMENT 17.2.1 Spatial Filtering and Noise Removal Mean filtering Mathematically,the mean filter is defined as a convolution between the image and a constant-valued NxN matrix 1 1 filtered=IK; - K N2 The N x N mean filter is a low-pass filter A low-pass filter reduces high-frequency components in the Fourier transform (FT) of the image IAEA Diagnostic Radiology Physics:A Handbook for Teachers and Students-17.2.1 Slide4 16/176 17.2 DETERMINISTICIMAGE PROCESSING AND FEATURE ENHANCEMENT 17.2.1 Spatial Filtering and Noise Removal Convolution and the Fourier transform The relationship between Fourier transform (FT) and convolution is F{AoB}=F{A}F{B} Convolution of a digital image with a matrix of constant values is the discrete equivalent of the convolution of a continuousimagefunction with the rect (boxcar) function The FT of the rect function is the sinc function So,mean filtering is equivalent to multiplying the FT of the image by the sinc function this mostly preserves the low-frequency components of the image and diminishes the high-frequency components IAEA Diagnostic Radiology Physics:A Handbook for Teachers and Students-17.2.1 Slide 5 17/176 17.2 DETERMINISTICIMAGE PROCESSING AND FEATURE ENHANCEMENT 17.2.1 Spatial Filtering and Noise Removal 17.2 DETERMINISTICIMAGE PROCESSING AND FEATURE ENHANCEMENT 17.2.1 Spatial Filtering and Noise Removal Image smoothing Mean filtering is an example of an image smoothing operation Smoothing and removal of high-frequency noise can help human observers understand medical images Smoothing is also an important intermediate step for advanced image analysis algorithms Modern image analysis algorithms involve numerical optimization and require computation of derivatives of functions derived from image data smoothing helps make derivative computation numerically stable IAEA Diagnostic Radiology Physics:A Handbook for Teachers and Students-17.2.1 Slide 7 19/176 17.2 DETERMINISTICIMAGE PROCESSING AND FEATURE ENHANCEMENT 17.2.1 Spatial Filtering and Noise Removal Ideal Low-Pass Filter The so-called ideal low-pass filter cuts off all frequencies above a certain threshold in the FT of the image in the Fourier domain, this is achieved by multiplying the FT of the image by a cylinder-shaped filter generated by rotating a one-dimensional rect function around the origin theoretically, the same effect is accomplished in the image domain by convolution with a one-dimensional sinc function rotated around the origin Assumes that images are periodic functions on an infinite domain in practice, most images are not periodic convolution with the rotated sinc function results in an artefact called ringing Another drawback of the ideal low-pass filter is the computational cost which is very high in comparison to mean filtering IAEA Diagnostic Radiology Physics:A Handbook for Teachers and Students-17.2.1 Slide 8 20/176  17.2 DETERMINISTICIMAGE PROCESSING AND FEATURE ENHANCEMENT 17.2.1 Spatial Filtering and Noise Removal Ideal low-pass filter and ringing artefact The ideal low-pass filter, i.e. The original image The image after a sinc function rotated around convolution with the low- the centre of the image pass filter.Notice how the bright intensity of the rib bones on the right of the image is replicated in the soft tissue to the right IAEA Diagnostic RadiologyPhysics:A Handbook for Teachers and Students-17.2.1 Slide 9 (21/176 17.2 DETERMINISTICIMAGE PROCESSING AND FEATURE ENHANCEMENT 17.2.1 Spatial Filtering and Noise Removal Gaussian Filtering The Gaussian filter is a low-pass filter that is not affected by the ringing artefact In the continuous domain,the Gaussianfilter is defined as the normal probability density function with standard deviation o, which has been rotated about the origin in x,y space Formally,theGaussian filter is definedas 202 G.x,y 2 where the value is called the width of the Gaussian filter IAEA Diagnostic Radiology Physics:A Handbook for Teachers and Students-17.2.1 Slide 10 22/176 17.2 DETERMINISTICIMAGE PROCESSING AND FEATURE ENHANCEMENT 17.2.1 Spatial Filtering and Noise Removal FT of Gaussian filter The FT of the Gaussian filter is also a Gaussian filter with reciprocal width 1/o FGxy=Gnv) where n,u are spatial frequencies IAEA Diagnostic Radiology Physics:A Handbook for Teachers and Students-17.2.1 Slide 11 23/176 17.2 DETERMINISTICIMAGE PROCESSING AND FEATURE ENHANCEMENT 17.2.1 Spatial Filtering and Noise Removal Discrete Gaussian filter The discrete Gaussian filter is a (2N+1)x(2N+1) matrix Its elements, G, are given by G=Gi-N-1,j-N-1) The size of the matrix,2N+1,determines how accurately the discrete Gaussian approximates the continuous Gaussian A common choice is N>=3o IAEA Diagnostic Radiology Physics:A Handbook for Teachers and Students-17.2.1 Slide 12 24/176 17.2 DETERMINISTICIMAGE PROCESSING AND FEATURE ENHANCEMENT 17.2.1 Spatial Filtering and Noise Removal Examples of Gaussian filters 0.005 E2 0-01 5011 0.005 301 A continuous2D Gaussian A discrete21x21 with=2 Gaussian filter with =2 IAEA Diagnostic Radiology Physics:A Handbook for Teachers and Students-17.2.1 Slide 13 (25/176) 17.2 DETERMINISTICIMAGE PROCESSING AND FEATURE ENHANCEMENT 17.2.1 Spatial Filtering and Noise Removal Application of the Gaussian filter - To apply low-pass filtering to a digital image,we perform convolution between the image and the Gaussian filter this is equivalent to multiplying the FT of the image by a Gaussian filter with width 1/o The Gaussian function decreases very quickly as we move away from the peak at the distance 4o from the peak, the value of the Gaussian is only 0.0003 of the value at the peak Convolution with the Gaussian filter removes high frequencies in the image low frequencies are mostly retained the larger the standard deviation of the Gaussian filter, the smoother the result of the filtering IAEA Diagnostic Radiology Physics:A Handbook for Teachers and Students-17.2.1 Slide 14 26/176 17.2 DETERMINISTICIMAGE PROCESSING AND FEATURE ENHANCEMENT 17.2.1 Spatial Filtering and Noise Removal An image convolved with Gaussian filters with different widths 17.2 DETERMINISTICIMAGE PROCESSING AND FEATURE ENHANCEMENT 17.2.1 Spatial Filtering and Noise Removal Median Filtering The median filter replaces each pixel in the image with the median of the pixel values in an N x N neighbourhood Taking the median of a set of numbers is a non-linear operation therefore,median filtering cannon be represented as convolution The median filter is useful for removing impulse noise,a type of noise where some isolated pixels in the image have very high or very low intensity values The disadvantage of median filtering is that it can remove important features, such as thin edges IAEA Diagnostic Radiology Physics:A Handbook for Teachers and Students-17.2.1 Slide 16 28/176 17.2 DETERMINISTICIMAGE PROCESSING AND FEATURE ENHANCEMENT 17.2.1 Spatial Filtering and Noise Removal Example of Median Filtering Original image Image degraded by The result of filtering The result of filtering adding "salt and the degraded image with a 5x5 median pepper'noise.The with a 5x5 mean filter filter.Much of the salt intensity of a tenth of and pepper noise has the pixels has been been removed-but replaced by 0 or 255 some of the fine lines in the image have also been removed by the filtering IAEA Diagnostic Radiology Physics:A Handbook for Teachers and Students-17.2.1 Slide 17 (29/176 17.2 DETERMINISTICIMAGE PROCESSING AND FEATURE ENHANCEMENT 17.2.1 Spatial Filtering and Noise Removal Edge-preserving smoothing and de-noising When we smooth an image,we remove high-frequency components This helps reduce noise in the image,but it also can removeimportant high-frequency features such as edges an edge in image processing is a discontinuity in the intensity function for example, in an X ray image, the intensity is discontinuous along the boundaries between bone and soft tissue Some advanced filtering algorithms try to remove noise in images without smoothing edges e.g.the anisotropic diffusion algorithm (Perona and Malik) IAEA Diagnostic Radiology Physics:A Handbook for Teachers and Students-17.2.1 Slide 18 30/176 17.2 DETERMINISTICIMAGE PROCESSING AND FEATURE ENHANCEMENT 17.2.1 Spatial Filtering and Noise Removal Anisotropic Diffusion algorithm Mathematically,smoothing an image with a Gaussian filter is analogous to simulating heat diffusion in a homogeneous body In anisotropic diffusion, the image is treated as an inhomogeneous body,with different heat conductance at different places in the image near edges,the conductance is lower, so heat diffuses more slowly preventing the edge from being smoothed away away from edges, the conductance is faster The result is that less smoothing is applied near image edges The approach is only as good as our ability to detect image edges IAEA Diagnostic Radiology Physics:A Handbook for Teachers and Students-17.2.1 Slide 19 31/176 17.2 DETERMINISTICIMAGEPROCESSING AND FEATURE ENHANCEMENT 17.2.2 EDGE.RIDGE AND SIMPLE SHAPE DETECTION IAEA Diagnostic Radiology Physics:A Handbook for Teachers and Students-17.2.2 Slide 1 (32/176) 17.2 DETERMINISTICIMAGE PROCESSING AND FEATURE ENHANCEMENT 17.2.2 Edge, Ridge and Simple Shape Detection Edges One of the main applications of image processing and image analysis is to detect structures of interest in images In many situations,the structure of interest and the surrounding structures have different image intensities By searching for discontinuities in the image intensity function,we can find the boundaries of structures of interest these discontinuities are called edges for example, in an X ray image,there is an edge at the boundary between bone and soft tissue IAEA Diagnostic Radiology Physics:A Handbook for Teachers and Students-17.2.2 Slide2 (33/176 17.2 DETERMINISTICIMAGE PROCESSING AND FEATURE ENHANCEMENT 17.2.2 Edge, Ridge and Simple Shape Detection Edge detection Edge detection algorithms search for edges in images automatically Because medical images are complex, they have very many discontinuities in the image intensity most of these are not related to the structure of interest may be discontinuities due to noise,imaging artefacts, or other structures Good edge detection algorithms identify edges that are more likely to be of interest However, no matter how good an edge detection algorithm is, it wil frequently find irrelevant edges edge detection algorithms are not powerful enough to completely automatically identify structures of interest in most medical images instead, they are a helpful tool for more complex segmentation algorithms. as well as a useful visualization tool IAEA Diagnostic Radiology Physics:A Handbook for Teachers and Students-17.2.2 Slide 3 (34/176 17.2 DETERMINISTICIMAGE PROCESSING AND FEATURE ENHANCEMENT 17.2.2 Edge,Ridge and Simple Shape Detection Tube detection Some structures in medical images have very characteristic shapes For example,blood vessels are tube-like structures with gradually varying width two edges that are roughly parallel to each other This property can be exploited by special tube-detection algorithms IAEA Diagnostic Radiology Physics:A Handbook for Teachers and Students-17.2.2 Slide4 (35/176 17.2 DETERMINISTICIMAGE PROCESSING AND FEATURE ENHANCEMENT 17.2.2 Edge, Ridge and Simple Shape Detection 17.2 DETERMINISTICIMAGE PROCESSING AND FEATURE ENHANCEMENT 17.2.2 Edge, Ridge and Simple Shape Detection How image derivatives are computed An edge is a discontinuity in the image intensity Therefore,the directional derivative of the image intensity in the direction orthogonal to the edge must be large,as seen in the preceding figure Edge detection algorithms exploit this property In order to compute derivatives,we require a continuous function, but an image is just an array of numbers One solution is to use the finite difference approximation of the derivative IAEA Diagnostic Radiology Physics:A Handbook for Teachers and Students-17.2.2 Slide 6 (37/176 17.2 DETERMINISTICIMAGE PROCESSING AND FEATURE ENHANCEMENT 17.2.2 Edge, Ridge and Simple Shape Detection Finite difference approximation in 1D From the Taylor series expansion,it is easy to derive the following approximation of the derivative 28 where 8 is a real number Os is the error term, involving 8 to the power of two and greater when8 such that I(x) and J(o(x) are"similar"for all x in The meaning of "similar" depends on the application in the context of medical image analysis,"similar"usually means "describing the same anatomical location' however, in practice such anatomical similarity cannot be quantified, and similar" means "having similar image intensity features" IAEA Diagnostic Radiology Physics:A Handbook for Teachers and Students-17.4 Slide 3(128/176 17.4 IMAGE REGISTRATION 17.4 Characterisation of image registration problems There are many different types of image registration problems They can be characterizedby two main components the transformation model the similarity metric IAEA Diagnostic Radiology Physics:A Handbook for Teachers and Students-17.4 Slide 4 (129/176 17.4 IMAGE REGISTRATION 17.4 Image Registration 17.4.1 Transformation Models 17.4.2 Registration Similarity Metrics 17.4.3 The General Framework for Image Registration 17.4.4 Applications of Image Registration IAEA Diagnostic Radiology Physics:A Handbook for Teachers and Students-17.4 Slide 5(130/176) 17.4 IMAGE REGISTRATION 17.4.1 TRANSFORMATION MODELS IAEA Diagnostic Radiology Physics:A Handbook for Teachers and Students-17.4.1 Slide 1 (131/176) 17.4 IMAGE REGISTRATION 17.4 Image Registration 17.4.1 Transformation Models 17.4.2 Registration Similarity Metrics 17.4.3 The General Framework for Image Registration 17.4.4 Applications of Image Registration IAEA Diagnostic Radiology Physics:A Handbook for Teachers and Students-17.4.1 Slide2 132/176 17.4 IMAGE REGISTRATION 17.4.1Transformation Models Linear vs.non-linear transformations The transformation can take many forms The transformation is called linear when it has the form (x)=Ax+b where Ais an nxn matrix bis an n x1vector Otherwise,the transformation is non-linear IAEA Diagnostic Radiology Physics:A Handbook for Teachers and Students-17.4.1 Slide 3 (133/176) 17.4 IMAGE REGISTRATION 17.4.1Transformation Models Rigid vs.non-rigid transformations A special case of linear transformations are rigid transformations The matrix in rigid transformations is a rotation matrix Rigid transformations describe rigid motions They are used in applications when the object being imaged moves without being deformed Non-rigid linear transformations,as well as non-linear transformations,are called deformable transformations IAEA Diagnostic Radiology Physics:A Handbook for Teachers and Students-17.4.1 Slide 4 (134/176 17.3 IMAGESEGMENTATION 17.4.1Transformation Models 17.4 IMAGE REGISTRATION 17.4.1Transformation Models Parametric transformations Non-linear transformations can be parametric or non- parametric Parametric transformations have the form x=x+ Wfxe+Wfxe+.. i=1 where fx-R is a basis, such as the Fourier basis or the B-spline basis e, e are unit vectors in the cardinal coordinate directions WW. are the coefficients of the basis functions IAEA Diagnostic Radiology Physics:A Handbook for Teachers and Students-17.4.1 Slide 6 136/176 17.4 IMAGE REGISTRATION 17.4.1Transformation Models Parametric vs.non-parametric transformations Usually, a relatively small number of low-frequency basis functions is used to represent a parametric transformation The resulting transformations vary smoothly across Such transformations are called low-dimensional non- linear transformations Non-parametric transformations do not have such a parametric form Instead, at every point in ,a vector v(x) is defined,and the transformation is simply given by x=x+vx IAEA Diagnostic Radiology Physics:A Handbook for Teachers and Students-17.4.1 Slide 7 (137/176 17.4 IMAGE REGISTRATION 17.4.1Transformation Models Diffeomorphic transformations Diffeomorphic transformations are a special class of non-parametric deformable transformations they are differentiable on Q and have a differentiable inverse e.g.in one dimension (n=1), diffeomorphic transformations are monotonically increasing (or monotonically decreasing) functions Very useful for medical image registration because they describe realistic transformations of anatomy,without singularities such as tearing or folding Registration algorithms that restrict deformations to be diffeomorphic exploit the property that the composition of two diffeomorphic transformations is also diffeomorphic the deformation between two images is constructed by composing many infinitesimal deformations, each of which is itself diffeomorphic IAEA Diagnostic Radiology Physics:A Handbook for Teachers and Students-17.4.1 Slide 8 138/176 17.4 IMAGE REGISTRATION 17.4.2 REGISTRATIONSIMILARITY METRICS IAEA Diagnostic Radiology Physics:A Handbook for Teachers and Students-17.4.2 Slide 1 (139/176) 17.4 IMAGE REGISTRATION 17.4 Image Registration 17.4.1 Transformation Models 17.4.2 Registration Similarity Metrics 17.4.3 The General Framework for Image Registration 17.4.4 Applications of Image Registration IAEA Diagnostic Radiology Physics:A Handbook for Teachers and Students-17.4.2 Slide2 140/176) 17.4 IMAGEREGISTRATION 17.4.2 Registration Similarity Metrics Similarity metrics Image registration tries to match places in images that are similar Since true anatomical similarity is not known, surrogate measures based on image intensity are used Many metrics have been proposed We will only review three such metrics Mean squared intensity difference Mutual information Cross-correlation IAEA Diagnostic Radiology Physics:A Handbook for Teachers and Students-17.4.2 Slide 3 (141/176 17.4 IMAGEREGISTRATION 17.4.2 Registration Similarity Metrics Mean squared intensity difference The similarity is measured as difference in image intensity The similarity of images I and J is given by Sim(I,J [1(x)-J($(x))]dx 9 Simple to compute Appropriate when anatomically similar places can reasonably be expected to have similar image intensity values Not appropriate for registration of images with different modalities MRI registration,because MRI intensity values are not consistent across scans IAEA Diagnostic Radiology Physics:A Handbook for Teachers and Students-17.4.2 Slide4 (142/176 17.4 IMAGE REGISTRATION 17.4.2 Registration Similarity Metrics Mutual information (1 of 3) Very useful for multimodality image registration A pair of images of the body are acquired with different modalities in modality 1,bone may have intensity range 100-200 and soft tissue may have range10-20 in modality 2, bone may have intensity between 3000 and 5000 and soft tissue may have intensity between 10000 and 20000 The mean square intensity difference metric would return very large values if these two images are aligned properly Another metric is needed that does not directly compare the intensity values IAEA Diagnostic Radiology Physics:A Handbook for Teachers and Students-17.4.2 Slide 5 143/176 17.4 IMAGE REGISTRATION 17.4.2 Registration Similarity Metrics Mutual information (2 of 3) The mutual information metric is derived from information theory To compute mutual information between images I and J we treat the pairs of intensity values (IJ as samples from a pair of random variablesX,Y One such sample exists at each pixel Mutual information is a measure of how dependent random variablesXand Y are on each other IAEA Diagnostic Radiology Physics:A Handbook for Teachers and Students-17.4.2 Slide 6 (144/176 17.4 IMAGEREGISTRATION 17.4.2 Registration Similarity Metrics Mutual information (3 of 3) Mutual information is given by p(x,y) J J p(x,y)log pxpy) dxdy where p(x,y) is the joint density of X and Y p(x) is the marginal density of x, p(y) is the marginal density of y The marginal densities are estimated by the histograms of the images I and J The joint density is estimated by the two-dimensional joint histogram of the imagesI andJ IAEA Diagnostic Radiology Physics:A Handbook for Teachers and Students-17.4.2 Slide 7 (145/176 17.3 IMAGESEGMENTATION 17.4.2 Registration Similarity Metrics Illustration of the joint histogram used in the computation of the mutual information metric Axial slice from an Axial slice from Joint histogram PET slice Joint histogram MR image a PET image of the MR and rotated out of of the MRI and aligned with PET images alignment with misaligned the MRI the MRI PET slice IAEA Diagnostic Radiology Physics:A Handbook for Teachers and Students-17.4.2 Slide 8(146/176 17.4 IMAGE REGISTRATION 17.4.2 Registration Similarity Metrics Cross-correlation The cross-correlation metric is computed as follows at each pixel index k,we compute the correlation coefficient between the values of imageI in a small neighbourhood of pixels surrounding k,and the values of image J over the same neighbourhood the correlation coefficients are summed up over the whole image The cross-correlation metric is robust to noise because it considers neighbourhoods rather than individual pixels However,it is expensive in computation time IAEA Diagnostic Radiology Physics:A Handbook for Teachers and Students-17.4.2 Slide 9 (147/176 17.4IMAGE REGISTRATION 17.4.3 THE GENERAL FRAMEWORK FOR IMAGE REGISTRATION IAEA Diagnostic Radiology Physics:A Handbook for Teachers and Students-17.4.3 Slide 1 (148/176) 17.4 IMAGE REGISTRATION 17.4.3 The General Framework for Image Registration General algorithmic framework for image registration 17.4 IMAGE REGISTRATION 17.4.3 The General Framework for Image Registration General algorithmic framework -transformation Usually, one of the images is designated as a reference image and the other image is the moving image Transformations are applied to the moving image, while the reference image remains unchanged The transformation is defined by some set of parameters small set for linear registration bigger for parametric non-linear registration and very large for non-parametric non-linear registration IAEA Diagnostic Radiology Physics:A Handbook for Teachers and Students-17.4.3 Slide 3 (150/176 17.4 IMAGE REGISTRATION 17.4.3 The General Framework for Image Registration General algorithmic framework -similarity metric Some initial parameters are supplied usually these initial parameters correspond to the identity transformation The transformation is applied to the moving image this involves resampling and interpolation because the values of (x) fall between voxel centres The resampled imageJ(o(x)) is compared to the reference image I(x) using the similarity metric this results in a dissimilarity value registration seeks to minimize this dissimilarity value IAEA Diagnostic Radiology Physics:A Handbook for Teachers and Students-17.4.3 Slide 4 (151/176 17.4 IMAGE REGISTRATION 17.4.3 The General Framework for Image Registration General algorithmic framework - regularization prior In many registration problems, an additional term, called the regularization prior, is minimized This term measures the complexity of the transformation favours smooth,regular transformations over irregular transformations can be though of as an Occam's razor prior for transformations IAEA Diagnostic Radiology Physics:A Handbook for Teachers and Students-17.4.3 Slide 5 (152/176 17.4 IMAGE REGISTRATION 17.4.3 The General Framework for Image Registration General algorithmic framework - objective function value Together,the dissimilarity value and the regularization prior value are combined into an objective function value The gradient of the objective function with respect to the transformation parameters is also computed Numerical optimization updates the values of the transformation parameters so as to minimize the objective function IAEA Diagnostic Radiology Physics:A Handbook for Teachers and Students-17.4.3 Slide 6 (153/176 17.4 IMAGE REGISTRATION 17.4.4 APPLICATIONS OF IMAGE REGISTRATION IAEA Diagnostic Radiology Physics:A Handbook for Teachers and Students-17.4.4 Slide 1 (154/176) 17.4 IMAGE REGISTRATION 17.4.4 Applications of Image Registration Applications of Image Registration There are many image analysis problems that require image registration Different problems require different transformation models. and different similarity metrics We can group medical image registration problems into two general categories registration that accounts for differences in image acquisition registration that accounts for anatomical variability (image normalisation) IAEA Diagnostic Radiology Physics:A Handbook for Teachers and Students-17.4.4 Slide2 (155/176 17.4 IMAGEREGISTRATION 17.4.4 Applications of Image Registration Registration that accounts for differences in image acquisition In many biomedical applications,multiple images of the same subject are acquired images may have completely different modalities (MRI vs.CT,CT vs PET,etc.) images may be acquired on the same piece of equipment using different imaging parameters even when parameters are identical.the position of the subject in the scanner may change between images To co-analyse multiple images of the same subject, it is necessary to match corresponding locations in these images This is accomplished using image registration Within this category, there are several distinct subproblems that require different methodology IAEA Diagnostic Radiology Physics:A Handbook for Teachers and Students-17.4.4 Slide 3 156/176 17.4 IMAGE REGISTRATION 17.4.4 Applications of Image Registration Accounting for Subject's Motion When multiple images of a subject are acquired in a short span of time, the subject may move for example,in fMRI studies,hundreds of scans are acquired during an imaging session To analysethe scans,they must first be aligned,so that the differences due to subject motion are factored out Motion correction typically uses image registration with rigid transformation models Simple image similarity metrics suffice IAEA Diagnostic Radiology Physics:A Handbook for Teachers and Students-17.4.4 Slide4 (157/176 17.4 IMAGE REGISTRATION 17.4.4 Applications of Image Registration Alignment of Multi-Modality 3D Images Often information from different imaging modalities must be combined for purposes of visualisation, diagnosis, and analysis for example,CT and PET images are often co-analysed, with CT providing high-resolution anatomical detail, and PET capturing physiological measures,such as metabolism The images have very different intensity patterns so registration requires specialised image similarity metrics,such as mutual information Often rigid transformations suffice however,some modalities introduce geometric distortions to images and low-dimensional parametric transformations may be necessary to align images IAEA Diagnostic Radiology Physics:A Handbook for Teachers and Students-17.4.4 Slide 5 158/176 17.4 IMAGE REGISTRATION 17.4.4 Applications of Image Registration Alignment of 3D and 2D Imaging Modalities Sometimes registration is needed to align a 2D image of the subject to a 3D image this problem arises in surgical and radiotherapy treatment contexts a3D scan is acquired and used to plan the intervention during the intervention,X ray or angiographic images are acquired and used to ensure that the intervention is being performed according to the plan corrections to the intervention are made based on the imaging For this to work, image registration must accurately align images of different dimensions and different modality This is a challenging problem that typically requires the image registration algorithm to simulate 2D images via data from the 3D image IAEA Diagnostic Radiology Physics:A Handbook for Teachers and Students-17.4.4 Slide 6 (159/176 17.4 IMAGE REGISTRATION 17.4.4 Applications of Image Registration Registration that Accounts for Anatomical Variability (Image Normalisation) The other major application of image registration is to match corresponding anatomical locations in images of different subjects in images where the anatomy of a single subject has changed over time The term commonly used for this is image normalisation Again,there are several different applications IAEA Diagnostic Radiology Physics:A Handbook for Teachers and Students-17.4.4 Slide 7 (160/176 17.4 IMAGE REGISTRATION 17.4.4 Applications of Image Registration Cross-sectional morphometry (1 of 2) Often we are interested in measuring how the anatomy of one group of subjects differs from another in a clinical trial, we may want to compare the anatomy of a cohort receiving a trial drug to the cohort receiving a placebo We may do so by matching every image to a common template image using image registration with non-linear transformations We may then compare the transformations from the template to the images in one cohort to the transformations to the images in the other cohort IAEA Diagnostic Radiology Physics:A Handbook for Teachers and Students-17.4.4 Slide 8 161/176 17.4 IMAGE REGISTRATION 17.4.4 Applications of Image Registration Cross-sectional morphometry (2 of 2) Specifically, we may examine the Jacobian of each transformation TheJacobian of the transformation describesthe local change in volume caused by the transformation If an infinitesimal region in the template has volume 8V the transformation o maps this region into a region of volume 8V Then the ratio 8V,/ 8V,equals the determinant of the Jacobian of the transformation IAEA Diagnostic Radiology Physics:A Handbook for Teachers and Students-17.4.4 Slide 9 162/176 17.4 IMAGE REGISTRATION 17.4.4 Applications of Image Registration Longitudinal morphometry We may acquire multiple images of a subject at different time points when studying the effect on human anatomy of disease intervention aging To measure the differences over time,we can employ parametric or non-parametric deformable registration Because the overall anatomy does not change extensively between images,the regularisation priors and other parameters of registration may need to be different than for cross-sectional morphometry IAEA Diagnostic Radiology Physics:A Handbook for Teachers and Students-17.4.4 Slide 10163/176 17.5 OPEN-SOURCE TOOLS FORIMAGE ANALYSIS 17.5 IAEA Diagnostic Radiology Physics:A Handbook for Teachers and Students-17.5 Slide 1 (164/176) 17.5OPEN-SOURCE TOOLS FOR IMAGE ANALYSIS 17.5 Open-source tools for image analysis This section briefly reviews several mature image processing and analysis tools that were available freely on the Internet at the time of writing The reader can experiment with the techniques described in this chapter by downloading and running these tools Most tools run on Apple and PC computers (with Linux and Windows operating systems These are just of few of many excellent tools available to the reader The Neuroimaging InformaticsTools and Resources Clearinghouse (NITRC,http://www.nitrc.org) is an excellent portal for finding free image analysis software IAEA Diagnostic Radiology Physics:A Handbook for Teachers and Students-17.5 Slide 2(165/176 17.5OPEN-SOURCE TOOLS FOR IMAGE ANALYSIS 17.5 Open-source tools for image analysis: URLs ImageJ http://rsbweb.nih.gov/ij ITK-SNAP* http://itksnap.org FSL http://www.fmrib.ox.ac.uk/fsl OsiriX http://www.osirix-viewer.com 3D Slicer http://slicer.org * Disclaimer: the book chapter author is involved in development of ITK-SNAP IAEA Diagnostic Radiology Physics:A Handbook for Teachers and Students-17.5 Slide 3(166/176 17.5OPEN-SOURCE TOOLS FOR IMAGE ANALYSIS 17.5 ImageJ ImageJ provides a wide array of image processing operations that can be applied to2D and 3D images In addition to basic image processing (filtering,edge detection,resampling),ImageJ provides some higher-level image analysis algorithms ImageJ is written in Java ImageJ can open many common 2D image files, as well as DiCOM format medical imaging data IAEA Diagnostic Radiology Physics:A Handbook for Teachers and Students-17.5 Slide 4 (167/176 17.5OPEN-SOURCE TOOLS FOR IMAGE ANALYSIS 17.5 ITK-SNAP ITK-SNAP is a tool for navigation and segmentation of 3D medical imaging data ITK-SNAP implements the active contour automatic segmentation algorithmsby Caselles et al.(1997) and Zhu and Yuille (1996) It also provides a dynamic interface for navigation in 3D images Several tools for manual delineation are also provided. ITK-SNAP can open many 3D image file formats,including DICOM,NIfTI and Analyze IAEA Diagnostic Radiology Physics:A Handbook for Teachers and Students-17.5 Slide 5(168/176 17.5OPEN-SOURCE TOOLS FOR IMAGE ANALYSIS 17.5 FSL FSL is a software library that offers many analysis tools for MRI brain imaging data It includes tools for linear image registration (FLIRT),non linear image registration (FNIRT),automated tissue classification (FAST) and many others FSL supports NIfTI and Analyze file formats,among others IAEA Diagnostic Radiology Physics:A Handbook for Teachers and Students-17.5 Slide 6(169/176 17.5OPEN-SOURCE TOOLS FOR IMAGE ANALYSIS 17.5 OsiriX OsiriXis a comprehensive PACS workstation and DiCOM image viewer It offers a range of visualization capabilities and a built-in segmentation tool Surface and volume rendering capabilities are especially well-suited for CTdata OsiriX requires an Apple computer with MacOS X IAEA Diagnostic Radiology Physics:A Handbook for Teachers and Students-17.5 Slide 7 (170/176 17.5OPEN-SOURCE TOOLS FOR IMAGE ANALYSIS 17.5 3D Slicer Slicer is an extensive software platform for image display and analysis It offers a wide range of plug-in modules that provide automatic seqmentation,registration and statistical analysis functionality Slicer also includes tools for image-guided surgery Many file formats are supported IAEA Diagnostic Radiology Physics:A Handbook for Teachers and Students-17.5 Slide 8(171/176 17.BIBLIOGRAPHY 17. IAEA Diagnostic Radiology Physics:A Handbook for Teachers and Students-17.Bibliography Slide 1 (172/176 17.BIBLIOGRAPHY 17.Bibliography:Image Processing DUDA,R.,HART,P.,Use of the Hough transformation to detect lines and curves in pictures,Communications of the ACM15(1972)11-15 GONZALEZ,R.C.,WOODS,R.E.,EDDINS,S.,Digital image processing using MATLAB, Repr. with corr. [i.e.3rd ] edn,Prentice Hall, Upper Saddle River,NJ 2004)xvi,716 pp. LINDEBERG,T.,Edge Detection and Ridge Detection with Automatic Scale Selection,Int.J.Comput.Vision301998)117-156 PERONA,P.,MALIK, J.,Scale-Space and Edge Detection Using Anisotropic Diffusion,IEEE T.Pattern Anal 12(1990) IAEA Diagnostic Radiology Physics:A Handbook for Teachers and Students-17.Bibliography Slide 2(173/176 17.BIBLIOGRAPHY 17.Bibliography:Image Segmentation (1 of 2) CASELLES.V..KIMMEL.R.,SAPIRO,G.,Geodesic active contours,Int.J Comput.Vision22 (1997)61-79 COLLINS.D.L..ZIJDENBOS,A.P..KOLLOKIAN,V..SLED.J.G.,KABANI N.J.,HOLMES,C.J.,EVANS,A.C.Design and Construction of a Realistic Digital Brain Phantom,IEEE Transactions on Medical Imaging,17 (1998)463- 468. COOTES,T.F.,TAYLOR,C.J.,COOPER,D.H.,GRAHAM,J.,Active shape models -- their training and application, Computer Vision and Image Understanding 61(1995)38-59 COOTES,T.F.,EDWARDS,G.J.,TAYLOR,C.J.,Active Appearance Models IEEE Trans.Pattern Anal.Mach. Intell., IEEE Computer Society 23(2001) 681-685. KASS,M..WITKIN,A.,TERZOPOULOS,D.,Snakes:active contour models International Int.J.Comput.Vision11988)321-331. LI,S.Z.,Markov Random Field Modeling in Image Analysis, 3rd edn, Springer (2009). IAEA Diagnostic Radiology Physics:A Handbook for Teachers and Students-17.Bibliography Slide 3(174/176 17.BIBLIOGRAPHY 17.Bibliography:Image Segmentation (2 of 2) PHAM.D.L.,XU,C.,PRINCE,J.L.,Current methods in medical image segmentation,Annu Rev Biomed Eng 2 (2000)315-37. http://www.ncbi.nlm.nih.gov/entrez/query.fcgi?cmd=Retrieve&db=PubMed&do pt=Citation&list uids=11701515. ZHU,S.C.,YUILLE,A.,Region Competition:Unifying Snakes, Region Growing,and Bayes/MDL for Multiband Image Segmentation,IEEE Trans Pattern Anal.Mach.Intell.,IEEE Computer Society 181996)884-900. IAEA Diagnostic Radiology Physics:A Handbook for Teachers and Students-17.Bibliography Slide 4(175/176 17.BIBLIOGRAPHY 17.Bibliography:Image Registration MAINTZ,J.B.,VIERGEVER,M.A.,A survey of medical image registration, Med Image Anal 2 11998)1-36. http://www.ncbi.nlm.nih.gov/entrez/query.fcgi?cmd=Retrieve&db=PubMed&do pt=Citation&list uids=10638851. PLUIM.J.P..MAINTZ.J.B..VIERGEVER.M.A..Mutual-information-based registration of medical images: a survey, IEEE transactions on medical imaging 22 8 2003)986-1004. http://www.ncbi.nlm.nih.gov/entrez/query.fcgi?cmd=Retrieve&db=PubMed&do pt=Citation&list uids=12906253. IAEA Diagnostic Radiology Physics:A Handbook for Teachers and Students-17.Bibliography Slide 5(176/176