Sensor Technologies and Biological Sensing Lecture 6 PDF

BIOMIMETIC AND BIOINSPIRED SENSORS: VISION, HEARING Lecture 6 2024. 11. 06. Dr. Sándor Földi Sensor technologies and biological sensing  CONTENTS 1. Optics 2. Physiology of vision 3. C...

BIOMIMETIC AND BIOINSPIRED SENSORS: VISION, HEARING Lecture 6 2024. 11. 06. Dr. Sándor Földi Sensor technologies and biological sensing  CONTENTS 1. Optics 2. Physiology of vision 3. Classic camera 4. Bioinspired vision sensor technologies 5. Depth camera 6. Physiology of hearing 7. Bioinspired hearing sensor 3  BASIC OPTICS An image can be formed when light reflected from an object or scene (at the object plane), is brought to focus on a surface (at the image plane). In a camera: the sensor array is located at the image plane Sharpest image: with a converging lens or system of lenses 4 Reference: Lakhtakia, Akhlesh, and Raúl José Martín-Palma, eds. Engineered biomimicry. Newnes, 2013.  BASIC OPTICS Gaussian lens equation (previous slide’s Figure): 1 1 1 + = 𝑠𝑜 𝑠𝑖 𝑓 Simplest optical arrangement Focal length and most other optical parameters are dependent on the wavelength (λ) Optical infinity: an object distance that results in an image plane distance very close to the focal length For designers: 𝑠0 ≥ 100𝑓 For visual acuity exams: 𝑠0 ≈ 338𝑓 5  BASIC OPTICS Spatial sampling of the image: Center-to-center distance between sensor locations equals the reciprocal of the spatial sampling frequency. Like temporal sampling, limited by the sampling theorem: Only spatial frequencies in the image up to one-half the spatial sampling frequency can be sampled and reconstructed without aliasing. Aliasing: Most noticeable to humans with regard to periodic patterns, such as the stripes of a person’s tie or shirt, which when aliased tend to look broader and distorted 6  BASIC OPTICS – APERTURE SIZE No real world lens can focus light to an infinitesimally tiny point → Blur spots Effective aperture size: diameter, numerical aperture (in microscopy), f-number (in photography) Blur examples: 7 Reference: Lakhtakia, Akhlesh, and Raúl José Martín-Palma, eds. Engineered biomimicry. Newnes, 2013.  BASIC OPTICS– DEPTH OF FIELD Depth of Field: There is an axial distance over which objects are imaged with acceptable sharpness. DOF is smaller for larger apertures. Longer focal length lenses have a smaller DOF for a given aperture 8 Reference: Lakhtakia, Akhlesh, and Raúl José Martín-Palma, eds. Engineered biomimicry. Newnes, 2013.  BASIC OPTICS– FIELD OF VIEW Field of view: The span over which a given scene is imaged Typically, it isn’t determined by the aperture size The approximate FOV is determined only by the geometry of: Angular FOV is independent of object distance For an imaging sensor of size a in a given direction, the angular FOV in that direction is 2arctan(𝑎Τ2𝑠𝑖) 9 Reference: Lakhtakia, Akhlesh, and Raúl José Martín-Palma, eds. Engineered biomimicry. Newnes, 2013.  BASIC OPTICS – REFLECTION AND REFRACTION When light enters an optical system, it encounters a boundary between two different indices of refraction n Examples: Air: n = 1 Lens made of crown glass: n = 1.5 Snell’s law: 𝑠𝑖𝑛𝜃1 𝑛2 = 𝑠𝑖𝑛𝜃2 𝑛1 Predicts the angle of refraction The angle of reflection is equal to the angle of incidence Reflectance (R): the fraction of incident light intensity (power) that is reflected. Transmittance (T): the fraction of incident light intensity that is refracted. R+T = 1 10 Reference: Lakhtakia, Akhlesh, and Raúl José Martín-Palma, eds. Engineered biomimicry. Newnes, 2013.  BASIC OPTICS – ABERRATIONS Most aberrations are due to imperfections in lenses: Monochromatic: Only one wavelength is considered at a time Spherical, comma, astigmatism, field curvature, defocus, barrel distortion and pin-cushion distortion Chromatic: Multiple wavelengths are considered Primarily due to the fact that the refractive index of any material, including lens’ glass, is wavelength dependent Chromatic aberration appears in a color image as fringes of inappropriate color along edges that separate bright and dark regions of the image 11  BASIC OPTICS – ABERRATIONS Monochromatic aberration examples Reference: Xu, Z., Ning, Y., Jiang, T., & Cheng, X. (2017, May). Integrated design course of applied optics focusing on operating and maintaining abilities. In Education and 12 Training in Optics and Photonics. Optical Society of America.  BASIC OPTICS – ABERRATIONS Chromatic aberration examples: Reference: https://en.wikipedia.org/wiki/File:Chromatic_aberration_(comparison).jpg; https://en.wikipedia.org/wiki/File:Purple_fringing.jpg; 13 https://en.wikipedia.org/wiki/File:Chris-chromatic-aberration.png  BASIC OPTICS – FOURIER OPTICS A powerful and practical method for design considerations such as apertures, lenses, photodetector size and spatial sampling Point Spread Function (PSF): Every optical component has a PSF defined in the spatial domain at the focal plane, which describes how an infinitesimally small point of light (the optical equivalent of a Dirac delta function δ(xo, yo) at the object plane) is spread (or smeared) by that component. Optical Transfer Function (OTF): The Fourier transform of the PSF PSF is in the spatial domain, OTF is in the spatial frequency domain Modulation Transfer Function (MTF): The magnitude of the OTF Contrast Transfer Function (CTF): Line pair: used as a maximum frequency sinusoidal pattern Conversion between MTF and CTF is well known 14  RECAP – VISION In details: Lecture 2. 15 Reference: Lakhtakia, Akhlesh, and Raúl José Martín-Palma, eds. Engineered biomimicry. Newnes, 2013.  CLASSIC CCD CAMERA CCD: Charge Coupled Device Introduction: 1974 Bell Laboratory Popular since 1980 Majority of the cameras contains CCDs Cheap semiconductor technology No additional devices (e.g. AD converter, special amplifiers, digital circuits can be integrated to the same chip) Sensor matrix: Discrete sensors in space: discretization, pixelization Incoming photons generates free electrons Photons with too small energy (infrared) don’t generate electrons Photons with too high energy (ultra violet) is absorbed before reaching the sensor layer Quantum efficiency (typically 70%) The free electrons are trapped in potential holes (2-100,000 pieces of electrons) Blooming: overflow of too many electrons 16  CLASSIC CCD CAMERA Photoactive region and a transmission region made out of a shift register. Steps of sensing: The array is exposed to the image A control circuit causes each capacitor to transfer its contents to its neighbor (operating as a shift register). The last capacitor in the array dumps its charge into a charge amplifier, which converts the charge into a voltage. By repeating this process, the controlling circuit converts the entire contents of the array in the semiconductor to a sequence of voltages. 17 Reference: https://en.wikipedia.org/wiki/Charge-coupled_device  CLASSIC CCD CAMERA Bayer filter mosaic: a color filter array (CFA) for arranging RGB color filters on a square grid of photosensors. The filter pattern is half green, one quarter red and one quarter blue. Better color separation can be reached by three-CCD devices and a dichroic beam splitter prism, that splits the image into red, green and blue components. Each of the three CCDs is arranged to respond to a particular color. 18 Reference: https://en.wikipedia.org/wiki/Charge-coupled_device  BIOMIMETIC VISION SENSOR APPROACHES There are at least 10 known variants of animal eyes Two main groups: Noncompound eye Compound eye Noncompound Eye: Refractive cornea eye (Camera eye) Compound Eye: Apposition eye Optical superposition eye Neural superposition eye 19  CAMERA EYE Nearly all mammals, including humans, have camera eye The primary refractive power is due to the air/cornea optical interface Additional refractive effect is sometimes provided by an internal lens, such as the variable-shape crystalline lens that humans use to accomodate focus for close objects Relatively large aperture → permitting good light gathering and keeping the blur spot acceptably small in the short focal distance (good static acuity) Artificial vision system based on camera eye uses a single large-aperture lens or lens systems (mimicking the cornea and lens) combined with a relatively large, high resolution focal plane array of photodetectors (mimicking the photoreceptors in the retina) 20  CAMERA EYE Simplification of human eye: Helmholtz’s shematic eye (reduced eye model): Single refractive surface: 5.5 mm radius of curvature Posterior nodal distance: 16.7 mm Angular span: 16.7 𝑚𝑚 × tan(1°)=291.5 𝜇𝑚/𝑑𝑒𝑔 Reference: Bar-Cohen, Yoseph. "Nature as a model for mimicking and inspiration of new technologies." International Reference: Lakhtakia, Akhlesh, and Raúl José Martín-Palma, eds. Journal of Aeronautical and Space Sciences 13.1 (2012): 1-13. Engineered biomimicry. Newnes, 2013. 21  CAMERA EYE In a typical human eye: The optic disk has a mean diameter of 1800 μm Optic disk subtends an arc roughly 6.2° Not truly spherical shape Resolution limit (based only on the diffraction-limited point-spread function of the pupil): 1 min of arc; which assumes a 2 mm pupil and illumination wavelength of 550 nm Simple camera eye model Almost all the standard cameras and vision sensor are based on camera eye Focal plane array (FPA): typically rigidly attached to the sensor frame Focal length is fixed (except for zoom lenses) If some particular object distance is required: the optical center of the lens or lens system is moved axially to ensure that the image distance remains equal to the distance between the nodal point and the fixed FPA 22  BIOMIMETIC CAMERA EYE EXAMPLE EC-Eye (Electrochemical eye) 23 Reference: Gu, L., Poddar, S., Lin, Y., Long, Z., Zhang, D., Zhang, Q., & Fan, Z. (2020). A biomimetic eye with a hemispherical perovskite nanowire array retina. Nature, 581(7808), 278-282.  BIOMIMETIC CAMERA EYE EXAMPLE EC-Eye (Electrochemical eye) 24 Reference: Gu, L., Poddar, S., Lin, Y., Long, Z., Zhang, D., Zhang, Q., & Fan, Z. (2020). A biomimetic eye with a hemispherical perovskite nanowire array retina. Nature, 581(7808), 278-282.  BIOMIMETIC CAMERA EYE EXAMPLE EC-Eye (Electrochemical eye) 25 Reference: Gu, L., Poddar, S., Lin, Y., Long, Z., Zhang, D., Zhang, Q., & Fan, Z. (2020). A biomimetic eye with a hemispherical perovskite nanowire array retina. Nature, 581(7808), 278-282.  BIOMIMETIC CAMERA EYE EXAMPLE EC-Eye (Electrochemical eye) 26 Reference: Gu, L., Poddar, S., Lin, Y., Long, Z., Zhang, D., Zhang, Q., & Fan, Z. (2020). A biomimetic eye with a hemispherical perovskite nanowire array retina. Nature, 581(7808), 278-282.  COMPOUND EYE Many insects and other species have compound eye Advanced features: Wide field of view Depth perception by employing stereo vision Depending on specific species, a compound eye includes from tens to thousands of individual hexagonal-shaped facet lenses. Ommatidium: primary modular vision unit of the compound eye Interommatidial angle: angle between adjacent ommatidia, ranges from 1 to 2 degrees 27 Reference: Prabhakara, R. S., Wright, C. H., & Barrett, S. F. (2010). Motion detection: A biomimetic vision sensor versus a CCD camera sensor. IEEE Sensors Journal, 12(2), 298-307.  COMPOUND EYE After light enters the facet lens, it passes through and is focused by the crystalline cone. The cone is made up of a transparent solid material and focuses the impinging light on the proximal end of the rhabdom. The rhabdom channels the light to the photosensitive receptors called rhabdomeres 28  COMPOUND EYE TYPES Three basic insect vision configurations: Apposition (a) Superposition (b) Neural superposition (c) 29 Reference: Lakhtakia, Akhlesh, and Raúl José Martín-Palma, eds. Engineered biomimicry. Newnes, 2013.  COMPOUND EYE – VISUAL PROCESSING Fairly significant vision process take place early in the vision system (prior to the brain) Primitive forms of object recognition take place on the retina Center-surround motion-detection neurons: Directionally sensitive to different stimuli Different orientation sensitivities Feed a higher order convexity cell to produce optical flow Optical flow: The apparent motion of surfaces or objects in a scene, resulting from the motion difference between the observer and a scene 30  COMPOUND EYE – VISUAL PROCESSING Convexity cell: 31 Reference: Lakhtakia, Akhlesh, and Raúl José Martín-Palma, eds. Engineered biomimicry. Newnes, 2013.  COMPOUND EYE – VISUAL PROCESSING Motion processing models: Differential- or gradient- based models employing first and second derivatives to determine velocity Region- or feature-based matching to determine movement between adjacent temporal image scenes (frames) Phase-based models employing an array of band-pass filters that parse the incoming signal according to scale, speed, and orientation Energy- or frequency-based methods that quantify the output energy from velocity tuned Gabor filters 32  COMPOUND EYE – VISUAL PROCESSING Motion processing: 33 Reference: Lakhtakia, Akhlesh, and Raúl José Martín-Palma, eds. Engineered biomimicry. Newnes, 2013.  COMPOUND EYE – VISUAL PROCESSING Hassenstein-Reinhardt model of motion: 34 Reference: Lakhtakia, Akhlesh, and Raúl José Martín-Palma, eds. Engineered biomimicry. Newnes, 2013.  COMPOUND EYE – EXAMPLES Fly eye sensors Reference: Prabhakara, R. S., Wright, C. H., & Barrett, S. F. (2010). Motion detection: A biomimetic vision sensor versus a CCD camera sensor. IEEE Sensors Journal, 12(2), 298- 307.; 35 Lakhtakia, Akhlesh, and Raúl José Martín-Palma, eds. Engineered biomimicry. Newnes, 2013.  COMPOUND EYE – EXAMPLES Fly eye sensors: Ideal neural superposition Reference: Prabhakara, R. S., Wright, C. H., & Barrett, S. F. (2010). Motion detection: A biomimetic vision sensor versus a CCD camera sensor. IEEE Sensors Journal, 12(2), 298- 307.; 36 Lakhtakia, Akhlesh, and Raúl José Martín-Palma, eds. Engineered biomimicry. Newnes, 2013.  COMPOUND EYE – EXAMPLES Fly eye sensors: Incredible motion detection capabilities (motion hyperacuity) Reference: Prabhakara, R. S., Wright, C. H., & Barrett, S. F. (2010). Motion detection: A biomimetic vision sensor versus a CCD camera sensor. IEEE Sensors Journal, 12(2), 298- 307.; 37 Lakhtakia, Akhlesh, and Raúl José Martín-Palma, eds. Engineered biomimicry. Newnes, 2013.  COMPOUND EYE – EXAMPLES Mantis shrimp – Adaptive contrast vision Both color vision and polarization vision Three axis eye movement: pitch, yaw and roll → polarization contrast in their field of view can be adjusted in real time 38 Reference: Kim, DaeEun, and Ralf Möller. "Biomimetic whiskers for shape recognition." Robotics and Autonomous Systems 55.3 (2007): 229-243.  DEPTH CAMERAS Can be realized with two or more cameras Other solution: Time-of-flight cameras Usually IR light 39  DEPTH CAMERAS Challenges: Apply multiple cameras simultaneously Low resolution imaging Holes in depth map Segmentation Overlapping 40 Reference: https://web.media.mit.edu/~achoo/tr/3d_benefits_limits.pdf  RECAP – EAR STRUCTURE 41 Reference: Hall, John E., and Guyton AC. "Guyton and Hall textbook of medical physiology." (2006).  RECAP – ORGAN OF CORTI 42 Reference: Hall, John E., and Guyton AC. "Guyton and Hall textbook of medical physiology." (2006).  ARTIFICIAL HAIR CELLS 43 Reference: Liu, Chang. "Micromachined biomimetic artificial haircell sensors." Bioinspiration & biomimetics 2.4 (2007): S162.  BIOMIMETIC SOUND DETECTION Human hearing frequency range: 20 Hz to 20 kHz Sound pressure range: 20 μPa to 20 Pa Artificial hair cell Mimic the sound conversion into an electric signal by the inner hair cell and the amplification process generated by the outer hair cell MEMS technology, self-sensing and self-actuated cantilever Feedback control loop Reference: Lenk, C., Ekinci, A., Rangelow, I. W., & Gutschmidt, S. (2018, July). Active, artificial hair cells for biomimetic sound detection based on active cantilever technology. In 2018 40th Annual 44 International Conference of the IEEE Engineering in Medicine and Biology Society (EMBC) (pp. 4488-4491). IEEE.  BIOMIMETIC TYMPANIC MEMBRANE Tympanic membrane (TM) consists of three parts: malleus, pars tensa and pars flaccida Curved conical shape TM does not move as a typical diaphragm, because the pressure force is transmitted only from the center of the TM when the TM moves in response of the sound pressure Reference: Yoon, Jong-Yun, and Gi-Woo Kim. "Harnessing the bilinear nonlinearity of a 3D printed biomimetic diaphragm for acoustic sensor applications." Mechanical 45 systems and signal processing 116 (2019): 710-724.  BIOMIMETIC TYMPANIC MEMBRANE Fused deposition modeling technique: Two materials for 3D printing: thermo plastic elastomer (→durable, flexible diaphragm) and polivinyl alcohol (→support) Reference: Yoon, Jong-Yun, and Gi-Woo Kim. "Harnessing the bilinear nonlinearity of a 3D printed biomimetic diaphragm for acoustic sensor applications." Mechanical 46 systems and signal processing 116 (2019): 710-724.  BIOMIMETIC ARTIFICIAL BASILAR MEMBRANE Basilar membrane: Pseudo-resonant structure Different width, stiffness, mass, damping, and duct dimensions at different points along its length High-frequency sounds localize near the base of the cochlea, while low- frequency sounds localize near the apex Artificial basilar membrane: An acoustic transducer that mimics cochlear tonotopy, which is the passive mechanical frequency selectivity of the basilar membrane and acoustic-to- electrical energy conversion of the hair cells Different MEMS technology-based approaches 47  BIOMIMETIC ARTIFICIAL BASILAR MEMBRANE Trapezoidal membrane type Beam array type 48 Reference: Jang, J., Jang, J. H., & Choi, H. (2017). Biomimetic Artificial Basilar Membranes for Next‐Generation Cochlear Implants. Advanced healthcare materials, 6(21)  BIOMIMETIC ARTIFICIAL BASILAR MEMBRANE Flexible Piezoelectric zirconite titanite (PZT) thin film-based 49 Reference: Jang, J., Jang, J. H., & Choi, H. (2017). Biomimetic Artificial Basilar Membranes for Next‐Generation Cochlear Implants. Advanced healthcare materials, 6(21)  BIOMIMETIC ARTIFICIAL BASILAR MEMBRANE MEMS Piezoelectric aluminium nitride (AIN)-based 50 Reference: Jang, J., Jang, J. H., & Choi, H. (2017). Biomimetic Artificial Basilar Membranes for Next‐Generation Cochlear Implants. Advanced healthcare materials, 6(21)  BIOMIMETIC ARTIFICIAL BASILAR MEMBRANE Flexible Piezoelectric polyvinylidene diflouride (PVDF)-based 51 Reference: Jang, J., Jang, J. H., & Choi, H. (2017). Biomimetic Artificial Basilar Membranes for Next‐Generation Cochlear Implants. Advanced healthcare materials, 6(21)  BIOMIMETIC ARTIFICIAL BASILAR MEMBRANE Triboelectric-based 52 Reference: Jang, J., Jang, J. H., & Choi, H. (2017). Biomimetic Artificial Basilar Membranes for Next‐Generation Cochlear Implants. Advanced healthcare materials, 6(21)  SUMMARY – QUESTIONS Introduce briefly 4 important basic concepts of optics that is important in biomimetic vision sensors. What are the two main types of biomimetic eyes? What are the differences between them? In which situation are they more usable? What is the concept of compound eyes? Introduce the three primery insect vision configurations. How can motion hyperacuity be reached? How are biomimetic hair cells connected to hearing? Introduce a biomimetic basilar membrane sensor. 53  REFERENCES 1. Lakhtakia, Akhlesh, and Raúl José Martín-Palma, eds. Engineered biomimicry. Newnes, 2013. 2. Xu, Z., Ning, Y., Jiang, T., & Cheng, X. (2017, May). Integrated design course of applied optics focusing on operating and maintaining abilities. In Education and Training in Optics and Photonics. Optical Society of America. 3. Bar-Cohen, Yoseph. "Nature as a model for mimicking and inspiration of new technologies." International Journal of Aeronautical and Space Sciences 13.1 (2012): 1-13. 4. Gu, L., Poddar, S., Lin, Y., Long, Z., Zhang, D., Zhang, Q., & Fan, Z. (2020). A biomimetic eye with a hemispherical perovskite nanowire array retina. Nature, 581(7808), 278-282. 5. Prabhakara, R. S., Wright, C. H., & Barrett, S. F. (2010). Motion detection: A biomimetic vision sensor versus a CCD camera sensor. IEEE Sensors Journal, 12(2), 298-307. 6. Martín-Palma, Raúl J., and Mathias Kolle. "Biomimetic photonic structures for optical sensing." Optics & Laser Technology 109 (2019): 270-277. 7. Zhong, B., Wang, X., Gan, X., Yang, T., & Gao, J. (2020). A biomimetic model of adaptive contrast vision enhancement from mantis shrimp. Sensors, 20(16), 4588. 8. Auffarth, Benjamin, Bernhard Kaplan, and Anders Lansner. "Map formation in the olfactory bulb by axon guidance of olfactory neurons." Frontiers in systems neuroscience 5 (2011): 84. 54  REFERENCES 9. Lenk, C., Ekinci, A., Rangelow, I. W., & Gutschmidt, S. (2018, July). Active, artificial hair cells for biomimetic sound detection based on active cantilever technology. In 2018 40th Annual International Conference of the IEEE Engineering in Medicine and Biology Society (EMBC) (pp. 4488-4491). IEEE 10. Yoon, Jong-Yun, and Gi-Woo Kim. "Harnessing the bilinear nonlinearity of a 3D printed biomimetic diaphragm for acoustic sensor applications." Mechanical systems and signal processing 116 (2019): 710-724. 11. Jang, J., Jang, J. H., & Choi, H. (2017). Biomimetic Artificial Basilar Membranes for Next‐Generation Cochlear Implants. Advanced healthcare materials, 6(21), 1700674. 12. Liu, Chang. "Micromachined biomimetic artificial haircell sensors." Bioinspiration & biomimetics 2.4 (2007): S162. 13. Hall, John E., and Guyton AC. "Guyton and Hall textbook of medical physiology." (2006). 14. Lenk, C., Ekinci, A., Rangelow, I. W., & Gutschmidt, S. (2018, July). Active, artificial hair cells for biomimetic sound detection based on active cantilever technology. In 2018 40th Annual International Conference of the IEEE Engineering in Medicine and Biology Society (EMBC) (pp. 4488-4491). IEEE 55