Sensor Technologies and Biological Sensing (Lecture 8) PDF

BIOINSPIRED SENSORS: TACTILE Lecture 8 2024. 11. 20. Dr. Sándor Földi Sensor technologies and biological sensing  CONTENTS 1. Somatic sensation physiology 2. Biomimetic somatic/tactile sensors 3  RECAP – SOMATIC SENSATION Sensa...

BIOINSPIRED SENSORS: TACTILE Lecture 8 2024. 11. 20. Dr. Sándor Földi Sensor technologies and biological sensing  CONTENTS 1. Somatic sensation physiology 2. Biomimetic somatic/tactile sensors 3  RECAP – SOMATIC SENSATION Sensation from the skin, muscles, bones, tendons and joints is termed somatic sensation Variety of specialized somatic receptors: 4 Reference: Widmaier, Eric P., Hershel Raff, and Kevin T. Strang. Human physiology: The mechanisms of body function. McGrow-Hill, 2006.  SOMATIC SENSATION – MECHANORECEPTORS We will focus on mechanical sensation Human mechanoreceptors 5 Reference: Yi, Zhengkun, Yilei Zhang, and Jan Peters. "Biomimetic tactile sensors and signal processing with spike trains: A review." Sensors and Actuators A: Physical 269 (2018): 41-52.  SOMATIC SENSATION – MECHANORECEPTORS Pacinian corpuscles: Sensitivity to deep-pressure touch and high- frequency vibration Located beneath the bottom layer of the skin Fast adaptive receptors Oval shaped, ~1 mm length, ~0.6 mm diameter Meissner’s corpuscles: Sensitivity to light touch Located in the top layer of the skin Fast adaptive receptors Between 80 and 150 μm in length and between 20 and 40 μm in diameter 6 Reference: Widmaier, Eric P., Hershel Raff, and Kevin T. Strang. Human physiology: The mechanisms of body function. McGrow-Hill, 2006.  SOMATIC SENSATION – MECHANORECEPTORS Merkel discs: Information regarding pressure, vibration and texture Most sensitive of the four mechanoreceptors at low frequencies, at around 15 Hz Slowly adaptive receptors Ruffini cylinders: Sensitive to lateral skin stretch and contribute to the kinesthetic sense and control of finger position and movement They register mechanical deformation within joints, angle specificity of up to 2° Monitoring slippage of objects 7 Reference: Yi, Zhengkun, Yilei Zhang, and Jan Peters. "Biomimetic tactile sensors and signal processing with spike trains: A review." Sensors and Actuators A: Physical 269 (2018): 41-52.  HUMAN TACTILE STIMULUS Reference: Carey, L. M., Mak-Yuen, Y. Y., & Matyas, T. A. (2020). The functional Tactile Object Recognition Test: A unidimensional measure with excellent internal consistency for haptic 8 sensing of real objects after stroke. Frontiers in neuroscience, 14, 542590.  HUMAN TACTILE STIMULUS Shape perception Main contributors: Merkel discs (SA-I) and Meissner’s corpuscles (FA-I) Multiple receptors are required for shape recognition SA-I mechanoreceptors offer more information in terms of spatial features of a shape FA-I mechanoreceptors play an important role in the discrimination of fine differences in sharpness 9 Reference: https://i0.wp.com/sensoryproject.org/wp-content/uploads/2019/07/Screen-Shot-2019-07-22-at-08.55.32-e1629729108919.png?fit=1501%2C891&ssl=1  HUMAN TACTILE STIMULUS Fingertip force perception Humans can estimate the magnitude of the contact force regardless of the shape of the objects The force direction is recognized primarily during the dynamic force stimulation Static force stimulation improves the discrimination ability only to a limited extent Limit: 7.1° in 3D force direction can be discriminated Tactile afferents potentially contribute to the encoding of the fingertip forces Each type of afferent fires in response to the force stimuli from a preferred direction 10  HUMAN TACTILE STIMULUS Slip detection The direction of the evoked skin stretch can be perceived by SA-I afferent, which indicates the direction of impending slip. The slip detection during the interaction with both the single dot plate and the textured plate can be achieved due to FA afferents. FA-I and FA-II afferent provide a reliable spatiotemporal code. FA-I, FA-II and SA-I afferents can reliably signal slips with different sensitivity. Reference: Zangrandi, A., D’Alonzo, M., Cipriani, C., & Di Pino, G. (2021). Neurophysiology of slip sensation and grip reaction: insights for hand prosthesis control of slippage. Journal of 11 neurophysiology, 126(2), 477-492.  HUMAN TACTILE STIMULUS Hardness perception Tactile information is sufficient to discriminate the rubber specimen independent of the velocities and forces of specimen application For the spring cells, both the tactile and kinesthetic information contributed to the ability of hardness discrimination SA-I afferent population might play a vital role in encoding the hardness of the objects The mean firing rate of SA-I afferents increases steeply as the hardness of the objects increases. Moreover, the FA-I afferents encodes the hardness of objects as well despite the significantly lower sensitivity compared to SA-I afferents. On the contrary, the SA-II afferent plays a relatively minor role by affecting the object hardness solely in certain conditions. The hardness classification depends on whether the object conforms to the body. More specifically, an object is considered as soft if it conforms to the body, while an object is considered as hard if the body conforms to the object. 12  HUMAN TACTILE STIMULUS Texture perception The neural coding of tactile texture is closely related to the spatial variations in slowly adapting (SA) mechanoreceptive afferents. There is a consistent relationship between human judgements of fine surface roughness and SA-I spatial variations. However, a recent study showed that the tactile roughness discrimination relies mainly on vibration sensitive afferents. The FA-I mechanoreceptors and the FA-II mechanoreceptors are generally regarded to be responsible for modulating the vibratory stimuli occurring at the interface of explored surfaces and the fingers. Unlike previous work on artificial textured surfaces, Manfredi et al. focused on the natural scenes of texture perception. There are 55 textures explored in total, including a variety of everyday fabrics such as foam, vinyl, and leather. The authors suggested that characterizing the skin vibrations is the first step to understand the mechanisms underlying texture perception. 13  MECHANOSENSORY INFORMATION TRANSMISSION 14 Reference Yi, Zhengkun, Yilei Zhang, and Jan Peters. "Biomimetic tactile sensors and signal processing with spike trains: A review." Sensors and Actuators A: Physical 269 (2018): 41-52.  TACTILE TRANSDUCTION MECHANISMS Piezoresistive sensors The resistance of piezoresistive tactile sensors will change when it experiences an applied force. Easy and direct transduction mechanism from pressure to resistance change. The piezoresistive sensors can mainly fall into four categories: strain gauges, piezoresistors, percolative composites, and composites. Strain gauges: Strain is the amount of deformation of body due to an applied force. The resistance change is mostly due to geometry change. Electrical resistances variey nonlinearly with the amount of strain. Temperature sensitivity → Weathstone bridge configurations. The mechanical property of strain gauges are considered to be comparable to that of slow- adapting mechanoreceptors. 15 Reference: Stassi, S., Cauda, V., Canavese, G., & Pirri, C. F. (2014). Flexible tactile sensing based on piezoresistive composites: A review. Sensors, 14(3), 5296-5332.  TACTILE TRANSDUCTION MECHANISMS Piezoresistive sensors Piezoresistors: The working principle is based on the band theory of solids. The resistance change of piezoresistors is derived from a change in the band structure of a material, which results in a variation in the resistivity. Materials: silicon and germanium 16 Reference: Stassi, S., Cauda, V., Canavese, G., & Pirri, C. F. (2014). Flexible tactile sensing based on piezoresistive composites: A review. Sensors, 14(3), 5296-5332.  TACTILE TRANSDUCTION MECHANISMS Piezoresistive sensors Percolative composites: Percolative composites are formed by dispersed conductive fillers such as carbon black, graphite and carbon nanotubes within an insulating matrix. The conductivity of percolative composites is altered when a pressure is applied. If the density of conductive fillers are high, the conductivity will be greater, and the lower density results in lower conductivity. The relationship between the applied force and conductivity is nonlinear. Limitations: hysteresis, response with a long latency. 17 Reference: Stassi, S., Cauda, V., Canavese, G., & Pirri, C. F. (2014). Flexible tactile sensing based on piezoresistive composites: A review. Sensors, 14(3), 5296-5332.  TACTILE TRANSDUCTION MECHANISMS Piezoresistive sensors Quantum tunneling composites: They are composed of conductive fillers and insulating barrieres. They are perfect insulators in the normal state, while they are more or less perfect conductors when external force is applied. Principle: fillers can tunnel through the insulating barriers and form a conducting path, which makes it different from percolative composites Limits: similar to percolative composites. 18 Reference: Stassi, S., Cauda, V., Canavese, G., & Pirri, C. F. (2014). Flexible tactile sensing based on piezoresistive composites: A review. Sensors, 14(3), 5296-5332.  TACTILE TRANSDUCTION MECHANISMS Piezoelectric sensors Piezoelectric materials produce electrical charge when they are deformed or subjected to mechanical force and correspondingly a voltage is developed if the surfaces are electroded. Piezoelectric sensors particularly suitable to measure dynamic stimuli such as vibration. Limitation: high sensitivity to temperature change, inapplicable for static stimuli measurement. Piezoelectric sensors might measure very low frequency stimuli on the condition that they act as both an actuator and sensor. Polyvinylidene fluoride (PVDF) is a classic example of piezoelectric polymer that is preferred in tactile sensors. The mechanical property of PVDF films are considered to be comparable to that of fast-adapting mechanoreceptors. In comparison to PVDF sensors, lead zirconate titanate (PZT) offers an advantage of a relatively high piezoelectricity. However, PZT is not flexible and difficult to be formed on the curved surfaces. Both PVDF and PZT have been incorporated into robotic hand for slip detection. 19  TACTILE TRANSDUCTION MECHANISMS Capacitive sensors A capacitive sensor is comprised of two conductive plates separated by a dielectric material between them. An applied force can produce change either in the distance between the plates or its area, and hence the capacitance. Capacitive tactile sensors exhibit high spatial resolution, good frequency response, and a relatively broader measurement range. Limitations: the capacitive tactile sensors are easy to be affected by noise, and it is especially evident when the sensors are in array configurations. Therefore, the particular filter circuits often come with capacitive tactile sensors to eliminate this noise. Microelectromechanical systems (MEMS) technology has also been used to develop capacitive tactile sensors. 20  TACTILE TRANSDUCTION MECHANISMS Optical sensors Optical tactile sensors modulate the tactile stimuli in the physical quantities such as light intensity. The optical tactile system is generally comprised of a light source, an optical detector, and an optical waveguide. Optical tactile sensors are immune to electromagnetic interference (EML) and have a wide sensing range. Limitations: bulky sizes. I.e.: the optical sensor called the TacTip is able to localization resolution of 0.1 mm, which is much smaller than resolution between sensing elements. 21  TACTILE TRANSDUCTION MECHANISMS Magnetic sensors Magnetic tactile sensors modulate the tactile stimuli in the magnetic mode. For example, based on the principle of electromagnetic induction, a tactile magnetic sensor including a permanent magnet and a two- dimensional array of inductors. The displacement of the permanent magnet due to the applied external force generated an induction voltage for each inductor. Magnetic tactile sensors commonly have a high dynamic range. Limitations: bulky size like the optical tactile sensors. The magnetic sensor can be able to feel texture and measure fluid flow with a low power consumption. Magnetic tactile sensors are capable of detecting slip as well. Reference: Goka, M., Nakamoto, H., & Takenawa, S. (2010, October). A magnetic type tactile sensor by GMR elements and inductors. In 2010 IEEE/RSJ International Conference on 22 Intelligent Robots and Systems (pp. 885-890). IEEE.  TACTILE TRANSDUCTION MECHANISMS Magnetic sensors 23 Reference: Alfadhel, A., & Kosel, J. (2015). Magnetic nanocomposite cilia tactile sensor. Advanced Materials, 27(47), 7888-7892.  TACTILE SENSING – DEEP LEARNING Not only the sensor is biomimetic, but also the signal processing Rising popularity in application of deep neural networks Pattern recognition in different spike trains, measured signals Combination of sensors and neural network capabilities can lead to a human like sensation description It also has importance in prosthetics 24  BIOMIMETIC TACTILE SENSING TacTip: Soft optical tactile sensors with 3D-printed biomimetic morphologies Reference: Lepora, N. F., Church, A., De Kerckhove, C., Hadsell, R., & Lloyd, J. (2019). From pixels to percepts: Highly robust edge perception and contour following using deep learning and an optical biomimetic tactile sensor. IEEE Robotics and Automation Letters, 4(2), 2101-2107.; Ward-Cherrier, B., Pestell, N., Cramphorn, L., Winstone, B., Giannaccini, M. E., Rossiter, J., & Lepora, N. F. (2018). The tactip family: Soft optical tactile sensors with 3d-printed biomimetic morphologies. Soft 25 robotics, 5(2), 216-227.  BIOMIMETIC TACTILE SENSING TacTip: Soft optical tactile sensors with 3D-printed biomimetic morphologies Reference: Ward-Cherrier, B., Pestell, N., Cramphorn, L., Winstone, B., Giannaccini, M. E., Rossiter, J., & Lepora, N. F. (2018). The tactip family: Soft optical tactile sensors with 3d-printed 26  BIOMIMETIC TACTILE SENSING TacTip: Tapping around the object Reference: Lepora, N. F., Church, A., De Kerckhove, C., Hadsell, R., & Lloyd, J. (2019). From pixels to percepts: Highly robust edge perception and contour following using 27 deep learning and an optical biomimetic tactile sensor. IEEE Robotics and Automation Letters, 4(2), 2101-2107.  BIOMIMETIC TACTILE SENSING TacTip: Different constructions, applications Reference: Ward-Cherrier, B., Pestell, N., Cramphorn, L., Winstone, B., Giannaccini, M. E., Rossiter, J., & Lepora, N. F. (2018). The tactip family: Soft optical tactile sensors with 3d-printed 28 biomimetic morphologies. Soft robotics, 5(2), 216-227.  BIOMIMETIC TACTILE SENSING Digit: Gelsight surface based tactile sensor by META Reference: Lambeta, M., Chou, P. W., Tian, S., Yang, B., Maloon, B., Most, V. R.,... & Calandra, R. (2020). Digit: A novel design for a low-cost compact high-resolution tactile sensor with 29 application to in-hand manipulation. IEEE Robotics and Automation Letters, 5(3), 3838-3845.  BIOMIMETIC TACTILE SENSING Digit: Reference: Lambeta, M., Chou, P. W., Tian, S., Yang, B., Maloon, B., Most, V. R.,... & Calandra, R. (2020). Digit: A novel design for a low-cost compact high-resolution tactile sensor with 30 application to in-hand manipulation. IEEE Robotics and Automation Letters, 5(3), 3838-3845.  BIOMIMETIC TACTILE SENSING Digit sensor: Advantage: Available online for free Low manufacturing costs High-resolution tactile readings 60 fps Multiple purposes Limitations: Size Image quality Relatively slow for some applicaiton areas More qulaitative information than quantitative Realtively high computational requirements Reference: Lambeta, M., Chou, P. W., Tian, S., Yang, B., Maloon, B., Most, V. R.,... & Calandra, R. (2020). Digit: A novel design for a low-cost compact high-resolution tactile sensor with 31 application to in-hand manipulation. IEEE Robotics and Automation Letters, 5(3), 3838-3845.  BIOMIMETIC TACTILE SENSING DigiTac: Combination of the Digit and the TacTip sensor Reference: Lepora, N. F., Lin, Y., Money-Coomes, B., & Lloyd, J. (2022). Digitac: A digit-tactip hybrid tactile sensor for comparing low-cost high-resolution robot touch. IEEE Robotics and 32 Automation Letters, 7(4), 9382-9388.  BIOMIMETIC TACTILE SENSING OptoForce 3D tactile sensor: Developed at our faculty 3D sensing of the force vector affecting its dome 3 layers of silicone Light sensing elements (photodiodes) 33  BIOMIMETIC TACTILE SENSING OptoForce 3D tactile sensor: Lower detection limit: ~0.1 N Low energy consumption Fast sampling, the sampling frequency can be in MHz region Sensitive and robust Can only measure only one attacking point Relatively large size 34  BIOMIMETIC TACTILE SENSING OptoForce 3D tactile sensor 35  BIOMIMETIC TACTILE SENSING OptoForce 3D tactile sensor 36  BIOMIMETIC TACTILE SENSING OptoForce 3D tactile sensor 37  BIOMIMETIC TACTILE SENSING BioTac sensor 38 Reference: Wettels, Nicholas, Jeremy A. Fishel, and Gerald E. Loeb. "Multimodal tactile sensor." The Human Hand as an Inspiration for Robot Hand Development. Springer, Cham, 2014. 405-429.  BIOMIMETIC TACTILE SENSING Ionic Tactile Sensors (ITS): biomimetic skin, human like sensing Reference: Amoli, V., Kim, J. S., Kim, S. Y., Koo, J., Chung, Y. S., Choi, H., & Kim, D. H. (2020). Ionic tactile sensors for emerging human‐interactive technologies: a review of recent progress. 39 Advanced Functional Materials, 30(20), 1904532.  ITS – SENSING MECHANISM Supercapacitive ITS: a) Ionic microdroplet sensing device b) Ion gel film sensing device c) Sensor with nanofibrous iontronic layer Reference: Amoli, V., Kim, J. S., Kim, S. Y., Koo, J., Chung, Y. S., Choi, H., & Kim, D. H. (2020). Ionic tactile sensors for emerging human‐interactive technologies: a review of recent progress. 40 Advanced Functional Materials, 30(20), 1904532.  IONIC LIQUID-BASED ITS Reference: Amoli, V., Kim, J. S., Kim, S. Y., Koo, J., Chung, Y. S., Choi, H., & Kim, D. H. (2020). Ionic tactile sensors for emerging human‐interactive technologies: a review of recent progress. 41 Advanced Functional Materials, 30(20), 1904532.  STRUCTURAL ENGINEERING IN ITS Reference: Amoli, V., Kim, J. S., Kim, S. Y., Koo, J., Chung, Y. S., Choi, H., & Kim, D. H. (2020). Ionic tactile sensors for emerging human‐interactive technologies: a review of recent progress. 42 Advanced Functional Materials, 30(20), 1904532.  SUMMARY – QUESTIONS Briefly introduce the mechanoreceptors on the human hand. What kind of tactile stimulus sensations should be handled by a biomimetic tactile sensors? Introduce one of them in details. Describe at least three tactile transduction mechanism. How does a TacTip sensor work? Compared to the OptoForce sensor, what are the similarities and differences between the two system? How does the OptoForce sensor work? How does the Digit sensor works? What is a DigiTac sensor? What are the Ionic Tactile Sensors? Describe briefly an exact example. 43  REFERENCES 1. Yi, Zhengkun, Yilei Zhang, and Jan Peters. "Biomimetic tactile sensors and signal processing with spike trains: A review." Sensors and Actuators A: Physical 269 (2018): 41-52. 2. Lepora, N. F., Church, A., De Kerckhove, C., Hadsell, R., & Lloyd, J. (2019). From pixels to percepts: Highly robust edge perception and contour following using deep learning and an optical biomimetic tactile sensor. IEEE Robotics and Automation Letters, 4(2), 2101-2107. 7. Ward-Cherrier, B., Pestell, N., Cramphorn, L., Winstone, B., Giannaccini, M. E., Rossiter, J., & Lepora, N. F. (2018). The tactip family: Soft optical tactile sensors with 3d-printed biomimetic morphologies. Soft robotics, 5(2), 216-227. 8. Amoli, V., Kim, J. S., Kim, S. Y., Koo, J., Chung, Y. S., Choi, H., & Kim, D. H. (2020). Ionic tactile sensors for emerging human‐interactive technologies: a review of recent progress. Advanced Functional Materials, 30(20), 1904532. 9. Stassi, S., Cauda, V., Canavese, G., & Pirri, C. F. (2014). Flexible tactile sensing based on piezoresistive composites: A review. Sensors, 14(3), 5296-5332. 10. Goka, M., Nakamoto, H., & Takenawa, S. (2010, October). A magnetic type tactile sensor by GMR elements and inductors. In 2010 IEEE/RSJ International Conference on Intelligent Robots and Systems (pp. 885-890). IEEE. 11. Wettels, Nicholas, Jeremy A. Fishel, and Gerald E. Loeb. "Multimodal tactile sensor." The Human Hand as an Inspiration for Robot Hand Development. Springer, Cham, 2014. 405-429. 12. Carey, L. M., Mak-Yuen, Y. Y., & Matyas, T. A. (2020). The functional Tactile Object Recognition Test: A unidimensional measure with excellent internal consistency for haptic sensing of real objects after stroke. Frontiers in neuroscience, 14, 542590. 13. Lepora, N. F., Lin, Y., Money-Coomes, B., & Lloyd, J. (2022). Digitac: A digit-tactip hybrid tactile sensor for comparing low-cost high-resolution robot touch. IEEE Robotics and Automation Letters, 7(4), 9382-9388. 14. Lambeta, M., Chou, P. W., Tian, S., Yang, B., Maloon, B., Most, V. R.,... & Calandra, R. (2020). Digit: A novel design for a low-cost compact high-resolution tactile sensor with application to in-hand manipulation. IEEE Robotics and Automation Letters, 5(3), 3838-3845. 44