Demosaicing in ISP Pipeline Demosaicing PDF

Jack Yu 5/30/24 Demosaicing in ISP Pipeline Introduction Demosaicing is the process of turning raw light sensor data into a color image. In digital cameras, light color filter patterns are put in front of the light sensors such that each light sensor can only measure either red, green, or blue lig...

Jack Yu 5/30/24 Demosaicing in ISP Pipeline Introduction Demosaicing is the process of turning raw light sensor data into a color image. In digital cameras, light color filter patterns are put in front of the light sensors such that each light sensor can only measure either red, green, or blue light. Since you can only measure light of one of the three colors, the process of mosaicing is used to fill in missing pixel color data from neighboring light sensor readings. Goals - Create full rgb pixel image from bayer domain data values - Use interpolation algorithm to predict missing color data value - Deal with edge padding of image Design In essence, the hardware implementation deals with a 3x3 kernel that processes the data for one rgb pixel in the resulting image. The bayer color filter is used. So, the color data is being received in a 2-dimension array as seen below. In the kernel, the center pixel is the position that will be processed for the resulting rgb values, using the surrounding data values to fill in the other two color data for the pixel.  Pos 1 Pos 2 Pos 3 Pos 4 Pixel Pos 6 Pos 7 Pos 8 Pos 8 Next Kernel Position: Pos 1 Pos 2 Pos 3 Pos 4 Pixel Pos 6 Pos 7 Pos 8 Pos 9 Each position in the kernel is fed into the hardware implementation as a stream of values. Next, as we receive a 3x3 kernel of data, we have to process this color data to calculate the rgb value for this particular pixel. Since the bayer color filter is being utilized, the color of the center pixel can be easily derived. In the raw image, the index number of a red light sensor data occurs in only both odd rows and columns. Blue light data occur on only both even rows and columns. Everywhere else is green light color data. There is higher concentration of green light data since it is the most prevalent color wavelength that humans perceive. Then to calculate each of the color values we use interpolation.  0.25 0.5 0.25 0.5 1 0.5 0.25 0.5 0.25 B2 GB 3 B2 GR 1 R4 GR1 B2 GB 3 B2 In this kernel case, since Red is in the pixel position, its value will represent the value for red. Red : r_value * 1 = 4 * 1 = 4 Green: (gr_value(position 4) * 0.25) + (gr_value(position 6) * 0.25) + (gb_value(position 2) * 0.25) + (gb_value(position 8) * 0.25) = (1 * 0.25) + (1 * 0.25) + (3 * 0.25) + (3 * 0.25) = 2 Blue: (b_value(position 1) * 0.25) + (b_value(position 3) * 0.25) + (b_value(position 7) * 0.25) + (b_value(position 9) * 0.25) = (1 * 0.25) + (1 * 0.25) + (3 * 0.25) + (3 * 0.25) = 2 Final RGB pixel value tuple: (4, 2, 2) Implementation Core.cpp ____________________________________________________________________________ #include "core.h" void core(hls::stream &redOutStream, hls::stream &greenOutStream, hls::stream &blueOutStream, hls::stream &pixel_1, hls::stream &pixel_2, hls::stream &pixel_3, hls::stream &pixel_4, hls::stream &pixel_5, hls::stream &pixel_6, hls::stream &pixel_7, hls::stream &pixel_8, hls::stream &pixel_9, hls::stream &width, hls::stream &height) { #pragma HLS INTERFACE axis part=redOutStream #pragma HLS INTERFACE axis part=greenOutStream #pragma HLS INTERFACE axis part=blueOutStream #pragma HLS INTERFACE ap_none port=return #pragma HLS INTERFACE axis part=pixel_1 #pragma HLS INTERFACE axis part=pixel_2 #pragma HLS INTERFACE axis part=pixel_3 #pragma HLS INTERFACE axis part=pixel_4 #pragma HLS INTERFACE axis part=pixel_5 #pragma HLS INTERFACE axis part=pixel_6 #pragma HLS INTERFACE axis part=pixel_7 #pragma HLS INTERFACE axis part=pixel_8 #pragma HLS INTERFACE axis part=pixel_9 #pragma HLS INTERFACE axis part=width #pragma HLS INTERFACE axis part=height for(int idxPixel = 0; idxPixel < (4*4); ++idxPixel) { #pragma HLS PIPELINE ap_uint _pixel_1 = pixel_1.read(); ap_uint _pixel_2 = pixel_2.read(); ap_uint _pixel_3 = pixel_3.read(); ap_uint _pixel_4 = pixel_4.read(); ap_uint _pixel_5 = pixel_5.read(); ap_uint _pixel_6 = pixel_6.read(); ap_uint _pixel_7 = pixel_7.read(); ap_uint _pixel_8 = pixel_8.read(); ap_uint _pixel_9 = pixel_9.read(); ap_uint _width = width.read(); ap_uint _height = height.read(); float f_pixel_1 = float(_pixel_1); float f_pixel_2 = float(_pixel_2); float f_pixel_3 = float(_pixel_3); float f_pixel_4 = float(_pixel_4); float f_pixel_5 = float(_pixel_5); float f_pixel_6 = float(_pixel_6); float f_pixel_7 = float(_pixel_7); float f_pixel_8 = float(_pixel_8); float f_pixel_9 = float(_pixel_9); int i_width = int(_width); int i_height = int(_height); float f_red_sum; float f_green_sum; float f_blue_sum; if ((idxPixel % (2*_width)) > input[i][j]; } } printf("Load Image: "); printf("\n"); for (int idxRows = 0; idxRows < size; ++idxRows) { for (int idxCols = 0; idxCols < size; ++idxCols) { printf("%d ", input[idxRows][idxCols]); } printf("\n"); } printf("\n"); hls::stream redOutputStream; hls::stream blueOutputStream; hls::stream greenOutputStream; hls::stream p1; hls::stream p2; hls::stream p3; hls::stream p4; hls::stream p5; hls::stream p6; hls::stream p7; hls::stream p8; hls::stream p9; hls::stream width; hls::stream height; for (int idxRows = 0; idxRows < size; ++idxRows) { for (int idxCols = 0; idxCols < size; ++idxCols) { ap_uint pix1; ap_uint pix2; ap_uint pix3; ap_uint pix4; ap_uint pix5; ap_uint pix6; ap_uint pix7; ap_uint pix8; ap_uint pix9; ap_uint w; ap_uint h; if (idxRows == 0 and idxCols == 0){ pix1 = input[idxRows + 1][idxCols + 1]; pix2 = input[idxRows + 1][idxCols]; pix3 = input[idxRows + 1][idxCols + 1]; pix4 = input[idxRows][idxCols + 1]; pix5 = input[idxRows][idxCols]; pix6 = input[idxRows][idxCols + 1]; pix7 = input[idxRows + 1][idxCols + 1]; pix8 = input[idxRows + 1][idxCols]; pix9 = input[idxRows + 1][idxCols + 1]; } else if (idxRows == 0 and idxCols == size - 1){ pix1 = input[idxRows + 1][idxCols - 1]; pix2 = input[idxRows + 1][idxCols]; pix3 = input[idxRows + 1][idxCols - 1]; pix4 = input[idxRows][idxCols - 1]; pix5 = input[idxRows][idxCols]; pix6 = input[idxRows][idxCols - 1]; pix7 = input[idxRows + 1][idxCols - 1]; pix8 = input[idxRows + 1][idxCols]; pix9 = input[idxRows + 1][idxCols - 1]; } else if (idxRows == size - 1 and idxCols == 0){ pix1 = input[idxRows - 1][idxCols + 1]; pix2 = input[idxRows - 1][idxCols]; pix3 = input[idxRows - 1][idxCols + 1]; pix4 = input[idxRows][idxCols + 1]; pix5 = input[idxRows][idxCols]; pix6 = input[idxRows][idxCols + 1]; pix7 = input[idxRows - 1][idxCols + 1]; pix8 = input[idxRows - 1][idxCols]; pix9 = input[idxRows - 1][idxCols + 1]; } else if (idxRows == size - 1 and idxCols == size - 1){ pix1 = input[idxRows - 1][idxCols - 1]; pix2 = input[idxRows - 1][idxCols]; pix3 = input[idxRows - 1][idxCols - 1]; pix4 = input[idxRows][idxCols - 1]; pix5 = input[idxRows][idxCols]; pix6 = input[idxRows][idxCols - 1]; pix7 = input[idxRows - 1][idxCols - 1]; pix8 = input[idxRows - 1][idxCols]; pix9 = input[idxRows - 1][idxCols - 1]; } else if (idxRows == 0){ pix1 = input[idxRows + 1][idxCols - 1]; pix2 = input[idxRows + 1][idxCols]; pix3 = input[idxRows + 1][idxCols + 1]; pix4 = input[idxRows][idxCols - 1]; pix5 = input[idxRows][idxCols]; pix6 = input[idxRows][idxCols + 1]; pix7 = input[idxRows + 1][idxCols - 1]; pix8 = input[idxRows + 1][idxCols]; pix9 = input[idxRows + 1][idxCols + 1]; } else if (idxRows == size - 1){ pix1 = input[idxRows - 1][idxCols - 1]; pix2 = input[idxRows - 1][idxCols]; pix3 = input[idxRows - 1][idxCols + 1]; pix4 = input[idxRows][idxCols - 1]; pix5 = input[idxRows][idxCols]; pix6 = input[idxRows][idxCols + 1]; pix7 = input[idxRows - 1][idxCols - 1]; pix8 = input[idxRows - 1][idxCols]; pix9 = input[idxRows - 1][idxCols + 1]; } else if (idxCols == 0){ pix1 = input[idxRows - 1][idxCols + 1]; pix2 = input[idxRows - 1][idxCols];  pix3 = input[idxRows - 1][idxCols + 1]; pix4 = input[idxRows][idxCols + 1]; pix5 = input[idxRows][idxCols]; pix6 = input[idxRows][idxCols + 1]; pix7 = input[idxRows + 1][idxCols + 1]; pix8 = input[idxRows + 1][idxCols]; pix9 = input[idxRows + 1][idxCols + 1]; } else if (idxCols == size - 1){ pix1 = input[idxRows - 1][idxCols - 1]; pix2 = input[idxRows - 1][idxCols]; pix3 = input[idxRows - 1][idxCols - 1]; pix4 = input[idxRows][idxCols - 1]; pix5 = input[idxRows][idxCols]; pix6 = input[idxRows][idxCols - 1]; pix7 = input[idxRows + 1][idxCols - 1]; pix8 = input[idxRows + 1][idxCols]; pix9 = input[idxRows + 1][idxCols - 1]; } else { pix1 = input[idxRows - 1][idxCols - 1]; pix2 = input[idxRows - 1][idxCols]; pix3 = input[idxRows - 1][idxCols + 1]; pix4 = input[idxRows][idxCols - 1]; pix5 = input[idxRows][idxCols]; pix6 = input[idxRows][idxCols + 1]; pix7 = input[idxRows + 1][idxCols - 1]; pix8 = input[idxRows + 1][idxCols]; pix9 = input[idxRows + 1][idxCols + 1]; } w = ap_uint(size); h = ap_uint(size); p1.write(pix1); p2.write(pix2); p3.write(pix3); p4.write(pix4); p5.write(pix5); p6.write(pix6); p7.write(pix7); p8.write(pix8); p9.write(pix9); width.write(w); height.write(h); } } core(redOutputStream, greenOutputStream, blueOutputStream, p1, p2, p3, p4, p5, p6, p7, p8, p9, width, height); int redOut; int greenOut; int blueOut; std::cout