Image sensors with AI Chips

In May 2020, Sony released the “intelligent vision sensor”, which has an embedded AI processing capacity. Since AI processing is performed offline, it is possible to reduce data transfer delays and communication costs related to cloud based computations.

The new product has a laminated structure in which a pixel chip with an image size of 1 / 2.3 inch (diagonal 7.857mm) and a logic chip are stacked. The pixel chip has approximately 12.3 million illuminated pixels. The unit cell size is 1.55 x 1.55 μm. The sensitivity (standard value F5.6) is around 250 LSB.

In addition to the circuits that process regular image sensor signals, the logic chip integrates a proprietary DSP specialized for AI processing and a memory for saving AI models. This makes it possible to realize an offline AI system without adding a high-performance processor or external memory. “The world’s first intelligent vision sensor equipped with an AI processing function,” as Sony markets it.

The amount of data taken by the image sensor can be reduced by performing ISP processing and AI processing on the logic chip side and outputting the object as metadata. We believe that not outputting image information will also protect privacy. As the output data format, it is possible to select a non-processed captured image, an image in the ISP output format (YUV / RGB), a ROI image in which a specific area is cut out, et cetera.

High-speed AI processing has made it possible to track an object in real time. The new product performs ISP processing and high-speed AI processing with the logic chip (processing time is 3.1 msec for AI model “MobileNet V1” used in image analysis for mobile devices), so all processing can be performed within one frame of video. 

The user can rewrite the AI ​​model saved in the built-in memory with an arbitrary AI model. This makes it possible to change the processing function according to the context. As an example, Sony  mentioned the use of cameras installed in stores. The camera facing the entrance could count visitors. If you change the AI ​​model, you can use it also as a camera facing the shelf to detect missing items or as a camera installed on the ceiling to detect heat maps of visitors.

Two types of products are available: bare chip product “IMX500” and package product “IMX501”. The IMX500 is already being shipped, and the price (excluding tax) is 10,000 yen (USD 93). The IMX501 is supplied as a ceramic LGA with external dimensions of 12.5 x 15.0 mm. Sony plans to begin shipping it from June 2020, with a price (excluding tax) of 20,000 yen (USD 186).