Can Huawei Match Apple TrueDepth?

　　Now that Samsung’s Galaxy S9 is already on the market and Huawei’s P20 announcement is looming large (the Chinese mobile giant will hold a press conference here next Tuesday), imaging experts have declared 3D sensing the new battleground for the mobile industry.

　　However, far from clear is whether either Samsung or Huawei will be able to catch up with Apple’s iPhone X.

　　Because the bar set by Apple’s TrueDepth camera is so high, Pierre Cambou, activity leader for MEMS and imaging at Yole Développement, predicts that it may take a year or longer for competitors to offer 3D sensing technologies comparable to iPhone X.

　　Unlike the megapixel race of olden days, 3D sensing will be a tougher challenge for most smartphone vendors — because a 3D camera contains myriad components that need to be aligned. It also requires competent supply chain management. Cambou called the 3D camera “a bundle of sub-devices.” He said, “Remember Apple iPhone X?  Companies involved in it include: STMicroelectronics, LG Innotek, Foxconn, ams, Lumentum.”

　　As for Samsung’s Galaxy S9, some reviewers are already calling its front-facing sensing technology “a disappointment.”

　　A CNET story earlier this month said: “An in-depth look at Samsung's new biometrics verification system — and how it stacks up against the iPhone X’s Face ID — shows it's not quite safe enough for mobile payments.”

　　This is because “Samsung's facial recognition system uses a regular camera to create a 2D map of your face, contrasted with Apple's Face ID, which creates a complex 3D scan of your facial pattern.” People were able to fool Samsung's technology on last year's Galaxy S8 by using photos. Apparently, that trick still works with the S9.

　　Leading up to Huawei’s P20 launch scheduled next week, numerous press reports are singling out “triple cameras” as the new handset’s key feature.

　　Why triple cameras?

　　Are three cameras better than two?

　　We asked Eran Briman, vice president of marketing and business development at Corephotonics, a market leader in dual-camera technologies. He told us: “Triple cameras are a brand-new territory that's just about to begin and yet to prove its value.”

　　Noting that triple cameras could come in many forms and configurations, he said they could serve different goals. These include “low light performance, zoom, depth, higher resolution, and image quality as a whole,” he explained.

　　According to Briman, triple-camera combinations could include a) a color (RGB) camera, a monochrome camera, and a third camera providing either ultra-wide field of view (FoV) or a 2x telephoto narrow FoV; b) a fisheye (ultra-wide-angle) camera, a standard wide camera and a 2x telephoto camera; or c) a standard wide-angle, a 2x telephoto and a 5x telephoto to enable super-zoom capability. He noted, “Other configurations are also possible, including various image sensors,” Briman said.

　　But here’s the thing. “All of these triple cameras can provide more accurate depth information compared to dual camera designs,” said Briman.

　　“The main reason is the fact that the baseline (distance between the cameras) would be larger, especially cameras one and three. The larger the baseline, the more accurate the depth is (but also the software has greater challenges, dealing with occlusions, synchronization and others),” Briman said.

　　He added, “Also, by combining information from three cameras, theoretically you could gain better depth information.”

　　In other words, Huawei’s triple cameras appear to illustrate the company’s effort to enhance depth-sensing technology. While no confirmation is available, Huawei’s suspected 3D sensing partner is Qualcomm. Last summer, Qualcomm announced that it’s been working with Himax Technologies to make a fully integrated structured light module (SliM). The San Diego giant said then that its 3D depth-sensing camera module was slated for mass production in the first quarter of this year.

　　Who’s who in 3D sensing

　　If smartphone vendors are indeed so hot for depth sensing, who are their key technology providers?

　　Before naming names, here’s a recap on how Apple’s 3D camera works.

　　As Yole previously explained, in enabling the front of the iPhone X to identify its owner’s face and unlock the phone, Apple combined a ToF proximity detector with an infrared “structured light” camera that can either use uniform “flood” or “dot-pattern” illumination.

　　First, the iPhone X combines an infrared camera with a flood illuminator that projects uniform infrared light forward. It then takes images, which, in turn, trigger a face-detection algorithm.

　　This function, however, isn’t meant to run all the time. The infrared camera linked to the ToF proximity sensor signals the camera to take a picture when it detects a face. The iPhone X then activates its dot pattern projector to take an image.

　　Both the regular and dot-pattern images then go to the application processing unit (APU), which puts them through a neural network trained to recognize the owner and unlock the phone.

　　Yole’s Cambou noted that no 3D image is computed at this point. The 3D information is contained in the dot-pattern image. “To run 3D applications, the same APU can use another algorithm [that] computes the depth map of the image,” he said.

　　He added, “The iPhone X takes advantage of the massive processing power available in the A11 chip, as structured light approaches are known to be computationally intensive. The use of a neural network is the key technology that made it possible.”

　　Basically, there are three different types of technologies that make 3D sensing possible: stereo, structured light and time of flight (ToF) sensing. Yole put together a table as follows, illustrating strengths and limitations of each of those depth-sensing technologies.

　　Since each 3D camera consists of a host of sub-devices, here’s a rundown Yole shared with us.

　　First, stereo or active stereo is realized through the combination of the followings:  two global shutter NIR cameras offered by Omnivision, STMicroelectronics; NIR Illumination (optional) supplied by Osram, Lumentum, Finisar, ams; and 3D hardware accelerator (vision processor) provided by Iniutive, Intel Movidius.

　　Second, for structured light, necessary sub-devices include: 1 global shutter NIR cameras, offered by Omnivision, STMicroelectronics; structured NIR illumination (optional), provided by ams, Himax, Namuga, Goertek; and 3D reconstruction software. Such software is developed by companies like Apple Primesense, Mantis, Namuga and Orbbec.

　　Third, the ToF category needs a ToF camera, supplied by PMD, Sony/Softkinetic, and NIR illumination (optional) from companies such as Osram, Lumentum, Finisar, ams.

　　Cambou noted both ams and STMicroelectronics offer ToF proximity detectors, but based on only a few pixels. Neither company has developed a ToF camera yet, he said.

　　Hurdles for Apple competitors

　　Assuming that iPhone X’s TrueDepth is as far ahead of everybody else as Yole says, what exactly is missing from other 3D-sending implementations?

　　Cambou believes that the “neural engine” is one of the key enablers and that’s also a key hurdle for Apple’s competitors. “Qualcomm apparently has rushed a ‘me too’ solution but... nothing happened yet,” he noted.

　　Then, “You need the complete 2D-3D camera system, that fits in the front side of a smartphone, below $15 price point,” he said. This is not impossible but it’s really hard, he said.

　　Further, “This complete system must achieve a certain degree of biometric identification performance,” Cambou added. “If a picture is enough to fool the system, then you don't have a solution.”

　　He observed, “I think that is where the problem lies. Apple knew from the start what it wanted a 3D camera for. It worked on user interface including a biometric solution, while competition was trying to sell IKEA furniture on the other side of the phone.”

　　Cambou added, “The level of performance needed for the Apple identification system has set the standard way above competitions’ know-how. Struggles at Himax are an indication that the application was not well understood by the OEMs.”

　　Beyond Qualcomm-Himax collaborations, are there other examples?

　　Cambou noted that Mediatek is in the 3D-sensing game as an APU provider. Mediatek is striving to provide CNN — somewhat similar to Apple’s neural engine — for biometric check. Apparently, Mediatek would provide a CNN accelerator on future solution for Xiaomi, and it is integrating an Orbbec-designed 3D camera, Cambou added.

　　Wild card

　　We can all debate forever which 3D camera modules are more secure, if the identification system is to be used for mobile payments. But one thing we shouldn’t forget, said Cambou, is the China factor.

　　In the end, which 3D sensing is good enough for China’s Alipay, for example, will be decided by China. China’s vast population already depending on tools like WeChat and Alipay makes China the default decider of the next-generation technology.