This article is based on an article from the Japanese edition of Engadget and was created using the translation tool Deepl.
An app that uses LiDAR, the terrain acquisition sensor of the latest iPad Pro (2020 model), to render effects in real time on the real world is now available for the iPad.
Effectron, developed by Gunma Prefecture's Anffexion Inc., allows users to draw 3D effects in real time on terrain acquired from the iPad Pro's onboard LiDAR sensor. There are 20 different effects to choose from, and it is possible to turn your home into a grid-covered space like in the movie TRON, or turn your office hallway into a cave to explore.
This is how AR apps will evolve with LiDAR
Currently, there are other AR apps that apply effects to the real world, such as MEISAI. This app can be used on iPhone XS/XR or later without LiDAR, but how advanced are the effects by using LiDAR? We interviewed the developer of Effectron and got the following answers.
The difference between using LiDAR and MEISAI
First of all, let's talk about the technical aspects of the app. In the case of MEISAI, the ARKit3's people occlusion feature is used to separate the background from the person and apply the effect. It's like applying an effect to one background layer and one person layer separately. MEISAI doesn't capture the actual depth, but thanks to Kitasenju Design's excellent technology and visual sense, the depth of the effect can be felt (I've never actually talked to them about this, so this is within my expectations).
Our Effectron, on the other hand, actually creates a 3D mesh, so it's not a pseudo effect, but rather a 3D model of the same scale and shape that is placed in accordance with the movement of the camera when the room is captured. In a future update, we plan to add a hit detection function to the mesh generated by reading the terrain with LiDAR (a process that prevents 3D objects from slipping through the mesh even if they are placed on it), so that the character can run around and make it pile up with snow. We can't do this on-screen hit detection on real-world objects without detecting the terrain with LiDAR.
There are also many advantages to using Effectorn at this stage. We can store a large area of mesh, so if you get the interior terrain of a building and look at the building from the outside, you can see the interior structure of the building because the mesh is still intact. It might be easier to understand if you use the grid effect and darken the background.
About the future use of LiDAR
As I mentioned earlier, with the LiDAR sensor attached, the expression of pseudo-AR can be greatly improved by creating a terrain mesh. Real space can be converted to 3D in real time. It's easy to understand if you imagine Pokémon Go. In the current version of Pokémon Go, Pikachu doesn't look like he's in real space. It only detects and places them on a flat surface, so they look like they're floating on the screen. But if you use LiDAR to create a terrain mesh, you can make Pikachu run around on the ground, bump into walls, jump over steps, etc. as if he is actually in the real space.
There are also many ways to use this application outside of entertainment, such as real estate. By dividing the mesh obtained from ARKit's LiDAR into walls, ceilings, chairs, etc. using the automatic discrimination function, you can change the wallpaper and try different colors, change the wood material of the floor, and so on. I hadn't been very interested in AR until now, but with the installation of LiDAR on the iPad Pro, the range of expression and use of AR has expanded to a completely different level, which is why I created Effectron.
This article is based on an article from the Japanese edition of Engadget and was created using the translation tool Deepl. The Japanese edition of Engadget does not guarantee the accuracy or reliability of this article.