Question Time: Do you want to create an avatar of your own?

For example, like this ⬇

Or, like this ⬇

From Hatsune Miku to the popular Samsung assistant, Sam, or LVMH, the world's largest luxury goods group, creating a virtual ambassador, virtual avatars are becoming big news along with discussions of the metaverse. In the busy schedule of the v3.5 update, Cocos developers, under the "good words and persuasion" of Sister C, began to explore the field of virtual humans.

Freer and lighter, indulge in a smooth experience

Nowadays, there are more and more virtual people everywhere. They can be virtual idols, entertainment stars, Web anchors selling goods, sign language interpreters, brand ambassadors, corporate employees... Virtual people are becoming active in various industries and fields and expanding our imagination.

The popularity of the metaverse has undoubtedly brought more attention to virtual humans. Although the metaverse is currently only a vague concept, it is foreseeable that virtual humans will be one of the core elements of the future metaverse world. One of the core factors restricting its development lies in technical capabilities. The technical level requirements are much higher than you'd think.

As a mature commercial 3D engine, Cocos has accumulated underlying technical capabilities and rich practical experience in development, which can significantly reduce the time and cost for developers to understand the key technologies of virtual humans. In short - Cocos is building virtual people from five key technologies of modeling, mouth shape, motion capture, rendering, and AI that help bring more realism to virtual humans.

This is just an exploration and experiment that Cocos has done in the field of virtual humans at this stage. With the progress of engine 3D rendering technology and due to the lightweight and high-efficiency characteristics of Cocos, developers can create sufficiently realistic 3D images and achieve the best virtual human effect with lower computing requirements, covering most computing devices and operating systems.

Modeling, driving, rendering, running live in the editor

As mentioned, Cocos has realized the key technologies required to create virtual people. These 3D images support the complete human skeleton and facial rigging running in the Cocos engine in real-time.

Cocos provides 3D characters with outstanding real-time rendering performance on mobile devices. Users are now able to run Cocos 3D characters with a maximum of 300k triangle and 60fps on devices with QualcommKryo™ 585 CPU.

Point 1: Highly customizable character editing

The preset character model has as many as 263 face and body adjustment coefficients, allowing users to develop unique avatars independently. At the same time, AI modeling technology will enable users to generate real-life digital avatars from photos quickly.

Point 2: Character dynamic capture and collection technology

After the skeletal animation is built, the virtual human has unlimited potential for physical activity. With just a photo, a video, and without the need for professional motion capture actors, your avatar can have an outstanding performance for those watching.

Cocos has implemented a general-purpose camera for lower-precision motion capture. Currently, we are trying to solve the problem of motion capture occlusion positioning and 3D space relationship through an RGB-D camera. At the same time, based on AI training, the 3D spatial location of hands, feet, and other parts of the image that are not visible are complemented, and the relationship between the front and back of the torso occlusion is automatically predicted and completed.

Point 3: Facial system capture technology

Cocos implements high-precision real-time expression capture technology based on video capture. It uses a large amount of video data to train the model. Finally, it generates high-quality continuous deformation (Blendshape) weight output, which is used to drive a complete virtual character, real-time expression capture effect, and offline effect. Though not 100% accurate, it can achieve a high-precision capture.

Point 4: Lip-Drive Technology

The speech lip drive is used to recognize the phonemes in the audio, train the model through a large amount of data AI, and drive the 3D character mouth using a mesh transformation to achieve high-fidelity voice performance.

Point 5: AI skeleton binding technology

Perfect bone standards and bone retargeting are the keys to driving characters, and a rich action library can quickly improve production efficiency. Currently, Cocos is testing the visual fast binding function of human skeletons to create a fast binding between imported models and standard bones to quickly adapt to the actions in the Cocos action library and drive characters to move properly.

The Cocos engine has always adhered to the principles of a low threshold to learn, cross-platform, and being lightweight. We will continue to add new capabilities, optimize performance and user experience, and iterate underlying technologies to bring ease of use and stability to developers in various fields. The authoring tool will be available for everyone in the future so users and developers can build their virtual characters.