Preface to Tactile Robotics: Immersion and Presence

Last week, I attended a lecture by Dr. Qi LU from MiraclePlus, where he touched upon the topic of "existential experience" within the metaverse. Dr. Qi described the infrastructure of the metaverse, including virtual worlds and virtual beings, as "3D existential embodiments," and the business environments of the metaverse in both consumer (C-end) and business (B-end) sectors as "existential e-commerce" and "existential marketing."

Transitioning from "immersion" to "existence," as metaverse technologies are expected to increasingly envelop our living environments, I find this assessment to be particularly insightful. The underlying intention is to enhance the authenticity of the metaverse experience.

When we introduce the concept of "existence" to robotics, I can't help but deduce that it seems to be a natural extension of the idea. Robots are not only present in the metaverse; they coexist with us in reality, with a physicality that transcends the digital realm. They truly see us, touch us, and interact with us in various capacities, from manufacturing in our factories to performing surgeries in our hospitals to crafting a cup of coffee in a café.

We seem to be on the verge of answering the question: as robots take over our senses of sight, hearing, smell, taste, and touch, what constitutes their "sense of existence"? Is it their physicality? Their materiality? Or their interaction with the physical world?

However, Buddha said: "All things are empty of inherent existence."

In other words, there is no such thing as a robot; a robot is merely a phenomenon that emerges when various conditions come together. When these conditions are met, the robot appears; when they dissipate, the robot vanishes.

Karl Marx stated: "A person is a sum of their social relations."

There is no fixed and unchanging entity called a robot. We can simply understand this to mean that the statement negates the physicality of robots.

Like you, I detest circuitous wordplay that, in an attempt to form conceptual consistency, almost strangles all freewheeling imagination.

---

I've been entangled in this conundrum for a few days, until the moment of writing, when I suddenly felt relieved.

The Buddhist concept of emptiness acknowledges the existence of phenomena but not their essential substance, authenticity, or permanence. All things are in constant flux under the influence of dependent origination. In my system of thought, the principle of dependent origination and emptiness holds true.

Thus, I realized that I only need to define phenomena and conditions, not essence. Despite years of training that tell me this is unscientific, I find that the path has widened.

---

Now, let me return to the main topic and provide a definition for the "immersion" and "presence" of robots:

• Immersion: The subject of "immersion" acknowledges being an outsider, recognizing that the sensations are virtual, and the robot itself is merely a substitute for peripheral sensory organs.
• Sensory organ substitution: sight, hearing, smell, taste, and touch.
• Presence: The subject of "presence" is fully engaged, acknowledging that sensations are real. The robot is not only a sensory substitute but also possesses a sense of self and psychological motivation.
• The first layer is the substitution of sensory organs: sight, hearing, smell, taste, and touch.
• The second layer is the substitution of "proprioception": with the distinction between self and others, further processing sensory information and regulating internal functions through the nervous system, mostly at the level of movement and rapid decision-making (subconscious).
• The third layer is psychological motivation: including individual consciousness and subconsciousness, as well as collective social consciousness and subconsciousness.

---

The subject of "immersion" is generally human, and what the machine needs to do is to substitute for human senses. The machine itself does not have a sense of self, and all sensory information is still processed by humans. Here, the machine needs to present the five senses of sight, hearing, smell, taste, and touch as realistically as possible, making it indistinguishable for people between virtual and real sensory information, fully immersing them into the virtual world. Of course, there are also hybrid approaches of virtual and reality.

The information of the five senses, defined phenomenally, can be divided into perception and presentation. Currently, both visually and auditorily, the technology is undoubtedly the most mature, with cameras, displays, microphones, and speakers perfectly achieving the perception and presentation of sight and sound. Furthermore, binocular cameras, lidar, VR glasses, and AR glasses have upgraded visual presentation from two-dimensional to three-dimensional.

The perception and presentation of smell and taste often depend on the identification and synthesis of chemical components. Existing sensors cannot achieve the function of the human or animal nose, only identifying relatively simple components, and the synthesis of odors depends on the mixing of different raw materials. There are also methods that directly stimulate nerves through electricity, which are not very mature in practical applications.

Tactile sensation is the most widespread and difficult of the five senses to implement.

Many people refer to simple pressure as touch, including single-point pressure sensors based on capacitive, resistive, electromagnetic principles, and single-point vibration simulators based on voice coil motors. In fact, human touch is much more complex, with over 100 types of ion channel proteins responsible for tactile perception, four of which are most commonly used. Phenomenally speaking, they are responsible for the compression and stretching of the skin's superficial and deep layers. Combined, the human hand's tactile sensation is of high resolution and dimension, capable of distinguishing the geometric form and motion of touch. We refer to this type of touch as "fine touch."

The combination we can expect to have a better sense of immersion is likely to be three-dimensional vision + three-dimensional hearing + simple olfaction + fine touch.

And fine touch is also the key to greatly enhancing the experience of immersion.

---

The subject of "presence" is generally the machine, and humans become parallel subjects in this interactive environment, where machines and humans can cooperate with each other and can be considered as operational objects for each other.

Here, in addition to having sensory substitution, what is more important for the machine is to have a sufficient sense of proprioception. The machine knows it is a machine, and humans are humans; it knows its senses, its limbs, its environment, and the movements it is making.

And the key to proprioception is still the machine's sense of touch, or multi-sensory integration mainly based on fine touch.

As Goethe said in his poem, lovers caress with "hands that can see," and snails have a "tactile vision"; the French proverb says "fingers are tipped with eyes."

The human body is covered with tactile sensation; you can imagine losing sight, hearing, smell, and taste, but you cannot imagine living without touch.

This is the sense of proprioception that touch brings, an instinctive sense of self that humans and animals possess, but unfortunately, machines do not. Fine touch gives machines this instinct.

Another question is, will machines have psychological motivation? The nature of this question is indefinable, as mentioned earlier, everything is empty of inherent existence. Human consciousness is also scientifically inexplicable; if we must find an explanation, Matthew Fisher from UCSB believes that human consciousness, memory, and thought are entangled quantum states, which need to be explained by quantum theory. Therefore, we believe that machines will not soon acquire autonomous psychological motivation.

However, we can make machines have simulated psychological motivation; consciousness can be simulated through biomimetic dynamic models, which at least appears to be safe.

---

Undoubtedly, touch is the most critical part of both "immersion" and "presence."

What we do with touch involves very interdisciplinary subjects, including numerical computation, mechanics, optics, machine learning, and mechanical structure, all of which are very demanding disciplines. But I think what is remarkable is that we have made it fun. I hope young employees will say the company is "so much fun." I still have a belief that as long as you make anything fun, more people will follow you.

---

I will expand on the above and publish it in the form of feature articles. Due to my scattered writing habits, the editors who have urged me to contribute in the past have suffered greatly, so I sincerely apologize and suggest that editors wait until I have written most of the content before contacting me. Although the contributions will be irregular, I will try to finish writing within six months.

It's the college entrance examination season again, and I still remember and am very fortunate for the essay topic of that year, "If memory can be transplanted." As a science student who writes more code than words, it was this essay that could have unexpectedly scored full marks that has given me a little confidence in writing to this day. In 40 minutes, a science fiction novel made me leave the examination room with a smile and also opened a science fiction life for me.

However, the topic of Tactile Robotics is no longer science fiction; it is the technology and products that we have been working on day after day for years, turning science fiction into reality.

Below is the table of contents I have planned for this topic. I may not publish them in order, and I may change their titles.

In addition, since this content is closely related to the company's products, I also encourage my colleagues to publish their own opinions and articles on the public account. I believe that from different perspectives, our readers will benefit more.

---

Weiran YUAN
June 2022, Shenzhen

---

Topic Name: Tactile Robotics

Table of Contents:

• Preface: Immersion and Presence
• Human Tactile
• Machine Tactile
• Dynamic Tactile
• Proactive Tactile
• Machine Tactile for Humans
• Multisensory Integration
• Human Consciousness
• Machine Consciousness