About the project
VR 3D modeling tools, 2022
l changeable x w changeable x h 200 mm (x x w x h inch)
Fusion360, Keyshot, Unity
Arduino, Soft urethane, silicone, Stainless steel, TPU
VR Clay is a haptic tool that enables sculpting in a VR environment, allowing users to feel the texture and sensation of clay with their hands. The discovery of pottery fragments in China in 2012 suggests that humans have been developing clay sculpting techniques for thousands of years. However, today’s predominant technologies involve using keyboards and mice on 2D monitors and printing with 3D printers, which, while maximizing accessibility and productivity, risk the loss of traditional sculpting techniques.
University sculpture departments are gradually disappearing, and even physical sculptors are required to learn digital sculpting. Many find the transition challenging due to the high entry barriers of digital sculpting. Surveys and interviews have revealed that professional digital sculptors and design students prefer physical sculpting techniques for their intuitive and instinctive nature, as adapting to new computer applications is burdensome.
VR-Clay leverages these familiar and instinctive sculpting skills in a digital format. This innovation allows for the continued development of traditional sculpting techniques, lowers the entry barriers for physical sculptors into digital sculpting, and saves digital sculptors from the time-consuming process of adapting to new applications. The integration of intuitive and instinctive clay sculpting techniques with digital tools promises more productive and creative activities.
Simulation with the prototype
Motivation
With the advancement of computer technology, humanity is losing the traditional sculpting techniques developed since they first began creating with clay.
This skill is an intuitive and instinctive behavior for humans. It develops from a very young age through tactile play. Even without instructions or education, humans learn to imagine and enjoy creating things by feeling and manipulating clay, honing their skills through tactile experience.
How can humanity utilize the accumulated physical sculpting techniques in the realm of computers? How can physical sculptors employ their skills in a computer environment? How can computer sculptors create using their hands rather than relying on keyboards and mice? And are these questions even necessary? To address these curiosities, I conducted the following interviews and research.
Analyze: Sculpture students and professional sculptors expressed a strong desire to work with tangible materials. However, along with industrial design students, they also recognized the necessity of computer-based sculpting, despite finding it challenging to master.
Surveys and interviews revealed that both Sculpture and Industrial Design students struggled with visualizing three-dimensional objects on a flat surface. This indicated a need for a more intuitive, immersive environment to facilitate their work. Virtual Reality (VR) presents a promising solution by offering a stereoscopic workspace, enhancing the 3D modeling experience.
Conclusion:
Consequently, I embarked on researching a VR modeling tool that simulates the tactile sensation of working with clay. Interestingly, as demonstrated in Figures 1-3, 2-3, and 3-3, all participant groups indicated that an advanced modeling tool with touch sensitivity would be beneficial. Notably, sculptors favored clay for initial concept development, whereas industrial design students preferred traditional pens and pencils. In interviews, industrial designers mentioned that using pencil and paper allowed them greater freedom to express their ideas, and they felt a disconnect between their 2D sketches and the 3D models.
This feedback underscores the advantage of conceptualizing in 3D from the outset for projects intended to be realized in three dimensions. Therefore, the market potential for a 3D sculpting tool extends beyond sculptors to include industrial designers, highlighting a broader application for this innovative technology.
Interview
With fine artists based on modeling
a1. What is your modeling process?
- Generating Ideas
- Rapid Shape Prototyping with Clay
- Sketching on Paper or Tablet
- Physical Modeling
a-2. What are the advantages of working with physical modelling?
- The viewers can share the feeling of the artist touching the clay.
- The artist’s emotions can be shared with the viewers. Viewers can feel the sincerity of the sculptor.
a-3. The matter has its own uniqueness, value, and vitality.
- The physical space creates the atmosphere of the work.
a-4. What are the downsides of physical sculpting? It is affected by gravity.
- The probability of failure is high. The financial burden is enormous. There is a risk of breakage.
a-5. What are computer sculpting apps similar to physical sculpting?
- There is nothing.
B. About virtual sculpting
b-1. What are the advantages of working with digital sculpting?
- It is possible to import previous work, copy, and paste it.
- It takes up less space.
- Less hard on the body.
b-2. What are the downsides of working digitally?
- It takes too much time to learn. It is not familiar.
- Barriers to entry are too high.
- Prediction is possible. leaves no aftertaste Expression is limited.
- It does not exist in the real world.
- It is too real to be unrealistic.
- What happens in computer work is imitating, not real.
b-3. What needs to be improved in digital modelling?
- I wish I could touch the clay.
- I hope that unexpected results can be produced.
- I hope I did not expect the result. I want the soil to harden.
- I wish the concept of time were applied to materials over time.
.
C. About physical sculpting
C-1. What is your modelling process?
- Ideation – Sketching on the paper or tablet – Physical modelling
C-2. What are the advantages of working with physical modelling?
- The user can understand the form more intuitively. Users can accurately grasp the volume and texture It can be created without learning the program.
- I can make it the way I want it to feel.
- Physical work is faster than computer work.
- It is possible to understand the properties of materials.
- The sculptor can communicate the material.
- The sculptor can feel the touch.
C-3 What are the downsides of physical sculpting?
- Expensive, Time consuming
- My body aches, and I get dusty.
C-4 What are computer sculpting apps similar to physical sculpting?
- Although “Zbrush” is similar to a physical modelling operation, it is not very similar.
D. About virtual sculpting
D-1. What are the advantages of working with digital sculpting?
- Less hard on the body.
- I can work without time and space constraints
D-2. What are the downsides of working digitally?
- It is difficult to convey the sense of touch and texture used by the artist to the viewer.
- It may be difficult to easily understand the shape three- dimensionally at once.
- In order to implement an idea, you need to be proficient in the program you use.
- You have to learn to program, you can’t see the real thing
- I cannot feel the material with my hands.
- It is difficult to grasp the light (illumination) and the three- dimensional shape of the model on the monitor.
- this is not alive
- The audience cannot feel the faint emotion, the pleasure they cannot do.
D-3. What needs to be improved in digital modelling?
- Keyboards and mice are not enough for sculpting.
- We need tools to replace them.
- I wish I could touch and feel it in real time
- It needs a simple way of working
Survey
1-1. To sculpture students at 22 universities in the UK and Korea
1-2. To the members of Royal Society Of Sculptors
1-3. To the Industrial design students From 22 Universities In England And Korea
2. What can be experienced in physical sculpting that cannot be replicated in computer sculpting?
2-1. To sculpture students at 22 universities in the UK and Korea
2-2. To the members of Royal Society Of Sculptors
2-3. To the Industrial design students From 22 Universities In England And Korea
3. If computer sculpting is developed to be similar to physical sculpting, who will benefit the most from computer sculpting?
3-1. To sculpture students at 22 universities in the UK and Korea
3-2. To the members of Royal Society Of Sculptors
3-3. To the Industrial design students From 22 Universities In England And Korea
4-1. To sculpture students at 22 universities in the UK and Korea
4-2. To the members of Royal Society Of Sculptors
4-3. To the Industrial design students From 22 Universities In England And Korea
Two different hypothesis & an experiment
How to recognize the shape of the clay and how to convert it into a digital signal?
If you can determine the mass of the clay and the location of a certain amount of clay, you will be able to digitally shape the clay.
HYPOTHESIS 1.
Weight and mass measurement
A-1) The amount of clay attached to the
frame for modelling should be measured.
A-2) The amount of clay that the sculptor is
newly attaching should be measured.
A-3) It scans only the place where the
shape of the clay changes and reads the
data.
B. SOLUTIONS
B-1) Only a certain amount of clay with a fixed weight is used.
B-2) The 3D scanner quickly scans only the area where the clay is changing and sends
the data to a computer.
B-3) It scans only the place where the
shape of the clay changes and reads the
data.
HYPOTHESIS 2.
X, Y, Z COORDINATES
The smart clay particles transmit their X, Y and Z coordinates to the computer.
A. REQUIREMENTS
A-1) The particles of smart clay must be small enough for the sculptor to do delicate work.
A-2) The particles of smart clay must be able to stick together.
A-3) The smart clay particles send their X, Y and Z coordinates to the computer.
B. SOLUTIONS
B-1) Because MEMS is a particle as small as dust, it allows the sculptor to express delicately. (ex. MEMS)
B-2) Smart particles can be agglomerated using magnets or nets.
(ex. Digital clay for VR / Alexandre, et al. 2004)
B-3) Smart particles have as much power as a computer’s CPU.(Digital Clay modules)
C. PROBLEMS
C-1) No problem.
C-2) Smart particles have different properties from clay used by sculptors.
C-3) 3. There is no evidence that smart particles can track vertical coordinates.
Experiment 1
MEASURABLE MAGNET CLAY
Hypothesis: The result of this will be able to quickly reflect the changes in the mass and shape of the modelling to the 3D modelling software.
Experimental method: The quantified iron bag correctly reflects the mass change of the model. Leap Motion detects the part where the hand touches the model, and the 3D scanner scans only the part that the model
touches.
Materials: Iron powder, balloons of various shapes, magnetic clay,
magnets, metal clay stand holder
Result: The hypothesis is confirmed, but the sculptor’s work is hindered by several installed devices. Also, the sculptor’s hand interferes with the scanning.
Reflection: In order to avoid technology competition and find
new approaches, I have to find a way that doesn’t depend on the
performance of the scanner or another way to scan clay the fastest.
The second approach. SMART CLAY STAND HOLDER
How to recognize the shape of the clay and how to convert it into a digital signal?
Even if the clay was digitized in the first experiment, the resolution of the clay shape was inevitably reduced. Therefore, if clay is scanned in real time using existing 3D scanning technology, it will be possible to digitally shape the clay.
1. Find a scanning method that does not disturb the sculptor.
2. Explore the pinnacle of advanced 3D scan technology.
3. Find a technology that can replace the scanner.
I discovered techniques to scan clay from the inside. But the opinions of experts were not positive. So, I sought to combine it with other related technologies. And I found different ways that data could be transmitted, not only by scanning but also by the image of the clay.
Tech 1.
GPR
For the sculptor to work without being disturbed by the scanner, the scanner must be inside the clay. If there is a scanner on top of the clay stent holder and you can work by covering it with clay, you will be able to freely model models larger than the size of the device.
Techniques for scanning objects through solids include sound and radar technology. From
inside the clay, the distance between the clay and the device can be calculated by measuring
the length of a time period, which starts with the speaker sending a sound to the surface
of the clay, and ends with the microphone receiving the sound to be reflected back.
The problem with this approach is that microphone has to quickly scan the whole
sides of the clay because the microphone doesn’t know where the sound has to be picked up by the directional
microphone is coming from.
GPR (Ground Penetrating Radar) +360 degree 3D scanner
Investigations related to radar technology have shown high potential. GPR is a technology that scans the ground. Because it can lighten the clay-like ground, we found great promise in this technology. The downside of GRP is its high price.
But its price is proportional to the depth of the land being explored. The sculptor’s clay requires a very shallow scan depth compared to the ground, so it can be produced at a relatively low cost. The picture below is a GPR made for 300 US dollars. If this technology were combined with a 360 degree 3D scanner technology, I was hoping that clay could be scanned from the inside of it. However, the problem with this technique is that it cannot scan shapes like bending snakes.
Tech 2.
4D scanning
For the sculptor to work without being disturbed by the scanner, the scanner must be inside the clay. If there is a scanner on top of the clay stent holder and you can work by covering it with clay, you will be able to freely model models larger than the size of the device.
I investigated the potential of quick 3D scanning technology. The technology of 3D scanning is currently a highly competitive field within the industry and is developing rapidly. The video shown here was created using a 3D scan. (The exact terminology for this technology has yet to be determined.)
Viewers can see the person in this video from various angles, such as the side and back, while the video is being played. This indicates that the 3D scanner captured at approximately 24 frames per second. Although it is currently impossible to stream captured 3D scenes in real-time, they can be produced as videos. However, many companies claim that real-time streaming will become a reality within a few years. This technology is widely used in movies, entertainment, and performances.
This technology also means that when a sculptor creates clay models, it is possible to update the changes in the clay in real-time within 3D modeling software.
Result
Tech 3.
Realtime scanning and Importing video to VR
Among the accessible technologies, smartphone-based scanning stands out. This software determines coordinates when first scanning an object, completes 3D modeling using X, Y, and Z coordinates, and updates the model in real-time as the coordinates change. This approach does not require expensive technology and offers high usability. However, it is challenging to apply this technology to clay modeling. The issue arises because new coordinates are frequently created, and the sculptor’s movements can cause existing coordinates to disappear.
I explored an easier way to scan clay and display it on a monitor. A software called LIV enables the transmission of objects photographed in the real world to the virtual world. This technology is similar to displaying physical clay on a monitor, requiring information to be rendered on the plane of the monitor or VR glasses, much like a 3D modeling application. By installing this software on a computer and setting up two or more cameras on VR glasses, users can view objects in three dimensions through the VR glasses.
Experiment to Import video to VR
Hypothesis: If an image of an object is streamed in real-time to a 3D modelling application, the sculptor can see the same effect as real-time streaming of a 3D modelling file to a 3D modelling application by scanning
the object in 3D.
Experimental method: The experiment designer installs the LIV application on the VR device and the computer application. Install a webcam on your VR glasses. Set up a green background and place the clay and clay standing holders in front of it. The sculptor works on clay sculptures in this setting
Materials: Webcam, lightings, green-coloured backdrop, clay, clay stand holder, Oculus Quest 2, monitor, LIV(application)
Result
When using digital sculpting tools, there is a limitation that physical space is required.
The third approach. VR clay
Inspiration for VR Clay Development
Development to resolve issues such as tactile sensation and use of computer tools discovered in previous experiments
EXP with Leap Motion
EXP with Balloons
Frustrated with my Leap Motion experiments, I decided to take a break and watch a movie. I always keep clay on my desk for sketching and have a habit of handling it while watching movies. During this break, I found myself absentmindedly touching a thick silicone airbag I had made for another project, but my mind perceived it as clay. Realizing this, I had an epiphany: this silicone airbag could simulate clay in the VR world.
Inspired by this idea, I created a prototype by attaching balloons to gloves, which mimicked the sensation of handling clay. This innovative approach suggested a new direction for VR modeling tools, combining tactile feedback with virtual sculpting.
Additionally, I tested the elasticity of various materials such as condoms and balloons. I found that a urethane condom with a Shore hardness of 60A could be stretched small enough to pass through the center of a pen but could also expand significantly like a balloon. I then evaluated the hardness, elasticity, and tactile properties of various stretchy objects around the house.
After experimenting with these household items, I purchased materials like latex and urethane silicone with Shore hardness ranging from 10A to 35A. Through this process, I discovered that the urethane with a Shore hardness of 35A offered the best clay-like feel—flexible yet firm enough to simulate the tactile sensation of real clay.
The development of VR Clay
Development to resolve issues such as tactile sensation and use of computer tools discovered in previous experiments
Relevant tech: Unity & Softrobotic
When studying 3D modeling tools, I focused on the sculptor’s skills and practical habits to refine the modeling process. Sculptors typically hold a lump of clay in both hands, using one hand to pinch off small amounts as needed, predominantly utilizing their thumb and index finger. To accommodate these natural movements, I designed an ergonomic controller that ensures the sculptor’s finger and hand motions do not interfere with the device.
In the VR world, when the user picks up a large piece of virtual clay with their palm, the soft urethane in the controller’s palm area expands, simulating the feel of real clay.
UNITY Integration
Even without adhering strictly to the guidelines below, the clay properties can still be maintained and can be implemented using open-source resources.
Static Floor Elements: The plates and columns of the floor remain unchanged. They are unaffected by gravity and can be rotated, lifted, or lowered using a two-handed Leap Motion control.
Particle-Based Clay Simulation:
- The virtual clay is represented as a mass of small spherical particles. For example, if the diameter of the entire lump is 200 units, each particle would have a diameter of 1 unit.
- These small particles tend to adhere to the tip of a column when the distance between them is less than 2mm.
- Particles also tend to stick to each other and the column when within a 2mm proximity.
- The particles physically react to every interaction point of the Leap Motion, allowing users to detach and attach particles from the main chunk and the column.
Adjustable Adhesion:
- The degree to which particles stick to each other and to the column can be adjusted numerically, allowing for customizable interaction dynamics.
By integrating these features, I aimed to create a VR modeling tool that closely mimics the tactile experience of working with clay, enhancing both precision and immersion for digital sculptors.
Prototype of VR Clay
Arduino Schedule for VR Clay Simulation
Initially, I developed an inflator using an injection motor. However, the injection motor’s speed was insufficient, resulting in a slow response from the Arduino device when the user interacted with the virtual clay in the VR world.
To address this issue, I modified the setup. The device was connected via a hose to two tanks: one for inflating with pressure and one for releasing air. I installed a valve on the hose and connected it to the Arduino, which was integrated with the Unity project. This setup allowed the Arduino to control the inflator more effectively, providing a more responsive and realistic interaction with the virtual clay.
User test of VR Clay
Henry Parkin | sculptor
Usability (7/10): The controller seemed very intuitive, but I did not experience it in VR.
Professionalism (10/10): The user test was conducted with clear intent and high professionalism.
Technology (6/10): Although I did not experience the VR element, I think sculpting in VR is a very creative and exciting new use for soft robotic technology. The quality of the pneumatic component in the handle is high.
Targeting (10/10): For the above reasons, the targeting was very precise.
Creativity (10/10): Sculpting in VR using soft robotic technology is a highly creative and innovative approach.
Design (9/10): I wonder if the solid parts of the controller could be compliant mechanically, so that even the hard parts respond to the user’s grip.
Min jeong Kim | sculptor
Usability (9/10): The controller seemed very intuitive and easy to use, even though I did not experience it in VR.
Professionalism (10/10): The user test was conducted with clear intent and high professionalism.
Technology (9/10): Although I did not experience the VR element, sculpting in VR is a very creative and exciting new use for soft robotic technology. The quality of the pneumatic component in the handle is high.
Targeting (10/10): For the above reasons, the targeting was very precise.
Creativity (10/10): Sculpting in VR using soft robotic technology is a highly creative and innovative approach. This tool can lead people to use clay as well as 3D programming simultaneously, which is efficient and beneficial.
Design (9/10): The design is excellent, but I wonder if the solid parts of the controller could be compliant mechanically so that even the hard parts respond to the user’s grip.
- Advice: It was an amazing prototype. I saw the potential of this tool to be useful and helpful to people working with 3D. It can efficiently combine the use of clay and 3D programming.
Empowering Creativity
in the VR Era
Insights on the VR Market
The VR market is growing faster than any other industry. As surveys have shown, computer modelers, sculptors, and industrial designers are eagerly awaiting the day when those who produce three-dimensional results can work in a three-dimensional space. Despite this demand, there are currently only two applications related to modeling on the Oculus Quest 2, which holds the largest market share in the VR industry.
Consider the timeline when computer 3D modeling applications were first released. This was before VR 3D modeling applications were developed. Now is the right time to make bold investments. Customer demand is overflowing.
Prior Arts
The outcome of this project is not about encountering numerous competitors in the market. The purpose of this project is neither an academic experiment nor a device targeting a VR space that will one day become more affordable. It is not even a haptic glove designed for gaming. The purpose of this project is to empower artists and designers to create beautiful and innovative results right now. Through this project, their ideas can be expressed more freely, making their dreams a reality.
Current Development Status
To increase accessibility and reduce costs, I am developing this device not as a standalone tool, but as an accessory that can be integrated with existing VR controllers (Oculus Quest 3). Additionally, I have minimized the size of the Arduino device to enhance its commercial viability.
