Tripo Doodle (TripoSG Scribble): Bringing 3D Ideas to Life Instantly with Sketch and Text

At SIGGRAPH Asia 2024's "Real-Time LIVE!", VAST (Tripo) unveiled Tripo Doodle (internally referred to as TripoSG Scribble), an interactive tool that transforms simple 2D sketches and text prompts into detailed 3D models in real-time. Traditional 3D modeling often involves steep learning curves and significant time investment, particularly in the initial stages of blocking out shapes. Tripo Doodle dramatically lowers this barrier, offering an intuitive, lightning-fast workflow that empowers both seasoned professionals and newcomers to visualize and iterate on 3D concepts with unprecedented ease. As demonstrated live, users can literally "doodle" their way to 3D assets, focusing creative energy on the idea rather than complex tooling.

VAST (Tripo) is thrilled to announce the open-sourcing of key components and insights behind it.

What is Tripo Doodle? From SIGGRAPH Stage to Your Fingertips

Tripo Doodle is a new interface designed to radically simplify and accelerate the 3D creation process. It allows users to:

Sketch Intuitively: Draw simple 2D shapes and lines on a canvas.
Add Text Prompts: Provide semantic context or specific attributes via text.
Generate Instantly: See a detailed 3D model appear and update in real-time based on these inputs.

Debuting at SIGGRAPH Asia 2024 Real-Time LIVE!, Tripo Doodle captivated audiences by showcasing a future where 3D modeling is as fluid and accessible as doodling. It directly addresses the often steep learning curve and time-consuming nature of traditional 3D workflows, particularly the initial asset blocking phase, empowering creators to focus purely on their ideas.

Core Technology: Extending TripoSG for Real-Time Interaction

Tripo Doodle isn't built from scratch; it leverages the power of VAST's state-of-the-art TripoSG foundation model and extends it with specific innovations for real-time, multimodal interaction:

TripoSG Base Model: The underlying engine is TripoSG, an image-to-3D shape generation model. It allows for the high-fidelity generation of 3D meshes directly from conditioning inputs (typically images in the base model). It's trained on curated data using precise Signed Distance Function (SDF) representations managed by a custom Variational Autoencoder (VAE).
Multimodal Conditioning (Sketch + Text): Tripo Doodle enhances TripoSG by incorporating mechanisms to understand and integrate both sketch and text inputs simultaneously.
- Sketch Guidance: The 2D drawing provides strong geometric constraints, defining the core shape, structure, and pose.
- Text Guidance: Natural language prompts steer the semantic interpretation, influencing object type, style, and specific features (e.g., adding "dragon" transforms a generic monster sketch).
Real-Time Optimization (e.g., Distillation): To achieve the near-instantaneous generation speeds essential for the interactive "doodling" experience, techniques such as CFG distillation are employed. A smaller, optimized model is trained to replicate the output of the larger TripoSG model, enabling rapid inference suitable for real-time updates based on continuous user input.

Bringing Ideas to Life

The SIGGRAPH Asia 2024 Real-Time LIVE! demonstration illustrated Tripo Doodle's power:

Effortless Creation: Simple sketches of a plant, table, ring, or monster were instantly transformed into 3D objects.
Live Iteration: The 3D models updated dynamically as sketches were drawn, erased, or refined, and as text prompts were added or changed (e.g., turning a generic monster into a "turtle monster" or a "dragon monster" with added wings).
Creative Exploration: The "Randomize" function allowed users to quickly cycle through different valid 3D interpretations of the same sketch/text input.
Accessibility: The "Doodle 1v1" segment, where audience members competed to create monsters in under 30 seconds, highlighted how intuitive and fast the tool is, even for first-time users. Examples like the "tomato monster" and the "caterpillar monster" showcased the creative (and sometimes surprising!) results achievable in seconds.

Explore Further

VAST is committed to advancing the field through open collaboration. Both TripoSG Scribble and TripoSG are open-sourced.

We invite the research and developer community to explore TripoSG and the concepts behind Tripo Doodle, build upon them, and help shape the future of 3D AI.