Stroke3D: Lifting 2D strokes into rigged 3D model via latent diffusion models

Stroke3D applications: generation, structural editing, and animation

We present Stroke3D, a novel framework that generates rigged 3D meshes from user-drawn strokes and language instructions. We show versatile downstream applications, including generation from different viewpoints, structural editing by adding strokes or modifying joint positions, and final animation. Skeleton color represents depth in 3D space.

Overview

Abstract

Rigged 3D assets are fundamental to 3D deformation and animation. However, existing 3D generation methods face challenges in generating animatable geometry, while rigging techniques lack fine-grained structural control over skeleton creation. To address these limitations, we introduce Stroke3D, a novel framework that directly generates rigged meshes from user inputs: 2D drawn strokes and a descriptive text prompt. Our approach pioneers a two-stage pipeline that separates the generation into: 1) Controllable Skeleton Generation, where the Skeletal Graph VAE (Sk-VAE) encodes the skeleton's graph structure into a latent space and the Skeletal Graph DiT (Sk-DiT) generates a skeletal embedding. The generation process is conditioned on both text for semantics and 2D strokes for explicit structural control, with the VAE decoder reconstructing the final high-quality 3D skeleton; and 2) Enhanced Mesh Synthesis via TextuRig and SKA-DPO, where we synthesize a textured mesh conditioned on the generated skeleton. We enhance an existing skeleton-to-mesh model with TextuRig—a dataset of textured and rigged meshes with captions curated from Objaverse-XL—and employ SKA-DPO, guided by a skeleton-mesh alignment score, to further improve geometric fidelity. Together, our framework enables a more intuitive workflow for creating ready-to-animate 3D content. To the best of our knowledge, our work is the first to generate rigged 3D meshes conditioned on user-drawn 2D strokes. Extensive experiments demonstrate that Stroke3D produces plausible skeletons and high-quality meshes.

01 Pipeline

Method

During training, Sk-VAE encodes a skeleton graph into a latent space, and Sk-DiT learns to generate latent embeddings conditioned on 2D strokes and text. After training with TextuRig, SKA-DPO further refines SKDream using a skeleton-mesh alignment reward.

02 Results

Skeleton Generation

Comparison with RigNet, SKDream, MagicArticulate, and UniRig.

03 Results

Mesh Generation

Comparison with SKDream, together with ablations of SKA-DPO and TextuRig.

04 Data

TextuRig Dataset

Analysis of the TextuRig dataset used for mesh generation.

Reference

BibTeX

@inproceedings{zhao2026stroke3d,
  title={{Stroke3D}: Lifting 2D strokes into rigged 3D model via latent diffusion models},
  author={Zhao, Ruisi and Zheng, Haoren and Yang, Zongxin and Fan, Hehe and Yang, Yi},
  booktitle={The Fourteenth International Conference on Learning Representations},
  year={2026},
  url={https://openreview.net/forum?id=VgOWxor3LV}
}