diff --git a/Readme.md b/Readme.md index f26f160..d3f282e 100644 --- a/Readme.md +++ b/Readme.md @@ -1,47 +1,99 @@
-# 👤 Live Avatar +

+ Live Avatar Teaser +

-**Streaming Real-time Infinite Inference for Interactive Avatars** +

🎬 Live Avatar: Streaming Real-time Audio-Driven Avatar Generation with Infinite Length

+ +

+Yubo Huang1,2 · +Hailong Guo1,3 · +Fangtai Wu1,4 · +Shifeng Zhang1 · +Shijie Huang1 · +Qijun Gan4 · +Lin Liu2 · +Sirui Zhao2,* · +Enhong Chen2,* · +Jiaming Liu1,‡ · +Steven Hoi1 +

+ +

+1 Alibaba Group    +2 University of Science and Technology of China    +3 Beijing University of Posts and Telecommunications    +4 Zhejiang University +

+ +

+* Corresponding authors.    ‡ Project leader. +

-arXiv -HuggingFace -Demo -Github - - - -
-Live Avatar Teaser - -

- Live Avatar achieves high-fidelity, low-latency, and infinite-duration avatar generation.
- Compatible with NVIDIA RTX 4090 & H800 GPUs. -

+arXiv HuggingFace Github Project Page
---- +> **TL;DR:** **Live Avatar** is an algorithm–system co-designed framework that enables real-time, streaming, infinite-length interactive avatar video generation. Powered by a **14B-parameter** diffusion model, it achieves **20 FPS** on **5×H800** GPUs with **4-step** sampling and supports **Block-wise Autoregressive** processing for **10,000+** second streaming videos. +

+ +

+ +--- +## ✨ Highlights + + + +> - ⚡ **​​Real-time Streaming Interaction**​​ - Achieve 20 FPS real-time streaming with low latency +> - ♾️ ​​**​​Infinite-length Autoregressive Generation**​​​​ - Support 10,000+ second continuous video generation +> - 🎨 ​​**​​Generalization Performances**​​​​ - Strong generalization across cartoon characters, singing, and diverse scenarios + + +--- ## 📰 News -- **[2025/11]** Code and inference scripts released. +- **[2025.12.02]** The code will be open source in early December. +- **[2025.12.02]** We release Paper and demo page Website. --- -## 📖 Abstract +## 📑 Todo List - -
- Framework Architecture -
+### 🌟 **Early December** (core code release) -> **Abstract:** *Recent studies have demonstrated the effectiveness of directly aligning diffusion models... [Insert your abstract here]. In this work, we propose Live Avatar, a framework that enables streaming real-time infinite inference...* +- ✅ Release the paper +- ✅ Release the demo website +- ⬜ Release inference code +- ⬜ Release checkpoints on Hugging Face +- ⬜ Release Gradio demo +- ⬜ Experimental real-time streaming inference on H800 GPUs. + - ⬜ Distribution-matching distillation to 4 steps + - ⬜ Timestep-forcing pipeline parallelism ---- +### ⚙️ **Later updates** -## 🛠️ Installation +- ⬜ Optimized real-time streaming inference on RTX 4090 / A100 GPUs. + - ⬜ Distribution-matching distillation to 3 steps + - ⬜ Timestep-forcing pipeline parallelism + - ⬜ SVD quantization + - ⬜ SageAttention integration +- ⬜ Run with very low VRAM +- ⬜ TTS integration +- ⬜ ComfyUI support +- ⬜ 1.3B model + + -## 📥 Download Models + + + ---- + -## 🚀 Inference + ## 📝 Citation If you find this project useful for your research, please consider citing our paper: ```bibtex -@article{yourname2025liveavatar, - title={Live Avatar: Directly Aligning the Full Diffusion Trajectory with Fine-Grained Human Preference}, - author={Author One and Author Two and Author Three}, - journal={arXiv preprint arXiv:2509.XXXXX}, - year={2025} +@article{placeholder } ``` @@ -118,5 +166,5 @@ If you find this project useful for your research, please consider citing our pa We would like to express our gratitude to the following projects: * [CausVid](https://github.com/tianweiy/CausVid) -* [SelfForcing](https://github.com/guandeh17/Self-Forcing) +* [Longlive](https://github.com/NVlabs/LongLive) * [WanS2V](https://humanaigc.github.io/wan-s2v-webpage/) \ No newline at end of file diff --git a/assets/liveavatar1.pdf b/assets/liveavatar1.pdf deleted file mode 100644 index d4c5c0f..0000000 Binary files a/assets/liveavatar1.pdf and /dev/null differ diff --git a/liveavatar/models/wan/causal_s2v_pipeline_infinite.py b/liveavatar/models/wan/causal_s2v_pipeline_infinite.py index 7cd2107..2b5dc78 100644 --- a/liveavatar/models/wan/causal_s2v_pipeline_infinite.py +++ b/liveavatar/models/wan/causal_s2v_pipeline_infinite.py @@ -889,13 +889,7 @@ class WanS2V: m=(mask[0][0].detach().to(torch.float16).cpu().numpy()>0.5).astype(np.uint8)*255; Image.fromarray(m.squeeze()).save("tmp/mask/mask.png") else: audio_emb, nr = self.encode_audio(audio_path, infer_frames=infer_frames) - # print(f"nr: {nr}") - # print(f"audio_emb num clip: {audio_emb.shape[-1]//infer_frames}") - # assert audio_emb.shape[-1]//infer_frames == nr - # num_repeat_clip = 3334 // nr + 1 #10000 seconds - # print(f"num_repeat_clip: {num_repeat_clip}") - # nr = nr * num_repeat_clip - # audio_emb = torch.cat([audio_emb]*num_repeat_clip, dim=-1) + self.audio_encoder.model.to("cpu") if num_repeat is None or num_repeat > nr: