Introducing the Best AI Image Generator: FLUX Model's Breakthrough Enhancements

For more cutting-edge AI technology news, please follow us: 【closerAI ComfyUI】. The FLUX ecosystem has entered the era of second-level output! The most powerful accelerator, nunchaku, has been tested: generating images in one second with memory use reduced by three times! It has become a productivity tool.

Hello everyone, I'm Jimmy. In this edition, I am introducing an explosive project, the FLUX model. We all know it requires a lot of memory; running it for text-to-image or image-to-image tasks with some optimizations could take around 1.5 to 2 minutes with my 8GB graphics card, and that's without adding LORA, fill, redux, or other controls. When additional controls are implemented, the time required increases further. Although the quality of the generated images is better than other models, the longer generation time is a significant downside. While there are enhanced acceleration solutions like teacha and sagettention, they still fall short of being true productivity tools in terms of efficiency.

Now, a new technological direction has emerged—the 4-bit diffusion model inference engine, nunchaku. The recent version 0.2 has propelled FLUX to generate images at a speed of one second, and the quality of the images is nearly identical to that of the original model.

ComfyUI-nunchaku Introduction

Nunchaku is a 4-bit diffusion model inference engine developed by MIT Han Lab, specifically designed to optimize generative AI models like Stable Diffusion and Flux. Its core technological breakthroughs include:

• SVDQuant Quantization Technology: By employing Singular Value Decomposition (SVD) and Kernel Fusion, model weights and activation values are compressed to 4 bits, reducing memory usage by 3.6 times (e.g., a 16GB graphics card can run the originally required 50GB Flux.1-dev model). This solves the traditional issue of image blurriness with 4-bit quantization, achieving an LPIPS quality index of only 0.326 (close to the original 0.573), which is visually indistinguishable.

• Multi-modal Ecosystem Compatibility: Perfect support for Flux models, LoRA, ControlNet, and multi-GPU architectures (NVIDIA Ampere/Ada/A100). Enhancements for text-to-image, ControlNet repainting, repairs, etc. yield an 8.7 times speed increase.

• Hardware-level Optimization: Optimized for the NVIDIA CUDA architecture, supporting FP16/FP8 mixed-precision computation, achieving image generation in just 3 seconds on a 16GB graphics card (compared to the original 111 seconds).

ComfyUI Node: https://github.com/mit-han-lab/ComfyUI-nunchaku
Wheel Address: https://huggingface.co/mit-han-lab/nunchaku/tree/main

Version 0.2.0 released on April 5! This version introduces multi-LoRA and ControlNet support, enhancing performance with FP16 attention and First-Block Cache. Additionally, it adds compatibility for 20 series GPUs to FLUX.1-redux.

Installation

1. Download the node and place it in comfyUI/custom_nodes.

2. Install the wheel from the above address using PIP. Make sure to have PyTorch version 2.5 or later; if your version is insufficient, please upgrade and back up your Python packages in advance as a precaution. Download the wheel matching your version.
   

3. Download the model and place it in comfyUI/models/unet
   

Restart ComfyUI Experience and Optimization Plan

While its node package contains workflows, directly using its workflow for generating images may result in unstable quality; you might feel it is comparatively inferior. Additionally, loading the models takes some time, but it is within an acceptable range. Once the model is loaded, generating images after drawing cards or modifying parameters becomes extremely fast.

Using the optimized workflow, I generated images in 23 seconds. There’s not much to say about its workflow since it's 4 times faster than FLUX, with comparable quality. However, because it’s accelerated and a 4-bit quantized model, there is some loss, but it’s minimal. Here, I mainly want to share some insights and thoughts on our optimization strategies for your reference. Previously, we introduced the magnification repair plan for FLUX: 【closerAI ComfyUI】Strong! New ideas for magnification and repair! A tool that balances efficiency and quality. The essential high-definition magnification scheme necessary for generating FLUX images must be learned!

The key insight is combining the scheduler kl_optimal's high-definition repair approach with nunchaku. We at closerAI built a super-accelerated image generation workflow based on this integration, as shown in the figure below:

The above images are generated based only on the original image size and style references, without any magnification or LORA integration. Our workflow thoughts are as follows:

1. Use the nunchaku FLUX basic text-to-image workflow for the first sampling. Here, we employ the SVDQ-int4-flux.1-dev model, with other reference settings as shown in the figure below:
   
   

2. Integrate with kl_optimal for second sampling.
   
   
   

What if your local computational power is insufficient?

For friends with limited local computational power, we recommend using online ComfyUI for your experience: runninghub.cn

Nunchaku FLUX workflow experience address: https://www.runninghub.cn/ai-detail/1910201953526448130
Registration address: https://www.runninghub.cn/?utm_source=kol01-RH151

Final Thoughts

The advantages of nunchaku FLUX include ultra-fast image generation, which resolves the slow generation issue with FLUX. Moreover, its ecosystem is gradually improving and supports FLUX's fill, redux, ControlNet, and multi-LoRA, making it a potential substitute for FLUX in image generation tasks. Highly recommended for use. As a productivity tool, it is entirely sufficient for generating base images.

This has been an introduction to the stable diffusion ComfyUI built by the closerAI team and the FLUX nunchaku super-accelerated image generation workflow. You can try building your workflow based on these ideas. Additionally, you can also obtain corresponding workflows on our closerAI membership site.

Thank you for reading my article! If you found it helpful, please give it a thumbs up, and if you'd like to receive updates promptly, feel free to star my work ⭐. See you next time!