Skip to content

⚡️Hybird Context Parallelism

Choose a tag to compare

View all tags
@DefTruth DefTruth released this 20 Oct 10:44
· 638 commits to main since this release
v1.0.6
fa3a6e8
This commit was created on GitHub.com and signed with GitHub’s verified signature.
GPG key ID: B5690EEEBB952194
Verified
Learn about vigilant mode.

cache-dit is compatible with context parallelism. Currently, we support the use of Hybrid Cache + Context Parallelism scheme (via NATIVE_DIFFUSER parallelism backend) in cache-dit. Users can use Context Parallelism to further accelerate the speed of inference! For more details, please refer to 📚examples/parallelism.

from cache_dit import ParallelismConfig

cache_dit.enable_cache(
    pipe_or_adapter, 
    cache_config=DBCacheConfig(...),
    # Set ulysses_size > 1 to enable ulysses style context parallelism.
    parallelism_config=ParallelismConfig(ulysses_size=2),
)
# Then, run with torchrun cmd:
# torchrun --nproc_per_node=2 parallel_cache.py
Assets 2
Loading