CS Knowledge Base

#

sequence-parallelism

Here are 6 public repositories matching this topic...

NVlabs / Long-RL

Long-RL: Scaling RL to Long Sequences (NeurIPS 2025)

reinforcement-learning multi-modality long-sequence large-language-models sequence-parallelism efficient-ai

Updated Sep 24, 2025
Python

InternLM / InternEvo

InternEvo is an open-sourced lightweight training framework aims to support model pre-training without the need for extensive dependencies.

pytorch multi-modal gemma pipeline-parallelism transformers-models tensor-parallelism llava llm-training internlm flash-attention zero3 llm-framework sequence-parallelism internlm2 ring-attention deepspeed-ulysses llama3 910b

Updated Aug 21, 2025
Python

xrsrke / pipegoose

Large scale 4D parallelism pre-training for 🤗 transformers in Mixture of Experts *(still work in progress)*

transformers moe data-parallelism distributed-optimizers model-parallelism megatron mixture-of-experts pipeline-parallelism huggingface-transformers megatron-lm tensor-parallelism large-scale-language-modeling 3d-parallelism zero-1 sequence-parallelism

Updated Dec 14, 2023
Python

AlibabaPAI / FlashModels

Fast and easy distributed model training examples.

deep-learning pytorch zero data-parallelism model-parallelism distributed-training xla tensor-parallelism llm fsdp sequence-parallelism

Updated Nov 26, 2024
Python

handdl / model-parallelisms

Minimal tensor, sequence, and pipeline parallelism for LLaMA.

model-parallelism pipeline-parallelism tensor-parallelism sequence-parallelism

Updated May 14, 2026
Python

harshita-mp / transformer-sequence-parallelism

Implements sequence parallelism for Transformer layers using MPI4Python and multi-GPU acceleration, exploring scalability trade-offs and communication bottlenecks in distributed attention.

deep-learning parallel-computing transformers pytorch high-performance-computing multi-gpu mpi4py distributed-training sequence-parallelism

Updated Dec 11, 2025
Python

Improve this page

Add a description, image, and links to the sequence-parallelism topic page so that developers can more easily learn about it.

Curate this topic

Add this topic to your repo

To associate your repository with the sequence-parallelism topic, visit your repo's landing page and select "manage topics."