Preserve
Synthetic Data Generation,
20-30x Faster
Stop generating tokens. Start permuting sentences.
160ms vs 3-5 seconds
Old vs New: Synthetic Data Generation
Why attention-based permutation beats token generation
Old Method
LLM-Driven Generation
New Method
Attention-Based Permutation
Key Insight
Instead of generating new text token-by-token, we reuse existing sentences in valid orders based on their semantic dependencies extracted from attention patterns.
Technical Deep Dive
How we achieve O(n) speedup with adaptive constraint relaxation
Pattern Extraction
We run a single forward pass through the model with output_attentions=True, then filter to only "structural" attention heads that have >30% cross-sentence attention.
These structural heads reveal which sentences semantically depend on which others. For example: "It does X" attends strongly to the sentence defining "it".
The Unique Solution
We convert dependencies into a DAG (Directed Acyclic Graph) where sentences are nodes and edges represent semantic dependencies. Then we generate permutations via topological sorts of this DAG.
The Challenge: "Almost Linear" DAGs
When DAGs are too restrictive (avg <2.5 dependencies/node), we get limited permutation diversity.
Our Innovation: Adaptive Constraint Relaxation
We dynamically detect restrictive DAGs and apply strategies:
- →Relax position constraints
- →Allow "sibling" swaps (sentences sharing dependencies)
- →Use fallback strategies for diversity
Result: O(seqlen) speedup while preventing semantic breakage and ensuring permutation diversity
Semantic Preservation
Dependencies maintained
Lightning Fast
No token generation
High Diversity
Multiple valid permutations
Join the Waitlist
Be among the first to access Preserve and revolutionize your synthetic data generation