Loubna Ben Allal
Lewis Tunstall
Dana Aubakirova
Maurizio Fiusco
Edward Beeching
Kashif Rasul
Leandro von Werra
Qiuyi Li
Diego di Bernardo
Combining DNA and language
When we train language models on text, we expect them to learn its underlying patterns. At best, they learn its structure as well as the most skilled humans, but we don’t expect them to find patterns unknown to us.
DNA is different: it is not a language we constructed, and our understanding of it is limited. Training models on DNA could let them learn the patterns we already know and uncover new ones in its structure and function.
DNA encodes biological instructions in an alphabet of four letters: A, G, T, and C. Genomic foundation models, such as our earlier Carbon-3B, learn patterns from large collections of DNA sequences, just as language models learn patterns from text. Researchers use them to score sequences, estimate the effects of mutations, or predict which bases come next, but these models cannot explain a prediction or answer a question about a sequence.
Language models trained mainly on text can explain biological concepts, reason about experiments and discuss findings from the literature. But they learn biology largely through descriptions, with little direct exposure to DNA itself. Bringing the two together would let a model connect patterns in DNA with decades of knowledge recorded in papers and databases, so researchers can ask about a specific sequence and discuss what it means.
Earlier work such as BioReason connected the two by pairing a separate DNA encoder with a language-model decoder. Carbon-X instead trains a single 14B decoder on both DNA and English: DNA is simply another set of tokens in the same model.
Input
Hybrid tokenizer
Shared embedding matrix
Qwen3-14B backbone
Output
Application: predicting the effect of a mutation. Given an 8,192-base window of human DNA with a single A→G change, Carbon-X correctly identifies the variant as pathogenic and explains why: the change disrupts the AG at a splice site. Of the other models, only GPT-6.1 Sol at maximum effort got it right; Opus 5.5 reached the right label for the wrong reason, and the rest answered benign, were blocked, or ran out of tokens.
How we built Carbon-X
We built Carbon-X by adapting Qwen3-14B to DNA through midtraining, supervised fine-tuning, and reinforcement learning. Three challenges shaped the recipe: learning from genomes without losing language ability, building reliable supervision where existing models were weak teachers, and turning sequence knowledge into better answers using verifiable rewards.
Learning DNA without losing language
Teaching Qwen3 DNA required balancing new capabilities against the language skills it already had. We started with a hybrid tokenizer, extending Qwen3’s vocabulary with 4,096 six-base DNA tokens, plus tokens for sequence boundaries and partial sequences. This gives genomic sequences a consistent representation instead of the irregular fragments produced by text tokenization. Six-base tokens also encode DNA more compactly than BPE tokenizers, using roughly a third as many tokens as Qwen3’s tokenizer.
The design builds on our earlier Carbon-3B results, where six-base tokenization outperformed byte-pair encoding, and follows NatureLM’s approach to combining biological sequences with text. Carbon-X processes both token types within a single model and can generate responses containing DNA and natural language.
A larger model, a lower learning rate, and sustained training gave us the best balance between DNA and language performance. Our 4B experiments showed that higher learning rates accelerated DNA learning but caused substantial forgetting on English tasks. We scaled the recipe to Qwen3-14B and trained on 300 billion tokens combining genomic sequences, biology text, and general English.
Training data
The final run used a 300B-token mixture made mostly of raw DNA from Carbon-3B’s data mixture, built from OpenGenome2 and GENERator, English text from web datasets and PDFs, and instruction data to help retain conversational and instruction-following abilities. The rest paired DNA with text: documents mined and filtered from web pages, papers and forums, plus synthetic examples in which Gemini analyzes annotated RefSeq sequences.
Midtraining ablations
DNA and English favored different learning rates. In our midtraining ablations, higher learning rates improved DNA performance but caused substantial forgetting, while lower ones preserved English but slowed progress on DNA; no setting worked well for both. Training the randomly initialized DNA embeddings first, with the rest of the model frozen, helped a little but did not resolve the conflict.
We tried different techniques for initializing DNA embeddings, separate learning rates for the embeddings and transformer body, protecting layers with conflicting DNA and English gradients, merging DNA and English specialists, and increasing English replay. None recovered strong performance in both domains.
What worked: a lower learning rate, a larger model and a longer run. Instead of the learning rate at which the model learned DNA fastest, we chose one at which DNA improved steadily, if more slowly, without losing English, and trained long enough to reach good performance on DNA tasks. At 4B, even these lower learning rates reduced English performance; at 14B, the same approach preserved most of it. We also used cosine decay: warmup–stable–decay keeps the learning rate high for most of training, which worsened forgetting in our experiments.
Midtraining unlocked gains that RL alone did not deliver. On FrontierVEP, RL applied directly to Qwen3-14B failed to rise above the 50% random-chance baseline throughout our training runs. Starting from the midtrained checkpoint, the same approach moved beyond chance and continued improving.
Building supervision we could trust
Higher benchmark scores did not guarantee grounded answers. The model still invented database references and misidentified genes, motivating supervision that could be checked against the supplied DNA.
Standard distillation offered no reliable shortcut: frontier models frequently refused requests or hallucinated, while open models struggled with the underlying tasks. We initially used RL rollouts from our midtrained model for self-distillation, but these responses carried unsupported claims into the supervision.
We worked with domain experts to build deterministic verifiers that check claims against the input sequence. These checks helped us construct grounded training responses. The resulting SFT model achieved higher overall FrontierVEP performance than our self-distilled model.
Improving with verifiable rewards
We then used reinforcement learning to improve performance on tasks with checkable outcomes. Starting from the SFT model, we trained on mutation-effect prediction, DNA question answering, sequence recovery, and sequence-position tasks, alongside instruction-following tasks. Task-specific verifiers supplied the rewards.
RL improved performance across all five DNA benchmarks, with the largest gains on mutation-effect prediction: approximately 20 percentage points on BioReason Non-SNV and 10 points on BioReason Coding over the SFT checkpoint.
Putting Carbon-X to work
Carbon-X can serve as a DNA specialist within a larger agent or run sequence-analysis tasks directly. Its low inference cost and support for local deployment make both approaches accessible on researchers’ own infrastructure.
A DNA specialist for other models
Using Carbon-X as a sub-agent improved Kimi K3’s performance in our agent evaluation, raising its DNA intelligence index from 49.4% to 60.5%—an improvement of 11.1 percentage points across five benchmarks.
DNA analysis at scale
Large-scale genomic studies can require millions of predictions. We estimate that running inference on 2.4 million genetic variants whose effects on health remain uncertain would cost about $300 with Carbon-X, compared with $32,000 to $4 million using the frontier AI models evaluated, depending on the model and reasoning setting.
Running Carbon-X locally
Carbon-X works with llama.cpp, so you can ask questions about DNA sequences locally, without sending them to an external model provider. For researchers working with sensitive or unpublished sequences, this keeps the data within their own infrastructure. The demo below shows Carbon-X running locally on a MacBook.
The importance of being open
As frontier labs move into biology, researchers should not have to depend on a handful of closed APIs. They need strong models they can run on their own hardware and afford to use at scale, while keeping sensitive data private.
Routine requests can trigger refusals. Safeguards matter in biology, but many closed models also refuse to answer legitimate biology questions. This has a measurable impact on benchmark performance. For example, on FrontierVEP, which tests whether mutations are benign or harmful, Gemini 4 Argon’s accuracy drops from 58% to 38% as reasoning effort increases, alongside a rise in refusals. We see a similar pattern with Claude Opus 5.5 on NextBase, which tests a model’s ability to generate DNA sequences: its score drops from 17.7% at low reasoning effort to just 1.1% at maximum effort.
Performance also depends on whether a model attempts the task. Several frontier models refused a large share of legitimate genomic requests. For example, on the NextBase task, the refusal rates for Opus 5.5 and Gemini 4 Argon increased significantly with reasoning effort, leading to a corresponding drop in performance.
Limitations
Carbon-X is a starting point for research on connecting DNA and language. We hope the model and training recipe help others investigate how knowledge learned from sequences can support reliable answers in natural language. Carbon-X has not been clinically validated and further work is needed to make its sequence-based answers consistently accurate and grounded.
- Exact sequence operations: The model can miscount bases, confuse coordinates, or make incorrect edits, particularly when locating exact positions in long sequences. One direction is to use it as a specialist within a larger assistant, combining Carbon-X’s sequence capabilities with tools that perform exact operations.
- Missing context and hallucinations: The model may invent a sequence’s identity or function when the prompt does not provide enough information. Include known annotations and relevant context where available, and verify its biological claims.
- Short DNA tasks and general biology: Frontier models remain stronger on short DNA exercises and textbook biology questions.
- Long sequence generation: Quality degrades when the model is asked to generate long DNA sequences.
Safety and dual use
We're releasing Carbon-X to make DNA research more accessible and support local use for sensitive data. Openness also enables independent scrutiny of the model’s limitations and potential dual-use risks.
With this in mind, we excluded eukaryote-infecting viral genomes from our DNA training data and evaluated Carbon-X on sequence-editing tasks with potential dual-use relevance. We found limited editing reliability and no advantage over readily available deterministic tools on those tasks. However, further research is needed to fully assess potential misuse risks, including after fine-tuning or integration with other systems.
Acknowledgements
Carbon-X is a joint collaboration between the research teams at Hugging Face, Zhongguancun Academy, and the Telethon Institute of Genetics and Medicine/University of Naples “Federico II”.
