Token Efficient Data Format for LLMs | Dweve HEDL

Choose the audience that matches your question

The page contains three selectable readings of the same subject.

Multi-agent orchestration with HEDL throughout.

Sub-millisecond retrieval for richer prompts.

Install HEDL, run the MCP server in front of your tools, and watch token usage drop. Benchmark corpus in the repository.

Same semantics, denser encoding, restored exactly.

Remove the proxy and you are back on JSON.

Rust core with zero dependencies and no unsafe code.

No vendor in path. Conversion runs locally.

HEDL ships as a Rust library, MCP server, and WASM component. The core parser has zero dependencies and no unsafe code; conversion is local, bidirectional, and lossless across the supported formats.

Library, MCP server, and WASM expose the same lossless conversion path.

Agent pipelines, retrieval, batch extraction, anywhere LLM tokens are billed by volume. The savings compound as tool call chains grow longer. Same data goes in, same answers come out, the only thing that changes is the number on the invoice.

Schema parser, positional encoder, MCP adapter

Drop in the MCP server and your agent stack speaks HEDL upstream and JSON downstream with no application code change. The proxy logs token counts before and after on every request. A Rust crate, a CLI, and a WASM build cover embedded and browser use.

Measured on 571 extraction tasks across 7 datasets, run on Mistral Large, DeepSeek Chat, and NVIDIA GLM-4.7.

The values you meant to send. In HEDL, nearly all that remains.

A quote pair around every string, even the unambiguous ones.

Braces and brackets marking structure the schema already knows.

The same field names, paid again on every record.

JSON is the universal exchange format, but as an LLM input it is wasteful. Repeated field names, structural brackets, and quote marks around every string are bytes you pay for that carry no meaning. Isolate each part below to see exactly what your token budget is buying.

Average across the 7 benchmark datasets.

We did not assume LLMs would understand a denser format. We tested it. In one measured run across 571 structured extraction questions on 7 real-world datasets, HEDL used 56 percent fewer tokens than JSON and recorded the highest accuracy among the formats in that run. The result belongs to that model, corpus and test setup.

HEDL declares a schema once, then encodes records positionally. No repeated field names. No brackets for nesting. Strings unquoted by default. Pick a dataset and watch the token count fall while the data stays identical. The round-trip is lossless.

Designed and maintained in Europe, with conversion running wherever you deploy it.

The Rust core has zero dependencies and no unsafe code.

It does not phone home. Nothing crosses a border you did not approve.

No vendor in the path. Conversion runs locally on your own machines.

Conversion runs on your own machines, in an EU region, or fully offline. HEDL does not phone home and needs no external service, so structured records stay inside the boundary you already operate.

Remove the proxy and every downstream system continues on plain JSON.

There is no big-bang migration to plan. Put the converter in front of one AI workflow, compare the logged token counts, then expand only where the saving holds. The path is reversible: remove the proxy and every downstream system continues on plain JSON.

Remove the converter; databases and reports keep receiving ordinary JSON.

It converts at the edge, so your databases and reports never change.

Parquet for data lakes and Neo4j Cypher for graph databases, both directions.

JSON, YAML, XML and CSV, the shapes most teams already exchange every day.

Nobody has to rip anything out. HEDL converts to and from the formats your systems already use, in both directions, and sits in front of your AI tools as a single drop-in. The dense version only exists on the trip to the model. Everything downstream sees the shapes it always did.

The meter scales the same rate up and down, so you can read off a figure close to your real usage.

The saving is simply the tokens HEDL removes. Less to send means less to pay for, at the same data.

The published reference: a service at one billion tokens a month saves 1,680 dollars a month by switching from JSON.

Tokens are billed by volume, and HEDL sends 56 percent fewer of them for the same data. Choose the size that looks like your usage and watch the monthly and yearly figures change. The published reference point is a service at one billion tokens a month, which saves 1,680 dollars a month.

Every AI call is priced by the token, and much of that budget is syntax rather than data. HEDL sent the same request in 56 percent fewer tokens in the measured run and showed a +10.3pp accuracy difference from JSON there. That is evidence from this setup, alongside a lower bill, without changing downstream systems.

Benchmark corpus and round-trip checks make the effect visible before production.

It speaks plain JSON to the rest of your stack. Remove it and you are back on JSON, byte for byte.

One compact layer sits before the model; the rest of your systems continue speaking JSON.

HEDL adds one compact exchange layer before the model while every database, report, and application continues speaking JSON. Run conversion locally, measure it on one workflow, and remove it without a data migration if the numbers do not hold.

Fewer words to pay for, with the same answer at the end.

Every compact message is restored and checked before the answer reaches you.

It only changes how the message is folded for sending.

You never see HEDL or touch it. It sits behind the helper.

HEDL is one small piece that makes AI helpers cheaper to run, without changing what they say back to you. You do not install it, you do not set it up, and you do not have to understand it. Here is the whole promise, in four short lines.

No. The full message is restored before it is used.

The full version comes back word for word.

It is normal to have questions, and there are no silly ones. Here are the five people ask most often, each with a plain answer and nothing hidden behind it. Tap a question to read the answer.

The labels are written once, not on every line.

You will never start HEDL or even see it. But every time one of those AI helpers is doing a job for you, it is quietly there, making the helper cheaper to run. Pick a moment and see what it is doing behind the curtain.

The compact message is turned back into the full version and checked before it is used.

Yes. The short way folds back into the long way, word for word.

No. The meaning stays exactly the same, every time.

No. You never see it or set it up. It does its work on its own.

Think of writing the same address on a hundred envelopes. You could write it in full every time, or write it once and keep it short on the rest. The letters still arrive. HEDL does the short way for you, and here is what that means for your side of things.

No technical words, no charts. Just the idea, the way you would explain it to a friend. HEDL is a shorter way for computers to say the same thing to an AI helper, so the same answer costs less to get.

HEDL sits in front of your AI tools and sends the same requests in 56 percent fewer tokens while speaking plain JSON to the rest of your stack. Nothing downstream changes: the result is a lower token bill, an accuracy difference measured in one benchmark run, and conversion that can run on your own machines without moving data to another service.

You do not need to know any of the technical words to follow this. When a computer talks to one of those AI helpers, it sends a lot of words, and every word is paid for, a little like a phone bill counted by the word. HEDL is a way of saying exactly the same thing with far fewer words. Nothing is left out, the meaning stays the same, and the answer you get back does not change. It just costs less, and you never have to touch it.

Every API call to a large language model is priced by the token. Most of that token budget is not data; it is overhead: repeated field names, nested brackets, quote marks around every string. HEDL sent exactly the same information using 56 percent fewer tokens in one benchmark run across 571 extraction tasks and 7 datasets, with a +10.3pp accuracy difference from JSON in that run. Drop in the MCP server and your existing JSON pipeline keeps working exactly as before.

HEDL (Hierarchical Entity Data Language) is an open-source, token-efficient text data format under Apache 2.0 for LLMs and structured records. Its Rust reader, writer, validators and servers keep the format inspectable and reversible.

HEDL (Hierarchical Entity Data Language) is an open-source, token-efficient table format under Apache 2.0 for LLM requests. Start locally with cargo add hedl; repeated labels disappear while conversion stays local and reversible. In one measured run, 500 flat records were 48.7% smaller than minified JSON.

HEDL (Hierarchical Entity Data Language) is an open-source text format under Apache 2.0 for lists sent to LLMs. It writes the column labels once at the top, then keeps the same facts in each row.

Add the Rust library with cargo add hedl or install the CLI with cargo install hedl-cli. Run the reader locally and inspect the bytes yourself.