Token Efficient Data Format for LLMs | Dweve HEDL

Choose the audience that matches your question

The page contains three selectable readings of the same subject.

HEDL (Hierarchical Entity Data Language) is an open-source, token-efficient text data format under Apache 2.0 for LLMs and structured records. Its Rust reader, writer, validators and servers keep the format inspectable and reversible.

HEDL (Hierarchical Entity Data Language) is an open-source, token-efficient table format under Apache 2.0 for LLM requests. Start locally with cargo add hedl; repeated labels disappear while conversion stays local and reversible. In one measured run, 500 flat records were 48.7% smaller than minified JSON.

HEDL (Hierarchical Entity Data Language) is an open-source text format under Apache 2.0 for lists sent to LLMs. It writes the column labels once at the top, then keeps the same facts in each row.

Add the Rust library with cargo add hedl or install the CLI with cargo install hedl-cli. Run the reader locally and inspect the bytes yourself.

Sub-millisecond retrieval for richer prompts.

Multi-agent orchestration with HEDL throughout.

Benchmark corpus and methodology ship in the repository.

more correct answers per token than JSON

Higher is better. HEDL returns 2.4x more correct answers per token than JSON.

Higher is better. HEDL extracts 80.4%, 10.3 points above JSON.

Shorter is better. HEDL carries the same answers in 56 percent fewer tokens than JSON.

JSON · YAML · XML · CSV · Parquet · TOON

48.7% smaller vs minified JSON in this run

Bidirectional, lossless conversion. The same data semantics, encoded more densely, restored exactly.

A drop-in proxy. Remove it and your stack is back on plain JSON, with no migration to undo.

The Rust core has zero dependencies and no unsafe code; shipped tools keep their runtime dependencies explicit.

No vendor sits in the path. Conversion runs locally, so records never leave your own infrastructure.

Figures follow the published reference: a service at 1B tokens a month saves $1,680 a month.

You never start any of this. It happens on its own, every time a helper is at work, whichever of these you are doing.

HEDL only changes the shape of the message for the trip there and back. What you started with comes back exactly the same, with nothing added and nothing lost.

You ask the helper to take something in one shape and hand it back in another, neat and ready to use.

A plain list repeats the same little labels on every single line. HEDL writes the labels once at the top, then only the entries, so the list takes far fewer words to send.

You ask the helper to go down a long list, a list of names, orders, or appointments, and tidy it up.

A long document is a lot of words to send. Folded the HEDL way, twice as much of it fits in one go, so the helper sees more of the page at once.

You hand the helper a long letter or report and ask it to pull out the parts you care about.

Behind the curtain, the helper had to read a lot of background first. HEDL folds that background up small, so it reads the same thing with far less to chew through.

You type a question into one of those AI helpers and it writes back an answer in a moment.

Same answer, said in fewer words. That is all that changes.

You can turn the short way back into the long way at any time, word for word.

You wonder whether you can ever get the full version back if you need it.

Nothing is left out. The shorter way carries exactly the same meaning as the long way.

You worry that saying it shorter might leave something out or change the meaning.

The same message, said with far fewer words, so the bill at the end is smaller.

Every word the helper reads is paid for, even the words it repeats over and over.

It sits in front of your AI tools and converts at the edge. Your existing systems keep sending and receiving the formats they already use.

Your databases, reports, and spreadsheets see the same shapes as before. The denser format only exists on the trip to and from the model.

Seven formats enter and leave through the same lossless bridge; the compact representation exists only around the model call.

The model receives the dense version. Your stack keeps speaking the formats it always did.

One drop-in converter sits between your systems and the model.

saved per month at one billion tokens by switching JSON to HEDL.

wasted per API call on repeated keys for a 10,000 user dataset.

Schema once. Then only the data, comma separated.

Now multiply the structural bytes by ten thousand records.

The values you actually meant to send. In HEDL, this is nearly all that remains.

A quote pair wrapped around every string, even the unambiguous ones.

Object braces and array brackets that mark structure the schema already knows.

The same field names, "id", "name", "role", paid again on every single record.

One compact layer sits before the model while the rest of your stack continues speaking JSON.

It speaks HEDL to the model and plain JSON back to your own systems, so the rest of your stack does not change. Take it out and you are exactly where you started.

Conversion can run on your own machines, in an EU region, or fully offline without an external service in the path.

Token reduction, answer accuracy, and lossless round trips are measured across real datasets before rollout.

Insert it before one workflow, compare the result, then expand only where it pays.

HEDL is one small piece that makes AI helpers cheaper to run, without changing what they say back to you. You do not install it, you do not set it up, and you do not have to understand it. Here is the short version of what it promises.

Because it says the same thing in fewer words, the helpers that use it cost less to run.

Every compact message is restored and checked before the helper uses it, so nothing disappears on the way.

It only changes how the message is folded for sending. The meaning is kept exactly the same.

You never see HEDL or touch it. It sits behind the AI helper and does its work on its own.

If you are still unsure, that is fine. You never have to touch HEDL for it to do its job.

Tap a question to read the plain answer. There are no wrong ones to ask.

The helpers you use cost money to run, counted a little like a phone bill by the word. Saying the same thing in fewer words means a smaller bill, which keeps those helpers cheaper to offer.

Every compact message is restored and checked before it is used. If the full version does not come back exactly, the conversion fails instead of quietly dropping part of it.

Yes. The short way folds back into the long way whenever it needs to, word for word, with nothing added and nothing lost. It is like folding a letter to post it, then unfolding it to read.

Can the full version always be brought back?

No. It only changes how the message is folded up for the trip to the helper and back. The meaning is kept exactly the same, so the answer you read is the answer you would have got anyway.

No. There is nothing for you to download, switch on, or learn. HEDL sits behind the AI helpers other people build, and it does its work on its own. You will never see it.

Do I have to install or set up anything?

You do not have to install anything or understand any of this. It works quietly in the background, so the helpers you use cost less and stay just as clear.

Imagine you write the same address on a hundred envelopes. You could write the full address every time. Or you could write it once, then just the house number on the rest. The letters still arrive. You simply did less writing. That is the whole idea behind HEDL.

You can change the short way back into the long way whenever you want, word for word, with nothing changed. Like folding a letter to fit an envelope, then unfolding it later to read.

Every word sent to an AI helper is paid for, a little like a phone bill counted by the word. Saying the same thing in fewer words means a smaller bill, with the same answer at the end.

When a computer talks to one of these AI helpers, it sends a lot of words. HEDL is a way of saying exactly the same thing with far fewer words. Nothing is left out, it is just said more plainly.

Your team places the converter in front of a single AI workflow. Nothing else in the stack is touched, and the rest of the business carries on exactly as before.

The converter records the token count before and after on every request, so you get a plain side-by-side of what you used to send and what you send now.

Once the saving is proven on the pilot, the same drop-in sits in front of the rest of your AI workflows. The saving repeats wherever tokens are billed by volume.

If it ever stops paying off, remove it and you are back on plain JSON.

Start on one workflow, prove it, then decide. No big-bang migration.

Same data in, same answers out. The only thing that changes is the number on the invoice.

The round trip is lossless, so the same data comes back exactly as it went in.

Remove the proxy and you are back on plain JSON, byte for byte, with no data migration to undo.

It sits in front of your tools and speaks plain JSON to the rest of your stack. Nothing downstream changes.

For every thousand tokens you pay for, you get more usable answers back than with JSON.