This tests various parsers agains the toml-test suite.
"Decoder tests" check if valid TOML documents are processed correctly, "encoder tests" checks writing TOML files, and "invalid input" checks that the parser rejects invalid TOML documents. Implementations are marked as "compliant" (✅) if both the "valid" and "encoder" tests pass, otherwise they're marked as non-compliant (❌). Usually it's not a big deal to accept some technically invalid TOML; almost all failing "invalid" tests are uncommon edge cases.
The benchmarks are a simple decode of a 15k and 5M file. Results are displayed as "good", "slow", and "very slow" only. Doing a fully fair comparison is not straight-forward, and it's not a contest on who can write the fastest parser – it's just to establish that performance is within reasonable bounds.
Name | Decoder tests | Encoder tests | Invalid input | 15k file | 5M file | |||
---|---|---|---|---|---|---|---|---|
❌ | C++ | nullptr-toml | fail: 3 | pass: 188(doesn't include encoder) | fail: 0 | pass: 371(TODO) | (TODO) | output |
✅ | C++ | toml++ | fail: 0 | pass: 191fail: 0 | pass: 191fail: 0 | pass: 371good | good | output |
✅ | C++ | toml11 | fail: 0 | pass: 191fail: 0 | pass: 191fail: 0 | pass: 371good | slow | output |
✅ | C | toml-c | fail: 0 | pass: 191(doesn't include encoder) | fail: 14 | pass: 357good | slow | output |
❌ | C | tomlc99 | fail: 8 | pass: 183(doesn't include encoder) | fail: 78 | pass: 293good | slow | output |
❌ | C# | tomlyn | fail: 3 | pass: 188(TODO) | fail: 5 | pass: 366slow | good | output |
✅ | Dart | toml.dart | fail: 0 | pass: 191fail: 0 | pass: 191fail: 0 | pass: 371good | good | output |
❌ | Fortran | toml-f | fail: 11 | pass: 180fail: 12 | pass: 179fail: 32 | pass: 339good | very slow | output |
✅ | Go | go-toml | fail: 0 | pass: 191fail: 0 | pass: 191fail: 0 | pass: 371good | slow | output |
✅ | Go | toml | fail: 0 | pass: 191fail: 0 | pass: 191fail: 13 | pass: 358good | good | output |
❌ | Guile | toml | fail: 5 | pass: 186fail: 4 | pass: 187fail: 14 | pass: 357(TODO) | (TODO) | output |
❌ | Haskell | toml-parser | fail: 1 | pass: 190fail: 0 | pass: 191fail: 2 | pass: 369good | good | output |
❌ | Haskell | toml-reader | fail: 2 | pass: 189(doesn't include encoder) | fail: 2 | pass: 369good | slow | output |
❌ | JS | deno-toml | fail: 28 | pass: 162fail: 88 | pass: 102fail: 125 | pass: 246(TODO) | (TODO) | output |
✅ | JS | j-toml | fail: 0 | pass: 191fail: 0 | pass: 191fail: 14 | pass: 357good | good | output |
✅ | JS | smol-toml | fail: 0 | pass: 190fail: 0 | pass: 190fail: 14 | pass: 357good | good | output |
✅ | JS | toml-eslint-parser | fail: 0 | pass: 191(doesn't include encoder) | fail: 9 | pass: 362good | good | output |
❌ | Lisp | clop | fail: 4 | pass: 187(doesn't include encoder) | fail: 0 | pass: 371(TODO) | (TODO) | output |
❌ | Lua | tinytoml | fail: 4 | pass: 186(doesn't include encoder) | fail: 1 | pass: 370(TODO) | (TODO) | output |
❌ | OCAML | otoml | fail: 3 | pass: 188fail: 8 | pass: 183fail: 52 | pass: 319(TODO) | (TODO) | output |
❌ | Odin | toml_parser | fail: 10 | pass: 181(doesn't include encoder) | fail: 205 | pass: 166(TODO) | (TODO) | output |
❌ | Python | toml | fail: 18 | pass: 173fail: 14 | pass: 177fail: 73 | pass: 298good | slow | output |
✅ | Python | tomli | fail: 0 | pass: 191fail: 0 | pass: 191fail: 0 | pass: 371good | good | output |
✅ | Python | tomlkit | fail: 0 | pass: 191fail: 0 | pass: 191fail: 6 | pass: 365slow | very slow | output |
✅ | Python | tomllib | fail: 0 | pass: 191(doesn't include encoder) | fail: 0 | pass: 371good | good | output |
❌ | Racket | toml-racket | fail: 2 | pass: 189fail: 7 | pass: 184fail: 1 | pass: 370(TODO) | (TODO) | output |
❌ | Ruby | perfect_toml | fail: 0 | pass: 191fail: 1 | pass: 190fail: 9 | pass: 362good | good | output |
❌ | Ruby | toml-rb | fail: 20 | pass: 171fail: 28 | pass: 163fail: 70 | pass: 301slow | very slow | output |
❌ | Ruby | tomlrb | fail: 1 | pass: 190(doesn't include encoder) | fail: 20 | pass: 351good | slow | output |
❌ | Rust | basic-toml | fail: 21 | pass: 170(TODO) | fail: 4 | pass: 367good | good | output |
✅ | Rust | boml | fail: 0 | pass: 191(doesn't include encoder) | fail: 134 | pass: 237good | good | output |
❌ | Rust | taplo | fail: 6 | pass: 185(TODO) | fail: 3 | pass: 368good | very slow | output |
✅ | Rust | toml | fail: 0 | pass: 191fail: 0 | pass: 191fail: 0 | pass: 371good | good | output |
✅ | Rust | toml_edit | fail: 0 | pass: 191fail: 0 | pass: 191fail: 0 | pass: 371good | good | output |
❌ | Swift | TOMLDecoder | fail: 1 | pass: 190(doesn't include encoder) | fail: 5 | pass: 366(TODO) | (TODO) | output |