Benchmarks

MASEval includes pre-implemented benchmarks for evaluating multi-agent systems.

Beta Benchmarks

Several benchmarks are currently in Beta. They have been implemented carefully, but these are highly complex systems and we have not yet validated the results against the original implementations. Use with caution when comparing with existing results or original paper numbers. Contributions and compute donations welcome!

MACS is the only benchmark that has been fully validated.

Adding Custom Benchmarks

You can also create your own benchmarks by subclassing the Benchmark class. See the Five-a-Day example for a complete walkthrough.

Licensing

For detailed source and licensing information for each benchmark's data, see BENCHMARKS.md.