As we continue to test our Fractal Tree Indexing with MongoDB, I’ve been updating my benchmark infrastructure so I can compare performance, correctness, and resource utilization.  Sysbench has long been a standard for testing MySQL performance, so I created a version that is compatible with MongoDB.  You can grab my current version of Sysbench for MongoDB here.

So what exactly is Sysbench?  According to the Sysbench homepage, “Sysbench is a modular, cross-platform and multi-threaded benchmark tool for evaluating OS [Operating System] parameters that are important for a system running a database under intensive load.”

• Sysbench schema
• 16 copies of the same collection, named sbtest1 … sbtest16, each with 10 million documents
• each has a secondary index on “k”
• documents structured as follows

{
"_id" : 1 … 10000000
"k"   : random integer between 1 and 10000000
"c"   : 10 segments of 11 random digits plus hyphen
        (119 characters total)
"pad" : 5 segments of 11 random digits plus hyphen
        (59 characters total)
}

• Sysbench workload – a single sysbench “transaction” consists of the following operations
• each client thread chooses a random collection to perform all the following operations on
• all operations are keyed by _id, sequentially if a range operation
• 10 random point lookups of “c”
• 1 range lookup of “c” of 100 documents
• 1 sum of “k” of 100 documents using the aggregation framework
• 1 ordered range lookup of “c” of 100 documents
• 1 distinct range lookup of “c” of  100 documents
• update attribute “k” (indexed) in 1 random document
• update attribute “c” (unindexed) in 1 random document
• delete 1 random document by _id, then insert of a new document using the same _id

Benchmark Environment

• Sun x4150, (2) Xeon 5460, 16GB RAM, StorageTek Controller (256MB, write-back), 4x10K SAS/RAID 0
• Ubuntu 10.04 Server (64-bit), ext4 files system
• MongoDB v2.2.3 and MongoDB v2.2.0 + Fractal Tree Indexes

Benchmark Results – Performance

Throughput at all concurrency levels was higher than MongoDB.  Our largest win was at 8 concurrent threads where we are 133% faster (17.38 tps vs. 40.50 tps).  Not bad for a first pass, as we have many more ideas to come that will push our line higher.

Benchmark Results – Compression

Compared to MongoDB’s file system size (61.36GB), our zlib size on disk takes up only 31.36% the space (19.24GB).  Compression has always been a strength of Fractal Tree Indexes, and some data can compress much more than this. I recently wrote another blog about our compression abilities on a different data set.

We will continue to share our results with the community and get people’s thoughts on applications where this might help, suggestions for next steps, and any other feedback. Please drop us a line if you are interested in becoming a beta tester.

I’ll be presenting my benchmarking infrastructure at Percona Live in April.  If you are attending the show be sure to stop by our booth and learn more about TokuDB.