October 18, 2015

MongoDB 3.0 WiredTiger compression results

MongoDB 3.1.9 was released last week http://blog.mongodb.org/post/130823293808/mongodb-319-is-released

3.1.9 is a development release so it is not intended for production use but the changes will be rolled into the 3.2.0 release which will be the next stable version.

One of the big changes that happened at 3.0 was the introduction of a new storage engine called WiredTiger. You can read more about it here https://www.mongodb.com/blog/post/whats-new-mongodb-30-part-3-performance-efficiency-gains-new-storage-architecture.

WiredTiger brought document-level concurrency and compression support. mmapv1 is the default storage engine. To use WiredTiger you had to run mongod with a special --storageEngine wiredTiger flag. This is going to change in 3.2.0. WiredTiger is going to become the default storage engine so today I gave it a try with some sample data.

I took a sample database that was on mongodb 2.6.11 using mmapv1 and converted it to mongodb 3.2.0-rc0 with wiredtiger. The process to do this is to backup the database with mongodump upgrade mongodb to 3.2.0 and make sure wiredtiger is the storage engine and then use mongorestore to restore the database back.

MongoDB doesn’t start if you change the storageEngine to wiredTiger and there are files in dbPath that were created with mmapv1 so you need to move that folder if you have an existing database.

Also of note wiredtiger is not available in the 32bit builds of 3.2.0 so you need a 64bit OS.

After importing the database here are the outputs of db.objects.stats()

mongodb 2.6.11 + mmapv1

{ "ns" : "0.objects", "count" : 2668342, "size" : 513690384, "avgObjSize" : 192, "storageSize" : 629637120, "totalIndexSize" : 357896224, "indexSizes" : { "_id_" : 86583840, "expireAt_1" : 45654784, "_key_1_score_-1" : 109084192, "_key_1_value_-1" : 116573408 }
}

mongodb 3.2.0-rc + wiredtiger

{ "ns" : "0.objects", "count" : 2668337, "size" : 345978824, "avgObjSize" : 129, "storageSize" : 109842432, "totalIndexSize" : 97681408, "indexSizes" : { "_id_" : 24162304, "expireAt_1" : 10874880, "_key_1_score_-1" : 22380544, "_key_1_value_-1" : 40263680 }
}

Both databases have 2.6 million documents. If you look closely you will notice the crazy reduction in both data and index sizes.

0_1445123961489_upload-a2eb95d4-8305-4723-ac04-95b7c9c87c38

If you are interested in how this works I suggest you read the compression options in 3.0+ and wired tiger https://www.mongodb.com/blog/post/new-compression-options-mongodb-30.

You can already benefit from this if you are on mongodb 3.0+, although you have to mongodump/switch to wiredtiger/mongorestore. If you do the switch post your before/after usage stats here!

© 2014 – 2023 NodeBB, Inc. — Made in Canada.