Ken Chen's Blog

MongoPush tool is another bold stroke in my serendipitous MongoDB career. It is a push based migration tool for MongoDB clusters. It provides high parallelism, supports topology transformation, is resumable, offers progress monitoring, and supports migrating subsets of data.  With MongoDB Atlas gaining popularity and the success of the Keyhole , I was frequently asked whether a migration feature can be added to the Keyhole. The other existing migration solutions often fail because of firewall restrictions or other limitations.  Keyhole serves a different purpose as an analysis tool, so rather than confuse its function, I created MongoPush as a separate tool.  Push-Based Solution My goal was simple, to develop a tool that can perform a sharded cluster migration when the Atlas Live Migration Service is not suitable.  MongoDB provides the mongomirror tool to push data to MongoDB Atlas from behind firewalls, but mongomirror can only migrate data between replica sets...

WiredTiger is the default storage engine starting in MongoDB v3.2.  In the diagram below, the red arrows show the flow of data through compression/decompression steps among the WiredTiger cache, file system cache and data files. Ideally, if all data, including indexes, can fit into the WiredTiger cache, accessing documents would be more efficient than having to access disk files frequently.  In reality, memory resources are shared to support other tasks such as file system cache, user connections, and database operations, for example, sorting.  At startup, if the WiredTiger cache size is not defined, the maximum memory allocated to the WiredTiger cache size is about a little less than 50% of the RAM. When experiencing performance degradation, users always intend to tune the WiredTiger cache size.  It is recommended to use the default value unless you have a small working set that can possibly fit into WiredTiger cache or the compressed data size can fit into file sy...