MongoPush Your GridFS to Atlas

-

GridFS is one of the most popular features of MongoDB and many implementations use GridFS as their file system to store binary files, such as PDF documents and images.  Atlas is also gaining popularity among corporations and has become the home of many mission critical databases.  In this blog, I will be solving an interesting use case of migrating a replica set to Atlas, but the existing storage size is greater than the maximum Atlas allows.

Use Case

For applications use GridFS and without a high transaction rate, a replica set works well. The maximum storage size Atlas allows is 4TB but the storage provided by Cloud providers can go as high as 16TB on a single volume. The challenge is to migrate 10TB compressed data into a 4TB bucket. To scale out, converting to a sharded cluster is recommended, but the team is seeking a way to complete data migration to Atlas immediately with minimum application changes.

The Officially Supported Way

Let’s first examine the officially supported method.  Because the existing 10TB data is more than the 4TB limit Atlas allows, an upgrade to a sharded cluster is what MongoDB recommends.  The first step is to shard all fs.chunks collections with the shard key {files_id: 1, n: 1} before using Atlas Live Migration service.  However, for those MongoDB clusters behind corporate firewalls and are not accessible by the Atlas Live Migration service, this is simply not an option.

The MongoPush Way

It would be ideal if we can migrate a replica set directly to a sharded cluster on Atlas.  Using MongoPush makes it possible with a few steps below:

  1. Create a sharded cluster on Atlas.  The number of shards depends on the storage requirement.

  2. Use MongoPush to create all indexes on the target cluster.

  3. Shard all fs.chunks collections on {files_id: 1, n: 1} and ensure there is at least one chunk on each shard.

  4. Use MongoPush to copy data.

This is probably the simplest way for those looking at eliminating years of technical debts in one shot.  I have already discussed similar steps in another blog Change Shard Key and Migrate to Atlas Using MongoPush.

The Online Archive Way

Uses are cost conscious and always look for ways to reduce cost.  To “expand” the storage size of a replica set beyond the 4TB limit, Atlas Online Archive seems an ideal place to store infrequently used documents.  Online Archive is a feature that archives infrequently accessed data from an Atlas cluster to a MongoDB-managed read-only Data Lake on a S3 storage. Once the data is archived, a unified view of Online Archive and Atlas data is available from a new connection endpoint. But, here comes the challenges. Can Online Archive move data (and quickly enough) during data migration to prevent the 10TB data overflowing the 4TB limit? The answer is yes if the collections are properly indexed and archives are properly partitioned. The instructions are as follows.


Configuration and Migration Steps

There are two collections to support GridFS; one stores file information and the other keeps the sliced file data.  Documents of a file in both collections have to be archived together.  Atlas provides web user interface as well as APIs to configure Online Archive.  In the steps followed, I’ll use the APIs to demonstrate.  The steps are summarized as follows:

  1. Create indexes of {archive: 1} on GridFS collections of the source cluster.

  2. Identify documents in GridFS collections to be archived and add a boolean field archive and set the value to true.

  3. Use MongoPush to create all indexes in the target cluster.

  4. Configure Online Archive using APIs

  5. Use MongoPush to copy data. 

Online Archive API Examples

Configure db.fs.files

curl --user "${PUBLIC-KEY}:${PRIVATE-KEY}" --digest \
--header "Content-Type: application/json" \
--include \
--request POST "https://cloud.mongodb.com/api/atlas/v1.0/groups/<group_id>/clusters/target/onlineArchives?pretty=true" \
--data '
{
        "dbName": "db",
        "collName": "fs.files",
        "partitionFields": [
                {
                        "fieldName": "_id",
                        "order": 0
                }],
        "criteria": {
                "type": "CUSTOM",
                "query": "{ \"archive\": true }"
  }
}'

Configure db.fs.chunks

curl --user "${PUBLIC-KEY}:${PRIVATE-KEY}" --digest \
--header "Content-Type: application/json" \
--include \
--request POST "https://cloud.mongodb.com/api/atlas/v1.0/groups/<group_id>/clusters/target/onlineArchives?pretty=true" \
--data '
{
        "dbName": "db",
        "collName": "fs.chunks",
        "partitionFields": [
                {
                        "fieldName": "files_id",
                        "order": 0
                }],
        "criteria": {
                "type": "CUSTOM",
                "query": "{ \"archive\": true }"
  }
}'


This solution works but has its own disadvantages.  Files retrieved from S3 buckets have higher read latency than from MongoDB clusters. In addition, Atlas provides 3 different connection endpoints, including a federated one, to access data from different data sources.  This implementation certainly requires minor application changes to handle different data sources.  Unless cost consideration outweighs performance considerations and applications maintainability, I would recommend upgrading to a sharded cluster over this solution.

Summary

The solutions discussed in this blog provide 3 solutions to migrate a large amount of GridFS data to Atlas.  Alternatively, for cost consideration, one should consider storing binary files in S3 buckets or other cheaper storages and only store metadata in MongoDB.  To migrate big data to MongoDB Atlas is a time consuming task.  I hope this provides helpful information to assist your Atlas migration project.  Please post your feedback in the comments.
Location: Atlanta, GA, USA

Comments

Post a Comment

Popular posts from this blog

Build and Download Keyhole

-
Image
=== updated 1/3/2023 ================================================================ For MongoDB JSON logs analytics, use my improved version from https://github.com/simagix/hatchet . ================================================================================== Keyhole is an open source project and the source codes are available at GitHub .  It is recommended to build your own Keyhole executables from the source codes.  Alternatively, Keyhole executables can be downloaded from Docker Hub , and available executables are for Linux, macOS, and Windows. Build from GitHub Golang has to be installed before you can build.  The commands below show how to clone from GitHub and build a Keyhole executable: mkdir -p $GOPATH/src/github.com/simagix cd $GOPATH/src/github.com/simagix git clone --depth 1 https://github.com/simagix/keyhole.git cd keyhole go build main/keyhole.go I provide a build.sh script to build Keyhole on macOS or Linux.  Simply run the script: ./build.sh I encourage you to u
Read more

Survey Your Mongo Land

-
Image
Keyhole was still under active development after I published the Peek at your MongoDB Clusters like a Pro with Keyhole: Part 1 post in December 2019.  Since then, I have added many features and also changed the format of output files from json to bson.  I’ll use this post to summarize new features and modified functions not available in the previous post.  Recap A couple months into Keyhole development, a clear direction was surfacing: I needed a tool to fetch all information of a MongoDB cluster in one shot.  I joked about this function the “Know the enemy and hit them where it hurts” feature.  The -allinfo option is perhaps the most commonly used one among users.  Using the -allinfo option, Keyhole collects host and build info, replication stats, server status, size and storage stats of databases and collections, index details and usages, and sharding configurations.  The information was collected from the primary member of a replica set or a mongos of a sharded cluster.  Later,
Read more

Maobi - Reports Generator for Keyhole

-
Image
The Maobi project was created to generate HTML reports by reading stats collected by Keyhole.  The word maobi is a phonetic translation of Chinese paintbrush or ink brushes, which are commonly used in Chinese calligraphy as well as Japanese calligraphy.  They are also used in Chinese painting and other brush painting styles.  When I was little, it was required in school to use an ink brush to write or to paint pictures.  It took practice and effort to create something pleasant to the eyes. The idea of naming the tool Maobi was a reflection of the time we all spent on preparing reports and presentations. Keyhole started as a command line data collecting and analytics tool. With more data to be presented, it became difficult to include all metrics within a terminal window. So, the project began. Maobi is like a coworker Keyhole can only dream of, one who is smart, friendly, stylish but not flashy, and has contributed greatly to Keyhole’s popularity. Installation The easiest way to
Read more