Streamline Keyhole and Maobi Reports Creation

 In this short article, I will share the instructions on how to streamline Keyhole and Maobi to create HTML reports.  Like I have explained in the Maobi - Reports Generator for Keyhole, Maobi was originally created for my own use and the idea was to create a web interface to present results instead of sharing text-based terminal screens.

During the time of developing Maobi, I wanted to review the reports first before sharing them.  Therefore, the design was to have Keyhole output metadata to a file, and Maobi can generate reports from the metadata.  Over the years, automating the report creation has been brought up many times, mainly from those who have strong wills to automate everything.  Beginning Maobi v1.7.0, I have made the automation possible by enhancing the API and also adding a -maobi flag to Keyhole v1.1.0.

Upload Metadata

To create an HTML report, you can upload the metadata, generated by Keyhole, to Maobi using a curl command.  To start Maobi, use the command below:

docker run -d -p 3333:3030 simagix/maobi

Keyhole writes metadata to files, and using different flags creates files with different suffixes.  The filename formats supported directly by Maobi are:

  • <hostname>-stats.bson.gz: output of keyhole -allinfo command

  • <hostname>-index.bson.gz: output of keyhole -index command

  • <hostname>-log.bson.gz: output of keyhole -loginfo command

In the example below, ${metadata_file} represents a metadata file, for example, host.example.com_27017-stats.bson.gz, and ${maobi_url} is the Maobi API, for example, http://api.example.com:3333/upload.

curl -s -F "file=@${metadata_file}" ${maobi_url} -o example.html

After executing the above command, a report, example.html, is created.

Keyhole Integration

A new flag -maobi was added to enable Keyhole to send the metadata automatically to Maobi and create an HTML report.   The ${conn_str} in the following examples is a MongoDB connection string, for example, mongodb+srv://user:secret@example.mongodb.net.  Note that if the -maobi flag is not included in the command, Keyhole works the same way as before.

Cluster Info and Metrics

The command below outputs two files; one contains metadata and the other is an HTML report.  Refer to my blog Survey Your Mongo Land for details about collecting cluster metrics and related information.

keyhole -maobi ${maobi_url} -allinfo ${conn_str}

Indexes Usages and Recommendations

The command below also outputs two files, a metadata file and an HTML report.  My blog Index Responsibly has details on collecting index usages and evaluating existing indexes.

keyhole -maobi ${maobi_url} -index ${conn_str}

Log Analytics

Three files will be created from each log file using the command below, and they are:

  • A metadata file

  • A TSV (Tab Separated Values) file

  • An HTML report

keyhole -maobi ${maobi_url} -loginfo ${log_file}

This Feel The Pulse of Mongo blog has additional details on log analytics.

What’s Next?

The initial idea was to design Keyhole and Maobi as two components and use API to exchange data.  Keyhole was designed to be a data collecting agent and Maobi, on the other hand, was for the presentation layer.  I’ll add more functions as needed to march toward the original goal.

~ ken.chen@simagix.com

Related Blogs

  1. Maobi - Reports Generator for Keyhole

  2. Survey Your Mongo Land

  3. Feel The Pulse of Mongo

  4. Index Responsibly

Comments

Popular posts from this blog

Build and Download Keyhole

MongoPush - Push-Based MongoDB Atlas Migration Tool

Survey Your Mongo Land