Streamline Keyhole and Maobi Reports Creation

-

 In this short article, I will share the instructions on how to streamline Keyhole and Maobi to create HTML reports.  Like I have explained in the Maobi - Reports Generator for Keyhole, Maobi was originally created for my own use and the idea was to create a web interface to present results instead of sharing text-based terminal screens.

During the time of developing Maobi, I wanted to review the reports first before sharing them.  Therefore, the design was to have Keyhole output metadata to a file, and Maobi can generate reports from the metadata.  Over the years, automating the report creation has been brought up many times, mainly from those who have strong wills to automate everything.  Beginning Maobi v1.7.0, I have made the automation possible by enhancing the API and also adding a -maobi flag to Keyhole v1.1.0.

Upload Metadata

To create an HTML report, you can upload the metadata, generated by Keyhole, to Maobi using a curl command.  To start Maobi, use the command below:

docker run -d -p 3333:3030 simagix/maobi

Keyhole writes metadata to files, and using different flags creates files with different suffixes.  The filename formats supported directly by Maobi are:

  • <hostname>-stats.bson.gz: output of keyhole -allinfo command

  • <hostname>-index.bson.gz: output of keyhole -index command

  • <hostname>-log.bson.gz: output of keyhole -loginfo command

In the example below, ${metadata_file} represents a metadata file, for example, host.example.com_27017-stats.bson.gz, and ${maobi_url} is the Maobi API, for example, http://api.example.com:3333/upload.

curl -s -F "file=@${metadata_file}" ${maobi_url} -o example.html

After executing the above command, a report, example.html, is created.

Keyhole Integration

A new flag -maobi was added to enable Keyhole to send the metadata automatically to Maobi and create an HTML report.   The ${conn_str} in the following examples is a MongoDB connection string, for example, mongodb+srv://user:secret@example.mongodb.net.  Note that if the -maobi flag is not included in the command, Keyhole works the same way as before.

Cluster Info and Metrics

The command below outputs two files; one contains metadata and the other is an HTML report.  Refer to my blog Survey Your Mongo Land for details about collecting cluster metrics and related information.

keyhole -maobi ${maobi_url} -allinfo ${conn_str}

Indexes Usages and Recommendations

The command below also outputs two files, a metadata file and an HTML report.  My blog Index Responsibly has details on collecting index usages and evaluating existing indexes.

keyhole -maobi ${maobi_url} -index ${conn_str}

Log Analytics

Three files will be created from each log file using the command below, and they are:

  • A metadata file

  • A TSV (Tab Separated Values) file

  • An HTML report

keyhole -maobi ${maobi_url} -loginfo ${log_file}

This Feel The Pulse of Mongo blog has additional details on log analytics.

What’s Next?

The initial idea was to design Keyhole and Maobi as two components and use API to exchange data.  Keyhole was designed to be a data collecting agent and Maobi, on the other hand, was for the presentation layer.  I’ll add more functions as needed to march toward the original goal.

~ ken.chen@simagix.com

Related Blogs

  1. Maobi - Reports Generator for Keyhole

  2. Survey Your Mongo Land

  3. Feel The Pulse of Mongo

  4. Index Responsibly
Location: Atlanta, GA, USA

Comments

Post a Comment

Popular posts from this blog

Build and Download Keyhole

-
Image
=== updated 2/6/2023 ================================================================ For MongoDB JSON logs analytics, see Streamline Your MongoDB Log Analysis with Hatchet . ================================================================================== Keyhole is an open source project and the source codes are available at GitHub .  It is recommended to build your own Keyhole executables from the source codes.  Alternatively, Keyhole executables can be downloaded from Docker Hub , and available executables are for Linux, macOS, and Windows. Build from GitHub Golang has to be installed before you can build.  The commands below show how to clone from GitHub and build a Keyhole executable: mkdir -p $GOPATH/src/github.com/simagix cd $GOPATH/src/github.com/simagix git clone --depth 1 https://github.com/simagix/keyhole.git cd keyhole go build main/keyhole.go I provide a build.sh script to build Keyhole on macOS or Linux.  Simply run the script: ./build.sh I encourage you to use the scr
Read more

Survey Your Mongo Land

-
Image
Keyhole was still under active development after I published the Peek at your MongoDB Clusters like a Pro with Keyhole: Part 1 post in December 2019.  Since then, I have added many features and also changed the format of output files from json to bson.  I’ll use this post to summarize new features and modified functions not available in the previous post.  Recap A couple months into Keyhole development, a clear direction was surfacing: I needed a tool to fetch all information of a MongoDB cluster in one shot.  I joked about this function the “Know the enemy and hit them where it hurts” feature.  The -allinfo option is perhaps the most commonly used one among users.  Using the -allinfo option, Keyhole collects host and build info, replication stats, server status, size and storage stats of databases and collections, index details and usages, and sharding configurations.  The information was collected from the primary member of a replica set or a mongos of a sharded cluster.  Later,
Read more

Maobi - Reports Generator for Keyhole

-
Image
The Maobi project was created to generate HTML reports by reading stats collected by Keyhole.  The word maobi is a phonetic translation of Chinese paintbrush or ink brushes, which are commonly used in Chinese calligraphy as well as Japanese calligraphy.  They are also used in Chinese painting and other brush painting styles.  When I was little, it was required in school to use an ink brush to write or to paint pictures.  It took practice and effort to create something pleasant to the eyes. The idea of naming the tool Maobi was a reflection of the time we all spent on preparing reports and presentations. Keyhole started as a command line data collecting and analytics tool. With more data to be presented, it became difficult to include all metrics within a terminal window. So, the project began. Maobi is like a coworker Keyhole can only dream of, one who is smart, friendly, stylish but not flashy, and has contributed greatly to Keyhole’s popularity. Installation The easiest way to
Read more