Keyhole - How It Began
Taking a MongoDB consulting engineer position was a surprised move to my family and friends back in 2017. I have been known as an imaginative and innovative software engineer and a great mentor. I rather read about new technologies or write software to solve problems than go to endless meetings. After I left Ericsson during its great force reduction, I had different thoughts and wanted to see how people would use MongoDB.
The Birth of Keyhole
Working as a consulting engineer, I am still a software developer by heart. I am a true believer of automating day-to-day tasks and use the time to do other more meaningful things. The fact is that I was already familiar with MongoDB as a product before joining MongoDB. I was a customer for more than 5 years and I created tools to monitor and analyze software and hardware performance including MongoDB.
The idea of creating a performance analytic tool started brewing during my second shadow with my manager Dean Johnson. I noticed the customer panicked when Dean asked him to install Java and POCDriver to validate a MongoDB installation on Windows servers. So, I thought why couldn’t we simply provide a tool for different platforms available to download and execute without installing an interpreter such as Java or Python.
In 2018, The Keyhole project began as a tool to validate MongoDB installation and simulated loads to measure transaction rates and performance. It worked, but there were already other tools available. I later added logs parsing, index evaluation, and templated random data population functions. All these ideas were borrowed from what I did in Ericsson, and they were about all the features in the early stage of Keyhole.
My colleague Brad Bishop was the one inspired me to add more analytics features to Keyhole. Internally, we had a script to collect MongoDB server status and a Python tool to generate reports. Both were contributions from my fellow colleagues but not actively maintained. After working with Brad debugging these tools, I gave up and decided to add these features to Keyhole. The Maobi project, a Keyhole reports generator, was also a result of this inspiration. I’ll write more about Maobi in another post. Brad and Jay Pearson became early adopters of Keyhole and Maobi. They provided many invaluable feedbacks and contributed to the success of the two projects.
I often found myself having free time when I was on the road. After an eye-sore experience from viewing small FTDC charts using an internal tool, I decided to add FTDC viewing functionality to Keyhole. I chose Grafana as UI to speed up my development, and focused on data processing. This venture did not go as smoothly as I have hoped because of little information available. Still polishing my Golang skills, I started with the mgo driver, the Go driver before the official mongo-go-driver, and reversely engineered to turn Keyhole into a single JSON server to support a Grafana interface. In late 2018, I presented Keyhole in one of the team meetings and found another open source projects supporter, Diana Esteves, who later provided information about publishing blogs to promote Keyhole.
By the mid 2019, Keyhole has become a reliable performance analytics tool for MongoDB. It has become popular among new consulting engineers. Diana suggested publishing blogs on what Keyhole could do. But, I was first reluctant to write about Keyhole because of the resistance from the team meeting. What changed my mind was Dean who saw Keyhole as a great tool to promote the MongoDB ecosystem. Since then three blog posts were published on mongodb.com.
Writing codes in Chinglish is ok, but writing blog posts was not. Gratefully, my colleagues came to rescue and help review my blogs. Many thanks for Diana, Jay, Emre Ucal, Lana Kikic, Eric Reid, and Eoin Brazil.
Keyhole is to promote the MongoDB ecosystem and my contribution to the open source community. It already has a lot of features to analyze MongoDB clusters. I'll continue adding new functionalities when necessary. Keyhole was actively enhanced and brought critics about the outdated blog posts. As a result, more posts will be published here following this one because I have total control on this site.