It feels like it was just the other week when I found out who my teammates were for the Open Source Product. Having just completed two smaller projects within a week, the idea of having five weeks dedicated to a production level application, was both exciting and daunting. When I learned I would be working with Tyler and Wesley, two really easy going and passionate engineers, I was thrilled. I knew right off the bat that we would be able to create a fantastic Open Source Product. What I didn’t know was how we would actually accomplish this, and what blocks we would end up facing down the road.
IDEATION WEEK
Pretty early on during the ideation week, we came across the topic of message brokers. While message brokers were not part of the core curriculum at Codesmith, we did notice a trend in the market which had message brokers experience listed as a common requirement for software engineer roles. Since nobody on the team had worked with message brokers before, we were all intrigued by it’s design and utility. And with that, what is said to be one of the toughest weeks at Codesmith, became time we dedicated to research on this amazing thing called Apache Kafka and how it can be used to handle large volumes of small bits of data at lightning speeds.
Initial diagram of application layout for Monokl
BUILDING THE OPEN SOURCE PRODUCT
We decided to build a developer tool, aimed at providing visualization and monitoring to a Kafka cluster. We started by diagraming our UI, and the different pages we envisioned for our app. We got to present our ideas to other members of the FTRI team, and share our excitement with them. We even got to work with a designer to create a logo that would convey what our app represented. All of these creative decisions were what sprung us into the deep dive of learning a suite of new technologies to bring our vision to life.
Logo and Icon we received from our designer
Now that we had the layout of our application, we had to learn more about the technology we were going to be monitoring, Apache Kafka. Learning the workings of each component within a Kafka cluster required extensive research that we didn’t fully anticipate. There was a steep learning curve, which ended up being a huge undertaking before we could begin writing any code.
After spending about a week researching and walking through tutorials, we had a solid understanding of Brokers, Producers, Consumers, Topics, and Partitions (all Kafka components that work together to send large streams of data), the next question we had to answer was how to actually access the metrics and incorporate them into our application. Enter JMX Exporter and Prometheus. The combination of these two were the key to unlocking access to the 50+ metrics available from an up and running Kafka cluster. It required more research and referencing other applications to learn that we actually needed to configure our Kafka cluster with specifications in order to connect it to Prometheus using JMX Exporter. But after we figured out how to do that, we were in metric heaven and could see the beautiful volumes of data.
Final product showing producer metrics for an active Kafka cluster
OVERCOMING CHALLENGES & LAUNCHING THE PRODUCT
All of this to say, our team experienced more than just a few blocks similar to the ones I’ve already mentioned, which required us to refine our scope for the MVP more than a few times. This was a huge lesson in learning how to prioritize tasks, manage our time, and when to work together or apart on a new feature.
The thing about going through tough technical challenges, is that you get to experience huge wins after you come out of them. It almost offsets the stress and frustration that comes with it, almost. In complete honesty, the challenges we faced were harder for some of us than others. This meant that if someone was feeling discouraged, it was the rest of our responsibility to be the cheerleader for the group.
When we had a win, big or small, we would celebrate those to the fullest. That’s what kept us being able to put one foot in front of the other. The experience of working on this Open Source Product significantly enhanced my ability to be an empathetic and compassionate engineer.
Throughout the five weeks we worked on our Open Source Product, I learned a lot about myself as an engineer and as a team player. As I take this next step in my career, I will bring with me the confidence to lead a team, vocalize my ideas, and communicate when I’ve hit blocks. By overcoming the challenges we faced, we became stronger engineers, and I would not trade this experience for anything.
Blog written by Savitri B., Codesmith Full-Time Remote Immersive Cohort 3