From Vague Interests to Actionable Ideas - Deep Dive Into the Production Project

Ideation Week is often described by Codesmith alums as “the hardest week of the program”. It’s counterintuitive at first glance. After more than a month of 11-hour days followed by back-to-back three-day sprints, what could be so difficult about an unstructured week of brainstorming? Eventually, my production project team would build an npm (Node Package Manager) package to facilitate data streaming with Apache Kafka. But it took us almost the entire week to narrow down Kafka as the technology the majority of us were most interested in working with, and another two days to hone in on specific problems we wanted to solve for Node.js developers working with Kafka.

Our demo application consuming streaming sales data from a Kafka cluster

Ideation Week is hard for two main reasons. First, there are almost no constraints on the technologies we could focus on, so narrowing down to one core technology seemed to reject other, equally appealing possibilities. Second, developing open source tools is more difficult than using them. That might sound obvious, but what I mean is that being a contributor requires one to be both attuned to the shortcomings of existing processes and invested enough to provide an alternative.

"I especially valued the time I spent pairing with my teammates, as each engineer brought their own temperament to the project and taught me something about myself."

• Day 1: All four team members made a list of technologies they were interested in working on and each narrowed it down to their Top 5. Interestingly, at this stage, we were only in complete agreement about a) wanting to work with GraphQL and b) developing our project in TypeScript. Three of us were highly interested in Kafka (the fourth was open to it but hesitant). We were also intrigued (distracted?) by several open source technologies with less mature ecosystems, like Svelte.
• Days 2-3: We started to research our technologies of interest more deeply while jotting down potential product ideas. We looked at open issues in GitHub repos and carefully dissected discussions of pain points among JavaScript developers working with Kafka and GraphQL. We also solicited the opinions of our own developer friends. Some of our initial ideas that ended up on the cutting room floor, such as a VS Code extension to detect dead code and a middleware monitor for Node.js servers, were not specific to Kafka or GraphQL at all. Ultimately, we decided against a general purpose devtool for our production project because we thought it would be a missed opportunity to develop expertise in a new technology.
• Days 4-5 (and the weekend leading up to Minimum Viable Product [MVP] presentations): We chose to focus on Kafka because we were interested in real-time data streaming. While there are good Node.js clients for Apache Kafka, the learning curve for developers building architectures utilizing Kafka in their web applications for the first time can be rather steep. We wanted to simplify the process by abstracting away the repetitive code otherwise needed to set up WebSocket connections between each app and the Kafka cluster.

While my summary of Ideation Week makes it sound like our team progressed smoothly from brainstorming to fine-tuning, the reality was much more chaotic and we often moved back and forth between stages.

Production Project Coding

After we set up the basic skeleton for the reference architecture of our package, we split off to implement individual features—though we often pair programmed at crucial junctures, where an extra pair of eyes was invaluable. I especially valued the time I spent pairing with my teammates, as each engineer brought their own temperament to the project and taught me something about myself. I consider myself an analytical thinker by nature, largely due to my academic training. As a result, my mind naturally observes all of the possibilities and obstacles that could arise prior to diving into a project. This tendency toward slow deliberation helped our team to reign in in scope creep and spotting small, insidious bugs, and it was complemented by other teammates’ more experiential, get-your-hands-dirty styles.

Schematic for AtomicKafka

By Launch Day, we had a library that was much more modular and customizable than we did at MVP, two demo applications showcasing how the library works, detailed documentation, and a smart-looking landing page. Tackling Kafka for our production project has further piqued my interest in distributed systems and event streaming, and the data nerd in me wants to dig into analytic pipelines for streaming data next. (That being said, I love the front-end too, and I even “don’t hate” CSS!)

I’m very proud of what AtomicKafka has been able to accomplish and welcome developers to create new issues and pull requests.

Blog written by Vicki Y., Codesmith NY Cohort 25