Developer Experience: Complex Software Development

Assessing Codebase Complexity

We asked Marius to evaluate the codebase complexity at adjoe:

When evaluating the complexity of our codebase on a scale from 1 to 10, the answer varies based on perspective. In terms of the bigger picture and the integration of system components, the code base can be quite complex. However, within each individual service, we strive to keep things simple, avoiding unnecessary complexities. While some technical debt was accrued during the initial stages of our company, we are actively paying it off by implementing cleaner and well-architected solutions.

Source: Medium

The Cross-Functional Team Structure

Here we asked Marius to describe his organization’s R&D department structure:

At our organization, we prioritize cross-functional teams consisting of 4 to 10 developers. This structure ensures seamless collaboration between different skill sets, reducing delays, prioritizing feedback, and eliminating the need for cumbersome cross-team coordination. By grouping teams based on products or services, developers can work together end-to-end, enhancing efficiency and minimizing interdependencies. We don’t have an R&D department in the traditional sense of a team that only focuses on research. However, we conduct regular events, generally once a quarter for two days, where we encourage our developers to try out new technologies and experiment with new ways of solving our existing problems.

Identifying Delivery Process Blind Spots

The blind spot is in the delivery process?

One of the blind spots we are addressing in our delivery process is deployment safety. Deployment safety is one area we are currently looking to improve. When we deploy to production, we are on high alert for any new alarms or changes in our dashboards. When something goes wrong, we initialize a rollback. Having an additional safety net around these deployments, such as blue-green deployments or canary deployments, would allow us to deploy to production with greater confidence.

Source: https://imgur.com/

It is important for us to remove any kind of fear associated with deployments; we want a blame-free environment where people can express their ideas, run experiments and learn from any potential failure. To achieve this without compromising on the reliability of the system, we need to build for failure. The idea is for when one service goes down or has an issue, it should have as little impact on other parts of the system as possible.

Optimal Testing Phases with Short Feedback Loops

The next three questions were about feedback processes on development:

In our development process, we emphasize short feedback loops and advocate for testing at multiple stages. Local testing is highly encouraged, as it allows developers to catch issues early on. However, as the system expands, it becomes necessary to utilize pipelines and test environments. Rolling your changes out to production is always a test in its own right. No matter how much you test beforehand, there will always be at least a slim possibility of something going wrong in production. Robust monitoring and quick rollback capabilities are essential in mitigating any unforeseen challenges.

Effective Feedback Channels

Feedback plays a crucial role in the growth and development of our developers. Regular 1-on-1 sessions with team leads serve as an important feedback channel, providing an opportunity to discuss performance, address concerns, and ensure open lines of communication. This two-way feedback process enables continuous improvement and ensures that developers receive the guidance they need to excel.

The Three Pillars of Observability

We use all three pillars of observability: logs, metrics, and traces. We are currently using a mix of CloudWatch, Sentry, and Prometheus to gather these signals. Alarms from these three sources are then gathered in Squadcast to have a central overview. We have built an in-house solution that evaluates each alarm and sends it to the responsible team. The team that receives the alarm knows the services they are responsible for and can, equipped with logs, metrics, and traces, quickly find the root cause of any issue. We’re always open to new observability solutions.

Measuring Developer Productivity

The last question we asked was about measuring productivity:

We believe that measuring developer productivity is not a straightforward task. Instead, we emphasize open communication and regular 1-on-1 discussions to assess the performance of team members. By fostering an environment where expectations are clear and concerns can be openly addressed, we encourage a culture of continuous growth. Factors impacting productivity are considered holistically, ensuring a fair evaluation process.

What is Digma

A runtime linter that lets developers quickly identify risky code, potential errors, and bottlenecks in complex codebases. Digma is an IDE plugin that helps developers write better code faster by analyzing code runtime data. It enables rapid development in complex projects by linting and detecting issues as they appear, highlighting possible risks in code, and providing code change analysis and context.

Learn how Digma works: Here