Agile software development with microservices

Organization

The app developers of the retail chain are not distributed horizontally among teams for backends, frontends, and middleware, but vertically, i.e., one workgroup develops all layers of a functional module.

If too many developers work on a large application simultaneously, focus and responsibility are lost. The individual developer can no longer think through all the technical aspects from front to back. With microservices modules, on the other hand, this remains possible. When back-end developers deliver complicated interfaces that front-end developers have to build on top of, trouble is programmed in the literal sense of the word.

The classic model of horizontal layering of apps and teams has had its day. Today, you slice both vertically along functionally complete modules that split into microservices. In this way, developers better understand the purpose of the app, get on with their work more quickly, there are fewer interfaces between teams, and the app composed of microservices is easier to maintain.

Methods and tools

How well and quickly developers work depends primarily on the available methods and tools. The ongoing integration (continuous integration, CI) and delivery (continuous delivery, CD) of the microservices play a decisive role in the success of the project. In CI, the iteratively developed contributions of the team are automatically inserted into the overall app, and their function and interaction are tested in the process. Continuous delivery means delivering the interim status after each programming round in order to accelerate the productive use of the software, obtain user feedback and optimize the app iteratively.

The entire software pipeline must be automated. Every rollout costs time and money. The developers contact IT headquarters, tickets for the rollout are created and processed, sometimes the team waits hours or days for the colleague at headquarters to press a button. Frustrating. No one needs that. Here, automation of the rollout saves the budget – and the nerves!

Deployment governance is also important, a procedure that records all changes and documents the current status of the app in a binding manner. Essentially, this means release management, including definition of the scope of functions and version control.

Only if every change to the app can be tracked in detail do we know where the error analysis should start in the event of problems in live operation.

High-level automation streamlines not only the distribution of software, but also the infrastructure on which the app runs. The preferred method of deploying new software quickly and securely is to run it in a virtual container, or containerization for short.

The rollout should include the software and the infrastructure on which it is built. The method to be used is called Infrastructure as Code or programmable infrastructure. Virtualization with containers has only advantages, no disadvantages. The company keeps its software stack under control, and the individual development teams can optimally support their rollouts. A container contains everything needed, from environment variables to libraries and the Java version to the runtime environment.

However, software may only be put into operation with strict monitoring and automatic messages in the event of critical conditions.

If we map a business process with several microservices, we need to know in detail how the interaction works. This is not easy. We need metrics for both the overall process and the individual microservices. Just measuring and analyzing the services is not enough. We need to monitor the entire process as a chain of microservices.

Each development team must monitor its microservices itself throughout the entire lifecycle. This responsibility must never be delegated to a central office. What you can delegate are the infrastructure tasks, such as: Which tool do we use for monitoring, who sets up the tool, who monitors it? Such tasks are even better handled by a central office, especially since it ensures efficiency in the teams by providing specifications.

Our interlocutor considers the widespread central storage and analysis of log data for process control, which is recommended by some sources, to be less relevant. However, a business process mapped with microservices must remain traceable in its sequence by means of so-called correlation IDs.

Experience shows that errors in distributed processes can hardly be found with the central log system alone. When analyzing it, the teams point their fingers at each other again. Even if the log system works with correlation IDs, the IT knowledge is distributed among the different teams and a certain blinkered mentality prevails.

Nevertheless, central logging proves useful. In competent hands, it helps to minimize the personnel effort required for error analysis. However, this is where technical experts are more in demand than developers. However, even if errors can be located and corrected more quickly, this does not necessarily mean that the entire development project is accelerated.

Work culture

A consistent agile working culture promotes fast, precisely fitting software development. However, the benefits of agile working can also be implemented at team level.

For agile, autonomous work, you need interdisciplinary teams. Only then can each team deliver a complete, well-rounded contribution. In addition, teams need to coordinate less with each other.

Small teams are ideal, especially since this prevents polarization within a larger department.

With small units, the project management effort is reduced. The teams work autonomously, and there are fewer interdependencies to consider during rollout. The key here is to choose a methodology that supports agile collaboration and scaling. Whether you choose Scrum of Scrums, SAFe, LeSS or the Spotify model is basically irrelevant. The main thing is to have collaboration structures in place in the first place. Of course, individual agile teams or projects will always run into walls in conservatively managed companies. Some will struggle with rigid structures and cornerstones. This costs energy, slows down software development, and wears down the agile teams. Nevertheless, software can be developed agilely even in a traditional work culture. Sooner or later, other departments and ultimately the entire company will want to work agilely. Often, it is even better to start small and transform one department at a time.

Freedom

Agile software development means giving teams largely free rein. But what are the limits of this freedom? Does it also extend to the choice of development environment, frameworks and tools? The decisive factors here are the time and costs incurred until a new developer is working productively.

A developer must be able to work in the new system environment at any time from day one. Companies should measure how long it takes for a new colleague to write the first line of code. A maximum of five minutes would be good. To achieve this, IT headquarters must do a number of things. It has to standardize, including the development environment, which will not suit everyone because it restricts the choice of work tools.

The operating system of the programming computer should not play a role. The developer should only have to install a Docker container with the working environment on his computer before he gets started. Only then is he independent of the computer and platforms.

The toolsets should also be prescribed. This saves many a discussion about preferred methods and languages. Standardization increases operational reliability not only from a technical perspective, but also from a legal perspective. This criterion is central for internally developed software.

In terms of technology, the freedom should be limited to choosing, for example, between a relational database, a document store, a NoSQL database or a Big Data bank. From the wide range of options, IT headquarters should provide one at a time. Matching the function of a microservice, the team determines the type of persistence of the data processed in it and then uses the option given for it. However, when it comes to whether a team programs in Java, Scala or Python, I have a firm opinion: no to the free choice of programming language for each microservice.

There must be ready-made software stacks in the company on which the developers are trained and about which they exchange information. The choice of a particular version of the stack can be left to the team.

I would even specify frameworks and libraries. At the very least, the selection should be subject to IT governance. I doubt that the development teams are aware of all the legal consequences that can arise from the choice of a library. There are too many licensing models. If you’re not careful, choosing a library commits you to publishing the source code of all apps developed with it. So at the very least, the company needs a process for approving and controlling libraries and other working tools.

Specifications must be stored in a central location. Basically, the development, maintenance and enforcement of internal standards is a full-time job. Decisions on such specifications are made by committees made up of representatives of the teams affected, so-called guilds, in an open process. However, it is also clear that it is not possible to think up a stack variant for every potential problem.

Advantages of standardization:

The smaller the selection of methods and tools, the shorter the training period and the more routine the handling. In addition, the teams can reinforce each other at short notice if they are all technically on the same level. Training courses are also easier to organize. However, if a team experiments with alternative approaches on its own, it soon produces isolated solutions that can only be integrated into the work of the others with a great deal of effort.

Conclusion

The project experience of our interlocutor in agile software development differs in some points from the “pure doctrine” from the specialist literature. In the expert’s environment, a tighter structure has proven itself, which curtails the developers’ freedom of choice to a certain extent in favor of greater standardization. The extent to which this model is suitable for other companies would have to be clarified on a case-by-case basis through an inventory and needs analysis.

We thank our interview partners for the interesting insights.

The interview was conducted by our experts for agile software development with microservices Claus Ried and Dirk Siegel.