Self-Contained-Systems Light & Shadow

17 Minutes reading time

Self-Contained-Cystem, what is it?

Self-Contained-System, referred to the following short SCS, is an architectural approach. The idea is to map requirements using individual, functionally independent systems. These systems include everything for mapping the necessary requirements, starting with data management, the actual business logic and finally a complete user interface. This makes these systems self-contained.

Each SCS should be looked after by its own development team. In addition to the technical separation of the systems, this architectural approach should also facilitate the organizational separation. By avoiding dependencies, the teams also become more independent and therefore more flexible.

That’s the theory. In the following paragraphs I would like to address the advantages and also possible problems when implementing the SCS approach.

Important basics

The idea of defining functionally independent systems is not new. Eric Evans has coined the term Domain-driven Design, short DDD. DDD clearly focuses on the technical requirements and the resulting technical structure. DDD offers design patterns to document and to implement the technical requirements. In the DDD world is a professionally enclosed area a Bounded-Context.

The SCS approach is based precisely on these Bounded-Contexts and expands its definition. In addition to the technical logic, the user interface is also implemented in this context. An important point here is that when a functionality is called, there should be no synchronous communication with components outside the SCS. If communication takes place, it should only take place asynchronously.

This has several consequences. It makes the system more resistant to possible errors such as timeouts. The implementation of the logic is also simpler, since no patterns, such as a bulk head or a circuit breaker, are switched upstream of the communication and any fallback scenarios have to be implemented. The other consequence is the prevention of the further breakdown of the Bounded-Context into further artifacts, an approach which is also often called "microservices". Why now this? In the DDD world a Bounded-Context is a unit which encapsulates all the necessary logic. When a change is made to this unit, such as a database extension, this change will probably apply to more than just the database and covers all "layers" of Bounded-Context. Now, if these layers are not distributed across "Micro Services" the scope of such changes is easier to overlook and it might not result into the need of introducing breaking changes to external APIs.

The explicit denial of synchronous communication is also intended to reduce the risk of building a distributed monolith with far too many and slow interfaces. So the motto is: less is more. Or to put it more pragmatically: what is not there does not make mistakes.

A few words about the organization

The SCS architecture approach has consequences for the organization and the team structure. As already mentioned in the introduction, each SCS should be looked after by a responsible team. This means that the team needs all necessary skills to do so. These competencies would be, for example, product ownership, development and QA competencies and also Ops or UX. By bundling these competencies in a team, the team becomes capable of acting and can pick up speed in the implementation of new features. So ideally every team member is in the "full stack".

Architecture decisions affecting only one SCS are called Micro-Architecture decisions. These decisions can be made and revised quickly as the scope is limited to an SCS. There are local decisions, such as which Java version, which framework, whether a hexagonal structure is used or which persistence API is used.

This contrasts with Macro-Architecture decisions. These decisions affect the entire system landscape, usually have a wider scope, require more coordination effort and are usually more difficult to revise. Such decisions are, for example, the number of SCSs, which cross-system protocols are to be used or which standards for the presentation technology are to be complied with. Macro-Architecture is therefore also and mainly organizational architecture!

How big can an SCS and the supervising team get? That is a very interesting question and there is no absolute answer to it. One team should oversee one SCS, possibly several, depending on the technical structure and the resulting complexity. The driving factor behind the team size is therefore exclusively the professional scope.

In SCS so it is not just a collection of technical principles, but rather a guideline for organizing development teams. Without adjustments within the organization, the approach cannot show its advantages. Ideally, this corresponds to the organization of the Context-map from the DDD world.

Data storage

An SCS should contain all functionalities that are necessary for the execution of the use cases it contains. This also includes the data required for this. This poses a number of challenges at the data storage level. An " order management " SCS probably needs the data for the orders and also has data sovereignty over this data. It probably also needs the customer data, but may not have data sovereignty over this data. It uses this data, is dependent on it, but does not manage it. How can this problem be implemented?

The answer is: data replication.

With data replication, the technical replication of the data is meant, and not the technical replication, as would be possible with a master-slave database setup, for example. What’s the difference here? Replication on a technical level, i.e. via DB replication, would reveal too many implementation details and tie the systems together too tightly. Changes to the database schema no longer only affect one SCS. Technical and organizational dependencies would arise, and this is exactly what the SCS approach is supposed to prevent.

Another argument is that not all systems need a full copy of the data, and they probably need to access the data differently. Certain data may also not be allowed to leave the original system, as they are particularly in need of protection. For example, system A can also be optimized for write access to customer data, and system B can only be optimized for read access. This then results in other requirements for the database to be used, the database schema, forms of normalization, etc.

What are the options now? Event sourcing in conjunction with a transaction log has proven to work well. A change to a customer data record in the " customer management " SCS would generate the event "customer was changed" with the new customer data and write it to a transaction log. The " order management " SCS would consume this event from the transaction log, extract the data necessary for order management and write to its own database.

Solutions like Apache Kafka are very interesting as a transaction log. In addition to the fact that this is pull solution and that the complexity at the Kafka server level can be kept very small, another aspect is much more important. Kafka enables a journey through time! The consumer of events can reset his "offset" within the transaction log in order to process events from the past again. This functionality is important for a disaster recovery!

Errors can and will happen, be it technical problems or just bugs in the software. Through the time travel we can install a new version of the software, consume events from the past and repair any broken data sets afterwards. A functionality that turned out to be very useful!

A disadvantage of this data replication is the resulting redundancy and the associated space consumption. If, for example, several million item data records are replicated across multiple systems, the size should not be underestimated and can also be a cost driver. The license model of the database must also be taken into account from a cost perspective. Each SCS should get its own database server, for example, during maintenance work on a database server, this maintenance does not have an effect on multiple systems. Licensing costs can explode. Here it is important to weigh up the risks, advantages and disadvantages and to find a pragmatic solution, which is also documented and communicated.

When it comes to data replication, the issue of data protection must not be neglected. Where can customer data be stored? What about credit card details? What options for auditing must be provided? Which data must be subsequently erasable? Which data must be anonymized, possibly also afterwards? Does this only affect the target system or also the transaction log in between? What about GDPR compliance? Are there any additional requirements? The responsible data protection officer must be consulted in any case, because technical data replication can very quickly result in a technical and legal problem.

The most important thing, the business logic

When implementing the business logic, there are a few things to consider from a conceptual point of view. SCS build on a core of approaches from Domain-Driven design. Domain-driven Design also helps in the implementation of the business logic. Patterns such as aggregate, entity, value object and repository help to structure and implement the business logic well. A hexagonal architecture further supports these patterns and helps to obtain a modular and easily testable system structure. Time to learn and deepen your knowledge about Domain-driven Design is never wasted!

Problems arise with the implementation of use cases that cannot be completely implemented within an SCS because, for example, suppliers or business partners are involved. We stay with the example of the " Order Management " SCS and look at the application, "Buy and stock reserve four items" more closely. When a customer orders something, the order should be booked in the system. Depending on whether the item is still in stock or not, the stock should be reserved for the customer or a reorder should be initiated. The result of the processing should be displayed to the customer.

This use case has it all!

Posting an order in the system is easy to implement. The reservation of stocks and the reordering are interesting. The warehouse management is probably from another SCS is responsible, just as the re-ordering. Communication should take place asynchronously in order to decouple the systems. Nevertheless, the result of the processing should be displayed to the customer after the order has been received. What options do we have? From the point of view of design patterns, the SAGA pattern is interesting. The SAGA pattern is intended to help implement distributed transactions. In addition to the technical implementation of the pattern, the technical requirements must also be implemented, such as calling up warehouse management and re-ordering including error handling! From a technical point of view, this is already challenging enough. In this case, error handling in particular is driven more from a technical than a technical point of view. What happens, for example, if the call to warehouse management takes too long? What happens if an error occurs while reordering? Should the system try again? How do I stop processing completely? And how do I get the whole thing meaningfully integrated into the SCS?

One possible solution here is to use a lightweight workflow engine, which in the "Order Management" SCS embedded is. A Workflow-Engine brings with the technical structure necessary for the correct implementation of the SAGA patterns. The workflow definition can very well with those in charge are coordinated and transparent. The focus on the implementation of the business logic is not lost. Embedded workflow engines have another advantage: as a rule, the workflow definition forms a unit with the SCS. A change in one usually also affects the other. It therefore makes sense to version, package and also deploy them together.

In this context, is still said that the call of another system should not be a synchronous call, as cross-system communication should be asynchronously. The call is the sending of an event to the target system and waiting for a corresponding response event from the target system including timeout handling, etc. Workflow engines are particularly helpful when it comes to modeling and implementing such functionalities!

We coukd implement the "Buy and inventory reserve" usecase like this: When you click on the "Buy" button inside the " Order Management " SCS the corresponding workflow on the workflow engine is started. If this workflow runs through successfully within a few dozen milliseconds, the order management can respond to the customer with the corresponding status of the order. If the workflow does not go through completely, we can also inform the customer of this with a corresponding note, and then later notify him of the processing via e-mail or push notification. This mechanism helps to implement the application correctly to scale better peak loads on systems and also helps to implement user-friendly error handling.

I already advertised Apache Kafka, now I also advertise a workflow-engine. An interesting and very lightweight solution is for example Zeebe from Camunda.

Business logic should be encapsulated in an SCS. However, this does not mean that Java, for example, has to be used to implement all use cases. Especially with data-intense applications, it may be useful to rely on SQL instead of ORM frameworks like JPA to use. The use of SQL does not subvert the encapsulation, it just brings business logic and data closer together. This is an example of the microarchitecture that is locally restricted to an SCS. These architecture and design decisions should be made by the responsible SCS team.

The user interface

Each SCS should contain the complete user interface for the use cases it contains. Ideally, it is a web application, i.e. the user interface is HTML, CSS and JavaScript.

Whole books can be written on this subject alone. In connection with SCS, however, special attention must be paid to the integration of several SCS into a uniform user interface.

As a user, I would like to use different functionalities. These functionalities should lead to the responsible SCS via "Link". Taking any online shop as an example, the question arises as to where, for example, the top navigation that is displayed on almost every page comes from. Should this functionality be implemented redundantly in several SCS?

The answer is of course: no.

There is a system that is responsible for top navigation. Another system is responsible for calculating recommendations, for example, and another system can, for example, generate article lists based on filter criteria. Each of these systems delivers an "HTML snippet" for precisely this part of the page.

The individual page components must be combined into one unit. There are several options here. One possibility would be to use a content management system that does the aggregation. This option is the most powerful, but it also raises the organizational question of who is in charge of and further developing this system. In addition, new dependencies between competent teams emerge and not least a single point of failure for the whole architecture. Another, somewhat lighter weight option is to integrate the page layout and placeholders for non-system content into the SCS in the form of templates. These placeholders could be SSI directives (server-side includes) or ESI directives (edge-side includes). The resolution of these placeholders would then be carried out by an upstream web server or varnish cache. In practice, I would rather recommend a lightweight solution and expand it later if this is really necessary. Once it has been brought into production, it is difficult to reduce complexity.

When it comes to ESI or caching, details need to be taken into account when it comes to personalized content. We want to achieve the highest possible cache hit rate. However, personalized content is usually very difficult to cache. One solution is to always display the same basic HTML structure for all users and to reload the personalized content via JavaScript. This strategy also makes sense without a cache in the background. In connection with Progressive-Enhancement interesting possibilities arise. Why, for example, deliver content when it is not even visible on the target device because the screen is too small? Why want to view videos when the target device can’t? All of these questions can only be answered on the target device and not on the application server. The server only delivers the basic structure of the page via semantic HTML. The rest is decided and assembled on the target device.

How do we get a uniform user experience? Is there a central repository for styles? What happens to JavaScript code? Every SCS has the styles and the JavaScript in the abdomen. The trick is to drive componebt-based approach in connection with a modularization of stylesheets and the JavaScript. A component is, for example, the top navigation or an article list. Each component has the necessary HTML, CSS and JavaScript. There should be as little global styles are defined as possible. This is the only way to implement functionalities independently and within an SCS and ultimately to build speed in development and short time-to-market. Here it is important to find a pragmatic way.

The consequence is that there is no longer a global stylesheet for everything. This is intentional, but from UX’s point of view it is a nightmare as a first reaction. However, if we take a closer look at large online shops such as Amazon, we also find that the page layout in the item lists is completely different than in the checkout. Probably because these pages are served by a different system. This has never bothered me as a customer, but as a computer scientist I notice it. Presumably, a pragmatic decision was made here between a uniform layout, independence from systems and teams, and development time and development costs.

This chapter is very HTML l knotty. In fact, HTML is basically the choice for the presentation layer when implementing an SCS. It becomes problematic when mixed presentation technologies have to be used. This is particularly the case when certain use cases are to be made available via app. If the app is a hybrid app and a WebView is already integrated, HTML can be used. However, if the app should use the native capabilities of the end device or be completely offline-capable, a different interface must be made available for the app. In this case, the SCS approach reaches its limit, as the user interface is no longer part of the SCS, and probably also the app development is no longer part of the actual SCS team. In this case, the compromise would be to add a (REST) API to the SCS and make this available to the app. The functionality then remains encapsulated in the SCS, only the app-specific part of the presentation layer is externalized.

Lessons learned

A lot more can be written about SCS as an architecture and organizational approach. Nevertheless, I would like to draw a line and this article with a short lessons learned here to complete.

SCS can help organize and structure the work of larger development teams. But also for smaller projects and teams it may be an interesting approach that the focus clearly on requirements without distracting on technical implementation details.

The principles behind Domain-driven Design find are wonderful application here. The principle, less is more, is also reflected in the current trends of software development and the return to a few, but important and correct "Services". As with all architectural decisions, this is not a matter of choosing between black and white. There is always a trade-off between advantages and disadvantages and life with the associated consequences.

Which framework and which programming language are the best to implement an SCS? In my opinion, they are all equally suitable as long as the technical requirements can be implemented with them and, of course, the development team can work productively and sustainably with them. The SCS approach should help to use the most sensible technology for each SCS. In practice, a compromise has to be found between innovative approaches and technological growth.

The core concepts from the SCS architecture can also be transferred to Swing or JavaFX applications. This approach is not popular, but it is possible. Desktop applications still have their authorization. An interesting combination from my professional past is Swing and Canoo UltraLightClient, ULC for short. ULC is essentially server-side Swing with a presentation layer that is distributed via WebStart. This example is a bit older, as the technology stack suggests. We didn’t call it SCS at the time either, but the approach was similar. If I wanted to get angry, I would say SCS can also be implemented with the Java portlet specification and a good, old J2EE application server. My point: with a little creativity, the concepts can also be profitable to something unfamiliar areas and also outside the Java world applied. Basically they are timeless and have only been rediscovered by SCS.

My experiences with the SCS architecture approach are positive. My only criticism is that the SCS "Manifesto" is coarse grained and ironically too many details are left open. The point “Macro-Architecture” in particular could provide more details and information, as this is exactly where the factors for a successful implementation are hidden. I hope this article has shed a little light on the darkness.

With this in mind, I would like to thank you for your attention. I am of course available for questions, comments and criticism. Many Thanks!

Links / Sources / References:

Self-contained systems - https://scs-architecture.org/

Evans, Eric : Domain-driven Design: Tackling Complexity in the Heart of Software

Git revision: b5b5126

Loading comments...