DYNAMOS: Dynamic Microservice Composition for Data-Exchange Systems, Lessons Learned

Data exchange has become increasingly important in modern business and research. Consequently, many initiatives are being developed worldwide to facilitate open data exchange in secure distributed market-places. Ideally, each party maintains control over their data and implements access through legal contracts, in the form of programmable policy. Such policy would express where the data exchange takes place, and who has access to the data. Inspired by how traditional Operating Systems abstract the complexities of computer architectures into standardized core functions, this research focuses on abstracting different data exchange patterns into a unified set of core data exchange microservices that adhere to agreed-upon data exchange policies. DYNAMOS implements a distributed data exchange platform and recreates real-life data exchange use cases. It is designed to be self-adaptive, utilizing extendable algorithms to generate dynamic microservice compositions and dynamically choose archetype patterns, influenced by policy, user input, or system events. In our study, we highlight key insights from our experience with a dynamic microservice platform. Employing sidecars for communication abstraction, protocol buffers for strict interface definition, and ephemeral single-use jobs for improved security emerged as pivotal strategies. However, these approaches do introduce a tradeoff between operational speed and especially system complexity.