Hassle-free Cloud Infrastructure Orchestrator

The DeepSquare project is building a sustainable and decentralized cloud computing ecosystem that allows anyone to request professional grade compute resources all over the world. Borderless, actor agnostic, and empowering professional cloud with decentralized infrastructure, this is the web as it should be. The ecosystem is made up of all the participants: engineers, artists, developers, end users, token-holders, applications, and service providers. And the overall vision is to empower the professional cloud with the decentralized, interoperable, and heterogeneous DeepSquare network in a sustainable and environmentally friendly way.

Thus, we introduce the newest addition to DeepSquare’s portfolio: Cluster Factory. Cluster Factory is a high-performance computing (HPC) multi-cloud infrastructure orchestrator where the software stack is designed and developed in such a way that it automates the process of managing, deploying, and monitoring of compute-optimized clusters. The software combines best-in-class solutions from HPC, Cloud, and DevOps to manage clusters along with the combination of GitOps practice which helps admins to manage their infrastructure in a reproducible and scalable way.

This state-of-the-art turnkey solution offers a comprehensive HPC (High-performance computing) stack that also includes Slurm, MPI, and DFS. This provides organizations the ability to deploy and scale their infrastructure layer in a unified and cloud-agnostic way. All of it is multi-cluster, and it will be the foundation of the software stack that is required to join the DeepSquare Grid. It was stated by Cristophe Lillo, the lead HPC architect at DeepSquare as well as one of the lead masterminds behind this solution:

“We look forward to getting a large amount of traction from High-Performance Computing centers and other types of companies that use large compute workloads as this is one of the latest compute-optimized infrastructure stacks”

This software is one of the latest methods to manage HPC clusters with all the benefits from Kubernetes and GitOps. While also using the “hardcore old school” characteristics that the HPC industry brings for well-optimized large compute workloads, it is a new way of managing infrastructure in a declarative manner. Thus without a doubt, Cluster Factory is a quick-to-configure and easy-to-use management solution. With regards to the time needed for cluster deployment, it was also stated by Lillo that:

“Depending on if the provider already possesses an HPC cluster in production and already has an idea of the services that ClusterFactory presents through equivalent experiences and configurations, it only takes a matter of hours, If someone already uses Kubernetes, it won’t be a problem as it is not required of the user to comply with the grid software stack. If someone is agnostic, everything can be deployed from scratch using the ClusterFactory”

Even though ClusterFactory can be compared to ClusterVision or Bright Cluster Manager, ClusterFactory is quite different and unique. While being open-source the software also merges various industries such as Cloud and HPC. In order to manage those, it uses Kubernetes, and none of the HPC cluster managers available nowadays is following this trend.

Also, ClusterFactory is GitOps compliant; this indicates that the Git repository is the source of truth for the infrastructure. Every engineer can track as well as modify their infrastructure straight from their preferred Git hosting service such as Github, this does not require them to interact with Kubernetes, and everything is deployed automatically through pull-based continuous deployment. As everything is configured declaratively on Git, all of it can be deployed in just seconds, in the expected state. ClusterFactory brings HPC to Kubernetes and it can be leveraged in regional clouds, HPC centers, and hosting providers to offer HPC – while also covering other workloads.

ClusterFactory also enables numerous compute servers to be fastened together, pulling the most of their computing power while avoiding the use of heavy abstraction layers (e.g. virtualization) and using more than one server at the same time to provide solutions to problems in a much more efficient way. Because the DeepSquare Grid is a decentralized network of high-performance computing clusters that is managed by ClusterFactory, it allows users to join the grid and capitalize on their spare computing resources.

“Since we believe in modularity, community-driven projects, and decentralization, ClusterFactory is open source, exceedingly configurable as well as GitOps ready to make the deployment of production-ready compute clusters simple and fun.”

Code and Documentation provides more information on the technical aspects of Cluster Factory.

The team is already focused on adding more to the core project and a large part of it is related to Data Management- DeepSquare User Portal which consists of user organization, permissions, and the data management part. This is an extremely important achievement and it’s just the start of our development, and there is a lot more to come! Said in short, the DeepSquare project is ready to challenge the status quo and disrupt the cloud industry by turning everything on its head. Thus it is headed towards a very bright future.