reCluster

Description

I participated in this project during my Master's thesis. I'll start by citing it to explain what it's about:

ReCluster is a project that aims to provide clustering software that adheres to specific characteristics, addressing clearly identified and declared problems. From a political perspective, the issues that reCluster seeks to mitigate and resolve are related to the lack of reuse of decommissioned hardware that still possesses useful computing power but is not being utilized. This results in a reduction in the environmental impact of these resources by acting both on the actual reuse of hardware and the extension of the useful life cycle of devices such as computers, which represent a type of waste, the e-waste, that is particularly expensive and complex to dispose of. In the design and construction of reCluster, special attention has also been paid to energy efficiency, enabling the creation of instances that are extremely frugal in terms of energy consumption and resource usage. This characteristic, combined with the fundamental ability to operate with highly heterogeneous hardware, makes reCluster extremely flexible and suitable for use in critical conditions, compensating for the lack of dedicated infrastructures and machines. The first application field for this system is precisely that of self-hosting services within universities. From a technical perspective, reCluster utilizes and combines various external software components, including Kubernetes, PostgreSQL, NodeJS, GraphQL, and Prisma. It also includes its own components that allow the management of the cluster according to the criteria specified by the project itself. One of the requirements is to minimize dependencies on other software to keep the system overall lightweight and compact. The operating system on which reCluster is designed to run is Linux, specifically the Alpine Linux distribution. Custom disk images of this distribution are provided, with all the necessary packages pre-installed for execution. Depending on the workload provided, reCluster can scale up or down the number of active and used machines to deliver services. This goal is achieved by automatically turning machines on and off based on the need to have them available for workload distribution.

My contribution

I was the second developer of reCluster, and my focus was concentrated on some specific areas:

• Increased hardware compatibility
• Improved usability
• Monitoring
• Documentation

Since this is a project that I worked on almost full-time for several months, on this page I will limit myself to providing a summary description of what my interventions were. For a complete and detailed analysis, you can download my thesis from here.

• Here you can find my GitHub repository of the project.
• In this repository, instead, the software and 3D files of the device I designed are available.

Increased compatibility

One of the software programs on which the operation of the entire system depends (used for hardware benchmarking) was available in a compiled version with restrictions that prevented it from working with dated processors that did not have support for some instruction sets more recent than their production date. I implemented the detection of the problem and the possibility of compiling from source automatically, in order to guarantee the correct functioning of the dependency in question, testing it.

I then acted on the process of managing network interfaces during installation. The presence of Wake On Lan support was mandatory for the completion of the installation and the correct functioning of all software components. Thus, wireless interfaces and those without Wake On Lan support were excluded. I also modified the classification of interfaces, and the requirements according to which an interface can be considered valid and usable. This intervention was necessary in order to implement other functionalities that I will talk about below. In this way, the variety of supported device classes has been exponentially expanded.

One of the fundamental interventions was the implementation of additional strategies for the automated ignition of the nodes that make up the cluster. In doing this, I acted both at the software and hardware level. I modified the server component and the installer accordingly, acting at the database, API, service implementation and logical execution procedures level. The result was the implementation of 4 strategies:

• Always on: the nodes always remain on and automatic shutdown is never performed.
• Wake on Lan: the WOL standard is used where supported to turn on the nodes.
• Smart plug: a smart plug is used that takes advantage of the automatic restart of computers after power removal.
• Button press device: an IoT device designed and built by me is used that allows the actuation of the power button of the computers that act as nodes remotely and automatically when necessary.

Regarding the device for pressing the power button, I designed it in CAD, using Fusion360, and then 3D printed and assembled it. I also created the specific firmware and the tools to test and configure it.

Usability

To improve the usability of the software, I implemented a series of graphical interfaces that clearly guide the user within the installation process. In this way, those who use reCluster are facilitated and do not risk getting lost in the numerous configuration steps and choices. I also made sure that any case was covered in the logical flow, and all exceptions were managed clearly. I also automated the procedure where possible and necessary.

Monitoring

I also created a simple and effective web interface from which it is possible to monitor the status of the system. It was created respecting the architectural principles of the project, that is, using the narrowest possible set of tools, in order to guarantee maximum compatibility with any modern browser.

Documentation

Finally, I produced practical documentation, a real user manual, implemented within the TiddlyWiki tool, which allows easy updating and consultation, as well as the creation of links between the various sections. In this way, even a user who has never used reCluster can succeed in installation and use almost immediately. You can consult the documentation here.

Current developments

I am currently working to extend compatibility to ARM processors (for example, Raspberry), initially excluded due to the lack of the pre-compiled package of the Sysbench software, used for benchmarking. Since compilation from source is now available, it is not difficult to extend compatibility to this class of devices as well.

The project continues to be developed at the University of Trento. Here is the project page.