Computers

Kafka Cluster using x8 Raspberry Pi

3+5 node Kafka cluster made up of 8 quad-core 2GB RAM Raspberry 4s, 16-port Gb switch x2 6-port USB power supplies, total cost $300, capable of processing 158 messages per second comfortably competing with my $1400 dual CPU 16-core Xeon 128GB HP workstation that costs about $1.50 per day vs this cluster’s less than $0.25 per day and practically no heat generation and dissipation (using wattage monitor device). The next goal is to convert this cluster into a fully operational Kubernetes cluster and if successful add 8 more nodes with redundancy for high availability and add Spark and AI components to make it a general-purpose computing environment for my home use. I used another server with VMWare Workstation to generate messages on several topics to determine any performance differences, none was found at least in this smaller set of test scenarios. Each Raspberry Pi was loaded with Ubuntu 18.04 LTS operating system.

I used my spare time during this pandemic to experiment with newer technologies such as Kafka and Kubernetes. Kafka logs are saved on a 128GB flash drive on each Raspberry Pi board. The diagram below shows the logical architecture of the setup.

Kuberntes Cluster using x2 Turing Pi

Under construction...

High Performance Computer

Under construction...