Tag Archives: MySQL

Moving to a larger cluster

I’ve been building a larger version of my Banana Pi cluster. It was originally built with just four nodes, but I’ve extended it to 8. There are
now four database servers running MySQL which are synchronized using master/slave replication. There are four web servers with Nginx and uWSGI
running Pyplate.

I plan on adding another four Banana Pis as web servers. Pages are cached on the web servers, so most pages are served without needing to access
the database. There isn’t much load on the database servers, so it shouldn’t be a problem to have more web servers than database servers.

The web servers are synchronized using rsync. I’ve set up ssh keys so that I can use rsync over ssh without having to enter passwords. The web
root directory and the Pyplate directory are synchronized from a master web server to the workers in the web server cluster.

I’ve moved pyplate.com to the new cluster, and I’m working on new sites that will also be hosted on the cluster.

I’m using Ganglia to monitor both clusters. It’s almost set up the way I want, except the master web server doesn’t appear in the Ganglia UI.
I’ll check the settings in gmond.conf on the node I’m having problems with.

Advertisements

Running several sites on a Banana Pi server

I’ve been doing some work on Pyplate CMS.  I’ve upgraded it so that I can run several sites on a single server, and I’ve also added support for MySQL to Pyplate.  I’m calling the new version Pyplate Multi-Site.

Now I can create a very simple mass blogging system on my Banana Pi cluster by installing Pyplate Multi-Site on each server in the cluster and make sure that they are synchronized.  This is an improvement on the Raspberry Pi cluster which can only serve one site.

I haven’t released the updated code yet.  I need to do more testing and update the Pyplate installation scripts.

So far I have only tested Pyplate Multi-site with Nginx.  If I want to use it with Apache, I will need to use mod vhosts in order to set a different web root folder for each site.  I haven’t tested it yet, but I will over the next few weeks.

In the short term I need to keep working on the Banana Pi cluster and get the upgraded version of it online so that I can start building some new sites.

Python utility for managing a cluster of database servers

I’m still working on some code to manage replication on a cluster of Banana Pi MySQL servers. It’s coming on quite well. I’ve got to a point where I can

  • demote a master,
  • promote another master,
  • add servers to the cluster,
  • copy the database to another server.

I’ve published the code on Github at https://github.com/Pyplate/db-cluster-utils.  There’s information on how to use DB Cluster Utils on my Banana Pi site at http://banoffeepiserver.com/mysql/python-database-cluster-management-utility/.

I was contacted earlier today by the owner of tinyonestore.com, asking if I would mind if they linked to my site from their site. I guess they wanted to check that they weren’t going to swamp my site with traffic, but that hasn’t been a problem so far. Looks like they have some interesting products like the Radxa Rock Pro.

GlusterFS and MySQL clusters running on Banana Pi servers

I’ve set up a four node MySQL cluster using Banana Pi servers.  There’s one master and three slaves replicating a MySQL database.

4 Banana Pi servers

4 node Banana Pi cluster

The first time I tried this I set up a small two node cluster and set up replication by typing commands at the MySQL shell prompt.  I didn’t want to do that again when I set up a 4 node cluster, so I wrote some Python scripts to do the work for me.  You can check them out at https://github.com/Pyplate/db-cluster-utils.

For detailed information on how I set this up, check out this page:http://banoffeepiserver.com/mysql/automate-mysql-replication-on-four-nodes.html.

I’ve also built a cluster of Banana Pi servers for use with the GlusterFS file distributed system.  First I built a stack of hard disks:

4 SATA hard disks

4 SATA hard disks

I connected a 4 node Banana Pi cluster to the stack, and installed GlusterFS on each Banana Pi.  Data is replicated on two of the disks, so if one disk breaks, no data will be lost.  You can read about how I set this up at http://banoffeepiserver.com/glusterfs/setting-up-glusterfs-on-four-banana-pi-servers.html

4 Banana Pis and 4 hard disks

4 node Gluster volume with SATA disks running on Banana Pi servers

Once I’ve finished testing these clusters, I can integrate them with my cluster of Banana Pi web servers.

My latest site, BanoffeePiServer.com, has been running smoothly since I set it up. I’m really pleased with its performance. At the moment it’s hosted on a small cluster of Banana Pi servers. I now have 10 more Banana Pis, so I’m planning on building a more sophisticated cluster.

I’ve been working on adding support for MySQL to Pyplate. This will mean I can set up a cluster of database servers, and offload the database from the nodes running the Nginx.

I have got as far as setting up a MySQL database and importing data into it. I’ve modified Pyplate to remotely access the MySQL database instead of the local SQLite database.

Next, I need to set up three more database nodes, and set up database replication. Then when I make changes to one of the databases, the changes will automatically be flushed through to the other database nodes.

I’ve got a lot more testing to do, but eventually I will release an update of Pyplate with added support for MySQL.

Once I’ve installed the MySQL cluster, I’ll look into setting up four more nodes running the Gluster distributed file system.

MySQL and Python

I’ve just published a post about using MySQL with Python on  a Raspberry Pi.  It follows on from the previous posts about SQLite, and it’s only a basic introduction into MySQL.  It’s enough to show you how to install MySQL, set up a database, and write Python code to access it.

There are still a lot of SQL related topics to cover: accessing databases over a network, Object Relational Models and much more.  I’ll write more articles on SQL eventually, but I want to write some articles about PWM and GPIO first.