Category Archives: Uncategorized

Server Monitoring

I’ve tested a few different monitoring tools for my Banana Pi servers.  The ones I’ve looked at so far are pretty simple, but they would be OK if I was just monitoring one Pi.

I need to be able to monitor clusters of servers, so I’m going to take a look at Ganglia.

Python utility for managing a cluster of database servers

I’m still working on some code to manage replication on a cluster of Banana Pi MySQL servers. It’s coming on quite well. I’ve got to a point where I can

  • demote a master,
  • promote another master,
  • add servers to the cluster,
  • copy the database to another server.

I’ve published the code on Github at https://github.com/Pyplate/db-cluster-utils.  There’s information on how to use DB Cluster Utils on my Banana Pi site at http://banoffeepiserver.com/mysql/python-database-cluster-management-utility/.

I was contacted earlier today by the owner of tinyonestore.com, asking if I would mind if they linked to my site from their site. I guess they wanted to check that they weren’t going to swamp my site with traffic, but that hasn’t been a problem so far. Looks like they have some interesting products like the Radxa Rock Pro.

GlusterFS and MySQL clusters running on Banana Pi servers

I’ve set up a four node MySQL cluster using Banana Pi servers.  There’s one master and three slaves replicating a MySQL database.

4 Banana Pi servers

4 node Banana Pi cluster

The first time I tried this I set up a small two node cluster and set up replication by typing commands at the MySQL shell prompt.  I didn’t want to do that again when I set up a 4 node cluster, so I wrote some Python scripts to do the work for me.  You can check them out at https://github.com/Pyplate/db-cluster-utils.

For detailed information on how I set this up, check out this page:http://banoffeepiserver.com/mysql/automate-mysql-replication-on-four-nodes.html.

I’ve also built a cluster of Banana Pi servers for use with the GlusterFS file distributed system.  First I built a stack of hard disks:

4 SATA hard disks

4 SATA hard disks

I connected a 4 node Banana Pi cluster to the stack, and installed GlusterFS on each Banana Pi.  Data is replicated on two of the disks, so if one disk breaks, no data will be lost.  You can read about how I set this up at http://banoffeepiserver.com/glusterfs/setting-up-glusterfs-on-four-banana-pi-servers.html

4 Banana Pis and 4 hard disks

4 node Gluster volume with SATA disks running on Banana Pi servers

Once I’ve finished testing these clusters, I can integrate them with my cluster of Banana Pi web servers.

Setting up a home page on a Pyplate site

I was recently asked how to set up a home page on a Pyplate web site. You need to do two things to make this work:
1. Create a new layout in the Site Settings field in the Admin area,
2. Edit the database record for page /main.html to use the new layout when this page is served.

To add a new layout, go to the Admin Area’s Site Settings page, and look for the post layout formatting code. It should look something like this, depending on which version of Pyplate you’re using:

 if layout == "post_layout":
 return post_layout.format(
 banner=pyplate.getFile(pyplate.getCMSRoot() + "/template/banner.html"),
 navbar=pyplate.getFile(pyplate.getCMSRoot() + "/template/navbar.html"),
# navbar=pyplate.get_navbar(dbase),
 banner_ad=pyplate.getFile(pyplate.getCMSRoot() + "/template/bananapi_responsive.html"),
 sidebar=sidebar_str,
# sharing=pyplate.get_sharing_template(),
 sharing=pyplate.get_sharing_buttons(path, page.title,page.description),
 breadcrumbs=pyplate.get_breadcrumbs(dbase, page.posts[0]),
 posts=post_str,
 section_menu=pyplate.get_category_menu(dbase, page.category, True, "More from this category:"),
 comments=pyplate.get_comment_template(),
 footer=pyplate.getFile(pyplate.getCMSRoot() + "/template/footer.html"))

Copy that code and paste it directly below the post_layout code. On the first line of th enew block, change post_layout to homepage_layout

#
# Copy the post_layout and change the layout string to homepage_layout
#
 if layout == "homepage_layout":
 return post_layout.format(
 banner=pyplate.getFile(pyplate.getCMSRoot() + "/template/banner.html"),
 navbar=pyplate.getFile(pyplate.getCMSRoot() + "/template/navbar.html"),
# navbar=pyplate.get_navbar(dbase),
 banner_ad=pyplate.getFile(pyplate.getCMSRoot() + "/template/bananapi_responsive.html"),
 sidebar=sidebar_str,
# sharing=pyplate.get_sharing_template(),
 sharing=pyplate.get_sharing_buttons(path, page.title,page.description),
 breadcrumbs=pyplate.get_breadcrumbs(dbase, page.posts[0]),
#
# Instead of using the post_str variable to fill the posts field, 
# use this line 
#
 posts=pyplate.getPost(pyplate.getCMSRoot() + "/content/main.html", "/content/main.html", ),

 section_menu=pyplate.get_category_menu(dbase, page.category, True, "More from this category:"),
 comments=pyplate.get_comment_template(),
 footer=pyplate.getFile(pyplate.getCMSRoot() + "/template/footer.html"))

Next, set the layout field to homepage_layout in the database.  You need to edit the layout field for /main.html. If you’re using a newer version of Pyplate with imort/export control in the Admin Area, then you can just export the database. Open /usr/share/pyplate/database/postindex.xml, and look for the record that contains main.html. Change the layout field to look like this:

<layout>homepage_layout</layout>

Click the import button to import the changes into the database. Then restart your server.

If you’re using an earlier version of Pyplate with no import/export buttons, you need to copy the database from /usr/share/pyplate/database/pyplate.db to a PC. Then you can install Sqlite Studio to edit the database: http://sqlitestudio.pl/. Copy the updated file back to your server and restart it.

CSS Media Queries

I’ve updated three of my web sites (pyplate.com, raspberrywebserver.com and banoffeepiserver.com) to make them mobile friendly. I’ve been reading about responsive web design, and it’s pretty simple.

I added media queries to my CSS code. Media queries are a way of grouping CSS rules for different screen sizes. The widths of several divs need to reduced for smaller screens. For screen widths below 800px, the sidebar needs to be displayed below the main content. This is done by making the sidebar float to the same side as the main content.

I added the following code to the end of my CSS file:

@media (max-width: 800px) 
{    
    #banner
    {
        height:150px;
        padding-left:15px;
    }

    #bannerleft
    {
         text-indent:1px;
    }
    #bannerright
    {
         padding-top:10px;
         text-align:left;
         float:left;
    }
    #post_header
    {
        height:50px;
    }
    #midpage
    {
        width:100%;
    }
    #container
    {
        width:100%;
    }
    #sidebararea
    {
        width:100%;
        float:none;
        margin-left:0px;
        margin-top:0;
    }
}
/*@media (max-width: 800px) {
    #sidebararea
    {
        width:100%;
        float:none;
        margin-left:0px;
        margin-top:0;
    }
}*/
.bannerad { width: 320px; height: 50px; }
@media (min-width:500px) {
  .bannerad { width: 500px; height: 60px; }
  .main .navbar { min-width: 100%; }
}
@media (min-width:800px) { 
  .navbar { min-width: 1100px; }
  #banner { min-width: 1100px; }
  .main .navbar { min-width: 100%; }
  .bannerad { width: 778px; height: 90px; } 
}
@media (min-width:1000px) {
  .navbar { min-width: 1100px; }
  .main .navbar { min-width: 100%; }
  #banner { min-width: 1100px; }
}

Now when I reduce the width of my browser, the layout changes to suit mobile screens. If you open raspberrywebserver.com in a browser and reduce the width of the browser window, you should see the layout change. You may need to refresh the page after you change the browser width.

My latest site, BanoffeePiServer.com, has been running smoothly since I set it up. I’m really pleased with its performance. At the moment it’s hosted on a small cluster of Banana Pi servers. I now have 10 more Banana Pis, so I’m planning on building a more sophisticated cluster.

I’ve been working on adding support for MySQL to Pyplate. This will mean I can set up a cluster of database servers, and offload the database from the nodes running the Nginx.

I have got as far as setting up a MySQL database and importing data into it. I’ve modified Pyplate to remotely access the MySQL database instead of the local SQLite database.

Next, I need to set up three more database nodes, and set up database replication. Then when I make changes to one of the databases, the changes will automatically be flushed through to the other database nodes.

I’ve got a lot more testing to do, but eventually I will release an update of Pyplate with added support for MySQL.

Once I’ve installed the MySQL cluster, I’ll look into setting up four more nodes running the Gluster distributed file system.

ARM powered server farm

My Banana Pi site has been up for a few days now, and I’m very happy with its performance.  I’ve put my clusters together to form a nice neat stack:

ARM server farm

ARM server farm

The Banana Pi servers are in the rack on the left.  The other two racks contain Raspberry Pi servers.

Banana Pi and Raspberry Pi server farm

Banana Pi and Raspberry Pi server farm

You can read about how I set up the Banana Pi cluster here: http://banoffeepiserver.com/banana-pi-server-cluster/.

Banana Pi server cluster

Banana Pi server cluster

Banana Pi server cluster

I built a Raspberry Pi cluster a few months ago, and it’s been very reliable. Performance has been surprisingly good, and the cluster has shown to handle a reasonable amount of traffic.

I recently got my hands on four Banana Pis, a Raspberry Pi clone with a dual core processor and gigabit ethernet.  I’ve built another cluster and got it on line at BanoffeePiServer.com.  This cluster has half the number of nodes in the RPi cluster, but it’s much more powerful.

I’ve used a newer version of Pyplate CMS which is much more efficient, and I’ve used a much more efficient mechanism to synchronize the servers.  Pyplate uses an SQLite database, so there’s no need for database replication.  The entire CMS can be copied to each node with rsync.

This page explains how I built the Banana Pi cluster.

A new version of Pyplate CMS is now available

It’s a long time since I posted here. I’ve been working on Pyplate, my Python CMS. Over the last few months I’ve made a lot of changes to the UI. Most site administration tasks can now be done via the Pyplate UI.

I’ve modified Pyplate to use the WSGI server interface instead of CGI. With CGI, the Python interpreter has to be loaded everytime a script is executed. Using WSGI means that a Python interpreter is kept in memory between requests. When a script needs to be executed it can run immediately because the interpreter has already been loaded. This has no effect on page load times of static pages, but dynamic pages are served much more quickly. The difference is huge.

There are different implementations of WSGI for different servers. In order to use WSGI with Apache, libapache2-mod-wsgi must be installed. This ensures that a python interpreter is loaded into each Apache thread. The downside of this is that each thread takes up more memory.

With Nginx, people often use set up a uWSGI server.  Nginx must be configured to pass requests for scripted pages to the uWSGI server. The uWSGI server executes a script, and returns the result to Nginx, which then passes the page back to the user who requested it.

I’ve also implemented a very simple webserver based on the WSGI class in the Python library. This server can handles static pages. If a requested file can’t be found, the server hands the request to Pyplate.  Pyplate generates the page, and returns it to the server.

The Python server is not very fast. It’s slower than Apache, and enabling page caching doesn’t make much difference. On the upside, it is very easy to set up.

I’ve added some security patches to Pyplate. A security token is embedded in each form generated by Pyplate. When a user clicks on the submit button, the security token is sent to the server with the POST request, and validated. I’ve also made some changes to the cookie code to make it harder to steal user’s cookies.

User information is now stored more securely. A salt is used to encrypt each user’s password, and a different salt is generated for each user.

I think Pyplate has come a very long way. It’s now a much more complete, more usable CMS. It’s still pretty simple, and small enough to run quickly.

What’s next? I’m working on a site built on a Banana Pi cluster. I’m building it using Pyplate, and it’s going to be based on my Raspberry Pi site. I may add a few more articles to my Raspberry Pi site, but eventually I plan to get back to working on Pyplate.

 

 

Pyplate now works with Nginx

I’ve written an install script that installs Pyplate for use with Nginx.  You can read the instructions here: http://www.pyplate.com/install-pyplate.

I’ve set up a Pyplate forum so that you can ask questions about Pyplate and get help with any technical issues.