What the UEC is Good For?

Is it ready for production usage?

At the moment the private Ubuntu Enterprise Cloud (we tested Ubuntu Server Maverick Meerkat 10.10) is not yet ready for production usage. There are a lot of issues with the images and getting the cloud to launch the instances with consistency. Furthermore Hybridfox, the Firefox add-on tool to manage the cloud, is still a bit rough on the edges e.g. the sorting tool is not always functioning properly. Not to mention that there’s really no simple way of creating backups and/or snapshots of the instances.

The successful testing

We did not get the UEC to do what we planned originally. But we did get it to do something. We concluded that the UEC functions well as a tool for short-time projects such as rendering or folding@home purposes, after finishing the work you’d take the system down and set it up for the next task.

When trying out the Folding@Home, we successfully installed Origami on 64 VM instances. Arttu got Irssi IRC-client and WordPress client running in the VMs, but due to the lack of backups and/or snapshot functionality, these were lost after a reboot. We also wanted to try out running desktops in the VMs, but the instances we got to launch didn’t have enough harddisk space to finish the installation, so worked around it by installing a KVM in the VM and installed successfully Slax and Mint Debian.

A big cloud

The cloud at its biggest was about 55 nodes. This was because the resources we had in use were limited to three laboratories with 20 computers in each. We thought that there would have been no reason for us to build a bigger cloud had we had more resources at our disposal.

Conclusions

There’s not much we can say about the next iteration, Ubuntu Server 11.04 Natty Narwhal, and if the mentioned shortcomings are fixed. Our testing environment was limited and in other usage during our testing, so we were really unable to do anything without someone else using the laboratory network. This did create conflicts such as when someone set up their own DHCP in the lab net our nodes registered the IP-addresses from the new DHCP instead of the lab’s DHCP server.

All in all it is great that Ubuntu is developing such tools for open source communities, and after further development it could be an excellent tool for small enterprises to create their own private clouds.

Demonstrating the SilverCloudProject 15.12.2010

The last week of the project has kicked off, and we’ll be demonstrating our project tomorrow 9 am Wednesday 15.12.2010 in room 5013 on the 5th floor of Haaga-Helia Pasila.

There will be a magnificent spectacle of VMs running inside KVMs: the formidable Slax vs the legendary Mintdebian – which will run more smoothly in a virtual virtual enviroment?!

silverclouder@silvercloudproject: ~$ echo Welcomes you all!

Origami in SilverCloud

As the project is turning in to it’s final phase, we have been discussing how we are going to demonstrate the usage of the cloud. One thing that Kaarlo wanted to point out was that how we would be able to demonstrate the calculating power we have in the cloud. So he suggested that we’d take part in the Folding@home -project by installing Origami on our VM’s. This is where the scalability factor would come to nice play, when we add nodes to the cloud, we’ll be able to increase the amount of VM’s calculating Origami. Arttu wrote a script how to install the Origami on VM’s by just adding the last digits of the ip-address the CC is issuing the VM’s.

for S in ( add here variables for S ) ; do screen ssh -i id-silverkey 
ubuntu@172.28.5.$S 'sudo apt-get -y install origami 
&& sudo apt-get -y install ia32-libs && 
sudo origami install -t 197910'; done

At the moment the cloud is running on 13 nodes with 30 VM’s.Here’s our projectgroups stats page. We are thinking that this would be a good way to measure how the cloud can be used for e-science.

What, Cloud?

There’s much talk about cloud computing and cloud services, and not everyone’s familiar with the concepts. So only to clarify everything to myself I’d like to figure out which is which, and what are these grid computing, server farms and data centers.

If we go back in history, and think of the first schema’s telephone engineers draw, they used a cloud pattern to represent the complex network of telephone lines. Naturally because the early internet used mainly these same telephone lines, they started to use the cloud pattern to represent the internet itself. Nowadays if something is on the internet, one can claim it to be in “the cloud”. But what are these cloud computing services then?

Ok, we take a bunch of hardware, connect them together and install an OS which knows how to communicate with each other, we have a little cloud. Now, if I take a file and upload it to my new little cloud, it’s not on the net. This is one point people tend to get wrong – cloud is not the internet. When I connect my little cloud to the net, and make it a service I sell people – say virtual hosting – it changes, and the files are on the net in the cloud.

This is what cloud services mean – a bunch of hardware tied together connected to the net. What’s different with regular hosting is that it is possible to scale these services according need. Need may depend on various factors – say certain time of the day, e.g. some services are not as active during daytime, you do not need the same processing power in the night as you might during day. You can rent 50 VM’s for the night; while you need 100 VM’s during the day to take care of the traffic. Roughly said, you save 25% costs!
Other way to benefit from the scalability is to do what Google and Facebook are doing while they are growing and growing they simultaneously are scaling up their services, with installing massive 10,000 node clusters to their cloud. They are not counting the power of the clusters, but the cubical space of the storage facilities they set the clusters in.

Grids Academics

As opposed to cloud computing, grid computing is net dependent. According to a major grid computing net site www.gridcafe.org, grid computing can be compared to the power grid. Like the electric we use in our day-to-day lives in our homes, or while working at the office we don’t think where the electricity is originated, and with that same principle we can just plug our computer in the grid and enjoy the unlimited computing power that the collective grid of the computers people are using in their homes provide.
As all computers in the world that are on-line do create considerable amount of processing power, gridding them together creates a resource that can be used for scientific research (e-science) such as simulating molecules in purpose of researching for new drug candidates, or for study of alternate fuels, such as fusion power.

Grids provide a good platform to create science, the cloud does as well. The difference is that a grid can be installed on home computers, which are connected with internet using a middle-software, and the computers are controlled by their owners, and the computing resource can be limited using the middleware – however the owner of the computer chooses.

In this essence grids are used by communities or even create communities around the usage. It can be a scientific community doing research as described before, or art community using the shared resource of the grid in rendering of a 3D animated short film.

Data Centers and Server Farms

While grids are mainly used by communities, data centers are used by large organizations that can afford them. Data centers are just what they sound like: large warehouses full of computer racks. They are used to store data and provide the needed processing power to run an organizations’ key business applications. They also often are used for cluster computing. You can have a cloud functioning in a data center as well as a server farm.

One example of such application that requires a massive amount of processing power could be rendering service special effects for a feature film (render farm). Or to run massive net applications like Google does with its mail service, maps, docs and the search engine.

The data centers of the day are so big, that they create massive amounts of heat and are built in arctic areas such as Iceland or northern Canada or even heartland Siberia to save in cooling expenses (Finland is marketing itself as a such cool country as well). What’s different with cloud compared to server farm is not the fact that farms are comprised of multiple computers like clouds, but that farms are not functioning under a single OS, instead there are OS’s in each physical drive instead of a collective OS with clusters and nodes, makes the biggest difference between the two. Both are scalable resources and both use virtualization technology.

Wrapping up my Cloud

There are many services already on the net, that are called “as a Service”, these are  referred as SaaS (Software as a Service), Iaas (Infrastructure as a Service), Paas (Platform as a Service) and Eaas (Everything as a Service). If you use these pay-as-you go manner on the net, they are services provided by public cloud service. Private cloud would be a service running in own data center for your own organization as an intranet or a private secured service on the net.

One of the best examples of Saas is Google with Gmail and Apps, with their hundreds of different applications developed for pretty much anything you can do business online with. What cloud service bring to the table in addition to the lower costs with on-demand scalability is the dynamic platform of virtual machines. You can run anything on a VM just like a regular physical computer, only that you are not restricted by your physical location. One only needs to have a working internet connection to access the cloud.

Cloud technology is going to keep on developing, but the question is how it is going develop – is it going to fuse with grid computing forming new virtual communities that not only share the computing power, but whole OS’s with shared desktops, such as you could imagine in a class room or in office or why not at home with your family? Tomorrow is a new day, and it’s going to be cloudy.

Sources:

M. Armburst, A.Fox, R. Griffith, A.Joseph, R. Katz, A. Konwinsky, G. Lee, D. Patterson, A. Rabkin, I. Stoica, M. Zaharia 2010: A View of Cloud Computing. Communications of the ACM

J. Anderson, Elon Univ. L. Rainie, Pew Research Center 2010: The Future of Cloud Computing. Pew Research Center’s Internet & American Life Project

www.google.com/a 2009: Cloud Computing – Latest Buzzword or a Glimpse of the Future. Google, Inc.

http://www.rackspacecloud.com/what_is_cloud_computing

http://www.gridcafe.org/

Still No Cloud In the Sky – Only Reflections

We’re almost in December, and finding a solution to the problem is in process. What’s happening now is that we’ve altered the basic architecture of the cloud so that the CC and CLC are on one removable track, and one NC on a different removable track. We changed it this setup, because we got one computer from the lab for our own use. We are thinking that this would solve the problem we have with losing the instances. After installing the cloud we tested it so that we ran a VM in the cloud and shut down the NC. The result is not looking good, the VM was still there, but we couldn’t connect it again with SSH.

Oh, and the winter is really getting on here in Finland – this morning the temperature got down to -20C!

No Save State

Not much to blog this time. Only that we’re stuck with the following problem. We are doing our project in laboratory class, that is in other use as well, and every time we wrap up things, and leave the lab for next class, our nodes get wiped with other OS’s. We have few mobile tracks we use for CLC, two CC’s and two NC’s.

The problem is now that, when we get in the lab next day, put in the mobile tracks and start up the cloud, we’ve lost most of our instances, and the ones we have left are scrambled up with the new IP’s the DHCP is issuing them.

One solution would be to backup the instances making them into tar-files, but at the moment Kaarlo’s not getting it work.

In addition, snow just keeps falling from the clouds in the sky. Once again Finland seems to be arrived in perpetual bliss that is winter with lots of snow. It’ll soon be Christmas, and we’ve a long way to go trying to figure out how we get the backups working with the instances.

A Few Good VM’s

A lot good things happened yesterday Nov. 17. Instead of using the images that can be downloaded from the Ubuntu Cloud User Interface Store, Kaarlo made the images himself. The whole problem with the VM’s running sporadically, originate from the images that can be downloaded from the Store. “Actually you can throw the images in the Store away” says Kaarlo, when asking if there were any way of getting them working.

The way to get an image that will work with consistency, is – and this is important – to do it on other physical machine than the CLC. Mikko found a solution from here. Apparently the image needs to be bundled in a guest OS, not the one running the cloud.

First you want the machine you are bundling the image to be able to communicate with the CLC, sothe machine needs to have credentials installed. Second the bundling is done with the following lines: (full description here.)

TIMESTAMP=$(date +%Y%m%d%H%M%S)
RELEASE=maverick
ARCH=amd64    # Or this might be i386
[ $ARCH = "amd64" ] && IARCH=x86_64 || IARCH=i386
UEC_IMG=$RELEASE-server-uec-$ARCH
URL=http://uec-images.ubuntu.com/$RELEASE/current/
[ ! -e $UEC_IMG.tar.gz ] &&  wget $URL/$UEC_IMG.tar.gz
uec-publish-tarball $UEC_IMG.tar.gz $RELEASE-$TIMESTAMP

Kaarlo opted to script it, so all he had to do after was to run the script, and after a while the image showed up in Hybridfox images tab. Running the image resulted fully functioning VM. The image was tested to run few times with 100% functioning VM’s.

Oh, and we got a nice fun snow cover on the ground this morning. The traffic was hell like it always is with the first decent snowfall.

Another Day In the Lab

Ok, we got a little breakthrough with the VM’s. On Nov. 15th we sat down again and set up the cloud. We have been struggling with the VM’s, and how to get them running in the system. The bug is described here: https://bugs.launchpad.net/ubuntu/lucid/+source/eucalyptus/+bug/566792

Rebooting the instances over and over again in Hybridfox, we managed to get around 10-15 VM’s running. In addition we got the VM’s running eight Irssi-clients. I guess one way to get around the bug with starting up the instances is just rebooting them over and over again, just like one documented solution suggests.

The installation process is described here (in finnish)

silver_log_01_installation.doc

The installation process is described here (in english)

silvercloud_install_description01

First Steps With the Ubuntu Enterprise Cloud

It’s been a shaky start, considering that we started the project with the Ubuntu Enterprise Cloud with fairly low knowledge of the whole system, and the first couple of weeks have been just about getting familiar with the system.

Now, that we got our project organization in order, we are going to start writing about the project much more. The blogging should have started much earlier, only everyone was too excited to get to work on with the Cloud itself, and blogging fell little bit on the background.

On the first session on Nov. 4th we got a basic Cloud running with few nodes, and managed to run a virtual machine on it after some struggle. problems we encountered were getting the Hybridfox to connect the Cloud and to get the virtual machine running on the Cloud. Hybridfox we got working by restarting the Firefox (too simple a solution, except in the Windows-world!) and virtual machines just began working by themselves after restarting the whole Cloud.

Nov. 9th we started working on a plan Kaarlo drew. The idea was to create a Cloud with a single Cloud Controller with two Clusters with a few nodes in both and to get multiple VM’s running. After first successful installation session we were optimistic about getting the Cloud working right away. The Cloud itself started working, but the VM’s didn’t start. We just couldn’t get a single VM running by the end of the day, we decided to try again next day.

The Plan Kaarlo came up with

Nov. 10th we started again from scratch, and managed to get a couple VM’s running after multiple restarts. But we weren’t able to connect the VM’s using SSH. Our thinking is that the IP’s we issued the Clusters were wrong, so we requested an IP range for our Cloud usage. And that’s where we are now.

SilverCloud Project launch

Hi!

I’m happy to make an announcement that SilverCloud project will be launched during the next month, as soon as a project plan (0.1) is accepted by the project coordinator Tero Karvinen.

BR,

Arttu