Ceph - Crush Map has Legacy Tunables

I upgraded Ceph from the old Dumpling version to the latest Jewel version. In addition to the OSDs not able to start up due to some permission settings on /var/lib/ceph (we need to change the permission settings recursively to ceph:ceph), I am also having this HEALTH_WARN messages:

indra@sc-test-nfs-01:~$ ceph status
    cluster d3dc01a3-c38d-4a85-b040-3015455246e6
     health HEALTH_WARN
            too many PGs per OSD (512 > max 300)
            crush map has legacy tunables (require bobtail, min is firefly)
            crush map has straw_calc_version=0
     monmap e3: 3 mons at {sc-test-ceph-01=192.168.3.3:6789/0,sc-test-ceph-02=192.168.3.4:6789/0,sc-test-nfs-01=192.168.3.2:6789/0}
            election epoch 50, quorum 0,1,2 sc-test-nfs-01,sc-test-ceph-01,sc-test-ceph-02
     osdmap e100: 3 osds: 3 up, 3 in
      pgmap v965721: 704 pgs, 6 pools, 188 MB data, 59 objects
            61475 MB used, 1221 GB / 1350 GB avail
                 704 active+clean

To resolve the problem is very simple, use below command:

ceph osd crush tunables optimal

indra@sc-test-nfs-01:~$ ceph osd crush tunables optimal
adjusted tunables profile to optimal

Ceph status after the adjustment:

indra@sc-test-nfs-01:~$ ceph status
    cluster d3dc01a3-c38d-4a85-b040-3015455246e6
     health HEALTH_WARN
            too many PGs per OSD (512 > max 300)
     monmap e3: 3 mons at {sc-test-ceph-01=192.168.3.3:6789/0,sc-test-ceph-02=192.168.3.4:6789/0,sc-test-nfs-01=192.168.3.2:6789/0}
            election epoch 50, quorum 0,1,2 sc-test-nfs-01,sc-test-ceph-01,sc-test-ceph-02
     osdmap e101: 3 osds: 3 up, 3 in
      pgmap v965764: 704 pgs, 6 pools, 188 MB data, 59 objects
            61481 MB used, 1221 GB / 1350 GB avail
                 704 active+clean

The warning messages related to the crush map are gone. Yay!

PS. Ignore the "too many PGs per OSD" warning, due to I have limited number of OSDs and too many pools and PGs on my test environment.

Source: https://www.mail-archive.com/ceph-users@lists.ceph.com/msg10225.html
Reference: http://docs.ceph.com/docs/master/rados/operations/crush-map/

New Templates: CloudLinux, SUSE Linux and Dokku PaaS

This article is originally posted on SimplerCloud's blog here.

In the past couple of weeks, we have added some new operating systems and applications into the OS templates line-up: CloudLinux, SUSE Linux Enterprise Server and Dokku PaaS template for Ubuntu.

CloudLinux



CloudLinux is a CentOS/RHEL-based operating system which is built and suitable for shared hosting environment. It provides isolation between users to avoid the “bad neighbour effect” where one user might be consuming a lot of the server’s resources such as CPU, memory or I/O, affecting the performance of other sites hosted on the same server. This, in turn, will improve the stability of the overall server.

CloudLinux also has some other features, such as CageFS, which provides virtualised per-user file system that uniquely encapsulates each user, preventing users from seeing each other and viewing sensitive information, thus improving security. Some other features of CloudLinux include MySQL Governor, SecureLinks and PHP Selector.

To select CloudLinux OS template, choose “CloudLinux 6.4 (64-bit) (excluding license)” from the list of OS templates when you make your servelet order. CloudLinux is not an open source operating system, you will need to purchase the license separately. More information about CloudLinux operating system can be found on their website.

SUSE Linux Enterprise Server




SUSE Linux Enterprise server is a Linux-based operating system built by SUSE, specialised to deliver mission-critical IT services efficiently and cost effectively. Similar to CloudLinux, SUSE Linux Enterprise Server is not free, and you would need to purchase the license separately.

Take note that the vm_initialize script to initialize the data disk doesn’t work on SUSE Linux, so you would need to initialize the data-disk manually by following the instruction in our knowledge-base article here.

To select the SUSE Linux Enterprise Server template, please choose “SUSE Linux ES 11 SP3 (64-bit”) from the list of OS templates when making your servelet order. More information about SUSE Linux Enterprise Server operating system can be found on their website.

Dokku PaaS

Dokku is a mini-Heroku powered by Docker and written in less than 100 lines of bash. Once it’s set up on your servelet, you can push Heroku-compatible applications to it via Git. This is suitable for developers who like to use Heroku as their PaaS (Platform as a Service), and this will provide your own, single-host version of Heroku running on your servelet. We are using Ubuntu 12.04.3 LTS operating system for the Dokku PaaS template.

To have the Dokku pre-installed on your Ubuntu servelet, please select “Ubuntu 12.04.3 (64-bit) + Dokku” from the list of OS templates when ordering your servelet. You would need to configure Dokku after the servelet is provisioned by following the instructions on our knowledge-base article here.

More information about Dokku can be found on their website here and on the author’s blog here.

SimplerCloud New Feature: Announcing Snapshot Backups to Secure Your Data

This article is originally posted on SimplerCloud's blog here.



Wow, it has been quite some time since we updated this blog. Amongst the things that kept us busy are upgrading our CEPH storage capacity and unrolling a major upgrade to CloudStack 4.2.

But the biggest thing of all is an important new feature that we’ve worked very hard to bring out  – snapshot backups.

A snapshot backup is a way for high-availability systems to avoid downtime and still enable backup by performing the backup on a snapshot—a read-only copy of the data set frozen at a point in time—while allowing applications to continue writing to their data (definition partly taken from Wikipedia).

This is a major feature crucial to any system administrator’s server management strategy. And we are pleased to announce that snapshot backups of our customers’ servelets is now available after this major upgrade.  With this new feature, customers can perform snapshot backups of both the root-disk and data-disk of their servelets, directly from our customer’s portal. This feature is chargeable based on the size of the snapshot backup and how long the snapshot is being stored in our storage system.

How do you make a snapshot backup?

To make a backup snapshot of either your root-disk or data-disk (or both), you can log in to our customer portal, choose Services > Simple Menu / Power Menu (depends on your servelet’s category). Click on the hostname of your servelet, then choose Storage > Disks. You will see the list of your disks (both your data-disk and root-disk).



To make a snapshot of either the data-disk or the root-disk, click on “Create a new snapshot” next to the disk. There will be a prompt: “Are you sure you wish to create new backup?”, click “OK”.



The window will then show the snapshot area of the storage section, showing the list of your existing snapshots, including the one you are creating. If this is the first time you have created a snapshot, then you will only see the snapshot which you are creating, with the status “Pending.”



Snapshots will take some time, depending on the size of your disk. Once snapshot is completed, the status of the snapshot will change to “Built” instead of Pending.

 

How can you restore data from the snapshot backup?

If you want to recover any data from any of your snapshots, you will need to open a support ticket with us, and let us know which snapshot you would like to restore data from. Please kindly provide us with two information:
- The date when the snapshot was created (e.g. 2013-10-12 22:51:17+0800)
- The disk name (e.g. #DATA-2080)

You can find the above two information from your servelet control panel > Storage > Snapshots.



We will then convert the snapshot into a disk volume, and temporarily attach the disk volume into your existing servelet, so that you can copy the data over to your servelet. Once you have done with the copying, we will detach back the temporary volume.

At the moment, the process has to be done manually. However, we are looking into possibility of making this process automated for the convenience of all our customers.

More information about our snapshot backup feature can be found on our knowledge base articles at this URL:

http://my.simplercloud.com/index.php?/knowledgebase/category/29/snapshot-backup/

SimplerCloud's New Feature: we now have free CloudFlare CDN on all Servelets

This article is originally posted on SimplerCloud's blog here.

Yes, you heard right.  CloudFlare’s CDN (Content Delivery Network) services are now available for all customers in their Servelet control panel.  There are no installation fees, and with this new feature, you can create, enable and manage your CloudFlare CDN directly through your Servelet panel so that everything is conveniently centralized.

What is CloudFlare?

CloudFlare is a multi-award winning CDN (Content Delivery Network) service which helps to accelerate and secure your websites hosted on your servelets by providing caching and proxying service for your websites. CloudFlare can help you to protect your websites from malicious visitors, save your servelet’s bandwidth and reduce the loading time of your websites.

CloudFlare also has multiple CDN nodes all around the globe, and the nodes will keep a copy of your website’s cache. All your websites’ visitors will be redirected and served by the nearest CDN node, thus increasing the speed and reducing the users’ website load time.

Read more about CloudFlare on their web site

How does CloudFlare CDN work?

Once you have activated the CloudFlare CDN service for any of your domains hosted in our name servers, our name servers will then make the necessary changes to point your website’s domain (normally http://www.YOUR-DOMAIN) to CloudFlare CDN network. At the same time, our control panel will connect to CloudFlare CDN network’s API to register your account and domain into the CDN network.

This allows CloudFlare to accelerate your website traffic, since all requests to your website will be routed through CloudFlare CDN network. CloudFlare’s advanced CDN technology will help to screen your website’s traffic from malicious visitors, cache static content of your website and optimise your web content.

What are the advantages of using CloudFlare CDN?

The advantages of using CloudFlare CDN:

1. Improve the overall performance of your website.
2. Protect your website from malicious threats.
3. Computer infection alert to visitors of your website.
4. Offline browsing mode of your website when your servelet is down.
5. Overall lower CPU usage of your servelet.

What are the caveats of CloudFlare CDN I need to take note of?

Some caveats and limitation of enabling CloudFlare CDN service for your website:

1. Your web statistics might not reflect the actual number of visitors to your site, because some of the requests are handled by CloudFlare CDN nodes and not handled by your web server directly.
2. Because CloudFlare CDN nodes cache static contents from your website, any changes to your content might not be reflected immediately when your visitors visit your page.
3. CloudFlare CDN service will only handle requests going to http://www.YOUR-DOMAIN, while requests going to [YOUR-DOMAIN] would normally still be handled by your web server.
4. CloudFlare CDN service (the free plan) doesn’t support SSL.

OK, I am very interested in trying the service.  How do I sign up?

The CloudFlare CDN service works together with DNS Hosting service, so you will need to use our DNS Hosting service (which is also free) for the CloudFlare CDN service to work.

To order the CloudFlare CDN service, login to the customer portal > Order > CDN Services. Choose “CloudFlare CDN” and then click “Order Now!”

How do I configure the CloudFlare CDN service?

To manage the CloudFlare CDN service, login to the customer portal and then go to Services > CDN Services > CloudFlare CDN. You will see the CloudFlare CDN console panel.

Click on “Managed Websites”. You will see the list of domains which are hosted in SimplerCloud’s name servers under “DNS Hosting service”. If you do not subscribe to DNS Hosting service or you haven’t add any domains under DNS hosting service, you will not see any domains listed here.

To enable CloudFlare CDN service for a particular domain, just click the grey cloud next to the domain name. Once the service is activated, the cloud’s colour will change to orange.

What will happen to my domain once the CloudFlare CDN service is activated?

Once the CloudFlare CDN service is activated for your domain, the DNS records for your domain will automatically be changed to point to the CloudFlare CDN service’s IP address rather than the actual IP address of your servelet.

Your website (e.g. http://www.YOUR-DOMAIN) will be served by CloudFlare CDN service rather than by your servelet directly.

Summary

Our control panel is now integrating our name servers (through our DNS Hosting service) and CloudFlare CDN services (through CloudFlare API), allowing you to enable CloudFlare CDN services for your websites hosted on your servelets with just a click of a button (a cloud image, to be exact). All the required processes: the creation of CloudFlare account, registering of your domain names into your CloudFlare account and enabling your domain name to use CloudFlare CDN services (by changing the DNS records of the domain hosted in our name servers to CloudFlare node IP addresses) are being done automatically in the back-end. And furthermore, the service is provided for free to all our customers.

So, what are you waiting for? Host your domain names in our name servers and try the CloudFlare CDN services for your website, for free.

Servelet Templates for cPanel and Plesk

This article is originally posted on SimplerCloud's blog here.

Good news for web-hosters!  We have released servelet templates for cPanel and Plesk. You can find them on the list of templates for plan M and above. With these templates, you will not need to install cPanel or Plesk manually yourselves. You can just select the templates (either CentOS+cPanel or Ubuntu+Plesk) and the control panel applications will be pre-installed for you, upon the provisioning of your servelets. You will just have to do the initial configuration, by accessing to the web interface of the control panels, for you to start using them, almost immediately! The templates are provided for free, for the convenience of all our customers. Take note that the templates come without licenses.

On top of that, you can also buy cPanel licenses from us. The price starts from $15.95 for cPanel/WHM VPS Optimized License, which is the most suitable license for your servelets. You will just need to provide the IP address and hostname of your cPanel servelet, and the license will be active immediately after you made your payment. It’s just that simple! 

We are also currently in the midst of talking to Parallels for us to start providing Plesk licenses for you, for the convenience of all our customers.

More information about how to install cPanel and Plesk on our servelets, and also on how to purchase cPanel license through our customer portal, can be found in our knowledge base.

Introducing SimplerCloud!

This article is originally posted here.

It has been very exciting days (and nights) for all of us here at SimplerCloud in the past month. Building a top-notch and world-class, built-from-the-ground cloud infrastructure is not easy, but we are very happy that we are now coming to this stage that we are ready to roll-out our services. We built SimplerCloud after realizing that there are no Asian cloud computing providers who can provide a true automated and 100% self-manageable cloud servers with high-performing, fast SSD servers and painless, instant deployment and on-the-fly scaling.

The main features of our services are our virtual machines (VM), which we call Servelets. They are on-demand, scalable virtualized servers accessible over the Internet. They are set up instantly, very easy to manage, and extremely reliable. Those Servelets are deployed in real-time and will be ready for use just minutes upon confirmation of the order. The most important thing is that they are more affordable than anything found in the market today.

And when we say it’s affordable, we really mean it. Our smallest XS Servelet plan only costs US$10 a month, without any contract commitments. This comes with 1 GB RAM, 20 GB blazing-fast SSD storage, 1 vCPU core and 1 TB of data transfer. It will be very difficult to get that kind of price for a cloud server, not only here in Singapore, but also in Asia.

Of course, our ability to bring the price down doesn’t sacrifice on quality. We want to provide cloud services which are easy, instant and reliable. All our servelets sit on very fast systems powered by SSD (solid-state) drives in a distributed and scalable storage system on high performance 10Gbps links. SSD uses integrated circuit assemblies as memory to store data persistently, without any moving mechanical components, unlike traditional hard drives. The result is a blazing fast I/O performance for your servelets, 3-10 times normal hard drives.

In addition to our Servelets, we also provide additional services such as domain registrations and SSL certificate. For domain registrations, we are currently providing TLD (top level domain) registrations for .com, .org, .net, .biz, .info and .name, starting from US$ 14.95 per month! More TLDs will be added soon, including country-level ccTLDs such as .SG and .ID. We also provides free DNS hosting for our customers. As for SSL certificate, we provide basic and wildcard SSL certificates starting from just US$ 29.90 per year.

All the above services can be ordered and managed through our simple and easy-to-use, multi-lingual control panel. At the moment both our website and our control panel are in English and Bahasa Indonesia, and more languages will be added soon. You can browse all the services that we provide, order any of the services instantly and start using those services almost immediately! Payment can be made using Paypal or credit card. You can also manage all the services that you have with us, console in to your Servelets, manage your domains and more.

So, what are you waiting for? Please visit our website, create yourself an account on our customer portal (it’s free) and start jumping into the real cloud bandwagon.

Manual Compile of libvirt to Resolve CloudStack and Ceph RBD Storage Issue

I installed CloudStack 4.1.0 on Ubuntu 12.04.2 LTS (precise) server. Initially I wanted to use Ubuntu 13.04 (raring) but CloudStack only provides packages repository for Ubuntu 12.04. I used KVM as the hypervisor hosts, also running Ubuntu 12.04.2 LTS, and use Ceph RBD (RADOS Block Device) for the primary storage for CloudStack.

The default libvirt version on Ubuntu 12.04 doesn’t support Ceph RBD as primary storage. I followed this instruction from Wido to get libvirt version 1.0.2, which can support RBD storage pool support. However, I had an issue whereby the libvirt is reporting wrong RBD storage pool’s disk usage / allocation information.

root@hv-kvm-02:~# virsh pool-info bab81ce8-d53f-3a7d-b8f6-841702f65c89
Name:           bab81ce8-d53f-3a7d-b8f6-841702f65c89
UUID:           bab81ce8-d53f-3a7d-b8f6-841702f65c89
State:          running
Persistent:     no
Autostart:      no
Capacity:       5.47 TiB
Allocation:     34819.02 TiB
Available:      5.47 TiB

As a result, VM instance creation failed because the RBD storage pool is reported as having insufficient disk space and CloudStack wasn’t able to find a suitable /available storage pool.

2013-07-15 11:15:28,313 DEBUG [cloud.storage.StorageManagerImpl] (Job-Executor-3:job-168) Checking pool: 208 for volume allocation [Vol[227|vm=225|ROOT]], maxSize : 15828044742656, totalAllocatedSize : 1769538048, askingSize : 8589934592, allocated disable threshold: 0.85
2013-07-15 11:15:28,313 DEBUG [storage.allocator.AbstractStoragePoolAllocator] (Job-Executor-3:job-168) Checking if storage pool is suitable, name: sc-image ,poolId: 209
2013-07-15 11:15:28,313 DEBUG [storage.allocator.AbstractStoragePoolAllocator] (Job-Executor-3:job-168) Is localStorageAllocationNeeded? false
2013-07-15 11:15:28,313 DEBUG [storage.allocator.AbstractStoragePoolAllocator] (Job-Executor-3:job-168) Is storage pool shared? true
2013-07-15 11:15:28,317 DEBUG [cloud.storage.StorageManagerImpl] (Job-Executor-3:job-168) Checking pool 209 for storage, totalSize: 6013522722816, usedBytes: 38283921137336466, usedPct: 6366.305226067051, disable threshold: 0.85
2013-07-15 11:15:28,317 DEBUG [cloud.storage.StorageManagerImpl] (Job-Executor-3:job-168) Insufficient space on pool: 209 since its usage percentage: 6366.305226067051 has crossed the pool.storage.capacity.disablethreshold: 0.85
2013-07-15 11:15:28,317 DEBUG [storage.allocator.FirstFitStoragePoolAllocator] (Job-Executor-3:job-168) FirstFitStoragePoolAllocator returning 1 suitable storage pools
2013-07-15 11:15:28,317 DEBUG [cloud.deploy.FirstFitPlanner] (Job-Executor-3:job-168) Checking suitable pools for volume (Id, Type): (228,DATADISK)
2013-07-15 11:15:28,317 DEBUG [cloud.deploy.FirstFitPlanner] (Job-Executor-3:job-168) We need to allocate new storagepool for this volume
2013-07-15 11:15:28,319 DEBUG [cloud.deploy.FirstFitPlanner] (Job-Executor-3:job-168) Calling StoragePoolAllocators to find suitable pools
2013-07-15 11:15:28,319 DEBUG [storage.allocator.FirstFitStoragePoolAllocator] (Job-Executor-3:job-168) Looking for pools in dc: 6 pod:6 cluster:6 having tags:[rbd]
2013-07-15 11:15:28,322 DEBUG [storage.allocator.FirstFitStoragePoolAllocator] (Job-Executor-3:job-168) FirstFitStoragePoolAllocator has 1 pools to check for allocation
2013-07-15 11:15:28,322 DEBUG [storage.allocator.AbstractStoragePoolAllocator] (Job-Executor-3:job-168) Checking if storage pool is suitable, name: sc-image ,poolId: 209
2013-07-15 11:15:28,322 DEBUG [storage.allocator.AbstractStoragePoolAllocator] (Job-Executor-3:job-168) Is localStorageAllocationNeeded? false
2013-07-15 11:15:28,322 DEBUG [storage.allocator.AbstractStoragePoolAllocator] (Job-Executor-3:job-168) Is storage pool shared? true
2013-07-15 11:15:28,326 DEBUG [cloud.storage.StorageManagerImpl] (Job-Executor-3:job-168) Checking pool 209 for storage, totalSize: 6013522722816, usedBytes: 38283921137336466, usedPct: 6366.305226067051, disable threshold: 0.85
2013-07-15 11:15:28,326 DEBUG [cloud.storage.StorageManagerImpl] (Job-Executor-3:job-168) Insufficient space on pool: 209 since its usage percentage: 6366.305226067051 has crossed the pool.storage.capacity.disablethreshold: 0.85
2013-07-15 11:15:28,326 DEBUG [storage.allocator.FirstFitStoragePoolAllocator] (Job-Executor-3:job-168) FirstFitStoragePoolAllocator returning 0 suitable storage pools
2013-07-15 11:15:28,326 DEBUG [cloud.deploy.FirstFitPlanner] (Job-Executor-3:job-168) No suitable pools found for volume: Vol[228|vm=225|DATADISK] under cluster: 6
2013-07-15 11:15:28,326 DEBUG [cloud.deploy.FirstFitPlanner] (Job-Executor-3:job-168) No suitable pools found
2013-07-15 11:15:28,326 DEBUG [cloud.deploy.FirstFitPlanner] (Job-Executor-3:job-168) No suitable storagePools found under this Cluster: 6
2013-07-15 11:15:28,326 DEBUG [cloud.deploy.FirstFitPlanner] (Job-Executor-3:job-168) Could not find suitable Deployment Destination for this VM under any clusters, returning.
2013-07-15 11:15:28,332 DEBUG [cloud.vm.UserVmManagerImpl] (Job-Executor-3:job-168) Destroying vm VM[User|Indra-Test-3] as it failed to create on Host with Id:null
2013-07-15 11:15:28,498 DEBUG [cloud.capacity.CapacityManagerImpl] (Job-Executor-3:job-168) VM state transitted from :Stopped to Error with event: OperationFailedToErrorvm's original host id: null new host id: null host id before state transition: null
2013-07-15 11:15:29,125 INFO [user.vm.DeployVMCmd] (Job-Executor-3:job-168) com.cloud.exception.InsufficientServerCapacityException: Unable to create a deployment for VM[User|Indra-Test-3]Scope=interface com.cloud.dc.DataCenter; id=6

After consulting the CloudStack users’ mailing list and logging a bug report on Apache’s JIRA here without any success, I managed to resolve the problem by compiling and installing the latest version of libvirt. This is how I did it on my KVM hypervisor hosts running on Ubuntu 12.04.2 LTS servers:

1. Download the latest libvirt version (1.1.0) from libvirt’s FTP site, and extract it:

ftp://libvirt.org/libvirt/libvirt-1.1.0.tar.gz

2. Install the required packages for compiling libvirt:

apt-get install librbd-dev
apt-get install libpciaccess-dev

3. Compile libvirt with RBD storage support, and set the required prefixes to overwrite the existing default libvirt on the Ubuntu server:

./autogen.sh --prefix=/usr --bindir=/usr/bin --sbindir=/usr/sbin --with-storage-rbd

4. After the compilation has been completed, check the logs visually to confirm that RBD storage support is enabled, then do the installation:

make
make install

5. Restart the KVM hosts after the installation is done, and then verify that the latest version of libvirt has been installed:

libvirtd --version
virsh --version

The command “virsh pool-info” is now showing the correct “ allocation” amount:

root@hv-kvm-02:~# virsh pool-info d433809b-01ea-3947-ba0f-48077244e4d6
Name: d433809b-01ea-3947-ba0f-48077244e4d6
UUID: d433809b-01ea-3947-ba0f-48077244e4d6
State: running
Persistent: no
Autostart: no
Capacity: 5.47 TiB
Allocation: 328.00 B
Available: 5.47 TiB

CloudStack will then be able to utilise the RBD storage pool when creating VM instances.