Category Archives: Behind the Scenes

Behind the Scenes: Security Patches & Private Network Updates

It has been a little while since we posted an update on what we’re working on behind the scenes here at Dathorn, so I wanted to take a quick moment to share. While it is easy to notice new features or see us working on your helpdesk tickets, much of our work goes completely unnoticed. That is, after all, our goal. We try to perform all updates, maintenance, etc. without any impact to your service.

While software patches and security updates are an ongoing battle, there have been a few noteworthy items as of late. You may have already heard about “Stack Clash”, a local privilege escalation vulnerability present in most Linux and BSD systems. Fortunately, we were able to quickly protect our servers against this without any service interruption, thanks to KernelCare. Traditionally, such kernel updates would require a reboot of each server but that has long been a thing of the past for us.

A few security issues have also been addressed in OpenVPN, including a remote code execution vulnerability. While serious, all of our instances were patched immediately and the severity of this particular issue for us was much lower since we only use OpenVPN internally for accessing certain private resources, such as IPMI on our servers. This had a much greater impact on VPN providers. Continue reading

Behind the Scenes: CentOS 5, LVE Stats 2, PHP 7.1 & EA 4

There’s been a lot happening here under the radar over the past month, so this gives us a great opportunity to post another of our “Behind the Scenes” updates. To start, we completed migration of all remaining CentOS 5 servers due to its end of life date, March 31st, quickly approaching. One of these servers was our primary DNS server (ns1) which was seamlessly migrated to a new CentOS 7 server without any service interruption.

CloudLinux’s LVE Stats 2 made its way out in a stable release, completely overhauling how system resource usage data is recorded on our servers. CPU and RAM usage is now recorded with much greater precision but perhaps most important is the new snapshot functionality. Now, when a CPU or RAM usage fault occurs, a snapshot of the account’s running processes is recorded. This allows you to go back and see what was running when a fault occurred, which is very helpful in identifying what caused it. Previously, no such information was available unless you actively witnessed the fault occurring.

PHP 7.1 was officially released and shortly thereafter available on all of our servers via CloudLinux’s PHP Selector. Our servers now offer PHP versions 5.4, 5.5, 5.6, 7.0 and 7.1. Version 5.6 is the default on new cPanel accounts and this can easily be changed via the “Select PHP Version” link in cPanel. Continue reading

Behind the Scenes: Server / PHP Upgrades & cPanel 58

It has been a little while since our last “Behind the Scenes” post so here’s another quick update of what’s been happening here at Dathorn. First, we are very happy to announce that we’ve completed the upgrade and migration of all of our hosting servers. Every single client is now on a server utilizing our latest hardware and software packages. This includes items like MariaDB, PHP Selector and, of course, pure SSD RAID 10 storage. The only mechanical drives that we’re using now are solely for backup storage and eventually these will be phased out as SSD capacity continues to grow.

Since all of our servers are now utilizing CloudLinux’s PHP Selector, we no longer have to perform scripted rebuilds of PHP and related dependencies as they are updated. As such, we will no longer be announcing these minor updates on our forums since they really don’t have any impact for our clients to be concerned about. You can always keep an eye on “Alt-PHP” updates on the CloudLinux blog if these items interest you.

Earlier this month we completed the deployment of cPanel version 58 across all of our servers. Most visible among the updates would be the changes made to Paper Lantern which you may have already noticed. There were, however, a number of system changes that aren’t so visible. One that you should be aware of is that the “Trash” folder on email accounts is now included in the email account’s disk usage. Believe it or not, this wasn’t the case before and it just didn’t make much sense. This often created confusion because email accounts would be reported as using far less disk space than they actually were. While this doesn’t change the cPanel account’s disk usage at all, it does allow email disk usage to be more accurately reported and managed, which is long overdue.

Going forward through October, we do still have some internal services that we need to migrate to new servers, such as our primary name server (ns1) and some backup servers. This is necessary as we approach the end of life for RHEL 5 / CentOS 5. We’ll be taking this opportunity to replace some older Adaptec RAID controllers with newer LSI ones as well. This maintenance won’t have any impact on clients but I did want to provide some insight on what we have planned here. That’s all for now, we hope you’re enjoying the cooler Fall weather as we are!

Behind the Scenes: Billing Migration & ImageTragick

We have successfully completed the migration of all clients from our old billing system (Ubersmith) to our new one that is integrated within our existing portal. For some of you, tomorrow will be the first time you’ll experience the new system when your service renews. We would love to hear any feedback you might have!

Being able to integrate our billing into our existing support mechanisms has been a tremendous help in simplifying processes both for us and our clients. We look forward to growing the available feature set going forward, including the ability for you to manage hosting plan changes without our intervention.

imagetragick

Earlier this month you may have heard of a new vulnerability in ImageMagick named “ImageTragick“. ImageMagick is a software suite used to create or edit many different types of images. One of the most common use cases involves their “convert” utility which is used to convert images from one type to another and resize them. cPanel, for instance, uses it as do a number of image gallery related scripts, shopping carts, etc. Because of this, the potential impact of ImageTragick was quite high.

Although it is fairly trivial to create a policy file or simply update ImageMagick to address the issues, care had to be taken to make sure all instances of ImageMagick were addressed. In many cases, our servers had two or three different versions of ImageMagick that needed to be taken care of. It’s not uncommon for it to be installed on the server as a general package in addition to the versions that both cPanel and CloudLinux provide. As always, you can rest assured that we’ve taken all possible steps to address these new attack vectors and will continue to monitor for further issues going forward.

Behind the Scenes: SRS, Billing Migration & Server Upgrades

The Sender Policy Framework (SPF) is a great tool to help validate email senders and detect email spoofing. We recommend that all domains have a proper SPF record configured for this reason. However, SPF has long caused problems with forwarders and this is now no longer an issue. As of cPanel 54, which was installed on all servers earlier this month, Sender Rewriting Scheme (SRS) is fully supported out of the box. Previously some forwarded emails would get rejected by the destination mail server due to the SPF check failing. SRS now automatically rewrites the envelope sender such that forwarded emails will still pass SPF checks. You don’t need to do anything to activate SRS, it was enabled on all servers on March 9th.

Last week another batch of clients was migrated to our new billing system that is built-in to our existing portal. The response has been overwhelmingly positive as it eliminates a lot of unnecessary confusion with having a separate billing system. We hope to have all of our clients migrated within the next month or two and appreciate any feedback you might have.

Lastly, we will be resuming our server upgrades in the upcoming month (April). These took a backseat while we finished up our billing system and continued evaluating the servers that had already been upgraded. Thus far we are very pleased with these upgrades and, like you, we can’t wait to have everything migrated to pure SSD environments. You’ll receive a notification when your server is scheduled to be upgraded if it hasn’t been already. We expect to have all remaining servers upgraded by the end of August.

Coming Soon: Server Upgrades

With many hosts your account will remain on a particular server until it fails in some way or another. That’s not the way we do things here at Dathorn. We like to be proactive with upgrading or replacing our servers to help avoid failures that happen more frequently as hardware ages. This also gives us a great opportunity to deploy new configurations, operating system versions, etc. so that we can continue adding value to our services.

Over the next several months we will be going through this process once again. All of our existing shared and reseller hosting servers will be upgraded by means of migrating to a new server. You will receive a ticket notification via our client portal once your particular server has been scheduled. Aside from announcing this I wanted to quickly highlight some of the more important changes that will take place as a part of these migrations.

First up are the hardware changes and the most important of these concerns the local storage. Over the past 13 years we’ve gone from SATA storage to SAS drives to our current hybrid SSD / SATA arrays. Now we’re very excited to be migrating to pure SSD storage. All servers will be utilizing new 12Gbps LSI RAID controllers with a minimum of six 1TB SSDs in RAID 10. The performance that we’ve been able to get from these new systems is simply amazing. While other providers may charge extra for (or not even offer) such high speed storage, all of our clients are being upgraded to it free of charge.

Continue reading

Adobe Flash & GHOST: Critical glibc Vulnerability

flash logo

Lately it seems there has been no shortage of critical vulnerabilities being discovered in commonly used software. In the past couple of weeks alone, Adobe has had to release patched versions of Flash to address a trio of publicized zero day vulnerabilities. While as a host that doesn’t really impact us directly, it should be a top priority for anyone browsing the web. The vast majority of end-user computer infections come from malicious content taking advantage of such vulnerabilities. These can often lead to your login information being compromised which certainly does become an issue for us. As always, please be sure you’re staying up-to-date with these Flash patches as well as those for your operating system, web browser, Java, etc. Continue reading

Drupal SQL Injection Vulnerability – CVE-2014-3704

drupal_logo-blue

On October 15th a very serious SQL injection vulnerability was discovered in Drupal that exists in all 7.x versions prior to 7.32. The severity of this vulnerability led to quick exploitation of it within approximately 7 hours of it having been publicized. Fortunately the provided patch to address this issue was quite simple and easy to apply. In fact, the patch only changed one line of code in the includes/database/database.inc file. Because of this we opted to go ahead and pro-actively apply the patch to all installations of Drupal 7.x on our servers. In less than an hour we had protected all of our clients’ Drupal installations from being exploited by this vulnerability. Beyond that it helped to protect our servers from attackers that were exploiting this vulnerability to run other malicious scripts. Affected clients should still upgrade their Drupal to the latest version as soon as possible.

Overall we were very pleased that this was so easily addressed on our end and we will certainly look into options like this going forward as new vulnerabilities in popular scripts are discovered. This incident shows how important it is for you to stay on top of script, plugin, and theme updates. Within a mere 7 hours of publicizing this vulnerability, it was being actively exploited. We highly recommend that you sign up for security related mailing lists for the scripts that you are using if they are available. This will give you the best chance at protecting yourself when (not if) a vulnerability like this comes to light.

Behind the Scenes: Shellshock & PHP 5.4

Here’s another quick update on what’s been going on here behind the scenes at Dathorn. As you may have heard, critical bugs were discovered in the popular Linux shell, bash. This event, dubbed “Shellshock”, started to publicly unfold about two weeks ago.

shellshock-bugThe details of these vulnerabilities can be a bit difficult to follow given the number of different patches that were posted. It even required a few quick, consecutive updates from some Linux distributions just to get it right. It seemed like each time a new patch was released someone else was able to poke holes in it, finding new methods to exploit and turning bash into a bit of swiss cheese. Continue reading

Behind the Scenes: Switches, VPN & RDP

As a continuation of our “Behind the Scenes” blog series I wanted to highlight some maintenance that we’ve completed over the past two weeks.

Switches

For starters, we’ve gone through all of our switches and have applied firmware updates as necessary. For the most part these were relatively minor but some did address potential denial of service vulnerabilities. Overall this helps us to avoid problems in the future and keeping up with new updates is generally a good idea anyways. These updates went off without a hitch thanks to our ability to first test them on equivalent spares that we keep on hand. Hardware failing in some form or another is simply a fact of life and we plan for this by building in redundancy and keeping spares on hand. We make sure to have full configuration backups for all of our switches should they need to be restored. On key switches the spares are even pre-loaded with the current configuration and powered on in-cabinet such that only swapping of network cabling would be necessary to recover from a failure. As great timing would have it, our data center also performed similar maintenance this morning on their networking equipment per our scheduled network maintenance announcement. This was likewise completed without issue. Continue reading