Category Archives: Behind the Scenes

Hardware Upgrades Complete!

Our goal is to provide the best performing, most reliable service that we can and a lot of this comes down to the hardware that we use. Newer hardware can easily provide performance benefits but proper deployment of it is crucial as far as reliability and security is concerned. We always try to avoid any service interruption whenever possible and building redundancies into our infrastructure has allowed us to do this more easily.

Over the past five months, we’ve been working diligently to overhaul our entire hosting infrastructure.  The individual server upgrades were a big part of this process and these were all completed by the end of July. Behind the scenes, though, our work has gone far beyond just the server upgrades. We’ve also upgraded every other piece of hardware that has a role in providing our services. All switches, power distribution, and even cabling has received upgrades in some form or another.

  • Switches – All of our public and private network switches have been replaced which provides greater performance and continued security updates. This was done without impacting any services and zero downtime on our public network. Our private network has received an even bigger overhaul with it now being entirely 10G, allowing for even better backup performance.
  • Power Distribution – All of our in-rack PDUs have been replaced with new zero U PDUs. This has saved at least 4U per cabinet and cleaned them up considerably. The new PDUs also have additional control and monitoring features. Because all of our gear has redundant power, we were able to swap these out without any power loss or downtime.
  • Cabling – As a result of the PDU changes, we wanted to re-cable everything to clean it up and allow for even better airflow. New slim Cat6A cables handle 1G connectivity and most 10G connectivity is over DAC.

Many other items were upgraded over these past few months as well. I just wanted to provide a brief look behind the scenes here that you wouldn’t otherwise know about. We’re very focused on providing the best service possible and our continual hardware upgrades play a large role in this.

Our New Client Portal!

Back in March, we refreshed our website and blog, making them responsive and far more mobile friendly. Since then, we’ve be revamping are administrative panel and now finally our client portal. It’s taken longer than we had hoped due to all of the server upgrades taking priority but we’re now very excited to unveil our freshly updated client portal! Below is just a quick comparison of the login pages on iOS.

All of the prior functionality within the portal is still present and organized much the same so it will be very familiar. We’re hoping to add a few more features in the coming months. If you have a moment, we’d love to get your feedback either here on our blog or in a ticket.

Behind the Scenes: Server Status, Monitoring & Website Refresh

Over the past few months we’ve been setting up and fine tuning a new in-house monitoring solution. This new system is now live and allows us to better monitor all aspects of our hosting services. As part of this, we’ve added a live status page to our website at status.dathorn.com. This page queries our monitoring system every minute to display the most accurate information and is a great first resource if you believe there are any issues. Rest assured that we will have already been alerted of any problems reported there.

Also, as you may have noticed by now, we just completed a minor refresh of our website. It now offers a greatly improved experience for mobile and high resolution devices. Over the next few months we plan to extend this to our portal as well so stay tuned!

Behind the Scenes: cPanel 70, cPanel 72 & Mobile Portal

It has been a little while since our last behind the scenes update so I wanted to take a quick moment to share a few things.

Last week, we completed deployment of cPanel 70 to all of our servers. While there weren’t a lot of visible changes in this version, you may have noticed the new “Email Accounts” interface in cPanel.

To start, you’ll see the interface is now broken down into four different tabs that allow you to add email accounts, view / modify them, manage the default account, and manage other configuration settings. This page will default to the list view of email accounts where you’ll now see the “Access Webmail” link more prominently. You can now also select the “Manage Suspension” option to suspend or hold outgoing email for a single email account. Continue reading

Behind the Scenes: Celebrating 15 Years & More

As we reach the end of 2017, I wanted to give a quick update on what’s been happening here behind the scenes over the past few months.

In our previous post, we covered our switch from SpamTitan to SpamExperts as a paid inbound email filtering option. Over the past couple of months we’ve now also migrated all outbound email filtering from SpamTitan to a completely custom setup utilizing SpamAssassin, ClamAV, and other software. This now completes our migration away from SpamTitan and has resulted in far better email filtering accuracy.

We’ve also been working to expand our offsite backup capacity and were finally able to complete this a couple of weeks ago. This was done in preparation for our not so secret plan to once again increase all of our hosting plan limits for everyone in the near future. We hope to roll these changes out during the first quarter of 2018.

The cPanel 66 and 68 updates have come and gone as well. There really wasn’t too much included that you’d have noticed from an end user standpoint. cPanel 70 will be available soon and will be much of the same as it mainly focuses on bug fixes.

Most importantly, though, is that December 12th officially marked our 15th year in business! There aren’t many hosting companies out there that can claim that, let alone without a change in ownership. We’ve been a resource you can rely on for 15 years and fully intend to remain as such for many more years to come. Thank you all very much for your continued partnerships and we wish you all a very Happy New Year!

Behind the Scenes: Security Patches & Private Network Updates

It has been a little while since we posted an update on what we’re working on behind the scenes here at Dathorn, so I wanted to take a quick moment to share. While it is easy to notice new features or see us working on your helpdesk tickets, much of our work goes completely unnoticed. That is, after all, our goal. We try to perform all updates, maintenance, etc. without any impact to your service.

While software patches and security updates are an ongoing battle, there have been a few noteworthy items as of late. You may have already heard about “Stack Clash”, a local privilege escalation vulnerability present in most Linux and BSD systems. Fortunately, we were able to quickly protect our servers against this without any service interruption, thanks to KernelCare. Traditionally, such kernel updates would require a reboot of each server but that has long been a thing of the past for us.

A few security issues have also been addressed in OpenVPN, including a remote code execution vulnerability. While serious, all of our instances were patched immediately and the severity of this particular issue for us was much lower since we only use OpenVPN internally for accessing certain private resources, such as IPMI on our servers. This had a much greater impact on VPN providers. Continue reading

Behind the Scenes: CentOS 5, LVE Stats 2, PHP 7.1 & EA 4

There’s been a lot happening here under the radar over the past month, so this gives us a great opportunity to post another of our “Behind the Scenes” updates. To start, we completed migration of all remaining CentOS 5 servers due to its end of life date, March 31st, quickly approaching. One of these servers was our primary DNS server (ns1) which was seamlessly migrated to a new CentOS 7 server without any service interruption.

CloudLinux’s LVE Stats 2 made its way out in a stable release, completely overhauling how system resource usage data is recorded on our servers. CPU and RAM usage is now recorded with much greater precision but perhaps most important is the new snapshot functionality. Now, when a CPU or RAM usage fault occurs, a snapshot of the account’s running processes is recorded. This allows you to go back and see what was running when a fault occurred, which is very helpful in identifying what caused it. Previously, no such information was available unless you actively witnessed the fault occurring.

PHP 7.1 was officially released and shortly thereafter available on all of our servers via CloudLinux’s PHP Selector. Our servers now offer PHP versions 5.4, 5.5, 5.6, 7.0 and 7.1. Version 5.6 is the default on new cPanel accounts and this can easily be changed via the “Select PHP Version” link in cPanel. Continue reading

Behind the Scenes: Server / PHP Upgrades & cPanel 58

It has been a little while since our last “Behind the Scenes” post so here’s another quick update of what’s been happening here at Dathorn. First, we are very happy to announce that we’ve completed the upgrade and migration of all of our hosting servers. Every single client is now on a server utilizing our latest hardware and software packages. This includes items like MariaDB, PHP Selector and, of course, pure SSD RAID 10 storage. The only mechanical drives that we’re using now are solely for backup storage and eventually these will be phased out as SSD capacity continues to grow.

Since all of our servers are now utilizing CloudLinux’s PHP Selector, we no longer have to perform scripted rebuilds of PHP and related dependencies as they are updated. As such, we will no longer be announcing these minor updates on our forums since they really don’t have any impact for our clients to be concerned about. You can always keep an eye on “Alt-PHP” updates on the CloudLinux blog if these items interest you.

Earlier this month we completed the deployment of cPanel version 58 across all of our servers. Most visible among the updates would be the changes made to Paper Lantern which you may have already noticed. There were, however, a number of system changes that aren’t so visible. One that you should be aware of is that the “Trash” folder on email accounts is now included in the email account’s disk usage. Believe it or not, this wasn’t the case before and it just didn’t make much sense. This often created confusion because email accounts would be reported as using far less disk space than they actually were. While this doesn’t change the cPanel account’s disk usage at all, it does allow email disk usage to be more accurately reported and managed, which is long overdue.

Going forward through October, we do still have some internal services that we need to migrate to new servers, such as our primary name server (ns1) and some backup servers. This is necessary as we approach the end of life for RHEL 5 / CentOS 5. We’ll be taking this opportunity to replace some older Adaptec RAID controllers with newer LSI ones as well. This maintenance won’t have any impact on clients but I did want to provide some insight on what we have planned here. That’s all for now, we hope you’re enjoying the cooler Fall weather as we are!

Behind the Scenes: Billing Migration & ImageTragick

We have successfully completed the migration of all clients from our old billing system (Ubersmith) to our new one that is integrated within our existing portal. For some of you, tomorrow will be the first time you’ll experience the new system when your service renews. We would love to hear any feedback you might have!

Being able to integrate our billing into our existing support mechanisms has been a tremendous help in simplifying processes both for us and our clients. We look forward to growing the available feature set going forward, including the ability for you to manage hosting plan changes without our intervention.

imagetragick

Earlier this month you may have heard of a new vulnerability in ImageMagick named “ImageTragick“. ImageMagick is a software suite used to create or edit many different types of images. One of the most common use cases involves their “convert” utility which is used to convert images from one type to another and resize them. cPanel, for instance, uses it as do a number of image gallery related scripts, shopping carts, etc. Because of this, the potential impact of ImageTragick was quite high.

Although it is fairly trivial to create a policy file or simply update ImageMagick to address the issues, care had to be taken to make sure all instances of ImageMagick were addressed. In many cases, our servers had two or three different versions of ImageMagick that needed to be taken care of. It’s not uncommon for it to be installed on the server as a general package in addition to the versions that both cPanel and CloudLinux provide. As always, you can rest assured that we’ve taken all possible steps to address these new attack vectors and will continue to monitor for further issues going forward.

Behind the Scenes: SRS, Billing Migration & Server Upgrades

The Sender Policy Framework (SPF) is a great tool to help validate email senders and detect email spoofing. We recommend that all domains have a proper SPF record configured for this reason. However, SPF has long caused problems with forwarders and this is now no longer an issue. As of cPanel 54, which was installed on all servers earlier this month, Sender Rewriting Scheme (SRS) is fully supported out of the box. Previously some forwarded emails would get rejected by the destination mail server due to the SPF check failing. SRS now automatically rewrites the envelope sender such that forwarded emails will still pass SPF checks. You don’t need to do anything to activate SRS, it was enabled on all servers on March 9th.

Last week another batch of clients was migrated to our new billing system that is built-in to our existing portal. The response has been overwhelmingly positive as it eliminates a lot of unnecessary confusion with having a separate billing system. We hope to have all of our clients migrated within the next month or two and appreciate any feedback you might have.

Lastly, we will be resuming our server upgrades in the upcoming month (April). These took a backseat while we finished up our billing system and continued evaluating the servers that had already been upgraded. Thus far we are very pleased with these upgrades and, like you, we can’t wait to have everything migrated to pure SSD environments. You’ll receive a notification when your server is scheduled to be upgraded if it hasn’t been already. We expect to have all remaining servers upgraded by the end of August.