Tackling a Tough IT Enterprise Environment
inSync Computer Solutions was initially contracted to provide programming services. This client had 500 employees, many were graphic designers, and had a large online presence to major department stores. inSync helped with some custom interfaces & Shopify. We were contacted for network support because of a Ransomware incident.
Our client was attacked by a highly sophisticated foreign entity on a weekend morning. The systems were in the process of being encrypted but an employee noticed the unusual activity and literally unplugged their systems, stopping the encryption. inSync was asked to come in to evaluate their systems. We audited their IT infrastructure at five different facilities and presented our findings to their C-level staff. Our findings included:
No backups, local or remote;
Failing infrastructure including servers, switches...;
No antivirus protection on 225 machines including desktops, laptops, servers…resulting in another attack;
Poor on & offboarding procedures of personnel resulting in significant O365 overbillings;
Inadequate digital file storage of custom artwork;
A list of priorities
1st: Purchased & Installed Palo Alto Firewalls
Our first priority was to purchase, install, activate all licenses & subscription, & configure the Palo Alto firewalls, an enterprise cybersecurity platform which provides network security, cloud security, endpoint protection, and cloud-delivered security services. The firewalls also allowed site to site connection & VPN for secure remote connection.
2nd: Backed Up Servers
There were no server backups of data/applications, or of their media. Backups are crucial protecting data in case of a system crash, data corruption, hard drive failure, ransomware attack...We recommended Veeam Software, an industry leading backup, recovery & replication software. And the client purchased new NAS drives to backup data locally & at one of their remote locations. Quest, a cloud backup, provided a daily offsite backup. Altogether there are 3 redundant daily backups complete - locally, at one of their remote locations, and in the Cloud.
3rd: Managing Office 365 - Email
First & foremost, inSync would never recommend Microsoft 365 for a company this large. While engineers love this solution, the CFO's do not. It is very expensive &, from our viewpoint, it doesn't eliminate problems it just introduces new ones. So managing your monthly Microsoft 365 invoice should be a priority.
Unfortunately, their email server was also mismanaged resulting in a catastrophic failure and the server was not backed up. Their email was down for a number of days and was eventually migrated to Microsoft 365 resulting in a fairly significant impact on repetitive overhead. $8,000/month.
Their Microsoft 365 account was also mismanaged resulting in the payment of thousands of dollars per month in unnecessary charges. Our client has since deleted a number of email accounts and has also received a discounted rate from our vendor resulting in some savings.
The on & off boarding of personnel, by your help desk personnel, should include Microsoft 365 as part of this process.
4th: Barracuda - Additional Protection for O365 Clients
Prior to our being contracted, two of our clients in Microsoft 365, were hit with very sophisticated phishing scams which resulted in $800,000 of losses. This particular client had a vendor that did a significant amount of business with them. The vendor's email had been hijacked. They sent an email to the Accounts Payable clerk and told her there was a change in banks. The clerk said she would need this on company letterhead. The hacker provided the information on letterhead via email. The vendor's invoices was paid to this hacker's bank account for a loss of $500,000.
If our client is in Microsoft 365, we always recommend Barracuda Essentials Complete which provides advanced email security, backup archiving & compliance - additional protection for Microsoft 365. Because it archives all email, Sharepoint & One Drive, it allows a step down in licensing as well. Barracuda Essentials Complete includes:
Virus scanning and filtering
Email-borne malware protection
Archiving for compliance
Automatic encrypted emails
Email, SharePoint and OneDrive backup
Advanced threat protection
4th: Inconsistent Antivirus Installed on Machines
Symantec Antivirus was missing on 225 desktops so we purchased the additional licenses to cover the remaining unprotected desktops, laptops & servers. When replacing desktop & laptops, we made sure the help desk personnel had processes in place so antivirus would always be included when installing new machines.
5th: Purchased the StarWind Cluster Solution to Replace Aging Servers
Our client's corporate IT infrastructure was, for the most part, aged out. We did not consider their IT environment to be stable. 20 of the 22 servers were not under warranty. 18 of the 22 physical servers were 7 years or older. And these servers were all a single point of failure. Microsoft has monthly security updates for their operating systems which are not being done because of the fragility of their equipment. And some of their servers had operating systems that were end-of life & required upgrading.
The cluster solution purchased included 3 servers for redundancy, and all of the Microsoft licensing necessary.
Starwinds 3-Server Cluster Solution $117,142
6th: SQL Server Replication - In Process
Our client's SQL Server Went Down Recently. It is mission critical & the SQL Server went down for at least a half hour. It has 40 distinct databases and requires 24/7 uptime. It’s was not the first time it has gone down. Our engineer believes it may be faulty memory but, in order to properly diagnose, Dell needs to run tests with the server offline. Since it is crucial to operations, Dell is unable to perform the tests.
inSync recommended a 2nd server be used as a replication target.
New SQL Server Cost Including Hardware & Software $104,749
7th: Qumulo Solution for Media & Artwork - In Process
Our client owned 250 terrabytes of custom artwork that were currently hosted on 2 NAS drives whose warranties expire in 2021. 300 graphic designers were competing for network and storage resources to work on large graphics files. It was extremely slow & time was wasted waiting for directories to list their contents.
We presented an enterprise storage system proposal to solve these challenges. Eliminating management complexity, providing data integrity and recoverability. An enterprise-class system with optimal performance and reliability.
Qumulo Hardware & Software Solution $184,401
Conclusion: As a result of the numerous failures & inadequate security, we replaced their senior engineering staff of 3 with one of our senior engineers who works only 24 hours per week. It's important to note that our goal is to reduce labor costs but it can only be effectively accomplished if companies invest in their IT infrastructure. With proper investment in servers, firewalls, switches, backup hardware & software...& 24/7 monitoring, labor can be reduced significantly. Our client is saving a lot of money and they are in much better shape than in previous years with their in-house personnel.