RSMUS.comWhen (not if) your environment is breached and the forensics auditors are reviewing the environment, will they find you were doing everything reasonable? Or will they discover some very basic blunders that you assumed were taken care of but, in fact, were not – either by your internal teams or outsourced experts? This series will focus on critical technologies within your environment and the top 10 list of controls and practices that could cost you your job if missing.
We will begin this series with firewalls, since most organizations depend on this one device to separate their internal network from the world. It is the first device to see the attack coming and the last device to see your data leaving. It is one of the first technologies to be examined by forensics auditors to determine the scope of a breach and, perhaps, was the underlying reason for the breach. Today, most organizations know the importance of a firewall, and therefore outsource a good deal of firewall-related services to third-party providers. Whether in-house or outsourced, the same assumptions are being made that the firewall is properly maintained and configured, and yet the same old mistakes continue to live on. Here are the top 10 assumptions related to firewalls that you need to ensure are covered.
Assumption #1: Firewall logging is enabled and archived.
Criticality: Failing to archive firewall logs is the equivalent of financial institutions not recording their security camera feeds. If you experience a data breach, forensics auditors can use the logs to help determine the scope of the breach. Knowing if the breach impacted a handful of customers or having no idea at all and having to assume it impacted all customers can be the difference in keeping your job. Litigation involving employees’ inappropriate use of the network is yet another example where firewall logs can be critical. Know that your firewalls have traffic logging enabled and the log data is retrievable.
Ensure: On a bi-yearly basis, ask your IT staff or third party provider for the following:
1. How long are firewall logs maintained? If you haven’t had this conversation with your outsource provider or your internal IT staff, you are probably already in trouble. Ask the question and make sure everyone is on the same page. If no policy exists, one year is a good rule of thumb.
2. Please provide the logs in an exported format (csv, txt, etc.) for a 24 hour period beginning at 12:00AM on (pick the earliest date in accordance with your log retention policy). You will want to ensure a file is provided and not just a printed report. Forensics teams will want access to the raw file in order to run their own searches and toolsets against the data. This question ensures logs are truly exportable and that the retention policy was met.
3. Please provide the logs in an exported format for the last 24 hours. This output ensures that your logging is still working properly. More often than not, a log server has been retired or moved without updating the firewall with the new server details. The firewall is still configured for logging, but the logs haven’t been collected.
Assumption #2: Firewall logs contain all the data necessary if called upon.
Criticality: Not having the correct level of details within your firewall logs is almost as bad as not having the logs at all. Firewalls have varying degree of details that can be exported from the device – typically using syslog. For most firewalls, the informational level will export the desired traffic details such as source/destination IP addresses, ports, and traffic volumes. This will increase your storage requirements, but this data is critical to a forensics review.
Ensure: Once the firewall logs have been provided to you, use a simple firewall log analyzer to answer the question below to ensure the correct level of logging is enabled:
1. Does the report contain top source/destination IP addresses?
2. Does the report contain top protocol details such as HTTP, HTTPS, SMTP, etc.?
3. Does the report contain volume-related data such as the amount of traffic associated with IP addresses or protocols?
You should be able to answer “yes” to the above questions. If you are not able to answer these question from a report based on the exported log data, your firewall may not be exporting the desired level of detail.
Assumption #3: Firewalls are being patched.
Criticality: With most firewalls exposed directly to the Internet, they experience constant attempts to exploit vulnerabilities within the firewall code itself. Keeping firewalls patched is critical to reducing your risk of someone breaching your environment through a firewall related vulnerability. Firewall manufacturers are releasing code fixes and operating system enhancements on a frequent basis and often have patches available at least every quarter.
Ensure: On a bi-yearly basis, ask your IT staff or third party provider for the following:
1. How often are firewalls updated or reviewed for appropriate updates? Any hesitation you note to answer this question should be an immediate red flag that a process is not well defined. If an answer is provided such as “quarterly,” ask to see evidence of this process (trouble tickets, change logs, review logs, etc.).
2. Please provide a list of our firewalls and their software version level and associated release date. If you have multiple firewalls, look for variations in release levels as that could indicate that a process doesn’t exist to maintain updated code levels across all devices. Release dates that are older than a year is an immediate cause for concern that firewalls are not being updated at all.
Assumption #4: Public access to our systems is properly restricted.
Criticality: Systems exposed to inbound Internet access are also under constant streams of malicious traffic prodding for weaknesses. As services move to the cloud and systems and IP addresses are reallocated, updates to the firewall to remove access to retired systems and services is often overlooked. Assigning IP addresses to new systems for which old firewall rules exist could inadvertently open public access to private systems.
Ensure: On a bi-yearly basis or after major system upgrades or migrations, ask your IT staff or third party provider for the following:
1. What systems are accessible by the public? Make sure you know the public IP address, private IP address (if translated), the services or ports that are open, and who it is opened to.
2. Is access to these systems still required? Verify that the access permissions to these hosts from the public network is truly required. Don’t be surprised if you find access rules allowing public access to systems and services that have long been retired. If you don’t know what the access is for, work with your team and vendors to find out!
Assumption #5: High risk systems are deployed within a DMZ.
Criticality: It is amazing how many organizations have DMZs or other form of segmentation configured within their environment but fail to correctly deploy systems within it. A DMZ or demilitarized zone is a network where untrusted devices and systems reside to provide segmentation from trusted devices and systems. What are untrusted systems? We typically define that as any device or system that you don’t control, such as a vendor-provided router or server that they maintain or any system or device that the public can “touch.” Time and again, history has shown that if the public were allowed to access a system, even if only on a single port (www-80), they have been able to somehow take control of that system through some security flaw. In the case of vendor managed devices, you cannot rely on someone else to keep you safe – period. You have no idea who has access to their device, if they are patching it, or what configuration errors they may have made that could leave you vulnerable. By placing these devices into a DMZ and segmenting them from your network, the gateway device for that DMZ network (typically a firewall), restricts traffic flows from these devices in the event they were compromised. A firewall/IPS might also detect malicious traffic originating from these devices which could notify you of a compromised device before the damage can spread. Without a DMZ, these un-trusted devices would have direct network access to all of your internal systems and potential malicious traffic flows from a compromised device could be extremely difficult to detect.
Ensure: On a bi-yearly basis or whenever an untrusted system is deployed within your environment, ask your IT staff or third party provider for the following:
1. What does each DMZ host have access to? Create a chart that details out each un-trusted host within the DMZ and list the internal and external systems and ports that the DMZ host is allowed to communicate with.
2. Is access to these systems required? Verify that the access permissions identified above are truly required. Limit this traffic flow to the minimal level of access possible. We would suggest that these hosts are not permitted to the Internet (any public IP). You can grant access to individual public IPs for updates or vendor sites, but in most cases these systems do not require access to every public IP on the planet.
Assumption #6: Our firewall restricts outbound access.
Criticality: Nothing can be more embarrassing than finding out a data leak occurred through a port on your firewall that was never needed for business reasons. Organizations have thought for years that it was the public interface of your firewall that required you to lock ports down while the internal network is free to go wherever they wanted since they are “trusted.” This assumption has been responsible for many data leaks and malware outbreaks that could have been prevented.
Ensure: On a bi-yearly basis, ask your IT staff or third party provider for the following:
1. What ports and protocols are allowed outbound by the firewall to any destination? The first step is actually knowing what you are allowing to leave your environment.
2. Are all of these services required? Once you have an inventory, begin to understand where you might be able to lock things down further. Most organizations really only require a few ports for web browsing such as ports 80 and 443. If you find out that you are allowing anything out on any port, you have a lot of work to do. In those cases, it is often best to begin blocking high risk ports further up in your access lists. Next, begin to put in rules ahead of the “permit any” statement for traffic that is required. Eventually, once you believe you have your inventory of ports permitted, remove the “permit any” statement and see who screams. Make exceptions as they are identified and verified as business relevant.
Assumption #7: We don’t require a high availability firewall solution.
Criticality: We cannot count the number of organizations over the years that have found out the hard way how critical their firewall is to their business. The reliance on the Internet to provide services such as end-of-day reporting, payroll uploads, and order processing are only a few examples that many organizations can only go a few minutes without. We also can’t forget access to vendor services that should be behind your firewall if a vendor provides a managed device to access their systems.
Ensure: On a bi-yearly basis and as new services are deployed, ask your various departments and managers:
1. What critical services do you rely on the Internet for on a recurring basis? Identifying this information may not be as easy as you think. Your IT staff will most likely be unable to answer all of these questions other than what previous experience has taught them when an outage occurred. Asking your different departments will be the best course of action. Don’t expect them to know if the application uses the firewall or not. Just ask them what daily processes are critical to the business and then sit down with them and determine the application’s dependencies.
2. How quickly could we restore services in the event of a firewall failure? Some organizations might have a backup site that has another firewall ready to take over. Other organizations rely on support contracts to get a new firewall in place in the event of a failure. If a support contract doesn’t meet the recovery requirements of the business, you might simply buy another firewall as a cold spare or configure it in a high availability set if supported. One more quick assumption we’ll throw in here; an expensive same-day on-site replacement contract doesn’t always mean same-day hours! Companies large and small in rural areas and urban areas have found out the hard way that a replacement part wasn’t available in the timeframe they had been promised in their contract. Sure, they may have gotten their money back on the contract but that was nothing to the loss and embarrassment experienced from the outage. It is also amazing how many organizations have kept these contracts in place for so long that they could have purchased a spare device over the course of the contract.
Assumption #8: Our firewalls aren’t that old.
Criticality: Many organizations rely on their manufacturer’s product lifecycle to determine when to upgrade their security appliance. End-of-sale or end-of-life announcements typically provide a three to five year window during which the appliance will still have access to software updates or hardware replacement in the event of a failure. For some manufacturers, software updates will cease before the hardware replacement warranty period. In other words, for a window of a couple of years in some cases, the hardware can still be replaced but patches or software updates are no longer available. Another important concept reminds us that the threats we are trying to mitigate with these systems ultimately determines whether or not our device is still relevant. If your device is too old to update to the latest code chain or it simply doesn’t have the technology or features necessary to combat the threats you are concerned about, you may need to upgrade sooner than your depreciation schedule.
Ensure: On a yearly basis, ask your IT staff or third party provider for the following:
1. Are there any end-of-life (EoL) or end-of-sale (EoS) announcements for our appliance? Never assume that just because you recently purchased an appliance there won’t be any of these notices. Your technology partner may not be keeping track of these product lifecycles and in fact sold you a system with an active notice. Always keep track of your own inventories and awareness of their lifecycle announcements.
2. Is the manufacturer keeping pace with security vulnerabilities? A sure sign your firewall should be replaced is when critical security vulnerabilities are announced but the manufacturer has delayed code updates for your particular model but more recent models from the same manufacturer are being patched.
Assumption #9: We are safer using firewalls from multiple vendors.
Criticality: The logic behind this assumption seems sound in that a single security flaw that targets a single vendor could bypass all of your defenses. Therefore, deploying either multiple firewalls from different vendors or firewalls from a separate vendor as your traditional route/switch environment is safer. The problem is virtually no breaches have been associated to a single vendor security bug when compared to configuration errors or missing patches within these vendor products. When we run into these multi-vendor networks, for companies large and small, we most often find missing updates, poorly followed best practices, and rulesets that look like Swiss cheese. The reason is simple, when forced to split attention across diverse products, organizations never really master any of them. When you don’t understand something, you tend to leave it alone once it is working.
Ensure: If your environment consists of multi-vendor firewall products, ask the following on a yearly basis to either your internal IT staff or your vendor:
1. What are the written upgrade procedures for each of our firewall platforms? At the very least, confirm that actual upgrade procedures exist for each of the vendor platforms you have in use. If procedures don’t exist, this could be a sign that upgrades are not being performed as routinely as you might think. Also, compare code inventories using the steps found in Assumption #3. If you see that one vendor platform is being updated more often than another, this is yet another sign of potential weaknesses in resources and skillsets necessary to maintain multi-vendor platforms.
2. Who is certified on each of our firewall platforms? A certification by itself doesn’t mean you have an expert on that product. It does however provide an indication of some level of dedication, training allocation, and experience that your staff or vendor exhibits for each platform. If no certifications exist for one product or the other, dig deeper. Ask what routine maintenance is being performed on each platform, what hardening procedures are in place, what tickets have been opened with each manufacturer over the last 12 months. If you get little response, it might be time to hire a third party to perform a detailed review of each platform’s configuration.
Assumption #10: We have a firewall so we are safe.
Criticality: If your security begins and ends with your firewall, get your resume ready. If the breach is big enough to make headlines (local or national), you probably won’t want to list your current employer as previous experience. Your adversaries are best at breaking down these single lines of defense. No doubt a firewall is required, but you need to ensure your defenses go beyond a single appliance at your border.
Ensure: On a yearly basis, ask your IT staff or third party provider for the following:
1. Besides our firewall, what else do we have deployed to defend against malicious activity? Think about things such as anti-virus, e-mail filtering, network segmentation, IPS/IDS, etc. If all of these things are integrated into a single firewall appliance, you are probably exposed.
2. Do we have good detective and corrective controls in place in addition to our preventative controls? Firewalls for the most part fall into a preventative control category. In other words, most people associate them with preventing malicious activity in the first place. Detective controls assume your preventative measures have failed and you are now trying to detect malicious activity in your network. Corrective controls are measures to correct the offense once detected within the network. For many organizations, most of their controls fall into the preventative category, including their firewall. If your list of controls leaves the detective or corrective categories empty, the odds are against you for quickly finding and correcting malicious activity when it happens. That doesn’t mean going out and buying more controls to populate those categories. Rather, a better approach is to work with a network/security architect to understand what you do have and how best to align the architecture with your security strategies and risk appetite.
Conclusions
Ensuring these controls are in place and not just an assumption can make a big difference in how your peers view your handling and preparedness when a breach occurs. Proving how small a breach was can save your company big time in terms of costs and brand damage compared to not having any evidence and having to make big assumptions as to the potential impact of the breach.
Here is a checklist to review with your team and/or outsourced providers:
1. Firewall logging is enabled and archived
a. How long are firewall logs maintained?
b. Please provide the logs in an exported format (csv, txt, etc.) for a 24 hour period beginning at 12:00AM on (pick the earliest date in accordance with your log retention policy).
2. Firewall logs contain the desired details.
a. Does the report contain top source/destination IP addresses?
b. Does the report contain top protocol details such as HTTP, HTTPS, SMTP, etc.?
c. Does the report contain volume related data such as the amount of traffic associated with IP addresses or protocols?
3. Firewalls are being patched.
a. How often are firewalls updated?
b. Please provide a list of our firewalls and their software version level and associated release date.
4. Public access to our systems is restricted.
a. What systems are accessible by the public (any IP address)?
b. Is access to these systems still required?
5. High-risk systems are deployed within a DMZ.
a. What does each DMZ host have access to?
b. Is access to these systems required?
6. Our firewall restricts outbound access.
a. What ports and protocols (services) are allowed outbound by the firewall to any destination?
b. Are all of these services required?
7. We don’t require a high availability firewall solution.
a. What critical services do you rely on the Internet for on a recurring basis?
b. How quickly could we restore services in the event of a firewall failure?
8. Our firewalls aren’t that old.
a. Are there any end-of-life (EoL) or end-of-sale (EoS) announcements for our appliance?
b. Is the manufacturer keeping pace with security vulnerabilities as they are announced?
9. We are safer using firewalls from multiple vendors.
a. What are the written upgrade procedures for each of our firewall platforms?
b. Who is certified on each of our firewall platforms?
10. We have a firewall so we are safe.
a. Besides our firewall, what else do we have deployed to defend against malicious activity?
b. Do we have good detective and corrective controls in place in addition to our preventative controls?
To speak with an RSM security and firewall consultant email us or call 800.274.3978. Visit our website to learn more about RSM’s security and compliance services.
Scott Hermanson
Scott leads the national network and unified communication solutions team, which encompasses network cyber-defense technologies, transport systems and unified communication platforms. Prior to joining RSM in 2003, Scott worked for a software company as a senior network engineer where he was responsible for the design and implementation of data and voice networks to support financial transactions in excess of over $1 million every minute and up to 800,000 online traders. Scott also has an extensive background in network design and architecture. He has designed infrastructures to support both front and back-office financial transactions with a variety of firms. Scott has great discipline in the field of network documentation and operational procedures. He has created web-based systems to capture network-based move, add and change requests, and a live documentation management system. He also has detailed experience for the implementation of network monitoring and management tools from a variety of vendors. In order to accommodate government regulation of financial-based networks, Scott has designed networks for five nines of availability. During his employment with a software company, the core network designed by Scott was able to switch all 800,000 users and over a dozen back-end connections to a remote recovery facility in less than three minutes. Switching services to the remote facility was performed once per month to ensure clients of the business continuance plan.