The next generation firewall (NGFW) was invented by a gentleman named Jean-Luc Picard on September 28, 1987, but it would be several years before terrestrial bound enterprises (no pun intended) would adopt the technology. But before we dive into the topic of testing let’s take a look at Palo Alto Networks’ Theory of firewall evolution. After all, Palo Alto Networks is credited with creating the first generation of next generation firewalls.

Stateless Firewalls

The first-generation firewall lacked a sophisticated marketing team and therefore was simply called a firewall. Developed by Digital Equipment Corporation (DEC) in 1988, or AT&T in 1989, and commercialized by Checkpoint in the early 1990s depending on which source you choose. As Kelly Jackson Higgins explained in a 2008 Dark Reading article, multiple people claimed to be or were credited with being the father of the firewall, while others said “It’s not my kid.” The technology behind these early firewalls was packet filtering. Due to the narrow scope of technology, testing these firewalls was fairly easy. Are the packets filtered according to the rule set, and are ports and destinations correctly handled? Back then testers probably didn’t have the throughput to test the primitive throughput of these early devices. These firewalls were stateless, meaning that packet filtering was predominantly content agnostic; they inspected packet headers. By way of analogy, while attending a company party (a long time ago) an African American friend of a Caucasian (me) and I exchanged company ID badges before entering the venue. The person (firewall) checking badges looked at the badges and saw the big Microsoft logo. The source was the company and the destination was the venue, and we had the right badges (access control list) and were quickly waved on. That we were bipeds with credentials and addresses was enough. We easily got past the stateless “firewall.” A show dog walking on two feet was turned away because he didn’t have a Microsoft badge (access control). If you were to test a stateless firewall, using that analogy, the firewall worked as designed.

Stateful Firewalls

Next came the stateful firewall. The stateful firewall added the ability to inspect whole packets. Stateful firewalls added additional context awareness, robust logging, some degree of forgery prevention, and more. Arguably, stateful firewalls were among the earliest security technologies to use heuristics. These types of heuristic rules remain alive and well today in many products. It’s good stuff. Testing the stateful firewalls became significantly more complex. Of course, with deep packet analysis, context awareness, robust logging, and more, throughput took a hit. Imagine the badge checker in the analogy above looking at the badge, comparing the picture to the person, frisking people for contraband, etc. Throughput would take a massive hit. Adding more entrances with more badge checkers would be a way to mitigate the performance hit, but essentially it comes down to security needs. And so, testing became significantly more complicated. A variety of spoofing attacks, content delivered using multiple protocols, access control, logging, and more became necessary. Throughput is a very important metric. In a previous life while testing the stated maximum throughput of a network appliance, the appliance caught fire.  With throughput being a significant differentiator, testers must validate honest performance claims versus “Liar, liar, chips on fire” claims.

Stateless firewalls are still relevant today. For the most part, it comes down to the value of the resource being protected and throughput needs, with cost being a potential factor.

Advanced Enterprise Firewall

Sigh… Warp drives are about the only thing we don’t have to test. ADVANCED ENTERPRISE FIREWALLs still provide the functionality of stateless and stateful firewalls, but add additional capabilities designed to punish salaried testers with overtime. Operating systems and applications keep becoming increasingly complex, and attack surfaces broader and more porous. As such, security products, such as Advanced Enterprise Firewalls have had to add more robust and comprehensive defenses. In addition to an increased set of testing parameters, the need for quality analysis becomes magnitudes more essential. I’ll get back to quality analysis vs piss poor sub-par analysis later on.

Advanced Enterprise Firewalls embrace and extend the capabilities of stateful firewalls. Advanced enterprise Firewalls provide deeper packet analysis, application awareness and controls, intrusion prevention systems, greater network visibility, the ability to consume threat intelligence feeds, and more. Many, if not all Advanced Enterprise Firewalls have antimalware integrated into the appliance. Some of the capabilities SecureIQLab is testing for include

• Identify applications with full application context awareness
• Identify and block threats that try to use “known good” ports and protocol
• Identify and block threats that try to use evasive tactics such as non-standard ports or “port hopping”
• Identify and block threats that are encrypted with SSL
• Identify users, groups, and locations and apply policy regardless of IP address
• Identify and block outbound data leaks
• Identify and block outbound botnet command and control communications
• Provide global visibility and granular policy management

The need for Advanced Enterprise Firewall stemmed from the fact that attacks have increasingly become complex with incorporation of multiple attack steps. These steps include low and advanced level reconnaissance to data exfiltration. Traditional firewalls are inadequate to map these attack path comprehensively. Furthermore, use of historical data to predict future behavior has become quite common and threat data has been increasingly used to block attack. Use of Machine learning technique (ML) has been increasingly ingrained in Advanced Enterprise Firewall to develop either model that can be used to stop future attacks without the use of signatures. Some of these Firewalls are also capable are generating signatures dynamically and apply appropriate mitigation mechanism such as dropping connection. SecureIQLab intends to include various test cases in categories such as Exfiltration to assess device’s capability in correctly inspecting and enforcing policy to analyze traffic via protocols such as DNS, and ICMP.

No matter how accurate or extensive the testing, the output is data and not information. Information requires quality analysis. Before I continue, let me give you an example of piss poor sub-par analysis.

Many alleged experts have cited vulnerabilities in antivirus products as making you less secure. They expound upon how vulnerabilities that have been found (and patched) can be exploited with really bad outcomes. OK, but where’s the useful analysis?

Does anti-virus (antimalware) make you less secure than not having antivirus? If the answer given is yes, then quantify it. Put up or shut up. Are we talking home, enterprise, or both? Enterprises keep extensive logs. How many threats in a specified, and reasonable amount of time are logs showing that malware is being deflected? This is a particularly important metric as block rates will vary when measuring efficacy of known threats vs zero days. I could go on and on. This is a case of garbage in, garbage out. The garbage input is an unqualified opinion vs empirical data. The garbage out is stating unsubstantiated opinions as fact.

When testing security products, accurate and relevant empirical data must be available for analysis. This begins with a solid methodology. SecureIQLab is a member of the Antimalware Testing Standards Organization (AMTSO). SecureIQLab being a member of AMTSO, has been a pioneer in expanding the scope of AMTSO testing to include much larger range of security products. Cloud Web Application Firewall tests were a demonstration of that pioneering activity. SecureIQLab is expanding this list further by adding Advanced Enterprise Firewall, Secure Service Edge, Secure Access Service Edge, Container Security & API Security tests as demand for such technology grows due to the demand of remote-workforce and dev-ops activities.

Extensive experience is required for comprehensive testing. At SecureIQLab we have over 50 years combined experience as enterprise users, testers, and analysts. Like the motto of State Farm Insurance says, we’ve seen a thing or two.

There are many more data points that you will see included in our analysis of our testing of Advanced Enterprise Firewalls, web application firewalls (WAFs), API security, and more.

The “best” security product is worthless to an enterprise that can’t afford it. Our reports include a comprehensive return on security investment (ROSI) analysis based upon empirical data to help IT professionals determine how best to optimize their security value for the multiple security products required to minimize their attack surface.

Randy Abrams
Senior Security Analyst
SecureIQLab