We are making some progress on our PCI Compliance Project. Our last update focused on PCI scanning vendor selection. We now have our first pass results back.
We initially selected four vendors for our audits:
We did not get a reply from SecurityMetrics about a complimentary test audit. Not even an reply, saying no we don't provide test audits. I scored this as a large negative given that we do already have an account with them and have referred many clients to them in the past.
Starting the Scans
Starting the scans at the providers was easy. The ControlScan and ScanAlert (HackerSafe) control panels were easier to use. I had an issue with Comodo due to the use of many pop-ups, which I have set to block. Once I allowed pop-ups from Comodo, I could run the audits.
Initial Results
I've been seeing some issues with ScanAlert. I've seen different scan results over very short periods of time. The scanners were finding new vulnerabilities for very basic items, such as open ports. To be easy to manage, the results have to be consistent from scan to scan. This may have just been a temporary issue, but I did see it on 3 different servers over the past month.
Port Identification
A good look over the scans show pretty much the same results with respect to open ports. The scans did differ in their descriptions of the ports. For example, all scans found port 5353, which runs avahi-daemon, a mDNS service known as Bonjour or Redezvous. Only the Comodo scan identified this correctly. The other two found the port but did not identify it. This was also true with port 78 and 790 (RPC ports). None of the scans missed ports when compared to a manual Nmap scan.
Vulnerability Levels
The PCI DSS splits risks into different risk levels, ranging from 1-5 based on the security impact.
Level 5-Urgent-With this level of vulnerability, hackers can compromise the entire host. This vulnerability type allows hackers to have complete access to full file-system read and write capabilities, remote execution of commands as a root or administrator user, as well as the presence of backdoors and Trojans.
Level 4-Critical-Gives hackers partial access to file-systems and also provides them with remote user capabilities. These vulnerabilities expose highly sensitive information.
Level 3-High-Gives hackers access to information stored on the host, including security settings. It sets up misuse of the host by intruders. Examples include access to specific files, denial of service attacks, directory browsing, mail relaying.
Level 2-Medium-Gives hackers a chance to research attacks against the host, and access to some sensitive information from the host, such as exact versions of services.
Level 1--Low vulnerabilities expose information, such as open ports. Information can be obtained by hackers on configuration.
To further muddy things, the standard then defines any items Level 3 and higher as "High Risk". To pass a PCI compliance scan you cannot have any high risk items. You can have a large number of Level 1-2 items and pass but you cannot have any Level 3 items. In managing PCI compliance, I find I spend most of my time documenting false positives regarding High Risk vulnerabilities.
Report Formats
All providers provided PDF export functions. This is very handy to send reports to clients and go through a long list of exploits. The Comodo report was the most tedious as it lacked the polish and summary levels of the other companies.
Comodo's online view of the report was equally confusing. The layout made it difficult to
get an overview of the security risk.
ScanAlert and ControlScan both produced good online and interactive results. The ControlScan reporting method made the most sense if you have to send this data to someone else. The high level summaries and break downs of exploits make it possible to get a quick overview of the security threats.
I plan to go over the ScanAlert and ControlScan reports this week and compare the results. I will likely not bother with the Comodo scan as the layout is very poor. If you are interested you can find the reports here:
Control Scan Report
Comodo Report
Scan Alert Report
Comments (4)
Thanks for posting the test results.
Have you noticed any issue with the "Port Discovery Scan" service offered by ScanAlert ? About the scan not identifying uncommon ports like 4223.
It looks to me like it just scans using basic nmap or nmap -F which does not scan the entire 1-65535 range by default.
Thanks.
Posted by Chanchal | August 22, 2008 3:16 PM
Posted on August 22, 2008 15:16
I don't remember exactly but I think you can specify the port range. I know you can with ControlScan. Most of these scanners are based on Nessus, which may or may not use NMap.
The major issue we had with ScanAlert when we tested was differential scan alerts. Ports were found on a second scan that were not identified on the first scan. No changes were made on the server.
If you are ever in doubt about what ports are open on your system, install nmap and run it locally with the full port range. This way firewall and upstream blocks do not get in the way.
Posted by Jeff Huckaby | August 22, 2008 3:22 PM
Posted on August 22, 2008 15:22
We also have the same "differential scan alerts" problem with ScanAlert. It doesnt detect all ports every time it scans.
Thanks for sharing your experience, Jeff.
Posted by Chanchal | September 11, 2008 5:25 PM
Posted on September 11, 2008 17:25
As a manged service provider, getting different results between runs is a major headache. We had a client pay us to secure his server. We eliminated all of issues, closed out the case and sent it to invoicing. The next day, he wondered why we had closed the ticket because he was out of compliance. This went on for 5 days. We gain considerable efficiency by sitting down and handling the scan mediation in one sitting. If we have to go back and forth for 5 days, then the work takes a lot longer and becomes more costly for our client.
Posted by Jeff Huckaby | September 15, 2008 12:16 PM
Posted on September 15, 2008 12:16