Benchmark Testing of Meltdown and Spectre Patches

Meltdown and Spectre are vulnerabilities in modern processors that can be exploited to steal data.  These flaws may result in data being stolen by attackers through Speculative Execution Side-Channel attacks. There are actually three separate vulnerabilities that comprise Meltdown and Spectre and two steps to fully remediate them.

Name

Description

CVE

Remediation

Meltdown

Rogue data cache load

CVE-2017-5754

Operating System patch

Spectre Variant 1

Bounds check bypass

CVE-2017-5753

Operating System patch

Spectre Variant 2

Branch target injection

CVE-2017-5715

Firmware update

 

It has been reported that remediating Meltdown and Spectre may decrease workstation performance by 5%-30%.  Also, due to the two-step remediation approach, it can be difficult to determine the vulnerability state of a given workstation.

We decided to evaluate these issues.  We executed a series of tests with two goals in mind: 1) demonstrate how a Meltdown and Spectre Validation tool can be used to identify vulnerability status and, 2) study the performance impact of the Meltdown and Spectre patches.

Test Approach:

We executed three test cycles:

·       Cycle 1: Tests of unpatched workstations (i.e. vulnerable to both Meltdown & Spectre)

·       Cycle 2: Tests of workstations with the Windows patch applied

·       Cycle 3: Tests of workstations with both Windows and BIOS patched

Each test cycle consisted of three tests:

·       Test 1: Execute Meltdown and Spectre vulnerability tool to determine vulnerability status

·       Test 2: Perform workstation boot benchmark to assess performance impact

·       Test 3: Perform 7zip benchmark to assess performance impact

Tests were executed on three workstations:

Model

OS

Processor

Memory

Original Firmware

Patched Firmware

HP ProBook 640 G1

Windows 10 - 1607

Intel R Core™ i5 4200M

12 GB

HP L77 Ver.01.39

HP L77 Ver.01.41

HP Elite 8460P

Windows 7 -7601

Intel R Core™ i5 2410

8 GB

HP 68SCF Ver. F.08

Unavailable

Surface 3

Windows 10 - 1709

Intel R Atom™ X7-Z8700

4 GB

AmericanMegaTrend Inc 1.51116.178

Unavailable

 

Test Tools:

The Meltdown and Spectre Validation tool checks your workstation’s Speculation Control Settings to determine if it is vulnerable to Speculative Execution Side-Channel attacks. It also gives suggested actions with regards on how to protect the workstation in the event that it is not protected.

Workstation boot benchmark tests were performed using the Microsoft Performance Toolkit, which is bundled with the Windows Assessment and Deployment Kit.  Each workstation was booted 10-20 times per test cycle and the 90th percentile results of the “BootDoneDuringExplorer” boot phase were reported.  BootDoneDuringExplorer represents the period from when the kernel is invoked until when the desktop is ready for user input.

7zip benchmark tests were conducted by measuring the time to compress a 1.59GB dataset using 7zip v16.04.  The dataset contains .doc, .jpeg, and .html files.  Compressions were repeated 20 times per machine per test cycle and the 90th percentile results were reported.

Test Results:

The Meltdown and Spectre Validation tool correctly identified vulnerability status and provided detailed recommendations on how to remediate the various vulnerabilities.

Boot benchmark tests resulted in boot times that were up to 23% slower after applying the OS patch.  7zip benchmark tests varied: in one of five tests, compressions were 13% faster after applying the OS patch, and in the other four tests compressions were unaffected or slightly slower. Given other industry studies on the performance impact of the Meltdown and Spectre patches, this result was somewhat unexpected.  We expected 7zip compression duration to be more affected than boot times.  We are planning additional tests to study this further but determining the reason for the slower boot times is beyond the scope of this blog.

The most significant finding was that firmware updates are not yet available for two (of three) of our test workstations.  After some research and a discussion with HP support, we learned that firmware updates may not be ready for quite some time or may never be provided for some older, out-of-warranty hardware models.  This means certain hardware will remain vulnerable to Spectre.

Test Result Summary

 

Comparisons of Benchmark Tests

  

Results from the Meltdown and Spectre Validation Tool

Conclusion:

We found that the Meltdown and Spectre Validation tool reliably determined vulnerability status of the test workstations.  We also observed some performance degradation on boot time after applying the Meltdown and Spectre operating system patch, though our full performance study was incomplete due to unavailability of firmware patches for all test workstations.  As firmware patches become more available, we will seek to complete the testing and will update the blog.

The main takeaway is that the approach for mitigating the Meltdown and Spectre vulnerabilities remains a very fluid situation.  It is important to monitor industry sources for latest mitigation recommendations and it is important to check early and often with your hardware vendor for their plans to provide a BIOS update to mitigate Spectre.

Additional Information:

Intel publishes their own set of benchmark test results as well as the tools they use to complete these tests. For more information please see the link below.

https://newsroom.intel.com/wp-content/uploads/sites/11/2018/01/Blog-Benchmark-Table.pdf

 Authors: Syed Rizvi, Consultant and Jessica Clark, Consultant II

Copyright © 1998 - 2019 Olenick. All Rights Reserved | Terms and Conditions
 
   

This site uses cookies to provide you with a more responsive and personalized service. By using this site you agree to our use of cookies.

Please read our cookie policy for more information on the cookies we use. Olenick’s privacy policy is available here.

More information Ok Decline