Every so often, we do a review and find a surprising nugget of information that we may have missed otherwise. Indeed, in our Intel Xeon Gold 6240R benchmarks and review piece we found a SKU that is a better value than when it was first announced due to a list price discount happening at some point. Let us look at that discount, and what that means to its competitive positioning.
Intel Xeon Gold 6240R Overview
The Xeon Gold 6240R most closely resembles the Intel Xeon Platinum 8260. What is interesting is that the Platinum 8260 we featured twice on STH. In our Intel Xeon Platinum 8260 Benchmarks and Review piece, we used a quad-socket node. We also used the Platinum 8260L in our Performance Check to validate that there was no difference between the standard Xeon Platinum SKU and the L model in terms of performance. As a fun aside, the Intel Xeon Gold 6240R is a 2x UPI link chip so it does not fit in the quad-socket scenario. It is also a standard memory part, so it does not fit in the high-memory scenario.
Key stats for the Intel Xeon Gold 6240R: 24 cores / 48 threads with a 2.4GHz base clock and 4.0GHz turbo boost. There is 35.75MB of onboard L3 cache. The CPU features a 165W TDP. These are $2200 list price parts. If you want to see more on the spec side, here is the ARK page for these CPUs.
Here is the lscpu output for the Intel Xeon Gold 6240R:
Somewhere between when Intel issued us launch pricing for the refresh, and this review going live, Intel seems to have decreased the list price. In our Big 2nd Gen Intel Xeon Scalable Refresh Brings Competition Anew piece we had the figure from Intel as $2445 for the Xeon Gold 6240R. We double-checked other coverage from that time period and that is what others recorded as well about eight months ago. Now, the list price is $2200.
As one can see on the right side of our comparison chart, taking a 24-core CPU and lowering the price by $245 means we effectively get a roughly $10/ core or a 10% discount in this case. We are going to talk more about market dynamics in the market analysis section of this review. The key takeaway is that this 10% discount makes the price per core more inline with AMD competition. We will also note the Xeon Platinum 8260 that this is effectively replacing in the mainstream dual-socket market is $4702. With the new pricing, we are getting a discount of around 53% off of what we had nine months prior to this review.
We mentioned the lack of the quad-socket capability for the Gold 6240R versus the Platinum 8260, but there is a dual-socket impact as well. The lack of the third UPI channel means that in dual-socket configurations the Xeon Gold 6240R has a 2x UPI link connection for socket-to-socket communication. A Xeon Platinum 8260 can have up to 3x UPI links. Enabling those three UPI links uses more power and increases trace count/ complexity in a motherboard. As a result, not all systems support this feature and that is why we noted it in our Supermicro BigTwin SYS-2029BZ-HNR Review.
Not all systems are designed for a 3x UPI link in a dual-socket configuration, especially since this adds costs. In some markets such as for hyper-converged servers as it adds ~50% more socket-to-socket capacity. As a 165W part, this is likely less of a concern than with the higher wattage 205W TDP parts.
In our benchmarks, we are going to investigate the performance impact. First, we are going to take a look at the test configurations then get into details.
Intel Xeon Gold 6240R Test Configuration
We are using a testbed that is designed for the higher-205W TDPs that some of the new refresh parts can hit, specifically the Supermicro SYS-2029UZ-TN20R25M or “2029UZ-TN20R25M” server. We published our Supermicro 2029UZ-TN20R25M Review recently if you want an in-depth look at the machine.
The Supermicro 2029UZ-TN20R25M is a 2U dual-socket server that is part of the company’s “Ultra” line meant to compete in the higher-end of the server market. We requested this server specifically because it has 20x NVMe SSD bays, it supports Intel Optane DCPMM, and it has built-in 25GbE. 25GbE is a major networking trend and we have already started doing overviews of 25GbE TOR switches such as the Ubiquiti UniFi USW-Leaf 48x 25GbE and 6x 100GbE switch overview and the Edgecore AS7712-32X Switch Overview. We have done adapter reviews such as the Supermicro AOC-S25G-i2S, Dell EMC 4GMN7 Broadcom 57404, and the Mellanox ConnectX-4 Lx. We also have more 100GbE switch reviews in the publishing queue so we wanted to start focusing on the new systems.
This Supermicro 2029UZ-TN20R25M platform is significant for another reason. It supports 205W TDP CPUs. That is a feature not every dual Xeon server has. For this reason, we wanted to use the Supermicro 2029UZ-TN20R25M which is a higher-end platform capable of handling this range of refresh CPUs. We do not need it for lower TDP SKUs like the Xeon Gold 6240R, but for higher-end SKUs we do. Here is the basic configuration:
System: Supermicro 2029UZ-TN20R25M
Memory: 12x 32GB DDR4-2933 DDR4 DRAM
OS SSD: 1x Intel DC S3710 400GB Boot
NVMe SSDs: 4x Intel DC P4510 2TB
Overall, this is a fairly simple configuration, but we are focused on CPU performance here. We are taking a further step and we tested these CPUs both in single and dual-socket configurations, so we will have some results of both views. The Supermicro Ultra server we are using we recommend using only in dual-socket configurations, however, we thought this would be a good way to spice the review up.
Next, let us get to performance before moving on to our market analysis section.
Intel Xeon Gold 6240R Performance
For this exercise, we are using our legacy Linux-Bench scripts which help us see cross-platform “least common denominator” results we have been using for years as well as several results from our updated Linux-Bench2 scripts. Starting with our 2nd Generation Intel Xeon Scalable refresh benchmarks, we are adding a number of our workload testing features to the mix as the next evolution of our platform.
At this point, our benchmarking sessions take days to run and we are generating well over a thousand data points. We are also running workloads for software companies that want to see how their software works on the latest hardware. As a result, this is a small sample of the data we are collecting and can share publicly. Our position is always that we are happy to provide some free data but we also have services to let companies run their own workloads in our lab, such as with our DemoEval service. What we do provide is an extremely controlled environment where we know every step is exactly the same and each run is done in a real-world data center, not a test bench.
We are going to show off a few results, and highlight a number of interesting data points in this article.
Python Linux 4.4.2 Kernel Compile Benchmark
This is one of the most requested benchmarks for STH over the past few years. The task was simple, we have a standard configuration file, the Linux 4.4.2 kernel from kernel.org, and make the standard auto-generated configuration utilizing every thread in the system. We are expressing results in terms of compiles per hour to make the results easier to read:
Setting the stage here, we are going to have a number of benchmark results with a core set and then varying a few interesting data points into the mix. This is the Intel Xeon Gold 6240R in the context of virtually the entire public AMD EPYC 7002 series dual-socket capable stack and is the first review we are slipping a few of the AMD EPYC 7H12 numbers into. Expect a review on that one soon. We also have a number of Intel Xeon offerings including adjacent SKUs and both refresh and pre-refresh parts.
c-ray 1.1 Performance
We have been using c-ray for our performance testing for years now. It is a ray tracing benchmark that is extremely popular to show differences in processors under multi-threaded workloads. We are going to use our 8K results which work well at this end of the performance spectrum.
This is a benchmark that we started to use several years ago. There are architectural reasons the AMD Zen and Zen 2 chips perform extremely well here. Instead of looking at AMD versus Intel, it is best to look at Intel v. Intel here. There are a number of tests where the Intel Xeon Gold 6240R is able to slightly edge-out the Intel Xeon Platinum 8260. We generally expect their performance to be close given specs.
7-zip Compression Performance
7-zip is a widely used compression/ decompression program that works cross-platform. We started using the program during our early days with Windows testing. It is now part of Linux-Bench.
Another storyline worth tracking here is the difference between the Gold 6240R and the Gold 6230R. We have not released our Gold 6230R review yet, however, it is a unique 26 core part. While the Gold 6230R has 26 cores, they are generally hitting lower clock speeds which means we often see the 24-core Gold 6240R perform better.
NAMD is a molecular modeling benchmark developed by the Theoretical and Computational Biophysics Group in the Beckman Institute for Advanced Science and Technology at the University of Illinois at Urbana-Champaign. More information on the benchmark can be found here. With GROMACS we have been working hard to support AVX-512 and AVX2 architectures. Here are the comparison results for the legacy data set:
Here we see the Xeon Gold 6240R slightly out-pace the AMD EPYC 7F52 frequency optimized part. The EPYC 7F52 is a 16-core part so this shows that 24 Intel Xeon cores even at a more mundane clock speed, can be competitive with 24 Intel Xeon cores. We can also see a substantial performance bump over the Intel Xeon Gold 6240. The model names may be similar, but the performance profiles and resources are very different.
OpenSSL is widely used to secure communications between servers. This is an important protocol in many server stacks. We first look at our sign tests:
Here are the verify results:
This is a workload that Intel may argue can be offloaded to QAT accelerators. Still, it shows something very interesting in that it is one of the few instances where we see the two extra cores of the Gold 6230R lead to more performance. In reality, our results were within a margin of error between the two but it is a good core v. clock speed trade-off example.
UnixBench Dhrystone 2 and Whetstone Benchmarks
Some of the longest-running tests at STH are the venerable UnixBench 5.1.3 Dhrystone 2 and Whetstone results. They are certainly aging, however, we constantly get requests for them, and many angry notes when we leave them out. UnixBench is widely used so we are including it in this data set. Here are the Dhrystone 2 results:
Here are the whetstone results:
In terms of price parity, the AMD EPYC 7452 is the closer SKU albeit at about 10% less per processor. The EPYC 7452 has 32 cores at lower clock speeds and lower TDP. On a performance per core basis, the Gold 6240R is better since the EPYC 7452 has around 33% more cores but only provides around 27% better performance. Of course, this is at a lower cost and even 27% better performance allows for very tangible consolidation benefits if one is not as concerned about per-core licensing.
Chess is an interesting use case since it has almost unlimited complexity. Over the years, we have received a number of requests to bring back chess benchmarking. We have been profiling systems and now use the results in our mainstream reviews:
For some additional context, the Intel Xeon Gold 6248R is a higher clocked 24-core part with more TDP headroom for only $700 more. If one is concerned about per-core performance, the Gold 6248R makes more sense. We are also going to mention here that AMD has 24-core parts as well. The AMD EPYC 7402 has 24-cores like the Gold 6240R but has a $1,783 list price or a 19% lower list price than the Xeon. At the original $2445 this was a 27% delta. It makes sense why this SKU saw a quick list price cut since the EPYC 7402 offering more performance, with the same core count, and what would have been at a 27% discount would have made the Xeon Gold 6240R less attractive. Even a 19% discount for less performance from the same core count seems a bit hard to justify, but it is better.
STH STFB KVM Virtualization Testing
One of the other workloads we wanted to share is from one of our DemoEval customers. We have permission to publish the results, but the application itself being tested is closed source. This is a KVM virtualization-based workload where our client is testing how many VMs it can have online at a given time while completing work under the target SLA. Each VM is a self-contained worker.
At this point, we generally know how this type of solution will perform. There is a nice uplift over the Xeon Gold 5218R here.
The last benchmark we wanted to look at is SPECrate2017_int_base performance. Specifically, we wanted to show the difference between what we get with Intel Xeon icc and AMD EPYC AOCC results. Server vendors get better results than we do, but this gives you an idea of where we are at in terms of what we have seen:
We received some comments noting we were a bit behind vendor results for some of our other tests, a fact that we noted in those reviews. Here we wanted to show both what we measured, and what server OEMs are measuring comparing the two chips. If the actual values are important for your RFP, then we suggest looking at the published results for the server(s) you are evaluating.
Here we see a fairly typical pattern. We pulled out some of our results and just left the AMD EPYC 7742 results here. While the EPYC 7H12 is faster, we wanted to just demonstrate the top end of a 225W TDP CPU which is what we are going to see as the top-end for many mainstream sockets.
Next, we are going to get into the “so what” and discuss market positioning for the processor before giving our final words.
Intel Xeon Gold 6240R Market Positioning
These chips are not released in a vacuum instead, they have competition on both the Intel and AMD sides. When you purchase a server and select a CPU, it is important to see the value of a platform versus its competitors. We are going to first discuss Intel v. AMD competition then look to Intel Xeon v. Intel Xeon.
Intel Xeon Gold 6240R v. AMD EPYC
We are going to exclude the Frequency Optimized EPYC and Intel Xeon parts here since the Xeon Gold 6240R is not designed for that market. Here we have the main segments of the AMD EPYC and 2nd Generation Intel Xeon Scalable processors on a $(USD) List per core basis:
With the price change from the $2445 to $2200, the price per core went from over $100 to just over $90. That is still a long way away from the two closest AMD EPYC competitors the EPYC 7452 and EPYC 7402. Here, the EPYC 7402 offers more performance with the same 24-core count. The AMD EPYC 7452 offers slightly lower pricing but more performance due to its 33% core count bump to 32 cores. Platform costs and deal-specific pricing may make the pricing deltas moot, but based on our testing these CPUs are more competitive in the $1500 range against the EPYC competition than even the $2200 range.
AMD offers a larger memory footprint with up to 4TB in 8-channel DDR4-3200 versus 1TB in 6-channel DDR4-2933. AMD has more PCIe I/O with 64, 96, or 128 PCIe Gen4 lanes per socket compared to Intel’s 48x PCIe Gen3 lanes. Intel has features such as AVX-512 and VNNI instructions (DL Boost) that AMD does not have but are new for 2nd Generation Intel Xeon Scalable. Perhaps the biggest feature Intel has is Optane DCPMM support. We should note here that DCPMM support is limited by memory capacity since the Gold 6240R is not an “L” SKU. Still, if one wants fast storage, DCPMM is a significant option that AMD does not have. If you are not using small DCPMM footprints, nor the new instructions, then AMD is extremely competitive.
To us, it would be very hard to recommend the Gold 6240R over the EPYC 7452 or EPYC 7402 given the list pricing. Intel has a broad portfolio, albeit smaller with SK hynix Acquiring the Intel NAND Storage Business. As a result, we are only looking at CPUs here not deal priced platforms so there is some margin to play with. Also, given the 2nd Gen Xeon platforms are an aging generation of technology still with only PCIe Gen3 support, an advantage is that components such as motherboards, risers, and cables for these Xeon systems are a bit less expensive. The cost of that, of course, is the PCIe Gen4 expansion the EPYCs have, but if cost is a major motivator, that is a benefit.
Intel Xeon Gold 6240R v. Intel Xeon
Intel made a strong competitive move in early 2020 with the Big 2nd Gen Intel Xeon Scalable Refresh altering the competitive landscape. Intel effectively dropped prices across most of its mainstream CPUs.
The challenge here is defining the Intel competition even with the Xeon Refresh. The Xeon Gold 6240R is only around $700 less at list pricing than the Gold 6248R. If software license costs are significant, then the Gold 6248R is a better buy since the list price delta and power delta will be offset by much higher software license costs.
If one truly does not need more performance and is instead focused on cost optimization, then the lower-clocked and lower-power Intel Xeon Gold 5220R starts to make more sense.
We do think that the Xeon Platinum 8260 effectively becomes a tough sell with the Gold 6240R. For high-end 3x UPI link systems, it will make more sense to upgrade to higher-end SKUs and push for per-deal pricing.
This is the challenge for the Intel Xeon Gold 6240R. It is a higher core count SKU with 24 cores at a time when Intel has a maximum of 28 cores per socket. The lower clock and power consumption make it not necessarily the best per-core performance, nor the lowest price per core. As a result, this is trying to hit a sweet spot in the Xeon range for those customers who do not want to take advantage of the lower pricing and better performance from AMD EPYC SKUs in this range.
With the recent, and stark, Intel DCG Enterprise segment miss where revenue was down 47%, and in contrast, AMD EPYC share is rising, perhaps the market has already figured this out. The Intel Xeon Gold 6240R is still a great chip. If it was released in 2017 or early 2019 (which one can argue it was with the Platinum 8260), then this would be a killer offering. Instead, as we have seen with the next-generation Intel Xeon Ice Lake CPUs getting delayed, these are SKUs thoroughly caught in the dynamics of the 2021 Intel Ice Pickle.
In Q4 2020 or Q1 2021 if you are a STH reader and the Xeon Gold 6240R is in your configuration, our best advice is to push for significant discounts. All of the systems vendors and both AMD and Intel know the competitive situation the Xeon Gold 6240R finds itself in. These are actually nice CPUs if you are in the market for this sweet spot, however, if you are reading STH you should now be aware of what to discuss in terms of alternative configurations to minimize your TCO with these systems.