StackOverflow2013

Note that there are some explanatory texts on larger screens.

plurals

PO
primarykey
Id
8391601
data
AcceptedAnswerId
0
AnswerCount
0
ClosedDate
CommentCount
30
CommunityOwnedDate
CreationDate
2011-12-05T20:43:55.563
FavoriteCount
0
LastActivityDate
2013-08-16T16:46:41.493
LastEditDate
2013-08-16T16:46:41.493
LastEditorUserId
922184
OwnerUserId
922184
ParentId
8389648
PostTypeId
2
Score
449
ViewCount
0
LastEditorDisplayName
text
Body
I've done this exact task before. But it was mainly to measure power consumption and CPU temperatures. The following code (which is fairly long) achieves close to optimal on my Core i7 2600K. The key thing to note here is the massive amount of manual loop-unrolling as well as interleaving of multiplies and adds... The full project can be found on my GitHub: <a href="https://github.com/Mysticial/Flops">https://github.com/Mysticial/Flops</a> <h1>Warning:</h1> If you decide to compile and run this, pay attention to your CPU temperatures!!! Make sure you don't overheat it. And make sure CPU-throttling doesn't affect your results! Furthermore, I take no responsibility for whatever damage that may result from running this code. Notes: <ul> <li>This code is optimized for x64. x86 doesn't have enough registers for this to compile well.</li> <li>This code has been tested to work well on Visual Studio 2010/2012 and GCC 4.6. ICC 11 (Intel Compiler 11) surprisingly has trouble compiling it well.</li> <li>These are for pre-FMA processors. In order to achieve peak FLOPS on Intel Haswell and AMD Bulldozer processors (and later), FMA (Fused Multiply Add) instructions will be needed. These are beyond the scope of this benchmark.</li> </ul> <pre><code>#include <emmintrin.h> #include <omp.h> #include <iostream> using namespace std; typedef unsigned long long uint64; double test_dp_mac_SSE(double x,double y,uint64 iterations){ register __m128d r0,r1,r2,r3,r4,r5,r6,r7,r8,r9,rA,rB,rC,rD,rE,rF; // Generate starting data. r0 = _mm_set1_pd(x); r1 = _mm_set1_pd(y); r8 = _mm_set1_pd(-0.0); r2 = _mm_xor_pd(r0,r8); r3 = _mm_or_pd(r0,r8); r4 = _mm_andnot_pd(r8,r0); r5 = _mm_mul_pd(r1,_mm_set1_pd(0.37796447300922722721)); r6 = _mm_mul_pd(r1,_mm_set1_pd(0.24253562503633297352)); r7 = _mm_mul_pd(r1,_mm_set1_pd(4.1231056256176605498)); r8 = _mm_add_pd(r0,_mm_set1_pd(0.37796447300922722721)); r9 = _mm_add_pd(r1,_mm_set1_pd(0.24253562503633297352)); rA = _mm_sub_pd(r0,_mm_set1_pd(4.1231056256176605498)); rB = _mm_sub_pd(r1,_mm_set1_pd(4.1231056256176605498)); rC = _mm_set1_pd(1.4142135623730950488); rD = _mm_set1_pd(1.7320508075688772935); rE = _mm_set1_pd(0.57735026918962576451); rF = _mm_set1_pd(0.70710678118654752440); uint64 iMASK = 0x800fffffffffffffull; __m128d MASK = _mm_set1_pd(*(double*)&iMASK); __m128d vONE = _mm_set1_pd(1.0); uint64 c = 0; while (c < iterations){ size_t i = 0; while (i < 1000){ // Here's the meat - the part that really matters. r0 = _mm_mul_pd(r0,rC); r1 = _mm_add_pd(r1,rD); r2 = _mm_mul_pd(r2,rE); r3 = _mm_sub_pd(r3,rF); r4 = _mm_mul_pd(r4,rC); r5 = _mm_add_pd(r5,rD); r6 = _mm_mul_pd(r6,rE); r7 = _mm_sub_pd(r7,rF); r8 = _mm_mul_pd(r8,rC); r9 = _mm_add_pd(r9,rD); rA = _mm_mul_pd(rA,rE); rB = _mm_sub_pd(rB,rF); r0 = _mm_add_pd(r0,rF); r1 = _mm_mul_pd(r1,rE); r2 = _mm_sub_pd(r2,rD); r3 = _mm_mul_pd(r3,rC); r4 = _mm_add_pd(r4,rF); r5 = _mm_mul_pd(r5,rE); r6 = _mm_sub_pd(r6,rD); r7 = _mm_mul_pd(r7,rC); r8 = _mm_add_pd(r8,rF); r9 = _mm_mul_pd(r9,rE); rA = _mm_sub_pd(rA,rD); rB = _mm_mul_pd(rB,rC); r0 = _mm_mul_pd(r0,rC); r1 = _mm_add_pd(r1,rD); r2 = _mm_mul_pd(r2,rE); r3 = _mm_sub_pd(r3,rF); r4 = _mm_mul_pd(r4,rC); r5 = _mm_add_pd(r5,rD); r6 = _mm_mul_pd(r6,rE); r7 = _mm_sub_pd(r7,rF); r8 = _mm_mul_pd(r8,rC); r9 = _mm_add_pd(r9,rD); rA = _mm_mul_pd(rA,rE); rB = _mm_sub_pd(rB,rF); r0 = _mm_add_pd(r0,rF); r1 = _mm_mul_pd(r1,rE); r2 = _mm_sub_pd(r2,rD); r3 = _mm_mul_pd(r3,rC); r4 = _mm_add_pd(r4,rF); r5 = _mm_mul_pd(r5,rE); r6 = _mm_sub_pd(r6,rD); r7 = _mm_mul_pd(r7,rC); r8 = _mm_add_pd(r8,rF); r9 = _mm_mul_pd(r9,rE); rA = _mm_sub_pd(rA,rD); rB = _mm_mul_pd(rB,rC); i++; } // Need to renormalize to prevent denormal/overflow. r0 = _mm_and_pd(r0,MASK); r1 = _mm_and_pd(r1,MASK); r2 = _mm_and_pd(r2,MASK); r3 = _mm_and_pd(r3,MASK); r4 = _mm_and_pd(r4,MASK); r5 = _mm_and_pd(r5,MASK); r6 = _mm_and_pd(r6,MASK); r7 = _mm_and_pd(r7,MASK); r8 = _mm_and_pd(r8,MASK); r9 = _mm_and_pd(r9,MASK); rA = _mm_and_pd(rA,MASK); rB = _mm_and_pd(rB,MASK); r0 = _mm_or_pd(r0,vONE); r1 = _mm_or_pd(r1,vONE); r2 = _mm_or_pd(r2,vONE); r3 = _mm_or_pd(r3,vONE); r4 = _mm_or_pd(r4,vONE); r5 = _mm_or_pd(r5,vONE); r6 = _mm_or_pd(r6,vONE); r7 = _mm_or_pd(r7,vONE); r8 = _mm_or_pd(r8,vONE); r9 = _mm_or_pd(r9,vONE); rA = _mm_or_pd(rA,vONE); rB = _mm_or_pd(rB,vONE); c++; } r0 = _mm_add_pd(r0,r1); r2 = _mm_add_pd(r2,r3); r4 = _mm_add_pd(r4,r5); r6 = _mm_add_pd(r6,r7); r8 = _mm_add_pd(r8,r9); rA = _mm_add_pd(rA,rB); r0 = _mm_add_pd(r0,r2); r4 = _mm_add_pd(r4,r6); r8 = _mm_add_pd(r8,rA); r0 = _mm_add_pd(r0,r4); r0 = _mm_add_pd(r0,r8); // Prevent Dead Code Elimination double out = 0; __m128d temp = r0; out += ((double*)&temp)[0]; out += ((double*)&temp)[1]; return out; } void test_dp_mac_SSE(int tds,uint64 iterations){ double *sum = (double*)malloc(tds * sizeof(double)); double start = omp_get_wtime(); #pragma omp parallel num_threads(tds) { double ret = test_dp_mac_SSE(1.1,2.1,iterations); sum[omp_get_thread_num()] = ret; } double secs = omp_get_wtime() - start; uint64 ops = 48 * 1000 * iterations * tds * 2; cout << "Seconds = " << secs << endl; cout << "FP Ops = " << ops << endl; cout << "FLOPs = " << ops / secs << endl; double out = 0; int c = 0; while (c < tds){ out += sum[c++]; } cout << "sum = " << out << endl; cout << endl; free(sum); } int main(){ // (threads, iterations) test_dp_mac_SSE(8,10000000); system("pause"); } </code></pre> Output (1 thread, 10000000 iterations) - Compiled with Visual Studio 2010 SP1 - x64 Release: <pre><code>Seconds = 55.5104 FP Ops = 960000000000 FLOPs = 1.7294e+010 sum = 2.22652 </code></pre> The machine is a Core i7 2600K @ 4.4 GHz. Theoretical SSE peak is 4 flops * 4.4 GHz = 17.6 GFlops. This code achieves 17.3 GFlops - not bad. Output (8 threads, 10000000 iterations) - Compiled with Visual Studio 2010 SP1 - x64 Release: <pre><code>Seconds = 117.202 FP Ops = 7680000000000 FLOPs = 6.55279e+010 sum = 17.8122 </code></pre> Theoretical SSE peak is 4 flops * 4 cores * 4.4 GHz = 70.4 GFlops. Actual is 65.5 GFlops. <hr> <h2>Let's take this one step further. AVX...</h2> <pre><code>#include <immintrin.h> #include <omp.h> #include <iostream> using namespace std; typedef unsigned long long uint64; double test_dp_mac_AVX(double x,double y,uint64 iterations){ register __m256d r0,r1,r2,r3,r4,r5,r6,r7,r8,r9,rA,rB,rC,rD,rE,rF; // Generate starting data. r0 = _mm256_set1_pd(x); r1 = _mm256_set1_pd(y); r8 = _mm256_set1_pd(-0.0); r2 = _mm256_xor_pd(r0,r8); r3 = _mm256_or_pd(r0,r8); r4 = _mm256_andnot_pd(r8,r0); r5 = _mm256_mul_pd(r1,_mm256_set1_pd(0.37796447300922722721)); r6 = _mm256_mul_pd(r1,_mm256_set1_pd(0.24253562503633297352)); r7 = _mm256_mul_pd(r1,_mm256_set1_pd(4.1231056256176605498)); r8 = _mm256_add_pd(r0,_mm256_set1_pd(0.37796447300922722721)); r9 = _mm256_add_pd(r1,_mm256_set1_pd(0.24253562503633297352)); rA = _mm256_sub_pd(r0,_mm256_set1_pd(4.1231056256176605498)); rB = _mm256_sub_pd(r1,_mm256_set1_pd(4.1231056256176605498)); rC = _mm256_set1_pd(1.4142135623730950488); rD = _mm256_set1_pd(1.7320508075688772935); rE = _mm256_set1_pd(0.57735026918962576451); rF = _mm256_set1_pd(0.70710678118654752440); uint64 iMASK = 0x800fffffffffffffull; __m256d MASK = _mm256_set1_pd(*(double*)&iMASK); __m256d vONE = _mm256_set1_pd(1.0); uint64 c = 0; while (c < iterations){ size_t i = 0; while (i < 1000){ // Here's the meat - the part that really matters. r0 = _mm256_mul_pd(r0,rC); r1 = _mm256_add_pd(r1,rD); r2 = _mm256_mul_pd(r2,rE); r3 = _mm256_sub_pd(r3,rF); r4 = _mm256_mul_pd(r4,rC); r5 = _mm256_add_pd(r5,rD); r6 = _mm256_mul_pd(r6,rE); r7 = _mm256_sub_pd(r7,rF); r8 = _mm256_mul_pd(r8,rC); r9 = _mm256_add_pd(r9,rD); rA = _mm256_mul_pd(rA,rE); rB = _mm256_sub_pd(rB,rF); r0 = _mm256_add_pd(r0,rF); r1 = _mm256_mul_pd(r1,rE); r2 = _mm256_sub_pd(r2,rD); r3 = _mm256_mul_pd(r3,rC); r4 = _mm256_add_pd(r4,rF); r5 = _mm256_mul_pd(r5,rE); r6 = _mm256_sub_pd(r6,rD); r7 = _mm256_mul_pd(r7,rC); r8 = _mm256_add_pd(r8,rF); r9 = _mm256_mul_pd(r9,rE); rA = _mm256_sub_pd(rA,rD); rB = _mm256_mul_pd(rB,rC); r0 = _mm256_mul_pd(r0,rC); r1 = _mm256_add_pd(r1,rD); r2 = _mm256_mul_pd(r2,rE); r3 = _mm256_sub_pd(r3,rF); r4 = _mm256_mul_pd(r4,rC); r5 = _mm256_add_pd(r5,rD); r6 = _mm256_mul_pd(r6,rE); r7 = _mm256_sub_pd(r7,rF); r8 = _mm256_mul_pd(r8,rC); r9 = _mm256_add_pd(r9,rD); rA = _mm256_mul_pd(rA,rE); rB = _mm256_sub_pd(rB,rF); r0 = _mm256_add_pd(r0,rF); r1 = _mm256_mul_pd(r1,rE); r2 = _mm256_sub_pd(r2,rD); r3 = _mm256_mul_pd(r3,rC); r4 = _mm256_add_pd(r4,rF); r5 = _mm256_mul_pd(r5,rE); r6 = _mm256_sub_pd(r6,rD); r7 = _mm256_mul_pd(r7,rC); r8 = _mm256_add_pd(r8,rF); r9 = _mm256_mul_pd(r9,rE); rA = _mm256_sub_pd(rA,rD); rB = _mm256_mul_pd(rB,rC); i++; } // Need to renormalize to prevent denormal/overflow. r0 = _mm256_and_pd(r0,MASK); r1 = _mm256_and_pd(r1,MASK); r2 = _mm256_and_pd(r2,MASK); r3 = _mm256_and_pd(r3,MASK); r4 = _mm256_and_pd(r4,MASK); r5 = _mm256_and_pd(r5,MASK); r6 = _mm256_and_pd(r6,MASK); r7 = _mm256_and_pd(r7,MASK); r8 = _mm256_and_pd(r8,MASK); r9 = _mm256_and_pd(r9,MASK); rA = _mm256_and_pd(rA,MASK); rB = _mm256_and_pd(rB,MASK); r0 = _mm256_or_pd(r0,vONE); r1 = _mm256_or_pd(r1,vONE); r2 = _mm256_or_pd(r2,vONE); r3 = _mm256_or_pd(r3,vONE); r4 = _mm256_or_pd(r4,vONE); r5 = _mm256_or_pd(r5,vONE); r6 = _mm256_or_pd(r6,vONE); r7 = _mm256_or_pd(r7,vONE); r8 = _mm256_or_pd(r8,vONE); r9 = _mm256_or_pd(r9,vONE); rA = _mm256_or_pd(rA,vONE); rB = _mm256_or_pd(rB,vONE); c++; } r0 = _mm256_add_pd(r0,r1); r2 = _mm256_add_pd(r2,r3); r4 = _mm256_add_pd(r4,r5); r6 = _mm256_add_pd(r6,r7); r8 = _mm256_add_pd(r8,r9); rA = _mm256_add_pd(rA,rB); r0 = _mm256_add_pd(r0,r2); r4 = _mm256_add_pd(r4,r6); r8 = _mm256_add_pd(r8,rA); r0 = _mm256_add_pd(r0,r4); r0 = _mm256_add_pd(r0,r8); // Prevent Dead Code Elimination double out = 0; __m256d temp = r0; out += ((double*)&temp)[0]; out += ((double*)&temp)[1]; out += ((double*)&temp)[2]; out += ((double*)&temp)[3]; return out; } void test_dp_mac_AVX(int tds,uint64 iterations){ double *sum = (double*)malloc(tds * sizeof(double)); double start = omp_get_wtime(); #pragma omp parallel num_threads(tds) { double ret = test_dp_mac_AVX(1.1,2.1,iterations); sum[omp_get_thread_num()] = ret; } double secs = omp_get_wtime() - start; uint64 ops = 48 * 1000 * iterations * tds * 4; cout << "Seconds = " << secs << endl; cout << "FP Ops = " << ops << endl; cout << "FLOPs = " << ops / secs << endl; double out = 0; int c = 0; while (c < tds){ out += sum[c++]; } cout << "sum = " << out << endl; cout << endl; free(sum); } int main(){ // (threads, iterations) test_dp_mac_AVX(8,10000000); system("pause"); } </code></pre> Output (1 thread, 10000000 iterations) - Compiled with Visual Studio 2010 SP1 - x64 Release: <pre><code>Seconds = 57.4679 FP Ops = 1920000000000 FLOPs = 3.34099e+010 sum = 4.45305 </code></pre> Theoretical AVX peak is 8 flops * 4.4 GHz = 35.2 GFlops. Actual is 33.4 GFlops. Output (8 threads, 10000000 iterations) - Compiled with Visual Studio 2010 SP1 - x64 Release: <pre><code>Seconds = 111.119 FP Ops = 15360000000000 FLOPs = 1.3823e+011 sum = 35.6244 </code></pre> Theoretical AVX peak is 8 flops * 4 cores * 4.4 GHz = 140.8 GFlops. Actual is 138.2 GFlops. <hr> Now for some explanations: The performance critical part is obviously the 48 instructions inside the inner loop. You'll notice that it's broken into 4 blocks of 12 instructions each. Each of these 12 instructions blocks are completely independent from each other - and take on average 6 cycles to execute. So there's 12 instructions and 6 cycles between issue-to-use. The latency of multiplication is 5 cycles, so it's just enough to avoid latency stalls. The normalization step is needed to keep the data from over/underflowing. This is needed since the do-nothing code will slowly increase/decrease the magnitude of the data. So it's actually possible to do better than this if you just use all zeros and get rid of the normalization step. However, since I wrote the benchmark to measure power consumption and temperature, I had to make sure the flops were on "real" data, rather than zeros - as the execution units may very well have special case-handling for zeros that use less power and produce less heat. <hr> <h2>More Results:</h2> <ul> <li>Intel Core i7 920 @ 3.5 GHz</li> <li>Windows 7 Ultimate x64</li> <li>Visual Studio 2010 SP1 - x64 Release</li> </ul> Threads: 1 <pre><code>Seconds = 72.1116 FP Ops = 960000000000 FLOPs = 1.33127e+010 sum = 2.22652 </code></pre> Theoretical SSE Peak: 4 flops * 3.5 GHz = 14.0 GFlops. Actual is 13.3 GFlops. Threads: 8 <pre><code>Seconds = 149.576 FP Ops = 7680000000000 FLOPs = 5.13452e+010 sum = 17.8122 </code></pre> Theoretical SSE Peak: 4 flops * 4 cores * 3.5 GHz = 56.0 GFlops. Actual is 51.3 GFlops. My processor temps hit 76C on the multi-threaded run! If you runs these, be sure the results aren't affected by CPU throttling. <hr> <ul> <li>2 x Intel Xeon X5482 Harpertown @ 3.2 GHz</li> <li>Ubuntu Linux 10 x64</li> <li>GCC 4.5.2 x64 - (-O2 -msse3 -fopenmp)</li> </ul> Threads: 1 <pre><code>Seconds = 78.3357 FP Ops = 960000000000 FLOPs = 1.22549e+10 sum = 2.22652 </code></pre> Theoretical SSE Peak: 4 flops * 3.2 GHz = 12.8 GFlops. Actual is 12.3 GFlops. Threads: 8 <pre><code>Seconds = 78.4733 FP Ops = 7680000000000 FLOPs = 9.78676e+10 sum = 17.8122 </code></pre> Theoretical SSE Peak: 4 flops * 8 cores * 3.2 GHz = 102.4 GFlops. Actual is 97.9 GFlops.
Tags
Title
singulars
PostAcceptedAnswerId
1. This table or related slice is empty.
PostParentId
1. POHow do I achieve the theoretical maximum of 4 FLOPs per cycle?
 singulars
 PostTypePostTypeId
 PTQuestion
PostTypePostTypeId
1. PTAnswer
UserLastEditorUserId
1. USMysticial
UserOwnerUserId
1. USMysticial
plurals
PostLinksPostIdRelatedPostId
1. This table or related slice is empty.
PostLinksRelatedPostIdPostId
1. This table or related slice is empty.
PostsAcceptedAnswerId
1. POHow do I achieve the theoretical maximum of 4 FLOPs per cycle?
 singulars
 PostTypePostTypeId
 PTQuestion
PostsParentIdCreationDate
1. This table or related slice is empty.
VotesPostIdCreationDate
1. VO
 singulars
 PostPostId
 PO
 UserUserId
 This table or related slice is empty.
 VoteTypeVoteTypeId
 VTUpMod
2. VO
 singulars
 PostPostId
 PO
 UserUserId
 This table or related slice is empty.
 VoteTypeVoteTypeId
 VTUpMod
3. VO
 singulars
 PostPostId
 PO
 UserUserId
 This table or related slice is empty.
 VoteTypeVoteTypeId
 VTUpMod
CommentsPostId

Querying!

Guidance

A row detail

Detail views are divided into sections. All the information in the data section comes from columns in the selected row. The other sections display data from other, related rows.

Related data can be related in a to-one or a to-many fashion. Captions of data related in a to-many fashion link to a list view showing a filtered view of the table.

Try moving around until you find a non-empty to-many entry and click on the label to get to one. You can move back to the root by clicking on the database name in the header.