This post is still in progress, read at your own risk. But feel free to comment
Trouble with benchmarking
I am thinking of writing a library to throw all kinds of rate limiting methods into it and even add distributed implementations into it.
It is on https://github.com/YesYouKenSpace/go-ratelimit/tree/main
I just encountered a problem I cannot really understand. I suspect my benchmarking results are not accurate. The first time i ran it, as expected, the
SyncMap + Load > LoadOrStore implementation outperforms the rest in every metric. Well as usual, i would run it once more just to be safe and guess what Map + Mutex outperforms it by almost 50%… Well maybe it was a fluke… But no! After a few more runs, it became more certain that the results always varies. They simply are not consistent, heck look at the below result, even in the same go test , the same implementation can have such different results‣
BenchmarkLiGr/name:SyncMap_+_Load_>_Store;number_of_keys:1028-10 3939787 385.9 ns/op 109 B/op 1 allocs/op BenchmarkLiGr/name:SyncMap_+_Load_>_Store;number_of_keys:1028-10 2101011 545.4 ns/op 194 B/op 1 allocs/op BenchmarkLiGr/name:SyncMap_+_Load_>_Store;number_of_keys:1028-10 2448114 527.0 ns/op 122 B/op 1 allocs/op BenchmarkLiGr/name:Map_+_Mutex;number_of_keys:1028-10 1407477 835.5 ns/op 64 B/op 1 allocs/op BenchmarkLiGr/name:Map_+_Mutex;number_of_keys:1028-10 1426149 948.7 ns/op 64 B/op 1 allocs/op BenchmarkLiGr/name:Map_+_Mutex;number_of_keys:1028-10 1398201 976.9 ns/op 64 B/op 1 allocs/op BenchmarkLiGr/name:Map_+_RWMutex;number_of_keys:1028-10 2802433 572.3 ns/op 139 B/op 1 allocs/op BenchmarkLiGr/name:Map_+_RWMutex;number_of_keys:1028-10 3247977 591.6 ns/op 94 B/op 1 allocs/op BenchmarkLiGr/name:Map_+_RWMutex;number_of_keys:1028-10 2652538 561.2 ns/op 64 B/op 1 allocs/op BenchmarkLiGr/name:SyncMap_+_LoadOrStore;number_of_keys:1028-10 2021199 550.4 ns/op 160 B/op 3 allocs/op BenchmarkLiGr/name:SyncMap_+_LoadOrStore;number_of_keys:1028-10 2616325 541.2 ns/op 160 B/op 3 allocs/op BenchmarkLiGr/name:SyncMap_+_LoadOrStore;number_of_keys:1028-10 2537360 528.2 ns/op 160 B/op 3 allocs/op BenchmarkLiGr/name:SyncMap_+_Load_>_LoadOrStore;number_of_keys:1028-10 2553331 579.3 ns/op 64 B/op 1 allocs/op BenchmarkLiGr/name:SyncMap_+_Load_>_LoadOrStore;number_of_keys:1028-10 2564904 665.7 ns/op 94 B/op 1 allocs/op BenchmarkLiGr/name:SyncMap_+_Load_>_LoadOrStore;number_of_keys:1028-10 2423139 585.9 ns/op 64 B/op 1 allocs/op BenchmarkLiGr/name:SyncMap_+_LoadOrStore;number_of_keys:2046-10 2627156 612.2 ns/op 160 B/op 3 allocs/op BenchmarkLiGr/name:SyncMap_+_LoadOrStore;number_of_keys:2046-10 2417246 517.7 ns/op 160 B/op 3 allocs/op BenchmarkLiGr/name:SyncMap_+_LoadOrStore;number_of_keys:2046-10 2506152 656.2 ns/op 160 B/op 3 allocs/op BenchmarkLiGr/name:SyncMap_+_Load_>_LoadOrStore;number_of_keys:2046-10 2483011 604.0 ns/op 64 B/op 1 allocs/op BenchmarkLiGr/name:SyncMap_+_Load_>_LoadOrStore;number_of_keys:2046-10 2466342 663.7 ns/op 64 B/op 1 allocs/op BenchmarkLiGr/name:SyncMap_+_Load_>_LoadOrStore;number_of_keys:2046-10 2640567 653.6 ns/op 80 B/op 1 allocs/op BenchmarkLiGr/name:SyncMap_+_Load_>_Store;number_of_keys:2046-10 2752107 631.5 ns/op 64 B/op 1 allocs/op BenchmarkLiGr/name:SyncMap_+_Load_>_Store;number_of_keys:2046-10 2464326 638.9 ns/op 64 B/op 1 allocs/op BenchmarkLiGr/name:SyncMap_+_Load_>_Store;number_of_keys:2046-10 2565309 619.5 ns/op 64 B/op 1 allocs/op BenchmarkLiGr/name:Map_+_Mutex;number_of_keys:2046-10 1452709 847.0 ns/op 64 B/op 1 allocs/op BenchmarkLiGr/name:Map_+_Mutex;number_of_keys:2046-10 1401331 984.5 ns/op 64 B/op 1 allocs/op BenchmarkLiGr/name:Map_+_Mutex;number_of_keys:2046-10 1464988 985.7 ns/op 64 B/op 1 allocs/op BenchmarkLiGr/name:Map_+_RWMutex;number_of_keys:2046-10 2534672 725.6 ns/op 64 B/op 1 allocs/op BenchmarkLiGr/name:Map_+_RWMutex;number_of_keys:2046-10 2552768 634.8 ns/op 64 B/op 1 allocs/op BenchmarkLiGr/name:Map_+_RWMutex;number_of_keys:2046-10 2600450 691.6 ns/op 64 B/op 1 allocs/op BenchmarkLiGr/name:Map_+_RWMutex;number_of_keys:8192-10 5175163 515.5 ns/op 64 B/op 1 allocs/op BenchmarkLiGr/name:Map_+_RWMutex;number_of_keys:8192-10 3650242 531.1 ns/op 64 B/op 1 allocs/op BenchmarkLiGr/name:Map_+_RWMutex;number_of_keys:8192-10 3847472 635.4 ns/op 141 B/op 1 allocs/op BenchmarkLiGr/name:SyncMap_+_LoadOrStore;number_of_keys:8192-10 5191035 1004 ns/op 268 B/op 3 allocs/op BenchmarkLiGr/name:SyncMap_+_LoadOrStore;number_of_keys:8192-10 4098939 578.9 ns/op 160 B/op 3 allocs/op BenchmarkLiGr/name:SyncMap_+_LoadOrStore;number_of_keys:8192-10 4057634 557.9 ns/op 160 B/op 3 allocs/op BenchmarkLiGr/name:SyncMap_+_Load_>_LoadOrStore;number_of_keys:8192-10 5381188 607.2 ns/op 64 B/op 1 allocs/op BenchmarkLiGr/name:SyncMap_+_Load_>_LoadOrStore;number_of_keys:8192-10 4172462 516.0 ns/op 64 B/op 1 allocs/op BenchmarkLiGr/name:SyncMap_+_Load_>_LoadOrStore;number_of_keys:8192-10 4170790 552.2 ns/op 64 B/op 1 allocs/op BenchmarkLiGr/name:SyncMap_+_Load_>_Store;number_of_keys:8192-10 6394174 754.4 ns/op 100 B/op 1 allocs/op BenchmarkLiGr/name:SyncMap_+_Load_>_Store;number_of_keys:8192-10 4384192 494.0 ns/op 64 B/op 1 allocs/op BenchmarkLiGr/name:SyncMap_+_Load_>_Store;number_of_keys:8192-10 3737553 583.2 ns/op 64 B/op 1 allocs/op BenchmarkLiGr/name:Map_+_Mutex;number_of_keys:8192-10 1854177 700.9 ns/op 65 B/op 1 allocs/op BenchmarkLiGr/name:Map_+_Mutex;number_of_keys:8192-10 1799601 809.3 ns/op 65 B/op 1 allocs/op BenchmarkLiGr/name:Map_+_Mutex;number_of_keys:8192-10 1638244 776.6 ns/op 65 B/op 1 allocs/op BenchmarkLiGr/name:SyncMap_+_LoadOrStore;number_of_keys:16384-10 2996959 480.4 ns/op 161 B/op 3 allocs/op BenchmarkLiGr/name:SyncMap_+_LoadOrStore;number_of_keys:16384-10 6250885 614.8 ns/op 160 B/op 3 allocs/op BenchmarkLiGr/name:SyncMap_+_LoadOrStore;number_of_keys:16384-10 6253971 555.8 ns/op 160 B/op 3 allocs/op BenchmarkLiGr/name:SyncMap_+_Load_>_LoadOrStore;number_of_keys:16384-10 3453082 376.7 ns/op 65 B/op 1 allocs/op BenchmarkLiGr/name:SyncMap_+_Load_>_LoadOrStore;number_of_keys:16384-10 6633942 571.8 ns/op 64 B/op 1 allocs/op BenchmarkLiGr/name:SyncMap_+_Load_>_LoadOrStore;number_of_keys:16384-10 5831298 530.7 ns/op 64 B/op 1 allocs/op BenchmarkLiGr/name:SyncMap_+_Load_>_Store;number_of_keys:16384-10 3222685 402.1 ns/op 65 B/op 1 allocs/op BenchmarkLiGr/name:SyncMap_+_Load_>_Store;number_of_keys:16384-10 6877593 776.6 ns/op 67 B/op 1 allocs/op BenchmarkLiGr/name:SyncMap_+_Load_>_Store;number_of_keys:16384-10 6414254 492.5 ns/op 64 B/op 1 allocs/op BenchmarkLiGr/name:Map_+_Mutex;number_of_keys:16384-10 1786554 656.9 ns/op 66 B/op 1 allocs/op BenchmarkLiGr/name:Map_+_Mutex;number_of_keys:16384-10 1340845 762.4 ns/op 70 B/op 1 allocs/op BenchmarkLiGr/name:Map_+_Mutex;number_of_keys:16384-10 1581866 645.9 ns/op 66 B/op 1 allocs/op BenchmarkLiGr/name:Map_+_RWMutex;number_of_keys:16384-10 3105381 491.4 ns/op 65 B/op 1 allocs/op BenchmarkLiGr/name:Map_+_RWMutex;number_of_keys:16384-10 5623940 622.8 ns/op 64 B/op 1 allocs/op BenchmarkLiGr/name:Map_+_RWMutex;number_of_keys:16384-10 5684859 585.9 ns/op 64 B/op 1 allocs/op
Current Hypotheses
- Busy machine - I am running on my laptop, which could be busy enough especially since running so many go routines (millions), the benchmarking could be sensitive to any other running processes’ contention
- Compiler optimisations - I thought i have eliminated them with
runtime.KeepAlivethough…
I have no idea now… I am just going to take a break and think about this later.
