project(go-ratelimit): genesis, perplexed by benchmarking #1
project(go-ratelimit): genesis, perplexed by benchmarking #1
type
status
date
slug
summary
tags
category
icon
password
Tweet

Trouble with benchmarking

I am thinking of writing a library to throw all kinds of rate limiting methods into it and even add distributed implementations into it. It is on https://github.com/YesYouKenSpace/go-ratelimit/tree/main
 
I just encountered a problem I cannot really understand. I suspect my benchmarking results are not accurate. The first time i ran it, as expected, the SyncMap + Load > LoadOrStore implementation outperforms the rest in every metric. Well as usual, i would run it once more just to be safe and guess what Map + Mutex outperforms it by almost 50%… Well maybe it was a fluke… But no! After a few more runs, it became more certain that the results always varies. They simply are not consistent, heck look at the below result, even in the same go test , the same implementation can have such different results (just look at SyncMap_+_LoadOrStore ).
 

Current Hypotheses

  • Busy machine - I am running on my laptop, which could be busy enough especially since running so many go routines (millions), the benchmarking could be sensitive to any other running processes’ contention
  • Compiler optimisations - I thought i have eliminated them with runtime.KeepAlive though…
 
I have no idea now… I am just going to take a break and think about this later.

References

  1. https://eli.thegreenplace.net/2023/common-pitfalls-in-go-benchmarking/
  1. https://www.p99conf.io/2023/08/16/how-to-write-accurate-benchmarks-in-go/
  1. https://leveluppp.ghost.io/benchmarks-in-go-can-be-surprising/
 
Learning PinescriptTIFU: I used .npmignore
Loading...