project(go-ratelimit): genesis, perplexed by benchmarking #1
type
status
date
slug
summary
tags
category
icon
password
Tweet
Trouble with benchmarking
I am thinking of writing a library to throw all kinds of rate limiting methods into it and even add distributed implementations into it.
It is on https://github.com/YesYouKenSpace/go-ratelimit/tree/main
I just encountered a problem I cannot really understand. I suspect my benchmarking results are not accurate. The first time i ran it, as expected, the
SyncMap + Load > LoadOrStore
implementation outperforms the rest in every metric. Well as usual, i would run it once more just to be safe and guess what Map + Mutex
outperforms it by almost 50%… Well maybe it was a fluke… But no! After a few more runs, it became more certain that the results always varies. They simply are not consistent, heck look at the below result, even in the same go test
, the same implementation can have such different results (just look at SyncMap_+_LoadOrStore
).Current Hypotheses
- Busy machine - I am running on my laptop, which could be busy enough especially since running so many go routines (millions), the benchmarking could be sensitive to any other running processes’ contention
- Compiler optimisations - I thought i have eliminated them with
runtime.KeepAlive
though…
I have no idea now… I am just going to take a break and think about this later.