Spaces:
Running
Tradeoff energy bost / performance
Thanks for this great tool that enables us to compute the energy cost of our LLM, very useful.
But the leaderboard shows ranks only according to the energy cost, while the most important is obviously the tradeoff between energy cost and performance. Isn't there a way to measure this tradeoff? Otherwise, we should all use distilgpt2 ! ;-)
Thank you,
Christophe
Hi Christophe! Of course the tradeoff between energy consumption and performance is important, but every user may assess it differently as it depends on their deployment environment and their goals. This leaderboard enables users to know more about the energy consumption of AI models, while the Open LLM Leaderboard and the LLM Perf Leaderboard enables them to get more information about their performance. The combination of these 3 leaderboards helps to make an informed decision taking into account accuracy, speed and energy consumption.
If you believe there is an objective way of measuring this tradeoff, do not hesitate to propose it here or to open a PR :)
I understand it is not the purpose of this leaderboard but I also agree with @cerisara : AI Energy score alone does not really make sense. It would be better to have a performance measure in the same table (accuracy, EER etc) and even better if we had an automatically generated graph with x-axis as performance/measure and y-axis as AI Energy Score.
But this page is already really interesting and helpful as it is. 🤗