I think you're looking for VisualEditor/Gadgets and related pages, especially VisualEditor/Gadgets/Creating a custom command and VisualEditor/Gadgets/Add a tool. Please feel free to improve the documentation as you go along.
If you are looking for simple examples, then m:User:Whatamidoing (WMF)/global.js has a couple of small things that might be interesting to you (e.g., a way to add nowiki formatting).