"A lot" is of somewhat the same nature the term "reasonable" is. All factors you mentioned matter. But, to put it short:
- 10 callbacks is ok. 100 may slow your site down a bit.
- Simple callbacks are ok. Be careful with heavy ones. For example, "pagelist" plugin works as a callback. There are several more plugins which query the database and do lots of stuff but are simply embedded in templates as callbacks. We call such plugins "widgets". Such widgets are "heavy", so it's reasonable to keep just a few of them.
May the Source be with you!