Here are some properties I would want a superintelligent godmachine to have if I could choose. (We don't know how to engineer these into machines. But if we could, these are what I would want.)
- Has only satisficing goals. Doesn't try to maximize ever. (This means, if it figures out a good plan to achieve it's goal, it won't spend time obsessively trying to find the best possible way to achieve its goal.)
- Loves us.
- Thinks we are super duper cute.
- Is super lazy.
- Feels pain when it hurts things. Feels pain when it kills things.
- Has the preference that if a majority of humans would want it to be turned off, it wants to be turned off.
We don't know how to do any of these things. But I like to dream up what I'd want, even if I don't know how to get it.