The way that utilitarianism is normally stated is a terrible way to think about it which leads to real errors. Wikipedia:
Utilitarianism [holds] that the proper course of action is the one that maximizes utility
This idea strikes terror into people’s hearts: this is a standard that no-one could possibly live up to. Even Toby Ord occasionally buys himself a treat. This is the heart of the demandingness objection to utilitarianism. I think this definition says both too much and too little; it comments only on the highest point on the scale, where a better definition can illuminate the whole scale. I would rather say this:
A course of action that results in higher utility is proportionally better.
So yes, it’s better to give all your money to GiveDirectly than to spend it all on a yacht. But it’s also better to give £5 to GiveDirectly than nothing; and having given £5, you can feel good that your action is better than giving £0 and wonder if you might give £10 which would be better still.
People spend a lot of effort on trying to work out whether they reach the bar on their actions, and where the bar should be. They are hard on themselves for not reaching the bar they set, and worry that if they stopped being hard on themselves they would slide back and fail to achieve what they could. Utilitarianism, by the first definition, seems to set a bar so high that you can’t hope to reach it. But the truth is, there is no bar; there’s an infinitely long scale of utility. And so the question is not “is this the very best I can do”, but “can I do better than this? How much better?”
(A post I’ve been meaning to write for some time finally prompted by a blog post by Julia Wise, which in turn arose out of a conversation with me, her and Jeff Kaufman)
Minds aren't magic 
Probably the most useful thing I learned from Dawkins is that the ideal is not a choice. Real choices are between two alternatives with different utility. He was talking about alleles and biological utility, but it’s good advice.
As for utilitarianism, I think the “predicting outcomes” objection and related problem of agency is the main one. If I take a life to save more lives society will not trust or welcome that decision outside of simplistic scenarios.
I think it’s worth noting that the two are less equivalent than they might seem on the surface.
As you might recall, I’m broadly anti the idea of utility functions. I’ve recently come around to the idea that something like what you’re describing is probably still a good decision making technique (or at least a better decision making technique than just going with your gut). However I tend to think that maximizing a utility function that makes perfectly good sense for the sort of decisions you typically actually make is an actively bad idea.
The reason being that a utility functions is mostly trained on the decisions it gets used for, so will tend to ignore details that aren’t important in those decisions but that turn out to be super important in the macro scale. Generally you aren’t presented with decisions that look like maxima for your trained utility, so it’s not been well tested in that region and thus is not likely to produce good results.
This is also why I tend to be suspicious of “And that’s why you should give all your money to charity” or Pascal’s mugging conclusions. I think they’re more likely to be showing a breakdown of your model in edge cases you haven’t incorporated due to lack of experience than they are to be demonstrating real insight.
Alternately, your utilitarian calculations are only as useful as your predictive ability. With smaller stakes, it’s easier to be accurate in the first place and easier to verify after the fact. To a risk-averse mind, this sounds much more sensible.