In some Essbase config file settings a zero value indicates an "unlimited" setting. In this case when I tested a zero value it actually set the query result limit to 0 cells. So I don't think there's a way to set it to unlimited. I'm not quite sure why anyone would set the limit to 0 but it's there for you if you need it.
2. Does Essbase run the query and then quit or just figure out that the query is too big and never bother running it?
From the minimal testing I did on this, it seems like Essbase figures out how large the query is going to be and throws an error if it is too big, never actually asking the database for any data. I tested this by creating two identical cubes with different QUERYRESULTLIMIT settings. Then I fired an MDX statement that took about 7 seconds on the cube with a larger setting. The cube with the smaller setting returned the error immediately. More extensive testing on this might be necessary.