We had incremental improvements in most releases from CDH5.5 -> CDH5.12, then a big improvement in CDH5.13 (there was a big rework of the spill-to-disk code), then I expect we'll see another big increase in reliability in CDH6.1 based on the work we've already done. I don't know what version you were experimenting with but it's gotten much better over time. Despite process memory good enough for execution, the query should definitely be able to stay under a query mem_limit by spilling to disk! I've put a lot of work into making that work more reliably over the last couple of years. I believe that even if it exceeds the query limits, it should rather spill data to disk than cancel the query.Ĭan someone reason this error. Note : the query was executed with mem_limit of 5GB. QueryMemory limit exceeded: Error occurred on backend ***** by fragment e4a777f7045b132:53b698be0000004c Memory left in process limit: 59.51 GB Memory left in query limit: -218.77 KB Query(e4a777f7045b132:53b698be00000000): memory limit exceeded. Despite executing a query with a memory limit set(say 5GB) and the impala deamon having sufficient memory, the query gets cancelled owing the error 'memory exceeded'.

Memory limit exceeded facebook flexify