Memory issue in Python. Dictionary v.s. Database (or can you combine both?) -

i'm trying find patterns of words huge input. using dictionary purpose, , after hours program crashed memoryerror.

i modified program. created database via mysqldb , inserted there values of pattern-index. every word check if in index , if not write index value. problem database approach slow.

i wondering if there way combine dictionaries , database example:

if ram <90% usage:     seek dict     append dict else:     if not (seek dict):         seek database         append database 

using dictionary same purpose of inputting 100 kb of data takes ~1.5 sec

using database same input takes ~84 sec

original input 16 gb . not know yet how take process.

short answer (detailed answer come):

your use of mysql poor, don't want commit @ use database extension of memory. removing commmit should give big improvement

better using mysql use leveldb (pip install leveldb) sync = false

adjust following values memory want use

• block_cache_size = 512*1024*1024 #512mo #the more important
• write_buffer_size = 10*1024*1024 #10mo

as have memoryerror means have 32bits system means total memory enable process can't more 4 go adjust values fit in min(your system memory,4go)