java - Parallelizing many GET requests -


is there efficient way parallelize large numer of requests in java. have file 200,000 lines, each 1 needing request wikimedia. , have write part of response common file. i've pasted main part of code below reference.

while ((line = br.readline()) != null) {     count++;     if ((count % 1000) == 0) {         system.out.println(count + " tags parsed");         fbw.flush();         bw.flush();     }     //system.out.println(line);     string target = new string(line);     if (target.startswith("\"") && (target.endswith("\""))) {         target = target.replaceall("\"", "");     }     string url = "http://en.wikipedia.org/w/api.php?action=query&prop=revisions&format=xml&rvprop=timestamp&rvlimit=1&rvdir=newer&titles=";     url = url + urlencoder.encode(target, "utf-8");     url obj = new url(url);     httpurlconnection con = (httpurlconnection) obj.openconnection();     // optional default     con.setrequestmethod("get");     //add request header     //con.setrequestproperty("user-agent", user_agent);     int responsecode = con.getresponsecode();     //system.out.println("sending 'get' request url: " + url);     bufferedreader in = new bufferedreader(new inputstreamreader(con.getinputstream()));     string inputline;     stringbuffer response = new stringbuffer();     while ((inputline = in.readline()) != null) {         response.append(inputline);              }     document doc = loadxmlfromstring(response.tostring());     nodelist x = doc.getelementsbytagname("revisions");     if (x.getlength() == 1) {         string time = x.item(0).getfirstchild().getattributes().item(0).gettextcontent().substring(0,10).replaceall("-", "");         bw.write(line + "\t" + time + "\n");     } else if (x.getlength() == 2) {         string time = x.item(1).getfirstchild().getattributes().item(0).gettextcontent().substring(0, 10).replaceall("-", "");                   bw.write(line + "\t" + time + "\n");     } else {         fbw.write(line + "\t" + "null" + "\n");     } }

i googled , seems there 2 options. 1 create threads , other use called executor. provide little guidance on 1 more appropriate task?

if really, need via requests, recommend use threadpoolexecutor small thread pool (2 or 3) avoid overloading wikipedia servers. avoid lot of coding ...

also consider using apache httpclient libraries (with persistent connections!).

but better idea use database download option. depending on doing, may able choose 1 of smaller downloads. this page discusses various options.

note: wikipedia prefers people download database dumps (etcetera) rather pounding on web servers.


Comments

Post a Comment

Popular posts from this blog

basic authentication with http post params android -

i doing basic authentication passing username , password , using basicnamevaluepair sending post params response service. my method: public stringbuilder callservicehttppost(string username, string password, string type) { // create new httpclient , post header httpclient httpclient = new defaulthttpclient(); httppost httppost = new httppost(webservice + type); httpresponse response = null; stringbuilder total = new stringbuilder(); try { url url = new url(webservice + type); /*string base64encodedcredentials = base64.encodetostring((username + ":" + password).getbytes(), base64.url_safe | base64.no_wrap);*/ string base64encodedcredentials = "basic " + base64.encodetostring( (username + ":" + password).getbytes(), base64.no_wrap); httppost.setheader("auth...
Read more

vb.net - Virtual Keyboard commands -

i wondering difference between virtual these keyboard commands is: keyeventf_extendedkey , keyeventf_keyup is. everywhere have looked gives me description based off integers , not want know each of them does. you've tagged question vb.net, these have nothing @ vb.net. they're constants defined in windows header files, use win32 api functions. as far difference, can't tell looking @ values. individual values not particularly important, that's why named identifiers used. what's important used , documentation functions tells mean. the first one, keyeventf_extendedkey , used keybdinput structure (which used along e.g. sendinput function) pass information synthesized keyboard input. if flag used, means scan code should interpreted extended key. technically, means scan code preceded prefix byte value 224 (&he0 in hexadecimal notation). the second one, keyeventf_keyup , 1 of flags available use structure. means key being released (going up)...
Read more

c++ - End of file on pipe magic during open -

i have c++ application in starting process(wireshark) following. if (fp == null){ fp = popen(processpath, "r"); //processpath process want start if (!fp){ throw std::invalid_argument("cannot start process"); } fprintf(fp, d_msg);//d_msg input want provide process } else if(fp != null){ fprintf(fp, d_msg); } the problem when execute c++ application, start wireshark error end of file on pipe magic during open what should avoid that? also tried using mkfifo create named pipe , execute it. used this: if (fp == null){ system("mkfifo /tmp/mine.pcap"); fp = popen("wireshark -k -i /tmp/mine.pcap", "r"); if (!fp){ dout << "cannot start wireshark"<<std::endl; throw std::invalid_argument("cannot start wireshark"); } input = fopen("/tmp/mine.pcap", "wb"); fprintf(input , d_msg); fclose(input)...
Read more