scrapy duplicate filter with csv file -


im trying avoid scraping same information more once, run spider every morning scrape jobs job board, copy them excel , press remove duplicates list using url. in scrapy (i can change txt file csv). happy implement middleware

this pipleing trying use

class craigslistsamplepipeline(object):        def find_row_by_id(item):         open('urllog.txt', 'r') f:                # open txt file urls previous scrapes             urlx = [url.strip() url in f.readlines()] # extract each url             if urlx == item ["website_url"]:              # compare old url url being scraped             raise dropitem('item in db')      # skip record if in url list         return

im sure code wrong, can please suggest how can this, im new explaining each line me alot. hope question makes sense , can me

ive looked @ these posts help, not able solve problem:

how filter csv file using python script

scrapy - spider crawls duplicate urls

how filter duplicate requests based on url in scrapy

use in keyword. so:

 if item['website_url'] in urlx:       raise dropitem('item in db')

you loaded urlx file each line url. list. in keyword checks see if website url in list urlx. if is, returns true. keep in mind comparison case sensitive in example. may want call .lower() on website url , on urls loaded file.

there more efficient ways of doing this, assume want works.


Comments

Post a Comment

Popular posts from this blog

basic authentication with http post params android -

i doing basic authentication passing username , password , using basicnamevaluepair sending post params response service. my method: public stringbuilder callservicehttppost(string username, string password, string type) { // create new httpclient , post header httpclient httpclient = new defaulthttpclient(); httppost httppost = new httppost(webservice + type); httpresponse response = null; stringbuilder total = new stringbuilder(); try { url url = new url(webservice + type); /*string base64encodedcredentials = base64.encodetostring((username + ":" + password).getbytes(), base64.url_safe | base64.no_wrap);*/ string base64encodedcredentials = "basic " + base64.encodetostring( (username + ":" + password).getbytes(), base64.no_wrap); httppost.setheader("auth...
Read more

vb.net - Virtual Keyboard commands -

i wondering difference between virtual these keyboard commands is: keyeventf_extendedkey , keyeventf_keyup is. everywhere have looked gives me description based off integers , not want know each of them does. you've tagged question vb.net, these have nothing @ vb.net. they're constants defined in windows header files, use win32 api functions. as far difference, can't tell looking @ values. individual values not particularly important, that's why named identifiers used. what's important used , documentation functions tells mean. the first one, keyeventf_extendedkey , used keybdinput structure (which used along e.g. sendinput function) pass information synthesized keyboard input. if flag used, means scan code should interpreted extended key. technically, means scan code preceded prefix byte value 224 (&he0 in hexadecimal notation). the second one, keyeventf_keyup , 1 of flags available use structure. means key being released (going up)...
Read more

c++ - End of file on pipe magic during open -

i have c++ application in starting process(wireshark) following. if (fp == null){ fp = popen(processpath, "r"); //processpath process want start if (!fp){ throw std::invalid_argument("cannot start process"); } fprintf(fp, d_msg);//d_msg input want provide process } else if(fp != null){ fprintf(fp, d_msg); } the problem when execute c++ application, start wireshark error end of file on pipe magic during open what should avoid that? also tried using mkfifo create named pipe , execute it. used this: if (fp == null){ system("mkfifo /tmp/mine.pcap"); fp = popen("wireshark -k -i /tmp/mine.pcap", "r"); if (!fp){ dout << "cannot start wireshark"<<std::endl; throw std::invalid_argument("cannot start wireshark"); } input = fopen("/tmp/mine.pcap", "wb"); fprintf(input , d_msg); fclose(input)...
Read more