![\"memcached-1\"](\"/361a88243c7d3be246f1e738d71cc778/memcached1.webp\")
Index is a typical way to speed-up queries in normal database system. There is no difference between MongoDB and a document-based database system. This article gives insight about the index in MongoDB, for query optimization.
\n_id is an ObjectId object, 12-byte BSON type that guarantees uniqueness within the collection. The ObjectId is generated based on timestamp, machine ID, process ID, and a process-local incremental counter.
\nFor a single-field index and sort operations, the sort order (i.e. ascending or descending) of the index key does not matter because MongoDB can traverse the index in either direction. The value of index is the type of index. For example, 1 indicates ascending order and -1 specifies the descending order.
\ndb.friends.createIndex( { "name" : 1 } )Compound Field
\nThe order of fields listed in a compound index has significance. For instance, if a compound index consists of { userid: 1, score: -1 }, the index sorts first by userid and then, within each userid value, sorts by score.
\ndb.products.createIndex( { "item": 1, "stock": 1 } )Multiple Key
\nMongoDB uses multiple index to index the content in an array. MongoDB creates separate index entries for every element of the array. You do not need explicitly create multiple key.
\nA collection can have at most one text index.
\nPerformance cost for text index:
\ntext indexes can be large. They contain one index entry for each unique post-stemmed word in each indexed field for each document inserted.
\ntext indexes will impact insertion throughput because MongoDB must add an index entry for each unique post-stemmed word in each indexed field of each new source document.
db.reviews.createIndex( { comments: "text" } )Query content by its hashed value. The hash is a function to computed by its value. The hashed value is designed to be distinct value. The one advantage is it is so quick, which take O(1) at most but by contract the normal binary search tree will take O(Log(N)). Hash will be theoretically quicker than normal binary search tree implementation. But the disadvantage is hash index performing range search will be extremely slowly than normal index.
\nThis an example in python to build a hash index
\ndb.active.createIndex( { a: "hashed" } )As my previous blog, I use the python web Crawler library to help crawl the static website. For the Scrapy, there can be customize download middle ware, which can deal with static content in the website like JavaScript.
\nHowever, the Scrapy already helps us with much of the underlying implementation, for example, it uses it own dispatcher and it has pipeline for dealing the parsing word after download. One drawback for using such library is hard to deal with some strange bugs occurring because they run the paralleled jobs.
\nFor this tutorial, I want to show the structure of a simple and efficient web crawler.
\nFirst of all, we need a scheduler, who can paralleled the job. Because the most of the time is on the requesting. I use the gevent to schedule the jobs. Gevent uses the libevent as its underlying library, which combines the multithreading and event-based techniques to parallel the job.
\nThere is the sample code:
\nimport gevent\nfrom gevent import Greenlet\nfrom gevent import monkey\nfrom selenium import webdriver\nmonkey.patch_socket()\nclass WebCrawler:\n def __init__(self,urls=[],num_worker = 1):\n self.url_queue = Queue()\n self.num_worker = num_worker\n def worker(self,pid):\n driver = self.initializeAnImegaDisabledDriver() #initilize the webdirver\n#TODO catch the exception\n while not self.url_queue.empty():\n url = self.url_queue.get()\n self.driver.get(url)\n elem = self.driver.find_elements_by_xpath("//script | //iframe | //img") # get such element from webpage\n def run(self):\n jobs = [gevent.spawn(self.worker,i) for i in xrange(self.num_worker)]The next part is the headless browser part. I use the phantomjs with --webdriver=4444 --disk-cache=true --ignore-ssl-errors=true --load-images=false --max-disk-cache-size=100000. You can get the detailed option from their documents.
Phantomjs uses selenium webdriver as front-end to handle the request. However phantomjs is using the webkit and QT as its underlying browser and controller. It has memory leak bugs therefore the phantomjs will consume ton of memory and it only can use one core of your CPU but you can deploy many instances of the phantomjs on different ports. I wrote a daemon process to monitor the memory and its situation but later I realize I can use Perl script to get the status of process and when it exceeds the limits like 1G memory and send kill signal to the process.
\nTo speed up the crawler, I choose to use static browser to verify the website first because the website is bad written, there might be deadlock occurring so just skip them.
\n","frontmatter":{"title":"Write a highly efficient python Web Crawler","author":{"id":"Mark Duan","github":null,"avatar":null},"date":"July 14, 2015","updated_date":null,"tags":["Python","Coding"],"coverImage":{"childImageSharp":{"fluid":{"aspectRatio":1,"src":"/static/c068d4f41cc4f2a33e3bd2737ca105df/7fbdd/python-web-crawler.webp","srcSet":"/static/c068d4f41cc4f2a33e3bd2737ca105df/61e93/python-web-crawler.webp 200w,\n/static/c068d4f41cc4f2a33e3bd2737ca105df/1f5c5/python-web-crawler.webp 400w,\n/static/c068d4f41cc4f2a33e3bd2737ca105df/7fbdd/python-web-crawler.webp 610w","sizes":"(max-width: 610px) 100vw, 610px"}}}},"fields":{"authorId":"Mark Duan","slug":"/engineering/write-a-highly-efficient-python-web-crawler/"}}},{"node":{"id":"177215ed-17df-5704-99e4-6260d06c47fb","html":"The memcached is one of the most popular open source on-memory key-value caching systems. I will briefly talk about the design of memory management of memcached.
\n
/* powers-of-N allocation structures */\ntypedef struct {\n unsigned int size; /* sizes of items */\n unsigned int perslab; /* how many items per slab */\n void *slots; /* list of item ptrs */\n unsigned int sl_curr; /* total free items in list */\n unsigned int slabs; /* how many slabs were allocated for this class */\n void **slab_list; /* array of slab pointers */\n unsigned int list_size; /* size of prev array */\n unsigned int killing; /* index+1 of dying slab, or zero if none */\n size_t requested; /* The number of requested bytes */\n} slabclass_t;This is the struct declaration of slabclass_t. Each slab class contains the same size of chunk, but different classes have different chunk sizes. The size is calculated by this algorithm:
\n<!--while (++i < POWER_LARGEST && size The content value factor is defined when memcached memory is deployed with -f, which can change the size between slab classes. For this loop, the size is multiplied by a specified factor.
\n![\"memcached-2\"](\"/af60c92dc1627ea9224338bbf10df1a8/memcached-2.webp\")
You can get this information by adding -vvv and you can use the command
\nstats slabs under telnet connection ip portEvery time when a new memory needs to be allocated. It will scan the slab class to find the most suitable class to store the chunk.
\nFrom memcached’s wiki:
\nOverview: Memcached 1.4.11. Fixes race conditions and crashes introduced in 1.4.10. Adds the ability to rebalance and reassign slab memory.
\nLong running instances of memcached memory may run into an issue where all available memory has been assigned to a specific slab class (say items of roughly size 100 bytes). Later the application starts storing more of its data into a different slab class (items around 200 bytes). Memcached could not use the 100 byte chunks to satisfy the 200 byte requests, and thus you would be able to store very few 200 byte items.
\n1.4.11 introduces the ability to reassign slab pages. This is a beta feature and the commands may change for the next few releases, so please keep this in mind. When the commands are finalized they will be noted in the release notes
\nWhile slab reassign is a manual feature, there is also the start of an automatic memory reassignment algorithm.
\nFrom the source code in slabs.c we can see, memcached uses two threads to monitor the slabs class, one is to do maintenance and another one is to do the re-balance the class.
\nMemcached defines a global variable struct slab\\rebalance slab_rebal,_ which is used to store the start, end information of slab. s_clsid is the source slab id and d_clsid is the destination slab id. The detailed blog in Chinese memcached源码分析—–slab automove和slab rebalance could be helpful.
\nstruct slab_rebalance {\n void *slab_start;\n void *slab_end;\n void *slab_pos;\n int s_clsid;\n int d_clsid;\n int busy_items;\n uint8_t done;\n};Memcached implements its own memory pool, which is used to avoid system memory allocation and memory fragmentation. That will make your memory efficient and easy to manage. Here is a demo implementation of memory pool. Basically it is a large pre-allocated chunk of memory.
\n mem_avail) {\n return NULL;\n }\n /* mem_current pointer _must_ be aligned!!! */\n if (size % CHUNK_ALIGN_BYTES) {\n size += CHUNK_ALIGN_BYTES - (size % CHUNK_ALIGN_BYTES);\n }\n mem_current = ((char*)mem_current) + size;\n if (size