[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

I don't know whether this is a toy example, having grid of this size is not uncommon. True, it would make more sense to do distribute more work on each box, if there was any. One has to find a proper balance, as with many other things in life. I simply responded to a question by the OP. George On Thu, 21 Feb 2019 at 01:30, DL Neil <PythonList at danceswithmice.info> wrote: > George > > On 21/02/19 1:15 PM, george trojan wrote: > > def create_box(x_y): > > return geometry.box(x_y[0] - 1, x_y[1], x_y[0], x_y[1] - 1) > > > > x_range = range(1, 1001) > > y_range = range(1, 801) > > x_y_range = list(itertools.product(x_range, y_range)) > > > > grid = list(map(create_box, x_y_range)) > > > > Which creates and populates an 800x1000 ?grid? (represented as a flat > list > > at this point) of ?boxes?, where a box is a shapely.geometry.box(). This > > takes about 10 seconds to run. > > > > Looking at this, I am thinking it would lend itself well to > > parallelization. Since the box at each ?coordinate" is independent of all > > others, it seems I should be able to simply split the list up into chunks > > and process each chunk in parallel on a separate core. To that end, I > > created a multiprocessing pool: > > > I recall a similar discussion when folk were being encouraged to move > away from monolithic and straight-line processing to modular functions - > it is more (CPU-time) efficient to run in a straight line; than it is to > repeatedly call, set-up, execute, and return-from a function or > sub-routine! ie there is an over-head to many/all constructs! > > Isn't the 'problem' that it is a 'toy example'? That the amount of > computing within each parallel process is small in relation to the > inherent 'overhead'. > > Thus, if the code performed a reasonable analytical task within each box > after it had been defined (increased CPU load), would you then notice > the expected difference between the single- and multi-process > implementations? > > > > From AKL to AK > -- > Regards =dn >

- Prev by Date:
**Multiprocessing performance question** - Next by Date:
**Ask for help on using asynctk package** - Previous by thread:
**Multiprocessing performance question** - Next by thread:
**Multiprocessing performance question** - Index(es):