• site home
  • blog home
  • galleries
  • contact
  • underwater
  • the bleeding edge

the last word

Photography meets digital computer technology. Photography wins -- most of the time.

You are here: Home / The Last Word / More slit scan experimants — parallel processing

More slit scan experimants — parallel processing

December 3, 2016 JimK 11 Comments

This post is part of a series about some experiments I’m doing combining space and time in slit scan photographs. The series starts here.

When you take a set of 10,000 5200×3500 pixel images and run slit scan simulations against them, there are a lot of options:

  • What image to start with.
  • Which way to run the simulated scan.
  • How fast to move the simulated slit.
  • How diffuse the edges of the slit should be.

There are way more than 10,000 ways to process all those images, and probably several thousand that yield meaningfully different results. It takes between 20 minutes and two hours to process an image. That means that opportunities for experimentation are limited. I’ve tried reducing the size of the images, finding something that works, then  going back to the big images, and trying to get something similar. I don’t like that workflow, and it’s pretty hit and miss.

What to do?

Heck, this is photography! You already know the answer: pour money on it.

I called up MathWorks and ordered the Parallel Processing Toolkit for Matlab. The price wasn’t actually all that high. I spet about five minutes scanning the documentation, and found a tool that will work for me. It’s a construction similar to a for loop, but one that allows parallel execution. It’s called parfor. I was careful to write a loop for processing images in parallel where any iteration did not depend on the result of any other iteration. I was also careful to create any class instances inside the loop, so that each parallel interation would get its own class instance. 

When I ran it, I got a bunch of Matlab processes:

matlab-parral-resource

There’s a place  in the Matlab console for tuning the parallel execution. About the only thing I needed to do was to pick the maximum number of parallel workers so that the combined process set was just under being disk-limited, and that I had enough CPU capacity left to do my usual work while Matlab did its thing. I picked 24 workers, which gave me resource utilization like this:

matlab-parral-resource-5

 

Then I proceeded to make images like these:

14a-slid-rev-start-4858-ovr0-8px-edit

14a-slid-fwd-start-4558-ovr0-8px-edit

14a-slid-fwd-start-6058-ovr0-4px-edit

6a-slid-rev-start-12463-ovr0-5px-edit

14a-slid-rev-start-7558-ovr0-4px-edit

14a-slid-rev-start-4558-ovr0-4px-edit

The Last Word

← Sony a7RII FW 3.3 raw dark-field histograms, ISO 3200-25600 More slit scan experiments — the agony of choosing →

Comments

  1. Jack Hogan says

    December 4, 2016 at 5:10 am

    How much of a speed improvement did the parallel processing toolbox provide Jim? Do you use the GPU at all?

    Reply
    • Jim says

      December 4, 2016 at 7:33 am

      Running 24 workers, it’s about 24 times as fast. I’m not using the GPU, just parfor loops.

      Reply
  2. Bryn says

    December 4, 2016 at 7:24 am

    Very cool. Been thinking of speeding up Lightroom by putting images on 10gbe server and only operating on subsets on multiple different computers.

    10000 jpeg images could fit in ram if you are willing to throw money at it

    Reply
    • Jim says

      December 4, 2016 at 7:32 am

      Yes. I only (only?) have 256GB of RAM.

      Reply
      • Bryn says

        December 4, 2016 at 7:35 am

        And 197gb free for a ram disk 🙂 I’ve considered using a ramdisk for Lightroom but seemed dangerous. But for your read only situation it’d be perfect. Anyway I assume your disk speed limiter is on an ssd?

        Reply
        • Jim says

          December 4, 2016 at 8:09 am

          I’m considering a RAM disk, but I’d rather figure out a way to get Win 10 to use a bigger disk cache. I am also looking into a SSD striped array. I’m using spinning rust now.

          Reply
          • Bryn says

            December 4, 2016 at 8:12 am

            I recently invested in a couple of these:
            4TB Samsung PM863a Entrerprise SSD (refurb, less than 5TBW) w/ 3 Year Warranty & $1000 + free shipping

            https://slickdeals.net/share/iphone_app/t/9241219

            Sent using Slickdeals for iOS

            Reply
          • Bryn Forbes says

            February 26, 2017 at 10:52 am

            Did you ever try a ramdisk? I have 128gb of ram now, and thinking of preloading cinemadng files (individual file for every video frame), have done a few minutes of googling and seems like softperfect seems to be actively developed, and recommended in a number of comments on the web. Found anything better?

            Reply
            • JimK says

              February 26, 2017 at 10:58 am

              I don’t have enough RAM to fit a whole series of images in RAM. I have 256GB, which seems like a lot, but is not enough for that. However, by spawning 24 parallel Matlab workers and assigning each to a core, I can get the synthetic slit scan rendering to the point where the CPU load is about 70%, which is as close to CPU-bound as I’d like to be and still be able to use the computer for other things.

              Reply
  3. Bryn says

    December 4, 2016 at 7:45 am

    Also you could speed up your search by working on smaller jpegs then rerendering full resolution your favorites

    Reply
    • Jim says

      December 4, 2016 at 8:07 am

      I tried that before, and I may return to it if I can automate the translation of the parameters to process the small images to the equivalent ones for the big images. I’m sure it can be (approximately) done, but I just need to buckle down and write the code.

      Reply

Leave a Reply Cancel reply

Your email address will not be published. Required fields are marked *

May 2025
S M T W T F S
 123
45678910
11121314151617
18192021222324
25262728293031
« Apr    

Articles

  • About
    • Patents and papers about color
    • Who am I?
  • How to…
    • Backing up photographic images
    • How to change email providers
    • How to shoot slanted edge images for me
  • Lens screening testing
    • Equipment and Software
    • Examples
      • Bad and OK 200-600 at 600
      • Excellent 180-400 zoom
      • Fair 14-30mm zoom
      • Good 100-200 mm MF zoom
      • Good 100-400 zoom
      • Good 100mm lens on P1 P45+
      • Good 120mm MF lens
      • Good 18mm FF lens
      • Good 24-105 mm FF lens
      • Good 24-70 FF zoom
      • Good 35 mm FF lens
      • Good 35-70 MF lens
      • Good 60 mm lens on IQ3-100
      • Good 63 mm MF lens
      • Good 65 mm FF lens
      • Good 85 mm FF lens
      • Good and bad 25mm FF lenses
      • Good zoom at 24 mm
      • Marginal 18mm lens
      • Marginal 35mm FF lens
      • Mildly problematic 55 mm FF lens
      • OK 16-35mm zoom
      • OK 60mm lens on P1 P45+
      • OK Sony 600mm f/4
      • Pretty good 16-35 FF zoom
      • Pretty good 90mm FF lens
      • Problematic 400 mm FF lens
      • Tilted 20 mm f/1.8 FF lens
      • Tilted 30 mm MF lens
      • Tilted 50 mm FF lens
      • Two 15mm FF lenses
    • Found a problem – now what?
    • Goals for this test
    • Minimum target distances
      • MFT
      • APS-C
      • Full frame
      • Small medium format
    • Printable Siemens Star targets
    • Target size on sensor
      • MFT
      • APS-C
      • Full frame
      • Small medium format
    • Test instructions — postproduction
    • Test instructions — reading the images
    • Test instructions – capture
    • Theory of the test
    • What’s wrong with conventional lens screening?
  • Previsualization heresy
  • Privacy Policy
  • Recommended photographic web sites
  • Using in-camera histograms for ETTR
    • Acknowledgments
    • Why ETTR?
    • Normal in-camera histograms
    • Image processing for in-camera histograms
    • Making the in-camera histogram closely represent the raw histogram
    • Shortcuts to UniWB
    • Preparing for monitor-based UniWB
    • A one-step UniWB procedure
    • The math behind the one-step method
    • Iteration using Newton’s Method

Category List

Recent Comments

  • JimK on Goldilocks and the three flashes
  • DC Wedding Photographer on Goldilocks and the three flashes
  • Wedding Photographer in DC on The 16-Bit Fallacy: Why More Isn’t Always Better in Medium Format Cameras
  • JimK on Fujifilm GFX 100S II precision
  • Renjie Zhu on Fujifilm GFX 100S II precision
  • JimK on Fuji 20-35/4 landscape field curvature at 23mm vs 23/4 GF
  • Ivo de Man on Fuji 20-35/4 landscape field curvature at 23mm vs 23/4 GF
  • JimK on Fuji 20-35/4 landscape field curvature at 23mm vs 23/4 GF
  • JimK on Fuji 20-35/4 landscape field curvature at 23mm vs 23/4 GF
  • Ivo de Man on Fuji 20-35/4 landscape field curvature at 23mm vs 23/4 GF

Archives

Copyright © 2025 · Daily Dish Pro On Genesis Framework · WordPress · Log in

Unless otherwise noted, all images copyright Jim Kasson.