Tag Archives: gaps

Warning: That RAW image is not really RAW – and why it matters

On the left is an “Auto” Adjustment while the same data on the right is unadjusted. See below and you will discover that there is some serious misinformation on the web about ACR adjustments.

You may know that Photoshop does not know how to open raw files like NEF, CR2. Every time you open a raw file, it is actually opened by Adobe Camera RAW (ACR) which is an internal component common to Photoshop, Photoshop Elements and Lightroom. And there is an Adobe Camera Raw Defaults setting that is automatically applied per each camera type unless the user chooses custom settings. What you may not know is that I highly recommend stacking your star trail images without making any adjustments. Once you make adjustments, especially changes to contrast, tone curve, brightness, shadows or exposure you increase the visibility of gaps and noise.  I explain why this is so in my Down with the Noise Webinar, but for now, just take my word for it!

Confusion Abounds

Unfortunately it is quite complicated to remove the default Camera Raw adjustments due to conflicting details on web sites, including on Adobe’s own FAQ. As my experiments show, the “default” settings for ACR apply adjustments. Adobe says that using CTRL-R (CMD-R on a Mac) resets to the defaults for a RAW file, but it doesn’t reset everything because the default settings do have adjustments!  Below are the choices for selecting, saving and resetting Camera Raw Defaults – you find this menu in the upper right of the ACR display – see more illustrations below.

Fullscreen capture 2212013 85814 AM.bmp

In my tests with ACR 7.0  CTRL-R – which theoretically is the same operation as selecting Camera Raw Defaults – did not remove hand applied adjustments to clarity, tint, noise reduction, sharpening, vibration or saturation, tone curve, and other settings. What CTRL-R actually does is remove adjustments to Exposure, Contrast, Highlights, Shadows, Whites, and Blacks.  Using the Camera Raw Defaults (first highlighted choice in the list above) doesn’t get what you might expect!  So I went further. I set all the values to zero, then used Save Camera Raw Defaults, selected Camera Raw Defaults for the image and opened it using the Open Object button. When you use Open Object ACR creates a .XMP file – sometimes called a sidecar file – that I inspected to see what has been set.  The non-zero settings in the XMP file after saving my custom camera raw defaults and choosing Camera Raw Defaults included the following non-zero settings:

 crs:CameraProfile="Camera Faithful"

When I saved my own Camera Raw Defaults I turned on Chromatic Aberration and Lens Profile correction and overrode the white balance to “Camera Faithful” just to be sure that the new Defaults were actually using my saved default settings. But wait! There are still Brightness and Contrast adjustments listed even though I had set those values to zero.  It is also not clear whether it is applying a tone curve adjustment. The good news is that my saved defaults are NOT doing any sharpening or noise reduction whereas the ACR defaults (the default defaults?) do mess with those.

Further Experiments

Before I tried to set my own Camera Raw Defaults, I followed advice I found online. That is how I discovered that the default, Default Camera Raw settings include both sharpening and color noise reduction.

Camera RAW "Defaults"

Using the “Camera RAW Default” selection from the menu. Some changes are still being applied!

Notice how the settings file (XMP) contains adjustments for color noise reduction, a tone curve, and sharpening. The real head scratcher is that the side-car (XMP) file also shows adjustments to Shadows, Brightness and Contrast – which are NOT shown on the Basic (leftmost) settings panel for the image.  Not knowing the internals of Photoshop, I can not tell if the brightness, shadow and contrast adjustments are actually present or not.

The XMP file and the display do not agree.

Basic does not show adjustments that are in the XMP file!

Unfortunately there are sites that claim that using the CTRL-U (CMD-U) sets all the values to default. This is incorrect. CTRL-U toggles between automatic and not automatic  which is the clickable text Auto in the settings dialog. What I’ve called Default RAW Adjustments in my comparison photo at the top of this article is actually automatic adjustment – I was mislead! What is automatic? It is a roulette wheel whereby you let ACR take its best guess at what it thinks will look right.  Apparently it is pretty smart unless you let ACR do its automatic thing on a night image in which case the result will not be very pleasing.

Camera Raw 7.0  -  Canon EOS 40D 2212013 81342 AM.bmp

The stated Adobe method to reset to Camera Raw Defaults is to use CTRL-R (CMD-R on Mac). After using this magic sequence I see that there is still sharpening, a tone curve and much more.


Yeah, me too.

In fact, the default RAW setting can be per camera per ISO. The bottom line for me is that I do not trust ACR to not mess with my image unless I apply a Linear “Develop Settings” to all the images I’m going to load. And I am not even sure that some adjustments are not still being made despite my strong desire to have my images be unfooled around with.

But Why Do I Care?

A RAW file that has nether been sharpened nor had a tone curve applied looks flat and boringish. Why so boring? A digital camera records images in a linear fashion but our eyes don’t perceive things that way. To prevent people from squawking, ACR by default applies tonal adjustments to convert the raw data into something more adapted to what we see.

Of course you might ask why anyone would ever want to look at the un-adjusted image, and the answer is I wouldn’t want to either… but when stacking the fact that the pixels haven’t been diddled with beforehand makes the result better.

How do you get really RAW Raw Images?

For starters, you can set all of your Raw Defaults to Zero and save them as I noted in the Confusion Abounds section above. As a further belt-and-suspenders technique I also created a preset called “Linear” using the Save Settings menu. I apply the “Linear” preset to my images before I open them to force the sidecar files to be created. Whether ACR is still messing with some of the data is not clear.

But what about Cooked Images?

I don’t always go “really RAW” – I may tweak the settings in ACR for a more pleasing visual appeal. The literature indicates that ACR is a bit better at making adjustments than Photoshop is.  The good news is that you can have your cake and eat it too because no matter what you do in ACR it does not change the data – just the adjustments that are applied to that data.

Here is how I made adjustments to the same image shown earlier along with all the non-zero values from the .XMP (sidecar file).


So if RAW is so Complicated I Should Stick to JPEGS, right?

Heavens no!  If you shoot JPEGS rather than Raw you’re throwing away a lot of good data. The processing to convert the captured data into a JPEG involves lots of decisions made on your behalf, behind your back, and without the ability to change your mind later.  Yes, you can diddle with the image, but you will not get the results you might if you had not let that little conversion monster distort your pristine data. In other words, you’ll eventually regret what happened.



600 Rule?

You may have heard it elsewhere as the “600 rule”.  I first heard about the rule while visiting the Looney Bean in Bishop, California in 2008.  Five photographers sitting in a coffee shop poring over their laptops reviewing what they recently bagged are bound to start talking.  It was my good fortune that one of those present was the very talented Brenda Tharp who first quoted the 600 rule to me.

I, however, have repeated the rule as the “500 Rule” because I think 600 is overly optimistic.  What is the rule?  The rule states that the maximum length of an exposure with stars that doesn’t result in star streaks is achieved by dividing the effective focal length of the lens into the number 600.  A 50mm lens on a 35 mm camera, therefore would allow 600 / 50 = 12 seconds of exposure before streaks are noticeable.  That same 50 mm lens on a 1.6 crop factor camera would only allow 7.5 seconds of exposure.

But Wait. The Rule Isn’t All That Great!

The real number is quite subjective.  A little math reveals that on the Canon 5D Mark II (a full frame camera), with a 16mm lens a pin point star on the celestial equator moves from one pixel* to the next in 5.3 seconds.  But the 600 rule would allow 37 seconds of exposure and the 500 rule 31 seconds.  Both rules will produce streaks on the sensor! The visibility of those streaks will depend on the finished print size and viewing distance.  Print it large and stand close and the streaks will be obvious.

So what does a 30 second exposure look like at the pixel level:

3 Stars at 30 Seconds, 16mm on 35mm Sensor

Clearly those stars are streaking across about 5 pixels* just as the math would bear out.

What is going on here?  The Canon 5D Mark II images are 5634 x 3753 pixels* from a sensor that measures 36 x 24 millimeters. Dividing 36 by 5,634 reveals that the distance from the center of one pixel* to the next is a scant 0.00639 millimeters (or 6.4 microns).

The formula for calculating the distance in millimeters (d) that a star travels across a sensor due to the earths rotation looks like this:

d = t * f / 13750

Where t is time in seconds, f the effective focal length and 13750 is, well 13750.  Is the math scaring you a bit… don’t worry… we’re almost done. Earlier we calculated the pixel* to pixel distance as 0.00639, what we want to find is how long (t) it takes for a star to move that far on the sensor.

0.00639 = t * f / 13750

Solving for f = 16mm we get a t value of 5.3 seconds as I asserted earlier.

But how does that calculate out on a different sensor, the Canon 50D, for example?

The Canon 50D has 4770 pixels across 25.1 mm or an inter-pixel* distance of 0.0053 millimeters.  Substituting into the earlier equation we find that a star marches across a pixel on the 50D with the same 16mm lens in 2.83 seconds.  With a 50mm lens on the same camera… the bad news is the star is speeding from one pixel* to the next in less than a second!

What does an image look like with a 30 second exposure at 16mm on a full frame camera? Remember that the streaks will be 40% longer on the cropped Canon 50D.

Milky Way over Black Rock Desert, Nevada

30 Second Exposure – a close look shows elongated stars.

Scaled down to only 16% of the original image size or seen from a distance no streaking is obvious! We will try not to twitch knowing – because we pixel peeped – that the stars are really dashes not nice round pinpricks of light. And indeed only the eagle eyed are likely to notice the dash-like nature of the stars until the photo is printed large, say at 20 x 30 inches.

What Can We Conclude?

  1. Streaking starts a LOT sooner than any rule you may have learned.
  2. The time it takes to streak depends on the inter-pixel* distance (sensor density / mm) and the focal length.
  3. How much streaking to allow depends on your aesthetic tolerances.
  4. You can not get more or brighter stars by exposing longer; starlight has already given up on one pixel* and moved on to the next in just a few seconds.
  5. The longer the focal length, the more impossible it becomes to prevent streaking.
  6. Gaps in your star trails may be unavoidable if the inter-shot delay (normally 1 second) is long enough to skip pixels*.

Final Note

I carefully added asterisks* to every location where I wrote the word “pixel” in a way that might imply your camera collects light in pixels. You might be wondering why I did that. The answer is: your sensor is comprised of sensels, not pixels. It takes 4 to 9 sensels to create a single pixel depending on the de-mosaic-ing algorithm your camera uses. Maybe you aren’t that picky, but I didn’t want to hear complaints from the purists.

I particularly relish this epiphany because I reported long ago that “longer exposures do not result in more stars“.  I just never got around to doing the extra bit of math – or the experiments – to prove out my assertions.

Real Final Note

A commenter has rightfully taken me to task by pointing out that the perception of a streak is dependent on many things other than just the actual sensor values recorded. In particular, if the image is not enlarged much some streaking will be scarcely or completely unnoticeable because the feature will be too small for the eye to perceive.  The problem with this assertion is that it assumes a lot of preconditions: e.g. how large the print is, how far from the print a viewer stands, and the subjective experience of the viewer.  My real world experience has led me to conclude that it is a reasonable goal to keep the streaking to 2 to 3 pixels or less because that will provide the greatest possible usable magnification (finished viewing size).  There would be no point to collecting a high megapixel image if you can not produce a print proportionately larger or more detailed than a lower megapixel image!

Here is an example that makes my point. I love this image captured on a Canon 5D Mark II. When printed at 20×30″ and viewed at 4 feet there is some streaking. Perhaps only a critical eye would notice, but even an untrained eye will notice when viewed from two feet away.

Famous III [C_035478]

Why North?

I live and travel in the Northern Hemisphere. In fact I have yet to travel south of the equator, so my apologies to those of you from the southern half of the planet for my obvious northern bias.  I believe those of you in the bottom half of the planet can just substitute the word South for North everywhere and everything should be correct.

Grand View [C_009613-686br]

Looking North from Grandview Campground, White Mountains, Bishop, California. Shot at ISO 800, f/3.2 for 6 minutes each. Began at 10:11 PM and ended at 5:35 AM. That is 75 shots x 6 minutes = seven and a half hours.

The results obtained by shooting a long exposure at night depend quite a lot on which direction the camera is pointed. I favor long star exposures with a northern view for many reasons.

  1. Curvature of the star trails is strongest around the north star. Exposures of about 6 hours will appear to be full circles (24 hours of exposures are actually needed to make complete circles and that is not possible in one night except near the North Pole!).
  2. The moon will never intervene into the shot because the moon never passes through the northern sky.
  3. Cassiopeia and Ursa Major (the Big Dipper) are bright constellations that can always be found in the Northern Sky – so there is always some interesting sweep of stars possible. The region immediately around the North Star, however has dimmer stars which may only be captured through long exposures.
  4. With just a smidgeon of star hopping skill it is easy to find the north star which, weather permitting, is always visible in the night sky.
  5. The moon sweeps east to west giving long shadows from the right or left of the subject. And when the moon is highest in the sky it can cast strong face light.
  6. The sun also never appears in the northern sky so it is safe to leave a camera running from before sundown to after sun up. Camera damage can result from a long exposure pointed at the sun.
  7. Since the moon cannot enter into a northern shot a photo can be made regardless of the moon’s phase and for as long as I choose. For shots toward the East, South or West it is important to know the moon phase and location during the hours of shooting to prevent problems from flare or washout.
  8. The stars in the north move the slowest through the field of view which allows them to be brighter and reduces inter-exposure gaps in the trails.
  9. If I know my latitude I know how high to point the camera and be guaranteed to get a circle in the view.
  10. I do not need to know what constellations will be visible in the direction I will shoot.
  11. Two major meteor showers (the Perseids and Quadrantids) and 3 periodic meteor showers (the Giacobinids or Draconids, the Ursids and the Andromedids) are well placed in the northern sky.

There are a few detriments to pointing north, however:

  • Not every situation lends itself to a view from the south. Just as not every weather condition provides clear views to the north.
  • It takes a longer exposure to form a pleasing arc. And if you are like me, you will spend time scheming how to get a complete 360 degree circle (hint, it will take at least 3 nights of shooting over 8 months to get it!)
  • To get a circular arc, I must include at least 10 degrees or so above and below the North Star. The altitude of the north celestial pole constrains the choice of lenses to wide angle only except near the equator.

NOTE: Contrary to popular belief, Polaris, the North Star, is not the brightest star in the sky. Sirius is the brightest star. The brightest objects in the night sky are the moon, and the planets Venus, Mars, Jupiter and Saturn. Also while Polaris is quite NEAR to the North Celestial Pole, it’s not exactly there so even Polaris will make a trail.

Trouble with Long Exposures – Part 2 of 2

In the previous article I discussed 4 of the 6 most common problems that occur with long exposures.  Those problems are:

  1. Poor Focus
  2. Dim Stars (low contrast)
  3. Strange Colors
  4. Purple or Pink Glow

In this installment we tackle these two issues

  1. Gaps in Star Trails
  2. Lots of Noise (Colored Speckles)

Gaps in Star Trails

To oversimplify a bit there are four causes for gaps in star trails created from successive exposures:

  1. Camera limitations
  2. Camera or intervalometer misconfiguration
  3. Processing choices
  4. Weather conditions

Camera limitations: I described this issue in my article “How long does a 30 second exposure take?”  All the Canon cameras I own – including the top of the line 5D Mark II require 32.8 seconds to complete a single 30 second exposure. Well there you go: almost 3 seconds of time where there is no exposure. This problem can be compounded by two common misconfiguration blunders:

  • Failing to allow enough time between exposures when using an intervalometer. Or using the wrong drive mode on the camera.
  • Failing to turn off long exposure noise reduction.

To avoid intervalometer misconfiguration I operate in either continuous exposure mode or bulb mode. I use continuous exposure mode when my exposures will be many and a maximum of 30 seconds – e.g. when trying to capture meteors or planning for a time-lapse animation. In continuous exposure mode I set my intervalometer with a start delay and then program an exposure time of several hours… AND I put my camera in Manual, high-speed continuous exposure mode with a typical exposure of 30 seconds. You do not really need an intervalometer for this – a locking cable release is sufficient.

When I operate in bulb mode, I try to get a moderately long exposure. Usually in the 4 to 10 minute range depending on the sky conditions. In this setup it is very important to put the camera in Bulb exposure and program the intervalometer to leave a 3 second gap between one exposure and the next. I have recently discovered, however that the Canon 5D Mark II will work with my intervalometer set to 1 second intervals. That’s goodness. I am still trying to work out whether the problem is due more to the timer or the camera. I do know that in continuous exposure mode all my cameras require 32.8 seconds per each 30 second exposure. Failure to allow a long enough pause between exposures can cause unexpected results.

Photo 1: For the first half of the evening I mistakenly left long exposure noise reduction on. The result was that half of my shots occured at every-other eight minute intervals.

The “dotted lines” in the circle above were caused by leaving on long exposure noise reduction. The result was that the intervalometer timed an 8 minute exposure, waited three seconds and then pressed the shutter for the next 8 minute exposure. However 3 seconds after the exposure completed it was still doing long exposure noise reduction so that cycle was skipped until the intervalometer released the shutter for the next 3 second “off” interval.

I have gotten into the habit of setting my exposure length to 3 seconds less than what I want… e.g. 9:57 for a 10 minute exposure. I then set a 3 second inter-shot interval. I used to set a 10 minute exposure plus a 3 second gap – but the predictability of starting a new exposure every 10 minutes makes it easier to monitor what is going on.

Another cause for gaps: changing the battery. I can offer the following important tidbits when you need to change the battery.

  • Do not wait for your battery to be exhausted. A partial exposure may not stack well or be completely written to your card. Battery exhaustion will likely occur at an inopportune time.
  • Have everything at hand in advance of the change. For example, keep the battery in your front pocket where your body heat will keep it warm.
  • Practice a battery change BEFORE you start your exposures. Only by practice beforehand will you be able to discover that the battery compartment is blocked by your tripod, or impossible to reach, etc.
  • When you DO change batteries beware! Your camera settings may change dramatically!

Processing choices you make when stacking the star trails also affect whether your gaps will be inconspicuous. Do not do any sharpening until you complete your stacking – and even then avoid sharpening the star trails themselves. The method used to stack trails is significant. However, I have observed that people do not notice gaps even in this image of 19 8-minute exposures printed out at 20×30 inches.

Photo 2: Even though it is composed of 19 eight minute exposures the gaps are never noticed even when printed at 20x30.

Weather conditions can also introduce gaps. In a truly dark sky where clouds are not lit by city glow, moonlight or twilight, clouds become “black holes” and block starlight. Low or fast moving clouds can obscure some, most or all of one or more images in the set. This can be perplexing if you happen to be sleeping during exposures which started and ended with clear skies.  Another problem is dew which may form a fog that diminishes or eliminates some or all of the exposures. Vigilance with a rag, the use of a hood or a dew heater are your only weapons against dew.

Lots of Noise (Speckled Colors)

I purposefully left the noise in Photo 1. It’s quite noticeable in the rock silhouette at the lower right and appears mostly as red specs. Annoying? Well, yes, but it is not the end of the world.  In order of effectiveness here are your best approaches to keep the noise manageable:

  1. Shoot at a lower ISO (100 or 200)
  2. Shoot and stack shorter exposures – longer exposures generate more noise.
  3. Capture the foreground and the star trails separately. A better lit foreground will exhibit less noise.
  4. Shoot during colder seasons – lower temperatures result in lower noise.
  5. Control stray light with a lens hood – and close or cover your viewfinder while exposing.
  6. Use high ISO noise reduction
  7. Use noise reduction post processing tools. Chrominance noise is usually most in need of correction.
  8. Use long exposure noise reduction.

Hopefully you noticed that long exposure noise reduction (LENR) is last on the list. If you are trying to stack star trails it is impossible to get continuous trails with LENR on. It is also the least effective unless you are only going to shoot one shot.

Before we go much further, it is worthwhile to note that there are 4 causes of “noise” and each has a different source. The random speckles are usually what is meant by noise. Those random speckles are created by heat, limitations in the electronics, and things as bizarre as electromagnetic phenomenon like sunspots. No kidding. True noise is by nature random and LENR can not do a thing to combat random noise except to diminish it by reducing the luminance of the offending pixels – which also reduces the sharpness of your image. But there are 3 other kinds of noise that are not random though often lumped into the same general category: hot/stuck or degraded pixels, local heat noise (sometimes called amp glow), and high ISO noise. LENR is effective for these because they are not random.

Hot or stuck pixels usually appear as bright pink, red, blue, green, white or purple spots. They are caused by either electronic problems on the sensor chip or by the dyes used to detect the color.  A pixel detects the intensity of the color red by use of a red dye (inkjet droplet) over a sensor site. If that red dye is insufficiently thick, or missing altogether then that pixel location will always read hot if there is any light falling on it – and if the problem is electronic it may read hot even if no light is striking it. Dead or degraded pixels are just the opposite. Too much dye or dead electronics at a pixel site. Degraded pixels are stuck black or darker than the surrounding pixels and are seldom if ever noticed in night photography.

Locally caused heat noise is noticeable in some cameras and is due to the heat of electronics in proximity to the sensor. In my opinion this problem is a design flaw in the camera. However this kind of noise is repeatable so LENR can help correct it. The “Pink or Purple Glow” that results from this flaw was discussed in Part 1.

High ISO noise has an understandable parallel in the world of audio. Take nearly any cheap radio. Turn it up. At some point the sound will become distorted and harsh. This harshness is because there are limitations in the signal, the amplifier circuitry and the speaker used to produce the sound.  Increasing the ISO in your camera is the photographic equivalent of the audio scenario.  At some point amplifying the light measurements made at each pixel makes the noise more obvious.