David Hoese

Generating a time series of GOES-16 ABI channels

in «articles» by David Hoese
Tags: , , ,

In the endless search for the coolest satellite data visualization I've produced the above video after a request by a coworker. The video consists of GOES-16 ABI data on January 1st, 2019 from 10Z up to 24Z. Full disk images are used to show a time series of each channel from Channel 1 (C01) to Channel 16 (C16).

Disclaimer: I wasn't going to write this up, but changed my mind after the end result was kind of neat. The code shown here was copied after the fact and although ugly it should still work.

Loading the individual channels

The below python code uses SatPy to load ABI L1B NetCDF files, aggregate/average the pixels to a lower 2km resolution for the higher resolution bands, then saves them in 16 separate MPEG-4 video files. As a SatPy maintainer I'd really like all of the filename hackery to be made easier and added to SatPy. Unfortunately I haven't had time to implement it properly yet.

I used data I had access to from the SSEC Data Center. Information on accessing this data outside the SSEC is available on their website.

The below python code in a script ran in about 2.5 hours for the 14 hours of data being processed. I ran this on a development server with 40 logical cores. I used the default dask threaded scheduler which means it created a worker for all 40 cores and likely could have made it go faster by using less workers.

The Setup

We are accessing the Data Center data from a lustre file system and due to locking behavior of the HDF5 library we have to set the following environment variable.

In [ ]:
import os
os.environ['HDF5_USE_FILE_LOCKING'] = 'FALSE'

Generating Scenes

We then need to create a Scene object for each time-step of the data. As mentioned above I'd really like to make this easier in the future, but for now this is what I do in a pinch. I first find all of the channel 1 (C01) files for all times I want. I then take the C01 filename and globify it to match all other channels for that time.

After creating the Scene I load all of the channels (C01-C16) and resample the data (via aggregation) to a 2km resolution. The scene objects are yielded from the generator function to improve performance when using the MultiScene later on; creating the Scene objects when they are needed.

In [ ]:
from satpy import Scene, MultiScene
from glob import glob
from datetime import datetime
from dask.diagnostics import ProgressBar

base_dir = '/arcdata/goes/grb/goes16/{0:%Y}/{0:%Y_%m_%d_%j}/abi/L1b/RadF'
ds_names = ['C{:02d}'.format(x) for x in range(1, 17)]

def scene_generator(base_dir):
    dt = datetime(2019, 1, 1)
    base_dir = base_dir.format(dt)
    # 1200Z to 2359
    c01_files = sorted(glob(os.path.join(base_dir, 'OR_ABI-L1b-RadF-M3C01_G16_s{:%Y%j}[12]*.nc').format(dt)))
    for c01_file in c01_files:
        ctime_idx = c01_file.find('e{:%Y}'.format(dt))
        all_files = glob(c01_file.replace('C01', 'C??')[:ctime_idx] + '*.nc')
        assert len(all_files) == 16
        
        scn = Scene(reader='abi_l1b', filenames=all_files)
        scn.load(ds_names)
        new_scn = scn.resample(scn.min_area(), resampler='native')
        yield new_scn

After we pass the scene generator to the MultiScene we call the save_animation method to start saving the MPEG-4 videos. We specify the name and start_time fields in the filename which will be filled in when the first Scene is generated.

In [ ]:
with ProgressBar():
    mscn = MultiScene(scene_generator(base_dir))
    #mscn.load(['C{:02d}'.format(x) for x in range(1, 17)])
    #new_mscn = mscn.resample(resampler='native')
    mscn.save_animation('{name}_{start_time:%Y%m%d_%H%M%S}.mp4', fps=10, batch_size=4)

Combining the individual videos in to one

To join the individual videos together I used ffmpeg on the command line. First, we make the list of video files that will be merged:

In [ ]:
!for fn in C*.mp4; do echo "file '$fn'" >>channel_videos.txt; done

Then call ffmpeg with the concat option to do the join:

In [ ]:
!ffmpeg -f concat -i channel_videos.txt -c copy channel_videos.mp4

It was at this point that I noticed that the video was very choppy. Likely because of the default settings of the imageio library that SatPy uses to create the video (via ffmpeg). I tried using ffmpeg again to reduce the quality/bitrate for better play back:

In [ ]:
!ffmpeg -i channel_videos.mp4 -vcodec libx264 -crf 38 abi_channel_videos.compress2.mp4

I'm calling this good enough. In case you missed it the video is embedded near the top of this page.

blog comments powered by Disqus