Data Processing
All sky microwave maps from COBE in different stages of image processing. (Credit: NASA/COBE)
Have you ever wondered what data from a spacecraft actually look like? You may think that the picture you see from the Hubble Space Telescope or the Chandra X-ray Observatory is exactly what the satellite in orbit sends down to Earth, like broadcasts from a tv station, but that's not the case.
Data from a satellite, or "telemetry," is usually in a special format, like a code, designed so a lot of information can be transmitted in a very short time. It's the job of the Data Processing group to turn the telemetry or raw data from a satellite into something that astronomers can easily read and interpret.
Packets, and Realtime Data
Data is generally sent in packets: specific amounts of bit-sized data that have a time stamp. Transmission of these packets can be uneven. For example, part of the relay system may 'drop packets' due to static or transmission errors. Those dropped packets are retransmitted, so no data is lost. Also, stored data and realtime data can be transmitted at the same time, each with their different time stamps.
The end result of these scenarios is that the data received on the ground, while complete, may not be in the proper order. The packet's time stamp lets you reassemble the data in the order observed, rather than the order you receive it. Naturally, it makes sense to wait until your entire transmission is complete before doing the final sorting.
The sorting can be done by either the ground station, or your local processing center. For both, the method is the same. You store all the incoming telemetry on your computer. Then, when the transmission is complete (or all data tapes have been received from remote sites), you sort the packets by time stamps to create, essentially, a perfect copy of what had originally been on the satellite.
What Comes In To Data Processing
A satellite in Earth orbit has several types of data it may send to people on the ground. It can send science data, like observations of matter falling into black holes in the case of an X-ray telescope satellite. It can send "housekeeping" data about the spacecraft's health, like how the solar panels are doing and whether the battery is fully charged. And it can send "attitude" data, which tells exactly where the satellite is pointed and how high above the Earth it currently is.
For an astronomer to use a satellite, he or she needs to know all these things at once: *where* was the satellite looking, *what* did it see, and was everything functioning normally at the time? The problem is, the satellite can't tell you everything it's doing at once. The different types of data come from different systems on board the satellite: the telescope collects science data, the power system keeps track of the solar panels and any batteries, the "attitude control" system tells where the satellite is pointed, etc. When it's time to send data to the ground, the different systems take turns, like tv reporters on the evening news.
When you watch the evening news, you might hear what the government did today, then what the weather was like, and then what happened in sports. You don't hear one reporter say, "Today, at 3pm, the President gave a speech, the Yankees won the first of a double-header, and a bad storm ripped through Nebraska. Then, at 5pm, Congress worked late on some legislation, the Orioles had a great triple play, and there was flooding in Florida." Instead, one reporter tells you what the weather was like all over the country, and after that another reporter tells you everything that happened that day in sports, etc.
It's the same with an orbiting satellite: first, you might get all the data about where the satellite pointed that day, then all the things the telescope saw, then a read out of the spacecraft's health. The Ground System makes sure all the data are received correctly, then sends everything to Data Processing, where the tricky task of matching up the separate reports from the different satellite systems begins.
One Billion Bytes A Day
Data Processing is basically one big computer program: it takes in the individual subsystem reports the satellite sent down, puts them all together, and resorts them so that things that happened at the same time are all together in one file. These files are converted into a special easy-to-read format that astronomers use, called "FITS". FITS stands for "Flexible Image Transport Standard", in other words, a simple way for astronomers to write data that can be read on many different systems by very different programs. Before astronomers started commonly using FITS files, data from one satellite or telescope often couldn't be read without someone first writing a special program to read it.
Sorting data from a satellite doesn't sound like too big a task - what's all the fuss about Data Processing? Satellites today frequently send a *lot* of data to Earth, so Data Processing has to work pretty hard to match everything up. For example, NASA's X-ray Timing Explorer Satellite (XTE) generates about 450 Megabytes of "raw" data a day. The small "floppy" diskettes that personal computer use hold less than 1 Mb of data, so that's already more than 450 floppy disks of data coming from XTE a day. Also, the raw data is in a special compressed format, and packed into subsystem "reports", to boot.
Once Data Processing finishes putting day's worth of XTE data into FITS format, the data takes up even more space. Even though FITS is easy to use, it is not compressed. The data have also been broken up into individual files, so that everything that occurred on board the satellite at the same time is now in one file. When a day's worth of XTE data leaves Data Processing, it takes up over a Gigabyte of disk space, or more than 1,000 floppy diskettes. That's over 1,000,000,000 bytes of data! And XTE's Data Processing program is fast enough to process three times this much data in one 8-hour day.
How big does a computer program have to be to handle this much data? For XTE, Data Processing takes about 10,000 lines of computer code. As you can imagine, it took several programmers many months to write and test that much code. But it pays off - when finished, the Data Processing programs can run for years, processing all the data that comes from a satellite, faster than the satellite can send it to the ground!