| Search and Retrieve NOAO Data | Contact us for help Send feedback |
The process of finding and accessing NOAO data follows several basic steps:
- Searching for data using the query form
- Search for data from your own NOAO observing programs
- Search for any data in the NOAO Archive
- Search form parameters
- Search results
- Staging data for ftp retrieval
- Retrieving data directly from the Archive using cURL
- Working with your downloaded data
The Search Form
|
|
| Caution: | The archive metadata that can be searched are not always complete or perfect! The data come from many telescopes and instruments, each with its own data-taking system, usually written without archival legacy in mind. Useful information is not always stored in FITS header keywords in a standardized way. Some information may occasionally be incorrect or missing (e.g., object coordinates) due to problems with the telescope, instrument, or data-taking systems. Other parameters (e.g., object name) depend on observer input and thus may be missing, misleading or wrong. |
How to find your own NOAO dataIf you are a registered user who is the PI for NOAO observing programs, or a co-I who has been granted authorized access by the PI, select Search My Data from the Search Type menu. Then enter any other constraints that you wish (e.g., observing calendar date, telescope and instrument, etc.) and click Search. If you have not already logged in, you will be directed to the NVO Sign-On page, then back to this form.You may only retrieve and download proprietary data if you are the program PI or an authorized co-I, and have signed in with your registered NVO username! For information on administering co-I access to proprietary data, please see the tutorial on PI/Co-I data access. |
|
How to search all data in the NOAO ArchiveTo search for any NOAO data (not just those from your own observing programs), select Search All Data from the Search Type menu. Any user may search all NOAO data, with or without registering with the NVO or signing in. However, proprietary data can only be staged and retrieved by their owner, who must signed in as a registered user. |
|
Search form parameters
|
Target informationThis section lets you search for data by object name or coordinates.
Note: The search region is defined by constant RA and Dec limits that are one half the search box size away from the specified RA and Dec. This simplistic definition therefore distorts from a square box as one moves toward the celestial poles. |
|
ObservationThis section lets you search by information about the observing program, principle investigator, observing date, or filename.
Note: All FITS files are renamed to a standard nomenclature when they are stored in the archive. The original filenames assigned by the user, however, are recorded in a FITS header keyword, and this filename search uses those names to help find data by those original names. |
|
Telescope & InstrumentThis section lets you select telescopes and instruments to search, and to limit searches by exposure time.
|
|
Data productsThis section provides a set of checkboxes that let you select what sort of data products to search, most notably, raw or pipeline-reduced data, and to select various types of pipeline data products. You may select more than one type of product. If you check no boxes, the search will return all types of raw and reduced data products. The data product types are:
|
|
Search Results and Data SelectionThe results of a query appear in a table that will often be longer and wider than can be displayed in your browser, and there are vertical and horizontal scroll bars that can be used to view the whole table. A query may return many archived data sets (up to a current maximum of 1000 files for unregistered users, or 10000 for registered, signed-in users), but displays only 20 rows of results per page. You may navigate through the results by clicking the page numbers that appear near the top of the Results table. |
Sorting, Filtering and CategorizingThe header of the Results table shows the names of each column. Mousing over the column name gives information about the data in that column. Clicking on the column name will sort the results according to the values in that column.There is a pulldown menu labeled "Filter by". This lets you further restrict the search results to a subset defined by one of the data parameters. For example, you can filter by Telescope, PI, Observing date, etc. Type the value of the parameter into the box to the right of the menu; the Portal will assist you by giving you a list of options that match the string you are typing. Click Go to select only results matching that parameter value; you can revert to the full list of results by clicking Reset. There is also a pulldown menu labeled "Categorize by". This lets you select certain parameters by which to sort your results into separate "categories". Select one of these parameters, and you will see a new set of tabs appear above the Results table, with all of the parameter values from your search. In the example shown below, the user has categorized by Proposal ID, and a series of tabs appear at top organizing the search results by their NOAO program numbers. You can undo this and go back to the full results set by clicking the Uncategorize button. |
|
|
|
|
Selecting Data for Retrieval
|
There are several ways to select data sets for retrieval from the archive.
If you own proprietary data and have signed in, you may retrieve your data.
For unregistered users, only public (i.e., non-proprietary) data may be
selected for retrieval. The public release date for any data set is shown
in a column at the right hand side of the table (see right). If a data set is public,
the Access column will show a link marked Retrieve.
Clicking on this link will immediately start a data download for that file.
The leftmost column of the Results table (see left) includes checkboxes that you may use to select individual data sets. Also you can use the "Selection" pulldown menu, located just above the Results table (see below). This provides options for selecting or de-selecting all visible rows on the current page of Results, or all rows on all pages of the Results table. |
|
|
||
Staging Data for Retrieval
Once you have selected the data that you want, click the button marked "Stage selected rows", just above the table of results. A pop-up box will tell you the volume of data that you are staging, and ask you to confirm that you wish to proceed. This start the process of staging your data for ftp retrieval.
Select the tab labeled Staging Area in order to monitor the progress of your data staging, and to get instructions for download.
The Staging Area
|
The Staging Area is shown below, seen in the process of staging some data. The panels at left
give information about the staging status, the available storage space, and instructions and tips
for retrieving your data. At right there is a list of the data files that are being staged;
this may continue on multiple pages if you are staging more than 20 files. There are several
buttons and selection menus across the top of the page which let you control the staging process.
|
|||||||||||||||
|
The image staging status is summarized in a box at the upper left, showing how many files
have been queued for staging, successfully staged, or for which there
have been errors. The staging status codes have the following meanings:
The list at right also gives the staging status for each individual data set. The file names are shown in bold face when they have been staged successfully. |
||||||||||||||
FTP RetrievalWhen the Imaging Staging Status reports that all of your files have been staged, you may retrieve them by ftp from nvo.noao.edu. The username, password, and the location of your staging area are given at left.
|
|
If you have problems with stagingOccasionally, some data sets may fail to stage properly. If so, you may need to try to restart staging. At the top of the Staging Area page there are several buttons that can be used to control the staging process:
|
|
Cleaning up your FTP staging areaWhen you have finished downloading your data, we suggest that you click Clear my staging area to empty the ftp area and free up staging disk space. Staged data will automatically be deleted after approximately one week. |
|
Retrieving data directly from the Archive with cURL
As an alternative to staging your data for FTP retrieval, you can fetch data using cURL, a command-line tool that allows batch downloads directly from the Archive. In certain circumstances, this may be faster or more robust than standard FTP.
Working with your downloaded data
Data compression:
FITS data stored in the NOAO archive are compressed in order to save space and to speed download transfer.
Before 2010, data were compressed using gzip. From semester 2010A on, the Archive has begun to use "tile compression",
which is a method of handling data compression within the FITS standard (rather than externally compressing existing
FITS files). The NOAO Archive is now using the Rice compression algorithm to create tile-compressed FITS images;
this is substantially faster than standard gzip and achieves greater compression factors. The tile-compressed
data files are recognizable by their ".fz" extension.
Data from NOAO telescopes and instruments are assigned unique filenames when they are stored in the NOAO Science Archive. However, the new archive filenames are generally not very informative. If you are the PI of an observing program, you may find it useful to rename the data files that you retrieve from the archive to the names they had at the telescope.



