List datasets available on GDEX, with their IDs and titles.
The full catalog has ~1700 datasets — far too many to return at once.
Always pass `query` to filter by keyword unless the user specifically
wants to browse the whole catalog page by page.
Args:
query: Keyword(s) to filter by, matched case-insensitively as a substring
against dataset id and title. Leave empty to browse unfiltered.
limit: Max number of datasets to return (default 50, capped at 500)
offset: Number of matching datasets to skip, for paging through results
Parameterslimitqueryoffset
Return full metadata for a GDEX dataset (parameters, temporal range, spatial coverage, etc.).
Args:
dsid: Dataset ID (dNNNNNN), e.g. d083002
Parametersdsidrequired
Return one metadata field for a dataset.
Prefer describe_dataset when the user wants a general summary covering
several of these at once (abstract, temporal, spatial_coverage,
variables, data_formats, volume) — this tool is for pulling a single
field, including the fields describe_dataset doesn't cover
(publications, contributors, related_datasets, documentation).
Args:
dsid: Dataset ID (dNNNNNN), e.g. d083002
field: One of: abstract, variables, temporal, spatial_coverage,
publications, contributors, data_formats, volume,
related_datasets, documentation
Parametersdsidrequiredfieldrequired
Return a combined overview of a dataset — abstract, temporal coverage,
spatial coverage, variables, data formats, and volume — in a single call.
Prefer this over calling get_dataset_field repeatedly when the user wants
a general summary of a dataset. If one of the underlying fields fails to
load, it's returned as {"error": ...} rather than failing the whole call.
Args:
dsid: Dataset ID (dNNNNNN), e.g. d083002
Parametersdsidrequired
Return file groups for a dataset. Pass gindex to get child groups under a parent.
Groups nest (dataset -> format -> year -> month, or similar, varying by
dataset) and there's no way to predict a child's gindex in advance — each
group's gindex/url is dataset-specific and only knowable from the parent
response. To drill down, read the gindex (or url) off a row in this
response and pass that as the next call's gindex.
This never returns file rows, only groups — every response stays small
regardless of how many files the dataset holds, unlike get_dataset_files.
Descend until a call returns empty ({} or []): that means the gindex you
just called with is a leaf with no further subgroups, so it's safe to
call get_dataset_files there for the actual files. find_dataset_files
automates exactly this walk if you'd rather not do it by hand.
At the top level, watch for a "Kerchunk Reference Files" (or similar
ARCO-related) group alongside the raw-format groups. For an analysis
task, prefer pulling from there (see also has_arco/get_arco_variables)
over a raw data file when one's available — it avoids downloading a
whole file just to read a subset of it.
Args:
dsid: Dataset ID (dNNNNNN), e.g. d083002
gindex: Optional group index to fetch child groups
Parametersdsidrequiredgindex
Return a paginated file listing for a dataset or a specific group.
Depending on how deep gindex is in the group hierarchy, this returns
either actual file rows or another layer of subgroup summaries — there's
no way to tell in advance which you'll get. If you get subgroups, read
the gindex (or url) off a row and call again with that gindex to go one
level deeper; gindex values are dataset-specific and can't be guessed.
A shallow gindex on a large dataset can return a very large response
(thousands of files) — the file-row portion of the response is capped at
500 rows (look for "_truncated": true). Two ways to avoid hitting that
cap instead of drilling down group by group: pass filter_wfile with a
filename pattern (e.g. a date like "20220808") to filter down to matching
files, or page through a known group's results with `page`. filter_wfile
only filters actual file rows, so it has no effect at a gindex that's
still returning a subgroup summary rather than files — if a first attempt
comes back unfiltered, descend one level (see get_file_groups) and retry
there.
Args:
dsid: Dataset ID (dNNNNNN), e.g. d083002
gindex: Optional group index to filter files
page: Page number to fetch (for a group with more files than fit on one page)
filter_wfile: Filter files by name pattern, e.g. "20220808" to match a date
fl: File list source (defaults to "web" server-side)
Parametersfldsidrequiredpagegindexfilter_wfile
Search a dataset's file-group hierarchy for files matching a name
pattern (e.g. a date like "20220808"), without ever pulling a large,
context-blowing file listing.
Automates the pattern described in get_file_groups: recursively calls
get_file_groups, descending into every child gindex, until a gindex
returns no further children (a leaf group) — then calls get_dataset_files
there with filter_wfile=name_pattern and keeps only the matches. Prefer
this over manually drilling with get_file_groups/get_dataset_files when
you don't already know roughly where in the hierarchy to look.
A dataset's hierarchy can be large (hundreds of leaf groups), and this
tool has no way to know in advance which branches might contain a match,
so it may need to visit many groups to be thorough. Pass start_gindex if
you already know a good starting point (e.g. from a prior get_file_groups
call, or a related dataset's structure) to narrow and speed up the
search. If the number of groups visited hits max_groups_visited, the
search stops early and `stopped_early` comes back true — narrow with
start_gindex and retry, or raise the cap.
Args:
dsid: Dataset ID (dNNNNNN), e.g. d083002
name_pattern: Filename substring/pattern to match, e.g. "20220808" for a date
start_gindex: Optional group index to start the search from, instead of the dataset root
max_groups_visited: Safety cap on groups traversed before giving up (default 300)
Parametersdsidrequiredname_patternrequiredstart_gindexmax_groups_visited
Return the file-search datatypes available for a dataset (a subset of
"grid", "cyclone_fix", "sensor"). Call this before the other filesearch_*
tools to know which one(s) apply — a dataset only supports search for the
datatypes it actually contains.
Args:
dsid: Dataset ID (dNNNNNN), e.g. d083002
Parametersdsidrequired
get_filesearch_grid_filters
Return the valid parameter/product/grid/level codes and date range for
"grid" datatype file search on a dataset. Use this to discover the codes
to pass to get_filesearch_grid_files, optionally narrowed by any filters
you already know you want.
Args:
dsid: Dataset ID (dNNNNNN), e.g. d083002
valid_datetime_min: Restrict to data valid on/after "YYYY-MM-DD HH:MM"
valid_datetime_max: Restrict to data valid on/before "YYYY-MM-DD HH:MM"
parameters: Restrict to specified parameter code(s)
products: Restrict to specified product code(s)
grids: Restrict to specified grid code(s)
levels: Restrict to specified vertical level code(s)
Parametersdsidrequiredgridslevelsproductsparametersvalid_datetime_maxvalid_datetime_min
get_filesearch_cyclone_fix_filters
Return the valid date range and other filters for "cyclone_fix"
datatype file search on a dataset.
Args:
dsid: Dataset ID (dNNNNNN), e.g. d083002
valid_datetime_min: Restrict to data valid on/after "YYYY-MM-DD HH:MM"
valid_datetime_max: Restrict to data valid on/before "YYYY-MM-DD HH:MM"
Parametersdsidrequiredvalid_datetime_maxvalid_datetime_min
get_filesearch_sensor_filters
Return the valid date range and other filters for "sensor" datatype
file search on a dataset.
Args:
dsid: Dataset ID (dNNNNNN), e.g. d083002
valid_date_min: Restrict to data valid on/after "YYYY-MM-DD"
valid_date_max: Restrict to data valid on/before "YYYY-MM-DD"
Parametersdsidrequiredvalid_date_maxvalid_date_min
get_filesearch_grid_files
Search for data files containing "grid" datatype data, filtered by
parameter code(s) and optionally by time range, product, grid, or level.
Results are paginated; use get_filesearch_result_page with the returned
result_id to fetch additional pages. Use get_filesearch_grid_filters
first to find valid parameter/product/grid/level codes for this dataset.
Args:
dsid: Dataset ID (dNNNNNN), e.g. d083002
parameters: Parameter code(s) to search for (required, at least one)
valid_datetime_min: Restrict to data valid on/after "YYYY-MM-DD HH:MM"
valid_datetime_max: Restrict to data valid on/before "YYYY-MM-DD HH:MM"
products: Restrict to specified product code(s)
grids: Restrict to specified grid code(s)
levels: Restrict to specified vertical level code(s)
Parametersdsidrequiredgridslevelsproductsparametersrequiredvalid_datetime_maxvalid_datetime_min
get_filesearch_cyclone_fix_files
Search for data files containing "cyclone_fix" datatype data, optionally
filtered by time range. Results are paginated; use get_filesearch_result_page
with the returned result_id to fetch additional pages.
Args:
dsid: Dataset ID (dNNNNNN), e.g. d083002
valid_datetime_min: Restrict to data valid on/after "YYYY-MM-DD HH:MM"
valid_datetime_max: Restrict to data valid on/before "YYYY-MM-DD HH:MM"
Parametersdsidrequiredvalid_datetime_maxvalid_datetime_min
get_filesearch_sensor_files
Search for data files containing "sensor" datatype data, optionally
filtered by date range. Results are paginated; use get_filesearch_result_page
with the returned result_id to fetch additional pages.
Args:
dsid: Dataset ID (dNNNNNN), e.g. d083002
valid_date_min: Restrict to data valid on/after "YYYY-MM-DD"
valid_date_max: Restrict to data valid on/before "YYYY-MM-DD"
Parametersdsidrequiredvalid_date_maxvalid_date_min
get_filesearch_result_page
Return a page of results from a previous get_filesearch_*_files call, by
its result_id. Use this to page through file-search results beyond the
first page (see the "pagination" block of a files/results response for
num_pages and next_page).
Args:
dsid: Dataset ID (dNNNNNN), e.g. d083002
result_id: The result_id from a previous filesearch files/results response
page_num: Page number to retrieve
Parametersdsidrequiredpage_numrequiredresult_idrequired
Return data access options for a dataset — download links, Globus URLs, access methods.
Args:
dsid: Dataset ID (dNNNNNN), e.g. d083002
Parametersdsidrequired
Check whether Analysis-Ready Cloud-Optimized (ARCO) data is available for a dataset.
For analysis tasks, call this (and check for a "Kerchunk Reference Files"
group via get_file_groups) before reaching for raw data files — reading
through ARCO/kerchunk references avoids downloading whole files just to
subset them. When picking among kerchunk reference variants, always use
the one with "-osdf" in its name (see get_arco_variables) — other
variants' chunk targets can be internal paths that only resolve on
NCAR's network. Open it with xr.open_dataset(url, engine="kerchunk",
storage_options={"remote_protocol": "https", "lazy": True}) rather than
hand-building an fsspec reference filesystem.
Args:
dsid: Dataset ID (dNNNNNN), e.g. d083002
Parametersdsidrequired
Return the list of ARCO variables available for a dataset.
Each row's type (index 2) is either "zarr" or "reference" (kerchunk).
A "reference" variable is often listed multiple times, once per kerchunk
reference variant (plain, "-remote-https", "-remote-osdf", etc.) — always
pick the "-osdf" variant's URL; the others' chunk targets can be internal
paths that only resolve on NCAR's network. Open it with
xr.open_dataset(url, engine="kerchunk", storage_options={"remote_protocol":
"https", "lazy": True}) — that's better behaved than hand-building an
fsspec reference filesystem. A "zarr" row needs no variant picking — its
URL already routes through the OSDF director (rewritten automatically);
open it directly with xr.open_dataset(url, engine="zarr").
Results are capped at 300 rows (look for "truncated": true); narrow with
search_arco_variables if you hit the cap.
Args:
dsid: Dataset ID (dNNNNNN), e.g. d083002
Parametersdsidrequired
Search ARCO variables by name for a dataset.
As with get_arco_variables, a "reference" (kerchunk) match is often
listed once per variant — always pick the "-osdf" variant's URL (reachable
from anywhere), and open it with xr.open_dataset(url, engine="kerchunk",
storage_options={"remote_protocol": "https", "lazy": True}). A "zarr"
match needs no variant picking — its URL already routes through the OSDF
director (rewritten automatically); open it directly with
xr.open_dataset(url, engine="zarr").
Args:
dsid: Dataset ID (dNNNNNN), e.g. d083002
query: Search text to match against variable names
Parametersdsidrequiredqueryrequired
Return a GDEX portal-wide metric.
Args:
metric: One of: volume_downloaded, unique_users, total_datasets,
total_citations, gdex_volume, total_requests, top_datasets, ai_datasets
Parametersmetric
Return a per-dataset metric.
Args:
dsid: Dataset ID (dNNNNNN), e.g. d083002
metric: One of: users_month, users_year, volume_month, volume_year
Parametersdsidrequiredmetric
Return GDEX staff contacts, optionally filtered to a specific dataset.
Args:
dsid: Optional dataset ID. If omitted, returns all staff.
Parametersdsid
List all subsetting request statuses for the authenticated user. Requires GDEX_TOKEN.
Takes no parameters.
Check the status of a specific subsetting request. Requires GDEX_TOKEN.
Args:
rindex: Request index/ID
Parametersrindexrequired
Return the output files for a completed subsetting request. Requires GDEX_TOKEN.
Args:
rindex: Request index/ID
Parametersrindexrequired
get_control_file_template
Return the control file template for building a subsetting request for a dataset.
Args:
dsid: Dataset ID (dNNNNNN), e.g. d083002
Parametersdsidrequired
Check a subset request body against the dataset's control file template,
without submitting anything. Use this before submit_subset_request to catch
missing or unrecognized fields fast, instead of finding out from a failed
API call.
Args:
dsid: Dataset ID (dNNNNNN), e.g. d083002
request_json: JSON string of the subsetting request body to validate
Parametersdsidrequiredrequest_jsonrequired
Submit a data subset request to GDEX. Requires GDEX_TOKEN.
Args:
request_json: JSON string of the subsetting request body. Use get_control_file_template
to get the expected structure for a dataset, or validate_subset_request
to check it before submitting.
Parametersrequest_jsonrequired
submit_and_wait_for_request
Submit a subset request and poll its status until it finishes, fails, or
timeout_s elapses — instead of calling submit_subset_request and then
manually looping on check_request_status. Requires GDEX_TOKEN.
GDEX's API schema doesn't document the exact status vocabulary, so
"finished" is a best-effort match on the status text (words like
"complete" vs. "error"/"fail"). If the outcome comes back "timeout", that
means the request is still pending by our reading, not that it failed —
keep polling with check_request_status(rindex), or re-run this tool with
a longer timeout_s. If it comes back "unknown", the status payload didn't
contain a field we recognize; inspect the raw "status" value yourself.
Args:
request_json: JSON string of the subsetting request body (see get_control_file_template).
poll_interval_s: Seconds between status checks (default 15, minimum 5).
timeout_s: Give up and return the last-seen status after this many seconds (default 600).
Parameterstimeout_srequest_jsonrequiredpoll_interval_s
Delete a subsetting request and its output files. Requires GDEX_TOKEN.
Args:
rindex: Request index/ID to purge
Parametersrindexrequired