API

cognite.cdffs.spec.CdfFileSystem(*args, **kwargs)	File system interface to work with the CDF files.
cognite.cdffs.spec.CdfFile(fs, ...[, mode, ...])	CDF File interface to work with a specific file.

class cognite.cdffs.spec.CdfFileSystem(*args, **kwargs)

File system interface to work with the CDF files.

Provides an interface to the files that are stored in CDF. Only a handful of methods that are required to support the popular python packages are implemented.

protocol

(class attribute) Protocol name to use when interacting with CDF Files.

Type:

str

cdf_list_cache

Cache the file list results from CDF.

Type:

Dict

cdf_list_expiry_time

Expiry time in seconds to invalidate the file list cache.

Type:

int

file_metadata

File Metadata that a user can use when reading/writing the files to CDF.

Type:

FileMetadata

file_cache

Cache the contents of the files.

Type:

Dict

file_handler

File handler to manage the requests to a cloud storage.

Type:

FileHandler

max_download_retries

Maximum number of download retries allowed before exhausting.

Type:

int

download_retries

Flag to indicate enable/disable download retries.

Type:

bool

do_connect(connection_config: ClientConfig, **kwargs: Any | None) → None

Connect to CDF using the connection configurations provided by the user.

Parameters:

connection_config (ClientConfig) – Cognite client connection configurations.

Raises:

ValueError – When an invalid connection_config is provided.

split_path(path: str, validate_suffix: bool = True) → Tuple[str, str, str]

Split the path and extract root_dir, external_id and a filename.

Parameters:

• path (str) – Path to split.

• validate_suffix (bool) – Flag to validate if the file name must have a valid suffix.

Returns:

Returns a tuple with root_dir, external_id_prefix and external_id.

Return type:

Tuple

Raises:

ValueError – When an invalid input path is given.

cache_path(root_dir: str, external_id: str, file_size: int) → None

Cache the file details in dircache to allow subsequent calls to make use of the file details.

Parameters:

• root_dir (str) – Root directory for the file.

• external_id (str) – External Id for the file.

• file_size (int) – File size (in bytes).

ls(path: str, detail: bool = False, **kwargs: Any | None) → Any | List[str]

List the files based on the directory & external Id prefixes extracted from path.

Parameters:

• path (str) – Path to use to extract the list of files from Cdf.

• detail (bool) – Flag to specify if detail list is expected.

• **kwargs (Optional[Any]) – Set of keyword arguments to support additional filtration’s

• files. (when listing the) –

Returns:

Returns the list of files/directories that match the path given.

Return type:

List

Raises:

FileNotFoundError – When there are no files matching the path given.

makedirs(path: str, exist_ok: bool = True) → None

Create a directory at a path given.

Parameters:

• path (str) – Path to use to create a directory.

• exist_ok (bool) – Flag to specify if error can be ignored when directory already exists.

Raises:

FileExistsError – When the directory prefixes already exists.

mkdir(path: str, create_parents: bool = True, **kwargs: Any | None) → None

Create a directory at a path given.

Parameters:

• path (str) – Path to use to create a directory.

• create_parents (bool) – Flag to specify if parents needs to be created.

• **kwargs (Optional[Any]) – Set of keyword arguments to support additional options

• directory. (to create a) –

makedir(path: str, create_parents: bool = False, **kwargs: Any | None) → None

Create a directory at a path given.

Parameters:

• path (str) – Path to use to create a directory.

• create_parents (bool) – Flag to specify if parents needs to be created.

• **kwargs (Optional[Any]) – Set of keyword arguments to support additional options

• directory. (to create a) –

rm_file(path: str) → None

Remove the file at a given path.

Parameters:

path (str) – Path to use to remove the file.

Raises:

FileNotFoundError – When a file is not found.

rm_files(paths: List) → None

Remove the list of files.

Parameters:

paths (List) – List of files to remove.

Raises:

FileNotFoundError – When a file is not found.

exists(path: str) → bool

Check if the file exists at the given path.

Parameters:

path (str) – Absolute path to check.

Returns:

True/False to indicate if the file exists.

Return type:

bool

mv(source_path: str, destination_path: str, recursive: bool = False, maxdepth: int | None = None, **kwargs: Any | None) → None

Move the files and directories at a path given to a new path.

Parameters:

• source_path (str) – Path to use as source to move the files and directories.

• destination_path (str) – Path to use as destination.

• recursive (bool) – Flag to recursively move the files and directories.

• maxdepth (int) – Maximum depth to use when moving the files and directories.

• **kwargs (Optional[Any]) – Set of keyword arguments to support additional options

• directory. (to move) –

Raises:

NotImplementedError – Error as it is not supported.

cd(path: str, **kwargs: Any | None) → None

Change the directory to a path given.

Parameters:

• path (str) – Path to use to change directory.

• **kwargs (Optional[Any]) – Set of keyword arguments to perform change directory.

Raises:

NotImplementedError – Error as it is not supported.

open(path: str, mode: str = 'rb', block_size: int = 5242880, cache_options: Dict[Any, Any] | None = None, **kwargs: Any | None) → CdfFile

Open the file for reading and writing.

Parameters:

• path (str) – File name with absolute path to open.

• mode (str) – Mode to use when opening a file.

• block_size (str) – Block size to use when opening a file.

• cache_options – (Optional[Dict[Any, Any]]): Additional user defined options to work with caching.

• **kwargs (Optional[Any]) – Set of keyword arguments to allow additional options

• file. (when opening a) –

Returns:

An instance of a ‘CdfFile’ to allow reading/writing file to Cdf.

Return type:

CdfFile

read_file(external_id: str, start_byte: int | None = None, end_byte: int | None = None) → Any

Open and read the contents of a file.

Parameters:

• external_id (str) – External Id of the file to fetch the contents.

• start_byte (int) – Start byte for the file only if a specific portion of the file is needed.

• end_byte (int) – End offset for the file only if a specific portion of the file is needed.

Returns:

File contents as is from a cloud storage.

Return type:

bytes

Raises:

FileNotFoundError – When there is no file matching the external_id given.

cat(path: str | list, recursive: bool = False, on_error: str = 'raise', **kwargs: Any | None) → bytes | Any | Dict[str, bytes]

Open and read the contents of a file(s).

Parameters:

• path (str) – Path to use to extract the read the contents of file(s) from Cdf.

• recursive (bool) – Flag to recursively read multiple files.

• on_error (str) – Flag to indicate how to handle file read exceptions.

• **kwargs (Optional[Any]) – Set of keyword arguments to read the file contents.

Returns:

File contents for the file name given. Dict[str, bytes]: File contents for the list of files given - if list of files were given as an argument it will be returned a dictionary with key as path name and contents of the files as value for each path.

Return type:

bytes

Raises:

ValueError – When the path is empty.

class cognite.cdffs.spec.CdfFile(fs: CdfFileSystem, cognite_client: CogniteClient, path: str, directory: str, external_id: str, mode: str = 'rb', block_size: int = 5242880, cache_options: Dict[Any, Any] | None = None, **kwargs: Any | None)

CDF File interface to work with a specific file.

Provides an interface to a file to either read or write the data based on the mode defined.

cognite_client

Cognite client to work with Cdf.

Type:

ClientClient

root_dir

Root directory for the file.

Type:

str

external_id

External Id for the file.

Type:

str

all_bytes_caching

Flag to indicate if the cache type is all bytes caching.

Type:

bool

file_metadata

File Metadata that a user can use when opening a file to write.

Type:

FileMetadata

cognite.cdffs.file_handler.FileHandler()

Handle the requests to a cloud storage.

class cognite.cdffs.file_handler.FileHandler

Handle the requests to a cloud storage.

Cache the download URLs, download the file contents from a cloud storage using custom headers.

_url_container

Cache the file download URLs extracted from CDF.

Type:

Dict

session

Session object to interact with a cloud storage.

Type:

requests.Session

download_file(download_url: str, start_byte: int | None = None, end_byte: int | None = None) → bytes

Download the file from a cloud storage using the download URL & offsets provided.

Parameters:

• download_url (str) – download URL for the file.

• start_byte (int) – Start byte for the file only if a specific portion of the file is needed.

• end_byte (int) – End offset for the file only if a specific portion of the file is needed.

Returns:

Returns a file contents as is.

Return type:

bytes

Raises:

FileException – If unexpected status received or any connection errors.

add_or_update_url(external_id: str, download_url: str) → None

Add or update the download url to the cache.

Parameters:

• external_id (str) – External Id for the file.

• download_url (str) – Download URL for the file.

get_url(external_id: str) → Any

Get download url from a cache if they are valid.

Parameters:

external_id (str) – External Id for the file.

Returns:

Download URL for the file from cache if it is still valid.

Return type:

str

cognite.cdffs.credentials.FsCredentials([...])	Credentials.
cognite.cdffs.credentials.FsOAuthCredentials([...])	OAuth Credentials for cdffs.
cognite.cdffs.credentials.FsToken([...])	Token Credential.
cognite.cdffs.credentials.get_connection_config(...)	Construct Cognite Client from environment variables.

class cognite.cdffs.credentials.FsCredentials(_env_file: str | PathLike | List[str | PathLike] | Tuple[str | PathLike, ...] | None = '<object object>', _env_file_encoding: str | None = None, _env_nested_delimiter: str | None = None, _secrets_dir: str | PathLike | None = None, *, cognite_project: str | None = None, cdf_cluster: str | None = None)

Credentials.

cognite_project

Cdf Project name.

Type:

str

cdf_cluster

Cdf Cluster URL.

Type:

str

abstract get_credentials() → Any

Get credentials to create a client config.

get_client_config() → ClientConfig

Get Cognite client config.

Returns:

Cognite Client Config.

Return type:

ClientConfig

class cognite.cdffs.credentials.FsOAuthCredentials(_env_file: str | PathLike | List[str | PathLike] | Tuple[str | PathLike, ...] | None = '<object object>', _env_file_encoding: str | None = None, _env_nested_delimiter: str | None = None, _secrets_dir: str | PathLike | None = None, *, cognite_project: str | None = None, cdf_cluster: str | None = None, token_url: str | None = None, client_id: SecretStr | None = None, client_secret: SecretStr | None = None, scopes: str | List | None = None)

OAuth Credentials for cdffs.

token_url

Token url to fetch access tokens.

Type:

str

client_id

Client Id.

Type:

str

client_secret

Client Secret.

Type:

SecretStr

scopes

List of scopes as comma(,) separated string.

Type:

str

get_credentials() → OAuthClientCredentials

Construct credentials based on environment variables.

Returns:

Credentials to create a Cognite Client Config.

Return type:

OAuthClientCredentials

class cognite.cdffs.credentials.FsToken(_env_file: str | PathLike | List[str | PathLike] | Tuple[str | PathLike, ...] | None = '<object object>', _env_file_encoding: str | None = None, _env_nested_delimiter: str | None = None, _secrets_dir: str | PathLike | None = None, *, cognite_project: str | None = None, cdf_cluster: str | None = None, token: SecretStr | None = None)

Token Credential.

token

Token.

Type:

SecretStr

get_credentials() → Token

Construct token credential based on environment variable.

Returns:

Token credential to create a Cognite Client Config.

Return type:

Token