oa.util¶
oa utils
Functions
|
Divide an iterable into chunks/batches of a specific size. |
|
Concatenate a list of lists into a single list. |
|
Ensure that an OpenAI client is available, either by using the provided one or creating a new one. |
Make sure the module-level |
|
The OpenAI API key: oa's config store, then the environment, then ask the user. |
|
Return current package name |
|
|
Convert iso date string to utc integer timestamp. |
|
Serialize an iterable as JSONL bytes |
|
Deserialize JSONL bytes into a python iterable (dict or list of dicts) |
|
|
|
|
|
|
|
Returns a function that takes two inputs (i: int, obj: Any) and saves the serializer(obj) bytes in a file named index_to_filename(i) in the rootdir. |
|
|
|
Return the number of tokens in a string, under given model. |
|
Return a SimpleNamespace of extractors for the named paths |
|
Return the pricing info for the OpenAI API. |
|
Saves obj in a temp file, using serializer to serialize it, and returns its path. |
|
A decorator that will change the annotation and default of the parameters of the decorated function, sourcing them from |
|
|
|
|
|
Convert utc integer timestamp to more human readable iso format. |
Classes
|
A class to manage and monitor the processing of a collection of items, allowing customizable processing functions, status handling, and timing control, using keys to track items. |
Exceptions
No OpenAI API key is configured, and there is no one to ask for it. |
- exception oa.util.OpenAIKeyNotFoundError[source]¶
Bases:
ConfigNotFoundNo OpenAI API key is configured, and there is no one to ask for it.
- class oa.util.ProcessingManager(pending_items, processing_function, handle_status_function, wait_time_function, status_check_interval=5.0, max_cycles=None, completed_items=<factory>, cycles=0)[source]¶
Bases:
Generic[KT,VT,Result]A class to manage and monitor the processing of a collection of items, allowing customizable processing functions, status handling, and timing control, using keys to track items.
Use Case:
Ideal for scenarios where you have a set of items identified by keys that require periodic status checks until they reach a completed state.
Useful for managing batch jobs, asynchronous tasks, or any operations where items may not complete processing immediately.
Attributes:
- pending_items (
MutableMapping[KT, VT]): The mapping of keys to pending items. If an iterable is provided, it is converted to a dict using
dict(enumerate(iterable)).
- pending_items (
processing_function (
Callable[[VT], Tuple[Status, Result]]): A function that takes a value (item) and returns a tuple of(status, result).status(Status): Indicates the current state of the item (e.g.,'completed','in_progress','failed').result(Result): Additional data or context about the item’s processing result.
handle_status_function (
Callable[[VT, Status, Result], bool]): A function that decides whether to remove an item frompending_itemsbased on itsstatusandresult.Returns
Trueif the item should be removed (e.g., processing is complete or failed irrecoverably).
wait_time_function (
Callable[[float, Dict], float]): A function that determines how long to wait before the next processing cycle.Takes
cycle_duration(time taken for the last cycle) andlocals()dictionary as inputs.Returns the sleep time in seconds.
status_check_interval (
float): Desired minimum time (in seconds) between status checks. Defaults to5.0.max_cycles (
Optional[int]): Maximum number of processing cycles to perform. IfNone, there is no limit.completed_items (
MutableMapping[KT, Result]): Mapping of keys to results for items that have been processed.cycles (
int): Number of processing cycles that have been performed.
- process_items()[source]¶
Runs the processing loop until all pending items are processed or max_cycles is reached.
In each cycle:
Calls
process_pending_items()to process the current pending items.Increments the cycle count.
Calculates the duration of the cycle and determines how long to sleep before the next cycle using
wait_time_function.Sleeps for the calculated duration if there are still pending items.
The loop continues until:
pending_itemsis empty (all items have been processed), orThe number of cycles reaches
max_cycles, if specified.
- process_pending_items()[source]¶
Processes the pending items once.
This method iterates over each key-value pair in
pending_items, applies theprocessing_functionto determine its status, and then useshandle_status_functionto decide whether to remove the item frompending_items. Items removed are added tocompleted_itemswith their results.
- oa.util.chunk_iterable(iterable, chk_size, *, chunk_type=None)[source]¶
Divide an iterable into chunks/batches of a specific size.
Handles both mappings (e.g. dicts) and non-mappings (lists, tuples, sets…) as you probably expect it to (if you give a dict input, it will chunk on the (key, value) items and return dicts of these). Thought note that you always can control the type of the chunks with the
chunk_typeargument.- Parameters:
- Return type:
Iterator[list[TypeVar(T)] |tuple[TypeVar(T),...] |dict[TypeVar(KT),TypeVar(VT)]]- Returns:
An iterator of dicts if the input is a Mapping, otherwise an iterator of collections (list, tuple, set…).
Examples
>>> list(chunk_iterable([1, 2, 3, 4, 5], 2)) [[1, 2], [3, 4], [5]]>>> list(chunk_iterable((1, 2, 3, 4, 5), 3, chunk_type=tuple)) [(1, 2, 3), (4, 5)]>>> list(chunk_iterable({"a": 1, "b": 2, "c": 3}, 2)) [{'a': 1, 'b': 2}, {'c': 3}]>>> list(chunk_iterable({"x": 1, "y": 2, "z": 3}, 1, chunk_type=dict)) [{'x': 1}, {'y': 2}, {'z': 3}]
- oa.util.concat_lists(lists)[source]¶
Concatenate a list of lists into a single list.
>>> concat_lists([[1, 2], [3, 4], [5, 6]]) [1, 2, 3, 4, 5, 6]
- oa.util.ensure_oa_client(oa_client)[source]¶
Ensure that an OpenAI client is available, either by using the provided one or creating a new one.
- Return type:
OpenAI
- oa.util.ensure_openai_api_key()[source]¶
Make sure the module-level
openaiclient has an API key, resolving it lazily.The key is looked up (config store, environment, then asking the user) the first time something needs it, never at
import oatime, so importingoaworks – and dependents’ test suites can run – when no key is set. Resolution follows oa’s own order (store before environment), as when the key was set at import, and happens once: a key set onopenai.api_keyafterwards is respected. If no key is stored or in the environment, a key already set onopenai.api_keyis kept rather than asking the user. Thread-safe.- Return type:
- oa.util.get_api_key_from_config()[source]¶
The OpenAI API key: oa’s config store, then the environment, then ask the user.
Asking only happens when the user can be asked (see
_can_prompt_user); otherwise an informativeOpenAIKeyNotFoundErroris raised. Thread-safe: when several threads need the key at once, only one asks, and the others reuse the answer.
- oa.util.iso_date_to_utc_int(iso_date)[source]¶
Convert iso date string to utc integer timestamp. Inverse of utc_int_to_iso_date.
- Return type:
>>> iso_date_to_utc_int('2024-08-12T14:01:57+00:00') 1723471317
- oa.util.jsonl_dumps(x, encoding='utf-8')[source]¶
Serialize an iterable as JSONL bytes
- Return type:
>>> jsonl_dumps([{'a': 1}, {'b': 2}]) b'{"a": 1}\n{"b": 2}'
- oa.util.jsonl_loads_iter(src, *, get_lines=<method 'splitlines' of 'bytes' objects>, line_egress=operator.methodcaller('strip'))[source]¶
Deserialize JSONL bytes into a python iterable (dict or list of dicts)
>>> list(jsonl_loads(b'\n{"a": 1}\n\n{"b": 2}')) [{'a': 1}, {'b': 2}]
- oa.util.mk_local_files_saves_callback(rootdir=None, *, serializer=<built-in function dumps>, index_to_filename=<built-in method format of str object>, print_dir_path=True)[source]¶
Returns a function that takes two inputs (i: int, obj: Any) and saves the serializer(obj) bytes in a file named index_to_filename(i) in the rootdir. If rootdir, a temp dir is used.
- oa.util.num_tokens(text=None, model='gpt-3.5-turbo')[source]¶
Return the number of tokens in a string, under given model.
keywords: token count, number of tokens
- Return type:
- oa.util.oa_extractors_obj(**named_paths)[source]¶
Return a SimpleNamespace of extractors for the named paths
- oa.util.pricing_info(category=None, *, print_data_date=False)[source]¶
Return the pricing info for the OpenAI API.
Note
These are not live prices. Live prices can be found here:
The information pricing_info returns is taken from the file
openai_api_pricing_info.jsonin thedatadirectory of the package. To print a message with the data date, dopricing_info(print_data_date=True).
- oa.util.save_in_temp_dir(obj, serializer=<built-in function dumps>)[source]¶
Saves obj in a temp file, using serializer to serialize it, and returns its path.