Pinder loader#
The goal of this tutorial is to provide some hands-on examples of how one can leverage the pinder dataset in their ML workflow. Specifically, we will illustrate how you can use various utilities provided by pinder to write your own data pipeline.
While this tutorial will not go into details about how to write your own model, it will cover the basic groundwork necessary to interface with structures in pinder and the associated splits and metadata. You will of course want to implement your own featurization pipelines, data representations, etc. but the hope is that this tutorial clarifies how to access the data and make use of it in an ML framework.
Before proceeding with this tutorial section, you may find it helpful to review the existing tutorials available in pinder.
Specifcially, the tutorials covering:
Accessing and loading data for training#
In order to access the train and val splits for PINDER, please refer to the pinder documentation
Once you have downloaded the pinder dataset, either via the pinder package or directly through gsutil, you will have all of the necessary files for training.
To get a list of those systems and their split labels, refer to the pinder index.
We will start by looking at the most basic way to load items from the training and validation set: via PinderSystem objects
Recap: PinderSystem and Structure classes#
import torch
from pinder.core import get_index, PinderSystem
def get_system(system_id: str) -> PinderSystem:
return PinderSystem(system_id)
index = get_index()
train = index[index.split == "train"].copy()
system = get_system(train.id.iloc[0])
system
2024-09-13 20:38:56,468 | pinder.core.utils.cloud:375 | INFO : Gsutil process_many=download_to_filename, threads=4, items=5
2024-09-13 20:38:56,562 | pinder.core.utils.cloud.process_many:23 | INFO : runtime succeeded: 0.09s
PinderSystem(
entry = IndexEntry(
(
'split',
'train',
),
(
'id',
'8phr__X4_UNDEFINED--8phr__W4_UNDEFINED',
),
(
'pdb_id',
'8phr',
),
(
'cluster_id',
'cluster_24559_24559',
),
(
'cluster_id_R',
'cluster_24559',
),
(
'cluster_id_L',
'cluster_24559',
),
(
'pinder_s',
False,
),
(
'pinder_xl',
False,
),
(
'pinder_af2',
False,
),
(
'uniprot_R',
'UNDEFINED',
),
(
'uniprot_L',
'UNDEFINED',
),
(
'holo_R_pdb',
'8phr__X4_UNDEFINED-R.pdb',
),
(
'holo_L_pdb',
'8phr__W4_UNDEFINED-L.pdb',
),
(
'predicted_R_pdb',
'',
),
(
'predicted_L_pdb',
'',
),
(
'apo_R_pdb',
'',
),
(
'apo_L_pdb',
'',
),
(
'apo_R_pdbs',
'',
),
(
'apo_L_pdbs',
'',
),
(
'holo_R',
True,
),
(
'holo_L',
True,
),
(
'predicted_R',
False,
),
(
'predicted_L',
False,
),
(
'apo_R',
False,
),
(
'apo_L',
False,
),
(
'apo_R_quality',
'',
),
(
'apo_L_quality',
'',
),
(
'chain1_neff',
10.78125,
),
(
'chain2_neff',
11.1171875,
),
(
'chain_R',
'X4',
),
(
'chain_L',
'W4',
),
(
'contains_antibody',
False,
),
(
'contains_antigen',
False,
),
(
'contains_enzyme',
False,
),
)
native=Structure(
filepath=/home/runner/.local/share/pinder/2024-02/pdbs/8phr__X4_UNDEFINED--8phr__W4_UNDEFINED.pdb,
uniprot_map=None,
pinder_id='8phr__X4_UNDEFINED--8phr__W4_UNDEFINED',
atom_array=<class 'biotite.structure.AtomArray'> with shape (2556,),
pdb_engine='fastpdb',
)
holo_receptor=Structure(
filepath=/home/runner/.local/share/pinder/2024-02/pdbs/8phr__X4_UNDEFINED-R.pdb,
uniprot_map=/home/runner/.local/share/pinder/2024-02/mappings/8phr__X4_UNDEFINED-R.parquet,
pinder_id='8phr__X4_UNDEFINED-R',
atom_array=<class 'biotite.structure.AtomArray'> with shape (1358,),
pdb_engine='fastpdb',
)
holo_ligand=Structure(
filepath=/home/runner/.local/share/pinder/2024-02/pdbs/8phr__W4_UNDEFINED-L.pdb,
uniprot_map=/home/runner/.local/share/pinder/2024-02/mappings/8phr__W4_UNDEFINED-L.parquet,
pinder_id='8phr__W4_UNDEFINED-L',
atom_array=<class 'biotite.structure.AtomArray'> with shape (1198,),
pdb_engine='fastpdb',
)
apo_receptor=None
apo_ligand=None
pred_receptor=None
pred_ligand=None
)
Notice the printed PinderSystem object has the following properties:
native- the ground-truth dimer complexholo_receptor- the receptor chain (monomer) from the ground-truth complexholo_ligand- the ligand chain (monomer) from the ground-truth complexapo_receptor- the canonical apo chain (monomer) paired to the receptor chainapo_ligand- the canonical apo chain (monomer) paired to the ligand chainpred_receptor- the AlphaFold2 predicted monomer paired to the receptor chainpred_ligand- the AlphaFold2 predicted monomer paired to the ligand chain
These properties are pointers to Structure objects. The Structure object provides the most direct mode of access to structures and associated properties.
Note: not all systems have an apo and/or predicted structure for all chains of the ground-truth dimer complex!
As was the case in the example above, when the alternative monomers are not available, the property will have a value of None.
You can determine which systems have which alternative monomer pairings a priori by looking at the boolean columns in the index apo_R and apo_L for the apo receptor and ligand, and predicted_R and predicted_L for the predicted receptor and ligand, respectively.
For instance, we can load a different system that does have apo receptor and ligand as such:
apo_system = get_system(train.query('apo_R and apo_L').id.iloc[0])
receptor = apo_system.apo_receptor
ligand = apo_system.apo_ligand
receptor, ligand
2024-09-13 20:38:56,651 | pinder.core.utils.cloud:375 | INFO : Gsutil process_many=download_to_filename, threads=4, items=15
2024-09-13 20:38:57,171 | pinder.core.utils.cloud.process_many:23 | INFO : runtime succeeded: 0.52s
(Structure(
filepath=/home/runner/.local/share/pinder/2024-02/pdbs/3wdb__A1_P9WPC9.pdb,
uniprot_map=/home/runner/.local/share/pinder/2024-02/mappings/3wdb__A1_P9WPC9.parquet,
pinder_id='3wdb__A1_P9WPC9',
atom_array=<class 'biotite.structure.AtomArray'> with shape (1144,),
pdb_engine='fastpdb',
),
Structure(
filepath=/home/runner/.local/share/pinder/2024-02/pdbs/6ucr__A1_P9WPC9.pdb,
uniprot_map=/home/runner/.local/share/pinder/2024-02/mappings/6ucr__A1_P9WPC9.parquet,
pinder_id='6ucr__A1_P9WPC9',
atom_array=<class 'biotite.structure.AtomArray'> with shape (1193,),
pdb_engine='fastpdb',
))
We can now access e.g. the sequence and the coordinates of the structures via the Structure objects:
receptor.sequence
'PLGSMFERFTDRARRVVVLAQEEARMLNHNYIGTEHILLGLIHEGEGVAAKSLESLGISLEGVRSQVEEIIGQGQQAPSGHIPFTPRAKKVLELSLREALQLGHNYIGTEHILLGLIREGEGVAAQVLVKLGAELTRVRQQVIQLLSGY'
receptor.coords[0:5]
array([[-12.982, -17.271, -11.271],
[-14.36 , -17.069, -11.749],
[-15.261, -16.373, -10.703],
[-15.461, -15.161, -10.801],
[-14.842, -18.494, -12.077]], dtype=float32)
We can always access the underyling biotite AtomArray via the Structure.atom_array property:
receptor.atom_array[0:5]
array([
Atom(np.array([-12.982, -17.271, -11.271], dtype=float32), chain_id="R", res_id=2, ins_code="", res_name="PRO", hetero=False, atom_name="N", element="N", b_factor=0.0),
Atom(np.array([-14.36 , -17.069, -11.749], dtype=float32), chain_id="R", res_id=2, ins_code="", res_name="PRO", hetero=False, atom_name="CA", element="C", b_factor=0.0),
Atom(np.array([-15.261, -16.373, -10.703], dtype=float32), chain_id="R", res_id=2, ins_code="", res_name="PRO", hetero=False, atom_name="C", element="C", b_factor=0.0),
Atom(np.array([-15.461, -15.161, -10.801], dtype=float32), chain_id="R", res_id=2, ins_code="", res_name="PRO", hetero=False, atom_name="O", element="O", b_factor=0.0),
Atom(np.array([-14.842, -18.494, -12.077], dtype=float32), chain_id="R", res_id=2, ins_code="", res_name="PRO", hetero=False, atom_name="CB", element="C", b_factor=0.0)
])
For a more comprehensive overview of all of the Structure class properties, refer to the pinder system tutorial.
Using the PinderLoader to load, filter and transform systems#
While the PinderSystem object provides a self-contained access to structures associated with a dimer system, the PinderLoader provides a base abstraction for how to iterate over systems, apply optional filters and/or transforms, and return the systems as an iterator. This construct is covered in a different tutorial tutorial.
Using the PinderLoader is not necessary to load systems in your own framework. It is simply one of the provided mechanisms if you find it useful.
Pinder loader brings together filters, transforms and writers to create a generic PinderSystem iterator. It takes either a split name or a list of system IDs as input and can be used to sample alternative monomers to form dimer complexes to serve as e.g. features.
Loading a specific split#
Note: only the test dataset has a subset defined (pinder_s, pinder_xl, pinder_af2)
For train and val, you could just do:
train_loader = PinderLoader(split="train")
val_loader = PinderLoader(split="val")
import torch
from pinder.core import PinderLoader
from pinder.core.loader import filters
base_filters = [
filters.FilterByMissingHolo(),
filters.FilterSubByContacts(min_contacts=5, radius=10.0, calpha_only=True),
filters.FilterDetachedHolo(radius=12, max_components=2),
]
sub_filters = [
filters.FilterSubByAtomTypes(min_atom_types=4),
filters.FilterByHoloOverlap(min_overlap=5),
filters.FilterByHoloSeqIdentity(min_sequence_identity=0.8),
filters.FilterSubRmsds(rmsd_cutoff=7.5),
filters.FilterDetachedSub(radius=12, max_components=2),
]
loader = PinderLoader(
split="test",
subset="pinder_af2",
monomer_priority="holo",
base_filters = base_filters,
sub_filters = sub_filters
)
loader
PinderLoader(split=test, monomers=holo, systems=180)
len(loader)
180
data = loader[0]
print(f"Data is a {type(data)}")
system, feature_complex, target_complex = data
type(system), type(feature_complex), type(target_complex)
2024-09-13 20:38:59,347 | pinder.core.utils.cloud:375 | INFO : Gsutil process_many=download_to_filename, threads=4, items=7
2024-09-13 20:38:59,465 | pinder.core.utils.cloud.process_many:23 | INFO : runtime succeeded: 0.12s
Data is a <class 'tuple'>
(pinder.core.index.system.PinderSystem,
pinder.core.loader.structure.Structure,
pinder.core.loader.structure.Structure)
# You can also use it as an iterator
from tqdm import tqdm
loaded_ids = []
for (system, feature_complex, target_complex) in tqdm(loader):
loaded_ids.append(system.entry.id)
0%| | 0/180 [00:00<?, ?it/s]
1%| | 1/180 [00:00<00:54, 3.29it/s]
2024-09-13 20:39:00,077 | pinder.core.utils.cloud:375 | INFO : Gsutil process_many=download_to_filename, threads=4, items=5
2024-09-13 20:39:00,230 | pinder.core.utils.cloud.process_many:23 | INFO : runtime succeeded: 0.15s
1%| | 2/180 [00:00<01:08, 2.62it/s]
2024-09-13 20:39:00,513 | pinder.core.utils.cloud:375 | INFO : Gsutil process_many=download_to_filename, threads=4, items=5
2024-09-13 20:39:00,681 | pinder.core.utils.cloud.process_many:23 | INFO : runtime succeeded: 0.17s
2%|▏ | 3/180 [00:01<01:13, 2.41it/s]
2024-09-13 20:39:00,964 | pinder.core.utils.cloud:375 | INFO : Gsutil process_many=download_to_filename, threads=4, items=7
2024-09-13 20:39:01,216 | pinder.core.utils.cloud.process_many:23 | INFO : runtime succeeded: 0.25s
2%|▏ | 4/180 [00:01<01:31, 1.93it/s]
2024-09-13 20:39:01,645 | pinder.core.utils.cloud:375 | INFO : Gsutil process_many=download_to_filename, threads=4, items=7
2024-09-13 20:39:01,903 | pinder.core.utils.cloud.process_many:23 | INFO : runtime succeeded: 0.26s
3%|▎ | 5/180 [00:02<01:35, 1.83it/s]
2024-09-13 20:39:02,237 | pinder.core.utils.cloud:375 | INFO : Gsutil process_many=download_to_filename, threads=4, items=12
2024-09-13 20:39:02,635 | pinder.core.utils.cloud.process_many:23 | INFO : runtime succeeded: 0.40s
3%|▎ | 6/180 [00:03<01:50, 1.57it/s]
2024-09-13 20:39:03,049 | pinder.core.utils.cloud:375 | INFO : Gsutil process_many=download_to_filename, threads=4, items=7
2024-09-13 20:39:03,248 | pinder.core.utils.cloud.process_many:23 | INFO : runtime succeeded: 0.20s
4%|▍ | 7/180 [00:03<01:38, 1.76it/s]
2024-09-13 20:39:03,479 | pinder.core.utils.cloud:375 | INFO : Gsutil process_many=download_to_filename, threads=4, items=11
2024-09-13 20:39:03,928 | pinder.core.utils.cloud.process_many:23 | INFO : runtime succeeded: 0.45s
4%|▍ | 8/180 [00:04<01:53, 1.52it/s]
5%|▌ | 9/180 [00:04<01:35, 1.79it/s]
6%|▌ | 10/180 [00:05<01:37, 1.74it/s]
2024-09-13 20:39:05,284 | pinder.core.utils.cloud:375 | INFO : Gsutil process_many=download_to_filename, threads=4, items=7
2024-09-13 20:39:05,667 | pinder.core.utils.cloud.process_many:23 | INFO : runtime succeeded: 0.38s
6%|▌ | 11/180 [00:06<01:57, 1.44it/s]
7%|▋ | 12/180 [00:06<01:35, 1.75it/s]
2024-09-13 20:39:06,533 | pinder.core.utils.cloud:375 | INFO : Gsutil process_many=download_to_filename, threads=4, items=7
2024-09-13 20:39:06,831 | pinder.core.utils.cloud.process_many:23 | INFO : runtime succeeded: 0.30s
7%|▋ | 13/180 [00:07<01:38, 1.70it/s]
8%|▊ | 14/180 [00:07<01:23, 1.98it/s]
2024-09-13 20:39:07,471 | pinder.core.utils.cloud:375 | INFO : Gsutil process_many=download_to_filename, threads=4, items=7
2024-09-13 20:39:07,721 | pinder.core.utils.cloud.process_many:23 | INFO : runtime succeeded: 0.25s
8%|▊ | 15/180 [00:08<01:25, 1.92it/s]
2024-09-13 20:39:08,027 | pinder.core.utils.cloud:375 | INFO : Gsutil process_many=download_to_filename, threads=4, items=5
2024-09-13 20:39:08,238 | pinder.core.utils.cloud.process_many:23 | INFO : runtime succeeded: 0.21s
9%|▉ | 16/180 [00:08<01:36, 1.70it/s]
2024-09-13 20:39:08,772 | pinder.core.utils.cloud:375 | INFO : Gsutil process_many=download_to_filename, threads=4, items=7
2024-09-13 20:39:09,065 | pinder.core.utils.cloud.process_many:23 | INFO : runtime succeeded: 0.29s
9%|▉ | 17/180 [00:09<01:48, 1.50it/s]
2024-09-13 20:39:09,630 | pinder.core.utils.cloud:375 | INFO : Gsutil process_many=download_to_filename, threads=4, items=5
2024-09-13 20:39:09,879 | pinder.core.utils.cloud.process_many:23 | INFO : runtime succeeded: 0.25s
10%|█ | 18/180 [00:10<01:43, 1.56it/s]
2024-09-13 20:39:10,200 | pinder.core.utils.cloud:375 | INFO : Gsutil process_many=download_to_filename, threads=4, items=7
2024-09-13 20:39:10,462 | pinder.core.utils.cloud.process_many:23 | INFO : runtime succeeded: 0.26s
11%|█ | 19/180 [00:11<01:43, 1.56it/s]
2024-09-13 20:39:10,846 | pinder.core.utils.cloud:375 | INFO : Gsutil process_many=download_to_filename, threads=4, items=9
2024-09-13 20:39:11,203 | pinder.core.utils.cloud.process_many:23 | INFO : runtime succeeded: 0.36s
11%|█ | 20/180 [00:11<01:49, 1.46it/s]
2024-09-13 20:39:11,631 | pinder.core.utils.cloud:375 | INFO : Gsutil process_many=download_to_filename, threads=4, items=7
2024-09-13 20:39:11,891 | pinder.core.utils.cloud.process_many:23 | INFO : runtime succeeded: 0.26s
12%|█▏ | 21/180 [00:12<01:51, 1.43it/s]
2024-09-13 20:39:12,366 | pinder.core.utils.cloud:375 | INFO : Gsutil process_many=download_to_filename, threads=4, items=5
2024-09-13 20:39:12,659 | pinder.core.utils.cloud.process_many:23 | INFO : runtime succeeded: 0.29s
12%|█▏ | 22/180 [00:13<02:23, 1.10it/s]
2024-09-13 20:39:13,762 | pinder.core.utils.cloud:375 | INFO : Gsutil process_many=download_to_filename, threads=4, items=7
2024-09-13 20:39:14,094 | pinder.core.utils.cloud.process_many:23 | INFO : runtime succeeded: 0.33s
13%|█▎ | 23/180 [00:14<02:09, 1.21it/s]
2024-09-13 20:39:14,394 | pinder.core.utils.cloud:375 | INFO : Gsutil process_many=download_to_filename, threads=4, items=9
2024-09-13 20:39:14,679 | pinder.core.utils.cloud.process_many:23 | INFO : runtime succeeded: 0.29s
13%|█▎ | 24/180 [00:15<02:00, 1.29it/s]
2024-09-13 20:39:15,051 | pinder.core.utils.cloud:375 | INFO : Gsutil process_many=download_to_filename, threads=4, items=7
2024-09-13 20:39:15,276 | pinder.core.utils.cloud.process_many:23 | INFO : runtime succeeded: 0.23s
14%|█▍ | 25/180 [00:15<01:46, 1.45it/s]
2024-09-13 20:39:15,553 | pinder.core.utils.cloud:375 | INFO : Gsutil process_many=download_to_filename, threads=4, items=7
2024-09-13 20:39:15,807 | pinder.core.utils.cloud.process_many:23 | INFO : runtime succeeded: 0.25s
14%|█▍ | 26/180 [00:16<01:39, 1.55it/s]
2024-09-13 20:39:16,085 | pinder.core.utils.cloud:375 | INFO : Gsutil process_many=download_to_filename, threads=4, items=7
2024-09-13 20:39:16,293 | pinder.core.utils.cloud.process_many:23 | INFO : runtime succeeded: 0.21s
15%|█▌ | 27/180 [00:16<01:30, 1.68it/s]
2024-09-13 20:39:16,558 | pinder.core.utils.cloud:375 | INFO : Gsutil process_many=download_to_filename, threads=4, items=7
2024-09-13 20:39:16,801 | pinder.core.utils.cloud.process_many:23 | INFO : runtime succeeded: 0.24s
16%|█▌ | 28/180 [00:17<01:37, 1.56it/s]
2024-09-13 20:39:17,304 | pinder.core.utils.cloud:375 | INFO : Gsutil process_many=download_to_filename, threads=4, items=7
2024-09-13 20:39:17,518 | pinder.core.utils.cloud.process_many:23 | INFO : runtime succeeded: 0.21s
16%|█▌ | 29/180 [00:17<01:28, 1.70it/s]
2024-09-13 20:39:17,767 | pinder.core.utils.cloud:375 | INFO : Gsutil process_many=download_to_filename, threads=4, items=5
2024-09-13 20:39:17,926 | pinder.core.utils.cloud.process_many:23 | INFO : runtime succeeded: 0.16s
17%|█▋ | 30/180 [00:18<01:19, 1.90it/s]
2024-09-13 20:39:18,154 | pinder.core.utils.cloud:375 | INFO : Gsutil process_many=download_to_filename, threads=4, items=7
2024-09-13 20:39:18,403 | pinder.core.utils.cloud.process_many:23 | INFO : runtime succeeded: 0.25s
17%|█▋ | 31/180 [00:19<01:32, 1.61it/s]
18%|█▊ | 32/180 [00:19<01:21, 1.82it/s]
2024-09-13 20:39:19,374 | pinder.core.utils.cloud:375 | INFO : Gsutil process_many=download_to_filename, threads=4, items=7
2024-09-13 20:39:19,594 | pinder.core.utils.cloud.process_many:23 | INFO : runtime succeeded: 0.22s
18%|█▊ | 33/180 [00:20<01:17, 1.89it/s]
2024-09-13 20:39:19,863 | pinder.core.utils.cloud:375 | INFO : Gsutil process_many=download_to_filename, threads=4, items=7
2024-09-13 20:39:20,144 | pinder.core.utils.cloud.process_many:23 | INFO : runtime succeeded: 0.28s
19%|█▉ | 34/180 [00:20<01:21, 1.78it/s]
2024-09-13 20:39:20,494 | pinder.core.utils.cloud:375 | INFO : Gsutil process_many=download_to_filename, threads=4, items=5
2024-09-13 20:39:20,649 | pinder.core.utils.cloud.process_many:23 | INFO : runtime succeeded: 0.15s
19%|█▉ | 35/180 [00:21<01:12, 2.00it/s]
2024-09-13 20:39:20,853 | pinder.core.utils.cloud:375 | INFO : Gsutil process_many=download_to_filename, threads=4, items=7
2024-09-13 20:39:21,195 | pinder.core.utils.cloud.process_many:23 | INFO : runtime succeeded: 0.34s
20%|██ | 36/180 [00:21<01:20, 1.78it/s]
2024-09-13 20:39:21,553 | pinder.core.utils.cloud:375 | INFO : Gsutil process_many=download_to_filename, threads=4, items=7
2024-09-13 20:39:21,801 | pinder.core.utils.cloud.process_many:23 | INFO : runtime succeeded: 0.25s
21%|██ | 37/180 [00:22<01:27, 1.63it/s]
2024-09-13 20:39:22,287 | pinder.core.utils.cloud:375 | INFO : Gsutil process_many=download_to_filename, threads=4, items=7
2024-09-13 20:39:22,564 | pinder.core.utils.cloud.process_many:23 | INFO : runtime succeeded: 0.28s
21%|██ | 38/180 [00:24<02:43, 1.15s/it]
22%|██▏ | 39/180 [00:26<03:16, 1.39s/it]
22%|██▏ | 40/180 [00:27<02:36, 1.12s/it]
2024-09-13 20:39:27,123 | pinder.core.utils.cloud:375 | INFO : Gsutil process_many=download_to_filename, threads=4, items=7
2024-09-13 20:39:27,407 | pinder.core.utils.cloud.process_many:23 | INFO : runtime succeeded: 0.28s
23%|██▎ | 41/180 [00:28<02:43, 1.18s/it]
2024-09-13 20:39:28,435 | pinder.core.utils.cloud:375 | INFO : Gsutil process_many=download_to_filename, threads=4, items=7
2024-09-13 20:39:28,664 | pinder.core.utils.cloud.process_many:23 | INFO : runtime succeeded: 0.23s
23%|██▎ | 42/180 [00:29<02:20, 1.02s/it]
2024-09-13 20:39:29,090 | pinder.core.utils.cloud:375 | INFO : Gsutil process_many=download_to_filename, threads=4, items=11
2024-09-13 20:39:29,413 | pinder.core.utils.cloud.process_many:23 | INFO : runtime succeeded: 0.32s
24%|██▍ | 43/180 [00:30<02:08, 1.07it/s]
2024-09-13 20:39:29,836 | pinder.core.utils.cloud:375 | INFO : Gsutil process_many=download_to_filename, threads=4, items=7
2024-09-13 20:39:30,084 | pinder.core.utils.cloud.process_many:23 | INFO : runtime succeeded: 0.25s
24%|██▍ | 44/180 [00:31<02:17, 1.01s/it]
2024-09-13 20:39:31,009 | pinder.core.utils.cloud:375 | INFO : Gsutil process_many=download_to_filename, threads=4, items=7
2024-09-13 20:39:31,253 | pinder.core.utils.cloud.process_many:23 | INFO : runtime succeeded: 0.24s
25%|██▌ | 45/180 [00:32<02:13, 1.01it/s]
26%|██▌ | 46/180 [00:32<01:48, 1.23it/s]
2024-09-13 20:39:32,354 | pinder.core.utils.cloud:375 | INFO : Gsutil process_many=download_to_filename, threads=4, items=7
2024-09-13 20:39:32,591 | pinder.core.utils.cloud.process_many:23 | INFO : runtime succeeded: 0.24s
26%|██▌ | 47/180 [00:33<01:45, 1.26it/s]
2024-09-13 20:39:33,105 | pinder.core.utils.cloud:375 | INFO : Gsutil process_many=download_to_filename, threads=4, items=7
2024-09-13 20:39:33,291 | pinder.core.utils.cloud.process_many:23 | INFO : runtime succeeded: 0.19s
27%|██▋ | 48/180 [00:33<01:30, 1.46it/s]
2024-09-13 20:39:33,530 | pinder.core.utils.cloud:375 | INFO : Gsutil process_many=download_to_filename, threads=4, items=5
2024-09-13 20:39:33,716 | pinder.core.utils.cloud.process_many:23 | INFO : runtime succeeded: 0.19s
27%|██▋ | 49/180 [00:34<01:18, 1.68it/s]
2024-09-13 20:39:33,921 | pinder.core.utils.cloud:375 | INFO : Gsutil process_many=download_to_filename, threads=4, items=5
2024-09-13 20:39:34,102 | pinder.core.utils.cloud.process_many:23 | INFO : runtime succeeded: 0.18s
28%|██▊ | 50/180 [00:34<01:14, 1.75it/s]
28%|██▊ | 51/180 [00:35<01:05, 1.97it/s]
2024-09-13 20:39:34,792 | pinder.core.utils.cloud:375 | INFO : Gsutil process_many=download_to_filename, threads=4, items=7
2024-09-13 20:39:35,045 | pinder.core.utils.cloud.process_many:23 | INFO : runtime succeeded: 0.25s
29%|██▉ | 52/180 [00:35<01:08, 1.86it/s]
29%|██▉ | 53/180 [00:36<01:17, 1.63it/s]
2024-09-13 20:39:36,188 | pinder.core.utils.cloud:375 | INFO : Gsutil process_many=download_to_filename, threads=4, items=5
2024-09-13 20:39:36,354 | pinder.core.utils.cloud.process_many:23 | INFO : runtime succeeded: 0.17s
30%|███ | 54/180 [00:36<01:10, 1.80it/s]
2024-09-13 20:39:36,613 | pinder.core.utils.cloud:375 | INFO : Gsutil process_many=download_to_filename, threads=4, items=5
2024-09-13 20:39:36,763 | pinder.core.utils.cloud.process_many:23 | INFO : runtime succeeded: 0.15s
31%|███ | 55/180 [00:37<01:02, 2.00it/s]
2024-09-13 20:39:36,985 | pinder.core.utils.cloud:375 | INFO : Gsutil process_many=download_to_filename, threads=4, items=7
2024-09-13 20:39:37,218 | pinder.core.utils.cloud.process_many:23 | INFO : runtime succeeded: 0.23s
31%|███ | 56/180 [00:38<01:21, 1.51it/s]
2024-09-13 20:39:38,020 | pinder.core.utils.cloud:375 | INFO : Gsutil process_many=download_to_filename, threads=4, items=7
2024-09-13 20:39:38,516 | pinder.core.utils.cloud.process_many:23 | INFO : runtime succeeded: 0.50s
32%|███▏ | 57/180 [00:39<01:25, 1.43it/s]
2024-09-13 20:39:38,808 | pinder.core.utils.cloud:375 | INFO : Gsutil process_many=download_to_filename, threads=4, items=7
2024-09-13 20:39:39,108 | pinder.core.utils.cloud.process_many:23 | INFO : runtime succeeded: 0.30s
32%|███▏ | 58/180 [00:40<01:45, 1.15it/s]
2024-09-13 20:39:40,072 | pinder.core.utils.cloud:375 | INFO : Gsutil process_many=download_to_filename, threads=4, items=7
2024-09-13 20:39:40,306 | pinder.core.utils.cloud.process_many:23 | INFO : runtime succeeded: 0.23s
33%|███▎ | 59/180 [00:41<01:39, 1.21it/s]
2024-09-13 20:39:40,790 | pinder.core.utils.cloud:375 | INFO : Gsutil process_many=download_to_filename, threads=4, items=5
2024-09-13 20:39:40,948 | pinder.core.utils.cloud.process_many:23 | INFO : runtime succeeded: 0.16s
33%|███▎ | 60/180 [00:41<01:25, 1.41it/s]
2024-09-13 20:39:41,235 | pinder.core.utils.cloud:375 | INFO : Gsutil process_many=download_to_filename, threads=4, items=5
2024-09-13 20:39:41,405 | pinder.core.utils.cloud.process_many:23 | INFO : runtime succeeded: 0.17s
34%|███▍ | 61/180 [00:41<01:11, 1.67it/s]
34%|███▍ | 62/180 [00:42<01:18, 1.51it/s]
2024-09-13 20:39:42,394 | pinder.core.utils.cloud:375 | INFO : Gsutil process_many=download_to_filename, threads=4, items=7
2024-09-13 20:39:42,636 | pinder.core.utils.cloud.process_many:23 | INFO : runtime succeeded: 0.24s
35%|███▌ | 63/180 [00:43<01:15, 1.54it/s]
2024-09-13 20:39:43,007 | pinder.core.utils.cloud:375 | INFO : Gsutil process_many=download_to_filename, threads=4, items=7
2024-09-13 20:39:43,220 | pinder.core.utils.cloud.process_many:23 | INFO : runtime succeeded: 0.21s
36%|███▌ | 64/180 [00:43<01:10, 1.65it/s]
2024-09-13 20:39:43,513 | pinder.core.utils.cloud:375 | INFO : Gsutil process_many=download_to_filename, threads=4, items=7
2024-09-13 20:39:43,885 | pinder.core.utils.cloud.process_many:23 | INFO : runtime succeeded: 0.37s
36%|███▌ | 65/180 [00:46<02:19, 1.21s/it]
2024-09-13 20:39:46,146 | pinder.core.utils.cloud:375 | INFO : Gsutil process_many=download_to_filename, threads=4, items=7
2024-09-13 20:39:46,414 | pinder.core.utils.cloud.process_many:23 | INFO : runtime succeeded: 0.27s
37%|███▋ | 66/180 [00:47<02:16, 1.20s/it]
2024-09-13 20:39:47,319 | pinder.core.utils.cloud:375 | INFO : Gsutil process_many=download_to_filename, threads=4, items=7
2024-09-13 20:39:47,569 | pinder.core.utils.cloud.process_many:23 | INFO : runtime succeeded: 0.25s
37%|███▋ | 67/180 [00:48<01:58, 1.05s/it]
2024-09-13 20:39:48,009 | pinder.core.utils.cloud:375 | INFO : Gsutil process_many=download_to_filename, threads=4, items=5
2024-09-13 20:39:48,167 | pinder.core.utils.cloud.process_many:23 | INFO : runtime succeeded: 0.16s
38%|███▊ | 68/180 [00:48<01:36, 1.16it/s]
2024-09-13 20:39:48,444 | pinder.core.utils.cloud:375 | INFO : Gsutil process_many=download_to_filename, threads=4, items=5
2024-09-13 20:39:48,609 | pinder.core.utils.cloud.process_many:23 | INFO : runtime succeeded: 0.17s
38%|███▊ | 69/180 [00:49<01:19, 1.40it/s]
2024-09-13 20:39:48,815 | pinder.core.utils.cloud:375 | INFO : Gsutil process_many=download_to_filename, threads=4, items=7
2024-09-13 20:39:49,031 | pinder.core.utils.cloud.process_many:23 | INFO : runtime succeeded: 0.22s
39%|███▉ | 70/180 [00:49<01:18, 1.39it/s]
39%|███▉ | 71/180 [00:50<01:14, 1.46it/s]
2024-09-13 20:39:50,148 | pinder.core.utils.cloud:375 | INFO : Gsutil process_many=download_to_filename, threads=4, items=7
2024-09-13 20:39:50,710 | pinder.core.utils.cloud.process_many:23 | INFO : runtime succeeded: 0.56s
40%|████ | 72/180 [00:51<01:24, 1.28it/s]
2024-09-13 20:39:51,158 | pinder.core.utils.cloud:375 | INFO : Gsutil process_many=download_to_filename, threads=4, items=7
2024-09-13 20:39:51,357 | pinder.core.utils.cloud.process_many:23 | INFO : runtime succeeded: 0.20s
41%|████ | 73/180 [00:51<01:14, 1.44it/s]
2024-09-13 20:39:51,644 | pinder.core.utils.cloud:375 | INFO : Gsutil process_many=download_to_filename, threads=4, items=7
2024-09-13 20:39:51,918 | pinder.core.utils.cloud.process_many:23 | INFO : runtime succeeded: 0.27s
41%|████ | 74/180 [00:53<01:44, 1.02it/s]
2024-09-13 20:39:53,302 | pinder.core.utils.cloud:375 | INFO : Gsutil process_many=download_to_filename, threads=4, items=7
2024-09-13 20:39:53,544 | pinder.core.utils.cloud.process_many:23 | INFO : runtime succeeded: 0.24s
42%|████▏ | 75/180 [00:54<01:34, 1.11it/s]
2024-09-13 20:39:54,009 | pinder.core.utils.cloud:375 | INFO : Gsutil process_many=download_to_filename, threads=4, items=5
2024-09-13 20:39:54,150 | pinder.core.utils.cloud.process_many:23 | INFO : runtime succeeded: 0.14s
42%|████▏ | 76/180 [00:54<01:28, 1.18it/s]
2024-09-13 20:39:54,738 | pinder.core.utils.cloud:375 | INFO : Gsutil process_many=download_to_filename, threads=4, items=7
2024-09-13 20:39:54,964 | pinder.core.utils.cloud.process_many:23 | INFO : runtime succeeded: 0.23s
43%|████▎ | 77/180 [00:55<01:17, 1.33it/s]
2024-09-13 20:39:55,260 | pinder.core.utils.cloud:375 | INFO : Gsutil process_many=download_to_filename, threads=4, items=7
2024-09-13 20:39:55,482 | pinder.core.utils.cloud.process_many:23 | INFO : runtime succeeded: 0.22s
43%|████▎ | 78/180 [00:56<01:09, 1.46it/s]
2024-09-13 20:39:55,791 | pinder.core.utils.cloud:375 | INFO : Gsutil process_many=download_to_filename, threads=4, items=7
2024-09-13 20:39:56,176 | pinder.core.utils.cloud.process_many:23 | INFO : runtime succeeded: 0.39s
44%|████▍ | 79/180 [00:57<01:24, 1.20it/s]
2024-09-13 20:39:56,968 | pinder.core.utils.cloud:375 | INFO : Gsutil process_many=download_to_filename, threads=4, items=9
2024-09-13 20:39:57,309 | pinder.core.utils.cloud.process_many:23 | INFO : runtime succeeded: 0.34s
44%|████▍ | 80/180 [00:58<01:40, 1.00s/it]
45%|████▌ | 81/180 [00:59<01:35, 1.04it/s]
2024-09-13 20:39:59,238 | pinder.core.utils.cloud:375 | INFO : Gsutil process_many=download_to_filename, threads=4, items=7
2024-09-13 20:39:59,656 | pinder.core.utils.cloud.process_many:23 | INFO : runtime succeeded: 0.42s
46%|████▌ | 82/180 [01:00<01:32, 1.06it/s]
2024-09-13 20:40:00,141 | pinder.core.utils.cloud:375 | INFO : Gsutil process_many=download_to_filename, threads=4, items=6
2024-09-13 20:40:00,442 | pinder.core.utils.cloud.process_many:23 | INFO : runtime succeeded: 0.30s
46%|████▌ | 83/180 [01:00<01:19, 1.22it/s]
2024-09-13 20:40:00,676 | pinder.core.utils.cloud:375 | INFO : Gsutil process_many=download_to_filename, threads=4, items=7
2024-09-13 20:40:01,013 | pinder.core.utils.cloud.process_many:23 | INFO : runtime succeeded: 0.34s
47%|████▋ | 84/180 [01:01<01:17, 1.24it/s]
2024-09-13 20:40:01,452 | pinder.core.utils.cloud:375 | INFO : Gsutil process_many=download_to_filename, threads=4, items=7
2024-09-13 20:40:01,752 | pinder.core.utils.cloud.process_many:23 | INFO : runtime succeeded: 0.30s
47%|████▋ | 85/180 [01:02<01:14, 1.28it/s]
2024-09-13 20:40:02,173 | pinder.core.utils.cloud:375 | INFO : Gsutil process_many=download_to_filename, threads=4, items=5
2024-09-13 20:40:02,380 | pinder.core.utils.cloud.process_many:23 | INFO : runtime succeeded: 0.21s
48%|████▊ | 86/180 [01:02<01:02, 1.51it/s]
2024-09-13 20:40:02,562 | pinder.core.utils.cloud:375 | INFO : Gsutil process_many=download_to_filename, threads=4, items=5
2024-09-13 20:40:02,838 | pinder.core.utils.cloud.process_many:23 | INFO : runtime succeeded: 0.28s
48%|████▊ | 87/180 [01:03<00:56, 1.66it/s]
49%|████▉ | 88/180 [01:03<00:52, 1.76it/s]
49%|████▉ | 89/180 [01:04<00:45, 2.00it/s]
50%|█████ | 90/180 [01:04<00:42, 2.14it/s]
2024-09-13 20:40:04,241 | pinder.core.utils.cloud:375 | INFO : Gsutil process_many=download_to_filename, threads=4, items=5
2024-09-13 20:40:04,506 | pinder.core.utils.cloud.process_many:23 | INFO : runtime succeeded: 0.26s
51%|█████ | 91/180 [01:04<00:41, 2.16it/s]
51%|█████ | 92/180 [01:05<00:41, 2.11it/s]
2024-09-13 20:40:05,189 | pinder.core.utils.cloud:375 | INFO : Gsutil process_many=download_to_filename, threads=4, items=5
2024-09-13 20:40:05,383 | pinder.core.utils.cloud.process_many:23 | INFO : runtime succeeded: 0.19s
52%|█████▏ | 93/180 [01:05<00:40, 2.16it/s]
2024-09-13 20:40:05,628 | pinder.core.utils.cloud:375 | INFO : Gsutil process_many=download_to_filename, threads=4, items=7
2024-09-13 20:40:05,860 | pinder.core.utils.cloud.process_many:23 | INFO : runtime succeeded: 0.23s
52%|█████▏ | 94/180 [01:06<00:44, 1.94it/s]
2024-09-13 20:40:06,270 | pinder.core.utils.cloud:375 | INFO : Gsutil process_many=download_to_filename, threads=4, items=7
2024-09-13 20:40:06,588 | pinder.core.utils.cloud.process_many:23 | INFO : runtime succeeded: 0.32s
53%|█████▎ | 95/180 [01:07<00:46, 1.82it/s]
2024-09-13 20:40:06,900 | pinder.core.utils.cloud:375 | INFO : Gsutil process_many=download_to_filename, threads=4, items=5
2024-09-13 20:40:07,245 | pinder.core.utils.cloud.process_many:23 | INFO : runtime succeeded: 0.34s
53%|█████▎ | 96/180 [01:07<00:46, 1.81it/s]
2024-09-13 20:40:07,457 | pinder.core.utils.cloud:375 | INFO : Gsutil process_many=download_to_filename, threads=4, items=7
2024-09-13 20:40:07,710 | pinder.core.utils.cloud.process_many:23 | INFO : runtime succeeded: 0.25s
54%|█████▍ | 97/180 [01:08<00:46, 1.79it/s]
2024-09-13 20:40:08,043 | pinder.core.utils.cloud:375 | INFO : Gsutil process_many=download_to_filename, threads=4, items=7
2024-09-13 20:40:08,280 | pinder.core.utils.cloud.process_many:23 | INFO : runtime succeeded: 0.24s
54%|█████▍ | 98/180 [01:08<00:44, 1.85it/s]
2024-09-13 20:40:08,531 | pinder.core.utils.cloud:375 | INFO : Gsutil process_many=download_to_filename, threads=4, items=7
2024-09-13 20:40:08,852 | pinder.core.utils.cloud.process_many:23 | INFO : runtime succeeded: 0.32s
55%|█████▌ | 99/180 [01:09<00:46, 1.75it/s]
56%|█████▌ | 100/180 [01:09<00:39, 2.05it/s]
56%|█████▌ | 101/180 [01:10<00:36, 2.16it/s]
2024-09-13 20:40:09,869 | pinder.core.utils.cloud:375 | INFO : Gsutil process_many=download_to_filename, threads=4, items=5
2024-09-13 20:40:10,325 | pinder.core.utils.cloud.process_many:23 | INFO : runtime succeeded: 0.46s
57%|█████▋ | 102/180 [01:10<00:42, 1.83it/s]
2024-09-13 20:40:10,609 | pinder.core.utils.cloud:375 | INFO : Gsutil process_many=download_to_filename, threads=4, items=7
2024-09-13 20:40:10,860 | pinder.core.utils.cloud.process_many:23 | INFO : runtime succeeded: 0.25s
57%|█████▋ | 103/180 [01:11<00:43, 1.77it/s]
2024-09-13 20:40:11,236 | pinder.core.utils.cloud:375 | INFO : Gsutil process_many=download_to_filename, threads=4, items=5
2024-09-13 20:40:11,452 | pinder.core.utils.cloud.process_many:23 | INFO : runtime succeeded: 0.22s
58%|█████▊ | 104/180 [01:11<00:39, 1.93it/s]
2024-09-13 20:40:11,631 | pinder.core.utils.cloud:375 | INFO : Gsutil process_many=download_to_filename, threads=4, items=7
2024-09-13 20:40:11,847 | pinder.core.utils.cloud.process_many:23 | INFO : runtime succeeded: 0.22s
58%|█████▊ | 105/180 [01:12<00:45, 1.65it/s]
2024-09-13 20:40:12,441 | pinder.core.utils.cloud:375 | INFO : Gsutil process_many=download_to_filename, threads=4, items=7
2024-09-13 20:40:12,750 | pinder.core.utils.cloud.process_many:23 | INFO : runtime succeeded: 0.31s
59%|█████▉ | 106/180 [01:13<00:51, 1.44it/s]
2024-09-13 20:40:13,338 | pinder.core.utils.cloud:375 | INFO : Gsutil process_many=download_to_filename, threads=4, items=6
2024-09-13 20:40:13,571 | pinder.core.utils.cloud.process_many:23 | INFO : runtime succeeded: 0.23s
59%|█████▉ | 107/180 [01:14<00:46, 1.57it/s]
2024-09-13 20:40:13,838 | pinder.core.utils.cloud:375 | INFO : Gsutil process_many=download_to_filename, threads=4, items=7
2024-09-13 20:40:14,082 | pinder.core.utils.cloud.process_many:23 | INFO : runtime succeeded: 0.24s
60%|██████ | 108/180 [01:15<00:57, 1.26it/s]
2024-09-13 20:40:14,994 | pinder.core.utils.cloud:375 | INFO : Gsutil process_many=download_to_filename, threads=4, items=7
2024-09-13 20:40:15,272 | pinder.core.utils.cloud.process_many:23 | INFO : runtime succeeded: 0.28s
61%|██████ | 109/180 [01:15<00:54, 1.31it/s]
2024-09-13 20:40:15,691 | pinder.core.utils.cloud:375 | INFO : Gsutil process_many=download_to_filename, threads=4, items=5
2024-09-13 20:40:15,905 | pinder.core.utils.cloud.process_many:23 | INFO : runtime succeeded: 0.21s
61%|██████ | 110/180 [01:16<00:47, 1.47it/s]
62%|██████▏ | 111/180 [01:16<00:40, 1.72it/s]
2024-09-13 20:40:16,531 | pinder.core.utils.cloud:375 | INFO : Gsutil process_many=download_to_filename, threads=4, items=7
2024-09-13 20:40:16,738 | pinder.core.utils.cloud.process_many:23 | INFO : runtime succeeded: 0.21s
62%|██████▏ | 112/180 [01:17<00:38, 1.75it/s]
2024-09-13 20:40:17,075 | pinder.core.utils.cloud:375 | INFO : Gsutil process_many=download_to_filename, threads=4, items=7
2024-09-13 20:40:17,349 | pinder.core.utils.cloud.process_many:23 | INFO : runtime succeeded: 0.27s
63%|██████▎ | 113/180 [01:18<00:40, 1.64it/s]
2024-09-13 20:40:17,781 | pinder.core.utils.cloud:375 | INFO : Gsutil process_many=download_to_filename, threads=4, items=5
2024-09-13 20:40:18,119 | pinder.core.utils.cloud.process_many:23 | INFO : runtime succeeded: 0.34s
63%|██████▎ | 114/180 [01:18<00:43, 1.51it/s]
2024-09-13 20:40:18,590 | pinder.core.utils.cloud:375 | INFO : Gsutil process_many=download_to_filename, threads=4, items=7
2024-09-13 20:40:18,946 | pinder.core.utils.cloud.process_many:23 | INFO : runtime succeeded: 0.36s
64%|██████▍ | 115/180 [01:19<00:45, 1.44it/s]
64%|██████▍ | 116/180 [01:20<00:39, 1.61it/s]
2024-09-13 20:40:19,789 | pinder.core.utils.cloud:375 | INFO : Gsutil process_many=download_to_filename, threads=4, items=5
2024-09-13 20:40:20,005 | pinder.core.utils.cloud.process_many:23 | INFO : runtime succeeded: 0.22s
65%|██████▌ | 117/180 [01:20<00:35, 1.80it/s]
2024-09-13 20:40:20,189 | pinder.core.utils.cloud:375 | INFO : Gsutil process_many=download_to_filename, threads=4, items=7
2024-09-13 20:40:20,489 | pinder.core.utils.cloud.process_many:23 | INFO : runtime succeeded: 0.30s
66%|██████▌ | 118/180 [01:21<00:35, 1.74it/s]
2024-09-13 20:40:20,804 | pinder.core.utils.cloud:375 | INFO : Gsutil process_many=download_to_filename, threads=4, items=6
2024-09-13 20:40:21,032 | pinder.core.utils.cloud.process_many:23 | INFO : runtime succeeded: 0.23s
66%|██████▌ | 119/180 [01:21<00:34, 1.78it/s]
2024-09-13 20:40:21,337 | pinder.core.utils.cloud:375 | INFO : Gsutil process_many=download_to_filename, threads=4, items=7
2024-09-13 20:40:21,610 | pinder.core.utils.cloud.process_many:23 | INFO : runtime succeeded: 0.27s
67%|██████▋ | 120/180 [01:22<00:35, 1.68it/s]
67%|██████▋ | 121/180 [01:22<00:31, 1.87it/s]
2024-09-13 20:40:22,400 | pinder.core.utils.cloud:375 | INFO : Gsutil process_many=download_to_filename, threads=4, items=5
2024-09-13 20:40:22,707 | pinder.core.utils.cloud.process_many:23 | INFO : runtime succeeded: 0.31s
68%|██████▊ | 122/180 [01:23<00:31, 1.83it/s]
2024-09-13 20:40:22,973 | pinder.core.utils.cloud:375 | INFO : Gsutil process_many=download_to_filename, threads=4, items=5
2024-09-13 20:40:23,153 | pinder.core.utils.cloud.process_many:23 | INFO : runtime succeeded: 0.18s
68%|██████▊ | 123/180 [01:23<00:27, 2.04it/s]
2024-09-13 20:40:23,339 | pinder.core.utils.cloud:375 | INFO : Gsutil process_many=download_to_filename, threads=4, items=7
2024-09-13 20:40:23,596 | pinder.core.utils.cloud.process_many:23 | INFO : runtime succeeded: 0.26s
69%|██████▉ | 124/180 [01:24<00:27, 2.05it/s]
2024-09-13 20:40:23,817 | pinder.core.utils.cloud:375 | INFO : Gsutil process_many=download_to_filename, threads=4, items=7
2024-09-13 20:40:24,140 | pinder.core.utils.cloud.process_many:23 | INFO : runtime succeeded: 0.32s
69%|██████▉ | 125/180 [01:24<00:30, 1.80it/s]
2024-09-13 20:40:24,535 | pinder.core.utils.cloud:375 | INFO : Gsutil process_many=download_to_filename, threads=4, items=7
2024-09-13 20:40:24,874 | pinder.core.utils.cloud.process_many:23 | INFO : runtime succeeded: 0.34s
70%|███████ | 126/180 [01:25<00:37, 1.46it/s]
2024-09-13 20:40:25,522 | pinder.core.utils.cloud:375 | INFO : Gsutil process_many=download_to_filename, threads=4, items=5
2024-09-13 20:40:25,657 | pinder.core.utils.cloud.process_many:23 | INFO : runtime succeeded: 0.13s
71%|███████ | 127/180 [01:26<00:30, 1.74it/s]
2024-09-13 20:40:25,843 | pinder.core.utils.cloud:375 | INFO : Gsutil process_many=download_to_filename, threads=4, items=7
2024-09-13 20:40:26,078 | pinder.core.utils.cloud.process_many:23 | INFO : runtime succeeded: 0.24s
71%|███████ | 128/180 [01:26<00:33, 1.55it/s]
72%|███████▏ | 129/180 [01:27<00:39, 1.29it/s]
72%|███████▏ | 130/180 [01:28<00:32, 1.55it/s]
2024-09-13 20:40:28,073 | pinder.core.utils.cloud:375 | INFO : Gsutil process_many=download_to_filename, threads=4, items=7
2024-09-13 20:40:28,380 | pinder.core.utils.cloud.process_many:23 | INFO : runtime succeeded: 0.31s
73%|███████▎ | 131/180 [01:29<00:32, 1.49it/s]
2024-09-13 20:40:28,793 | pinder.core.utils.cloud:375 | INFO : Gsutil process_many=download_to_filename, threads=4, items=5
2024-09-13 20:40:28,956 | pinder.core.utils.cloud.process_many:23 | INFO : runtime succeeded: 0.16s
73%|███████▎ | 132/180 [01:29<00:34, 1.38it/s]
2024-09-13 20:40:29,651 | pinder.core.utils.cloud:375 | INFO : Gsutil process_many=download_to_filename, threads=4, items=5
2024-09-13 20:40:29,806 | pinder.core.utils.cloud.process_many:23 | INFO : runtime succeeded: 0.15s
74%|███████▍ | 133/180 [01:30<00:29, 1.58it/s]
2024-09-13 20:40:30,071 | pinder.core.utils.cloud:375 | INFO : Gsutil process_many=download_to_filename, threads=4, items=5
2024-09-13 20:40:30,246 | pinder.core.utils.cloud.process_many:23 | INFO : runtime succeeded: 0.17s
74%|███████▍ | 134/180 [01:30<00:26, 1.71it/s]
2024-09-13 20:40:30,541 | pinder.core.utils.cloud:375 | INFO : Gsutil process_many=download_to_filename, threads=4, items=7
2024-09-13 20:40:30,779 | pinder.core.utils.cloud.process_many:23 | INFO : runtime succeeded: 0.24s
75%|███████▌ | 135/180 [01:31<00:27, 1.63it/s]
2024-09-13 20:40:31,228 | pinder.core.utils.cloud:375 | INFO : Gsutil process_many=download_to_filename, threads=4, items=7
2024-09-13 20:40:31,502 | pinder.core.utils.cloud.process_many:23 | INFO : runtime succeeded: 0.27s
76%|███████▌ | 136/180 [01:32<00:31, 1.42it/s]
76%|███████▌ | 137/180 [01:32<00:27, 1.59it/s]
77%|███████▋ | 138/180 [01:33<00:23, 1.81it/s]
2024-09-13 20:40:32,965 | pinder.core.utils.cloud:375 | INFO : Gsutil process_many=download_to_filename, threads=4, items=5
2024-09-13 20:40:33,133 | pinder.core.utils.cloud.process_many:23 | INFO : runtime succeeded: 0.17s
77%|███████▋ | 139/180 [01:33<00:21, 1.93it/s]
2024-09-13 20:40:33,406 | pinder.core.utils.cloud:375 | INFO : Gsutil process_many=download_to_filename, threads=4, items=5
2024-09-13 20:40:33,587 | pinder.core.utils.cloud.process_many:23 | INFO : runtime succeeded: 0.18s
78%|███████▊ | 140/180 [01:34<00:23, 1.67it/s]
2024-09-13 20:40:34,195 | pinder.core.utils.cloud:375 | INFO : Gsutil process_many=download_to_filename, threads=4, items=7
2024-09-13 20:40:34,441 | pinder.core.utils.cloud.process_many:23 | INFO : runtime succeeded: 0.25s
78%|███████▊ | 141/180 [01:35<00:29, 1.32it/s]
2024-09-13 20:40:35,325 | pinder.core.utils.cloud:375 | INFO : Gsutil process_many=download_to_filename, threads=4, items=7
2024-09-13 20:40:35,615 | pinder.core.utils.cloud.process_many:23 | INFO : runtime succeeded: 0.29s
79%|███████▉ | 142/180 [01:36<00:27, 1.40it/s]
2024-09-13 20:40:35,938 | pinder.core.utils.cloud:375 | INFO : Gsutil process_many=download_to_filename, threads=4, items=7
2024-09-13 20:40:36,181 | pinder.core.utils.cloud.process_many:23 | INFO : runtime succeeded: 0.24s
79%|███████▉ | 143/180 [01:36<00:26, 1.38it/s]
2024-09-13 20:40:36,679 | pinder.core.utils.cloud:375 | INFO : Gsutil process_many=download_to_filename, threads=4, items=7
2024-09-13 20:40:36,941 | pinder.core.utils.cloud.process_many:23 | INFO : runtime succeeded: 0.26s
80%|████████ | 144/180 [01:37<00:25, 1.42it/s]
2024-09-13 20:40:37,334 | pinder.core.utils.cloud:375 | INFO : Gsutil process_many=download_to_filename, threads=4, items=7
2024-09-13 20:40:37,522 | pinder.core.utils.cloud.process_many:23 | INFO : runtime succeeded: 0.19s
81%|████████ | 145/180 [01:38<00:22, 1.58it/s]
81%|████████ | 146/180 [01:38<00:18, 1.86it/s]
2024-09-13 20:40:38,118 | pinder.core.utils.cloud:375 | INFO : Gsutil process_many=download_to_filename, threads=4, items=5
2024-09-13 20:40:38,362 | pinder.core.utils.cloud.process_many:23 | INFO : runtime succeeded: 0.24s
82%|████████▏ | 147/180 [01:38<00:17, 1.85it/s]
82%|████████▏ | 148/180 [01:39<00:15, 2.04it/s]
2024-09-13 20:40:39,037 | pinder.core.utils.cloud:375 | INFO : Gsutil process_many=download_to_filename, threads=4, items=7
2024-09-13 20:40:39,316 | pinder.core.utils.cloud.process_many:23 | INFO : runtime succeeded: 0.28s
83%|████████▎ | 149/180 [01:39<00:16, 1.85it/s]
2024-09-13 20:40:39,701 | pinder.core.utils.cloud:375 | INFO : Gsutil process_many=download_to_filename, threads=4, items=7
2024-09-13 20:40:39,912 | pinder.core.utils.cloud.process_many:23 | INFO : runtime succeeded: 0.21s
83%|████████▎ | 150/180 [01:40<00:15, 1.90it/s]
2024-09-13 20:40:40,190 | pinder.core.utils.cloud:375 | INFO : Gsutil process_many=download_to_filename, threads=4, items=5
2024-09-13 20:40:40,407 | pinder.core.utils.cloud.process_many:23 | INFO : runtime succeeded: 0.22s
84%|████████▍ | 151/180 [01:40<00:14, 2.04it/s]
2024-09-13 20:40:40,600 | pinder.core.utils.cloud:375 | INFO : Gsutil process_many=download_to_filename, threads=4, items=7
2024-09-13 20:40:40,821 | pinder.core.utils.cloud.process_many:23 | INFO : runtime succeeded: 0.22s
84%|████████▍ | 152/180 [01:41<00:14, 1.93it/s]
2024-09-13 20:40:41,181 | pinder.core.utils.cloud:375 | INFO : Gsutil process_many=download_to_filename, threads=4, items=7
2024-09-13 20:40:41,407 | pinder.core.utils.cloud.process_many:23 | INFO : runtime succeeded: 0.23s
85%|████████▌ | 153/180 [01:42<00:14, 1.83it/s]
2024-09-13 20:40:41,805 | pinder.core.utils.cloud:375 | INFO : Gsutil process_many=download_to_filename, threads=4, items=384
2024-09-13 20:40:51,786 | pinder.core.utils.cloud.process_many:23 | INFO : runtime succeeded: 9.98s
86%|████████▌ | 154/180 [01:52<01:30, 3.47s/it]
2024-09-13 20:40:52,084 | pinder.core.utils.cloud:375 | INFO : Gsutil process_many=download_to_filename, threads=4, items=7
2024-09-13 20:40:52,330 | pinder.core.utils.cloud.process_many:23 | INFO : runtime succeeded: 0.25s
86%|████████▌ | 155/180 [01:53<01:09, 2.76s/it]
2024-09-13 20:40:53,197 | pinder.core.utils.cloud:375 | INFO : Gsutil process_many=download_to_filename, threads=4, items=7
2024-09-13 20:40:53,409 | pinder.core.utils.cloud.process_many:23 | INFO : runtime succeeded: 0.21s
87%|████████▋ | 156/180 [01:54<00:50, 2.12s/it]
2024-09-13 20:40:53,815 | pinder.core.utils.cloud:375 | INFO : Gsutil process_many=download_to_filename, threads=4, items=7
2024-09-13 20:40:54,039 | pinder.core.utils.cloud.process_many:23 | INFO : runtime succeeded: 0.22s
87%|████████▋ | 157/180 [01:54<00:38, 1.66s/it]
2024-09-13 20:40:54,390 | pinder.core.utils.cloud:375 | INFO : Gsutil process_many=download_to_filename, threads=4, items=5
2024-09-13 20:40:54,530 | pinder.core.utils.cloud.process_many:23 | INFO : runtime succeeded: 0.14s
88%|████████▊ | 158/180 [01:54<00:27, 1.26s/it]
2024-09-13 20:40:54,735 | pinder.core.utils.cloud:375 | INFO : Gsutil process_many=download_to_filename, threads=4, items=5
2024-09-13 20:40:54,937 | pinder.core.utils.cloud.process_many:23 | INFO : runtime succeeded: 0.20s
88%|████████▊ | 159/180 [01:55<00:21, 1.02s/it]
2024-09-13 20:40:55,191 | pinder.core.utils.cloud:375 | INFO : Gsutil process_many=download_to_filename, threads=4, items=7
2024-09-13 20:40:55,403 | pinder.core.utils.cloud.process_many:23 | INFO : runtime succeeded: 0.21s
89%|████████▉ | 160/180 [01:56<00:21, 1.07s/it]
2024-09-13 20:40:56,360 | pinder.core.utils.cloud:375 | INFO : Gsutil process_many=download_to_filename, threads=4, items=7
2024-09-13 20:40:56,577 | pinder.core.utils.cloud.process_many:23 | INFO : runtime succeeded: 0.22s
89%|████████▉ | 161/180 [01:57<00:18, 1.01it/s]
2024-09-13 20:40:57,169 | pinder.core.utils.cloud:375 | INFO : Gsutil process_many=download_to_filename, threads=4, items=7
2024-09-13 20:40:57,367 | pinder.core.utils.cloud.process_many:23 | INFO : runtime succeeded: 0.20s
90%|█████████ | 162/180 [01:57<00:14, 1.21it/s]
2024-09-13 20:40:57,610 | pinder.core.utils.cloud:375 | INFO : Gsutil process_many=download_to_filename, threads=4, items=7
2024-09-13 20:40:57,843 | pinder.core.utils.cloud.process_many:23 | INFO : runtime succeeded: 0.23s
91%|█████████ | 163/180 [01:58<00:13, 1.28it/s]
2024-09-13 20:40:58,281 | pinder.core.utils.cloud:375 | INFO : Gsutil process_many=download_to_filename, threads=4, items=7
2024-09-13 20:40:58,480 | pinder.core.utils.cloud.process_many:23 | INFO : runtime succeeded: 0.20s
91%|█████████ | 164/180 [01:58<00:10, 1.46it/s]
92%|█████████▏| 165/180 [01:59<00:08, 1.76it/s]
2024-09-13 20:40:59,048 | pinder.core.utils.cloud:375 | INFO : Gsutil process_many=download_to_filename, threads=4, items=7
2024-09-13 20:40:59,229 | pinder.core.utils.cloud.process_many:23 | INFO : runtime succeeded: 0.18s
92%|█████████▏| 166/180 [01:59<00:07, 1.91it/s]
2024-09-13 20:40:59,464 | pinder.core.utils.cloud:375 | INFO : Gsutil process_many=download_to_filename, threads=4, items=7
2024-09-13 20:40:59,648 | pinder.core.utils.cloud.process_many:23 | INFO : runtime succeeded: 0.18s
93%|█████████▎| 167/180 [02:00<00:06, 1.98it/s]
2024-09-13 20:40:59,932 | pinder.core.utils.cloud:375 | INFO : Gsutil process_many=download_to_filename, threads=4, items=7
2024-09-13 20:41:00,133 | pinder.core.utils.cloud.process_many:23 | INFO : runtime succeeded: 0.20s
93%|█████████▎| 168/180 [02:00<00:06, 1.89it/s]
2024-09-13 20:41:00,511 | pinder.core.utils.cloud:375 | INFO : Gsutil process_many=download_to_filename, threads=4, items=7
2024-09-13 20:41:00,758 | pinder.core.utils.cloud.process_many:23 | INFO : runtime succeeded: 0.25s
94%|█████████▍| 169/180 [02:01<00:05, 1.86it/s]
2024-09-13 20:41:01,072 | pinder.core.utils.cloud:375 | INFO : Gsutil process_many=download_to_filename, threads=4, items=5
2024-09-13 20:41:01,245 | pinder.core.utils.cloud.process_many:23 | INFO : runtime succeeded: 0.17s
94%|█████████▍| 170/180 [02:01<00:04, 2.07it/s]
2024-09-13 20:41:01,423 | pinder.core.utils.cloud:375 | INFO : Gsutil process_many=download_to_filename, threads=4, items=5
2024-09-13 20:41:01,573 | pinder.core.utils.cloud.process_many:23 | INFO : runtime succeeded: 0.15s
95%|█████████▌| 171/180 [02:02<00:04, 2.07it/s]
96%|█████████▌| 172/180 [02:02<00:03, 2.30it/s]
2024-09-13 20:41:02,231 | pinder.core.utils.cloud:375 | INFO : Gsutil process_many=download_to_filename, threads=4, items=7
2024-09-13 20:41:02,435 | pinder.core.utils.cloud.process_many:23 | INFO : runtime succeeded: 0.20s
96%|█████████▌| 173/180 [02:03<00:03, 2.09it/s]
2024-09-13 20:41:02,810 | pinder.core.utils.cloud:375 | INFO : Gsutil process_many=download_to_filename, threads=4, items=7
2024-09-13 20:41:03,062 | pinder.core.utils.cloud.process_many:23 | INFO : runtime succeeded: 0.25s
97%|█████████▋| 174/180 [02:04<00:03, 1.56it/s]
2024-09-13 20:41:03,829 | pinder.core.utils.cloud:375 | INFO : Gsutil process_many=download_to_filename, threads=4, items=5
2024-09-13 20:41:04,010 | pinder.core.utils.cloud.process_many:23 | INFO : runtime succeeded: 0.18s
97%|█████████▋| 175/180 [02:04<00:02, 1.71it/s]
2024-09-13 20:41:04,282 | pinder.core.utils.cloud:375 | INFO : Gsutil process_many=download_to_filename, threads=4, items=5
2024-09-13 20:41:04,467 | pinder.core.utils.cloud.process_many:23 | INFO : runtime succeeded: 0.19s
98%|█████████▊| 176/180 [02:05<00:02, 1.49it/s]
2024-09-13 20:41:05,159 | pinder.core.utils.cloud:375 | INFO : Gsutil process_many=download_to_filename, threads=4, items=7
2024-09-13 20:41:05,384 | pinder.core.utils.cloud.process_many:23 | INFO : runtime succeeded: 0.22s
98%|█████████▊| 177/180 [02:06<00:02, 1.48it/s]
2024-09-13 20:41:05,857 | pinder.core.utils.cloud:375 | INFO : Gsutil process_many=download_to_filename, threads=4, items=7
2024-09-13 20:41:06,064 | pinder.core.utils.cloud.process_many:23 | INFO : runtime succeeded: 0.21s
99%|█████████▉| 178/180 [02:06<00:01, 1.38it/s]
2024-09-13 20:41:06,688 | pinder.core.utils.cloud:375 | INFO : Gsutil process_many=download_to_filename, threads=4, items=7
2024-09-13 20:41:06,899 | pinder.core.utils.cloud.process_many:23 | INFO : runtime succeeded: 0.21s
99%|█████████▉| 179/180 [02:07<00:00, 1.55it/s]
2024-09-13 20:41:07,147 | pinder.core.utils.cloud:375 | INFO : Gsutil process_many=download_to_filename, threads=4, items=5
2024-09-13 20:41:07,326 | pinder.core.utils.cloud.process_many:23 | INFO : runtime succeeded: 0.18s
100%|██████████| 180/180 [02:07<00:00, 1.67it/s]
100%|██████████| 180/180 [02:07<00:00, 1.41it/s]
Loading a specific list of systems#
systems = [
"1df0__A1_Q07009--1df0__B1_Q64537",
"117e__A1_P00817--117e__B1_P00817",
]
loader = PinderLoader(
ids=systems,
monomer_priority="holo",
base_filters = base_filters,
sub_filters = sub_filters
)
passing_ids = []
for item in loader:
passing_ids.append(item[0].entry.id)
systems_removed_by_filters = set(systems) - set(passing_ids)
systems_removed_by_filters
2024-09-13 20:41:08,139 | pinder.core.utils.cloud:375 | INFO : Gsutil process_many=download_to_filename, threads=4, items=7
2024-09-13 20:41:08,336 | pinder.core.utils.cloud.process_many:23 | INFO : runtime succeeded: 0.20s
set()
len(systems) == len(passing_ids)
True
Optional Pinder writer#
Without defining a writer for the PinderLoader, the loaded systems are available as a tuple of (PinderSystem, Structure, Structure) objects, containing the original PinderSystem and the sampled feature and target complexes, respectively.
If you want to explicitly write the (potentially transformed) structure objects to a custom location or in a custom format (e.g. PDB, pickle, etc.), you can implement a subclass of PinderWriterBase.
The default writer implements writing to PDB files (leveraging the Structure.to_pdb method on the structure objects).
from pinder.core.loader.writer import PinderDefaultWriter
from pathlib import Path
from tempfile import TemporaryDirectory
with TemporaryDirectory() as tmp_dir:
temp_dir = Path(tmp_dir)
loader = PinderLoader(
ids=systems,
monomer_priority="pred",
writer=PinderDefaultWriter(temp_dir)
)
assert set(loader.index.id) == set(systems)
for i, r in loader.index.iterrows():
loaded = loader[i]
pinder_id = r.id
system_dir = loader.writer.output_path / pinder_id
assert system_dir.is_dir()
print(list(system_dir.glob("af_*.pdb")))
[PosixPath('/tmp/tmp0luhkilv/117e__A1_P00817--117e__B1_P00817/af__P00817.pdb')]
[PosixPath('/tmp/tmp0luhkilv/1df0__A1_Q07009--1df0__B1_Q64537/af__Q07009.pdb'), PosixPath('/tmp/tmp0luhkilv/1df0__A1_Q07009--1df0__B1_Q64537/af__Q64537.pdb')]
Constructing torch datasets and dataloaders from pinder systems#
The remaining sections of this tutorial will be for those interested specifically in torch datasets and dataloaders.
Specifically, we will show how to:
Implement a PyTorch
Datasetto interface with pinder dataInclude apo and predicted monomers in the data pipeline, with an option to target specific monomer types or randomly sample from the available types
Leverage
PinderSystemand its associated methods to crop apo/predicted monomers to match the ground-truth holo monomersWrite filters and transforms that operate on
StructureobjectsIntegrate annotations in data filtering and featurization
Create example features to use for training (you will of course choose your own features)
Incorporate diversity sampling in the data loader
The pinder.core.loader.dataset module provides two example implementations of how to integrate the pinder dataset into a torch-based machine learning pipeline.
PinderDataset: A map-styletorch.utils.data.Datasetthat can be used with torchDataLoader’s.PPIDataset: Atorch_geometric.data.Datasetthat can be used with torch-geometricDataLoader’s. This dataset is designed to be used with thetorch_geometricpackage.
Together, the two datasets provide an example implementation of how to abstract away the complexity of loading and processing multiple structures associated with each PinderSystem by leveraging the following utilities from pinder:
pinder.core.PinderLoaderpinder.core.loader.filterspinder.core.loader.transforms
The examples cover two different batch data item structures to illustrate two different use-cases:
PinderDataset: A batch of(target_complex, feature_complex)pairs, wheretarget_complexandfeature_complexaretorch.Tensorobjects representing the atomic coordinates and atom types of the holo and sampled (decoy, holo/apo/pred) complexes, respectively.PPIDataset: A batch ofPairedPDBobjects, where the receptor and ligand are encoded separately in a heterogeneous graph, viatorch_geometric.data.HeteroData, holding multiple node and/or edge types in disjunct storage objects.
The remaining sections will be split into:
Using the
PinderDatasettorch datasetUsing the
PPIDatasettorch-geometric datasetHow you could implement your own dataset & dataloader
PinderDataset (torch Dataset)#
The PinderDataset is an example implementation of a torch.utils.data.Dataset that represents its data items as a dict containing the following key, value pairs:
target_complex: The ground-truth holo dimer, represented with a set of default properties encoded asTensor’sfeature_complex: The sampled dimer complex, representing “features”, also represented with a set of default properties encoded asTensor’sid: The pinder ID for the selected systemtarget_id: The IDs of the receptor and ligand holo monomers, concatenated into a single ID stringsample_id: The IDs of the sampled receptor and ligand holo monomers, concatenated into a single ID string. This can be useful for debugging purposes or generally tracking which specific monomers are selected when targeting alternative monomers (more on this shortly)
Each of the target_complex and feature_complex values are dictionaries with structural properties encoded by the pinder.core.loader.geodata.structure2tensor function by default:
atom_coordinatesatom_typesresidue_coordinatesresidue_typesresidue_ids
You can choose to use a different representation by overriding the default values of transform and target_transform.
It leverages the PinderLoader to apply optional filters and/or transforms, provide an interface for sampling alternative monomers, and exposes transform and target_transform arguments used by the torch Dataset API.
For more details on the torch Dataset APIs, please refer to the tutorials.
from pinder.core.loader import filters, transforms
from pinder.core.loader.dataset import PinderDataset
base_filters = [
filters.FilterByMissingHolo(),
filters.FilterSubByContacts(min_contacts=5, radius=10.0, calpha_only=True),
filters.FilterDetachedHolo(radius=12, max_components=2),
]
sub_filters = [
filters.FilterSubByAtomTypes(min_atom_types=4),
filters.FilterByHoloOverlap(min_overlap=5),
filters.FilterByHoloSeqIdentity(min_sequence_identity=0.8),
filters.FilterSubRmsds(rmsd_cutoff=7.5),
filters.FilterDetachedSub(radius=12, max_components=2),
]
# We can include Structure-level transforms (and filters) which will operate on the target and feature complexes
structure_transforms = [
transforms.SelectAtomTypes(atom_types=["CA", "N", "C", "O"])
]
train_dataset = PinderDataset(
split="train",
# We can leverage holo, apo, pred, random and random_mixed monomer sampling strategies
monomer_priority="random_mixed",
base_filters = base_filters,
sub_filters = sub_filters,
structure_transforms=structure_transforms,
)
assert len(train_dataset) == len(get_index().query('split == "train"'))
train_dataset
<pinder.core.loader.dataset.PinderDataset at 0x7fe7432a2d70>
Sampling alternative monomers#
The monomer_priority argument can be used to target different mixes of bound and unbound monomers to use for creating the decoy/feature complex.
The allowed values for monomer_priority are “apo”, “holo”, “pred”, “random” or “random_mixed”.
When monomer_priority is set to one of the available monomer types (holo, apo, pred), the same monomer type will be selected for both receptor and ligand.
When the monomer priority is “random”, a random monomer type will be selected from the set of monomer types available for both the receptor and ligand. This option ensures the same type of monomer is used for the receptor and ligand.
When the monomer priority is “random_mixed”, a random monomer type will be selected for each of receptor and ligand, separately.
Enabling the fallback_to_holo option (default) will enable silent fallback to holo when the monomer_priority is set to one of apo or pred, but the corresponding monomer is not available for the dimer.
This is useful when only one of receptor or ligand has an unbound monomer, but you wish to include apo or predicted structures in your workflow.
If fallback_to_holo is disabled, an error will be raised when the monomer_priority is set to one of apo or pred, but the corresponding monomer is not available for the dimer.
By default, when apo monomers are selected, the “canonical” apo monomer is used. Although a single canonical apo monomer should be used for eval, pinder provides multiple apo monomers paired to a single holo monomer (when available). In order to include these non-canonical/alternative monomers, you can specify use_canonical_apo=False when constructing the PinderLoader or PinderDataset objects.
data_item = train_dataset[0]
data_item
{'target_complex': {'atom_types': tensor([[0., 0., 0., ..., 0., 0., 0.],
[1., 0., 0., ..., 0., 0., 0.],
[1., 0., 0., ..., 0., 0., 0.],
...,
[1., 0., 0., ..., 0., 0., 0.],
[1., 0., 0., ..., 0., 0., 0.],
[0., 0., 1., ..., 0., 0., 0.]]),
'residue_types': tensor([[16.],
[16.],
[16.],
...,
[ 0.],
[ 0.],
[ 0.]]),
'atom_coordinates': tensor([[131.7500, 429.3090, 163.5360],
[132.6810, 428.2520, 163.1550],
[133.5150, 428.6750, 161.9500],
...,
[177.7620, 463.8650, 166.9020],
[177.4130, 465.0800, 167.7550],
[176.8000, 464.9490, 168.8150]]),
'residue_coordinates': tensor([[131.7500, 429.3090, 163.5360],
[132.6810, 428.2520, 163.1550],
[133.5150, 428.6750, 161.9500],
...,
[177.7620, 463.8650, 166.9020],
[177.4130, 465.0800, 167.7550],
[176.8000, 464.9490, 168.8150]]),
'residue_ids': tensor([ 4., 4., 4., ..., 182., 182., 182.])},
'feature_complex': {'atom_types': tensor([[0., 0., 0., ..., 0., 0., 0.],
[1., 0., 0., ..., 0., 0., 0.],
[1., 0., 0., ..., 0., 0., 0.],
...,
[1., 0., 0., ..., 0., 0., 0.],
[1., 0., 0., ..., 0., 0., 0.],
[0., 0., 1., ..., 0., 0., 0.]]),
'residue_types': tensor([[16.],
[16.],
[16.],
...,
[ 0.],
[ 0.],
[ 0.]]),
'atom_coordinates': tensor([[131.7500, 429.3090, 163.5360],
[132.6810, 428.2520, 163.1550],
[133.5150, 428.6750, 161.9500],
...,
[177.7620, 463.8650, 166.9020],
[177.4130, 465.0800, 167.7550],
[176.8000, 464.9490, 168.8150]]),
'residue_coordinates': tensor([[131.7500, 429.3090, 163.5360],
[132.6810, 428.2520, 163.1550],
[133.5150, 428.6750, 161.9500],
...,
[177.7620, 463.8650, 166.9020],
[177.4130, 465.0800, 167.7550],
[176.8000, 464.9490, 168.8150]]),
'residue_ids': tensor([ 4., 4., 4., ..., 182., 182., 182.])},
'id': '8phr__X4_UNDEFINED--8phr__W4_UNDEFINED',
'sample_id': '8phr__X4_UNDEFINED-R--8phr__W4_UNDEFINED-L',
'target_id': '8phr__X4_UNDEFINED-R--8phr__W4_UNDEFINED-L'}
# Since we used the default option of crop_equal_monomer_shapes, we should expect feature and target complex coords are identical shapes
assert (
data_item["feature_complex"]["atom_coordinates"].shape
== data_item["target_complex"]["atom_coordinates"].shape
)
data_item["feature_complex"]["atom_coordinates"].shape
torch.Size([1316, 3])
help(PinderDataset)
Help on class PinderDataset in module pinder.core.loader.dataset:
class PinderDataset(torch.utils.data.dataset.Dataset)
| PinderDataset(split: 'str | None' = None, index: 'pd.DataFrame | None' = None, metadata: 'pd.DataFrame | None' = None, monomer_priority: 'str' = 'holo', base_filters: 'list[PinderFilterBase]' = [], sub_filters: 'list[PinderFilterSubBase]' = [], structure_filters: 'list[StructureFilter]' = [], structure_transforms: 'list[StructureTransform]' = [], transform: 'Callable[[Structure], torch.Tensor | dict[str, torch.Tensor]]' = <function structure2tensor_transform at 0x7fe749c4f490>, target_transform: 'Callable[[Structure], torch.Tensor | dict[str, torch.Tensor]]' = <function structure2tensor_transform at 0x7fe749c4f490>, ids: 'list[str] | None' = None, fallback_to_holo: 'bool' = True, use_canonical_apo: 'bool' = True, crop_equal_monomer_shapes: 'bool' = True, index_query: 'str | None' = None, metadata_query: 'str | None' = None, pre_specified_monomers: 'dict[str, str] | pd.DataFrame | None' = None, **kwargs: 'Any') -> 'None'
|
| Method resolution order:
| PinderDataset
| torch.utils.data.dataset.Dataset
| typing.Generic
| builtins.object
|
| Methods defined here:
|
| __getitem__(self, idx: 'int') -> 'dict[str, dict[str, torch.Tensor] | torch.Tensor]'
|
| __init__(self, split: 'str | None' = None, index: 'pd.DataFrame | None' = None, metadata: 'pd.DataFrame | None' = None, monomer_priority: 'str' = 'holo', base_filters: 'list[PinderFilterBase]' = [], sub_filters: 'list[PinderFilterSubBase]' = [], structure_filters: 'list[StructureFilter]' = [], structure_transforms: 'list[StructureTransform]' = [], transform: 'Callable[[Structure], torch.Tensor | dict[str, torch.Tensor]]' = <function structure2tensor_transform at 0x7fe749c4f490>, target_transform: 'Callable[[Structure], torch.Tensor | dict[str, torch.Tensor]]' = <function structure2tensor_transform at 0x7fe749c4f490>, ids: 'list[str] | None' = None, fallback_to_holo: 'bool' = True, use_canonical_apo: 'bool' = True, crop_equal_monomer_shapes: 'bool' = True, index_query: 'str | None' = None, metadata_query: 'str | None' = None, pre_specified_monomers: 'dict[str, str] | pd.DataFrame | None' = None, **kwargs: 'Any') -> 'None'
| Initialize self. See help(type(self)) for accurate signature.
|
| __len__(self) -> 'int'
|
| ----------------------------------------------------------------------
| Data and other attributes defined here:
|
| __annotations__ = {}
|
| __parameters__ = ()
|
| ----------------------------------------------------------------------
| Methods inherited from torch.utils.data.dataset.Dataset:
|
| __add__(self, other: 'Dataset[T_co]') -> 'ConcatDataset[T_co]'
|
| ----------------------------------------------------------------------
| Data descriptors inherited from torch.utils.data.dataset.Dataset:
|
| __dict__
| dictionary for instance variables (if defined)
|
| __weakref__
| list of weak references to the object (if defined)
|
| ----------------------------------------------------------------------
| Data and other attributes inherited from torch.utils.data.dataset.Dataset:
|
| __orig_bases__ = (typing.Generic[+T_co],)
|
| ----------------------------------------------------------------------
| Class methods inherited from typing.Generic:
|
| __class_getitem__(params) from builtins.type
|
| __init_subclass__(*args, **kwargs) from builtins.type
| This method is called when a class is subclassed.
|
| The default implementation does nothing. It may be
| overridden to extend subclasses.
Torch DataLoader for PinderDataset#
The PinderDataset can be served by a torch.utils.data.DataLoader.
There is a convenience function pinder.core.loader.dataset.get_torch_loader for taking a PinderDataset and returning a DataLoader for the dataset object.
We can leverage the default collate_fn (pinder.core.loader.dataset.collate_batch) to merge multiple systems (Dataset items) to create mini-batches of tensors:
from pinder.core.loader.dataset import collate_batch, get_torch_loader
from torch.utils.data import DataLoader
batch_size = 2
train_dataloader = get_torch_loader(
train_dataset,
batch_size=batch_size,
shuffle=True,
collate_fn=collate_batch,
num_workers=0,
)
assert isinstance(train_dataloader, DataLoader)
assert hasattr(train_dataloader, "dataset")
# Get a batch from the dataloader
batch = next(iter(train_dataloader))
# expected batch dict keys
assert set(batch.keys()) == {
"target_complex",
"feature_complex",
"id",
"sample_id",
"target_id",
}
assert isinstance(batch["target_complex"], dict)
assert isinstance(batch["target_complex"]["atom_coordinates"], torch.Tensor)
feature_coords = batch["feature_complex"]["atom_coordinates"]
# Ensure batch size propagates to tensor dims
assert feature_coords.shape[0] == batch_size
# Ensure coordinates have dim 3
assert feature_coords.shape[2] == 3
2024-09-13 20:41:17,041 | pinder.core.utils.cloud:375 | INFO : Gsutil process_many=download_to_filename, threads=4, items=5
2024-09-13 20:41:17,221 | pinder.core.utils.cloud.process_many:23 | INFO : runtime succeeded: 0.18s
2024-09-13 20:41:17,721 | pinder.core.utils.cloud:375 | INFO : Gsutil process_many=download_to_filename, threads=4, items=5
2024-09-13 20:41:17,900 | pinder.core.utils.cloud.process_many:23 | INFO : runtime succeeded: 0.18s
Torch geometric Dataset#
# Make sure to install torch_cluster
# !pip install torch_cluster
from pinder.core.loader.dataset import PPIDataset
from pinder.core.loader.geodata import NodeRepresentation
nodes = {NodeRepresentation("atom"), NodeRepresentation("residue")}
train_dataset = PPIDataset(
node_types=nodes,
split="train",
monomer1="holo_receptor",
monomer2="holo_ligand",
limit_by=5,
force_reload=True,
parallel=False,
)
assert len(train_dataset) == 5
train_dataset
Processing...
2024-09-13 20:41:23,671 | pinder.core.utils.cloud:375 | INFO : Gsutil process_many=download_to_filename, threads=4, items=7
2024-09-13 20:41:23,777 | pinder.core.utils.cloud.process_many:23 | INFO : runtime succeeded: 0.11s
2024-09-13 20:41:24,515 | pinder.core.utils.cloud:375 | INFO : Gsutil process_many=download_to_filename, threads=4, items=5
2024-09-13 20:41:24,581 | pinder.core.utils.cloud.process_many:23 | INFO : runtime succeeded: 0.07s
2024-09-13 20:41:24,708 | pinder.core.utils.cloud:375 | INFO : Gsutil process_many=download_to_filename, threads=4, items=7
2024-09-13 20:41:24,841 | pinder.core.utils.cloud.process_many:23 | INFO : runtime succeeded: 0.13s
2024-09-13 20:41:24,980 | pinder.core.utils.cloud:375 | INFO : Gsutil process_many=download_to_filename, threads=4, items=7
2024-09-13 20:41:25,054 | pinder.core.utils.cloud.process_many:23 | INFO : runtime succeeded: 0.07s
2024-09-13 20:41:25,136 | pinder.core.loader.dataset:533 | INFO : Finished processing, only 5 systems
Done!
PPIDataset(5)
from torch_geometric.data import HeteroData
from pinder.core import get_index
# Here we check that we correctly capped the number of systems to load to 5 (via limit_by)
pindex = get_index()
raw_ids = set(train_dataset.raw_file_names)
assert len(raw_ids.intersection(set(pindex.id))) == 5
processed_ids = {f.stem for f in train_dataset.processed_file_names}
# Here we ensure that all 5 ids got processed and saved as .pt file on disk
assert len(processed_ids.intersection(set(pindex.id))) == 5
# Let's get an item from the dataset by index
data_item = train_dataset[0]
assert isinstance(data_item, HeteroData)
data_item
PairedPDB(
ligand_residue={
residueid=[286, 1],
pos=[286, 3],
edge_index=[2, 2860],
chain=[1],
},
receptor_residue={
residueid=[290, 1],
pos=[290, 3],
edge_index=[2, 2900],
chain=[1],
},
ligand_atom={
x=[2313, 12],
pos=[2313, 3],
edge_index=[2, 23130],
},
receptor_atom={
x=[2344, 12],
pos=[2344, 3],
edge_index=[2, 23440],
},
pdb={
id=[1],
num_nodes=1,
}
)
# We can also get an item by its system ID
data_item = train_dataset.get_filename("8phr__X4_UNDEFINED--8phr__W4_UNDEFINED")
data_item
PairedPDB(
ligand_residue={
residueid=[158, 1],
pos=[158, 3],
edge_index=[2, 1580],
chain=[1],
},
receptor_residue={
residueid=[171, 1],
pos=[171, 3],
edge_index=[2, 1710],
chain=[1],
},
ligand_atom={
x=[1198, 12],
pos=[1198, 3],
edge_index=[2, 11980],
},
receptor_atom={
x=[1358, 12],
pos=[1358, 3],
edge_index=[2, 13580],
},
pdb={
id=[1],
num_nodes=1,
}
)
train_dataset.print_summary()
PPIDataset (#graphs=5):
+------------+----------+----------+
| | #nodes | #edges |
|------------+----------+----------|
| mean | 2612.4 | 0 |
| std | 1631.6 | 0 |
| min | 999 | 0 |
| quantile25 | 1600 | 0 |
| median | 2343 | 0 |
| quantile75 | 2886 | 0 |
| max | 5234 | 0 |
+------------+----------+----------+
Number of nodes per node type:
+------------+------------------+--------------------+---------------+-----------------+-------+
| | ligand_residue | receptor_residue | ligand_atom | receptor_atom | pdb |
|------------+------------------+--------------------+---------------+-----------------+-------|
| mean | 140.2 | 152.8 | 1103.4 | 1215 | 1 |
| std | 90.2 | 87.5 | 738.9 | 717.1 | 0 |
| min | 58 | 62 | 426 | 452 | 1 |
| quantile25 | 78 | 97 | 622 | 802 | 1 |
| median | 121 | 144 | 958 | 1119 | 1 |
| quantile75 | 158 | 171 | 1198 | 1358 | 1 |
| max | 286 | 290 | 2313 | 2344 | 1 |
+------------+------------------+--------------------+---------------+-----------------+-------+
PairedPDB torch-geometric HeteroData object#
The PPIDataset represents its data items as PairedPDB objects, where the receptor and ligand are encoded separately in a heterogeneous graph, via torch_geometric.data.HeteroData, holding multiple node and/or edge types in disjunct storage objects.
It leverages the PinderLoader to apply optional filters and/or transforms and implements a caching system by saving processed systems to disk in .pt files.
The PairedPDB implements a conversion method .from_pinder_system that takes in a PinderSystem and converts it to a PairedPDB object.
For more details on the torch-geometric Dataset APIs, please refer to the tutorials.
from pinder.core.loader.geodata import PairedPDB
from torch_geometric.data import HeteroData
pinder_id = "3s9d__B1_P48551--3s9d__A1_P01563"
system = PinderSystem(pinder_id)
holo_data = PairedPDB.from_pinder_system(
system=system,
monomer1="holo_receptor", monomer2="holo_ligand",
node_types=nodes,
)
assert isinstance(holo_data, HeteroData)
expected_node_types = [
'ligand_residue', 'receptor_residue', 'ligand_atom', 'receptor_atom'
]
assert holo_data.num_nodes == 2780
assert holo_data.num_edges == 0
assert isinstance(holo_data.num_node_features, dict)
expected_num_feats = {
'ligand_residue': 0,
'receptor_residue': 0,
'ligand_atom': 12,
'receptor_atom': 12
}
for k, v in expected_num_feats.items():
assert holo_data.num_node_features[k] == v
assert holo_data.node_types == expected_node_types
holo_data
2024-09-13 20:41:25,992 | pinder.core.utils.cloud:375 | INFO : Gsutil process_many=download_to_filename, threads=4, items=15
2024-09-13 20:41:26,204 | pinder.core.utils.cloud.process_many:23 | INFO : runtime succeeded: 0.21s
PairedPDB(
ligand_residue={
residueid=[127, 1],
pos=[127, 3],
edge_index=[2, 1270],
chain=[1],
},
receptor_residue={
residueid=[180, 1],
pos=[180, 3],
edge_index=[2, 1800],
chain=[1],
},
ligand_atom={
x=[1032, 12],
pos=[1032, 3],
edge_index=[2, 10320],
},
receptor_atom={
x=[1441, 12],
pos=[1441, 3],
edge_index=[2, 14410],
}
)
# You can target specific monomers (apo/holo/pred) for the receptor and ligand
apo_data = PairedPDB.from_pinder_system(
system=system,
monomer1="apo_receptor", monomer2="apo_ligand",
node_types=nodes,
)
assert isinstance(apo_data, HeteroData)
assert apo_data.num_nodes == 3437
assert apo_data.num_edges == 0
assert isinstance(apo_data.num_node_features, dict)
expected_num_feats = {
'ligand_residue': 0,
'receptor_residue': 0,
'ligand_atom': 12,
'receptor_atom': 12
}
for k, v in expected_num_feats.items():
assert apo_data.num_node_features[k] == v
assert apo_data.node_types == expected_node_types
apo_data
PairedPDB(
ligand_residue={
residueid=[165, 1],
pos=[165, 3],
edge_index=[2, 1650],
chain=[1],
},
receptor_residue={
residueid=[212, 1],
pos=[212, 3],
edge_index=[2, 2120],
chain=[1],
},
ligand_atom={
x=[1350, 12],
pos=[1350, 3],
edge_index=[2, 13500],
},
receptor_atom={
x=[1710, 12],
pos=[1710, 3],
edge_index=[2, 17100],
}
)
Torch geometric DataLoader#
The PPIDataset can be served by a torch_geometric.DataLoader.
There is a convenience function pinder.core.loader.dataset.get_geo_loader for taking a PPIDataset and returning a DataLoader for the dataset object.
from pinder.core.loader.dataset import get_geo_loader
from torch_geometric.loader import DataLoader
loader = get_geo_loader(train_dataset)
assert isinstance(loader, DataLoader)
assert hasattr(loader, "dataset")
ds = loader.dataset
assert len(ds) == 5
loader
<torch_geometric.loader.dataloader.DataLoader at 0x7fe7432fdd50>
ds
PPIDataset(5)
Implementing your own PyTorch Dataset & DataLoader for pinder#
While the previous sections covered two example implementations of leveraging the torch and torch-geometric APIs, below we will illustrate how you can write your own PyTorch data pipeline to feed your model.
We will focus on writing a minimalistic example that simply fetches PDB files and returns the coordinates as tensors. We will also include an example of a sampler function for the dataloader to implement diversity sampling in your workflow.
Defining the Dataset#
Below we will write a barebones torch.utils.data.Dataset object that implements at a minimum:
__init__method__len__method returning the number of items in the dataset__getitem__method that returns an item in the dataset
We will also add an option to include apo and predicted monomer types by adding a monomer_priority argument to our dataset. The argument will be set to “random” by default, indicating that we want to randomly sample a monomer type from the set of monomers available for a given system. We could also target a specific monomer type by setting this argument to one of “holo”, “apo”, “predicted”. Not every system in the training set has apo or predicted structures available, so we will also add a fallback_to_holo argument to indicate whether we want to use holo monomers when the selected monomer type is not available.
We also define two interfaces for applying filters to our dataset:
metadata_filter: a query string to apply to the pinder metadata pandas DataFramesystem_filters: list[PinderFilterBase]: a list of filters that inheret a base class,PinderFilterBase, which serves as the abstraction layer for definingPinderSystem-based filtersstructure_filters: list[StructureFilter]: a list of filters that inheret a base class,StructureFilter, which serves as the abstraction layer for definingStructure-based filters
import numpy as np
from numpy.typing import NDArray
from torch.utils.data import Dataset
from pinder.core import get_index, get_metadata
from pinder.core.loader.loader import _create_target_feature_complex, select_monomer
from pinder.core.loader.structure import Structure
index = get_index()
metadata = get_metadata()
class CustomPinderDataset(Dataset):
def __init__(
self,
split: str,
monomer_priority: str = "random",
fallback_to_holo: bool = True,
crop_equal_monomer_shapes: bool = True,
use_canonical_apo: bool = True,
transform=None,
target_transform=None,
structure_filters: list[filters.StructureFilter] = [],
system_filters: list[filters.PinderFilterBase] = [],
metadata_filter: str | None = None,
max_load_attempts: int = 10,
) -> None:
# Which split/mode we are using for the dataset instance
self.split = split
self.monomer_priority = monomer_priority
self.fallback_to_holo = fallback_to_holo
self.crop_equal_monomer_shapes = crop_equal_monomer_shapes
# Optional transform and target transform to apply (will be covered shortly)
self.transform = transform
self.target_transform = target_transform
# Optional system-level filters to apply
self.system_filters = system_filters
# Optional structure filters to apply
self.structure_filters = structure_filters
# Maximum number of times to try sampling another index from the dataset until an exception is raised
self.max_load_attempts = max_load_attempts
# Whether we should use canonical apo structures (apo_R/L_pdb columns in pinder index) if apo monomers are selected
self.use_canonical_apo = use_canonical_apo
# Define the subset of the pinder index and metadata corresponding to the split of our dataset instance
self.index = index.query(f'split == "{split}"').reset_index(drop=True)
self.metadata = metadata[metadata["id"].isin(set(self.index.id))].reset_index(drop=True)
if metadata_filter:
try:
self.metadata = self.metadata.query(metadata_filter).reset_index(drop=True)
except Exception as e:
print(f"Failed to apply metadata_filter={metadata_filter}: {e}")
self.index = self.index[self.index["id"].isin(set(self.metadata.id))].reset_index(drop=True)
def __len__(self):
return len(self.index)
def __getitem__(self, idx: int) -> tuple[NDArray[np.double], NDArray[np.double]]:
valid_idx = False
attempts = 0
while not valid_idx and attempts < self.max_load_attempts:
attempts += 1
row = self.index.iloc[idx]
system = PinderSystem(row.id)
system = self.apply_system_filters(system)
if not isinstance(system, PinderSystem):
continue
selected_monomers = select_monomer(
row,
self.monomer_priority,
self.fallback_to_holo,
self.use_canonical_apo,
)
# With the system and selected_monomers objects, we can now create a pair of dimer complexes
# Below we leverage the existing utility from the PinderLoader (_create_target_feature_complex)
target_complex, feature_complex = _create_target_feature_complex(
system, selected_monomers, self.crop_equal_monomer_shapes, self.fallback_to_holo
)
valid_idx = self.apply_structure_filters(target_complex)
if not valid_idx:
# Try another index before raising IndexError
idx = random.choice(list(range(len(self))))
if not valid_idx:
raise IndexError(
f"Unable to find a valid item in the dataset satisfying filters at {idx} after {attempts} attempts!"
)
if self.transform is not None:
feature_complex = self.transform(feature_complex)
if self.target_transform is not None:
target_complex = self.target_transform(target_complex)
return feature_complex, target_complex
def apply_structure_filters(self, structure: Structure) -> bool:
pass_filters = True
for structure_filter in self.structure_filters:
if not structure_filter(structure):
pass_filters = False
break
return pass_filters
def apply_system_filters(self, system: PinderSystem) -> PinderSystem | bool:
for system_filter in self.system_filters:
if isinstance(system_filter, filters.PinderFilterBase):
if not base_filter(system):
return False
return system
def __repr__(self) -> str:
return f"CustomPinderDataset(split={self.split}, monomers={self.monomer_priority}, systems={len(self)})"
# Note the selected monomers indicated by Structure.pinder_id attributes. Since we enabled cropping, the feature and target complex AtomArray have identical shapes
test_data = CustomPinderDataset(split="test")
test_data[0]
(Structure(
filepath=/home/runner/.local/share/pinder/2024-02/pdbs/af__A0A229LVN5--af__A0A229LVN5.pdb,
uniprot_map=None,
pinder_id='af__A0A229LVN5--af__A0A229LVN5',
atom_array=<class 'biotite.structure.AtomArray'> with shape (2092,),
pdb_engine='fastpdb',
),
Structure(
filepath=/home/runner/.local/share/pinder/2024-02/test_set_pdbs/7rzb__A1_A0A229LVN5-R--7rzb__A2_A0A229LVN5-L.pdb,
uniprot_map=<class 'pandas.core.frame.DataFrame'> with shape (294, 14),
pinder_id='7rzb__A1_A0A229LVN5-R--7rzb__A2_A0A229LVN5-L',
atom_array=<class 'biotite.structure.AtomArray'> with shape (2092,),
pdb_engine='fastpdb',
))
In the above example, we wrote a torch Dataset that currently returns a tuple of Structure objects (one representing the “decoy” or sample to use for features and the other representing the ground-truth “label”)
Of course we wouldn’t use these Structure objects directly in our model. In your model, you will have to choose which features to compute and which data structure to use to represent them.
While the PinderDataset and PPIDataset datasets provide examples of feature encodings, below we will adjust the transform and target_transform objects to simply return NumPy NDArray objects as a default. Note: you can’t pass Structure objects to torch DataLoader, you must first convert them into array-like structures supported by the default torch collate_fn. You can implement your own collate functions to adjust this behavior.
def default_transform(structure: Structure) -> NDArray[np.double]:
return structure.coords
test_data = CustomPinderDataset(split="test", transform=default_transform, target_transform=default_transform)
test_data[0]
(array([[-12.621098 , -9.128866 , 17.258345 ],
[-13.660538 , -8.840156 , 16.26575 ],
[-13.38884 , -7.5286083, 15.5240555],
...,
[ 7.175483 , -19.776093 , 21.191608 ],
[ 9.592422 , -19.601938 , 19.197937 ],
[ 9.432584 , -17.757698 , 20.458265 ]], dtype=float32),
array([[-13.215324 , -11.076902 , 15.214827 ],
[-14.133494 , -10.301851 , 14.38616 ],
[-13.614427 , -8.882866 , 14.150835 ],
...,
[ 6.8049655, -15.96424 , 20.159506 ],
[ 9.54528 , -16.992254 , 18.400843 ],
[ 7.021378 , -15.329907 , 18.152586 ]], dtype=float32))
Implementing diversity sampling#
Here, we provide an example of how one might use torch.utils.data.WeightedRandomSampler. However, users are free to sample diversity any way they see fit. For this example, we are going to sample diversity inversely proportional to pinder cluster population size.
from torch.utils.data import WeightedRandomSampler
def inverse_cluster_size_sampler(dataset: PinderDataset, replacement: bool = True):
index = dataset.index
cluster_counts = (
index["cluster_id"].value_counts().rename("cluster_count")
)
index = index.merge(
cluster_counts, left_on="cluster_id", right_index=True
)
# undersample large clusters
cluster_weights = 1.0 / torch.tensor(index.cluster_count.values)
return WeightedRandomSampler(
weights=cluster_weights,
num_samples=len(
cluster_counts
),
replacement=replacement,
)
sampler = inverse_cluster_size_sampler(
test_data,
replacement=True,
)
sampler
<torch.utils.data.sampler.WeightedRandomSampler at 0x7fe7431c0d60>
Defining the dataloader#
Now that we have implemented a dataset and sampling function, we can tie everything together to implement the DataLoader.
from torch.utils.data import DataLoader
test_dataloader = DataLoader(
test_data,
batch_size=1,
# Mutually exclusive with sampler
shuffle=False,
sampler=sampler,
)
test_features, test_labels = next(iter(test_dataloader))
test_features, test_labels
2024-09-13 20:41:27,159 | pinder.core.utils.cloud:375 | INFO : Gsutil process_many=download_to_filename, threads=4, items=7
2024-09-13 20:41:27,404 | pinder.core.utils.cloud.process_many:23 | INFO : runtime succeeded: 0.25s
(tensor([[[-31.1970, 1.8040, -4.0020],
[-31.2980, 1.2740, -2.5900],
[-31.2110, 2.3760, -1.4720],
...,
[-23.9580, 0.2910, -7.9400],
[-25.0780, -1.9930, -7.8570],
[-24.8920, 0.9820, -7.0100]]]),
tensor([[[ 36.9863, -11.0252, 80.2713],
[ 37.7090, -9.8566, 80.9012],
[ 37.0190, -8.4626, 80.6714],
...,
[ 38.5636, 21.5941, 84.1964],
[ 36.0663, 21.6029, 84.6874],
[ 38.8361, 20.2134, 84.6802]]]))
Putting it all together, we can now get a train/val/test dataloader as such:
from typing import Any, Callable
def get_loader(
dataset: CustomPinderDataset,
sampler: torch.utils.data.Sampler | None = inverse_cluster_size_sampler,
batch_size: int = 2,
# shuffle is mutually exclusive with sampler
shuffle: bool = False,
num_workers: int = 0,
collate_fn: Callable[[list[tuple[NDArray[np.double], NDArray[np.double]]]], tuple[torch.Tensor, torch.Tensor]] | None = None,
**kwargs: Any,
) -> "DataLoader[CustomPinderDataset]":
return DataLoader(
dataset,
batch_size=batch_size,
shuffle=shuffle,
num_workers=num_workers,
sampler=sampler,
collate_fn=collate_fn,
**kwargs,
)
train_data = CustomPinderDataset(
split="train",
structure_filters=[filters.MinAtomTypesFilter()],
metadata_filter="(buried_sasa >= 500)",
transform=default_transform,
target_transform=default_transform,
)
train_dataloader = get_loader(
train_data,
sampler=inverse_cluster_size_sampler(train_data),
batch_size=1,
)
train_dataloader
<torch.utils.data.dataloader.DataLoader at 0x7fe7432fe0b0>
train_features, train_labels = next(iter(train_dataloader))
train_features, train_labels
2024-09-13 20:41:31,118 | pinder.core.utils.cloud:375 | INFO : Gsutil process_many=download_to_filename, threads=4, items=7
2024-09-13 20:41:31,335 | pinder.core.utils.cloud.process_many:23 | INFO : runtime succeeded: 0.22s
(tensor([[[416.1500, 261.4880, 421.8900],
[417.0670, 262.2200, 422.7540],
[417.2630, 263.6420, 422.2500],
...,
[462.5360, 215.6140, 453.5950],
[463.9440, 216.0950, 453.6580],
[461.5490, 213.9400, 448.6200]]]),
tensor([[[416.1500, 261.4880, 421.8900],
[417.0670, 262.2200, 422.7540],
[417.2630, 263.6420, 422.2500],
...,
[462.5360, 215.6140, 453.5950],
[463.9440, 216.0950, 453.6580],
[461.5490, 213.9400, 448.6200]]]))
Using a larger batch size (collate_fn)#
What if we want to use a larger batch size?
By default, the collate_fn used by the DataLoader is torch.utils.data._utils.collate.default_collate which expects to be able to stack tensors via torch.stack.
Since different pinder systems have a differing number of atomic coordinates, using a batch size greater than 1 will cause this function to raise a RuntimeError with a message like:
RuntimeError: stack expects each tensor to be equal size, but got [688, 3] at entry 0 and [1391, 3] at entry 1
We can leverage existing pinder utilities to adapt our default collate_fn to pad tensor dimensions with dummy values so that they can be stacked.
from pinder.core.loader.dataset import pad_and_stack
help(pad_and_stack)
Help on function pad_and_stack in module pinder.core.loader.dataset:
pad_and_stack(tensors: 'list[Tensor]', dim: 'int' = 0, dims_to_pad: 'list[int] | None' = None, value: 'int | float | None' = None) -> 'Tensor'
Pads a list of tensors to the maximum length observed along each dimension and then stacks them along a new dimension (given by `dim`).
Parameters:
tensors (list[Tensor]): A list of tensors to pad and stack
dim (int): The new dimension to stack along.
dims_to_pad (list[int] | None): The dimensions to pad
value (int | float | None, optional): The value to pad with, by default None
Returns:
Tensor: The padded and stacked tensor. Below are examples of input and output shapes
Example 1: Sequence features (although redundant with torch.rnn.utils.pad_sequence)
input: [(2,), (7,)], dim: 0
output: (2, 7)
Example 2: Pair features (e.g., pairwise coordinates)
input: [(4, 4, 3), (7, 7, 3)], dim: 0
output: (2, 7, 7, 3)
def collate_coordinates(batch, coords_pad_value: int = -100):
feature_coords = []
target_coords = []
for x in batch:
feat, target = x
if isinstance(feat, np.ndarray):
feat = torch.tensor(feat, dtype=torch.float32)
if isinstance(target, np.ndarray):
target = torch.tensor(target, dtype=torch.float32)
feature_coords.append(feat)
target_coords.append(target)
feature_coords = pad_and_stack(feature_coords, dim=0, value=coords_pad_value)
target_coords = pad_and_stack(target_coords, dim=0, value=coords_pad_value)
return feature_coords, target_coords
train_dataloader = get_loader(
train_data,
sampler=inverse_cluster_size_sampler(train_data),
collate_fn=collate_coordinates,
batch_size=2,
)
train_features, train_labels = next(iter(train_dataloader))
train_features, train_labels
2024-09-13 20:41:31,599 | pinder.core.utils.cloud:375 | INFO : Gsutil process_many=download_to_filename, threads=4, items=5
2024-09-13 20:41:31,759 | pinder.core.utils.cloud.process_many:23 | INFO : runtime succeeded: 0.16s
2024-09-13 20:41:31,871 | pinder.core.utils.cloud:375 | INFO : Gsutil process_many=download_to_filename, threads=4, items=7
2024-09-13 20:41:32,105 | pinder.core.utils.cloud.process_many:23 | INFO : runtime succeeded: 0.23s
(tensor([[[ 125.9140, 229.1700, 349.1470],
[ 127.1220, 229.5170, 349.8830],
[ 128.0540, 230.3360, 349.0100],
...,
[-100.0000, -100.0000, -100.0000],
[-100.0000, -100.0000, -100.0000],
[-100.0000, -100.0000, -100.0000]],
[[ 220.5350, 185.9060, -4.7080],
[ 221.1650, 184.9700, -3.7800],
[ 222.1990, 185.6760, -2.9080],
...,
[ 248.0230, 247.8970, 263.1650],
[ 246.9340, 250.9660, 263.6470],
[ 248.0100, 250.0080, 264.0270]]]),
tensor([[[ 125.9140, 229.1700, 349.1470],
[ 127.1220, 229.5170, 349.8830],
[ 128.0540, 230.3360, 349.0100],
...,
[-100.0000, -100.0000, -100.0000],
[-100.0000, -100.0000, -100.0000],
[-100.0000, -100.0000, -100.0000]],
[[ 220.5350, 185.9060, -4.7080],
[ 221.1650, 184.9700, -3.7800],
[ 222.1990, 185.6760, -2.9080],
...,
[ 248.0230, 247.8970, 263.1650],
[ 246.9340, 250.9660, 263.6470],
[ 248.0100, 250.0080, 264.0270]]]))