Filtering objects from a workdir

Learning python, how to filter


Usually, if you work with in data science field, you need to create several objects, like list, DataFrame, matrix, etc. Besides, you have to handle some files to read and make copies. For that and other actions, sometime you have to select some objects from your workdir.

The goal in this post is to show some options about how to filter objects from your workdir.

Listing and Filtering your objects

There are several options to list the objects from a workdir. We will use the packages os and glob.

First of all, load the packages:

import os
import glob

You can check your workdir:

# check workdir

or se your workdir:

# set your workdir

If you want see all files in your workdir:

>>> os.listdir()
['CORR_TEST_fm940glt.png', 'ET_Fev_Maio_Agosto_OK.xlsm', 'T_sup_Fev_Maio_Agosto_OK.xlsm', 'H_final_Fev_Maio_Agosto_OK.xlsm', '', '', 'Saldo_24h_Fev_Maio_Agosto_OK.xlsm', 'CORR_TEST_tmg43ws.png', 'Saldo_inst_Fev_Maio_Agosto_ok.xlsm', 'Fluxo_calor_solo_Fev_Maio_Agosto_OK.xlsm', 'Calor_latente_Fev_Maio_Agosto_ok.xlsm', '', 'NDVI_Fev_Maio_Agosto_OK.xlsm', 'Albedo_Fev_Maio_Agosto_OK.xlsm', '', 'IAF_Fev_Maio_Agosto_OK.xlsm']

If you want all xlsm files in your workdir, one option is:

>>> glob.glob('*.xlsm')
['ET_Fev_Maio_Agosto_OK.xlsm', 'T_sup_Fev_Maio_Agosto_OK.xlsm', 'H_final_Fev_Maio_Agosto_OK.xlsm', 'Saldo_24h_Fev_Maio_Agosto_OK.xlsm', 'Saldo_inst_Fev_Maio_Agosto_ok.xlsm', 'Fluxo_calor_solo_Fev_Maio_Agosto_OK.xlsm', 'Calor_latente_Fev_Maio_Agosto_ok.xlsm', 'NDVI_Fev_Maio_Agosto_OK.xlsm', 'Albedo_Fev_Maio_Agosto_OK.xlsm', 'IAF_Fev_Maio_Agosto_OK.xlsm']

Let’s suppose that you need to filter all .xlsm files that has T_ in the name:

>>> res = list(filter(lambda k: 'T_' in k, lst_xlsm))
>>> print(res)
['ET_Fev_Maio_Agosto_OK.xlsm', 'T_sup_Fev_Maio_Agosto_OK.xlsm']

If, just in case, you want to filter all .xlsm files that has NO T_ in the name (note the not):

lst_xlsm = glob.glob('*.xlsm')
>>> res = list(filter(lambda k: not 'T_' in k, lst_xlsm))
>>> print(res)
['H_final_Fev_Maio_Agosto_OK.xlsm', 'Saldo_24h_Fev_Maio_Agosto_OK.xlsm', 'Saldo_inst_Fev_Maio_Agosto_ok.xlsm', 'Fluxo_calor_solo_Fev_Maio_Agosto_OK.xlsm', 'Calor_latente_Fev_Maio_Agosto_ok.xlsm', 'NDVI_Fev_Maio_Agosto_OK.xlsm', 'Albedo_Fev_Maio_Agosto_OK.xlsm', 'IAF_Fev_Maio_Agosto_OK.xlsm']

Another usual case in data science a list of objects to filter. If you use dir() you get all your environment content:

>>> dir()
['__PYTHON_EL_native_completion_setup', '__annotations__', '__builtins__', '__code', '__doc__', '__loader__', '__name__', '__package__', '__pyfile', '__spec__', 'codecs', 'df_ET', 'df_ETaug', 'df_ETfev', 'df_ETmay', 'df_H', 'df_HEATFLOW', 'df_HEATFLOWaug', 'df_HEATFLOWfev', 'df_HEATFLOWmay', 'df_Haug', 'df_Hfev', 'df_Hmay', 'df_IAF', 'df_IAFaug', 'df_IAFfev', 'df_IAFmay', 'df_LE', 'df_LEaug', 'df_LEfev', 'df_LEmay', 'df_RN24', 'df_RN24aug', 'df_RN24fev', 'df_RN24may', 'df_RNins', 'df_RNinsaug', 'df_RNinsfev', 'df_RNinsmay', 'df_TsupK', 'df_TsupKaug', 'df_TsupKfev', 'df_TsupKmay', 'df_albaug', 'df_albedo', 'df_albfev', 'df_albmay', 'df_fulldata1807G', 'df_fullfm940glt', 'df_fullfm940glt_pivot', 'df_fulltmg43ws', 'df_fulltmg43ws_pivot', 'glob', 'lst_xlsm', 'np', 'os', 'pd', 'res']

If you need to get all objects with df_ in the begin:

>>> lst_ob = dir()
>>> res = [k for k in lst_ob if 'df_' in k]
>>> print(res)
['df_ET', 'df_ETaug', 'df_ETfev', 'df_ETmay', 'df_H', 'df_HEATFLOW', 'df_HEATFLOWaug', 'df_HEATFLOWfev', 'df_HEATFLOWmay', 'df_Haug', 'df_Hfev', 'df_Hmay', 'df_IAF', 'df_IAFaug', 'df_IAFfev', 'df_IAFmay', 'df_LE', 'df_LEaug', 'df_LEfev', 'df_LEmay', 'df_RN24', 'df_RN24aug', 'df_RN24fev', 'df_RN24may', 'df_RNins', 'df_RNinsaug', 'df_RNinsfev', 'df_RNinsmay', 'df_TsupK', 'df_TsupKaug', 'df_TsupKfev', 'df_TsupKmay', 'df_albaug', 'df_albedo', 'df_albfev', 'df_albmay', 'df_fulldata1807G', 'df_fullfm940glt', 'df_fullfm940glt_pivot', 'df_fulltmg43ws', 'df_fulltmg43ws_pivot']


If you keep organized the files names and the objects names you can filter them easily. We only showed some options to filter lists. You can create more ways to achieve the same result. Let me know you know another one.

Best regards






comments powered by Disqus