Redis user permissions (ACL) and python connection

-
 Recently, I was working on setting up a redis cluster along with separate users for readwrite and readonly. I setup the readonly user using the below privileges (ACLs):   user redisreadonly on >mySuperSecretPassword ~* resetchannels -@all +@read +ping +asking   Once this is setup, I wrote a simple program in python to connect to the redis cluster using the readonly credentials and print the number of keys.   #!/usr/bin/env python3 """ pip3 install redis """ from redis.cluster import RedisCluster, ClusterNode # ─────── Cluster connection ─────── startup_nodes = [ ClusterNode("redis_host_1", 6379), ClusterNode("redis_host_2", 6379), ClusterNode("redis_host_3", 6379) ] rc = RedisCluster(startup_nodes=startup_nodes,decode_responses=True,username='redisreadonly',password='mySuperSecretPassw...
Post a Comment

Elasticsearch: Get output in CSV format using curl and jq

-

 

If you've used elasticsearch, you'll know that each call to it has to be through REST APIs, either using Kibana, curl, postman or similar tools.

If you are like me, then curl is the way to go!

However, that does pose one problem - how does one select specific fields, akin to select statements of SQL/RDBMS DBs?

The answer is using jq, the lightweight and flexible command-line JSON processor.

 This is my use-case:

 I have an elasticsearch cluster in which daily snapshots are scheduled and I would like to get the output of snapshot name, snapshot repo, name of the SLM policy etc., in a comma separated row format so that I can directly use it to push that data to my backup catalog in PostgreSQL. So, I query "_snapshot/<snapshot_repo>/backup_name_start_string*" and the output is quite long and convoluted (the more the backups, the longer the output)!

Here, I present the way to shorten the output as well as to fetch it in row based CSV format, easier to understand and that can be used directly in insert statements of postgreSQL.

 Raw Output:

$ curl --silent --insecure -X GET --header "Authorization: ApiKey ${MY_ES_API_KEY}" "https://10.1.2.3:9200/_snapshot/my_repo/snapshot_starting_word*?sort=start_time&pretty"
{
  "snapshots" : [
    {
      "snapshot" : "my_snapshot_24-06-2025_06_01_15-3gjkpyrwqtuqevaz8bjg_w",
      "uuid" : "UpuAXk2RS7yaxQfZ8jtinQ",
      "repository" : "my_repo",
      "version_id" : 8525000,
      "version" : "8.18.0-8.18.3",
      "indices" : [
        ".ds-.kibana-event-log-ds-2025.06.13-000005",
        ".kibana-siem-rule-migrations-prebuiltrules",
        ".ds-.logs-deprecation.elasticsearch-default-2025.08.15-000002",
        ".kibana_security_session_1",
        ".ds-.slm-history-7-2025.06.19-000002",
        "my_main_es_index",
        ".apm-agent-configuration",
        ".ds-ilm-history-7-2025.06.22-000004",
        ".slo-observability.sli-v3.3",
        ".kibana_ingest_8.17.4_001",
        ".kibana_analytics_8.17.4_001",
        ".slo-observability.summary-v3.3",
        ".transform-internal-007",
        ".slo-observability.summary-v3.4",
        ".internal.alerts-observability.metrics.alerts-default-000001",
        ".monitoring-es-7-2025.06.22",
        ".transform-notifications-000002",
        ".kibana_task_manager_8.17.4_001",
        ".slo-observability.summary-v3.3.temp",
        ".tasks",
        ".ds-.logs-deprecation.elasticsearch-default-2025.06.16-000001",
        ".monitoring-kibana-7-2025.06.18",
        ".internal.alerts-observability.threshold.alerts-default-000001",
        ".ds-.kibana-event-log-ds-2025.06.23-000002",
        ".internal.alerts-ml.anomaly-detection-health.alerts-default-000001",
        ".internal.alerts-stack.alerts-default-000001",
        ".monitoring-es-7-2025.06.24",
        ".monitoring-kibana-7-2025.06.24",
        ".monitoring-es-7-2025.06.20",
        ".ds-.slm-history-7-2025.06.12-000001",
        ".monitoring-kibana-7-2025.06.20",
        ".internal.alerts-observability.slo.alerts-default-000001",
        ".apm-source-map",
        ".security-profile-8",
        ".apm-custom-link",
        ".internal.alerts-transform.health.alerts-default-000001",
        ".internal.alerts-observability.uptime.alerts-default-000001",
        ".monitoring-kibana-7-2025.06.19",
        ".kibana_alerting_cases_8.17.4_001",
        ".monitoring-kibana-7-2025.06.21",
        ".slo-observability.summary-v3.4.temp",
        ".ds-.kibana-event-log-ds-2025.06.06-000004",
        ".kibana_entities-definitions-1",
        ".internal.alerts-ml.anomaly-detection.alerts-default-000001",
        ".security-7",
        ".kibana-siem-rule-migrations-integrations",
        ".ds-.kibana-event-log-ds-2025.06.16-000001",
        ".kibana_usage_counters_8.17.4_001",
        ".internal.alerts-observability.apm.alerts-default-000001",
        ".ds-ilm-history-7-2025.06.23-000002",
        ".ds-.kibana-event-log-ds-2025.06.30-000003",
        ".monitoring-es-7-2025.06.21",
        ".ds-.kibana-event-log-ds-2025.06.20-000006",
        ".monitoring-kibana-7-2025.06.22",
        ".kibana_8.17.4_001",
        ".monitoring-es-7-2025.06.18",
        ".slo-observability.sli-v3.4",
        ".internal.alerts-security.alerts-default-000001",
        ".monitoring-es-7-2025.08.19",
        ".ds-ilm-history-7-2025.08.15-000003",
        ".ds-ilm-history-7-2025.07.16-000001",
        ".kibana_security_solution_8.17.4_001",
        ".geoip_databases",
        ".internal.alerts-observability.logs.alerts-default-000001",
        ".internal.alerts-default.alerts-default-000002",
        ".monitoring-es-7-2025.08.23",
        ".internal.alerts-default.alerts-default-000001",
        ".monitoring-kibana-7-2025.08.23"
      ],
      "data_streams" : [
        ".kibana-event-log-ds",
        ".slm-history-7",
        "ilm-history-7",
        ".logs-deprecation.elasticsearch-default"
      ],
      "include_global_state" : true,
      "metadata" : {
        "policy" : "my_snapshot_policy"
      },
      "state" : "SUCCESS",
      "start_time" : "2025-06-24T06:01:14.852Z",
      "start_time_in_millis" : 1750744874000,
      "end_time" : "2025-06-24T06:01:22.257Z",
      "end_time_in_millis" : 1750744882000,
      "duration_in_millis" : 7405,
      "failures" : [ ],
      "shards" : {
        "total" : 71,
        "failed" : 0,
        "successful" : 71
      },
      "feature_states" : [
        {
          "feature_name" : "geoip",
          "indices" : [
            ".geoip_databases"
          ]
        },
        {
          "feature_name" : "security",
          "indices" : [
            ".security-7",
            ".security-profile-8"
          ]
        },
        {
          "feature_name" : "transform",
          "indices" : [
            ".transform-internal-007"
          ]
        },
        {
          "feature_name" : "tasks",
          "indices" : [
            ".tasks"
          ]
        },
        {
          "feature_name" : "kibana",
          "indices" : [
            ".kibana_ingest_8.17.4_001",
            ".kibana_alerting_cases_8.17.4_001",
            ".kibana_8.17.4_001",
            ".apm-custom-link",
            ".kibana_security_solution_8.17.4_001",
            ".apm-agent-configuration",
            ".kibana_security_session_1",
            ".kibana_task_manager_8.17.4_001",
            ".kibana_usage_counters_8.17.4_001",
            ".kibana_entities-definitions-1",
            ".kibana_analytics_8.17.4_001"
          ]
        }
      ]
    },
    ...
    ...
    ...
    ...
  "total" : 9,
  "remaining" : 0
}

 

Expected Output: 

"snapshot_name","snapshot_repo","slm_policy_name","snapshot_status","snapshot_start_time","snapshot_start_time_in_millis","snapshot_end_time","snapshot_end_time_in_millis","snapshot_duration_in_millis","total_shards","successful_shards","failed_shards","total_snapshots"
"my_snapshot_24-06-2025_06_01_15-3gjkpyrwqtuqevaz8bjg_w","my_repo","my_slm_policy_name","SUCCESS","2025-06-24T06:01:14.852Z",1750744874000,"2025-06-24T06:01:22.257Z",1750744882000,7405,71,71,0,9
"my_snapshot_25-06-2025_06_01_15-3gjkpyrwqjueqvaz8zje_m","my_repo","my_slm_policy_name","SUCCESS","2025-06-25T06:01:14.852Z",1750831282000,"2025-06-24T06:02:21.257Z",1750831281000,7405,71,71,0,9
..
..

 

Solution - using JQ: 

curl --silent --insecure -X GET --header "Authorization: ApiKey ${MY_ES_API_KEY}" "https://10.1.2.3:9200/_snapshot/my_repo/<snapshot_starting_word>*?sort=start_time&pretty" | jq -r '. | .total as $t | (["snapshot_name", "snapshot_repo", "slm_policy_name", "snapshot_status", "snapshot_start_time", "snapshot_start_time_in_millis", "snapshot_end_time", "snapshot_end_time_in_millis", "snapshot_duration_in_millis", "total_shards", "successful_shards", "failed_shards", "total_snapshots"] | @csv), ((.snapshots[] | [.snapshot, .repository, .metadata.policy, .state,.start_time,.start_time_in_millis, .end_time, .end_time_in_millis, .duration_in_millis, .shards.total, .shards.successful, .shards.failed, $t]) | @csv)'

 

Explanation: 

The below piece of code variabalizes (converts to a variable) the total number of snapshots (shown above in the output as "total:"). So, this variable ($t) can be used down the line along with others. 


jq -r '. | .total as $t

 

Post that, the header row gets printed as a separate row first. NOTE: Any field/set of fields that need to be printed as CSV in jq, needs to be converted to an array first. Hence, the inclusion of the square braces [] 


jq -r '. | .total as $t

 

Post that, "snapshots" is converted to an array and printed and its output is then redirected to the next stage. In the next stage, the relevant fields are gleaned directly or from within nested directory objects or arrays (as the case may be) and printed as an array (i.e., enclosed within []) and then converted to CSV (using @csv) 


(.snapshots[] | [.snapshot, .repository, .metadata.policy, .state,.start_time,.start_time_in_millis, .end_time, .end_time_in_millis, .duration_in_millis, .shards.total, .shards.successful, .shards.failed, $t]) | @csv'


Comments

Post a Comment

Popular posts from this blog

Interesting Oracle Applications (EBS) Interview Questions

-
Below are some interesting interview questions that I have faced over the years. What is the difference between a concurrent manager and a cron job? Can one patch a CRS Oracle home with the CRS services up? Suppose, there are 2 nodes node1 and node2 with CRS cluster and one wants to patch node1. Can it be done with node2 being up? Suppose one fires a select statement and the data does not in the library cache, then the data, obviously, has to be fetched from the data files. Which background process is used for this purpose? What are the sizing requirements for say, a 100 concurrent users Oracle Apps Installation? What is the difference between a shared appl_top and a staged appl_top?
Read more

Modify retention period of workflow queues

-
Yesterday, there was a requirement to increase the retention period of WF_DEFERRED and WF_BPEL_Q so that the developers could troubleshoot issues involving business events. This can be done this way: 1. Check the retention period of existing workqueues. SQL> SELECT owner, name, retention FROM all_queues WHERE name LIKE 'WF%'; OWNER           NAME                                                RETENTION --------------- --------------------------------------------- ------------------------------ APPS                 WF_BPEL_Q                                                0 APPLSYS         WF_INBOUND_QUEUE              ...
Read more

Check if UTL_FILE and FND_FILE are working fine

-
Yesterday, while troubleshooting a vexing ORA-29280 error, I came across 2 useful PL/SQL anonymous blocks that can be used to test FND_FILE and UTL_FILE to find out whether they are indeed working fine. UTL_FILE set serveroutput on DECLARE file_location VARCHAR2(256) := '<first entry on utl_file_dir>'; file_name VARCHAR2(256) := 'utlfile1.lst'; file_text VARCHAR2(256) := 'THIS IS A TEST'; file_id UTL_FILE.file_type; BEGIN file_id := UTL_FILE.fopen(file_Location, file_name, 'W'); UTL_FILE.put_line(file_id, file_text); UTL_FILE.fclose(file_id); EXCEPTION WHEN UTL_FILE.INVALID_PATH THEN dbms_output.put_line('Invalid path ' || SQLERRM); WHEN OTHERS THEN dbms_output.put_line('Others '|| SQLCODE || ' ' || SQLERRM); END; / References: Metalink (MOS) Note ID: 261693.1 FND_FILE set serveroutput on exec FND_FILE.PUT_LINE(FND_FILE.LOG, 'THIS IS A TEST'); HTH....
Read more