top of page
BlogPageTop

Processing Time Calculator

Green Card Calculator

How to Pull Data from Oracle IDCS (Identity Cloud Services) Rest API

Oracle IDCS Rest API

Oracle IDCS has various rest APIs that can be used to pull data and you can utilize it further for data analytics. Let's see how we can pull data using simple shell scripts.


Table of Contents: Oracle IDCS Rest API


This Bash script automates the retrieval and processing of audit events data from an Oracle IDCS Rest API endpoint. It begins by checking for the existence of a parameter file, "param.txt", and sets environment variables accordingly. Using these variables, it generates a basic authentication token and validates its integrity. The script then retrieves the total number of qualified records from the API endpoint and proceeds to fetch the audit events data in paginated batches. Each batch is modified to a specific format and saved into JSON files with timestamps. Finally, it iterates through each batch until all records are retrieved and clean up temporary files upon completion, providing a seamless and efficient workflow for managing audit events.



Parameter File

Step 1: Create a parameter file "param.txt" which will contain the Customer ID, Customer secret, and organization URL (all in new line). You can leave the environment name as it is. Please note below values are just dummy values to showcase how you need to create a param file. Validate using Postman if your keys are working properly before running the script.


scripts> pwd
/home/hadoop/scripts
scripts> more param.txt
CID=
61rgrjk5869bjrvrb9999rbre20
CSEC=
01rgt-atbt-4956-9e77-15rjb74756nr64
ORG=
https://idcs-9bbrtj756bjer8gbk753gbvj8f7eh3.identity.oraclecloud.com
ENV=
QUAL

Main Script

Step 2: At the same location, create the shell script for pulling data in JSON format. A brief description is given before each section of code. You can name your shell script anything; just make sure the permissions are correct to execute. I kept it as 755.



Check if the Parameter File Exists

This script checks for the presence of a parameter file named "param.txt" and reads specific lines from it to determine the environment. Based on the environment, it provides a corresponding welcome message or indicates an invalid selection.


#!/bin/bash

[ -f ./param.txt ] && echo "Parameter file is present" || echo "Parameter file not found!! Create param.txt with CID, CSEC, ORG, and ENV details."

ENV=`head -8 ./param.txt | tail -1`

[ -z "$ENV" ] && echo "Environment variable is empty" || echo "Environment variable looks good"

case $ENV in
        DEV)  echo "Welcome to DEV environment!" ;;
        QUAL) echo "Welcome to QUAL environment!"

        CID=`head -2 ./param.txt | tail -1`
        CSEC=`head -4 ./param.txt | tail -1`
        ORG=`head -6 ./param.txt | tail -1`
        sleep 1;;

        PL)   echo "Welcome to ProdLike environment!" ;;
        PT)   echo "Welcome to ProdTest environment!" ;;
        PROD) echo "Welcome to PROD environment!" ;;

        *)    echo "Invalid environment selection!" ;;
esac

Create Basic Token

Generate the base64, basic_token at https://www.base64encode.org/. This line of code takes the values stored in $CID and $CSEC, combines them into a single string separated by a colon, encodes that string into Base64 format, and assigns the resulting encoded string to the variable basic_token.


basic_token=`echo -n $CID:$CSEC | base64 -w 0`


Function to Regenerate the Token

This function sends a request to obtain a new access token using the OAuth 2.0 client credentials flow, processes the response to extract the access token, and stores it in a file named access_token.tmp.


regenToken()
{
curl -X POST \
"$ORG/oauth2/v1/token" \
  -H "Authorization: Basic $basic_token" \
  -H "Cache-Control: no-cache" \
  -H "Content-Type: application/x-www-form-urlencoded" \
  -d "grant_type=client_credentials&scope=urn%3Aopc%3Aidm%3A__myscopes__" | awk -F[":"] '{print$2}' | awk -F[","] '{print$1}' | awk '{print substr($0, 2, length($0) - 2)}' > access_token.tmp

echo "New token is generated.. access_token refreshed!!"
}


Testing Token Validity

This portion of the main shell script checks the validity of an access token. It first reads the token from a file and then sends a request to validate it. If the token is invalid, it regenerates a new one and repeats the process. Finally, it updates the token file accordingly.


access_token=`more access_token.tmp`.

tokenTest=`curl -X POST   "$ORG/oauth2/v1/introspect"   -H "Authorization: Basic $basic_token"   -H "Cache-Control: no-cache"   -H "Content-Type: application/x-www-form-urlencoded"   -d token=$access_token | awk -F"," '{print$1}' | awk -F":" '{print$2}' | sed 's/[{}]//g'` 

if [ "$tokenTest" = "true" ]; then echo "Token is valid..";

else
  echo "Invalid token! Invoking func to pull new token.."
  regenToken
  access_token=`more access_token.tmp`
fi


Remove all the previous files. The script can be modified later to pull delta records only.

rm -f auditevents.idcs*

Pull totalResults Count

This script retrieves the total number of qualified records from an API endpoint "$ORG/admin/v1/AuditEvents" using a GET request. It includes an access token in the request headers for authorization. The response is processed using awk to extract the value associated with "totalResults". After obtaining the total number of records, it echoes this information and then waits for 5 seconds.


totalResults=`curl -X GET "$ORG/admin/v1/AuditEvents?&count=0" -H "Authorization: Bearer $access_token" -H "Cache-Control: no-cache" | awk -F"\"totalResults\"\:" '{print$2}' | awk -F"," '{print$1}'`

echo "Total number of qualified records: $totalResults"
sleep 5



Loop to Pull the Records

This script iterates through paginated API calls to retrieve audit events data from "$ORG/admin/v1/AuditEvents". It sets the pagination parameters and continuously fetches data until all records are obtained. Each batch of data is processed and saved into a JSON file named "auditevents.idcs.json".


itemsPerPage=1000
startIndex=1

while [ $startIndex -le $totalResults ]
do
echo "startIndex: $startIndex"

curl -X GET \
"$ORG/admin/v1/AuditEvents?&startIndex=$startIndex&count=$itemsPerPage" \
-H "Authorization: Bearer $access_token" \
-H "Cache-Control: no-cache" | awk -F"Resources" '{print$2}' | awk -F"startIndex" '{print$1}' | cut -c 4- | rev | cut -c 4- | rev > auditevents.idcs.json


Formatting the JSON Output

This script performs a search and replaces operation on a JSON file named "auditevents.idcs.json". It replaces occurrences of the string defined by the variable PAT with the string defined by REP_PAT. The modified content is then redirected to a new file with a timestamp appended to its name. Afterward, it increments the startIndex by 1000 to prepare for the next batch of data retrieval in the loop. This process repeats until all records are retrieved.


PAT=]},{\"idcsCreatedBy
REP_PAT=]}'\n'{\"idcsCreatedBy

sed "s/$PAT/$REP_PAT/g" auditevents.idcs.json  > auditevents.idcs.json_`date +"%Y%m%d_%H%M%S%N"`

startIndex=`expr $startIndex + 1000`
done

Remove the access token temp file at the end of the script:

rm -f access_token.tmp


Summary

In short, the shell script performs the following operations:


  • It checks for the presence of a parameter file named "param.txt" and sets environment variables accordingly.

  • It generates a basic authentication token.

  • It validates the token and regenerates it if invalid.

  • It retrieves the total number of qualified records and waits for 5 seconds.

  • It retrieves audit events data in paginated batches, modifies the data format, and saves it into JSON files with timestamps.

  • It iterates through each batch until all records are retrieved.

  • Finally, it cleans up temporary files


```bash
#!/bin/bash

[ -f ./param.txt ] && echo "Parameter file is present" || echo "Parameter file not found!! Create param.txt with CID, CSEC, ORG, and ENV details."

ENV=$(head -8 ./param.txt | tail -1)

[ -z "$ENV" ] && echo "Environment variable is empty" || echo "Environment variable looks good"

case $ENV in
    DEV)
        echo "Welcome to DEV environment!"
        ;;
    QUAL)
        echo "Welcome to QUAL environment!"
        CID=$(head -2 ./param.txt | tail -1)
        CSEC=$(head -4 ./param.txt | tail -1)
        ORG=$(head -6 ./param.txt | tail -1)
        sleep 1
        ;;
    PL)
        echo "Welcome to ProdLike environment!"
        ;;
    PT)
        echo "Welcome to ProdTest environment!"
        ;;
    PROD)
        echo "Welcome to PROD environment!"
        ;;
    *)
        echo "Invalid environment selection!"
        ;;
esac

basic_token=$(echo -n $CID:$CSEC | base64 -w 0)

regenToken() {
    curl -X POST \
    "$ORG/oauth2/v1/token" \
    -H "Authorization: Basic $basic_token" \
    -H "Cache-Control: no-cache" \
    -H "Content-Type: application/x-www-form-urlencoded" \
    -d "grant_type=client_credentials&scope=urn%3Aopc%3Aidm%3A__myscopes__" | awk -F[":"] '{print$2}' | awk -F[","] '{print$1}' | awk '{print substr($0, 2, length($0) - 2)}' > access_token.tmp

    echo "New token is generated.. access_token refreshed!!"
}

access_token=$(more access_token.tmp).

tokenTest=$(curl -X POST "$ORG/oauth2/v1/introspect" -H "Authorization: Basic $basic_token" -H "Cache-Control: no-cache" -H "Content-Type: application/x-www-form-urlencoded" -d token=$access_token | awk -F"," '{print$1}' | awk -F":" '{print$2}' | sed 's/[{}]//g')

if [ "$tokenTest" = "true" ]; then
    echo "Token is valid.."
else
    echo "Invalid token! Invoking func to pull new token.."
    regenToken
    access_token=$(more access_token.tmp)
fi

rm -f auditevents.idcs*

totalResults=$(curl -X GET "$ORG/admin/v1/AuditEvents?&count=0" -H "Authorization: Bearer $access_token" -H "Cache-Control: no-cache" | awk -F"\"totalResults\"\:" '{print$2}' | awk -F"," '{print$1}')

echo "Total number of qualified records: $totalResults"
sleep 5

itemsPerPage=1000
startIndex=1

while [ $startIndex -le $totalResults ]; do
    echo "startIndex: $startIndex"

    curl -X GET \
    "$ORG/admin/v1/AuditEvents?&startIndex=$startIndex&count=$itemsPerPage" \
    -H "Authorization: Bearer $access_token" \
    -H "Cache-Control: no-cache" | awk -F"Resources" '{print$2}' | awk -F"startIndex" '{print$1}' | cut -c 4- | rev | cut -c 4- | rev > auditevents.idcs.json

    PAT=]},{\"idcsCreatedBy
    REP_PAT=]}'\n'{\"idcsCreatedBy

    sed "s/$PAT/$REP_PAT/g" auditevents.idcs.json > auditevents.idcs.json_$(date +"%Y%m%d_%H%M%S%N")

    startIndex=$(expr $startIndex + 1000)
done

rm -f access_token.tmp
``` 

I've formatted the script for better readability while preserving its functionality.


Related Posts

Related Topics

Comments

Want to share your thoughts about this blog?

Disclaimer: Please note that the information provided on this website is for general informational purposes only and should not be taken as legal advice. Dataneb is a platform for individuals to share their personal experiences with visa and immigration processes, and their views and opinions may not necessarily reflect those of the website owners or administrators. 

 

While we strive to keep the information up-to-date and accurate, we make no representations or warranties of any kind, express or implied, about the completeness, accuracy, reliability, suitability, or availability with respect to the website or the information, products, services, or related graphics contained on the website for any purpose. Any reliance you place on such information is therefore strictly at your own risk. 

 

We strongly advise that you consult with a qualified immigration attorney or official government agencies for any specific questions or concerns related to your individual situation. We are not responsible for any losses, damages, or legal disputes arising from the use of information provided on this website. 

 

By using this website, you acknowledge and agree to the above disclaimer and Google's Terms of Use and Privacy Policy.

bottom of page