Shrinking the size of Oracle Virtual Box

First, zero fill your virtual disk. Boot the VM and run:

sudo dd if=/dev/zero of=/bigemptyfile bs=4096k status=progress
sudo rm -f /bigemptyfile

Shut down the VM and VirtualBox.

Then on the host run:

vboxmanage modifymedium x:\path\to\disk.vdi --compact

Note: You'll have to specify the path of vboxmanage (generally 'C:\Program Files\Oracle\VirtualBox').

ngrok - Time saver when you want to expose local host to internet

I was working with bitbucket pipelines and integrating with spinnaker webhooks. Since SPINNAKER does not have a direct interaction with bitbucket pipelines. I started using webhooks from spinnaker and able to trigger bitbucketpipeline from SPINNAKER by calling API end points.

    Now the real problem is I need to trigger the spinnaker pipeline from bitbucket. So Bitbucket should call the SPINNAKER API webhook end point. I setup everything in my local spinnaker and bitbucket server which is in cloud is not able to call the local end point . What I did is I setup a webhook in my bitbucket pipeline for each commit. Bitbucket trying to ping my local url and throw error like invalid uri. from few posts on Atlassian forums suggesting ngrok to expose localhost to internet (from security wise its dangerous if its not secured/ controlled) . 

    Then finally I tried ngrok and the options its providing is super cool. Its much simple to use for testing purposes. The download size also so less and the command is like 'ngrok.exe http portnumber' thats it. Its providing a simple dashboard to monitor the incoming requests (which is very good to know who is using our end points). also providing inspect functionality, status codes and able to view header and body of incoming and response . Also we can setit for time bound after that it dissolves itself. We can load the dashboard from localhost:4040

     

 

                                                     Its providing 2 end points one is http and the other is https to access local host from internet. for a guest user this session is limited to 2 hrs and you can able to see the remaining time displayed on the top. 

 

If you register for their services, they provide an account with auth token and dashboard where we can see all tunnels . Also we can override the 2 hrs limit. They also provide an API end point so that we can programmatically expose the end points and destroy them automatically after use which is super cool. 

 

Powershell - CURL error Ambiguous parameters in the command

 Powershell - CURL error Ambiguous parameters in the command :

I tried to invoke bitbucket pipelines from API using CURL command. I installed CURL first in powershell using chocolatey - Choco install curl . Then opened a new window and tried by prebuilt POST request in curl .

$ curl -X POST -is -u username:password `

  -H 'Content-Type: application/json' `

 https://api.bitbucket.org/2.0/repositories/test/testdemo/pipelines/ `

  -d '

  {

    "target": {

      "ref_type": "branch",

      "type": "pipeline_ref_target",

      "ref_name": "master"

    }

  }'

First I used line separator as "\" in end of each line which is shell format which is not recognized by powershell . Later I replaced "\" with back quote "`" character. 

After entering the command , Powershell throws error like "Parameter cannot be processed because the parameter name 'u' is ambiguous. Possible matches" 

Then I found that by default Powershell tried to call Invoke-Request alias instead of calling curl.exe. Stack overflow suggested to remove the alias and proceed. I removed the alias by executing the below command

Remove-item alias:curl

This did the magic and now my powershell is calling the bitbucket API to execute my pipeline.

Unable to find Root certificate Error in Fiddler

 Today, I tried to install Fiddler in one of the Virtual machine. Installation went well without any errors. When I tried to configure fiddler to decrypt https requests, it was not able to create the intercept certificates. 

I opened the fiddler logs tab on the right hand side and saw an error that getRootCertificate()call failed. Also in the initial lines of log, I found that Certmaker.dll is missing from the installation. I fixed the certmaker dll by copying the dll file from another VM where fiddler installed. This removes that line from the log but still not able to generate root certificate. Even the export the root certificate to Desktop failed. 

I tried opening fiddler as Administrator, disabled anti virus, looked at the certstore, but still getting the below error from fiddler

04:41:29:1927 !ERROR: Failed to generate Certificate using CertEnroll. System.Reflection.TargetInvocationException Exception has been thrown by the target of an invocation.

 < CertEnroll::CX509PrivateKey::Create: The requested operation cannot be completed. The computer must be trusted for delegation and the current user account must be configured to allow delegation. 0x80090345 (-2146892987 SEC_E_DELEGATION_REQUIRED)

Then after spending time on stack overflow and google the solution is with a registry key. Here is the fix :

Add the keyword "ProtectionPolicy" with DWord value as 1 in the below Location in the registry

[HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\Cryptography\Protect\Providers\df9d8cd0-1501-11d1-8c7a-00c04fc297eb]

Then I am able to trust the root certificate and https decryption worked well. 




Docker on Windows Failed to start

 All of a sudden, my docker on windows stopped working with an error  unexpected error occurred: Logon failure: the user has not been granted the requested logon type at this computer. (0x80070569).

)

I did couple of attempts and the error repeated. 

Finally followed this page and restarted the below services made the trick.

https://github.com/docker/for-win/issues/3542

Services restarted in services.msc:

1. Hyper-V Host Service

2. Windows Management Instrumentation Service. 

Google Lighthouse integration with Jenkins

 As part of Early Performance Testing, we decided to gather client side performance Metrics as well . Also the requirement to include this part of Jenkins CI/CD Process. 

I started installing Google lighthouse locally in node. When I ran against www.google.com, it ran like a charm. but the the real problem is to collect the stats for a page sitting beyond authentication. When started reading about that , google gave 2 options. 

1. Passing authentication cookie / header as extra headers while invoking the url

2. Using chrome in debug mode ( Manually we need to login) 

After discussion with architect, we took option 1 as strategical solution and option 2 as a Tactical solution. Our Jenkins running in windows machine and not able to launch chrome in debug mode via command prompt. 

Running Lighthouse locally : 

Here comes the issue to launch the chrome in debug mode and pass the remote debugging port to lighthouse commandline to run the test on pages behind authentication. As of now manual intervention is required to launch the browser and login to the application under test. But future need to automate this part using Sikuli or some other way. To launch chrome in debug mode, I installed chrome-debug in Npm and the same available in node modules

Then open a command prompt and go to the Node_Modules directory and run the chrome-debug. Once it opened chrome, it will output the port number in the command prompt itself.

Now run the lighthouse with remote port 60844

To run lighthouse with this open chrome session, open a command prompt and run the below command 

lighthouse  https://xyz.com/#/case --disable-storage-reset --port 60844 --output-path=./report.json --output json

Now the same chrome session will be highlighted in windows explorer and a new tab with the provided url will be opened. Once the report generation over, the newly opened tab will be closed automatically. 

The generated sample report will be like below:

Now lets see how to run this from Jenkins:

Jenkins Integration with LightHouse:

To integrate Lighthouse with Jenkins, Install NPM and lighthouse with flag -g in the jenkins server. 

npm install -g lighthouse

Then have chrome-debug files in Node_modules directory as in image 1. 

Install LighthouseReport plugin to Jenkins from plugin manager. Then restart Jenkins and come back to our freestyle Project. 

Go to Build section and choose Execute Windows batch command and enter the below lines of code.

Then choose lighthouse report and only enter /report.json. 

Now save the changes . Jenkins configuration is ready. 

In the jenkins server, open a chrome-debug session and update the port number in Jenkins build code. then click on build now. 

In the console you can able to view below log.

Once the run completed you can see the lighthouse Report on the left hand side , once you click on this the generated report will be loaded. A sample report below:

Todo: 

1. Configuring lighthouse-batch to run with more number of url's at a time
2. Automating browser login via sikuli
3. Moving all these to docker
4. Passing authenticated header

String modification with Groovy - Jmeter

 Today, while I am solving a old script, I need to remove a comma at the end in a string which is feed to construct a dynamic url in a request. 

The whole string was constructed by a beanshell post processor in JMeter and I dont want to poke my nose there to debug the logic. The issue I found was the output string created by the beanshell script is not in correct JSON format. After analyzing that I found to trim a comma at the end. so Instead of adding my code at the end of the beanshell, I added as part of a preprocessor to the new request. 

I attempted to use StringUtils class from apache library, but found it was not working. Before and after text processing, the number of the characters remains the same. 

//StringUtils.substring(Final, 0, Final.length() - 2);

//StringUtils.chop(Final);

I spend an hour , but no luck. So I moved to a JSR223 preprocessor and choose Groovy as scripting language. I tried my luck with minus function and index function on the string and it worked successfully.

Then I attempted substring function like below and it worked fine . //Final.substring(0,2); so finally solved my issue by writing a substring in Groovy.

import java.lang.*;

import java.util.*;

import org.apache.jmeter.util.JMeterUtils;

import org.apache.commons.lang3.StringUtils;

log.info("Final Results before trimming comma is "+vars.get("FinalQuery"));

Final = vars.get("FinalQuery");

//log.info("last ch is "+Final[-1]);

//log.info("fi is "+Final.substring(0,Final.length()-1));

Final =Final.substring(0,Final.length()-1);

//log.info("len is "+Final.length());

//Final.substring(0,2);

//StringUtils.substring(Final, 0, Final.length() - 2);

//StringUtils.chop(Final);

log.info("Final Results after  trimming comma is "+Final.toString());

//log.info("len is "+Final.length());

vars.put("FinalQuery",Final.toString());

Editing Average Response time for JMeter - Grafana

 In this topic, I explain the modifications I did in the Average Response time graph for Grafana - Jmeter integration. If you configured the integration, the graph will be displayed like this>

Here the data points are partially visibile and mostly we may not be able to get the most of this chart. Especially, if your test is having more number of transactions, then this will looks glumpsy. To modify this we can tweak some parameters of the grafana chart as below. 

To edit the chart click on the drop down icon next to chart legend (in this case Transaction Response Times) . In the menu, click on Edit .

Now you can get the edit screen so that we can customize. Note: If you dont have the edit rights in grafana, you may not be getting the menu. 

Look at the query in the query window (If you use influx db as a middle layer) . 

In case if you are not getting this query window you can click on the highlighted button to get text query editor. If you want to skip some transactions to be displayed in this graph, you can slightly edit the query. 

SELECT mean("pct90.0") FROM "$measurement_name" WHERE ("application" =~ /^$application$/) AND $timeFilter GROUP BY "transaction", time($__interval) fill(null)

You can add addtional filter in the query as below:

SELECT mean("pct90.0") FROM "$measurement_name" WHERE ("application" =~ /^$application$/) AND $timeFilter AND "transaction" !~ /XYZ*/ GROUP BY "transaction", time($__interval) fill(null)

In the place of XYZ you can put your regex or starting letters of the transactions to be hidden. If you want to show only certain transactions you can modify the query accordingly. 

By default, a transation named "all" will be displayed in the chart. you can hide that as well by tweaking like this.

SELECT mean("pct90.0") FROM "$measurement_name" WHERE ("application" =~ /^$application$/) AND $timeFilter AND "transaction" !~ /XYZ*/ AND  "transaction" = 'all' GROUP BY "transaction", time($__interval) fill(null)

Please note the single quotes and double quotes to avoid mistakes. In case if you want to remove the transaction response times of failed transactions you can set your query like this:

SELECT mean("pct90.0") FROM "$measurement_name" WHERE ("application" =~ /^$application$/) AND $timeFilter AND "transaction" !~ /XYZ*/ AND  "transaction" = 'all' AND "statut" = 'ok' GROUP BY "transaction", time($__interval) fill(null)

Now look at the graph visual settings. On the same screen, right hand side we can see three tabs for editing ( Panel, Field and Overrides) 

In the Panel - Visualization used to change the chart type but not right now .

Go to Panel - Display tab and look at the Area fill option. If you don't want to shade the chart lines change Area = 0. 

Another important option is how to display null values in the chart. Default is null which will show the lines as disconnected and leave more dots in the chart. Change the value as Connected which will connect the data points to make it smoother to view. 

After adjusting Area & null value display the chart will look like this:

Now lets adjust the Hover Tool tip. This is the one when we move the mouse cursor over the chart and by default it displays all transaction names and its response time graphs. Our aim is to highlight only the pointed line and its transaction name and detail instead of all . To do this, look for option called Hover Tool Tip in the display section and change to Single as below:

Dual Axis Graphs (Overlay Graphs) :

Let us see how we can create a overlay graph like Loadrunner analysis in Grafana. For example, I need to overlay Vusers (Active Threads in terms of JMeter) on Error count graph. You can pick up the Errors graph from the default Json and click on Edit as we shown above. In the query section, add another query to the graph by clicking on the +Query button. In the query editor enter the below query to plot Vusers on top of Errors graph. 

SELECT last("maxAT") FROM "$measurement_name" WHERE ("transaction" = 'internal' AND "application" =~ /^$application$/) AND $timeFilter GROUP BY time($__interval) fill(null)

Now you can see the Vusers (Active threads) line on top of the Errors graph. Some times default area shading will hide the errors. Lets go to the Panel --> Display and change Area Fill to zero. 

In the Legends section, turn on Average and current values for Errors for better clarity. 

To tune it better , lets change the graph to dual axis graph. One axis for Errors and the other for Active Threads. we need to click on the color line just before the  graph legend. I highlighted the area to click for better clarity. 

if we click on  the color line, we can get the axis tab and enable the Y-Axis:

Now we can configure the right Y-axis properties in the  Panel -->Axis section. Provide a label name, and if required change the units. The graph will look likes below:

By the same way we can overlay Vusers on top of Response time / hits per second graphs. 

Increasing JMeter JVM Memory

 This article applies for JMeter version 5.3.

By default, JVM memory size will be 1GB alloted to Jmeter which will not suffice if you run a test with more than 100 Vusers. Also if you have not increased the JVM Memory size you might be running the test with headless mode which will be difficult to monitor unless you setup Grafana / Blazemeter/ . 

To increse the JVM size you need to alter 2 files. Go to Jmeter Bin folder and open JMeter.bat file in a text editor of your choice.

Add these lines before get standard environment variables

set HEAP=-Xms512m -Xmx6g

set JVM_ARGS=-Xms512m -Xmx6g

Then open Jmeter.sh file

# add the Java9 args before the user given ones

JVM_ARGS="$JAVA9_OPTS $JVM_ARGS"

JVM_ARGS="-Xms1g -Xmx6g"

Now save both the files and the JVM Memory size increased to 6GB. 

Verification in Karate

Verification in Karate:

In this post, I am going to discuss about various level of verification we are going to apply in Karate..

Once we are able to successfully make the request (GET/POST/DELETE/...) now we need to validate the response . Response validation will go into three stages:
1. Response Status code validation (http)
2. Response Schema validation
3. Response value validations.

Karate has these abilities for all these validations with more in built functions. we can see them one by one.

1. Response Status code validation (HTTP)