RobotFramework Keyword: wget - 下載網頁上的資料

網站大都有提供下載功能，驗證下載功能與內容是否正常是必要的測試項目之一。然而，SeleniumLibrary並沒有提供類似的keyword。本篇文章分享我們解決這個問題的歷程。

起初我們透過外部程式去達到測試需求。要達成此目的，有幾個步驟:

1. 測試環境可以執行wget。
2. 取得下載位置的連結。
3. 設定Cookie。
4. 執行下載。
5. 驗證內容。

大部分linux安裝起來就有wget，如果沒有就自行安裝；而windows可以從這裡下載執行檔，將它放到你的測試環境中。

在這裡我使用的是Get Element Attribute去取得hyper link的內容:

Get Element Attribute | //a[@id='link_id']@href

假如你的網站是需要登入才能夠操作，這部分就是必要的。首先可以透過SeleniumLibrary提供的Get Cookies來取得當前session內容；它所拿到的內容是使用name1=value1; name2=value2;形式組成，只要挑你需要的屬性出來即可。對我們而言，JSESSIONID記載我們的登入資訊。

接著是要組成wget所需要的cookie file格式，詳細格式可以參考此連結，大致上如下:

${domain_ip}   FALSE   ${path}   FALSE   9999999999   JSESSIONID   ${jsession_id}

假如你是使用localhost連線，${domain_ip}就是localhost，否則就是要連線的機器IP。

假如參數連結位置為${fileUrl}而下載檔案路徑${fileName}，wget的執行指令如下:

${wget} --cookies=on --load-cookies ${cookie_file} --output-document ${fileName} ${fileUrl}

如果下載的位置為https，為了略過Certificate驗證，會搭配–no-check-certificate參數:

${wget} --cookies=on --no-check-certificate --load-cookies ${cookie_file} --output-document ${fileName} ${fileUrl}

這部分就是看各位的業務需求去做assertion。我的Keyword提供給大家參考:

Download File Wit Wget
    [Arguments]    ${fileUrl}    ${fileName}
    Log    ${fileUrl}
    ${cookies}    Get Cookies
    @{jessonid}    Split String    ${cookies}    JSESSIONID=
    ${location}    Get Location IP
    ${cookies}    Set Variable    ${location}\tFALSE\t//\tFALSE\t0\tJSESSIONID\t@{jessonid}[1]\r\n localhost\tFALSE\t//\tFALSE\t0\tJSESSIONID\t@{jessonid}[1]
    Log    ${cookies}
    Create File    ./cookies    ${cookies}
    ${currentdir}    Get Suite Directory
    ${isLinux}    Is Linux
    ${wget}    Set Variable If    '${isLinux}'!='TRUE'    ${currentdir}${/}..${/}commonResource${/}wget${/}wget    '${isLinux}'!='FALSE'    wget
    @{output}    Run And Return Rc And Output    ${wget} --cookies=on --no-check-certificate --load-cookies cookies --output-document ${fileName} \ ${fileUrl}
    Should Be Equal As Integers    @{output}[0]    0    @{output}[1]

由於我們將預設連線修改為https，並且只允許TLS1.2，這導致在舊版本的Linux如RHEL6.0上無法透過預設的wget進行測試。因此改為自己實作TestLibrary去提供wget的功能。我所使用的HttpClient版本為httpclient-4.3.5，我直接將程式分享給各位，這可能會根據你的需求或library版本需要做些微調:

Robot Keyword:

Download File
    [Arguments]    ${fileUrl}    ${fileName}
    Log    ${fileUrl}
    ${cookies}    Get Cookies
    @{jessonid}    Split String    ${cookies}    JSESSIONID=
    ${location}    Get Location IP
    ${ret}    wget    ${fileUrl}    ${fileName}    JSESSIONID=@{jessonid}[1]    ${location}
    log    ${ret}

Test Library WgetLibrary:

import java.io.File;
import java.io.FileOutputStream;
 
import javax.net.ssl.SSLContext;
 
import org.apache.commons.io.IOUtils;
import org.apache.http.client.methods.CloseableHttpResponse;
import org.apache.http.client.methods.HttpGet;
import org.apache.http.conn.ssl.SSLConnectionSocketFactory;
import org.apache.http.conn.ssl.SSLContextBuilder;
import org.apache.http.conn.ssl.TrustSelfSignedStrategy;
import org.apache.http.impl.client.BasicCookieStore;
import org.apache.http.impl.client.CloseableHttpClient;
import org.apache.http.impl.client.HttpClientBuilder;
import org.apache.http.impl.cookie.BasicClientCookie;
 
public class WgetLibrary {	
	 public static final String ROBOT_LIBRARY_SCOPE = "TEST CASE";
	 public static final String ROBOT_LIBRARY_VERSION = "1.0.0";
 
	private BasicCookieStore createCookieStore(String cookies, String domain, String path){
		if( cookies == null || cookies.isEmpty() )
			return null;
 
		BasicCookieStore cookieStore = new BasicCookieStore();
		String[] cookiePairs = cookies.split(";");
		for( String cookiePair : cookiePairs){
			String[] cookieToken = cookiePair.split("=");
			BasicClientCookie cookie = new BasicClientCookie(cookieToken[0], cookieToken[1]);
			cookie.setDomain(domain);
			cookie.setPath(path);
			cookieStore.addCookie(cookie);
		}
		return cookieStore;
	}
 
	public long wget(String url, String destFile, String cookies, String domain){
		return wget(url, destFile, cookies, domain, "/");
	}
 
	public long wget(String url, String destFile, String cookies, String domain, String path){
		CloseableHttpClient httpClient = null;
		FileOutputStream fos = null;
		CloseableHttpResponse response = null;
		try {
			HttpClientBuilder builder = HttpClientBuilder.create();
 
			SSLContext sslContext = new SSLContextBuilder().loadTrustMaterial(null, new TrustSelfSignedStrategy()).build();
			SSLConnectionSocketFactory sslSocketFactory = new SSLConnectionSocketFactory(sslContext, SSLConnectionSocketFactory.ALLOW_ALL_HOSTNAME_VERIFIER);
			builder.setSSLSocketFactory(sslSocketFactory);
 
			BasicCookieStore cookieStore = createCookieStore(cookies, domain, path);
			if( cookieStore != null ) {
				builder.setDefaultCookieStore(cookieStore);
			}
 
			httpClient = builder.build();
			response = httpClient.execute(new HttpGet(url));
			fos = new FileOutputStream(new File(destFile));
			IOUtils.copy(response.getEntity().getContent(), fos);
			return response.getStatusLine().getStatusCode();
		} catch( Exception e ){
			throw new RuntimeException(e);
		} finally {
			Cleaner.close(fos);
			Cleaner.close(response);
			Cleaner.close(httpClient);
		}
	}
}

• Format of cookies when using wget?
• How to Ignore Certificate Errors in Apache HttpClient 4.5
• javadoc - NoopHostnameVerifier
• javadoc - TrustSelfSignedStrategy
• RobotFramework - SeleniumLibrary
• HttpClient 4 – Send Custom Cookie

---